-
Notifications
You must be signed in to change notification settings - Fork 0
/
local-search.xml
1327 lines (636 loc) · 734 KB
/
local-search.xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
<?xml version="1.0" encoding="utf-8"?>
<search>
<entry>
<title>ES6 Promise用法 asyncawait异步处理同步化</title>
<link href="/2024/04/03/ES6%20Promise%E7%94%A8%E6%B3%95%20asyncawait%E5%BC%82%E6%AD%A5%E5%A4%84%E7%90%86%E5%90%8C%E6%AD%A5%E5%8C%96/"/>
<url>/2024/04/03/ES6%20Promise%E7%94%A8%E6%B3%95%20asyncawait%E5%BC%82%E6%AD%A5%E5%A4%84%E7%90%86%E5%90%8C%E6%AD%A5%E5%8C%96/</url>
<content type="html"><![CDATA[<span id="more"></span><h3 id="promise定义"><a href="#promise定义" class="headerlink" title="promise定义"></a>promise定义</h3><blockquote><p>promise是解决异步的方法，本质上是一个构造函数，可以用它实例化一个对象。对象身上有resolve、reject、all，原型上有then、catch方法。promise对象有三种状态：pending（初识状态/进行中）、resolved或fulfilled（成功）、rejected（失败）</p></blockquote><ul><li><strong>pending</strong> 表示”待定的，将发生的”，相当于是一个初始状态。创建Promise对象时，且没有调用resolve或者是reject方法，相当于是初始状态。这个初始状态会随着你调用resolve，或者是reject函数而切换到另一种状态。</li><li><strong>resolved</strong> 表示解决了，就是说这个承诺实现了。 要实现从pending到resolved的转变，需要在 创建Promise对象时，在函数体中调用了resolve方法。</li><li><strong>rejected</strong> 表示”拒绝，失败“。表示这个承诺没有做到，失败了。要实现从pending到rejected的转换，只需要在创建Promise对象时，调用reject函数。</li></ul><h3 id="以uniapp中请求为例"><a href="#以uniapp中请求为例" class="headerlink" title="以uniapp中请求为例"></a>以uniapp中请求为例</h3><ul><li><strong>uni.request</strong> 多重嵌套的请求</li></ul><figure class="highlight js"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br></pre></td><td class="code"><pre><code class="hljs js"><span class="hljs-title function_">getData</span>(<span class="hljs-params"></span>){<br> <span class="hljs-comment">//获取分类列表id</span><br> uni.<span class="hljs-title function_">request</span>({<br> <span class="hljs-attr">url</span>:<span class="hljs-string">"https://ku.qingnian8.com/dataApi/news/navlist.php"</span>,<br> <span class="hljs-attr">success</span>:<span class="hljs-function"><span class="hljs-params">res</span>=></span>{<br> <span class="hljs-keyword">let</span> id=res.<span class="hljs-property">data</span>[<span class="hljs-number">0</span>].<span class="hljs-property">id</span><br> <span class="hljs-comment">// 根据分类id获取该分类下的所有文章</span><br> uni.<span class="hljs-title function_">request</span>({<br> <span class="hljs-attr">url</span>:<span class="hljs-string">"https://ku.qingnian8.com/dataApi/news/newslist.php"</span>,<br> <span class="hljs-attr">data</span>:{<br> <span class="hljs-attr">cid</span>:id<br> },<br> <span class="hljs-attr">success</span>:<span class="hljs-function"><span class="hljs-params">res2</span>=></span>{<br> <span class="hljs-comment">//获取到一篇文章的id，根据文章id找到该文章下的评论</span><br> <span class="hljs-keyword">let</span> id=res2.<span class="hljs-property">data</span>[<span class="hljs-number">0</span>].<span class="hljs-property">id</span>;<br> uni.<span class="hljs-title function_">request</span>({<br> <span class="hljs-attr">url</span>:<span class="hljs-string">"https://ku.qingnian8.com/dataApi/news/comment.php"</span>,<br> <span class="hljs-attr">data</span>:{<br> <span class="hljs-attr">aid</span>:id<br> },<br> <span class="hljs-attr">success</span>:<span class="hljs-function"><span class="hljs-params">res3</span>=></span>{<br> <span class="hljs-comment">//找到该文章下所有的评论</span><br> <span class="hljs-variable language_">console</span>.<span class="hljs-title function_">log</span>(res3)<br> }<br> })<br> }<br> })<br> }<br> })<br>}<br></code></pre></td></tr></table></figure><ul><li>封装成<strong>promise</strong>对象</li></ul><figure class="highlight js"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br></pre></td><td class="code"><pre><code class="hljs js"><span class="hljs-attr">methods</span>: {<br> <span class="hljs-comment">//先获取导航分类接口，将结果进行返回，到调用函数的地方获取</span><br> <span class="hljs-title function_">getNav</span>(<span class="hljs-params">callback</span>){<br> <span class="hljs-keyword">return</span> <span class="hljs-keyword">new</span> <span class="hljs-title class_">Promise</span>(<span class="hljs-function">(<span class="hljs-params">resolve,reject</span>)=></span>{<br> uni.<span class="hljs-title function_">request</span>({<br> <span class="hljs-attr">url</span>:<span class="hljs-string">"https://ku.qingnian8.com/dataApi/news/navlist.php"</span>,<br> <span class="hljs-attr">success</span>:<span class="hljs-function"><span class="hljs-params">res</span>=></span>{<br> <span class="hljs-title function_">resolve</span>(res)<br> },<br> <span class="hljs-attr">fail</span>:<span class="hljs-function"><span class="hljs-params">err</span>=></span>{<br> <span class="hljs-title function_">reject</span>(err)<br> }<br> })<br> })<br> },<br><br><br> <span class="hljs-comment">//获取文章数据，将文章列表进行返回</span><br> <span class="hljs-title function_">getArticle</span>(<span class="hljs-params">id</span>){<br> <span class="hljs-keyword">return</span> <span class="hljs-keyword">new</span> <span class="hljs-title class_">Promise</span>(<span class="hljs-function">(<span class="hljs-params">resolve,reject</span>)=></span>{<br> uni.<span class="hljs-title function_">request</span>({<br> <span class="hljs-attr">url</span>:<span class="hljs-string">"https://ku.qingnian8.com/dataApi/news/newslist.php"</span>,<br> <span class="hljs-attr">data</span>:{<br> <span class="hljs-attr">cid</span>:id<br> },<br> <span class="hljs-attr">success</span>:<span class="hljs-function"><span class="hljs-params">res</span>=></span>{<br> <span class="hljs-title function_">resolve</span>(res)<br> },<br> <span class="hljs-attr">fail</span>:<span class="hljs-function"><span class="hljs-params">err</span>=></span>{<br> <span class="hljs-title function_">reject</span>(err)<br> }<br> })<br> })<br> },<br><br> <span class="hljs-comment">//获取文章下的所有评论</span><br> <span class="hljs-title function_">getComment</span>(<span class="hljs-params">id</span>){<br> <span class="hljs-keyword">return</span> <span class="hljs-keyword">new</span> <span class="hljs-title class_">Promise</span>(<span class="hljs-function">(<span class="hljs-params">resolve,reject</span>)=></span>{<br> uni.<span class="hljs-title function_">request</span>({<br> <span class="hljs-attr">url</span>:<span class="hljs-string">"https://ku.qingnian8.com/dataApi/news/comment.php"</span>,<br> <span class="hljs-attr">data</span>:{<br> <span class="hljs-attr">aid</span>:id<br> },<br> <span class="hljs-attr">success</span>:<span class="hljs-function"><span class="hljs-params">res</span>=></span>{<br> <span class="hljs-title function_">resolve</span>(res)<br> },<br> <span class="hljs-attr">fail</span>:<span class="hljs-function"><span class="hljs-params">err</span>=></span>{<br> <span class="hljs-title function_">reject</span>(err)<br> }<br> })<br> })<br>} <br></code></pre></td></tr></table></figure><ul><li><strong>promise</strong> 用 <strong>then</strong> 方式的链式调用</li></ul><figure class="highlight js"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><code class="hljs js"><span class="hljs-comment">//promise链式调用</span><br><span class="hljs-variable language_">this</span>.<span class="hljs-title function_">getNav</span>().<span class="hljs-title function_">then</span>(<span class="hljs-function"><span class="hljs-params">res</span>=></span>{<br> <span class="hljs-keyword">let</span> id=res.<span class="hljs-property">data</span>[<span class="hljs-number">0</span>].<span class="hljs-property">id</span>;<br> <span class="hljs-keyword">return</span> <span class="hljs-variable language_">this</span>.<span class="hljs-title function_">getArticle</span>(id);<br>}).<span class="hljs-title function_">then</span>(<span class="hljs-function"><span class="hljs-params">res</span>=></span>{<br> <span class="hljs-keyword">let</span> id=res.<span class="hljs-property">data</span>[<span class="hljs-number">0</span>].<span class="hljs-property">id</span>;<br> <span class="hljs-keyword">return</span> <span class="hljs-variable language_">this</span>.<span class="hljs-title function_">getComment</span>(id)<br>}).<span class="hljs-title function_">then</span>(<span class="hljs-function"><span class="hljs-params">res</span>=></span>{<br> <span class="hljs-variable language_">console</span>.<span class="hljs-title function_">log</span>(res)<br>})<br></code></pre></td></tr></table></figure><ul><li><strong>await / async</strong> 异步处理同步化</li></ul><blockquote><p>onload是函数，这个函数必须有async命令，在调用函数的部分，前面都加了一个await，这个命令的意思就是等这一行的异步方法执行成功后，将返回的值赋值给res变量，然后才能再走下一行代码，这就是将原来的异步编程改为了同步编程，这就是标题提到的“异步处理，同步化” </p></blockquote><figure class="highlight js"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><code class="hljs js"><span class="hljs-keyword">async</span> <span class="hljs-title function_">onLoad</span>(<span class="hljs-params"></span>) {<br> <span class="hljs-keyword">let</span> id,res;<br> res=<span class="hljs-keyword">await</span> <span class="hljs-variable language_">this</span>.<span class="hljs-title function_">getNav</span>();<br> id=res.<span class="hljs-property">data</span>[<span class="hljs-number">0</span>].<span class="hljs-property">id</span>;<br> res=<span class="hljs-keyword">await</span> <span class="hljs-variable language_">this</span>.<span class="hljs-title function_">getArticle</span>(id);<br> id=res.<span class="hljs-property">data</span>[<span class="hljs-number">0</span>].<span class="hljs-property">id</span>;<br> res=<span class="hljs-keyword">await</span> <span class="hljs-variable language_">this</span>.<span class="hljs-title function_">getComment</span>(id);<br> <span class="hljs-variable language_">console</span>.<span class="hljs-title function_">log</span>(res)<br>}<br></code></pre></td></tr></table></figure><blockquote><p>[!TIP]</p><p><a href="https://www.bilibili.com/read/cv18799030/">ES6 Promise的用法，async/await异步处理同步化 文章</a></p><p><a href="https://www.bilibili.com/video/BV1XW4y1v7Md/">ES6 Promise的用法，ES7 async/await异步处理同步化，异步处理进化史 视频</a></p></blockquote>]]></content>
<categories>
<category>前端</category>
</categories>
<tags>
<tag>es6</tag>
</tags>
</entry>
<entry>
<title>es6中for循环遍历各种使用场景</title>
<link href="/2024/04/03/es6%E4%B8%ADfor%E5%BE%AA%E7%8E%AF%E9%81%8D%E5%8E%86%E5%90%84%E7%A7%8D%E4%BD%BF%E7%94%A8%E5%9C%BA%E6%99%AF/"/>
<url>/2024/04/03/es6%E4%B8%ADfor%E5%BE%AA%E7%8E%AF%E9%81%8D%E5%8E%86%E5%90%84%E7%A7%8D%E4%BD%BF%E7%94%A8%E5%9C%BA%E6%99%AF/</url>
<content type="html"><![CDATA[<span id="more"></span><h4 id="for循环的各种使用场景"><a href="#for循环的各种使用场景" class="headerlink" title="for循环的各种使用场景"></a>for循环的各种使用场景</h4><ul><li><p><strong>forEach</strong></p><ul><li>性能上小于for，for循环直接操作索引，没有额外的函数调用</li><li>for可以使用break终止，forEach不支持跳出循环，使用return时类似于for的continue，结束当次循环</li><li>forEach不支持await异步等待</li></ul></li><li><p><strong>map</strong></p><ul><li><p>对数组遍历不破坏原数组，将会创建一个新数组，按照原始数组元素顺序依次执行给定的函数，map方法非常适合用于处理数组中的每个元素并生成新的数组</p></li><li><p><code>map(callbackFn)</code> callbackFn包含参数(element 数组中当前正在处理的元素, index 索引, array 数组本身)</p></li></ul><figure class="highlight xquery"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><code class="hljs xquery"><span class="hljs-keyword">let</span> arrs = [<span class="hljs-built_in">{name</span>:<span class="hljs-string">"华为"</span>,price:<span class="hljs-number">6999</span>},<span class="hljs-built_in">{name</span>:<span class="hljs-string">"苹果"</span>,price:<span class="hljs-number">9888</span>},<span class="hljs-built_in">{name</span>:<span class="hljs-string">"小米"</span>,price:<span class="hljs-number">4999</span>}]<br><span class="hljs-keyword">let</span> newArrs = arrs.<span class="hljs-keyword">map</span>(<span class="hljs-type">item</span>=>{<br><span class="hljs-keyword">return</span> {<br>...<span class="hljs-type">item</span>,<br>price:item.price+<span class="hljs-string">"元"</span>,<br><span class="hljs-built_in">number</span>:<span class="hljs-number">888</span><br>}<br>});<br></code></pre></td></tr></table></figure></li><li><p><strong>filter</strong></p><ul><li><p>过滤方法，会对原数组中的每个元素应用指定的函数，并返回一个新数组，其中包含符合条件的元素。原数组不会受到影响。</p></li><li><p>在filter回调函数中，满足true即可被处理到新函数中，false不做处理。</p></li><li><p><code>filter(callbackFn)</code> callbackFn包含参数(element 数组中当前正在处理的元素, index 索引, array 数组本身)</p></li></ul><figure class="highlight xquery"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><code class="hljs xquery"><span class="hljs-keyword">let</span> arrs = [<span class="hljs-number">5</span>,<span class="hljs-number">7</span>,<span class="hljs-number">8</span>,<span class="hljs-number">15</span>,<span class="hljs-number">22</span>,<span class="hljs-number">1</span>,<span class="hljs-number">2</span>];<br><span class="hljs-keyword">let</span> newArrs = arrs<span class="hljs-built_in">.filter</span>(<span class="hljs-type">item</span>=>{<br><span class="hljs-keyword">return</span> <span class="hljs-type">item</span>><span class="hljs-number">10</span><br>})<br></code></pre></td></tr></table></figure></li><li><p><strong>reduce</strong></p><ul><li><p>对数组中的每个元素按序执行一个指定方法，每一次运行 reducer 会将先前元素的计算结果作为参数传入。</p></li><li><p><code>reduce(callbackFn, initialValue)</code> callbackFn包含参数(prev上一次调用 callbackFn 的结果, current当前值, index索引-可选,array 数组本身-可选)，<strong>initialValue</strong> 第一次调用回调函数时初始值</p></li></ul><figure class="highlight pgsql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><code class="hljs pgsql">let arrs = [<span class="hljs-number">1</span>,<span class="hljs-number">2</span>,<span class="hljs-number">3</span>,<span class="hljs-number">4</span>];<br>let result = arrs.reduce((prev,<span class="hljs-keyword">current</span>,<span class="hljs-keyword">index</span>)=>{<br>console.log(prev,<span class="hljs-keyword">current</span>,<span class="hljs-keyword">index</span>);<br><span class="hljs-keyword">return</span> prev+<span class="hljs-keyword">current</span>;<br>},<span class="hljs-number">0</span>)<br></code></pre></td></tr></table></figure></li><li><p><strong>every</strong></p><ul><li><p>判断数组中所有元素是否满足函数中给定的条件，全部满足返回true,只要有一项不满足则返回false。</p></li><li><p><code>every(callbackFn)</code> callbackFn包含参数(element 数组中当前正在处理的元素, index 索引, array 数组本身)</p></li></ul><figure class="highlight maxima"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><code class="hljs maxima"><span class="hljs-built_in">let</span> arrs = [<span class="hljs-number">1</span>, <span class="hljs-number">2</span>, <span class="hljs-number">3</span>, <span class="hljs-number">4</span>, <span class="hljs-number">5</span>];<br>// 写成两行时记得要<span class="hljs-built_in">return</span><br><span class="hljs-built_in">let</span> result = arrs.<span class="hljs-built_in">every</span>(<span class="hljs-built_in">num</span> => <span class="hljs-built_in">num</span> > <span class="hljs-number">0</span>);<br></code></pre></td></tr></table></figure></li><li><p><strong>some</strong></p><ul><li><p>判断数组，只要有一个满足条件即返回true，全部不满足条件才会返回false。</p></li><li><p><code>some(callbackFn)</code> callbackFn包含参数(element 数组中当前正在处理的元素, index 索引, array 数组本身)</p></li></ul><figure class="highlight xquery"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><code class="hljs xquery"><span class="hljs-keyword">let</span> arrs = [<span class="hljs-number">55</span>,<span class="hljs-number">26</span>,<span class="hljs-number">3</span>,<span class="hljs-number">12</span>,<span class="hljs-number">39</span>];<br><span class="hljs-keyword">let</span> result = arrs.<span class="hljs-keyword">some</span>(<span class="hljs-type">item</span> => <span class="hljs-type">item</span> < <span class="hljs-number">10</span>);<br>console.log(result);<br></code></pre></td></tr></table></figure></li><li><p><strong>includes</strong></p><ul><li><p>判断一个数组是否包含一个指定的值，根据情况，如果包含则返回 true，否则返回 false。</p></li><li><p><code>includes(searchElement)</code> searchElement 需要查找的值。</p></li></ul><figure class="highlight arcade"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs arcade">const arrs = [<span class="hljs-number">1</span>, <span class="hljs-number">2</span>, <span class="hljs-number">3</span>];<br><span class="hljs-built_in">console</span>.<span class="hljs-built_in">log</span>(arrs.<span class="hljs-built_in">includes</span>(<span class="hljs-number">2</span>));<br></code></pre></td></tr></table></figure></li></ul><blockquote><p>[!TIP]<br>参考链接<br><a href="https://blog.csdn.net/qq_18798149/article/details/135089225">https://blog.csdn.net/qq_18798149/article/details/135089225</a></p></blockquote>]]></content>
<categories>
<category>前端</category>
</categories>
<tags>
<tag>es6 js</tag>
</tags>
</entry>
<entry>
<title>github上有趣的项目(值得star)</title>
<link href="/2024/03/28/github%E4%B8%8A%E6%9C%89%E8%B6%A3%E7%9A%84%E9%A1%B9%E7%9B%AE(%E5%80%BC%E5%BE%97star)/"/>
<url>/2024/03/28/github%E4%B8%8A%E6%9C%89%E8%B6%A3%E7%9A%84%E9%A1%B9%E7%9B%AE(%E5%80%BC%E5%BE%97star)/</url>
<content type="html"><![CDATA[<span id="more"></span><h3 id="类似-selenuium-的网页自动化工具"><a href="#类似-selenuium-的网页自动化工具" class="headerlink" title="类似 selenuium 的网页自动化工具"></a>类似 selenuium 的网页自动化工具</h3><blockquote><p>这是一款基于 Python 的网页自动化工具，支持 Chrome 和 Edge 等 Chromium 内核的浏览器。它将控制浏览器和收发请求两大功能合二为一，并提供了统一、简洁的接口，简单易用十分容易上手。该项目 v3.x 版本推出了 WebPage 摆脱对 selenium 的依赖，重新开发了底层逻辑，具有速度快、不易被网站识别、无需为不同版本浏览下载驱动等特点</p></blockquote><p><a href="https://github.com/g1879/DrissionPage">g1879/DrissionPage: 基于python的网页自动化工具。既能控制浏览器，也能收发数据包。可兼顾浏览器自动化的便利性和requests的高效率。功能强大，内置无数人性化设计和便捷功能。语法简洁而优雅，代码量少。 (github.com)</a></p><h3 id="实时直播和视频-AI-换脸程序"><a href="#实时直播和视频-AI-换脸程序" class="headerlink" title="实时直播和视频 AI 换脸程序"></a>实时直播和视频 AI 换脸程序</h3><blockquote><p>该项目可以对摄像头和本地视频文件中的人物，进行实时 AI 换脸，可用于 PC 直播、视频等场景。</p></blockquote><p><a href="https://github.com/iperov/DeepFaceLive">DeepFaceLive</a></p><h3 id="屏幕实时翻译工具"><a href="#屏幕实时翻译工具" class="headerlink" title="屏幕实时翻译工具"></a>屏幕实时翻译工具</h3><blockquote><p>免费开源的屏幕实时翻译工具。该项目可以对屏幕上选定区域内显示的文本进行实时翻译，可识别英语、俄语、中文等语言。用户可自行选择 Tesseract、WindowsOCR、EasyOCR 多种 OCR 引擎，以及包括谷歌翻译在内的多种翻译源。</p></blockquote><p><a href="https://github.com/Danily07/Translumo">Translumo</a></p><h3 id="将截图转化为代码"><a href="#将截图转化为代码" class="headerlink" title="将截图转化为代码"></a>将截图转化为代码</h3><blockquote><p>将截图转化为 HTML 代码的工具。该项目可以将屏幕截图转化为 HTML/JS/Tailwind CSS 代码，它使用 GPT-4 Vision 生成代码、DALL-E 3 生成相似的图片。</p></blockquote><p><a href="https://github.com/abi/screenshot-to-code">Screenshot-To-Code</a></p><h3 id="极简风格的-Vue-管理后台模板"><a href="#极简风格的-Vue-管理后台模板" class="headerlink" title="极简风格的 Vue 管理后台模板"></a>极简风格的 Vue 管理后台模板</h3><blockquote><p>一款极简风格的 Vue 管理后台。这是一个开源、免费、可商用的后台管理模板，基于 Vue3、Vite4、Pinia、Unocss 和 Naive UI 等前端最新技术栈。它简洁、轻量、风格清新，上手成本低，适合中小型项目或者个人项目。</p></blockquote><p><a href="https://github.com/zclzone/vue-naive-admin">Vue-Naive-Admin</a></p><h3 id="Redis-桌面客户端"><a href="#Redis-桌面客户端" class="headerlink" title="Redis 桌面客户端"></a>Redis 桌面客户端</h3><blockquote><p>一款轻量级的跨平台 Redis 桌面客户端。该项目是基于 WebView2 的 Redis 桌面客户端，拥有小巧的体积和精美的界面，同时支持中文。它提供了多种连接方式、分段加载、慢日志、转码显示等功能，可以在 Windows、Linux 和 macOS 系统上使用。</p></blockquote><p><a href="https://github.com/tiny-craft/tiny-rdm">Tiny-Rdm</a></p><h3 id="视频翻译和配音工具"><a href="#视频翻译和配音工具" class="headerlink" title="视频翻译和配音工具"></a>视频翻译和配音工具</h3><blockquote><p>开源的视频翻译和配音工具。该项目可以将视频从一种语言翻译成指定语言的视频，并自动生成和添加对应语言的字幕和配音。</p></blockquote><p><a href="https://github.com/jianchang512/pyvideotrans">Pyvideotrans</a></p>]]></content>
<categories>
<category>Git</category>
</categories>
<tags>
<tag>github git</tag>
</tags>
</entry>
<entry>
<title>feapder爬虫框架之轻量AirSpider用法示例</title>
<link href="/2024/03/27/feapder%E7%88%AC%E8%99%AB%E6%A1%86%E6%9E%B6%E4%B9%8B%E8%BD%BB%E9%87%8FAirSpider%E7%94%A8%E6%B3%95%E7%A4%BA%E4%BE%8B/"/>
<url>/2024/03/27/feapder%E7%88%AC%E8%99%AB%E6%A1%86%E6%9E%B6%E4%B9%8B%E8%BD%BB%E9%87%8FAirSpider%E7%94%A8%E6%B3%95%E7%A4%BA%E4%BE%8B/</url>
<content type="html"><![CDATA[<span id="more"></span><h2 id="安装"><a href="#安装" class="headerlink" title="安装"></a>安装</h2><p>精简版</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs shell">pip install feapder<br></code></pre></td></tr></table></figure><p>浏览器渲染版：</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs shell">pip install "feapder[render]"<br></code></pre></td></tr></table></figure><p>完整版：</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs shell">pip install "feapder[all]"<br></code></pre></td></tr></table></figure><p>三个版本区别：</p><ol><li>精简版：不支持浏览器渲染、不支持基于内存去重、不支持入库mongo</li><li>浏览器渲染版：不支持基于内存去重、不支持入库mongo</li><li>完整版：支持所有功能</li></ol><h2 id="使用"><a href="#使用" class="headerlink" title="使用"></a>使用</h2><blockquote><p>AirSpider是一款轻量爬虫，学习成本低。面对一些数据量较少，无需断点续爬，无需分布式采集的需求，可采用此爬虫</p></blockquote><p>创建模板命令：<code>feapder create -s air_spider_test</code> </p><p>请选择爬虫模板 <em>AirSpider</em> </p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-comment"># -*- coding: utf-8 -*-</span><br><span class="hljs-string">"""</span><br><span class="hljs-string">Created on 2024-03-26 11:57:14</span><br><span class="hljs-string">---------</span><br><span class="hljs-string">@summary:</span><br><span class="hljs-string">---------</span><br><span class="hljs-string">@author: zhangmingwei</span><br><span class="hljs-string">"""</span><br><br><span class="hljs-keyword">import</span> feapder<br><br><br><span class="hljs-keyword">class</span> <span class="hljs-title class_">AirSpiderTest</span>(feapder.AirSpider):<br> <span class="hljs-keyword">def</span> <span class="hljs-title function_">start_requests</span>(<span class="hljs-params">self</span>):<br> <span class="hljs-keyword">yield</span> feapder.Request(<span class="hljs-string">"https://spidertools.cn"</span>)<br><br> <span class="hljs-keyword">def</span> <span class="hljs-title function_">parse</span>(<span class="hljs-params">self, request, response</span>):<br> <span class="hljs-comment"># 提取网站title</span><br> <span class="hljs-built_in">print</span>(response.xpath(<span class="hljs-string">"//title/text()"</span>).extract_first())<br> <span class="hljs-comment"># 提取网站描述</span><br> <span class="hljs-built_in">print</span>(response.xpath(<span class="hljs-string">"//meta[@name='description']/@content"</span>).extract_first())<br> <span class="hljs-built_in">print</span>(<span class="hljs-string">"网站地址: "</span>, response.url)<br><br><br><span class="hljs-keyword">if</span> __name__ == <span class="hljs-string">"__main__"</span>:<br> AirSpiderTest().start()<br></code></pre></td></tr></table></figure>]]></content>
<categories>
<category>爬虫</category>
</categories>
<tags>
<tag>爬虫</tag>
</tags>
</entry>
<entry>
<title>feapder爬虫框架之任务TaskSpider用法示例</title>
<link href="/2024/03/27/feapder%E7%88%AC%E8%99%AB%E6%A1%86%E6%9E%B6%E4%B9%8B%E4%BB%BB%E5%8A%A1TaskSpider%E7%94%A8%E6%B3%95%E7%A4%BA%E4%BE%8B/"/>
<url>/2024/03/27/feapder%E7%88%AC%E8%99%AB%E6%A1%86%E6%9E%B6%E4%B9%8B%E4%BB%BB%E5%8A%A1TaskSpider%E7%94%A8%E6%B3%95%E7%A4%BA%E4%BE%8B/</url>
<content type="html"><![CDATA[<span id="more"></span><h2 id="安装"><a href="#安装" class="headerlink" title="安装"></a>安装</h2><p>精简版</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs shell">pip install feapder<br></code></pre></td></tr></table></figure><p>浏览器渲染版：</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs shell">pip install "feapder[render]"<br></code></pre></td></tr></table></figure><p>完整版：</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs shell">pip install "feapder[all]"<br></code></pre></td></tr></table></figure><p>三个版本区别：</p><ol><li>精简版：不支持浏览器渲染、不支持基于内存去重、不支持入库mongo</li><li>浏览器渲染版：不支持基于内存去重、不支持入库mongo</li><li>完整版：支持所有功能</li></ol><h2 id="使用"><a href="#使用" class="headerlink" title="使用"></a>使用</h2><blockquote><p>TaskSpider是一款分布式爬虫，内部封装了取种子任务的逻辑，内置支持从redis或者mysql获取任务，也可通过自定义实现从其他来源获取任务</p></blockquote><p>创建模板命令：<code>feapder create -s task_spider_test</code> </p><p>请选择爬虫模板 TaskSpider</p><figure class="highlight routeros"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br></pre></td><td class="code"><pre><code class="hljs routeros"><span class="hljs-comment"># -*- coding: utf-8 -*-</span><br><span class="hljs-string">""</span><span class="hljs-string">"</span><br><span class="hljs-string">Created on 2024-03-26 17:27:27</span><br><span class="hljs-string">---------</span><br><span class="hljs-string">@summary:</span><br><span class="hljs-string">---------</span><br><span class="hljs-string">@author: zhangmingwei01</span><br><span class="hljs-string">"</span><span class="hljs-string">""</span><br><br>import feapder<br><span class="hljs-keyword">from</span> feapder import ArgumentParser<br><span class="hljs-keyword">from</span> items.spider_data_item import SpiderDataItem<br><br><br>class SpiderTest(feapder.TaskSpider):<br> # 自定义数据库，若项目中有setting.py文件，此自定义可删除<br> __custom_setting__ = dict(<br> <span class="hljs-attribute">REDISDB_IP_PORTS</span>=<span class="hljs-string">"localhost:6379"</span>,<br> <span class="hljs-attribute">REDISDB_USER_PASS</span>=<span class="hljs-string">""</span>,<br> <span class="hljs-attribute">REDISDB_DB</span>=0,<br> <span class="hljs-attribute">MYSQL_IP</span>=<span class="hljs-string">"localhost"</span>,<br> <span class="hljs-attribute">MYSQL_PORT</span>=3306,<br> <span class="hljs-attribute">MYSQL_DB</span>=<span class="hljs-string">"data"</span>,<br> <span class="hljs-attribute">MYSQL_USER_NAME</span>=<span class="hljs-string">"root"</span>,<br> <span class="hljs-attribute">MYSQL_USER_PASS</span>=<span class="hljs-string">"pwd"</span>,<br> )<br><br> def add_task(self):<br> # 加种子任务 框架会调用这个函数，方便往redis里塞任务，但不能写成死循环。实际业务中可以自己写个脚本往redis里塞任务<br> self._redisdb.zadd(self._task_table, {<span class="hljs-string">"id"</span>: 1, <span class="hljs-string">"url"</span>: <span class="hljs-string">"https://www.baidu.com"</span>})<br><br> def start_requests(self, task):<br> task_id = task.id<br> url = task.url<br> yield feapder.Request(url, <span class="hljs-attribute">task_id</span>=task_id)<br><br> def parse(self, request, response):<br> # 提取网站title<br> title = response.xpath(<span class="hljs-string">"//title/text()"</span>).extract_first()<br> <span class="hljs-built_in">print</span>(<span class="hljs-string">"网站标题："</span>, title)<br> # 提取网站描述<br> <span class="hljs-built_in">print</span>(response.xpath(<span class="hljs-string">"//meta[@name='description']/@content"</span>).extract_first())<br> <span class="hljs-built_in">print</span>(<span class="hljs-string">"网站地址: "</span>, response.url)<br> item = SpiderDataItem(**{<br> <span class="hljs-string">"title"</span>: title,<br> <span class="hljs-string">"url"</span>: response.url,<br> })<br> yield item<br><br> # mysql 需要更新任务状态为做完 即 <span class="hljs-attribute">state</span>=1<br> # yield self.update_task_batch(request.task_id)<br><br><br><span class="hljs-keyword">if</span> __name__ == <span class="hljs-string">"__main__"</span>:<br> # 用mysql做任务表，需要先建好任务任务表<br> # spider = SpiderTest(<br> # <span class="hljs-attribute">redis_key</span>=<span class="hljs-string">"spider:test"</span>, # 分布式爬虫调度信息存储位置<br> # <span class="hljs-attribute">task_table</span>=<span class="hljs-string">"spider_test"</span>, # mysql中的任务表<br> # task_keys=[<span class="hljs-string">"id"</span>, <span class="hljs-string">"url"</span>], # 需要获取任务表里的字段名，可添加多个<br> # <span class="hljs-attribute">task_state</span>=<span class="hljs-string">"state"</span>, # mysql中任务状态字段<br> # )<br><br> # 用redis做任务表<br> spider = SpiderTest(<br> <span class="hljs-attribute">redis_key</span>=<span class="hljs-string">"spider:test2024"</span>, # 分布式爬虫调度信息存储位置<br> <span class="hljs-attribute">task_table</span>=<span class="hljs-string">"spider_test"</span>, # 任务表名<br> <span class="hljs-attribute">task_table_type</span>=<span class="hljs-string">"redis"</span>, # 任务表类型为redis<br> )<br><br> parser = ArgumentParser(<span class="hljs-attribute">description</span>=<span class="hljs-string">"SpiderTest爬虫"</span>)<br><br> parser.add_argument(<br> <span class="hljs-string">"--start_master"</span>,<br> <span class="hljs-attribute">action</span>=<span class="hljs-string">"store_true"</span>,<br> <span class="hljs-attribute">help</span>=<span class="hljs-string">"添加任务"</span>,<br> <span class="hljs-attribute">function</span>=spider.start_monitor_task,<br> )<br> parser.add_argument(<br> <span class="hljs-string">"--start_worker"</span>, <span class="hljs-attribute">action</span>=<span class="hljs-string">"store_true"</span>, <span class="hljs-attribute">help</span>=<span class="hljs-string">"启动爬虫"</span>, <span class="hljs-attribute">function</span>=spider.start<br> )<br><br> parser.start()<br><br> # 直接启动<br> spider.start() # 启动爬虫<br> spider.start_monitor_task() # 添加任务<br><br> # 通过命令行启动<br> # python spider_test.py --start_master # 添加任务<br> # python spider_test.py --start_worker # 启动爬虫<br></code></pre></td></tr></table></figure><ul><li>这里包含了入库的操作，在同级目录下创建<em>items</em>文件夹，新建一个 <em>spider_data_item.py</em></li></ul><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-comment"># -*- coding: utf-8 -*-</span><br><span class="hljs-string">"""</span><br><span class="hljs-string">Created on 2024-03-27 11:41:57</span><br><span class="hljs-string">---------</span><br><span class="hljs-string">@summary:</span><br><span class="hljs-string">---------</span><br><span class="hljs-string">@author: zhangmingwei01</span><br><span class="hljs-string">"""</span><br><br><span class="hljs-keyword">from</span> feapder <span class="hljs-keyword">import</span> Item<br><br><br><span class="hljs-keyword">class</span> <span class="hljs-title class_">SpiderDataItem</span>(<span class="hljs-title class_ inherited__">Item</span>):<br> <span class="hljs-string">"""</span><br><span class="hljs-string"> This class was generated by feapder</span><br><span class="hljs-string"> command: feapder create -i spider_data 1</span><br><span class="hljs-string"> """</span><br><br> __table_name__ = <span class="hljs-string">"spider_data"</span><br><br> <span class="hljs-keyword">def</span> <span class="hljs-title function_">__init__</span>(<span class="hljs-params">self, *args, **kwargs</span>):<br> <span class="hljs-comment"># self.id = kwargs.get('id')</span><br> self.title = kwargs.get(<span class="hljs-string">'title'</span>)<br> self.url = kwargs.get(<span class="hljs-string">'url'</span>)<br></code></pre></td></tr></table></figure><ul><li>需要先在数据库中建好表，库名为<em>data</em>，表名为<em>spider_data</em>，以下是建表语句</li></ul><figure class="highlight pgsql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><code class="hljs pgsql"><span class="hljs-keyword">CREATE</span> <span class="hljs-keyword">TABLE</span> `spider_data` (<br> `id` <span class="hljs-type">int</span>(<span class="hljs-number">10</span>) unsigned <span class="hljs-keyword">NOT</span> <span class="hljs-keyword">NULL</span> AUTO_INCREMENT,<br> `title` <span class="hljs-type">varchar</span>(<span class="hljs-number">255</span>) <span class="hljs-keyword">DEFAULT</span> <span class="hljs-keyword">NULL</span>,<br> `url` <span class="hljs-type">varchar</span>(<span class="hljs-number">255</span>) <span class="hljs-keyword">DEFAULT</span> <span class="hljs-keyword">NULL</span>,<br> <span class="hljs-keyword">PRIMARY KEY</span> (`id`)<br> ) ENGINE=InnoDB AUTO_INCREMENT=<span class="hljs-number">1</span> <span class="hljs-keyword">DEFAULT</span> CHARSET=utf8mb4;<br></code></pre></td></tr></table></figure><ul><li>实际业务中可能无 <em>add_task</em>方法，应自己去手动添加任务到redis中</li></ul><figure class="highlight apache"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs apache"><span class="hljs-comment"># 导入_redisdb</span><br><span class="hljs-attribute">_redisdb</span>.zadd(self._task_table, {<span class="hljs-string">"id"</span>: <span class="hljs-number">1</span>, <span class="hljs-string">"url"</span>: <span class="hljs-string">"https://www.baidu.com"</span>})<br></code></pre></td></tr></table></figure>]]></content>
<categories>
<category>爬虫</category>
</categories>
<tags>
<tag>爬虫</tag>
</tags>
</entry>
<entry>
<title>爬虫利器 pyppeteer 使用技巧</title>
<link href="/2022/08/22/%E7%88%AC%E8%99%AB%E5%88%A9%E5%99%A8%20pyppeteer%20%E4%BD%BF%E7%94%A8%E6%8A%80%E5%B7%A7/"/>
<url>/2022/08/22/%E7%88%AC%E8%99%AB%E5%88%A9%E5%99%A8%20pyppeteer%20%E4%BD%BF%E7%94%A8%E6%8A%80%E5%B7%A7/</url>
<content type="html"><![CDATA[<span id="more"></span><h2 id="一、拦截器简单用法"><a href="#一、拦截器简单用法" class="headerlink" title="一、拦截器简单用法"></a>一、拦截器简单用法</h2><p>拦截器作用于单个Page，即浏览器中的一个标签页。每初始化一个Page都要添加一下拦截器。拦截器实际上是</p><p>通过给各种事件添加回调函数来实现的。</p><p>事件列表可参见：pyppeteer.page.Page.Events</p><p>常用拦截器：</p><ul><li>request：发出网络请求时触发</li><li>response：收到网络响应时触发</li><li>dialog：页面有弹窗时触发</li></ul><p>使用request拦截器修改请求：</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-comment"># coding:utf8</span><br><span class="hljs-keyword">import</span> asyncio<br><span class="hljs-keyword">from</span> pyppeteer <span class="hljs-keyword">import</span> launch<br><br><span class="hljs-keyword">from</span> pyppeteer.network_manager <span class="hljs-keyword">import</span> Request<br><br><br>launch_args = {<br> <span class="hljs-string">"headless"</span>: <span class="hljs-literal">False</span>,<br> <span class="hljs-string">"args"</span>: [<br> <span class="hljs-string">"--start-maximized"</span>,<br> <span class="hljs-string">"--no-sandbox"</span>,<br> <span class="hljs-string">"--disable-infobars"</span>,<br> <span class="hljs-string">"--ignore-certificate-errors"</span>,<br> <span class="hljs-string">"--log-level=3"</span>,<br> <span class="hljs-string">"--enable-extensions"</span>,<br> <span class="hljs-string">"--window-size=1920,1080"</span>,<br> <span class="hljs-string">"--user-agent=Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.97 Safari/537.36"</span>,<br> ],<br>}<br><br><br><span class="hljs-keyword">async</span> <span class="hljs-keyword">def</span> <span class="hljs-title function_">modify_url</span>(<span class="hljs-params">request: Request</span>):<br> <span class="hljs-keyword">if</span> request.url == <span class="hljs-string">"https://www.baidu.com/"</span>:<br> <span class="hljs-keyword">await</span> request.continue_({<span class="hljs-string">"url"</span>: <span class="hljs-string">"https://www.baidu.com/s?wd=ip&ie=utf-8"</span>})<br> <span class="hljs-keyword">else</span>:<br> <span class="hljs-keyword">await</span> request.continue_()<br><br><br><span class="hljs-keyword">async</span> <span class="hljs-keyword">def</span> <span class="hljs-title function_">interception_test</span>():<br> <span class="hljs-comment"># 启动浏览器</span><br> browser = <span class="hljs-keyword">await</span> launch(**launch_args)<br> <span class="hljs-comment"># 新建标签页</span><br> page = <span class="hljs-keyword">await</span> browser.newPage()<br> <span class="hljs-comment"># 设置页面打开超时时间</span><br> page.setDefaultNavigationTimeout(<span class="hljs-number">10</span> * <span class="hljs-number">1000</span>)<br> <span class="hljs-comment"># 设置窗口大小</span><br> <span class="hljs-keyword">await</span> page.setViewport({<span class="hljs-string">"width"</span>: <span class="hljs-number">1920</span>, <span class="hljs-string">"height"</span>: <span class="hljs-number">1040</span>})<br><br> <span class="hljs-comment"># 启用拦截器</span><br> <span class="hljs-keyword">await</span> page.setRequestInterception(<span class="hljs-literal">True</span>)<br><br> <span class="hljs-comment"># 设置拦截器</span><br> <span class="hljs-comment"># 1. 修改请求的url</span><br> <span class="hljs-keyword">if</span> <span class="hljs-number">1</span>:<br> page.on(<span class="hljs-string">"request"</span>, modify_url)<br> <span class="hljs-keyword">await</span> page.goto(<span class="hljs-string">"https://www.baidu.com"</span>)<br><br> <span class="hljs-keyword">await</span> asyncio.sleep(<span class="hljs-number">10</span>)<br><br> <span class="hljs-comment"># 关闭浏览器</span><br> <span class="hljs-keyword">await</span> page.close()<br> <span class="hljs-keyword">await</span> browser.close()<br> <span class="hljs-keyword">return</span><br><br><br><span class="hljs-keyword">if</span> __name__ == <span class="hljs-string">"__main__"</span>:<br> loop = asyncio.get_event_loop()<br> loop.run_until_complete(interception_test())<br></code></pre></td></tr></table></figure><p>使用response拦截器获取某个请求的响应：</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-keyword">async</span> <span class="hljs-keyword">def</span> <span class="hljs-title function_">get_content</span>(<span class="hljs-params">response: Response</span>):<br> <span class="hljs-string">"""</span><br><span class="hljs-string"> # 注意这里不需要设置 page.setRequestInterception(True)</span><br><span class="hljs-string"> page.on("response", get_content)</span><br><span class="hljs-string"> :param response:</span><br><span class="hljs-string"> :return:</span><br><span class="hljs-string"> """</span><br> <span class="hljs-keyword">if</span> response.url == <span class="hljs-string">"https://www.baidu.com/"</span>:<br> content = <span class="hljs-keyword">await</span> response.text()<br> title = re.search(<span class="hljs-string">b"<title>(.*?)</title>"</span>, content)<br> <span class="hljs-built_in">print</span>(title.group(<span class="hljs-number">1</span>))<br></code></pre></td></tr></table></figure><p>清除页面所有弹窗:</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-keyword">async</span> <span class="hljs-keyword">def</span> <span class="hljs-title function_">handle_dialog</span>(<span class="hljs-params">dialog: Dialog</span>):<br> <span class="hljs-string">"""</span><br><span class="hljs-string"> page.on("dialog", get_content)</span><br><span class="hljs-string"> :param dialog: </span><br><span class="hljs-string"> :return: </span><br><span class="hljs-string"> """</span><br> <span class="hljs-keyword">await</span> dialog.dismiss()<br></code></pre></td></tr></table></figure><h2 id="二、拦截器实现切换代理"><a href="#二、拦截器实现切换代理" class="headerlink" title="二、拦截器实现切换代理"></a>二、拦截器实现切换代理</h2><p>一般情况下浏览器添加代理的方法为设置启动参数：</p><blockquote><p>–proxy-server=<a href="http://user:password@ip:port">http://user:password@ip:port</a></p></blockquote><p>例如：</p><figure class="highlight ada"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><code class="hljs ada">launch_args = {<br> <span class="hljs-string">"headless"</span>: <span class="hljs-literal">False</span>,<br> <span class="hljs-string">"args"</span>: [ <span class="hljs-string">"--proxy-server=http://localhost:1080"</span>,<br> <span class="hljs-string">"--user-agent=Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.97 Safari/537.36"</span>,<br> ],<br>}<br></code></pre></td></tr></table></figure><p>但此种方式的缺点很明显，只能在浏览器启动时设置。当需要切换代理时，只能重启浏览器，这个代价</p><p>就太高了，所以我们可以想想其他办法。</p><p>思路很简单：</p><ul><li>request拦截器可以修改请求属性并且返回自定义响应内容</li><li>使用第三方库来发送网络请求，并设置代理。然后封装响应内容返回给浏览器</li></ul><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-keyword">import</span> aiohttp<br><br>aiohttp_session = aiohttp.ClientSession(loop=asyncio.get_event_loop())<br><br>proxy = <span class="hljs-string">"http://127.0.0.1:1080"</span><br><span class="hljs-keyword">async</span> <span class="hljs-keyword">def</span> <span class="hljs-title function_">use_proxy_base</span>(<span class="hljs-params">request: Request</span>):<br> <span class="hljs-string">"""</span><br><span class="hljs-string"> # 启用拦截器</span><br><span class="hljs-string"> await page.setRequestInterception(True)</span><br><span class="hljs-string"> page.on("request", use_proxy_base)</span><br><span class="hljs-string"> :param request:</span><br><span class="hljs-string"> :return:</span><br><span class="hljs-string"> """</span><br> <span class="hljs-comment"># 构造请求并添加代理</span><br> req = {<br> <span class="hljs-string">"headers"</span>: request.headers,<br> <span class="hljs-string">"data"</span>: request.postData,<br> <span class="hljs-string">"proxy"</span>: proxy, <span class="hljs-comment"># 使用全局变量 则可随意切换</span><br> <span class="hljs-string">"timeout"</span>: <span class="hljs-number">5</span>,<br> <span class="hljs-string">"ssl"</span>: <span class="hljs-literal">False</span>,<br> }<br> <span class="hljs-keyword">try</span>:<br> <span class="hljs-comment"># 使用第三方库获取响应</span><br> <span class="hljs-keyword">async</span> <span class="hljs-keyword">with</span> aiohttp_session.request(<br> method=request.method, url=request.url, **req<br> ) <span class="hljs-keyword">as</span> response:<br> body = <span class="hljs-keyword">await</span> response.read()<br> <span class="hljs-keyword">except</span> Exception <span class="hljs-keyword">as</span> e:<br> <span class="hljs-keyword">await</span> request.abort()<br> <span class="hljs-keyword">return</span><br><br> <span class="hljs-comment"># 数据返回给浏览器</span><br> resp = {<span class="hljs-string">"body"</span>: body, <span class="hljs-string">"headers"</span>: response.headers, <span class="hljs-string">"status"</span>: response.status}<br> <span class="hljs-keyword">await</span> request.respond(resp)<br> <span class="hljs-keyword">return</span><br></code></pre></td></tr></table></figure><p>或者再增加一些缓存来节约一下带宽：</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-comment"># 静态资源缓存</span><br>static_cache = {}<br><span class="hljs-keyword">async</span> <span class="hljs-keyword">def</span> <span class="hljs-title function_">use_proxy_and_cache</span>(<span class="hljs-params">request: Request</span>):<br> <span class="hljs-string">"""</span><br><span class="hljs-string"> # 启用拦截器</span><br><span class="hljs-string"> await page.setRequestInterception(True)</span><br><span class="hljs-string"> page.on("request", use_proxy_base)</span><br><span class="hljs-string"> :param request:</span><br><span class="hljs-string"> :return:</span><br><span class="hljs-string"> """</span><br> <span class="hljs-keyword">global</span> static_cache<br> <span class="hljs-keyword">if</span> request.url <span class="hljs-keyword">not</span> <span class="hljs-keyword">in</span> static_cache:<br> <span class="hljs-comment"># 构造请求并添加代理</span><br> req = {<br> <span class="hljs-string">"headers"</span>: request.headers,<br> <span class="hljs-string">"data"</span>: request.postData,<br> <span class="hljs-string">"proxy"</span>: proxy, <span class="hljs-comment"># 使用全局变量 则可随意切换</span><br> <span class="hljs-string">"timeout"</span>: <span class="hljs-number">5</span>,<br> <span class="hljs-string">"ssl"</span>: <span class="hljs-literal">False</span>,<br> }<br> <span class="hljs-keyword">try</span>:<br> <span class="hljs-comment"># 使用第三方库获取响应</span><br> <span class="hljs-keyword">async</span> <span class="hljs-keyword">with</span> aiohttp_session.request(<br> method=request.method, url=request.url, **req<br> ) <span class="hljs-keyword">as</span> response:<br> body = <span class="hljs-keyword">await</span> response.read()<br> <span class="hljs-keyword">except</span> Exception <span class="hljs-keyword">as</span> e:<br> <span class="hljs-keyword">await</span> request.abort()<br> <span class="hljs-keyword">return</span><br><br> <span class="hljs-comment"># 数据返回给浏览器</span><br> resp = {<span class="hljs-string">"body"</span>: body, <span class="hljs-string">"headers"</span>: response.headers, <span class="hljs-string">"status"</span>: response.status}<br> <span class="hljs-comment"># 判断数据类型 如果是静态文件则缓存起来</span><br> content_type = response.headers.get(<span class="hljs-string">"Content-Type"</span>)<br> <span class="hljs-keyword">if</span> content_type <span class="hljs-keyword">and</span> (<span class="hljs-string">"javascript"</span> <span class="hljs-keyword">in</span> content_type <span class="hljs-keyword">or</span> <span class="hljs-string">"/css"</span> <span class="hljs-keyword">in</span> content_type):<br> static_cache[request.url] = resp<br> <span class="hljs-keyword">else</span>:<br> resp = static_cache[request.url]<br><br> <span class="hljs-keyword">await</span> request.respond(resp)<br> <span class="hljs-keyword">return</span><br></code></pre></td></tr></table></figure><h2 id="最后"><a href="#最后" class="headerlink" title="最后"></a>最后</h2><p>参考：<a href="https://www.cnblogs.com/dyfblog/p/10887940.html">pyppeteer进阶技巧</a></p>]]></content>
<categories>
<category>爬虫</category>
</categories>
<tags>
<tag>爬虫</tag>
</tags>
</entry>
<entry>
<title>用pandas生成excel文件示例，并调整excel的格式或样式</title>
<link href="/2022/06/27/%E7%94%A8pandas%E7%94%9F%E6%88%90excel%E6%96%87%E4%BB%B6%E7%A4%BA%E4%BE%8B%EF%BC%8C%E5%B9%B6%E8%B0%83%E6%95%B4excel%E7%9A%84%E6%A0%BC%E5%BC%8F%E6%88%96%E6%A0%B7%E5%BC%8F/"/>
<url>/2022/06/27/%E7%94%A8pandas%E7%94%9F%E6%88%90excel%E6%96%87%E4%BB%B6%E7%A4%BA%E4%BE%8B%EF%BC%8C%E5%B9%B6%E8%B0%83%E6%95%B4excel%E7%9A%84%E6%A0%BC%E5%BC%8F%E6%88%96%E6%A0%B7%E5%BC%8F/</url>
<content type="html"><![CDATA[<span id="more"></span><h2 id="用pandas生成excel"><a href="#用pandas生成excel" class="headerlink" title="用pandas生成excel"></a>用pandas生成excel</h2><ul><li>当我们有特殊的需求时，比如要修改excel的行宽列宽，还有字体样式等等</li></ul><h3 id="需求示例"><a href="#需求示例" class="headerlink" title="需求示例"></a>需求示例</h3><p><img src="https://img-blog.csdnimg.cn/f37a16db24b84301863ffaf4d486819e.png" alt="在这里插入图片描述"></p><h3 id="代码实现"><a href="#代码实现" class="headerlink" title="代码实现"></a>代码实现</h3><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-comment"># -*- coding: utf-8 -*-</span><br><span class="hljs-keyword">import</span> pandas <span class="hljs-keyword">as</span> pd<br><span class="hljs-keyword">from</span> datetime <span class="hljs-keyword">import</span> datetime, timedelta<br><br><br><span class="hljs-keyword">def</span> <span class="hljs-title function_">modify_excel_format</span>(<span class="hljs-params">excel_data, writer, df</span>):<br> <span class="hljs-comment"># ----------调整excel格式 ---------------</span><br> workbook = writer.book<br> fmt = workbook.add_format({<span class="hljs-string">"font_name"</span>: <span class="hljs-string">u"宋体"</span>})<br> col_fmt = workbook.add_format(<br> {<span class="hljs-string">'bold'</span>: <span class="hljs-literal">True</span>, <span class="hljs-string">'font_size'</span>: <span class="hljs-number">11</span>, <span class="hljs-string">'font_name'</span>: <span class="hljs-string">u'宋体'</span>, <span class="hljs-string">'border'</span>: <span class="hljs-number">1</span>, <span class="hljs-string">'bg_color'</span>: <span class="hljs-string">'#0265CB'</span>,<span class="hljs-string">'font_color'</span>: <span class="hljs-string">'white'</span>,<br> <span class="hljs-string">'valign'</span>: <span class="hljs-string">'vcenter'</span>, <span class="hljs-string">'align'</span>: <span class="hljs-string">'center'</span>})<br> detail_fmt = workbook.add_format(<br> {<span class="hljs-string">"font_name"</span>: <span class="hljs-string">u"宋体"</span>, <span class="hljs-string">'border'</span>: <span class="hljs-number">0</span>, <span class="hljs-string">'valign'</span>: <span class="hljs-string">'vcenter'</span>, <span class="hljs-string">'align'</span>: <span class="hljs-string">'center'</span>,<span class="hljs-string">'font_size'</span>: <span class="hljs-number">11</span>, <span class="hljs-string">'text_wrap'</span>: <span class="hljs-literal">True</span>})<br> worksheet1 = writer.sheets[<span class="hljs-string">'Sheet1'</span>]<br> <span class="hljs-keyword">for</span> col_num, value <span class="hljs-keyword">in</span> <span class="hljs-built_in">enumerate</span>(df.columns.values):<br> worksheet1.write(<span class="hljs-number">0</span>, col_num, value, col_fmt)<br> <span class="hljs-comment"># 设置列宽行宽</span><br> worksheet1.set_column(<span class="hljs-string">'A:F'</span>, <span class="hljs-number">20</span>, fmt)<br> worksheet1.set_row(<span class="hljs-number">0</span>, <span class="hljs-number">30</span>, fmt)<br> <span class="hljs-keyword">for</span> i <span class="hljs-keyword">in</span> <span class="hljs-built_in">range</span>(<span class="hljs-number">1</span>, <span class="hljs-built_in">len</span>(excel_data)+<span class="hljs-number">1</span>):<br> worksheet1.set_row(i, <span class="hljs-number">27</span>, detail_fmt)<br><br><br>excel_data = []<br>columns = [<span class="hljs-string">"用户"</span>, <span class="hljs-string">"名字"</span>, <span class="hljs-string">"标题"</span>, <span class="hljs-string">"类"</span>, <span class="hljs-string">"测试"</span>, <span class="hljs-string">"语言"</span>]<br>tmp = [<span class="hljs-string">"张同学"</span>, <span class="hljs-string">"张同学"</span>, <span class="hljs-string">"pd 生成excel"</span>, <span class="hljs-string">"pandas"</span>, <span class="hljs-string">"pd"</span>, <span class="hljs-string">"python"</span>]<br>excel_data.append(tmp)<br>df = pd.DataFrame(data=excel_data, columns=columns)<br>t = datetime.now().date() - timedelta(days=<span class="hljs-number">1</span>)<br><span class="hljs-keyword">with</span> pd.ExcelWriter(path=<span class="hljs-string">'demo-%d%02d%02d.xlsx'</span> % (t.year, t.month, t.day), engine=<span class="hljs-string">"xlsxwriter"</span>) <span class="hljs-keyword">as</span> writer:<br> df.to_excel(writer, sheet_name=<span class="hljs-string">'Sheet1'</span>, encoding=<span class="hljs-string">'utf8'</span>, header=<span class="hljs-literal">False</span>, index=<span class="hljs-literal">False</span>, startcol=<span class="hljs-number">0</span>, startrow=<span class="hljs-number">1</span>)<br> modify_excel_format(excel_data, writer, df)<br> writer.save()<br> <span class="hljs-built_in">print</span>(<span class="hljs-string">'ok!'</span>)<br><br></code></pre></td></tr></table></figure><ul><li>我的blog链接：<a href="https://waym1ng.github.io/2022/06/26/%E7%94%A8pandas%E7%94%9F%E6%88%90excel%E6%96%87%E4%BB%B6%E7%A4%BA%E4%BE%8B%EF%BC%8C%E5%B9%B6%E8%B0%83%E6%95%B4excel%E7%9A%84%E6%A0%BC%E5%BC%8F%E6%88%96%E6%A0%B7%E5%BC%8F/">用pandas生成excel文件示例，并调整excel的格式或样式</a></li></ul>]]></content>
<categories>
<category>python加油鸭</category>
</categories>
<tags>
<tag>pandas python 数据分析</tag>
</tags>
</entry>
<entry>
<title>Python 时间戳转换出错</title>
<link href="/2022/06/20/Python%20%E6%97%B6%E9%97%B4%E6%88%B3%E8%BD%AC%E6%8D%A2%E5%87%BA%E9%94%99/"/>
<url>/2022/06/20/Python%20%E6%97%B6%E9%97%B4%E6%88%B3%E8%BD%AC%E6%8D%A2%E5%87%BA%E9%94%99/</url>
<content type="html"><![CDATA[<span id="more"></span><ul><li>当我们想将时间戳转换成特定格式的时间字符串，比如带有年月日，以下写法可能会出现报错</li></ul><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs python">datetime.strftime(datetime.fromtimestamp(<span class="hljs-number">1655481600</span>), <span class="hljs-string">'%Y年%m月%d日 %H:%M:%S'</span>)<br></code></pre></td></tr></table></figure><blockquote><p>UnicodeEncodeError: ‘locale’ codec can’t encode character ‘\u5e74’ in position 2: encoding error</p></blockquote><ul><li>解决方案</li></ul><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-keyword">from</span> datetime <span class="hljs-keyword">import</span> datetime<br>datetime_str = datetime.strftime(datetime.fromtimestamp(<span class="hljs-number">1655481600</span>), <span class="hljs-string">'%Yn%my%dr %H:%M:%S'</span>).replace(<span class="hljs-string">'n'</span>,<span class="hljs-string">'年'</span>).replace(<span class="hljs-string">'y'</span>, <span class="hljs-string">'月'</span>).replace(<span class="hljs-string">'r'</span>, <span class="hljs-string">'日'</span>)<br><span class="hljs-built_in">print</span>(datetime_str) <span class="hljs-comment"># '2022年06月18日 00:00:00'</span><br>datetime_str2 = datetime.strftime(datetime.fromtimestamp(<span class="hljs-number">1655481600</span>), <span class="hljs-string">'%Y{y}%m{m}%d{d} %H:%M:%S'</span>).<span class="hljs-built_in">format</span>(y=<span class="hljs-string">'年'</span>, m=<span class="hljs-string">'月'</span>, d=<span class="hljs-string">'日'</span>)<br><span class="hljs-built_in">print</span>(datetime_str2) <span class="hljs-comment"># '2022年06月18日 00:00:00'</span><br></code></pre></td></tr></table></figure>]]></content>
<categories>
<category>python加油鸭</category>
</categories>
<tags>
<tag>python</tag>
</tags>
</entry>
<entry>
<title>Python try...except...时捕获异常时具体文件和行数</title>
<link href="/2021/07/08/Python%20try...except...%E6%97%B6%E6%8D%95%E8%8E%B7%E5%BC%82%E5%B8%B8%E6%97%B6%E5%85%B7%E4%BD%93%E6%96%87%E4%BB%B6%E5%92%8C%E8%A1%8C%E6%95%B0/"/>
<url>/2021/07/08/Python%20try...except...%E6%97%B6%E6%8D%95%E8%8E%B7%E5%BC%82%E5%B8%B8%E6%97%B6%E5%85%B7%E4%BD%93%E6%96%87%E4%BB%B6%E5%92%8C%E8%A1%8C%E6%95%B0/</url>
<content type="html"><![CDATA[<span id="more"></span><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-keyword">def</span> <span class="hljs-title function_">try_exception_test</span>():<br> <span class="hljs-keyword">try</span>:<br> a = <span class="hljs-number">0</span><br> b = <span class="hljs-number">1</span>/a<br> <span class="hljs-built_in">print</span>(b)<br> <span class="hljs-keyword">except</span> Exception <span class="hljs-keyword">as</span> e:<br> <span class="hljs-built_in">print</span>(e)<br> <span class="hljs-comment"># 发生异常所在的文件</span><br> <span class="hljs-built_in">print</span>(e.__traceback__.tb_frame.f_globals[<span class="hljs-string">"__file__"</span>])<br> <span class="hljs-comment"># 发生异常所在的行数</span><br> <span class="hljs-built_in">print</span>(e.__traceback__.tb_lineno)<br><br><span class="hljs-keyword">if</span> __name__ == <span class="hljs-string">'__main__'</span>:<br> try_exception_test()<br></code></pre></td></tr></table></figure><p>模拟一段会抛出异常的代码</p><p>执行结果：</p><blockquote><p>division by zero<br>C:/Users/admin01/Desktop/script/demo.py<br>4</p></blockquote><p>可以看到报错原因为<strong>division by zero</strong></p><p>文件位置为<strong>C:/Users/admin01/Desktop/script/demo.py</strong></p><p>行数为<strong>第4行</strong></p><p> 要是系统中不能实时打印出来的话，可以考虑加上 <strong>flush=True</strong></p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-built_in">print</span>(<span class="hljs-built_in">str</span>(e), flush=<span class="hljs-literal">True</span>)<br></code></pre></td></tr></table></figure>]]></content>
<categories>
<category>python加油鸭</category>
</categories>
<tags>
<tag>python debug</tag>
</tags>
</entry>
<entry>
<title>python 列表List转换成树形结构</title>
<link href="/2021/06/22/python%20%E5%88%97%E8%A1%A8List%E8%BD%AC%E6%8D%A2%E6%88%90%E6%A0%91%E5%BD%A2%E7%BB%93%E6%9E%84/"/>
<url>/2021/06/22/python%20%E5%88%97%E8%A1%A8List%E8%BD%AC%E6%8D%A2%E6%88%90%E6%A0%91%E5%BD%A2%E7%BB%93%E6%9E%84/</url>
<content type="html"><![CDATA[<span id="more"></span><ul><li><strong>原始数据</strong>：list中嵌套dict的数据格式</li><li><strong>转换结果</strong>：数结构的数据，children字段嵌套的形式，适用于前端树形结构的渲染</li></ul><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-keyword">def</span> <span class="hljs-title function_">list_to_tree</span>(<span class="hljs-params">data</span>):<br> root = []<br> node = []<br><br> <span class="hljs-comment"># 初始化数据，获取根节点和其他子节点list</span><br> <span class="hljs-keyword">for</span> d <span class="hljs-keyword">in</span> data:<br> d[<span class="hljs-string">"choice"</span>] = <span class="hljs-number">0</span><br> <span class="hljs-keyword">if</span> d.get(<span class="hljs-string">"parent_id"</span>) == <span class="hljs-number">0</span>:<br> root.append(d)<br> <span class="hljs-keyword">else</span>:<br> node.append(d)<br> <span class="hljs-comment"># print("root----",root)</span><br> <span class="hljs-comment"># print("node----",node)</span><br> <span class="hljs-comment"># 查找子节点</span><br> <span class="hljs-keyword">for</span> p <span class="hljs-keyword">in</span> root:<br> add_node(p, node)<br><br> <span class="hljs-comment"># 无子节点</span><br> <span class="hljs-keyword">if</span> <span class="hljs-built_in">len</span>(root) == <span class="hljs-number">0</span>:<br> <span class="hljs-keyword">return</span> node<br><br> <span class="hljs-keyword">return</span> root<br><br><br><span class="hljs-keyword">def</span> <span class="hljs-title function_">add_node</span>(<span class="hljs-params">p, node</span>):<br> <span class="hljs-comment"># 子节点list</span><br> p[<span class="hljs-string">"children"</span>] = []<br> <span class="hljs-keyword">for</span> n <span class="hljs-keyword">in</span> node:<br> <span class="hljs-keyword">if</span> n.get(<span class="hljs-string">"parent_id"</span>) == p.get(<span class="hljs-string">"theme_id"</span>):<br> p[<span class="hljs-string">"children"</span>].append(n)<br><br> <span class="hljs-comment"># 递归子节点，查找子节点的节点</span><br> <span class="hljs-keyword">for</span> t <span class="hljs-keyword">in</span> p[<span class="hljs-string">"children"</span>]:<br> <span class="hljs-keyword">if</span> <span class="hljs-keyword">not</span> t.get(<span class="hljs-string">"children"</span>):<br> t[<span class="hljs-string">"children"</span>] = []<br> t[<span class="hljs-string">"children"</span>].append(add_node(t, node))<br><br> <span class="hljs-comment"># 退出递归的条件</span><br> <span class="hljs-keyword">if</span> <span class="hljs-built_in">len</span>(p[<span class="hljs-string">"children"</span>]) == <span class="hljs-number">0</span>:<br> p[<span class="hljs-string">"choice"</span>] = <span class="hljs-number">1</span><br> <span class="hljs-keyword">return</span><br> <br><span class="hljs-keyword">if</span> __name__ == <span class="hljs-string">'__main__'</span>:<br> data_list = [{<span class="hljs-string">'parent_id'</span>: <span class="hljs-number">10023</span>, <span class="hljs-string">'theme_id'</span>: <span class="hljs-number">10024</span>, <span class="hljs-string">'theme_name'</span>: <span class="hljs-string">'英语三级'</span>},<br> {<span class="hljs-string">'parent_id'</span>: <span class="hljs-number">10022</span>, <span class="hljs-string">'theme_id'</span>: <span class="hljs-number">10023</span>, <span class="hljs-string">'theme_name'</span>: <span class="hljs-string">'英语二级'</span>},<br> {<span class="hljs-string">'parent_id'</span>: <span class="hljs-number">0</span>, <span class="hljs-string">'theme_id'</span>: <span class="hljs-number">10025</span>, <span class="hljs-string">'theme_name'</span>: <span class="hljs-string">'语文一级'</span>},<br> {<span class="hljs-string">'parent_id'</span>: <span class="hljs-number">10025</span>, <span class="hljs-string">'theme_id'</span>: <span class="hljs-number">10026</span>, <span class="hljs-string">'theme_name'</span>: <span class="hljs-string">'语文二级'</span>},<br> {<span class="hljs-string">'parent_id'</span>: <span class="hljs-number">0</span>, <span class="hljs-string">'theme_id'</span>: <span class="hljs-number">10022</span>, <span class="hljs-string">'theme_name'</span>: <span class="hljs-string">'英语一级'</span>}]<br> data_tree = list_to_tree(data_list)<br> <span class="hljs-built_in">print</span>(data_tree)<br> <span class="hljs-string">'''</span><br><span class="hljs-string"> 结果如下：</span><br><span class="hljs-string"> [</span><br><span class="hljs-string"> {</span><br><span class="hljs-string"> "parent_id": 0,</span><br><span class="hljs-string"> "theme_id": 10025,</span><br><span class="hljs-string"> "theme_name": "语文一级",</span><br><span class="hljs-string"> "choice": 0,</span><br><span class="hljs-string"> "children": [</span><br><span class="hljs-string"> {</span><br><span class="hljs-string"> "parent_id": 10025,</span><br><span class="hljs-string"> "theme_id": 10026,</span><br><span class="hljs-string"> "theme_name": "语文二级",</span><br><span class="hljs-string"> "choice": 1,</span><br><span class="hljs-string"> "children": []</span><br><span class="hljs-string"> }</span><br><span class="hljs-string"> ]</span><br><span class="hljs-string"> },</span><br><span class="hljs-string"> {</span><br><span class="hljs-string"> "parent_id": 0,</span><br><span class="hljs-string"> "theme_id": 10022,</span><br><span class="hljs-string"> "theme_name": "英语一级",</span><br><span class="hljs-string"> "choice": 0,</span><br><span class="hljs-string"> "children": [</span><br><span class="hljs-string"> {</span><br><span class="hljs-string"> "parent_id": 10022,</span><br><span class="hljs-string"> "theme_id": 10023,</span><br><span class="hljs-string"> "theme_name": "英语二级",</span><br><span class="hljs-string"> "choice": 0,</span><br><span class="hljs-string"> "children": [</span><br><span class="hljs-string"> {</span><br><span class="hljs-string"> "parent_id": 10023,</span><br><span class="hljs-string"> "theme_id": 10024,</span><br><span class="hljs-string"> "theme_name": "英语三级",</span><br><span class="hljs-string"> "choice": 1,</span><br><span class="hljs-string"> "children": []</span><br><span class="hljs-string"> }</span><br><span class="hljs-string"> ]</span><br><span class="hljs-string"> }</span><br><span class="hljs-string"> ]</span><br><span class="hljs-string"> }</span><br><span class="hljs-string"> ]</span><br><span class="hljs-string"> '''</span><br></code></pre></td></tr></table></figure>]]></content>
<categories>
<category>python加油鸭</category>
</categories>
<tags>
<tag>python</tag>
</tags>
</entry>
<entry>
<title>关于MySQL的优化思路</title>
<link href="/2021/06/11/%E5%85%B3%E4%BA%8EMySQL%E7%9A%84%E4%BC%98%E5%8C%96%E6%80%9D%E8%B7%AF/"/>
<url>/2021/06/11/%E5%85%B3%E4%BA%8EMySQL%E7%9A%84%E4%BC%98%E5%8C%96%E6%80%9D%E8%B7%AF/</url>
<content type="html"><![CDATA[<span id="more"></span><h2 id="SQL-优化步骤"><a href="#SQL-优化步骤" class="headerlink" title="SQL 优化步骤"></a>SQL 优化步骤</h2><p>当面对一个需要优化的 SQL 时，我们有哪几种排查思路呢？</p><h3 id="通过-show-status-命令了解-SQL-执行次数"><a href="#通过-show-status-命令了解-SQL-执行次数" class="headerlink" title="通过 show status 命令了解 SQL 执行次数"></a>通过 show status 命令了解 SQL 执行次数</h3><p>首先，我们可以使用 <strong>show status</strong> 命令查看服务器状态信息。show status 命令会显示每个服务器变量 variable_name 和 value，状态变量是只读的。如果使用 SQL 命令，可以使用 like 或者 where 条件来限制结果。like 可以对变量名做标准模式匹配。</p><p><img src="https://img-blog.csdnimg.cn/img_convert/df0228046f8cda9907e2e85259e81bd1.png" alt="图片"></p><p>图我没有截全，下面还有很多变量，读者可以自己尝试一下。也可以在操作系统上使用 <strong>mysqladmin extended-status</strong> 命令来获取这些消息。</p><p>但是我执行 mysqladmin extended-status 后，出现这个错误。</p><p><img src="https://img-blog.csdnimg.cn/img_convert/6e67bbb69e51e9eec7381637972c6ca7.png" alt="图片"></p><p>应该是我没有输入密码的原因，使用 <strong>mysqladmin -P3306 -uroot -p -h127.0.0.1 -r -i 1 extended-status</strong> 后，问题解决。</p><p>这里需要注意一下 show status 命令中可以添加统计结果的级别，这个级别有两个</p><ul><li><p>session 级：默认当前链接的统计结果</p></li><li><p>global 级：自数据库上次启动到现在的统计结果</p></li></ul><p>如果不指定统计结果级别的话，默认使用 session 级别。</p><p>对于 show status 查询出来的统计结果，有两类参数需要注意下，一类是以 <code>Com_</code> 为开头的参数，一类是以 <code>Innodb_</code> 为开头的参数。</p><p>下面是 Com_ 为开头的参数，参数很多，我同样没有截全。</p><p><img src="https://img-blog.csdnimg.cn/img_convert/fa90d54962f22320be3d3b99bb3b271e.png" alt="图片"></p><p>Com_xxx 表示的是每个 xxx 语句执行的次数，我们通常关心的是 select 、insert 、update、delete 语句的执行次数，即</p><ul><li><p>Com_select：执行 select 操作的次数，一次查询会使结果 + 1。</p></li><li><p>Com_insert：执行 INSERT 操作的次数，对于批量插入的 INSERT 操作，只累加一次。</p></li><li><p>Com_update：执行 UPDATE 操作的次数。</p></li><li><p>Com_delete：执行 DELETE 操作的次数。</p></li></ul><p>以 Innodb_ 为开头的参数主要有</p><ul><li><p>Innodb_rows_read：执行 select 查询返回的行数。</p></li><li><p>Innodb_rows_inserted：执行 INSERT 操作插入的行数。</p></li><li><p>Innodb_rows_updated：执行 UPDATE 操作更新的行数。</p></li><li><p>Innodb_rows_deleted：执行 DELETE 操作删除的行数。</p></li></ul><p>通过上面这些参数执行结果的统计，我们能够大致了解到当前数据库是以更新（包括插入、删除）为主还是查询为主。</p><p>除此之外，还有一些其他参数用于了解数据库的基本情况。</p><ul><li><p>Connections：查询 MySQL 数据库的连接次数，这个次数是不管连接是否成功都算上。</p></li><li><p>Uptime：服务器的工作时间。</p></li><li><p>Slow_queries：满查询次数。</p></li><li><p>Threads_connected：查看当前打开的连接的数量。</p></li></ul><p>下面这个博客汇总了几乎所有 show status 的参数，可以当作参考手册。</p><p><a href="https://blog.csdn.net/ayay/_870621/article/details/88633092">https://blog.csdn.net/ayay\_870621/article/details/88633092</a></p><h3 id="定位执行效率较低的-SQL"><a href="#定位执行效率较低的-SQL" class="headerlink" title="定位执行效率较低的 SQL"></a>定位执行效率较低的 SQL</h3><p>定位执行效率比较慢的 SQL 语句，一般有两种方式</p><ul><li>可以通过<strong>慢查询日志</strong>来定位哪些执行效率较低的 SQL 语句。</li></ul><p>MySQL 中提供了一个慢查询的日志记录功能，可以把查询 SQL 语句时间大于多少秒的语句写入慢查询日志，日常维护中可以通过慢查询日志的记录信息快速准确地判断问题所在。用 --log-slow-queries 选项启动时，mysqld 会写一个包含所有执行时间超过 long_query_time 秒的 SQL 语句的日志文件，通过查看这个日志文件定位效率较低的 SQL 。</p><p>比如我们可以在 my.cnf 中添加如下代码，然后退出重启 MySQL。</p><figure class="highlight arcade"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs arcade"><span class="hljs-built_in">log</span>-slow-queries = <span class="hljs-regexp">/tmp/my</span>sql-slow.<span class="hljs-built_in">log</span><br>long_query_time = <span class="hljs-number">2</span><br></code></pre></td></tr></table></figure><p>通常我们设置最长的查询时间是 2 秒，表示查询时间超过 2 秒就记录了，通常情况下 2 秒就够了，然而对于很多 WEB 应用来说，2 秒时间还是比较长的。</p><p>也可以通过命令来开启：</p><p>我们先查询 MySQL 慢查询日志是否开启</p><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs sql"><span class="hljs-keyword">show</span> variables <span class="hljs-keyword">like</span> "%slow%";<br></code></pre></td></tr></table></figure><p><img src="https://img-blog.csdnimg.cn/img_convert/f12c417c58bed5de2c46147200cc2906.png" alt="图片"></p><p>启用慢查询日志</p><figure class="highlight routeros"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs routeros"><span class="hljs-built_in">set</span> global <span class="hljs-attribute">slow_query_log</span>=<span class="hljs-string">'ON'</span>;<br></code></pre></td></tr></table></figure><p><img src="https://img-blog.csdnimg.cn/img_convert/57df048943a64aeb47a71c8fb0fc6877.png" alt="图片"></p><p>然后再次查询慢查询是否开启</p><p><img src="https://img-blog.csdnimg.cn/img_convert/02b2cbe1c1d5839fc921e1be79e48af0.png" alt="图片"></p><p>如图所示，我们已经开启了慢查询日志。</p><p>慢查询日志会在查询结束以后才记录，所以在应用反应执行效率出现问题的时候慢查询日志并不能定位问题，此时应该使用 <strong>show processlist</strong> 命令查看当前 MySQL 正在进行的线程。包括线程的状态、是否锁表等，可以实时的查看 SQL 执行情况。同样，使用<strong>mysqladmin processlist</strong>语句也能得到此信息。</p><p><img src="https://img-blog.csdnimg.cn/img_convert/8282291cafbc0201bb433c0cae1eb9df.png" alt="图片"></p><p>下面就来解释一下各个字段对应的概念</p><ul><li><p>Id ：Id 就是一个标示，在我们使用 kill 命令杀死进程的时候很有用，比如 kill 进程号。</p></li><li><p>User：显示当前的用户，如果不是 root，这个命令就只显示你权限范围内的 SQL 语句。</p></li><li><p>Host：显示 IP ，用于追踪问题</p></li><li><p>Db：显示这个进程目前连接的是哪个数据库，为 null 是还没有 select 数据库。</p></li><li><p>Command：显示当前连接锁执行的命令，一般有三种：查询 query，休眠 sleep，连接 connect。</p></li><li><p>Time：这个状态持续的时间，单位是秒</p></li><li><p>State：显示当前 SQL 语句的状态，非常重要，下面会具体解释。</p></li><li><p>Info：显示这个 SQL 语句。</p></li></ul><p>State 列非常重要，关于这个列的内容比较多，读者可以参考一下这篇文章</p><p><a href="https://blog.csdn.net/weixin/_34357436/article/details/91768402">https://blog.csdn.net/weixin\_34357436/article/details/91768402</a></p><p>这里面涉及线程的状态、是否锁表等选项，可以实时的查看 SQL 的执行情况，同时对一些锁表进行优化。</p><h3 id="通过-EXPLAIN-命令分析-SQL-的执行计划"><a href="#通过-EXPLAIN-命令分析-SQL-的执行计划" class="headerlink" title="通过 EXPLAIN 命令分析 SQL 的执行计划"></a>通过 EXPLAIN 命令分析 SQL 的执行计划</h3><p>通过以上步骤查询到效率低的 SQL 语句后，可以通过 EXPLAIN 或者 DESC 命令获取 MySQL 如何执行 SELECT 语句的信息，包括在 SELECT 语句执行过程中表如何连接和连接的顺序。</p><p>比如我们使用下面这条 SQL 语句来分析一下执行计划</p><figure class="highlight n1ql"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs n1ql"><span class="hljs-keyword">explain</span> <span class="hljs-keyword">select</span> * <span class="hljs-keyword">from</span> test1;<br></code></pre></td></tr></table></figure><p><img src="https://img-blog.csdnimg.cn/img_convert/230949e49f2eb9f0d75b11f6e3722df8.png" alt="图片"></p><p>上表中涉及内容如下</p><ul><li>select_type：表示常见的 SELECT 类型，常见的有 SIMPLE，SIMPLE 表示的是简单的 SQL 语句，不包括 UNION 或者子查询操作，比如下面这段就是 SIMPLE 类型。</li></ul><p><img src="https://img-blog.csdnimg.cn/img_convert/9c620f1fa2648014b79e1eb39d81da85.png" alt="图片"></p><p>PRIMARY ，查询中最外层的 SELECT（如两表做 UNION 或者存在子查询的外层的表操作为 PRIMARY，内层的操作为 UNION），比如下面这段子查询。</p><p><img src="https://img-blog.csdnimg.cn/img_convert/dcf7088fa50d696761693cf128fecc2c.png" alt="图片"></p><p>UNION，在 UNION 操作中，查询中处于内层的 SELECT（内层的 SELECT 语句与外层的 SELECT 语句没有依赖关系时）。</p><p>SUBQUERY：子查询中首个SELECT（如果有多个子查询存在），如我们上面的查询语句，子查询第一个是 sr（sys_role）表，所以它的 select_type 是 SUBQUERY。</p><ul><li><p>table ，这个选项表示输出结果集的表。</p></li><li><p>type，这个选项表示表的连接类型，这个选项很有深入研究的价值，因为很多 SQL 的调优都是围绕 type 来讲的，但是这篇文章我们主要围绕优化方式来展开的，type 这个字段我们暂时作为了解，这篇文章不过多深入。</p><p>type 这个字段会牵扯到连接的性能，它的不同类型的性能由好到差分别是</p><p>system ：表中仅有一条数据时，该表的查询就像查询常量表一样。</p><p>const ：当表中只有一条记录匹配时，比如使用了表主键（primary key）或者表唯一索引（unique index）进行查询。</p><p>eq-ref ：表示多表连接时使用表主键或者表唯一索引，比如</p><figure class="highlight pgsql"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs pgsql"><span class="hljs-keyword">select</span> A.text, B.text <span class="hljs-keyword">where</span> A.ID = B.ID<br></code></pre></td></tr></table></figure><p>这个查询语句，对于 A 表中的每一个 ID 行，B 表中都只能有唯一的 B.Id 来进行匹配时。</p><p>ref ：这个类型不如上面的 eq-ref 快，因为它表示的是因为对于表 A 中扫描的每一行，表 C 中有几个可能的行，C.ID 不是唯一的。</p><p>ref_or_null ：与 ref 类似，只不过这个选项包含对 NULL 的查询。</p><p>index_merge ：查询语句使用了两个以上的索引，比如经常在有 and 和 or 关键字出现的场景，但是在由于<strong>读取索引过多</strong>导致其性能有可能还不如 range（后面说）。</p><p>unique_subquery ：这个选项经常用在 in 关键字后面，子查询带有 where 关键字的子查询中，用 sql 来表示就是这样</p><figure class="highlight n1ql"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs n1ql">value IN (<span class="hljs-keyword">SELECT</span> primary_key <span class="hljs-keyword">FROM</span> single_table <span class="hljs-keyword">WHERE</span> some_expr)<br></code></pre></td></tr></table></figure><p>range ：<strong>索引范围查询</strong>，常见于使用 =，<>，>，>=，<，<=，IS NULL，<=>，BETWEEN，IN() 或者 like 等运算符的查询中。</p><p>index ：索引全表扫描，把索引从头到尾扫一遍。</p><p>all ：这个我们接触的最多了，就是全表查询，select * from xxx ，性能最差。</p></li></ul><p>上面就是 type 内容的大致解释，关于 type 我们经常会在 SQL 调优的环节使用 explain 分析其类型，然后改进查询方式，越靠近 system 其查询效率越高，越靠近 all 其查询效率越低。</p><p><img src="https://img-blog.csdnimg.cn/img_convert/af5bfcb5d65e0f0c7dec14a89a23d81e.png" alt="图片"></p><ul><li><p>possible_keys ：表示查询时，可能使用的索引。</p></li><li><p>key ：表示实际使用的索引。</p></li><li><p>key_len ：索引字段的长度。</p></li><li><p>rows ：扫描行的数量。</p></li><li><p>filtered ：通过查询条件查询出来的 SQL 数量占用总行数的比例。</p></li><li><p>extra ：执行情况的描述。</p></li></ul><p>通过上面的分析，我们可以大致确定 SQL 效率低的原因，一种非常有效的提升 SQL 查询效率的方式就是使用索引，接下来我会讲解一下如何使用索引提高查询效率。</p><h2 id="索引"><a href="#索引" class="headerlink" title="索引"></a>索引</h2><p>索引是数据库优化中最常用也是最重要的手段，通过使用不同的索引可以解决大多数 SQL 性能问题，也是面试经常会问到的优化方式，围绕着索引，面试官能让你造出火箭来，所以总结一点就是索引非常非常重！要！不只是使用，你还要懂其原！理！</p><h3 id="索引介绍"><a href="#索引介绍" class="headerlink" title="索引介绍"></a>索引介绍</h3><p>索引的目的就是用于快速查找某一列的数据，对相关数据列使用索引能够大大提高查询操作的性能。不使用索引，MySQL 必须从第一条记录开始读完整个表，直到找出相关的行，表越大查询数据所花费的时间就越多。如果表中查询的列有索引，MySQL 能够快速到达一个位置去搜索数据文件，而不必查看所有数据，那么将会节省很大一部分时间。</p><h3 id="索引分类"><a href="#索引分类" class="headerlink" title="索引分类"></a>索引分类</h3><p>先来了解一下索引都有哪些分类。</p><ul><li><p><code>全局索引(FULLTEXT)</code>：全局索引，目前只有 MyISAM 引擎支持全局索引，它的出现是为了解决针对文本的模糊查询效率较低的问题，并且只限于 CHAR、VARCHAR 和 TEXT 列。</p></li><li><p><code>哈希索引(HASH)</code>：哈希索引是 MySQL 中用到的唯一 key-value 键值对的数据结构，很适合作为索引。HASH 索引具有一次定位的好处，不需要像树那样逐个节点查找，但是这种查找适合应用于查找单个键的情况，对于范围查找，HASH 索引的性能就会很低。默认情况下，MEMORY 存储引擎使用 HASH 索引，但也支持 BTREE 索引。</p></li><li><p><code>B-Tree 索引</code>：B 就是 Balance 的意思，BTree 是一种平衡树，它有很多变种，最常见的就是 B+ Tree，它被 MySQL 广泛使用。</p></li><li><p><code>R-Tree 索引</code>：R-Tree 在 MySQL 很少使用，仅支持 geometry 数据类型，支持该类型的存储引擎只有MyISAM、BDb、InnoDb、NDb、Archive几种，相对于 B-Tree 来说，R-Tree 的优势在于范围查找。</p></li></ul><p>从逻辑上来对 MySQL 进行分类，主要分为下面这几种</p><ul><li><p>普通索引：普通索引是最基础的索引类型，它没有任何限制 。创建方式如下</p><figure class="highlight n1ql"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs n1ql"><span class="hljs-keyword">create</span> <span class="hljs-keyword">index</span> normal_index <span class="hljs-keyword">on</span> cxuan003(id);<br></code></pre></td></tr></table></figure><p><img src="https://img-blog.csdnimg.cn/img_convert/ff83638300efddbf54bba53bc5591f72.png" alt="图片"></p><p>删除方式</p><figure class="highlight n1ql"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs n1ql"><span class="hljs-keyword">drop</span> <span class="hljs-keyword">index</span> normal_index <span class="hljs-keyword">on</span> cxuan003;<br></code></pre></td></tr></table></figure><p><img src="https://img-blog.csdnimg.cn/img_convert/3d1e28a09fda0bd1320bcf7a7d8e535f.png" alt="图片"></p></li><li><p>唯一索引：唯一索引列的值必须唯一，允许有空值，如果是组合索引，则列值的组合必须唯一，创建方式如下</p><figure class="highlight n1ql"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs n1ql"><span class="hljs-keyword">create</span> <span class="hljs-keyword">unique</span> <span class="hljs-keyword">index</span> normal_index <span class="hljs-keyword">on</span> cxuan003(id);<br></code></pre></td></tr></table></figure><p><img src="https://img-blog.csdnimg.cn/img_convert/3197d902b85441d79053985591791795.png" alt="图片"></p></li><li><p>主键索引：是一种特殊的索引，一个表只能有一个主键，不允许有空值。一般是在建表的时候同时创建主键索引。</p><figure class="highlight sas"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><code class="hljs sas"><span class="hljs-keyword">CREATE</span> <span class="hljs-keyword">TABLE</span> <span class="hljs-keyword">table</span> (<br> id <span class="hljs-meta">int</span>(11) <span class="hljs-keyword">NOT</span> <span class="hljs-keyword">NULL</span> AUTO_INCREMENT ,<br> <span class="hljs-keyword">title</span> char(255) <span class="hljs-keyword">NOT</span> <span class="hljs-keyword">NULL</span> ,<br> <span class="hljs-keyword">PRIMARY</span> <span class="hljs-keyword">KEY</span> (id)<br>)<br></code></pre></td></tr></table></figure><p><img src="https://img-blog.csdnimg.cn/img_convert/3065fc966d06bd2c9fd82265328109fd.png" alt="图片"></p></li><li><p>组合索引：指多个字段上创建的索引，只有在查询条件中使用了创建索引时的第一个字段，索引才会被使用。使用组合索引时遵循最左前缀原则，下面我们就会创建组合索引。</p></li><li><p>全文索引：主要用来查找文本中的关键字，而不是直接与索引中的值相比较，目前只有 char、varchar，text 列上可以创建全文索引，创建表的适合添加全文索引</p><figure class="highlight pgsql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><code class="hljs pgsql"><span class="hljs-keyword">CREATE</span> <span class="hljs-keyword">TABLE</span> <span class="hljs-keyword">table</span> (<br> id <span class="hljs-type">int</span>(<span class="hljs-number">11</span>) <span class="hljs-keyword">NOT</span> <span class="hljs-keyword">NULL</span> AUTO_INCREMENT ,<br> title <span class="hljs-type">char</span>(<span class="hljs-number">255</span>) <span class="hljs-type">CHARACTER</span> <span class="hljs-keyword">NOT</span> <span class="hljs-keyword">NULL</span> ,<br> content <span class="hljs-type">text</span> <span class="hljs-type">CHARACTER</span> <span class="hljs-keyword">NULL</span> ,<br> <span class="hljs-type">time</span> <span class="hljs-type">int</span>(<span class="hljs-number">10</span>) <span class="hljs-keyword">NULL</span> <span class="hljs-keyword">DEFAULT</span> <span class="hljs-keyword">NULL</span> ,<br> <span class="hljs-keyword">PRIMARY KEY</span> (id),<br> FULLTEXT (content)<br>);<br></code></pre></td></tr></table></figure><p>当然也可以直接创建全局索引</p><figure class="highlight n1ql"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs n1ql"><span class="hljs-keyword">CREATE</span> FULLTEXT <span class="hljs-keyword">INDEX</span> index_content <span class="hljs-keyword">ON</span> article(content)<br></code></pre></td></tr></table></figure></li></ul><h3 id="索引使用"><a href="#索引使用" class="headerlink" title="索引使用"></a>索引使用</h3><p>索引可以在创建表的时候进行创建，也可以单独创建，下面我们采用单独创建的方式，我们在 cxuan004 上创建前缀索引</p><p><img src="https://img-blog.csdnimg.cn/img_convert/86c710e974bff5efb1e3372a8c506064.png" alt="图片"></p><p>我们使用 <code>explain</code> 进行分析，可以看到 cxuan004 使用索引的情况</p><p><img src="https://img-blog.csdnimg.cn/img_convert/92302060148d81ac76058bb9ed59c8fe.png" alt="图片"></p><p>如果不想使用索引，可以删除索引，索引的删除语法是</p><p><img src="https://img-blog.csdnimg.cn/img_convert/b1a310a78ad59670f45c6a5ceac8bc50.png" alt="图片"></p><p>索引使用细则</p><p>我们在 cxuan005 上根据 id 和 hash 创建一个复合索引，如下所示</p><figure class="highlight n1ql"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs n1ql"><span class="hljs-keyword">create</span> <span class="hljs-keyword">index</span> id_hash_index <span class="hljs-keyword">on</span> cxuan005(id,hash);<br></code></pre></td></tr></table></figure><p><img src="https://img-blog.csdnimg.cn/img_convert/ed7f8f7a8ce5c011f8ba3b7caad5bba9.png" alt="图片"></p><p>然后根据 id 进行执行计划的分析</p><figure class="highlight n1ql"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs n1ql"><span class="hljs-keyword">explain</span> <span class="hljs-keyword">select</span> * <span class="hljs-keyword">from</span> cxuan005 <span class="hljs-keyword">where</span> id = <span class="hljs-string">'333'</span>;<br></code></pre></td></tr></table></figure><p><img src="https://img-blog.csdnimg.cn/img_convert/c7bb3dc5a8850093384c494bf6bb0b2f.png" alt="图片"></p><p>可以发现，即使 where 条件中使用的不是复合索引（Id 、hash），索引仍然能够使用，这就是索引的前缀特性。但是如果只按照 hash 进行查询的话，索引就不会用到。</p><figure class="highlight n1ql"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs n1ql"><span class="hljs-keyword">explain</span> <span class="hljs-keyword">select</span> * <span class="hljs-keyword">from</span> cxuan005 <span class="hljs-keyword">where</span> hash=<span class="hljs-string">'8fd1f12575f6b39ee7c6d704eb54b353'</span>;<br></code></pre></td></tr></table></figure><p><img src="https://img-blog.csdnimg.cn/img_convert/2504cc78a4038c352b893c8605091846.png" alt="图片"></p><p>如果 where 条件使用了 like 查询，并且 <code>%</code> 不在第一个字符，索引才可能被使用。</p><p>对于复合索引来说，只能使用 id 进行 like 查询，因为 hash 列不管怎么查询都不会走索引。</p><figure class="highlight n1ql"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs n1ql"><span class="hljs-keyword">explain</span> <span class="hljs-keyword">select</span> * <span class="hljs-keyword">from</span> cxuan005 <span class="hljs-keyword">where</span> id <span class="hljs-keyword">like</span> <span class="hljs-string">'%1'</span>;<br></code></pre></td></tr></table></figure><p><img src="https://img-blog.csdnimg.cn/img_convert/d07b9c45713b4066fd774cf73e6e7102.png" alt="图片"></p><p>可以看到，如果第一个字符是 % ，则没有使用索引。</p><figure class="highlight n1ql"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs n1ql"><span class="hljs-keyword">explain</span> <span class="hljs-keyword">select</span> * <span class="hljs-keyword">from</span> cxuan005 <span class="hljs-keyword">where</span> id <span class="hljs-keyword">like</span> <span class="hljs-string">'1%'</span>;<br></code></pre></td></tr></table></figure><p><img src="https://img-blog.csdnimg.cn/img_convert/a8a2f477fd83cba63b1035686183b0d1.png" alt="图片"></p><p>如果使用了 % 号，就会触发索引。</p><p>如果列名是索引的话，那么对列名进行 NULL 查询，将会触发索引。</p><figure class="highlight n1ql"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs n1ql"><span class="hljs-keyword">explain</span> <span class="hljs-keyword">select</span> * <span class="hljs-keyword">from</span> cxuan005 <span class="hljs-keyword">where</span> id <span class="hljs-keyword">is</span> <span class="hljs-literal">null</span>;<br></code></pre></td></tr></table></figure><p><img src="https://img-blog.csdnimg.cn/20210611153727568.gif"></p><p>还有一些情况是存在索引但是 MySQL 并不会使用的情况。</p><ul><li><p>最简单的，如果使用索引后比不使用索引的效率还差，那么 MySQL 就不会使用索引。</p></li><li><p>如果 SQL 中使用了 OR 条件，OR 前的条件列有索引，而后面的列没有索引的话，那么涉及到的索引都不会使用，比如 cxuan005 表中，只有 id 和 hash 字段有索引，而 info 字段没有索引，那么我们使用 or 进行查询。</p><figure class="highlight pgsql"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs pgsql"><span class="hljs-keyword">explain</span> <span class="hljs-keyword">select</span> * <span class="hljs-keyword">from</span> cxuan005 <span class="hljs-keyword">where</span> id = <span class="hljs-number">111</span> <span class="hljs-keyword">and</span> <span class="hljs-keyword">info</span> = <span class="hljs-string">'cxuan'</span>;<br></code></pre></td></tr></table></figure></li></ul><p><img src="https://img-blog.csdnimg.cn/20210611153727569.gif"></p><ul><li><p>我们从 explain 的执行结果可以看到，虽然 possible_keys 选项上仍然有 id_hash_index 索引，但是从 key、key_len 可以得知，这条 SQL 语句并未使用索引。</p></li><li><p>在带有复合索引的列上查询不是第一列的数据，也不会使用索引。</p><figure class="highlight n1ql"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs n1ql"><span class="hljs-keyword">explain</span> <span class="hljs-keyword">select</span> * <span class="hljs-keyword">from</span> cxuan005 <span class="hljs-keyword">where</span> hash = <span class="hljs-string">'8fd1f12575f6b39ee7c6d704eb54b353'</span>;<br></code></pre></td></tr></table></figure><p><img src="https://img-blog.csdnimg.cn/img_convert/03a58d6c271ce79e80e7ad48db23fa34.png" alt="图片"></p></li><li><p>如果 where 条件的列参与了计算，那么也不会使用索引</p><figure class="highlight n1ql"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs n1ql"><span class="hljs-keyword">explain</span> <span class="hljs-keyword">select</span> * <span class="hljs-keyword">from</span> cxuan005 <span class="hljs-keyword">where</span> id + <span class="hljs-string">'111'</span> = <span class="hljs-string">'666'</span>;<br></code></pre></td></tr></table></figure><p><img src="https://img-blog.csdnimg.cn/img_convert/2721d3543748ece8a0a046effd71063b.png" alt="图片"></p></li><li><p>索引列使用函数，一样也不会使用索引</p><figure class="highlight pgsql"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs pgsql"><span class="hljs-keyword">explain</span> <span class="hljs-keyword">select</span> * <span class="hljs-keyword">from</span> cxuan005 <span class="hljs-keyword">where</span> concat(id,<span class="hljs-string">'111'</span>) = <span class="hljs-string">'666'</span>;<br></code></pre></td></tr></table></figure><p><img src="https://img-blog.csdnimg.cn/img_convert/ec5086d89874bafb56117264df1dbe62.png" alt="图片"></p></li><li><p>索引列使用了 like ，并且 <code>%</code> 位于第一个字符，则不会使用索引。</p></li><li><p>在 order by 操作中，排序的列同时也在 where 语句中，将不会使用索引。</p></li><li><p>当数据类型出现隐式转换时，比如 varchar 不加单引号可能转换为 int 类型时，会使索引无效，触发全表扫描。比如下面这两个例子能够显而易见的说明这一点</p><p><img src="https://img-blog.csdnimg.cn/img_convert/9d02aed0b543beca0f2e84ac7bc3c5be.png" alt="图片"></p></li><li><p>在索引列上使用 IS NOT NULL 操作</p><p><img src="https://img-blog.csdnimg.cn/img_convert/17062c9db14db1bd86a4f67ff492b256.png" alt="图片"></p></li><li><p>在索引字段上使用 <>，!=。不等于操作符是永远不会用到索引的，因此对它的处理只会产生全表扫描。</p><p><img src="https://img-blog.csdnimg.cn/img_convert/2685df910c22153926df2daaa0a61bf9.png" alt="图片"></p></li></ul><p>关于设置索引但是索引没有生效的场景还有很多，这个需要小伙伴们工作中不断总结和完善，不过我上面总结的这些索引失效的情景，能够覆盖大多数索引失效的场景了。</p><h3 id="查看索引的使用情况"><a href="#查看索引的使用情况" class="headerlink" title="查看索引的使用情况"></a>查看索引的使用情况</h3><p>在 MySQL 索引的使用过程中，有一个 <code>Handler_read_key</code> 值，这个值表示了<strong>某一行被索引值读的次数</strong>。Handler_read_key 的值比较低的话，则表明增加索引得到的性能改善不是很理想，可能索引使用的频率不高。</p><p>还有一个值是 <code>Handler_read_rnd_next</code>，这个值高则意味着查询运行效率不高，应该建立索引来进行抢救。这个值的含义是在数据文件中读下一行的请求数。如果正在进行大量的表扫描，Handler_read_rnd_next 的值比较高，就说明表索引不正确或写入的查询没有利用索引。</p><p><img src="https://img-blog.csdnimg.cn/img_convert/e304165a9e6cc79681c6fa5b090f58eb.png" alt="图片"></p><h2 id="MySQL-分析表、检查表和优化表"><a href="#MySQL-分析表、检查表和优化表" class="headerlink" title="MySQL 分析表、检查表和优化表"></a>MySQL 分析表、检查表和优化表</h2><p>对于大多数开发者来说，他们更倾向于解决简单 SQL的优化，而复杂 SQL 的优化交给了公司的 DBA 来做。</p><p>下面就从普通程序员的角度和你聊几个简单的优化方式。</p><h3 id="MySQL-分析表"><a href="#MySQL-分析表" class="headerlink" title="MySQL 分析表"></a>MySQL 分析表</h3><p>分析表用于分析和存储表的关键字分布，分析的结果可以使得系统得到准确的统计信息，使得 SQL 生成正确的执行计划。如果用于感觉实际执行计划与预期不符，可以执行分析表来解决问题，分析表语法如下</p><figure class="highlight pgsql"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs pgsql"><span class="hljs-keyword">analyze</span> <span class="hljs-keyword">table</span> cxuan005;<br></code></pre></td></tr></table></figure><p><img src="https://img-blog.csdnimg.cn/img_convert/c49e8e83471a76c90e8d61c481d2bcf3.png" alt="图片"></p><p>分析结果涉及到的字段属性如下</p><p>Table：表示表的名称；</p><p>Op：表示执行的操作，analyze 表示进行分析操作，check 表示进行检查查找，optimize 表示进行优化操作；</p><p>Msg_type：表示信息类型，其显示的值通常是状态、警告、错误和信息这四者之一；</p><p>Msg_text：显示信息。</p><p>对表的定期分析可以改善性能，应该成为日常工作的一部分。因为通过更新表的索引信息对表进行分析，可改善数据库性能。</p><h3 id="MySQL-检查表"><a href="#MySQL-检查表" class="headerlink" title="MySQL 检查表"></a>MySQL 检查表</h3><p>数据库经常可能遇到错误，比如数据写入磁盘时发生错误，或是索引没有同步更新，或是数据库未关闭 MySQL 就停止了。遇到这些情况，数据就可能发生错误： <strong>Incorrect key file for table: ‘ ‘. Try to repair it</strong>. 此时，我们可以使用 Check Table 语句来检查表及其对应的索引。</p><figure class="highlight pgsql"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs pgsql"><span class="hljs-keyword">check</span> <span class="hljs-keyword">table</span> cxuan005;<br></code></pre></td></tr></table></figure><p><img src="https://img-blog.csdnimg.cn/img_convert/0b2e1f7bc2c7acfbc25401d8c681cb93.png" alt="图片"></p><p>检查表的主要目的就是检查一个或者多个表是否有错误。Check Table 对 MyISAM 和 InnoDB 表有作用。Check Table 也可以检查视图的错误。</p><h3 id="MySQL-优化表"><a href="#MySQL-优化表" class="headerlink" title="MySQL 优化表"></a>MySQL 优化表</h3><p>MySQL 优化表适用于删除了大量的表数据，或者对包含 VARCHAR、BLOB 或则 TEXT 命令进行大量修改的情况。MySQL 优化表可以将大量的空间碎片进行合并，消除由于删除或者更新造成的空间浪费情况。它的命令如下</p><figure class="highlight abnf"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs abnf">optimize table cxuan005<span class="hljs-comment">;</span><br></code></pre></td></tr></table></figure><p><img src="https://img-blog.csdnimg.cn/img_convert/ceb7a348cb85e7d2315f62759a6052c7.png" alt="图片"></p><p>我的存储引擎是 InnoDB 引擎，但是从图可以知道，InnoDB 不支持使用 optimize 优化，建议使用 recreate + analyze 进行优化。optimize 命令只对 MyISAM 、BDB 表起作用。</p>]]></content>
<categories>
<category>SQL</category>
</categories>
<tags>
<tag>mysql</tag>
</tags>
</entry>
<entry>
<title>Dockerfile 最佳实践(Dockerfile指令的使用及建议)</title>
<link href="/2020/12/14/Dockerfile%20%E6%9C%80%E4%BD%B3%E5%AE%9E%E8%B7%B5(Dockerfile%E6%8C%87%E4%BB%A4%E7%9A%84%E4%BD%BF%E7%94%A8%E5%8F%8A%E5%BB%BA%E8%AE%AE)/"/>
<url>/2020/12/14/Dockerfile%20%E6%9C%80%E4%BD%B3%E5%AE%9E%E8%B7%B5(Dockerfile%E6%8C%87%E4%BB%A4%E7%9A%84%E4%BD%BF%E7%94%A8%E5%8F%8A%E5%BB%BA%E8%AE%AE)/</url>
<content type="html"><![CDATA[<span id="more"></span><h1 id="目录"><a href="#目录" class="headerlink" title="目录"></a><strong>目录</strong></h1><p> </p><p><a href="#dockerfile-%E6%9C%80%E4%BD%B3%E5%AE%9E%E8%B7%B5">Dockerfile 最佳实践</a></p><p><a href="#%E4%B8%80%E8%88%AC%E6%80%A7%E7%9A%84%E6%8C%87%E5%8D%97%E5%92%8C%E5%BB%BA%E8%AE%AE">一般性的指南和建议</a></p><p><a href="#%E5%AE%B9%E5%99%A8%E5%BA%94%E8%AF%A5%E6%98%AF%E7%9F%AD%E6%9A%82%E7%9A%84">容器应该是短暂的</a></p><p><a href="#%E4%BD%BF%E7%94%A8-dockerignore-%E6%96%87%E4%BB%B6">使用 .dockerignore 文件</a></p><p><a href="#%E4%BD%BF%E7%94%A8%E5%A4%9A%E9%98%B6%E6%AE%B5%E6%9E%84%E5%BB%BA">使用多阶段构建</a></p><p><a href="#%E9%81%BF%E5%85%8D%E5%AE%89%E8%A3%85%E4%B8%8D%E5%BF%85%E8%A6%81%E7%9A%84%E5%8C%85">避免安装不必要的包</a></p><p><a href="#%E4%B8%80%E4%B8%AA%E5%AE%B9%E5%99%A8%E5%8F%AA%E8%BF%90%E8%A1%8C%E4%B8%80%E4%B8%AA%E8%BF%9B%E7%A8%8B">一个容器只运行一个进程</a></p><p><a href="#%E9%95%9C%E5%83%8F%E5%B1%82%E6%95%B0%E5%B0%BD%E5%8F%AF%E8%83%BD%E5%B0%91">镜像层数尽可能少</a></p><p><a href="#%E5%B0%86%E5%A4%9A%E8%A1%8C%E5%8F%82%E6%95%B0%E6%8E%92%E5%BA%8F">将多行参数排序</a></p><p><a href="#%E6%9E%84%E5%BB%BA%E7%BC%93%E5%AD%98">构建缓存</a></p><p><a href="#dockerfile-%E6%8C%87%E4%BB%A4">Dockerfile 指令</a></p><p><a href="#from">FROM</a></p><p><a href="#label">LABEL</a></p><p><a href="#run">RUN</a></p><p><a href="#cmd">CMD</a></p><p><a href="#expose">EXPOSE</a></p><p><a href="#env">ENV</a></p><p><a href="#add-%E5%92%8C-copy">ADD 和 COPY</a></p><p><a href="#entrypoint">ENTRYPOINT</a></p><p><a href="#volume">VOLUME</a></p><p><a href="#user">USER</a></p><p><a href="#workdir">WORKDIR</a></p><p><a href="#%E5%AE%98%E6%96%B9%E9%95%9C%E5%83%8F%E7%A4%BA%E4%BE%8B">官方镜像示例</a></p><p><a href="#%E8%B5%84%E6%BA%90%E9%93%BE%E6%8E%A5">资源链接</a></p><p><a href="#%E5%AE%98%E6%96%B9%E7%BD%91%E7%AB%99">官方网站</a></p><p><a href="#%E5%AE%9E%E8%B7%B5%E5%8F%82%E8%80%83">实践参考</a></p><p><a href="#%E6%8A%80%E6%9C%AF%E4%BA%A4%E6%B5%81">技术交流</a></p><p><a href="#%E5%85%B6%E5%AE%83">其它</a></p><p><a href="#%E5%8F%82%E8%80%83%C2%A0">参考 </a></p><hr><h1 id="Dockerfile-最佳实践"><a href="#Dockerfile-最佳实践" class="headerlink" title="Dockerfile 最佳实践"></a>Dockerfile 最佳实践</h1><p>本附录是对 Docker 官方文档中 <a href="https://docs.docker.com/develop/develop-images/dockerfile_best-practices/">Best practices for writing Dockerfiles</a> 的理解与翻译。</p><h2 id="一般性的指南和建议"><a href="#一般性的指南和建议" class="headerlink" title="一般性的指南和建议"></a>一般性的指南和建议</h2><h3 id="容器应该是短暂的"><a href="#容器应该是短暂的" class="headerlink" title="容器应该是短暂的"></a>容器应该是短暂的</h3><p>通过 <code>Dockerfile</code> 构建的镜像所启动的容器应该尽可能短暂（生命周期短）。「短暂」意味着可以停止和销毁容器，并且创建一个新容器并部署好所需的设置和配置工作量应该是极小的。</p><h3 id="使用-dockerignore-文件"><a href="#使用-dockerignore-文件" class="headerlink" title="使用 .dockerignore 文件"></a>使用 <code>.dockerignore</code> 文件</h3><p>使用 <code>Dockerfile</code> 构建镜像时最好是将 <code>Dockerfile</code> 放置在一个新建的空目录下。然后将构建镜像所需要的文件添加到该目录中。为了提高构建镜像的效率，你可以在目录下新建一个 <code>.dockerignore</code> 文件来指定要忽略的文件和目录。<code>.dockerignore</code> 文件的排除模式语法和 Git 的 <code>.gitignore</code> 文件相似。</p><h3 id="使用多阶段构建"><a href="#使用多阶段构建" class="headerlink" title="使用多阶段构建"></a>使用多阶段构建</h3><p>在 <code>Docker 17.05</code> 以上版本中，你可以使用 多阶段构建 来减少所构建镜像的大小。</p><h3 id="避免安装不必要的包"><a href="#避免安装不必要的包" class="headerlink" title="避免安装不必要的包"></a>避免安装不必要的包</h3><p>为了降低复杂性、减少依赖、减小文件大小、节约构建时间，你应该避免安装任何不必要的包。例如，不要在数据库镜像中包含一个文本编辑器。</p><h3 id="一个容器只运行一个进程"><a href="#一个容器只运行一个进程" class="headerlink" title="一个容器只运行一个进程"></a>一个容器只运行一个进程</h3><p>应该保证在一个容器中只运行一个进程。将多个应用解耦到不同容器中，保证了容器的横向扩展和复用。例如 web 应用应该包含三个容器：web应用、数据库、缓存。</p><p>如果容器互相依赖，你可以使用 Docker 自定义网络 来把这些容器连接起来。</p><h3 id="镜像层数尽可能少"><a href="#镜像层数尽可能少" class="headerlink" title="镜像层数尽可能少"></a>镜像层数尽可能少</h3><p>你需要在 <code>Dockerfile</code> 可读性（也包括长期的可维护性）和减少层数之间做一个平衡。</p><h3 id="将多行参数排序"><a href="#将多行参数排序" class="headerlink" title="将多行参数排序"></a>将多行参数排序</h3><p>将多行参数按字母顺序排序（比如要安装多个包时）。这可以帮助你避免重复包含同一个包，更新包列表时也更容易。也便于 <code>PRs</code> 阅读和审查。建议在反斜杠符号 <code>\</code> 之前添加一个空格，以增加可读性。</p><p>下面是来自 <code>buildpack-deps</code> 镜像的例子：</p><figure class="highlight routeros"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><code class="hljs routeros"><span class="hljs-built_in">RUN</span> apt-<span class="hljs-built_in">get</span> update && apt-<span class="hljs-built_in">get</span> install -y \<br> bzr \<br> cvs \<br> git \<br> mercurial \<br> subversion<br></code></pre></td></tr></table></figure><h3 id="构建缓存"><a href="#构建缓存" class="headerlink" title="构建缓存"></a>构建缓存</h3><p>在镜像的构建过程中，Docker 会遍历 <code>Dockerfile</code> 文件中的指令，然后按顺序执行。在执行每条指令之前，Docker 都会在缓存中查找是否已经存在可重用的镜像，如果有就使用现存的镜像，不再重复创建。如果你不想在构建过程中使用缓存，你可以在 <code>docker build</code> 命令中使用 <code>--no-cache=true</code> 选项。</p><p>但是，如果你想在构建的过程中使用缓存，你得明白什么时候会，什么时候不会找到匹配的镜像，遵循的基本规则如下：</p><ul><li>从一个基础镜像开始（<code>FROM</code> 指令指定），下一条指令将和该基础镜像的所有子镜像进行匹配，检查这些子镜像被创建时使用的指令是否和被检查的指令完全一样。如果不是，则缓存失效。</li><li>在大多数情况下，只需要简单地对比 <code>Dockerfile</code> 中的指令和子镜像。然而，有些指令需要更多的检查和解释。</li><li>对于 <code>ADD</code> 和 <code>COPY</code> 指令，镜像中对应文件的内容也会被检查，每个文件都会计算出一个校验和。文件的最后修改时间和最后访问时间不会纳入校验。在缓存的查找过程中，会将这些校验和和已存在镜像中的文件校验和进行对比。如果文件有任何改变，比如内容和元数据，则缓存失效。</li><li>除了 <code>ADD</code> 和 <code>COPY</code> 指令，缓存匹配过程不会查看临时容器中的文件来决定缓存是否匹配。例如，当执行完 <code>RUN apt-get -y update</code> 指令后，容器中一些文件被更新，但 Docker 不会检查这些文件。这种情况下，只有指令字符串本身被用来匹配缓存。</li></ul><p>一旦缓存失效，所有后续的 <code>Dockerfile</code> 指令都将产生新的镜像，缓存不会被使用。</p><h2 id="Dockerfile-指令"><a href="#Dockerfile-指令" class="headerlink" title="Dockerfile 指令"></a>Dockerfile 指令</h2><p>下面针对 <code>Dockerfile</code> 中各种指令的最佳编写方式给出建议。</p><h3 id="FROM"><a href="#FROM" class="headerlink" title="FROM"></a>FROM</h3><p>尽可能使用当前官方仓库作为你构建镜像的基础。推荐使用 <a href="https://hub.docker.com/_/alpine/">Alpine</a> 镜像，因为它被严格控制并保持最小尺寸（目前小于 5 MB），但它仍然是一个完整的发行版。</p><h3 id="LABEL"><a href="#LABEL" class="headerlink" title="LABEL"></a>LABEL</h3><p>你可以给镜像添加标签来帮助组织镜像、记录许可信息、辅助自动化构建等。每个标签一行，由 <code>LABEL</code> 开头加上一个或多个标签对。下面的示例展示了各种不同的可能格式。<code>#</code> 开头的行是注释内容。</p><blockquote><p>注意：如果你的字符串中包含空格，必须将字符串放入引号中或者对空格使用转义。如果字符串内容本身就包含引号，必须对引号使用转义。</p></blockquote><figure class="highlight stata"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><code class="hljs stata"># <span class="hljs-keyword">Set</span> <span class="hljs-keyword">one</span> or <span class="hljs-keyword">more</span> individual labels<br><span class="hljs-keyword">LABEL</span> com.example.<span class="hljs-keyword">version</span>=<span class="hljs-string">"0.0.1-beta"</span><br><br><span class="hljs-keyword">LABEL</span> vendor=<span class="hljs-string">"ACME Incorporated"</span><br><br><span class="hljs-keyword">LABEL</span> com.example.release-date=<span class="hljs-string">"2015-02-12"</span><br><br><span class="hljs-keyword">LABEL</span> com.example.<span class="hljs-keyword">version</span>.is-production=<span class="hljs-string">""</span><br></code></pre></td></tr></table></figure><p>一个镜像可以包含多个标签，但建议将多个标签放入到一个 <code>LABEL</code> 指令中。</p><figure class="highlight livescript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><code class="hljs livescript"><span class="hljs-comment"># Set multiple labels at once, using line-continuation characters to break long lines</span><br>LABEL vendor=ACME<span class="hljs-string">\</span> Incorporated <span class="hljs-string">\</span><br> com.example.<span class="hljs-keyword">is</span>-beta= <span class="hljs-string">\</span><br> com.example.<span class="hljs-keyword">is</span>-production=<span class="hljs-string">""</span> <span class="hljs-string">\</span><br> com.example.version=<span class="hljs-string">"0.0.1-beta"</span> <span class="hljs-string">\</span><br> com.example.release-date=<span class="hljs-string">"2015-02-12"</span><br></code></pre></td></tr></table></figure><p>关于标签可以接受的键值对，参考 <a href="https://docs.docker.com/config/labels-custom-metadata/">Understanding object labels</a>。关于查询标签信息，参考 <a href="https://docs.docker.com/config/labels-custom-metadata/">Managing labels on objects</a>。</p><h3 id="RUN"><a href="#RUN" class="headerlink" title="RUN"></a>RUN</h3><p>为了保持 <code>Dockerfile</code> 文件的可读性，可理解性，以及可维护性，建议将长的或复杂的 <code>RUN</code> 指令用反斜杠 <code>\</code> 分割成多行。</p><p>apt-get</p><p><code>RUN</code> 指令最常见的用法是安装包用的 <code>apt-get</code>。因为 <code>RUN apt-get</code> 指令会安装包，所以有几个问题需要注意。</p><p>不要使用 <code>RUN apt-get upgrade</code> 或 <code>dist-upgrade</code>，因为许多基础镜像中的「必须」包不会在一个非特权容器中升级。如果基础镜像中的某个包过时了，你应该联系它的维护者。如果你确定某个特定的包，比如 <code>foo</code>，需要升级，使用 <code>apt-get install \-y foo</code> 就行，该指令会自动升级 <code>foo</code> 包。</p><p>永远将 <code>RUN apt-get update</code> 和 <code>apt-get install</code> 组合成一条 <code>RUN</code> 声明，例如：</p><figure class="highlight routeros"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><code class="hljs routeros"><span class="hljs-built_in">RUN</span> apt-<span class="hljs-built_in">get</span> update && apt-<span class="hljs-built_in">get</span> install -y \<br> package-bar \<br> package-baz \<br> package-foo<br></code></pre></td></tr></table></figure><p>将 <code>apt-get update</code> 放在一条单独的 <code>RUN</code> 声明中会导致缓存问题以及后续的 <code>apt-get install</code> 失败。比如，假设你有一个 <code>Dockerfile</code> 文件：</p><figure class="highlight routeros"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><code class="hljs routeros"><span class="hljs-keyword">FROM</span> ubuntu:18.04<br><br><span class="hljs-built_in">RUN</span> apt-<span class="hljs-built_in">get</span> update<br><br><span class="hljs-built_in">RUN</span> apt-<span class="hljs-built_in">get</span> install -y curl<br></code></pre></td></tr></table></figure><p>构建镜像后，所有的层都在 Docker 的缓存中。假设你后来又修改了其中的 <code>apt-get install</code> 添加了一个包：</p><figure class="highlight routeros"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><code class="hljs routeros"><span class="hljs-keyword">FROM</span> ubuntu:18.04<br><br><span class="hljs-built_in">RUN</span> apt-<span class="hljs-built_in">get</span> update<br><br><span class="hljs-built_in">RUN</span> apt-<span class="hljs-built_in">get</span> install -y curl nginx<br></code></pre></td></tr></table></figure><p>Docker 发现修改后的 <code>RUN apt-get update</code> 指令和之前的完全一样。所以，<code>apt-get update</code> 不会执行，而是使用之前的缓存镜像。因为 <code>apt-get update</code> 没有运行，后面的 <code>apt-get install</code> 可能安装的是过时的 <code>curl</code> 和 <code>nginx</code> 版本。</p><p>使用 <code>RUN apt-get update && apt-get install \-y</code> 可以确保你的 Dockerfiles 每次安装的都是包的最新的版本，而且这个过程不需要进一步的编码或额外干预。这项技术叫作 <code>cache busting</code>。你也可以显示指定一个包的版本号来达到 <code>cache-busting</code>，这就是所谓的固定版本，例如：</p><figure class="highlight routeros"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><code class="hljs routeros"><span class="hljs-built_in">RUN</span> apt-<span class="hljs-built_in">get</span> update && apt-<span class="hljs-built_in">get</span> install -y \<br> package-bar \<br> package-baz \<br> <span class="hljs-attribute">package-foo</span>=1.3.*<br></code></pre></td></tr></table></figure><p>固定版本会迫使构建过程检索特定的版本，而不管缓存中有什么。这项技术也可以减少因所需包中未预料到的变化而导致的失败。</p><p>下面是一个 <code>RUN</code> 指令的示例模板，展示了所有关于 <code>apt-get</code> 的建议。</p><figure class="highlight livescript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><code class="hljs livescript">RUN apt-get update && apt-get install -y <span class="hljs-string">\</span><br> aufs-tools <span class="hljs-string">\</span><br> automake <span class="hljs-string">\</span><br> build-essential <span class="hljs-string">\</span><br> curl <span class="hljs-string">\</span><br> dpkg-sig <span class="hljs-string">\</span><br> libcap-dev <span class="hljs-string">\</span><br> libsqlite3-dev <span class="hljs-string">\</span><br> mercurial <span class="hljs-string">\</span><br> reprepro <span class="hljs-string">\</span><br> ruby1.<span class="hljs-number">9.1</span> <span class="hljs-string">\</span><br> ruby1.<span class="hljs-number">9.1</span>-dev <span class="hljs-string">\</span><br> s3cmd=<span class="hljs-number">1.1</span>.* <span class="hljs-string">\</span><br> && rm -rf <span class="hljs-regexp">/var/lib/apt/lists/</span>*<br></code></pre></td></tr></table></figure><p>其中 <code>s3cmd</code> 指令指定了一个版本号 <code>1.1.*</code>。如果之前的镜像使用的是更旧的版本，指定新的版本会导致 <code>apt-get udpate</code> 缓存失效并确保安装的是新版本。</p><p>另外，清理掉 apt 缓存 <code>var/lib/apt/lists</code> 可以减小镜像大小。因为 <code>RUN</code> 指令的开头为 <code>apt-get udpate</code>，包缓存总是会在 <code>apt-get install</code> 之前刷新。</p><blockquote><p>注意：官方的 Debian 和 Ubuntu 镜像会自动运行 apt-get clean，所以不需要显式的调用 apt-get clean。</p></blockquote><h3 id="CMD"><a href="#CMD" class="headerlink" title="CMD"></a>CMD</h3><p><code>CMD</code> 指令用于执行目标镜像中包含的软件，可以包含参数。<code>CMD</code> 大多数情况下都应该以 <code>CMD ["executable", "param1", "param2"...]</code> 的形式使用。因此，如果创建镜像的目的是为了部署某个服务(比如 <code>Apache</code>)，你可能会执行类似于 <code>CMD ["apache2", "-DFOREGROUND"]</code> 形式的命令。我们建议任何服务镜像都使用这种形式的命令。</p><p>多数情况下，<code>CMD</code> 都需要一个交互式的 <code>shell</code> (bash, Python, perl 等)，例如 <code>CMD ["perl", "-de0"]</code>，或者 <code>CMD ["PHP", "-a"]</code>。使用这种形式意味着，当你执行类似 <code>docker run \-it python</code> 时，你会进入一个准备好的 <code>shell</code> 中。<code>CMD</code> 应该在极少的情况下才能以 <code>CMD ["param", "param"]</code> 的形式与 <code>ENTRYPOINT</code> 协同使用，除非你和你的镜像使用者都对 <code>ENTRYPOINT</code> 的工作方式十分熟悉。</p><h3 id="EXPOSE"><a href="#EXPOSE" class="headerlink" title="EXPOSE"></a>EXPOSE</h3><p><code>EXPOSE</code> 指令用于指定容器将要监听的端口。因此，你应该为你的应用程序使用常见的端口。例如，提供 <code>Apache</code> web 服务的镜像应该使用 <code>EXPOSE 80</code>，而提供 <code>MongoDB</code> 服务的镜像使用 <code>EXPOSE 27017</code>。</p><p>对于外部访问，用户可以在执行 <code>docker run</code> 时使用一个标志来指示如何将指定的端口映射到所选择的端口。</p><h3 id="ENV"><a href="#ENV" class="headerlink" title="ENV"></a>ENV</h3><p>为了方便新程序运行，你可以使用 <code>ENV</code> 来为容器中安装的程序更新 <code>PATH</code> 环境变量。例如使用 <code>ENV PATH /usr/local/nginx/bin:$PATH</code> 来确保 <code>CMD ["nginx"]</code> 能正确运行。</p><p><code>ENV</code> 指令也可用于为你想要容器化的服务提供必要的环境变量，比如 Postgres 需要的 <code>PGDATA</code>。</p><p>最后，<code>ENV</code> 也能用于设置常见的版本号，比如下面的示例：</p><figure class="highlight pgsql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><code class="hljs pgsql">ENV PG_MAJOR <span class="hljs-number">9.3</span><br><br>ENV PG_VERSION <span class="hljs-number">9.3</span><span class="hljs-number">.4</span><br><br>RUN curl -SL http://example.com/postgres-$PG_VERSION.tar.xz | tar -xJC /usr/src/postgress && …<br><br>ENV <span class="hljs-type">PATH</span> /usr/<span class="hljs-keyword">local</span>/postgres-$PG_MAJOR/bin:$<span class="hljs-type">PATH</span><br></code></pre></td></tr></table></figure><p>类似于程序中的常量，这种方法可以让你只需改变 <code>ENV</code> 指令来自动的改变容器中的软件版本。</p><h3 id="ADD-和-COPY"><a href="#ADD-和-COPY" class="headerlink" title="ADD 和 COPY"></a>ADD 和 COPY</h3><p>虽然 <code>ADD</code> 和 <code>COPY</code> 功能类似，但一般优先使用 <code>COPY</code>。因为它比 <code>ADD</code> 更透明。<code>COPY</code> 只支持简单将本地文件拷贝到容器中，而 <code>ADD</code> 有一些并不明显的功能（比如本地 tar 提取和远程 URL 支持）。因此，<code>ADD</code> 的最佳用例是将本地 tar 文件自动提取到镜像中，例如 <code>ADD rootfs.tar.xz</code>。</p><p>如果你的 <code>Dockerfile</code> 有多个步骤需要使用上下文中不同的文件。单独 <code>COPY</code> 每个文件，而不是一次性的 <code>COPY</code> 所有文件，这将保证每个步骤的构建缓存只在特定的文件变化时失效。例如：</p><figure class="highlight gradle"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><code class="hljs gradle"><span class="hljs-keyword">COPY</span> requirements.txt <span class="hljs-regexp">/tmp/</span><br><br>RUN pip install --requirement <span class="hljs-regexp">/tmp/</span>requirements.txt<br><br><span class="hljs-keyword">COPY</span> . <span class="hljs-regexp">/tmp/</span><br></code></pre></td></tr></table></figure><p>如果将 <code>COPY . /tmp/</code> 放置在 <code>RUN</code> 指令之前，只要 <code>.</code> 目录中任何一个文件变化，都会导致后续指令的缓存失效。</p><p>为了让镜像尽量小，最好不要使用 <code>ADD</code> 指令从远程 URL 获取包，而是使用 <code>curl</code> 和 <code>wget</code>。这样你可以在文件提取完之后删掉不再需要的文件来避免在镜像中额外添加一层。比如尽量避免下面的用法：</p><figure class="highlight awk"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><code class="hljs awk">ADD http:<span class="hljs-regexp">//</span>example.com<span class="hljs-regexp">/big.tar.xz /u</span>sr<span class="hljs-regexp">/src/</span>things/<br><br>RUN tar -xJf <span class="hljs-regexp">/usr/</span>src<span class="hljs-regexp">/things/</span>big.tar.xz -C <span class="hljs-regexp">/usr/</span>src/things<br><br>RUN make -C <span class="hljs-regexp">/usr/</span>src/things all<br></code></pre></td></tr></table></figure><p>而是应该使用下面这种方法：</p><figure class="highlight awk"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><code class="hljs awk">RUN mkdir -p <span class="hljs-regexp">/usr/</span>src/things \<br> && curl -SL http:<span class="hljs-regexp">//</span>example.com/big.tar.xz \<br> | tar -xJC <span class="hljs-regexp">/usr/</span>src/things \<br> && make -C <span class="hljs-regexp">/usr/</span>src/things all<br></code></pre></td></tr></table></figure><p>上面使用的管道操作，所以没有中间文件需要删除。</p><p>对于其他不需要 <code>ADD</code> 的自动提取功能的文件或目录，你应该使用 <code>COPY</code>。</p><h3 id="ENTRYPOINT"><a href="#ENTRYPOINT" class="headerlink" title="ENTRYPOINT"></a>ENTRYPOINT</h3><p><code>ENTRYPOINT</code> 的最佳用处是设置镜像的主命令，允许将镜像当成命令本身来运行（用 <code>CMD</code> 提供默认选项）。</p><p>例如，下面的示例镜像提供了命令行工具 <code>s3cmd</code>:</p><figure class="highlight css"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><code class="hljs css">ENTRYPOINT <span class="hljs-selector-attr">[<span class="hljs-string">"s3cmd"</span>]</span><br><br>CMD <span class="hljs-selector-attr">[<span class="hljs-string">"--help"</span>]</span><br></code></pre></td></tr></table></figure><p>现在直接运行该镜像创建的容器会显示命令帮助：</p><figure class="highlight routeros"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs routeros">$ docker <span class="hljs-built_in">run</span> s3cmd<br></code></pre></td></tr></table></figure><p>或者提供正确的参数来执行某个命令：</p><figure class="highlight dockerfile"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs dockerfile">$ docker <span class="hljs-keyword">run</span><span class="language-bash"> s3cmd <span class="hljs-built_in">ls</span> s3://mybucket</span><br></code></pre></td></tr></table></figure><p>这样镜像名可以当成命令行的参考。</p><p><code>ENTRYPOINT</code> 指令也可以结合一个辅助脚本使用，和前面命令行风格类似，即使启动工具需要不止一个步骤。</p><p>例如，<code>Postgres</code> 官方镜像使用下面的脚本作为 <code>ENTRYPOINT</code>：</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><code class="hljs bash"><span class="hljs-meta">#!/bin/bash</span><br><span class="hljs-built_in">set</span> -e<br><br><span class="hljs-keyword">if</span> [ <span class="hljs-string">"<span class="hljs-variable">$1</span>"</span> = <span class="hljs-string">'postgres'</span> ]; <span class="hljs-keyword">then</span><br> <span class="hljs-built_in">chown</span> -R postgres <span class="hljs-string">"<span class="hljs-variable">$PGDATA</span>"</span><br><br> <span class="hljs-keyword">if</span> [ -z <span class="hljs-string">"<span class="hljs-subst">$(ls -A <span class="hljs-string">"<span class="hljs-variable">$PGDATA</span>"</span>)</span>"</span> ]; <span class="hljs-keyword">then</span><br> gosu postgres initdb<br> <span class="hljs-keyword">fi</span><br><br> <span class="hljs-built_in">exec</span> gosu postgres <span class="hljs-string">"<span class="hljs-variable">$@</span>"</span><br><span class="hljs-keyword">fi</span><br><br><span class="hljs-built_in">exec</span> <span class="hljs-string">"<span class="hljs-variable">$@</span>"</span><br></code></pre></td></tr></table></figure><blockquote><p>注意：该脚本使用了 Bash 的内置命令 exec，所以最后运行的进程就是容器的 PID 为 1 的进程。这样，进程就可以接收到任何发送给容器的 Unix 信号了。</p></blockquote><p>该辅助脚本被拷贝到容器，并在容器启动时通过 <code>ENTRYPOINT</code> 执行：</p><figure class="highlight dockerfile"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><code class="hljs dockerfile"><span class="hljs-keyword">COPY</span><span class="language-bash"> ./docker-entrypoint.sh /</span><br><br><span class="hljs-keyword">ENTRYPOINT</span><span class="language-bash"> [<span class="hljs-string">"/docker-entrypoint.sh"</span>]</span><br></code></pre></td></tr></table></figure><p>该脚本可以让用户用几种不同的方式和 <code>Postgres</code> 交互。</p><p>你可以很简单地启动 <code>Postgres</code>：</p><figure class="highlight routeros"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs routeros">$ docker <span class="hljs-built_in">run</span> postgres<br></code></pre></td></tr></table></figure><p>也可以执行 <code>Postgres</code> 并传递参数：</p><figure class="highlight applescript"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs applescript">$ docker <span class="hljs-built_in">run</span> postgres postgres <span class="hljs-comment">--help</span><br></code></pre></td></tr></table></figure><p>最后，你还可以启动另外一个完全不同的工具，比如 <code>Bash</code>：</p><figure class="highlight applescript"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs applescript">$ docker <span class="hljs-built_in">run</span> <span class="hljs-comment">--rm -it postgres bash</span><br></code></pre></td></tr></table></figure><h3 id="VOLUME"><a href="#VOLUME" class="headerlink" title="VOLUME"></a>VOLUME</h3><p><code>VOLUME</code> 指令用于暴露任何数据库存储文件，配置文件，或容器创建的文件和目录。强烈建议使用 <code>VOLUME</code> 来管理镜像中的可变部分和用户可以改变的部分。</p><h3 id="USER"><a href="#USER" class="headerlink" title="USER"></a>USER</h3><p>如果某个服务不需要特权执行，建议使用 <code>USER</code> 指令切换到非 root 用户。先在 <code>Dockerfile</code> 中使用类似 <code>RUN groupadd \-r postgres && useradd \-r \-g postgres postgres</code> 的指令创建用户和用户组。</p><blockquote><p>注意：在镜像中，用户和用户组每次被分配的 UID/GID 都是不确定的，下次重新构建镜像时被分配到的 UID/GID 可能会不一样。如果要依赖确定的 UID/GID，你应该显示的指定一个 UID/GID。</p></blockquote><p>你应该避免使用 <code>sudo</code>，因为它不可预期的 TTY 和信号转发行为可能造成的问题比它能解决的问题还多。如果你真的需要和 <code>sudo</code> 类似的功能（例如，以 root 权限初始化某个守护进程，以非 root 权限执行它），你可以使用 <a href="https://github.com/tianon/gosu">gosu</a>。</p><p>最后，为了减少层数和复杂度，避免频繁地使用 <code>USER</code> 来回切换用户。</p><h3 id="WORKDIR"><a href="#WORKDIR" class="headerlink" title="WORKDIR"></a>WORKDIR</h3><p>为了清晰性和可靠性，你应该总是在 <code>WORKDIR</code> 中使用绝对路径。另外，你应该使用 <code>WORKDIR</code> 来替代类似于 <code>RUN cd ... && do-something</code> 的指令，后者难以阅读、排错和维护。</p><h2 id="官方镜像示例"><a href="#官方镜像示例" class="headerlink" title="官方镜像示例"></a>官方镜像示例</h2><p>这些官方镜像的 Dockerfile 都是参考典范：<a href="https://github.com/docker-library/docs">https://github.com/docker-library/docs</a></p><h1 id="资源链接"><a href="#资源链接" class="headerlink" title="资源链接"></a>资源链接</h1><h2 id="官方网站"><a href="#官方网站" class="headerlink" title="官方网站"></a>官方网站</h2><ul><li>Docker 官方主页：<a href="https://www.docker.com/">https://www.docker.com</a></li><li>Docker 官方博客：<a href="https://www.docker.com/blog/">https://www.docker.com/blog/</a></li><li>Docker 官方文档：<a href="https://docs.docker.com/">https://docs.docker.com/</a></li><li>Docker Hub：<a href="https://hub.docker.com/">https://hub.docker.com</a></li><li>Docker 的源代码仓库：<a href="https://github.com/moby/moby">https://github.com/moby/moby</a></li><li>Docker 路线图 <a href="https://github.com/docker/roadmap/projects">https://github.com/docker/roadmap/projects</a></li><li>Docker 发布版本历史：<a href="https://docs.docker.com/release-notes/">https://docs.docker.com/release-notes/</a></li><li>Docker 常见问题：<a href="https://docs.docker.com/engine/faq/">https://docs.docker.com/engine/faq/</a></li><li>Docker 远端应用 API：<a href="https://docs.docker.com/develop/sdk/">https://docs.docker.com/develop/sdk/</a></li></ul><h2 id="实践参考"><a href="#实践参考" class="headerlink" title="实践参考"></a>实践参考</h2><ul><li>Dockerfile 参考：<a href="https://docs.docker.com/engine/reference/builder/">https://docs.docker.com/engine/reference/builder/</a></li><li>Dockerfile 最佳实践：<a href="https://docs.docker.com/engine/userguide/eng-image/dockerfile_best-practices/">https://docs.docker.com/engine/userguide/eng-image/dockerfile_best-practices/</a></li></ul><h2 id="技术交流"><a href="#技术交流" class="headerlink" title="技术交流"></a>技术交流</h2><ul><li>Docker 邮件列表： <a href="https://groups.google.com/forum/#!forum/docker-user">https://groups.google.com/forum/#!forum/docker-user</a></li><li>Docker 的 IRC 频道：<a href="https://chat.freenode.net/#docker">https://chat.freenode.net#docker</a></li><li>Docker 的 Twitter 主页：<a href="https://twitter.com/docker">https://twitter.com/docker</a></li></ul><h2 id="其它"><a href="#其它" class="headerlink" title="其它"></a>其它</h2><ul><li>Docker 的 StackOverflow 问答主页：<a href="https://stackoverflow.com/search?q=docker">https://stackoverflow.com/search?q=docker</a></li></ul><h1 id="参考"><a href="#参考" class="headerlink" title="参考"></a>参考</h1><blockquote><p><a href="https://docker/_practice.gitee.io/zh-cn/">https://docker\_practice.gitee.io/zh-cn/</a></p></blockquote>]]></content>
<categories>
<category>Docker</category>
</categories>
<tags>
<tag>docker linux</tag>
</tags>
</entry>
<entry>
<title>Jupyter NoteBook 的快捷键使用指南</title>
<link href="/2020/09/25/Jupyter%20NoteBook%20%E7%9A%84%E5%BF%AB%E6%8D%B7%E9%94%AE%E4%BD%BF%E7%94%A8%E6%8C%87%E5%8D%97/"/>
<url>/2020/09/25/Jupyter%20NoteBook%20%E7%9A%84%E5%BF%AB%E6%8D%B7%E9%94%AE%E4%BD%BF%E7%94%A8%E6%8C%87%E5%8D%97/</url>
<content type="html"><![CDATA[<span id="more"></span><h1 id="Jupyter-NoteBook-的快捷键使用指南"><a href="#Jupyter-NoteBook-的快捷键使用指南" class="headerlink" title="Jupyter NoteBook 的快捷键使用指南"></a>Jupyter NoteBook 的快捷键使用指南</h1><p> </p><p>Jupyter Notebook 有两种键盘输入模式。即命令模式和编辑模式，这与 <a href="http://www.vim.org/">Vim</a> 有些类似。在编辑模式下，可以往单元中键入代码或文本，此时单元格被绿色的框线包围，且命令模式下的快捷键不生效。在命令模式下，可以用快捷键命令运行单元格，移动单元格，切换单元格编辑状态等等，此时的单元格被灰色的框线包围，且编辑模式下的快捷键不生效。</p><p>从命令模式进入编辑模式需按 <code>Enter</code> 键，从编辑模式切换到命令模式需按 <code>Esc</code> 键。</p><p>以下两表分别是对命令和编辑两种模式下快捷键的简单说明：</p><p> </p><h3 id="命令模式快捷键（按-Esc-键开启）"><a href="#命令模式快捷键（按-Esc-键开启）" class="headerlink" title="命令模式快捷键（按 Esc 键开启）:"></a>命令模式快捷键（按 Esc 键开启）:</h3><table><thead><tr><th>快捷键</th><th>作用</th><th>说明</th></tr></thead><tbody><tr><td>Enter</td><td>转入编辑模式</td><td></td></tr><tr><td>Shift-Enter</td><td>运行本单元，选中下个单元</td><td>新单元默认为命令模式</td></tr><tr><td>Ctrl-Enter</td><td>运行本单元</td><td></td></tr><tr><td>Alt-Enter</td><td>运行本单元，在其下插入新单元</td><td>新单元默认为编辑模式</td></tr><tr><td>Y</td><td>单元转入代码状态</td><td></td></tr><tr><td>M</td><td>单元转入 markdown 状态</td><td></td></tr><tr><td>R</td><td>单元转入 raw 状态</td><td></td></tr><tr><td>1</td><td>设定 1 级标题</td><td>仅在 markdown 状态下时建议使用标题相关快捷键，如果单元处于其他状态，则会强制切换到 markdown 状态</td></tr><tr><td>2</td><td>设定 2 级标题</td><td></td></tr><tr><td>3</td><td>设定 3 级标题</td><td></td></tr><tr><td>4</td><td>设定 4 级标题</td><td></td></tr><tr><td>5</td><td>设定 5 级标题</td><td></td></tr><tr><td>6</td><td>设定 6 级标题</td><td></td></tr><tr><td>Up</td><td>选中上方单元</td><td></td></tr><tr><td>K</td><td>选中上方单元</td><td></td></tr><tr><td>Down</td><td>选中下方单元</td><td></td></tr><tr><td>J</td><td>选中下方单元</td><td></td></tr><tr><td>Shift-K</td><td>连续选择上方单元</td><td></td></tr><tr><td>Shift-J</td><td>连续选择下方单元</td><td></td></tr><tr><td>A</td><td>在上方插入新单元</td><td></td></tr><tr><td>B</td><td>在下方插入新单元</td><td></td></tr><tr><td>X</td><td>剪切选中的单元</td><td></td></tr><tr><td>C</td><td>复制选中的单元</td><td></td></tr><tr><td>Shift-V</td><td>粘贴到上方单元</td><td></td></tr><tr><td>V</td><td>粘贴到下方单元</td><td></td></tr><tr><td>Z</td><td>恢复删除的最后一个单元</td><td></td></tr><tr><td>D,D</td><td>删除选中的单元</td><td>连续按两个 D 键</td></tr><tr><td>Shift-M</td><td>合并选中的单元</td><td></td></tr><tr><td>Ctrl-S</td><td>保存当前 NoteBook</td><td></td></tr><tr><td>S</td><td>保存当前 NoteBook</td><td></td></tr><tr><td>L</td><td>开关行号</td><td>编辑框的行号是可以开启和关闭的</td></tr><tr><td>O</td><td>转换输出</td><td></td></tr><tr><td>Shift-O</td><td>转换输出滚动</td><td></td></tr><tr><td>Esc</td><td>关闭页面</td><td></td></tr><tr><td>Q</td><td>关闭页面</td><td></td></tr><tr><td>H</td><td>显示快捷键帮助</td><td></td></tr><tr><td>I,I</td><td>中断 NoteBook 内核</td><td></td></tr><tr><td>0,0</td><td>重启 NoteBook 内核</td><td></td></tr><tr><td>Shift</td><td>忽略</td><td></td></tr><tr><td>Shift-Space</td><td>向上滚动</td><td></td></tr><tr><td>Space</td><td>向下滚动</td><td></td></tr></tbody></table><p> </p><h3 id="编辑模式快捷键（-按-Enter-键启动）"><a href="#编辑模式快捷键（-按-Enter-键启动）" class="headerlink" title="编辑模式快捷键（ 按 Enter 键启动）:"></a>编辑模式快捷键（ 按 Enter 键启动）:</h3><table><thead><tr><th>快捷键</th><th>作用</th><th>说明</th></tr></thead><tbody><tr><td>Tab</td><td>代码补全或缩进</td><td></td></tr><tr><td>Shift-Tab</td><td>提示</td><td>输出帮助信息，部分函数、类、方法等会显示其定义原型，如果在其后加 <code>?</code> 再运行会显示更加详细的帮助</td></tr><tr><td>Ctrl-]</td><td>缩进</td><td>向右缩进</td></tr><tr><td>Ctrl-[</td><td>解除缩进</td><td>向左缩进</td></tr><tr><td>Ctrl-A</td><td>全选</td><td></td></tr><tr><td>Ctrl-Z</td><td>撤销</td><td></td></tr><tr><td>Ctrl-Shift-Z</td><td>重做</td><td></td></tr><tr><td>Ctrl-Y</td><td>重做</td><td></td></tr><tr><td>Ctrl-Home</td><td>跳到单元开头</td><td></td></tr><tr><td>Ctrl-Up</td><td>跳到单元开头</td><td></td></tr><tr><td>Ctrl-End</td><td>跳到单元末尾</td><td></td></tr><tr><td>Ctrl-Down</td><td>跳到单元末尾</td><td></td></tr><tr><td>Ctrl-Left</td><td>跳到左边一个字首</td><td></td></tr><tr><td>Ctrl-Right</td><td>跳到右边一个字首</td><td></td></tr><tr><td>Ctrl-Backspace</td><td>删除前面一个字</td><td></td></tr><tr><td>Ctrl-Delete</td><td>删除后面一个字</td><td></td></tr><tr><td>Esc</td><td>切换到命令模式</td><td></td></tr><tr><td>Ctrl-M</td><td>切换到命令模式</td><td></td></tr><tr><td>Shift-Enter</td><td>运行本单元，选中下一单元</td><td>新单元默认为命令模式</td></tr><tr><td>Ctrl-Enter</td><td>运行本单元</td><td></td></tr><tr><td>Alt-Enter</td><td>运行本单元，在下面插入一单元</td><td>新单元默认为编辑模式</td></tr><tr><td>Ctrl-Shift–</td><td>分割单元</td><td>按光标所在行进行分割</td></tr><tr><td>Ctrl-Shift-Subtract</td><td>分割单元</td><td></td></tr><tr><td>Ctrl-S</td><td>保存当前 NoteBook</td><td></td></tr><tr><td>Shift</td><td>忽略</td><td></td></tr><tr><td>Up</td><td>光标上移或转入上一单元</td><td></td></tr><tr><td>Down</td><td>光标下移或转入下一单元</td><td></td></tr><tr><td>Ctrl-/</td><td>注释整行/撤销注释</td><td>仅代码状态有效</td></tr></tbody></table><p><strong>注：</strong> 如果快捷键被系统中的其它应用占用，则可能会失效</p><p><strong>来源：</strong> <a href="https://opus.konghy.cn/ipynb/jupyter-notebook-keyboard-shortcut.html">https://opus.konghy.cn/ipynb/jupyter-notebook-keyboard-shortcut.html</a></p>]]></content>
<categories>
<category>python加油鸭</category>
</categories>
<tags>
<tag>python markdown</tag>
</tags>
</entry>
<entry>
<title>PyMySQL 详解</title>
<link href="/2020/08/25/PyMySQL%20%E8%AF%A6%E8%A7%A3/"/>
<url>/2020/08/25/PyMySQL%20%E8%AF%A6%E8%A7%A3/</url>
<content type="html"><![CDATA[<span id="more"></span><p><a href="https://github.com/PyMySQL/PyMySQL">PyMySQL</a> 是一个纯 Python 实现的 MySQL 客户端操作库，支持事务、存储过程、批量执行等。</p><p>PyMySQL 遵循 Python 数据库 API v2.0 规范，并包含了 pure-Python MySQL 客户端库。</p><h2 id="安装"><a href="#安装" class="headerlink" title="安装"></a>安装</h2><p> </p><figure class="highlight cmake"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs cmake">pip <span class="hljs-keyword">install</span> PyMySQL<br></code></pre></td></tr></table></figure><h2 id="创建数据库连接"><a href="#创建数据库连接" class="headerlink" title="创建数据库连接"></a>创建数据库连接</h2><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-keyword">import</span> pymysql<br><br>connection = pymysql.connect(host=<span class="hljs-string">'localhost'</span>,<br> port=<span class="hljs-number">3306</span>,<br> user=<span class="hljs-string">'root'</span>,<br> password=<span class="hljs-string">'root'</span>,<br> db=<span class="hljs-string">'demo'</span>,<br> charset=<span class="hljs-string">'utf8'</span>)<br></code></pre></td></tr></table></figure><p>参数列表：</p><table><thead><tr><th>参数</th><th>描述</th></tr></thead><tbody><tr><td>host</td><td>数据库服务器地址，默认 localhost</td></tr><tr><td>user</td><td>用户名，默认为当前程序运行用户</td></tr><tr><td>password</td><td>登录密码，默认为空字符串</td></tr><tr><td>database</td><td>默认操作的数据库</td></tr><tr><td>port</td><td>数据库端口，默认为 3306</td></tr><tr><td>bind_address</td><td>当客户端有多个网络接口时，指定连接到主机的接口。参数可以是主机名或IP地址。</td></tr><tr><td>unix_socket</td><td>unix 套接字地址，区别于 host 连接</td></tr><tr><td>read_timeout</td><td>读取数据超时时间，单位秒，默认无限制</td></tr><tr><td>write_timeout</td><td>写入数据超时时间，单位秒，默认无限制</td></tr><tr><td>charset</td><td>数据库编码</td></tr><tr><td>sql_mode</td><td>指定默认的 SQL_MODE</td></tr><tr><td>read_default_file</td><td>Specifies my.cnf file to read these parameters from under the [client] section.</td></tr><tr><td>conv</td><td>Conversion dictionary to use instead of the default one. This is used to provide custom marshalling and unmarshaling of types.</td></tr><tr><td>use_unicode</td><td>Whether or not to default to unicode strings. This option defaults to true for Py3k.</td></tr><tr><td>client_flag</td><td>Custom flags to send to MySQL. Find potential values in constants.CLIENT.</td></tr><tr><td>cursorclass</td><td>设置默认的游标类型</td></tr><tr><td>init_command</td><td>当连接建立完成之后执行的初始化 SQL 语句</td></tr><tr><td>connect_timeout</td><td>连接超时时间，默认 10，最小 1，最大 31536000</td></tr><tr><td>ssl</td><td>A dict of arguments similar to mysql_ssl_set()’s parameters. For now the capath and cipher arguments are not supported.</td></tr><tr><td>read_default_group</td><td>Group to read from in the configuration file.</td></tr><tr><td>compress</td><td>Not supported</td></tr><tr><td>named_pipe</td><td>Not supported</td></tr><tr><td>autocommit</td><td>是否自动提交，默认不自动提交，参数值为 None 表示以服务器为准</td></tr><tr><td>local_infile</td><td>Boolean to enable the use of LOAD DATA LOCAL command. (default: False)</td></tr><tr><td>max_allowed_packet</td><td>发送给服务器的最大数据量，默认为 16MB</td></tr><tr><td>defer_connect</td><td>是否惰性连接，默认为立即连接</td></tr><tr><td>auth_plugin_map</td><td>A dict of plugin names to a class that processes that plugin. The class will take the Connection object as the argument to the constructor. The class needs an authenticate method taking an authentication packet as an argument. For the dialog plugin, a prompt(echo, prompt) method can be used (if no authenticate method) for returning a string from the user. (experimental)</td></tr><tr><td>server_public_key</td><td>SHA256 authenticaiton plugin public key value. (default: None)</td></tr><tr><td>db</td><td>参数 database 的别名</td></tr><tr><td>passwd</td><td>参数 password 的别名</td></tr><tr><td>binary_prefix</td><td>Add _binary prefix on bytes and bytearray. (default: False)</td></tr></tbody></table><h2 id="执行-SQL"><a href="#执行-SQL" class="headerlink" title="执行 SQL"></a>执行 SQL</h2><ul><li><p>cursor.execute(sql, args) 执行单条 SQL</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-comment"># 获取游标</span><br>cursor = connection.cursor()<br> <br><span class="hljs-comment"># 创建数据表</span><br>effect_row = cursor.execute(<span class="hljs-string">'''</span><br><span class="hljs-string">CREATE TABLE `users` (</span><br><span class="hljs-string">`name` varchar(32) NOT NULL,</span><br><span class="hljs-string">`age` int(10) unsigned NOT NULL DEFAULT '0',</span><br><span class="hljs-string">PRIMARY KEY (`name`)</span><br><span class="hljs-string">) ENGINE=InnoDB DEFAULT CHARSET=utf8</span><br><span class="hljs-string">'''</span>)<br> <br><span class="hljs-comment"># 插入数据(元组或列表)</span><br>effect_row = cursor.execute(<span class="hljs-string">'INSERT INTO `users` (`name`, `age`) VALUES (%s, %s)'</span>, (<span class="hljs-string">'mary'</span>, <span class="hljs-number">18</span>))<br> <br><span class="hljs-comment"># 插入数据(字典)</span><br>info = {<span class="hljs-string">'name'</span>: <span class="hljs-string">'fake'</span>, <span class="hljs-string">'age'</span>: <span class="hljs-number">15</span>}<br>effect_row = cursor.execute(<span class="hljs-string">'INSERT INTO `users` (`name`, `age`) VALUES (%(name)s, %(age)s)'</span>, info)<br> <br>connection.commit()<br></code></pre></td></tr></table></figure></li><li><p>executemany(sql, args) 批量执行 SQL</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-comment"># 获取游标</span><br>cursor = connection.cursor()<br> <br><span class="hljs-comment"># 批量插入</span><br>effect_row = cursor.executemany(<br> <span class="hljs-string">'INSERT INTO `users` (`name`, `age`) VALUES (%s, %s) ON DUPLICATE KEY UPDATE age=VALUES(age)'</span>, [<br> (<span class="hljs-string">'hello'</span>, <span class="hljs-number">13</span>),<br> (<span class="hljs-string">'fake'</span>, <span class="hljs-number">28</span>),<br> ])<br> <br>connection.commit()<br></code></pre></td></tr></table></figure></li></ul><p> </p><p>注意：INSERT、UPDATE、DELETE 等修改数据的语句需手动执行<code>connection.commit()</code>完成对数据修改的提交。</p><h2 id="获取自增-ID"><a href="#获取自增-ID" class="headerlink" title="获取自增 ID"></a>获取自增 ID</h2><figure class="highlight arduino"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs arduino">cursor.lastrowid<br></code></pre></td></tr></table></figure><h2 id="查询数据"><a href="#查询数据" class="headerlink" title="查询数据"></a>查询数据</h2><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-comment"># 执行查询 SQL</span><br>cursor.execute(<span class="hljs-string">'SELECT * FROM `users`'</span>)<br><br><span class="hljs-comment"># 获取单条数据</span><br>cursor.fetchone()<br><br><span class="hljs-comment"># 获取前N条数据</span><br>cursor.fetchmany(<span class="hljs-number">3</span>)<br><br><span class="hljs-comment"># 获取所有数据</span><br>cursor.fetchall()<br></code></pre></td></tr></table></figure><h2 id="游标控制"><a href="#游标控制" class="headerlink" title="游标控制"></a>游标控制</h2><p>所有的数据查询操作均基于游标，我们可以通过<code>cursor.scroll(num, mode)</code>控制游标的位置。</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs python">cursor.scroll(<span class="hljs-number">1</span>, mode=<span class="hljs-string">'relative'</span>) <span class="hljs-comment"># 相对当前位置移动</span><br>cursor.scroll(<span class="hljs-number">2</span>, mode=<span class="hljs-string">'absolute'</span>) <span class="hljs-comment"># 相对绝对位置移动</span><br></code></pre></td></tr></table></figure><h2 id="设置游标类型"><a href="#设置游标类型" class="headerlink" title="设置游标类型"></a>设置游标类型</h2><p>查询时，默认返回的数据类型为元组，可以自定义设置返回类型。支持5种游标类型：</p><ul><li>Cursor: 默认，元组类型</li><li>DictCursor: 字典类型</li><li>DictCursorMixin: 支持自定义的游标类型，需先自定义才可使用</li><li>SSCursor: 无缓冲元组类型</li><li>SSDictCursor: 无缓冲字典类型</li></ul><p>无缓冲游标类型，适用于数据量很大，一次性返回太慢，或者服务端带宽较小时。源码注释：</p><blockquote><p>Unbuffered Cursor, mainly useful for queries that return a lot of data, or for connections to remote servers over a slow network.</p><p>Instead of copying every row of data into a buffer, this will fetch rows as needed. The upside of this is the client uses much less memory, and rows are returned much faster when traveling over a slow network or if the result set is very big.</p><p>There are limitations, though. The MySQL protocol doesn’t support returning the total number of rows, so the only way to tell how many rows there are is to iterate over every row returned. Also, it currently isn’t possible to scroll backwards, as only the current row is held in memory.</p></blockquote><p>创建连接时，通过 cursorclass 参数指定类型：</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><code class="hljs python">connection = pymysql.connect(host=<span class="hljs-string">'localhost'</span>,<br> user=<span class="hljs-string">'root'</span>,<br> password=<span class="hljs-string">'root'</span>,<br> db=<span class="hljs-string">'demo'</span>,<br> charset=<span class="hljs-string">'utf8'</span>,<br> cursorclass=pymysql.cursors.DictCursor)<br></code></pre></td></tr></table></figure><p>也可以在创建游标时指定类型：</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs python">cursor = connection.cursor(cursor=pymysql.cursors.DictCursor)<br></code></pre></td></tr></table></figure><h2 id="事务处理"><a href="#事务处理" class="headerlink" title="事务处理"></a>事务处理</h2><ul><li><p>开启事务 <code>connection.begin()</code></p></li><li><p>提交修改 <code>connection.commit()</code></p></li><li><p>回滚事务 <code>connection.rollback()</code></p></li></ul><h2 id="防-SQL-注入"><a href="#防-SQL-注入" class="headerlink" title="防 SQL 注入"></a>防 SQL 注入</h2><ul><li><p>转义特殊字符 <code>connection.escape_string(str)</code></p></li><li><p>参数化语句 支持传入参数进行自动转义、格式化 SQL 语句，以避免 SQL 注入等安全问题。</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-comment"># 插入数据(元组或列表)</span><br>effect_row = cursor.execute(<span class="hljs-string">'INSERT INTO `users` (`name`, `age`) VALUES (%s, %s)'</span>, (<span class="hljs-string">'mary'</span>, <span class="hljs-number">18</span>))<br><br><span class="hljs-comment"># 插入数据(字典)</span><br>info = {<span class="hljs-string">'name'</span>: <span class="hljs-string">'fake'</span>, <span class="hljs-string">'age'</span>: <span class="hljs-number">15</span>}<br>effect_row = cursor.execute(<span class="hljs-string">'INSERT INTO `users` (`name`, `age`) VALUES (%(name)s, %(age)s)'</span>, info)<br><br><span class="hljs-comment"># 批量插入</span><br>effect_row = cursor.executemany(<br><span class="hljs-string">'INSERT INTO `users` (`name`, `age`) VALUES (%s, %s) ON DUPLICATE KEY UPDATE age=VALUES(age)'</span>, [<br> (<span class="hljs-string">'hello'</span>, <span class="hljs-number">13</span>),<br> (<span class="hljs-string">'fake'</span>, <span class="hljs-number">28</span>),<br>])<br></code></pre></td></tr></table></figure></li></ul><p> </p><h2 id="参考资料"><a href="#参考资料" class="headerlink" title="参考资料"></a>参考资料</h2><ul><li><a href="http://www.cnblogs.com/wt11/p/6141225.html">Python中操作mysql的pymysql模块详解</a></li><li><a href="http://www.cnblogs.com/liubinsh/p/7568423.html">Python之pymysql的使用</a></li></ul>]]></content>
<categories>
<category>SQL</category>
</categories>
<tags>
<tag>数据库 mysql python</tag>
</tags>
</entry>
<entry>
<title>Celery 的用法介绍</title>
<link href="/2020/08/18/Celery%20%E7%9A%84%E7%94%A8%E6%B3%95%E4%BB%8B%E7%BB%8D/"/>
<url>/2020/08/18/Celery%20%E7%9A%84%E7%94%A8%E6%B3%95%E4%BB%8B%E7%BB%8D/</url>
<content type="html"><![CDATA[<span id="more"></span><h2 id="celery介绍"><a href="#celery介绍" class="headerlink" title="celery介绍"></a><strong>celery介绍</strong></h2><h3 id="什么是celery"><a href="#什么是celery" class="headerlink" title="什么是celery"></a><strong>什么是celery</strong></h3><p><strong>这次我们来介绍一下Python的一个第三方模块celery，那么celery是什么呢？</strong></p><ul><li><code>celery是一个灵活且可靠的，处理大量消息的分布式系统，可以在多个节点之间处理某个任务。</code></li><li><code>celery是一个专注于实时处理的任务队列，支持任务调度。</code></li><li><code>celery是开源的，有很多使用者。</code></li><li><code>celery完全基于Python语言编写。</code></li></ul><p><strong>所以celery是一个任务调度框架，类似于Apache的airflow，当然airflow也是基于Python语言编写。不过有一点需要注意，celery是用来调度任务的，它本身并不具备任务的存储功能，而我们说在调度任务的时候肯定是要把任务存起来的，因此在使用celery的时候还需要搭配一些具备存储、访问功能的工具，比如：消息队列、Redis缓存、数据库等等。官方推荐的是消息队列RabbitMQ，个人认为有些时候使用Redis也是不错的选择，当然我们都会介绍。</strong></p><p><strong>那么celery都可以在哪些场景中使用呢？</strong></p><ul><li><code>异步任务：一些耗时的操作可以交给celery异步执行，而不用等着程序处理完才知道结果。比如：视频转码、邮件发送、消息推送等等。</code></li><li><code>定时任务：比如定时推送消息、定时爬取数据、定时统计数据等等</code></li></ul><p>参考：<a href="https://www.cnblogs.com/traditional/p/11788756.html">这是一篇很详细的celery用法介绍</a></p>]]></content>
<categories>
<category>python加油鸭</category>
</categories>
<tags>
<tag>python 数据库</tag>
</tags>
</entry>
<entry>
<title>RESTful 设计方法</title>
<link href="/2020/07/23/RESTful%20%E8%AE%BE%E8%AE%A1%E6%96%B9%E6%B3%95/"/>
<url>/2020/07/23/RESTful%20%E8%AE%BE%E8%AE%A1%E6%96%B9%E6%B3%95/</url>
<content type="html"><![CDATA[<span id="more"></span><h1 id="RESTful设计方法"><a href="#RESTful设计方法" class="headerlink" title="RESTful设计方法"></a>RESTful设计方法</h1><h2 id="1-域名"><a href="#1-域名" class="headerlink" title="1. 域名"></a>1. 域名</h2><p>应该尽量将API部署在专用域名之下。</p><figure class="highlight awk"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs awk">https:<span class="hljs-regexp">//</span>api.example.com<br></code></pre></td></tr></table></figure><p>如果确定API很简单，不会有进一步扩展，可以考虑放在主域名下。</p><figure class="highlight awk"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs awk">https:<span class="hljs-regexp">//</span>example.org<span class="hljs-regexp">/api/</span><br></code></pre></td></tr></table></figure><h2 id="2-版本（Versioning）"><a href="#2-版本（Versioning）" class="headerlink" title="2. 版本（Versioning）"></a>2. 版本（Versioning）</h2><p>应该将API的版本号放入URL。</p><figure class="highlight awk"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><code class="hljs awk">http:<span class="hljs-regexp">//</span>www.example.com<span class="hljs-regexp">/app/</span><span class="hljs-number">1.0</span>/foo<br><br>http:<span class="hljs-regexp">//</span>www.example.com<span class="hljs-regexp">/app/</span><span class="hljs-number">1.1</span>/foo<br><br>http:<span class="hljs-regexp">//</span>www.example.com<span class="hljs-regexp">/app/</span><span class="hljs-number">2.0</span>/foo<br></code></pre></td></tr></table></figure><p>另一种做法是，将版本号放在HTTP头信息中，但不如放入URL方便和直观。<a href="https://developer.github.com/v3/media/#request-specific-version">Github</a>采用这种做法。</p><p>因为不同的版本，可以理解成同一种资源的不同表现形式，所以应该采用同一个URL。版本号可以在HTTP请求头信息的Accept字段中进行区分（参见<a href="http://www.informit.com/articles/article.aspx?p=1566460">Versioning REST Services</a>）：</p><figure class="highlight apache"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><code class="hljs apache"><span class="hljs-attribute">Accept</span>: vnd.example-com.foo+json; version=<span class="hljs-number">1</span>.<span class="hljs-number">0</span><br><br><span class="hljs-attribute">Accept</span>: vnd.example-com.foo+json; version=<span class="hljs-number">1</span>.<span class="hljs-number">1</span><br><br><span class="hljs-attribute">Accept</span>: vnd.example-com.foo+json; version=<span class="hljs-number">2</span>.<span class="hljs-number">0</span><br></code></pre></td></tr></table></figure><h2 id="3-路径（Endpoint）"><a href="#3-路径（Endpoint）" class="headerlink" title="3. 路径（Endpoint）"></a>3. 路径（Endpoint）</h2><p>路径又称”终点”（endpoint），表示API的具体网址，每个网址代表一种资源（resource）</p><p><strong>(1) 资源作为网址，只能有名词，不能有动词，而且所用的名词往往与数据库的表名对应。</strong></p><p>举例来说，以下是不好的例子:</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><code class="hljs bash">/getProducts<br>/listOrders<br>/retreiveClientByOrder?orderId=1<br></code></pre></td></tr></table></figure><p>对于一个简洁结构，你应该始终用名词。 此外，利用的HTTP方法可以分离网址中的资源名称的操作。</p><figure class="highlight apache"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><code class="hljs apache"><span class="hljs-attribute">GET</span> /products ：将返回所有产品清单<br><span class="hljs-attribute">POST</span> /products ：将产品新建到集合<br><span class="hljs-attribute">GET</span> /products/<span class="hljs-number">4</span> ：将获取产品 <span class="hljs-number">4</span><br><span class="hljs-attribute">PATCH</span>（或）PUT /products/<span class="hljs-number">4</span> ：将更新产品 <span class="hljs-number">4</span><br></code></pre></td></tr></table></figure><p><strong>(2) API中的名词应该使用复数。无论子资源或者所有资源。</strong></p><p>举例来说，获取产品的API可以这样定义</p><figure class="highlight awk"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs awk">获取单个产品：http:<span class="hljs-regexp">//</span><span class="hljs-number">127.0</span>.<span class="hljs-number">0.1</span>:<span class="hljs-number">8080</span><span class="hljs-regexp">/AppName/</span>rest<span class="hljs-regexp">/products/</span><span class="hljs-number">1</span><br>获取所有产品: http:<span class="hljs-regexp">//</span><span class="hljs-number">127.0</span>.<span class="hljs-number">0.1</span>:<span class="hljs-number">8080</span><span class="hljs-regexp">/AppName/</span>rest/products<br></code></pre></td></tr></table></figure><h2 id="3-HTTP动词"><a href="#3-HTTP动词" class="headerlink" title="3. HTTP动词"></a>3. HTTP动词</h2><p>对于资源的具体操作类型，由HTTP动词表示。</p><p>常用的HTTP动词有下面四个（括号里是对应的SQL命令）。</p><ul><li>GET（SELECT）：从服务器取出资源（一项或多项）。</li><li>POST（CREATE）：在服务器新建一个资源。</li><li>PUT（UPDATE）：在服务器更新资源（客户端提供改变后的完整资源）。</li><li>DELETE（DELETE）：从服务器删除资源。</li></ul><p>还有三个不常用的HTTP动词。</p><ul><li>PATCH（UPDATE）：在服务器更新(更新)资源（客户端提供改变的属性）。</li><li>HEAD：获取资源的元数据。</li><li>OPTIONS：获取信息，关于资源的哪些属性是客户端可以改变的。</li></ul><p>下面是一些例子。</p><figure class="highlight gradle"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><code class="hljs gradle">GET /zoos：列出所有动物园<br>POST /zoos：新建一个动物园（上传文件）<br>GET <span class="hljs-regexp">/zoos/I</span>D：获取某个指定动物园的信息<br>PUT <span class="hljs-regexp">/zoos/I</span>D：更新某个指定动物园的信息（提供该动物园的全部信息）<br>PATCH <span class="hljs-regexp">/zoos/I</span>D：更新某个指定动物园的信息（提供该动物园的部分信息）<br><span class="hljs-keyword">DELETE</span> <span class="hljs-regexp">/zoos/I</span>D：删除某个动物园<br>GET <span class="hljs-regexp">/zoos/I</span>D/animals：列出某个指定动物园的所有动物<br><span class="hljs-keyword">DELETE</span> <span class="hljs-regexp">/zoos/I</span>D<span class="hljs-regexp">/animals/I</span>D：删除某个指定动物园的指定动物<br></code></pre></td></tr></table></figure><h2 id="4-过滤信息（Filtering）"><a href="#4-过滤信息（Filtering）" class="headerlink" title="4. 过滤信息（Filtering）"></a>4. 过滤信息（Filtering）</h2><p>如果记录数量很多，服务器不可能都将它们返回给用户。API应该提供参数，过滤返回结果。</p><p>下面是一些常见的参数。</p><figure class="highlight routeros"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><code class="hljs routeros">?<span class="hljs-attribute">limit</span>=10：指定返回记录的数量<br>?<span class="hljs-attribute">offset</span>=10：指定返回记录的开始位置。<br>?<span class="hljs-attribute">page</span>=2&per_page=100：指定第几页，以及每页的记录数。<br>?<span class="hljs-attribute">sortby</span>=name&order=asc：指定返回结果按照哪个属性排序，以及排序顺序。<br>?<span class="hljs-attribute">animal_type_id</span>=1：指定筛选条件<br></code></pre></td></tr></table></figure><p>参数的设计允许存在冗余，即允许API路径和URL参数偶尔有重复。比如，GET /zoos/ID/animals 与 GET /animals?zoo_id=ID 的含义是相同的。</p><h2 id="6-状态码（Status-Codes）"><a href="#6-状态码（Status-Codes）" class="headerlink" title="6. 状态码（Status Codes）"></a>6. 状态码（Status Codes）</h2><p>服务器向用户返回的状态码和提示信息，常见的有以下一些（方括号中是该状态码对应的HTTP动词）。</p><blockquote><ul><li>200 OK - [GET]：服务器成功返回用户请求的数据</li><li>201 CREATED - [POST/PUT/PATCH]：用户新建或修改数据成功。</li><li>202 Accepted - [*]：表示一个请求已经进入后台排队（异步任务）</li><li>204 NO CONTENT - [DELETE]：用户删除数据成功。</li><li>400 INVALID REQUEST - [POST/PUT/PATCH]：用户发出的请求有错误，服务器没有进行新建或修改数据的操作</li><li>401 Unauthorized - [*]：表示用户没有权限（令牌、用户名、密码错误）。</li><li>403 Forbidden - [*] 表示用户得到授权（与401错误相对），但是访问是被禁止的。</li><li>404 NOT FOUND - [*]：用户发出的请求针对的是不存在的记录，服务器没有进行操作，该操作是幂等的。</li><li>406 Not Acceptable - [GET]：用户请求的格式不可得（比如用户请求JSON格式，但是只有XML格式）。</li><li>410 Gone -[GET]：用户请求的资源被永久删除，且不会再得到的。</li><li>422 Unprocesable entity - [POST/PUT/PATCH] 当创建一个对象时，发生一个验证错误。</li><li>500 INTERNAL SERVER ERROR - [*]：服务器发生错误，用户将无法判断发出的请求是否成功。</li></ul></blockquote><p>状态码的完全列表参见<a href="http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html">这里</a>或<a href="https://zh.wikipedia.org/wiki/HTTP%E7%8A%B6%E6%80%81%E7%A0%81">这里</a>。</p><h2 id="7-错误处理（Error-handling）"><a href="#7-错误处理（Error-handling）" class="headerlink" title="7. 错误处理（Error handling）"></a>7. 错误处理（Error handling）</h2><p>如果状态码是4xx，服务器就应该向用户返回出错信息。一般来说，返回的信息中将error作为键名，出错信息作为键值即可。</p><figure class="highlight applescript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><code class="hljs applescript">{<br> <span class="hljs-keyword">error</span>: <span class="hljs-string">"Invalid API key"</span><br>}<br></code></pre></td></tr></table></figure><h2 id="8-返回结果"><a href="#8-返回结果" class="headerlink" title="8. 返回结果"></a>8. 返回结果</h2><p>针对不同操作，服务器向用户返回的结果应该符合以下规范。</p><ul><li>GET /collection：返回资源对象的列表（数组）</li><li>GET /collection/resource：返回单个资源对象</li><li>POST /collection：返回新生成的资源对象</li><li>PUT /collection/resource：返回完整的资源对象</li><li>PATCH /collection/resource：返回完整的资源对象</li><li>DELETE /collection/resource：返回一个空文档</li></ul><h2 id="9-超媒体（Hypermedia-API）"><a href="#9-超媒体（Hypermedia-API）" class="headerlink" title="9. 超媒体（Hypermedia API）"></a>9. 超媒体（Hypermedia API）</h2><p>RESTful API最好做到Hypermedia（即返回结果中提供链接，连向其他API方法），使得用户不查文档，也知道下一步应该做什么。</p><p>比如，Github的API就是这种设计，访问<a href="https://api.github.com/">api.github.com</a>会得到一个所有可用API的网址列表。</p><figure class="highlight awk"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><code class="hljs awk">{<br><span class="hljs-string">"current_user_url"</span>: <span class="hljs-string">"https://api.github.com/user"</span>,<br><span class="hljs-string">"authorizations_url"</span>: <span class="hljs-string">"https://api.github.com/authorizations"</span>,<br><span class="hljs-regexp">//</span> ...<br>}<br></code></pre></td></tr></table></figure><p>从上面可以看到，如果想获取当前用户的信息，应该去访问<a href="https://api.github.com/user">api.github.com/user</a>，然后就得到了下面结果。</p><figure class="highlight json"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><code class="hljs json"><span class="hljs-punctuation">{</span><br> <span class="hljs-attr">"message"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"Requires authentication"</span><span class="hljs-punctuation">,</span><br> <span class="hljs-attr">"documentation_url"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"https://developer.github.com/v3"</span><br><span class="hljs-punctuation">}</span><br></code></pre></td></tr></table></figure><p>上面代码表示，服务器给出了提示信息，以及文档的网址。</p><h2 id="10-其他"><a href="#10-其他" class="headerlink" title="10. 其他"></a>10. 其他</h2><p>服务器返回的数据格式，应该尽量使用JSON，避免使用XML。</p><h2 id="11-Flask-Restlul-的使用"><a href="#11-Flask-Restlul-的使用" class="headerlink" title="11.Flask-Restlul 的使用"></a>11.Flask-Restlul 的使用</h2><p><a href="https://juejin.im/post/5d1ffe466fb9a07efd472cee">Flask-RESTful是用于快速构建REST API的Flask扩展</a></p>]]></content>
<categories>
<category>RESTful</category>
</categories>
<tags>
<tag>restful</tag>
</tags>
</entry>
<entry>
<title>认识 RESTful 什么是RESTful</title>
<link href="/2020/07/23/%E8%AE%A4%E8%AF%86%20RESTful%20%E4%BB%80%E4%B9%88%E6%98%AFRESTful/"/>
<url>/2020/07/23/%E8%AE%A4%E8%AF%86%20RESTful%20%E4%BB%80%E4%B9%88%E6%98%AFRESTful/</url>
<content type="html"><![CDATA[<span id="more"></span><h1 id="认识RESTful"><a href="#认识RESTful" class="headerlink" title="认识RESTful"></a>认识RESTful</h1><p><strong>在前后端分离的应用模式里，后端API接口如何定义？</strong></p><ul><li><p>对于接口的请求方式与路径，每个后端开发人员可能都有自己的定义方式，风格迥异。</p></li><li><p>是否存在一种统一的定义方式，被广大开发人员接受认可的方式呢？</p></li><li><p>这就是被普遍采用的API的RESTful设计风格。</p></li><li><p>例如对于后端数据库中保存了商品的信息，前端可能需要对商品数据进行增删改查，那相应的每个操作后端都需要提供一个API接口：</p></li></ul><table><thead><tr><th>请求方法</th><th>请求地址</th><th>后端操作</th></tr></thead><tbody><tr><td>GET</td><td>/goods</td><td>获取所有商品</td></tr><tr><td>POST</td><td>/goods</td><td>增加商品</td></tr><tr><td>GET</td><td>/goods/1</td><td>获取编号为1的商品</td></tr><tr><td>PUT</td><td>/goods/1</td><td>修改编号为1的商品</td></tr><tr><td>DELETE</td><td>/goods/1</td><td>删除编号为1的商品</td></tr></tbody></table><p> </p><h2 id="说明"><a href="#说明" class="headerlink" title="说明"></a>说明</h2><ul><li>RESTful是一种开发理念。<strong>REST</strong>是<strong>设计风格</strong>而不是标准。</li><li>公司中的后端程序员在定义后端的接口时，可以选择是否遵守这种设计风格。</li><li>RESTful设计风格的API，每种请求方法，都对应后端一种数据库操作。</li></ul>]]></content>
<categories>
<category>RESTful</category>
</categories>
<tags>
<tag>设计模式 restful</tag>
</tags>
</entry>
<entry>
<title>文件上传 Flask报错</title>
<link href="/2020/07/23/%E6%96%87%E4%BB%B6%E4%B8%8A%E4%BC%A0%20%20Flask%E6%8A%A5%E9%94%99/"/>
<url>/2020/07/23/%E6%96%87%E4%BB%B6%E4%B8%8A%E4%BC%A0%20%20Flask%E6%8A%A5%E9%94%99/</url>
<content type="html"><![CDATA[<span id="more"></span><h4 id="报错信息：TypeError-expected-str-bytes-or-os-PathLike-object-not-FileStorage"><a href="#报错信息：TypeError-expected-str-bytes-or-os-PathLike-object-not-FileStorage" class="headerlink" title="报错信息：TypeError: expected str, bytes or os.PathLike object,not FileStorage"></a>报错信息：TypeError: expected str, bytes or os.PathLike object,not FileStorage</h4><ul><li> 上传一个文件 file 本来想通过open()来打开文件进行处理的，但是却报错了</li></ul><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-meta">@app.route(<span class="hljs-params"><span class="hljs-string">'/getfile'</span>, methods=[<span class="hljs-string">'POST'</span>]</span>)</span><br><span class="hljs-keyword">def</span> <span class="hljs-title function_">getfile</span>():<br> request_data = request.files[<span class="hljs-string">'file'</span>]<br> rsrcmgr = PDFResourceManager()<br> retstr = io.StringIO()<br> codec = <span class="hljs-string">'utf-8'</span><br> laparams = LAParams()<br> device = TextConverter(rsrcmgr, retstr, codec=codec, laparams=laparams)<br> fp = <span class="hljs-built_in">open</span>(request_data, <span class="hljs-string">'rb'</span>)<br> interpreter = PDFPageInterpreter(rsrcmgr, device)<br> password = <span class="hljs-string">""</span><br> maxpages = <span class="hljs-number">0</span><br> caching = <span class="hljs-literal">True</span><br> pagenos = <span class="hljs-built_in">set</span>()<br><br> <span class="hljs-keyword">for</span> page <span class="hljs-keyword">in</span> PDFPage.get_pages(fp, pagenos, maxpages=maxpages,<br> password=password,<br> caching=caching,<br> check_extractable=<span class="hljs-literal">True</span>):<br> interpreter.process_page(page)<br><br> text = retstr.getvalue()<br><br> fp.close()<br> device.close()<br> retstr.close()<br><span class="hljs-keyword">return</span> text<br></code></pre></td></tr></table></figure><ul><li> 报错</li></ul><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs bash">line 27, <span class="hljs-keyword">in</span> getfile fp = open(request_data, <span class="hljs-string">'rb'</span>).decode(<span class="hljs-string">"utf-8"</span>) TypeError: expected str, bytes or os.PathLike object, not FileStorage<br></code></pre></td></tr></table></figure><ul><li> 解决方案</li></ul><blockquote><p>The <code>request.files['file']</code> is an instance of a <a href="http://werkzeug.pocoo.org/docs/0.14/datastructures/#werkzeug.datastructures.FileStorage">FileStorage</a> class (see also <a href="http://flask.pocoo.org/docs/0.12/api/#flask.Request.files">http://flask.pocoo.org/docs/0.12/api/#flask.Request.files</a>), so you can’t do the <code>fp = open(request_data, 'rb')</code>. The FileStorage object contains a <code>stream</code> attribute that should point to an open temporary file, and probably you can pass that to <code>PDFPage.get_pages()</code></p></blockquote><blockquote><p>也就是说：Files [‘ file’]是 FileStorage 类的一个实例，所以不能执行 fp = open (request _ data，‘ rb’)</p><p>FileStorage 对象包含 <code>stream</code> 属性，该属性应该指向一个打开的临时文件，您可以将其传递给 PDFPage.get _ pages ()</p></blockquote><p> </p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-meta">@app.route(<span class="hljs-params"><span class="hljs-string">'/getfile'</span>, methods=[<span class="hljs-string">'POST'</span>]</span>)</span><br><span class="hljs-keyword">def</span> <span class="hljs-title function_">getfile</span>():<br> file = request.files[<span class="hljs-string">'file'</span>]<br> rsrcmgr = PDFResourceManager()<br> retstr = io.StringIO()<br> codec = <span class="hljs-string">'utf-8'</span><br> laparams = LAParams()<br> device = TextConverter(rsrcmgr, retstr, codec=codec, laparams=laparams)<br> interpreter = PDFPageInterpreter(rsrcmgr, device)<br> password = <span class="hljs-string">""</span><br> maxpages = <span class="hljs-number">0</span><br> caching = <span class="hljs-literal">True</span><br> pagenos = <span class="hljs-built_in">set</span>()<br><br> <span class="hljs-keyword">for</span> page <span class="hljs-keyword">in</span> PDFPage.get_pages(file.stream, pagenos, maxpages=maxpages,<br> password=password,<br> caching=caching,<br> check_extractable=<span class="hljs-literal">True</span>):<br> interpreter.process_page(page)<br><br> text = retstr.getvalue()<br><br> device.close()<br> retstr.close()<br><span class="hljs-keyword">return</span> text<br></code></pre></td></tr></table></figure>]]></content>
<categories>
<category>python加油鸭</category>
</categories>
<tags>
<tag>python web</tag>
</tags>
</entry>
<entry>
<title>进入 Docker 容器的几种方式 attach,ssh,nsenter,exec</title>
<link href="/2020/07/14/%E8%BF%9B%E5%85%A5%20Docker%20%E5%AE%B9%E5%99%A8%E7%9A%84%E5%87%A0%E7%A7%8D%E6%96%B9%E5%BC%8F%20attach,ssh,nsenter,exec/"/>
<url>/2020/07/14/%E8%BF%9B%E5%85%A5%20Docker%20%E5%AE%B9%E5%99%A8%E7%9A%84%E5%87%A0%E7%A7%8D%E6%96%B9%E5%BC%8F%20attach,ssh,nsenter,exec/</url>
<content type="html"><![CDATA[<span id="more"></span><p>在使用Docker创建了容器之后，大家比较关心的就是如何进入该容器了，其实进入Docker容器有好几多种方式，这里我们就讲一下常用的几种进入Docker容器的方法。</p><p>进入Docker容器比较常见的几种做法如下：</p><ul><li>使用docker attach</li><li>使用SSH</li><li>使用nsenter</li><li>使用exec</li></ul><h1 id="一、使用docker-attach进入Docker容器"><a href="#一、使用docker-attach进入Docker容器" class="headerlink" title="一、使用docker attach进入Docker容器"></a>一、使用docker attach进入Docker容器</h1><p> Docker提供了attach命令来进入Docker容器。</p><p> </p><p> 接下来我们创建一个守护态的Docker容器，然后使用docker attach命令进入该容器。</p><ol><li>$ sudo docker run -itd ubuntu:14.04 /bin/bash</li></ol><p> </p><p> </p><p> 然后我们使用docker ps查看到该容器信息，接下来就使用docker attach进入该容器</p><ol><li>$ sudo docker attach 44fc0f0582d9</li></ol><p> </p><p> 可以看到我们已经进入到该容器中了。</p><p> </p><p> 但在，使用该命令有一个问题。当多个窗口同时使用该命令进入该容器时，所有的窗口都会同步显示。如果有一个窗口阻塞了，那么其他窗口也无法再进行操作。</p><p> </p><p>因为这个原因，所以docker attach命令不太适合于生产环境，平时自己开发应用时可以使用该命令。</p><h1 id="二、使用SSH进入Docker容器"><a href="#二、使用SSH进入Docker容器" class="headerlink" title="二、使用SSH进入Docker容器"></a>二、使用SSH进入Docker容器</h1><p> 在生产环境中排除了使用docker attach命令进入容器之后，相信大家第一个想到的就是ssh。在镜像（或容器）中安装SSH Server，这样就能保证多人进入</p><p>容器且相互之间不受干扰了，相信大家在当前的生产环境中（没有使用Docker的情况）也是这样做的。但是使用了Docker容器之后不建议使用ssh进入到Docker容</p><p>器内。关于为什么不建议使用，请参考如下文章：</p><p><a href="http://www.oschina.net/translate/why-you-dont-need-to-run-sshd-in-docker?cmp">为什么不需要在 Docker 容器中运行 sshd</a></p><h1 id="三、使用nsenter进入Docker容器"><a href="#三、使用nsenter进入Docker容器" class="headerlink" title="三、使用nsenter进入Docker容器"></a>三、使用nsenter进入Docker容器</h1><p> 在上面两种方式都不适合的情况下，还有一种比较方便的方法，即使用nsenter进入Docker容器。关于什么是nsenter请参考如下文章：</p><p><a href="https://github.com/jpetazzo/nsenter">https://github.com/jpetazzo/nsenter</a></p><p>在了解了什么是nsenter之后，系统默认将我们需要的nsenter安装到主机中</p><p>如果没有安装的话，按下面步骤安装即可（注意是主机而非容器或镜像）</p><p>具体的安装命令如下：</p><ol><li>$ wget <a href="https://www.kernel.org/pub/linux/utils/util-linux/v2.24/util-linux-2.24.tar.gz">https://www.kernel.org/pub/linux/utils/util-linux/v2.24/util-linux-2.24.tar.gz</a> </li><li>$ tar -xzvf util-linux-2.24.tar.gz </li><li>$ cd util-linux-2.24/ </li><li>$ ./configure –without-ncurses </li><li>$ make nsenter </li><li>$ sudo cp nsenter /usr/local/bin</li></ol><p> </p><p>安装好nsenter之后可以查看一下该命令的使用。</p><p> <img src="https://imgconvert.csdnimg.cn/aHR0cHM6Ly9pbWFnZXMyMDE1LmNuYmxvZ3MuY29tL2Jsb2cvMTAyOTc0My8yMDE3MDIvMTAyOTc0My0yMDE3MDIwNzE1NTU0ODk5NC0zNjA4Njg4NjAucG5n?x-oss-process=image/format,png"></p><p> </p><p> nsenter可以访问另一个进程的名称空间。所以为了连接到某个容器我们还需要获取该容器的第一个进程的PID。可以使用docker inspect命令来拿到该PID。</p><p>docker inspect命令使用如下：</p><ol><li>$ sudo docker inspect –help</li></ol><p> </p><p>inspect命令可以分层级显示一个镜像或容器的信息。比如我们当前有一个正在运行的容器</p><p><img src="https://imgconvert.csdnimg.cn/aHR0cHM6Ly9pbWFnZXMyMDE1LmNuYmxvZ3MuY29tL2Jsb2cvMTAyOTc0My8yMDE3MDIvMTAyOTc0My0yMDE3MDIwNzE1NTYzNDkxNi01NDkyMTk1NjMucG5n?x-oss-process=image/format,png"></p><p> </p><p> </p><p>可以使用docker inspect来查看该容器的详细信息。</p><ol><li>$ sudo docker inspect 44fc0f0582d9</li></ol><p> </p><p> </p><p><img src="https://imgconvert.csdnimg.cn/aHR0cHM6Ly9pbWFnZXMyMDE1LmNuYmxvZ3MuY29tL2Jsb2cvMTAyOTc0My8yMDE3MDIvMTAyOTc0My0yMDE3MDIwNzE1NTcxNjk5NC0yMTI1MjkwMDA0LnBuZw?x-oss-process=image/format,png"></p><p> </p><p>由其该信息非常多，此处只截取了其中一部分进行展示。如果要显示该容器第一个进行的PID可以使用如下方式</p><ol><li>$ sudo docker inspect -f {<!-- -->{.State.Pid}} 44fc0f0582d9</li></ol><p> </p><p> </p><p> <img src="https://imgconvert.csdnimg.cn/aHR0cHM6Ly9pbWFnZXMyMDE1LmNuYmxvZ3MuY29tL2Jsb2cvMTAyOTc0My8yMDE3MDIvMTAyOTc0My0yMDE3MDIwNzE1NTgyMjUyNi0xODQ1Nzc4NDU2LnBuZw?x-oss-process=image/format,png"></p><p> </p><p>在拿到该进程PID之后我们就可以使用nsenter命令访问该容器了。</p><ol><li>$ sudo nsenter –target 3326 –mount –uts –ipc –net –pid</li></ol><p> </p><ol><li>$ sudo nsenter –target 3326 –mount –uts –ipc –net –pid</li></ol><p> </p><p>其中的3326即刚才拿到的进程的PID</p><p> </p><p>当然，如果你认为每次都输入那么多参数太麻烦的话，网上也有许多做好的脚本供大家使用。</p><p>地址如下：</p><p><a href="http://yeasy.gitbooks.io/docker_practice/content/container/enter.html">http://yeasy.gitbooks.io/docker_practice/content/container/enter.html</a></p><p><a href="http://www.tuicool.com/articles/eYnUBrR">http://www.tuicool.com/articles/eYnUBrR</a></p><p> </p><p><strong>四、使用docker exec进入Docker容器</strong></p><p> 除了上面几种做法之外，docker在1.3.X版本之后还提供了一个新的命令exec用于进入容器，这种方式相对更简单一些，下面我们来看一下该命令的使用：</p><ol><li>$ sudo docker exec –help</li></ol><p> </p><p> </p><p> <img src="https://imgconvert.csdnimg.cn/aHR0cHM6Ly9pbWFnZXMyMDE1LmNuYmxvZ3MuY29tL2Jsb2cvMTAyOTc0My8yMDE3MDIvMTAyOTc0My0yMDE3MDIwNzE2MDI0NDY4Mi0xMDE5MDcxMDkyLnBuZw?x-oss-process=image/format,png"></p><p> </p><p>接下来我们使用该命令进入一个已经在运行的容器</p><ol><li>$ sudo docker ps </li><li>$ sudo docker exec -it 775c7c9ee1e1 /bin/bash</li></ol><p> </p><p>转自：<a href="https://www.cnblogs.com/xhyan/p/6593075.html">https://www.cnblogs.com/xhyan/p/6593075.html</a></p>]]></content>
<categories>
<category>Docker</category>
</categories>
<tags>
<tag>docker linux ubuntu</tag>
</tags>
</entry>
<entry>
<title>Linux ubuntu pip install mysqlclient 报错问题解决</title>
<link href="/2020/07/13/Linux%20ubuntu%20pip%20install%20mysqlclient%20%E6%8A%A5%E9%94%99%E9%97%AE%E9%A2%98%E8%A7%A3%E5%86%B3/"/>
<url>/2020/07/13/Linux%20ubuntu%20pip%20install%20mysqlclient%20%E6%8A%A5%E9%94%99%E9%97%AE%E9%A2%98%E8%A7%A3%E5%86%B3/</url>
<content type="html"><![CDATA[<span id="more"></span><ul><li><h3 id="报错"><a href="#报错" class="headerlink" title="报错"></a>报错</h3></li></ul><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br></pre></td><td class="code"><pre><code class="hljs bash">Looking <span class="hljs-keyword">in</span> indexes: https://pypi.douban.com/simple<br>Collecting mysqlclient<br> Using cached https://pypi.doubanio.com/packages/a5/e1/e5f2b231c05dc51d9d87fa5066f90d1405345c54b14b0b11a1c859020f21/mysqlclient-2.0.1.tar.gz (87 kB)<br> ERROR: Command errored out with <span class="hljs-built_in">exit</span> status 1:<br> <span class="hljs-built_in">command</span>: /home/zunder/Desktop/env/djenv/bin/python3 -c <span class="hljs-string">'import sys, setuptools, tokenize; sys.argv[0] = '</span><span class="hljs-string">"'"</span><span class="hljs-string">'/tmp/pip-install-1qfymr2e/mysqlclient/setup.py'</span><span class="hljs-string">"'"</span><span class="hljs-string">'; __file__='</span><span class="hljs-string">"'"</span><span class="hljs-string">'/tmp/pip-install-1qfymr2e/mysqlclient/setup.py'</span><span class="hljs-string">"'"</span><span class="hljs-string">';f=getattr(tokenize, '</span><span class="hljs-string">"'"</span><span class="hljs-string">'open'</span><span class="hljs-string">"'"</span><span class="hljs-string">', open)(__file__);code=f.read().replace('</span><span class="hljs-string">"'"</span><span class="hljs-string">'\r\n'</span><span class="hljs-string">"'"</span><span class="hljs-string">', '</span><span class="hljs-string">"'"</span><span class="hljs-string">'\n'</span><span class="hljs-string">"'"</span><span class="hljs-string">');f.close();exec(compile(code, __file__, '</span><span class="hljs-string">"'"</span><span class="hljs-string">'exec'</span><span class="hljs-string">"'"</span><span class="hljs-string">'))'</span> egg_info --egg-base /tmp/pip-pip-egg-info-9lw6d9f3<br> cwd: /tmp/pip-install-1qfymr2e/mysqlclient/<br> Complete output (12 lines):<br> /bin/sh: 1: mysql_config: not found<br> /bin/sh: 1: mariadb_config: not found<br> /bin/sh: 1: mysql_config: not found<br> Traceback (most recent call last):<br> File <span class="hljs-string">"<string>"</span>, line 1, <span class="hljs-keyword">in</span> <module><br> File <span class="hljs-string">"/tmp/pip-install-1qfymr2e/mysqlclient/setup.py"</span>, line 15, <span class="hljs-keyword">in</span> <module><br> metadata, options = get_config()<br> File <span class="hljs-string">"/tmp/pip-install-1qfymr2e/mysqlclient/setup_posix.py"</span>, line 65, <span class="hljs-keyword">in</span> get_config<br> libs = mysql_config(<span class="hljs-string">"libs"</span>)<br> File <span class="hljs-string">"/tmp/pip-install-1qfymr2e/mysqlclient/setup_posix.py"</span>, line 31, <span class="hljs-keyword">in</span> mysql_config<br> raise OSError(<span class="hljs-string">"{} not found"</span>.format(_mysql_config_path))<br> OSError: mysql_config not found<br> ----------------------------------------<br>ERROR: Command errored out with <span class="hljs-built_in">exit</span> status 1: python setup.py egg_info Check the logs <span class="hljs-keyword">for</span> full <span class="hljs-built_in">command</span> output.<br></code></pre></td></tr></table></figure><ul><li><h3 id="解决"><a href="#解决" class="headerlink" title="解决"></a>解决</h3></li></ul><blockquote><p>apt-get install libmysqlclient-dev</p></blockquote><p> </p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br></pre></td><td class="code"><pre><code class="hljs bash">Reading package lists... Done<br>Building dependency tree <br>Reading state information... Done<br>The following additional packages will be installed:<br> libmysqlclient20 libssl-dev libssl-doc libssl1.0.0 zlib1g zlib1g-dev<br>The following NEW packages will be installed:<br> libmysqlclient-dev libmysqlclient20 libssl-dev libssl-doc zlib1g-dev<br>The following packages will be upgraded:<br> libssl1.0.0 zlib1g<br>2 upgraded, 5 newly installed, 0 to remove and 721 not upgraded.<br>Need to get 4,260 kB/5,394 kB of archives.<br>After this operation, 20.5 MB of additional disk space will be used.<br>Do you want to <span class="hljs-built_in">continue</span>? [Y/n] Y<br>Get:1 https://mirrors.tuna.tsinghua.edu.cn/ubuntu xenial-updates/main amd64 libmysqlclient20 amd64 5.7.30-0ubuntu0.16.04.1 [685 kB]<br>Get:2 https://mirrors.tuna.tsinghua.edu.cn/ubuntu xenial-updates/main amd64 zlib1g-dev amd64 1:1.2.8.dfsg-2ubuntu4.3 [167 kB]<br>Get:3 https://mirrors.tuna.tsinghua.edu.cn/ubuntu xenial-updates/main amd64 libssl-dev amd64 1.0.2g-1ubuntu4.16 [1,344 kB]<br>Get:4 https://mirrors.tuna.tsinghua.edu.cn/ubuntu xenial-updates/main amd64 libmysqlclient-dev amd64 5.7.30-0ubuntu0.16.04.1 [986 kB]<br>Get:5 https://mirrors.tuna.tsinghua.edu.cn/ubuntu xenial-updates/main amd64 libssl-doc all 1.0.2g-1ubuntu4.16 [1,078 kB]<br>Fetched 4,260 kB <span class="hljs-keyword">in</span> 2s (1,654 kB/s)<br>Preconfiguring packages ...<br>(Reading database ... 214443 files and directories currently installed.)<br>Preparing to unpack .../zlib1g_1%3a1.2.8.dfsg-2ubuntu4.3_amd64.deb ...<br>Unpacking zlib1g:amd64 (1:1.2.8.dfsg-2ubuntu4.3) over (1:1.2.8.dfsg-2ubuntu4) ...<br>Processing triggers <span class="hljs-keyword">for</span> libc-bin (2.23-0ubuntu3) ...<br>Setting up zlib1g:amd64 (1:1.2.8.dfsg-2ubuntu4.3) ...<br>Processing triggers <span class="hljs-keyword">for</span> libc-bin (2.23-0ubuntu3) ...<br>(Reading database ... 214443 files and directories currently installed.)<br>Preparing to unpack .../libssl1.0.0_1.0.2g-1ubuntu4.16_amd64.deb ...<br>Unpacking libssl1.0.0:amd64 (1.0.2g-1ubuntu4.16) over (1.0.2g-1ubuntu4) ...<br>Selecting previously unselected package libmysqlclient20:amd64.<br>Preparing to unpack .../libmysqlclient20_5.7.30-0ubuntu0.16.04.1_amd64.deb ...<br>Unpacking libmysqlclient20:amd64 (5.7.30-0ubuntu0.16.04.1) ...<br>Selecting previously unselected package zlib1g-dev:amd64.<br>Preparing to unpack .../zlib1g-dev_1%3a1.2.8.dfsg-2ubuntu4.3_amd64.deb ...<br>Unpacking zlib1g-dev:amd64 (1:1.2.8.dfsg-2ubuntu4.3) ...<br>Selecting previously unselected package libssl-dev:amd64.<br>Preparing to unpack .../libssl-dev_1.0.2g-1ubuntu4.16_amd64.deb ...<br>Unpacking libssl-dev:amd64 (1.0.2g-1ubuntu4.16) ...<br>Selecting previously unselected package libmysqlclient-dev.<br>Preparing to unpack .../libmysqlclient-dev_5.7.30-0ubuntu0.16.04.1_amd64.deb ...<br>Unpacking libmysqlclient-dev (5.7.30-0ubuntu0.16.04.1) ...<br>Selecting previously unselected package libssl-doc.<br>Preparing to unpack .../libssl-doc_1.0.2g-1ubuntu4.16_all.deb ...<br>Unpacking libssl-doc (1.0.2g-1ubuntu4.16) ...<br>Processing triggers <span class="hljs-keyword">for</span> libc-bin (2.23-0ubuntu3) ...<br>Processing triggers <span class="hljs-keyword">for</span> man-db (2.7.5-1) ...<br>Setting up libssl1.0.0:amd64 (1.0.2g-1ubuntu4.16) ...<br>Setting up libmysqlclient20:amd64 (5.7.30-0ubuntu0.16.04.1) ...<br>Setting up zlib1g-dev:amd64 (1:1.2.8.dfsg-2ubuntu4.3) ...<br>Setting up libssl-dev:amd64 (1.0.2g-1ubuntu4.16) ...<br>Setting up libmysqlclient-dev (5.7.30-0ubuntu0.16.04.1) ...<br>Setting up libssl-doc (1.0.2g-1ubuntu4.16) ...<br>Processing triggers <span class="hljs-keyword">for</span> libc-bin (2.23-0ubuntu3) ...<br></code></pre></td></tr></table></figure><blockquote><p>apt-get install libssl-dev</p></blockquote><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><code class="hljs bash">Reading package lists... Done<br>Building dependency tree <br>Reading state information... Done<br>libssl-dev is already the newest version (1.0.2g-1ubuntu4.16).<br>libssl-dev <span class="hljs-built_in">set</span> to manually installed.<br>0 upgraded, 0 newly installed, 0 to remove and 721 not upgraded.<br></code></pre></td></tr></table></figure><ul><li><h3 id="执行"><a href="#执行" class="headerlink" title="执行"></a>执行</h3></li></ul><blockquote><p> pip install mysqlclient -i <a href="https://pypi.douban.com/simple">https://pypi.douban.com/simple</a></p></blockquote><p> 成功安装！</p><p><img src="https://img-blog.csdnimg.cn/20200713115437135.png"></p>]]></content>
<categories>
<category>Linux</category>
</categories>
<tags>
<tag>linux ubuntu python</tag>
</tags>
</entry>
<entry>
<title>Python SQLAlchemy 连接MySQL的CURD操作 使用上下文管理 session</title>
<link href="/2020/07/08/Python%20SQLAlchemy%20%E8%BF%9E%E6%8E%A5MySQL%E7%9A%84CURD%E6%93%8D%E4%BD%9C%20%E4%BD%BF%E7%94%A8%E4%B8%8A%E4%B8%8B%E6%96%87%E7%AE%A1%E7%90%86%20session/"/>
<url>/2020/07/08/Python%20SQLAlchemy%20%E8%BF%9E%E6%8E%A5MySQL%E7%9A%84CURD%E6%93%8D%E4%BD%9C%20%E4%BD%BF%E7%94%A8%E4%B8%8A%E4%B8%8B%E6%96%87%E7%AE%A1%E7%90%86%20session/</url>
<content type="html"><![CDATA[<span id="more"></span><ul><li><h3 id="使用-contextmanager-来管理"><a href="#使用-contextmanager-来管理" class="headerlink" title="使用 contextmanager 来管理"></a>使用 contextmanager 来管理</h3></li></ul><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-keyword">from</span> sqlalchemy <span class="hljs-keyword">import</span> create_engine<br><span class="hljs-keyword">from</span> sqlalchemy.orm <span class="hljs-keyword">import</span> scoped_session,sessionmaker<br><br><br>db_connect = <span class="hljs-string">"mysql+pymysql://root:password@localhost:3306/db_name?charset=utf8"</span><br><br>create=create_engine(db_connect)<br>SessionType=scoped_session(sessionmaker(bind=create,expire_on_commit=<span class="hljs-literal">False</span>))<br><br><span class="hljs-keyword">def</span> <span class="hljs-title function_">GetSession</span>():<br> <span class="hljs-keyword">return</span> SessionType()<br><br><span class="hljs-keyword">from</span> contextlib <span class="hljs-keyword">import</span> contextmanager<br><br><span class="hljs-meta">@contextmanager</span><br><span class="hljs-keyword">def</span> <span class="hljs-title function_">session_socpe</span>():<br> session=GetSession()<br> <span class="hljs-keyword">try</span>:<br> <span class="hljs-keyword">yield</span> session<br> session.commit()<br> <span class="hljs-keyword">except</span>:<br> session.rollback()<br> <span class="hljs-keyword">raise</span><br> <span class="hljs-keyword">finally</span>:<br> session.close()<br></code></pre></td></tr></table></figure><ul><li><h3 id="用法"><a href="#用法" class="headerlink" title="用法"></a>用法</h3></li><li> 查询</li></ul><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-keyword">with</span> session_socpe() <span class="hljs-keyword">as</span> session:<br> obj = session.query(Model类).<span class="hljs-built_in">filter</span>(Model类.字段==参数).first()<br></code></pre></td></tr></table></figure><ul><li>增加</li></ul><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-keyword">with</span> session_socpe() <span class="hljs-keyword">as</span> session:<br> obj = Model类(字段<span class="hljs-number">1</span>=值<span class="hljs-number">1</span>, 字段<span class="hljs-number">2</span>=值<span class="hljs-number">2</span>, 字段<span class="hljs-number">3</span>=值<span class="hljs-number">3</span>)<br> session.add(obj)<br></code></pre></td></tr></table></figure><ul><li> 删除</li></ul><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-keyword">with</span> session_socpe() <span class="hljs-keyword">as</span> session:<br> obj = session.query(Model类).<span class="hljs-built_in">filter</span>(Model类.字段==参数).delete()<br></code></pre></td></tr></table></figure><ul><li>修改</li></ul><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-keyword">with</span> session_socpe() <span class="hljs-keyword">as</span> session:<br> obj = session.query(Model类).<span class="hljs-built_in">filter</span>(Model类.字段==参数).update({<span class="hljs-string">"修改字段"</span>:<span class="hljs-string">"修改值"</span>})<br></code></pre></td></tr></table></figure><blockquote><p>每次修改时，不用频繁地session.commit()</p><p>也方便了 万一出错后 的 rollback 回滚</p></blockquote>]]></content>
<categories>
<category>python加油鸭</category>
</categories>
<tags>
<tag>mysql 数据库 orm</tag>
</tags>
</entry>
<entry>
<title>Python SQLAlchemy 自动生成模型 models 文件</title>
<link href="/2020/07/08/Python%20SQLAlchemy%20%E8%87%AA%E5%8A%A8%E7%94%9F%E6%88%90%E6%A8%A1%E5%9E%8B%20models%20%E6%96%87%E4%BB%B6/"/>
<url>/2020/07/08/Python%20SQLAlchemy%20%E8%87%AA%E5%8A%A8%E7%94%9F%E6%88%90%E6%A8%A1%E5%9E%8B%20models%20%E6%96%87%E4%BB%B6/</url>
<content type="html"><![CDATA[<span id="more"></span><ul><li> 安装模块</li></ul><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs bash">pip3 install sqlacodegen<br></code></pre></td></tr></table></figure><ul><li> 执行</li></ul><p> </p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><code class="hljs bash">sqlacodegen mysql+pymysql://root:[email protected]:3306/db_name > test_model.py<br><br>root:mysql 用户<br>password:mysql 密码<br>db_name: 数据库名称<br>test_model.py:导出的名字<br><br>--tables <span class="hljs-built_in">test</span> : 可以指定<span class="hljs-built_in">test</span>数据表<br></code></pre></td></tr></table></figure><ul><li>查看 py 文件 得到以下</li></ul><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-keyword">from</span> sqlalchemy <span class="hljs-keyword">import</span> CHAR, Column, DateTime, Enum, String, Text, text<br><span class="hljs-keyword">from</span> sqlalchemy.dialects.mysql <span class="hljs-keyword">import</span> INTEGER<br><span class="hljs-keyword">from</span> sqlalchemy.ext.declarative <span class="hljs-keyword">import</span> declarative_base<br><br>Base = declarative_base()<br>metadata = Base.metadata<br><br><span class="hljs-keyword">class</span> <span class="hljs-title class_">SaveUser</span>(<span class="hljs-title class_ inherited__">Base</span>):<br> __tablename__ = <span class="hljs-string">'save_user'</span><br><br> <span class="hljs-built_in">id</span> = Column(INTEGER(<span class="hljs-number">11</span>), primary_key=<span class="hljs-literal">True</span>)<br> userName = Column(String(<span class="hljs-number">32</span>), unique=<span class="hljs-literal">True</span>)<br> password = Column(String(<span class="hljs-number">16</span>), index=<span class="hljs-literal">True</span>)<br> createTime = Column(DateTime)<br></code></pre></td></tr></table></figure><ul><li> 成功!</li></ul>]]></content>
<categories>
<category>python加油鸭</category>
</categories>
<tags>
<tag>python mysql 数据库</tag>
</tags>
</entry>
<entry>
<title>Django ORM 更新/修改操作</title>
<link href="/2020/07/01/Django%20ORM%20%E6%9B%B4%E6%96%B0!%E4%BF%AE%E6%94%B9%E6%93%8D%E4%BD%9C/"/>
<url>/2020/07/01/Django%20ORM%20%E6%9B%B4%E6%96%B0!%E4%BF%AE%E6%94%B9%E6%93%8D%E4%BD%9C/</url>
<content type="html"><![CDATA[<span id="more"></span><h1 id="model-update常规用法"><a href="#model-update常规用法" class="headerlink" title="model update常规用法"></a>model update常规用法</h1><p>假如我们的表结构是这样的</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-keyword">class</span> <span class="hljs-title class_">User</span>(models.Model):<br> username = models.CharField(max_length=<span class="hljs-number">255</span>, unique=<span class="hljs-literal">True</span>, verbose_name=<span class="hljs-string">'用户名'</span>)<br> is_active = models.BooleanField(default=<span class="hljs-literal">False</span>, verbose_name=<span class="hljs-string">'激活状态'</span>)<br></code></pre></td></tr></table></figure><p>那么我们修改用户名和状态可以使用如下两种方法：</p><p>方法一：</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs python">User.objects.<span class="hljs-built_in">filter</span>(<span class="hljs-built_in">id</span>=<span class="hljs-number">1</span>).update(username=<span class="hljs-string">'nick'</span>,is_active=<span class="hljs-literal">True</span>)<br></code></pre></td></tr></table></figure><p>方法二：</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><code class="hljs python">_t = User.objects.get(<span class="hljs-built_in">id</span>=<span class="hljs-number">1</span>)<br>_t.username=<span class="hljs-string">'nick'</span><br>_t.is_active=<span class="hljs-literal">True</span><br>_t.save()<br></code></pre></td></tr></table></figure><p>方法一适合更新一批数据，类似于mysql语句<code>update user set username='nick' where id = 1</code></p><p> </p><p>方法二适合更新一条数据，也只能更新一条数据，当只有一条数据更新时推荐使用此方法，另外此方法还有一个好处，我们接着往下看</p><h1 id="具有auto-now属性字段的更新"><a href="#具有auto-now属性字段的更新" class="headerlink" title="具有auto_now属性字段的更新"></a>具有auto_now属性字段的更新</h1><p>我们通常会给表添加三个默认字段</p><ul><li>自增ID，这个django已经默认加了，就像上边的建表语句，虽然只写了username和is_active两个字段，但表建好后也会有一个默认的自增id字段</li><li>创建时间，用来标识这条记录的创建时间，具有<code>auto_now_add</code>属性，创建记录时会自动填充当前时间到此字段</li><li>修改时间，用来标识这条记录最后一次的修改时间，具有<code>auto_now</code>属性，当记录发生变化时填充当前时间到此字段</li></ul><p>就像下边这样的表结构</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-keyword">class</span> <span class="hljs-title class_">User</span>(models.Model):<br> create_time = models.DateTimeField(auto_now_add=<span class="hljs-literal">True</span>, verbose_name=<span class="hljs-string">'创建时间'</span>)<br> update_time = models.DateTimeField(auto_now=<span class="hljs-literal">True</span>, verbose_name=<span class="hljs-string">'更新时间'</span>)<br> username = models.CharField(max_length=<span class="hljs-number">255</span>, unique=<span class="hljs-literal">True</span>, verbose_name=<span class="hljs-string">'用户名'</span>)<br> is_active = models.BooleanField(default=<span class="hljs-literal">False</span>, verbose_name=<span class="hljs-string">'激活状态'</span>)<br></code></pre></td></tr></table></figure><p><strong>当表有字段具有<code>auto_now</code>属性且你希望他能自动更新时，必须使用上边方法二的更新，不然auto_now字段不会更新</strong>，也就是：</p><p> </p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><code class="hljs python">_t = User.objects.get(<span class="hljs-built_in">id</span>=<span class="hljs-number">1</span>)<br>_t.username=<span class="hljs-string">'nick'</span><br>_t.is_active=<span class="hljs-literal">True</span><br>_t.save()<br></code></pre></td></tr></table></figure><h1 id="json-x2F-dict类型数据更新字段"><a href="#json-x2F-dict类型数据更新字段" class="headerlink" title="json/dict类型数据更新字段"></a>json/dict类型数据更新字段</h1><p>目前主流的web开放方式都讲究前后端分离，分离之后前后端交互的数据格式大都用通用的jason型，那么如何用最少的代码方便的更新json格式数据到数据库呢？同样可以使用如下两种方法：</p><p>方法一：</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs python">data = {<span class="hljs-string">'username'</span>:<span class="hljs-string">'nick'</span>,<span class="hljs-string">'is_active'</span>:<span class="hljs-string">'0'</span>}<br>User.objects.<span class="hljs-built_in">filter</span>(<span class="hljs-built_in">id</span>=<span class="hljs-number">1</span>).update(**data)<br></code></pre></td></tr></table></figure><ul><li>同样这种方法不能自动更新具有<code>auto_now</code>属性字段的值</li><li>通常我们再变量前加一个星号(*)表示这个变量是元组/列表，加两个星号表示这个参数是字典</li></ul><p>方法二：</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><code class="hljs python">data = {<span class="hljs-string">'username'</span>:<span class="hljs-string">'nick'</span>,<span class="hljs-string">'is_active'</span>:<span class="hljs-string">'0'</span>}<br>_t = User.objects.get(<span class="hljs-built_in">id</span>=<span class="hljs-number">1</span>)<br>_t.__dict__.update(**data)<br>_t.save()<br></code></pre></td></tr></table></figure><ul><li>方法二和方法一同样无法自动更新<code>auto_now</code>字段的值</li><li>注意这里使用到了一个<code>__dict__</code>方法</li></ul><p>方法三：</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><code class="hljs python">_t = User.objects.get(<span class="hljs-built_in">id</span>=<span class="hljs-number">1</span>)<br>_t.role=Role.objects.get(<span class="hljs-built_in">id</span>=<span class="hljs-number">3</span>)<br>_t.save()<br></code></pre></td></tr></table></figure><h1 id="ForeignKey字段更新"><a href="#ForeignKey字段更新" class="headerlink" title="ForeignKey字段更新"></a>ForeignKey字段更新</h1><p>假如我们的表中有Foreignkey外键时，该如何更新呢？</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-keyword">class</span> <span class="hljs-title class_">User</span>(models.Model):<br> create_time = models.DateTimeField(auto_now_add=<span class="hljs-literal">True</span>, verbose_name=<span class="hljs-string">'创建时间'</span>)<br> update_time = models.DateTimeField(auto_now=<span class="hljs-literal">True</span>, verbose_name=<span class="hljs-string">'更新时间'</span>)<br> username = models.CharField(max_length=<span class="hljs-number">255</span>, unique=<span class="hljs-literal">True</span>, verbose_name=<span class="hljs-string">'用户名'</span>)<br> is_active = models.BooleanField(default=<span class="hljs-literal">False</span>, verbose_name=<span class="hljs-string">'激活状态'</span>)<br> role = models.ForeignKey(Role, on_delete=models.CASCADE, null=<span class="hljs-literal">True</span>, verbose_name=<span class="hljs-string">'角色'</span>)<br></code></pre></td></tr></table></figure><p>方法一：</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs python">User.objects.<span class="hljs-built_in">filter</span>(<span class="hljs-built_in">id</span>=<span class="hljs-number">1</span>).update(role=<span class="hljs-number">2</span>)<br></code></pre></td></tr></table></figure><ul><li>最简单的方法，直接让给role字段设置为一个id即可</li><li>当然也可以用dict作为参数更新：</li></ul><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs python">User.objects.<span class="hljs-built_in">filter</span>(<span class="hljs-built_in">id</span>=<span class="hljs-number">1</span>).update(**{<span class="hljs-string">'username'</span>:<span class="hljs-string">'nick'</span>,<span class="hljs-string">'role'</span>:<span class="hljs-number">3</span>})<br></code></pre></td></tr></table></figure><p>方法二：</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs python">_role = Role.objects.get(<span class="hljs-built_in">id</span>=<span class="hljs-number">2</span>)<br>User.objects.<span class="hljs-built_in">filter</span>(<span class="hljs-built_in">id</span>=<span class="hljs-number">1</span>).update(role=_role)<br></code></pre></td></tr></table></figure><ul><li>也可以赋值一个实例给role</li><li>当然也可以用dict作为参数更新：</li></ul><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs python">_role = Role.objects.get(<span class="hljs-built_in">id</span>=<span class="hljs-number">1</span>)<br>User.objects.<span class="hljs-built_in">filter</span>(<span class="hljs-built_in">id</span>=<span class="hljs-number">1</span>).update(**{<span class="hljs-string">'username'</span>:<span class="hljs-string">'nick'</span>,<span class="hljs-string">'role'</span>:_role})<br></code></pre></td></tr></table></figure><p>方法三：</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><code class="hljs python">_t = User.objects.get(<span class="hljs-built_in">id</span>=<span class="hljs-number">1</span>)<br>_t.role=Role.objects.get(<span class="hljs-built_in">id</span>=<span class="hljs-number">3</span>)<br>_t.save()<br></code></pre></td></tr></table></figure><ul><li>注意：<strong>这里的role必须赋值为一个对象，不能写id</strong>，不然会报错<code>"User.role" must be a "Role" instance</code></li><li>当使用dict作为参数更新时又有一点不同，如下代码：</li></ul><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><code class="hljs python">_t = User.objects.get(<span class="hljs-built_in">id</span>=<span class="hljs-number">1</span>)<br>_t.__dict__.update(**{<span class="hljs-string">'username'</span>:<span class="hljs-string">'nick'</span>,<span class="hljs-string">'role_id'</span>:<span class="hljs-number">2</span>})<br>_t.save()<br></code></pre></td></tr></table></figure><ul><li>**Foreignkey外键必须加上<code>_id</code>**，例如：{‘role_id’:3}</li><li>role_id后边必须跟一个id（int或str类型都可），不能跟role实例</li></ul><h1 id="ManyToManyField字段更新"><a href="#ManyToManyField字段更新" class="headerlink" title="ManyToManyField字段更新"></a>ManyToManyField字段更新</h1><p>假如我们的表中有ManyToManyField字段时更新又有什么影响呢？</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-keyword">class</span> <span class="hljs-title class_">User</span>(models.Model):<br> create_time = models.DateTimeField(auto_now_add=<span class="hljs-literal">True</span>, verbose_name=<span class="hljs-string">'创建时间'</span>)<br> update_time = models.DateTimeField(auto_now=<span class="hljs-literal">True</span>, verbose_name=<span class="hljs-string">'更新时间'</span>)<br> username = models.CharField(max_length=<span class="hljs-number">255</span>, unique=<span class="hljs-literal">True</span>, verbose_name=<span class="hljs-string">'用户名'</span>)<br> is_active = models.BooleanField(default=<span class="hljs-literal">False</span>, verbose_name=<span class="hljs-string">'激活状态'</span>)<br> role = models.ForeignKey(Role, on_delete=models.CASCADE, null=<span class="hljs-literal">True</span>, verbose_name=<span class="hljs-string">'角色'</span>)<br> groups = models.ManyToManyField(Group, null=<span class="hljs-literal">True</span>, verbose_name=<span class="hljs-string">'组'</span>)<br></code></pre></td></tr></table></figure><p>m2m更新：m2m字段没有直接更新的方法，只能通过清空再添加的方法更新了</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><code class="hljs python">_t = User.objects.get(<span class="hljs-built_in">id</span>=<span class="hljs-number">1</span>)<br>_t.groups.clear()<br>_t.groups.add(*[<span class="hljs-number">1</span>,<span class="hljs-number">3</span>,<span class="hljs-number">5</span>])<br>_t.save()<br></code></pre></td></tr></table></figure><ul><li><p><code>add()</code>：m2m字段添加一个值，当有多个值的时候可用列表，参照上边例子</p><ul><li>_t.groups.add(2)</li><li>_t.groups.add(Group.objects.get(id=2))</li></ul></li><li><p><code>remove()</code>：m2m字段移除一个值，，当有多个值的时候可用列表，参照上边例子</p><ul><li>_t.groups.remove(2)</li><li>_t.groups.remove(Group.objects.get(id=2))</li></ul></li><li><p><code>clear()</code>：清空m2m字段的值</p></li></ul>]]></content>
<categories>
<category>Django</category>
</categories>
<tags>
<tag>python django</tag>
</tags>
</entry>
<entry>
<title>Django ORM select 查询操作</title>
<link href="/2020/07/01/Django%20ORM%20select%20%E6%9F%A5%E8%AF%A2%E6%93%8D%E4%BD%9C/"/>
<url>/2020/07/01/Django%20ORM%20select%20%E6%9F%A5%E8%AF%A2%E6%93%8D%E4%BD%9C/</url>
<content type="html"><![CDATA[<span id="more"></span><h2 id="基本操作"><a href="#基本操作" class="headerlink" title="基本操作"></a>基本操作</h2><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-comment"># 获取所有数据，对应SQL：select * from User</span><br>User.objects.<span class="hljs-built_in">all</span>()<br><br><span class="hljs-comment"># 匹配，对应SQL：select * from User where name = 'Uzi'</span><br>User.objects.<span class="hljs-built_in">filter</span>(name=<span class="hljs-string">'Uzi'</span>)<br><br><span class="hljs-comment"># 不匹配，对应SQL：select * from User where name != 'Uzi'</span><br>User.objects.exclude(name=<span class="hljs-string">'Uzi'</span>)<br><br><span class="hljs-comment"># 获取单条数据（有且仅有一条，id唯一），对应SQL：select * from User where id = 724</span><br>User.objects.get(<span class="hljs-built_in">id</span>=<span class="hljs-number">123</span>)<br></code></pre></td></tr></table></figure><h2 id="常用操作"><a href="#常用操作" class="headerlink" title="常用操作"></a>常用操作</h2><p> </p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-comment"># 获取总数，对应SQL：select count(1) from User</span><br>User.objects.count()<br>User.objects.<span class="hljs-built_in">filter</span>(name=<span class="hljs-string">'Uzi'</span>).count()<br><br><span class="hljs-comment"># 比较，gt:>，gte:>=，lt:<，lte:<=,对应SQL：select * from User where id > 724</span><br>User.objects.<span class="hljs-built_in">filter</span>(id__gt=<span class="hljs-number">724</span>)<br>User.objects.<span class="hljs-built_in">filter</span>(id__gt=<span class="hljs-number">1</span>, id__lt=<span class="hljs-number">10</span>)<br><br><span class="hljs-comment"># 包含，in，对应SQL：select * from User where id in (11,22,33)</span><br>User.objects.<span class="hljs-built_in">filter</span>(id__in=[<span class="hljs-number">11</span>, <span class="hljs-number">22</span>, <span class="hljs-number">33</span>])<br>User.objects.exclude(id__in=[<span class="hljs-number">11</span>, <span class="hljs-number">22</span>, <span class="hljs-number">33</span>])<br><br><span class="hljs-comment"># isnull：isnull=True为空，isnull=False不为空，对应SQL：select * from User where pub_date is null</span><br>User.objects.<span class="hljs-built_in">filter</span>(pub_date__isnull=<span class="hljs-literal">True</span>)<br><br><span class="hljs-comment"># like，contains大小写敏感，icontains大小写不敏感，相同用法的还有startswith、endswith</span><br>User.objects.<span class="hljs-built_in">filter</span>(name__contains=<span class="hljs-string">"sre"</span>)<br>User.objects.exclude(name__contains=<span class="hljs-string">"sre"</span>)<br><br><span class="hljs-comment"># 范围，between and，对应SQL：select * from User where id between 3 and 8</span><br>User.objects.<span class="hljs-built_in">filter</span>(id__range=[<span class="hljs-number">3</span>, <span class="hljs-number">8</span>])<br><br><span class="hljs-comment"># 排序，order by，'id'按id正序，'-id'按id倒叙</span><br>User.objects.<span class="hljs-built_in">filter</span>(name=<span class="hljs-string">'Uzi'</span>).order_by(<span class="hljs-string">'id'</span>)<br>User.objects.<span class="hljs-built_in">filter</span>(name=<span class="hljs-string">'Uzi'</span>).order_by(<span class="hljs-string">'-id'</span>)<br><br><span class="hljs-comment"># 多级排序，order by，先按name进行正序排列，如果name一致则再按照id倒叙排列</span><br>User.objects.<span class="hljs-built_in">filter</span>(name=<span class="hljs-string">'Uzi'</span>).order_by(<span class="hljs-string">'name'</span>,<span class="hljs-string">'-id'</span>)<br></code></pre></td></tr></table></figure><h2 id="进阶操作"><a href="#进阶操作" class="headerlink" title="进阶操作"></a>进阶操作</h2><p> </p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-comment"># limit，对应SQL：select * from User limit 3;</span><br>User.objects.<span class="hljs-built_in">all</span>()[:<span class="hljs-number">3</span>]<br><br><span class="hljs-comment"># limit，取第三条以后的数据，没有对应的SQL，类似的如：select * from User limit 3,10000000，从第3条开始取数据，取10000000条（10000000大于表中数据条数）</span><br>User.objects.<span class="hljs-built_in">all</span>()[<span class="hljs-number">3</span>:]<br><br><span class="hljs-comment"># offset，取出结果的第10-20条数据（不包含10，包含20）,也没有对应SQL，参考上边的SQL写法</span><br>User.objects.<span class="hljs-built_in">all</span>()[<span class="hljs-number">10</span>:<span class="hljs-number">20</span>]<br><br><span class="hljs-comment"># 分组，group by，对应SQL：select username,count(1) from User group by username;</span><br><span class="hljs-keyword">from</span> django.db.models <span class="hljs-keyword">import</span> Count<br>User.objects.values_list(<span class="hljs-string">'username'</span>).annotate(Count(<span class="hljs-string">'id'</span>))<br><br><span class="hljs-comment"># 去重distinct，对应SQL：select distinct(username) from User</span><br>User.objects.values(<span class="hljs-string">'username'</span>).distinct().count()<br><br><span class="hljs-comment"># filter多列、查询多列，对应SQL：select username,fullname from accounts_user</span><br>User.objects.values_list(<span class="hljs-string">'username'</span>, <span class="hljs-string">'fullname'</span>)<br><br><span class="hljs-comment"># filter单列、查询单列，正常values_list给出的结果是个列表，里边里边的每条数据对应一个元组，当只查询一列时，可以使用flat标签去掉元组，将每条数据的结果以字符串的形式存储在列表中，从而避免解析元组的麻烦</span><br>User.objects.values_list(<span class="hljs-string">'username'</span>, flat=<span class="hljs-literal">True</span>)<br><br><span class="hljs-comment"># int字段取最大值、最小值、综合、平均数</span><br><span class="hljs-keyword">from</span> django.db.models <span class="hljs-keyword">import</span> Sum,Count,Max,Min,Avg<br><br>User.objects.aggregate(Count(‘<span class="hljs-built_in">id</span>’))<br>User.objects.aggregate(Sum(‘age’))<br></code></pre></td></tr></table></figure><h2 id="时间字段"><a href="#时间字段" class="headerlink" title="时间字段"></a>时间字段</h2><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-comment"># 匹配日期，date</span><br>User.objects.<span class="hljs-built_in">filter</span>(create_time__date=datetime.date(<span class="hljs-number">2018</span>, <span class="hljs-number">8</span>, <span class="hljs-number">1</span>))<br>User.objects.<span class="hljs-built_in">filter</span>(create_time__date__gt=datetime.date(<span class="hljs-number">2018</span>, <span class="hljs-number">8</span>, <span class="hljs-number">2</span>))<br><br><span class="hljs-comment"># 匹配年，year，相同用法的还有匹配月month，匹配日day，匹配周week_day，匹配时hour，匹配分minute，匹配秒second</span><br>User.objects.<span class="hljs-built_in">filter</span>(create_time__year=<span class="hljs-number">2018</span>)<br>User.objects.<span class="hljs-built_in">filter</span>(create_time__year__gte=<span class="hljs-number">2018</span>)<br><br><span class="hljs-comment"># 按天统计归档</span><br>today = datetime.date.today()<br>select = {<span class="hljs-string">'day'</span>: connection.ops.date_trunc_sql(<span class="hljs-string">'day'</span>, <span class="hljs-string">'create_time'</span>)}<br>deploy_date_count = Task.objects.<span class="hljs-built_in">filter</span>(<br> create_time__range=(today - datetime.timedelta(days=<span class="hljs-number">7</span>), today)<br>).extra(select=select).values(<span class="hljs-string">'day'</span>).annotate(number=Count(<span class="hljs-string">'id'</span>))<br></code></pre></td></tr></table></figure><h2 id="Q-的使用"><a href="#Q-的使用" class="headerlink" title="Q 的使用"></a>Q 的使用</h2><p><strong>Q对象可以对关键字参数进行封装，从而更好的应用多个查询，可以组合&(and)、|(or)、~(not)操作符。</strong></p><p>例如下边的语句 </p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-keyword">from</span> django.db.models <span class="hljs-keyword">import</span> Q<br><br>User.objects.<span class="hljs-built_in">filter</span>(<br> Q(role__startswith=<span class="hljs-string">'sre_'</span>),<br> Q(name=<span class="hljs-string">'公众号'</span>) | Q(name=<span class="hljs-string">'Uzi'</span>)<br>)<br></code></pre></td></tr></table></figure><figure class="highlight html"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs html">转换成SQL语句如下：<br></code></pre></td></tr></table></figure><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs sql"><span class="hljs-keyword">select</span> <span class="hljs-operator">*</span> <span class="hljs-keyword">from</span> <span class="hljs-keyword">User</span> <span class="hljs-keyword">where</span> role <span class="hljs-keyword">like</span> <span class="hljs-string">'sre_%'</span> <span class="hljs-keyword">and</span> (name<span class="hljs-operator">=</span><span class="hljs-string">'公众号'</span> <span class="hljs-keyword">or</span> name<span class="hljs-operator">=</span><span class="hljs-string">'Uzi'</span>)<br></code></pre></td></tr></table></figure><figure class="highlight html"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><code class="hljs html">通常更多的时候我们用Q来做搜索逻辑，比如前台搜索框输入一个字符，后台去数据库中检索标题或内容中是否包含<br><br></code></pre></td></tr></table></figure><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br></pre></td><td class="code"><pre><code class="hljs python">_s = request.GET.get(<span class="hljs-string">'search'</span>)<br><br>_t = Blog.objects.<span class="hljs-built_in">all</span>()<br><span class="hljs-keyword">if</span> _s:<br> _t = _t.<span class="hljs-built_in">filter</span>(<br> Q(title__icontains=_s) |<br> Q(content__icontains=_s)<br> )<br><br><span class="hljs-keyword">return</span> _t<br>外键：ForeignKey<br>表结构：<br><span class="hljs-keyword">class</span> <span class="hljs-title class_">Role</span>(models.Model):<br> name = models.CharField(max_length=<span class="hljs-number">16</span>, unique=<span class="hljs-literal">True</span>)<br><br><br><span class="hljs-keyword">class</span> <span class="hljs-title class_">User</span>(models.Model):<br> username = models.EmailField(max_length=<span class="hljs-number">255</span>, unique=<span class="hljs-literal">True</span>)<br> role = models.ForeignKey(Role, on_delete=models.CASCADE)<br></code></pre></td></tr></table></figure><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br><span class="line">97</span><br><span class="line">98</span><br><span class="line">99</span><br><span class="line">100</span><br><span class="line">101</span><br><span class="line">102</span><br></pre></td><td class="code"><pre><code class="hljs python">正向查询:<br><span class="hljs-comment"># 查询用户的角色名</span><br>_t = User.objects.get(username=<span class="hljs-string">'Uzi'</span>)<br>_t.role.name<br>反向查询：<br><span class="hljs-comment"># 查询角色下包含的所有用户</span><br>_t = Role.objects.get(name=<span class="hljs-string">'Role03'</span>)<br>_t.user_set.<span class="hljs-built_in">all</span>()<br>另一种反向查询的方法：<br>_t = Role.objects.get(name=<span class="hljs-string">'Role03'</span>)<br><br><span class="hljs-comment"># 这种方法比上一种_set的方法查询速度要快</span><br>User.objects.<span class="hljs-built_in">filter</span>(role=_t)<br>第三种反向查询的方法：<br>如果外键字段有related_name属性，例如models如下：<br><br><span class="hljs-keyword">class</span> <span class="hljs-title class_">User</span>(models.Model):<br> username = models.EmailField(max_length=<span class="hljs-number">255</span>, unique=<span class="hljs-literal">True</span>)<br> role = models.ForeignKey(Role, on_delete=models.CASCADE,related_name=<span class="hljs-string">'roleUsers'</span>)<br>那么可以直接用related_name属性取到某角色的所有用户<br><br>_t = Role.objects.get(name = <span class="hljs-string">'Role03'</span>)<br>_t.roleUsers.<span class="hljs-built_in">all</span>()<br>M2M：ManyToManyField<br>表结构：<br><span class="hljs-keyword">class</span> <span class="hljs-title class_">Group</span>(models.Model):<br> name = models.CharField(max_length=<span class="hljs-number">16</span>, unique=<span class="hljs-literal">True</span>)<br><br><span class="hljs-keyword">class</span> <span class="hljs-title class_">User</span>(models.Model):<br> username = models.CharField(max_length=<span class="hljs-number">255</span>, unique=<span class="hljs-literal">True</span>)<br> groups = models.ManyToManyField(Group, related_name=<span class="hljs-string">'groupUsers'</span>)<br>正向查询:<br><span class="hljs-comment"># 查询用户隶属组</span><br>_t = User.objects.get(username = <span class="hljs-string">'Uzi'</span>)<br>_t.groups.<span class="hljs-built_in">all</span>()<br>反向查询：<br><span class="hljs-comment"># 查询组包含用户</span><br>_t = Group.objects.get(name = <span class="hljs-string">'groupC'</span>)<br>_t.user_set.<span class="hljs-built_in">all</span>()<br>同样M2M字段如果有related_name属性，那么可以直接用下边的方式反查<br><br>_t = Group.objects.get(name = <span class="hljs-string">'groupC'</span>)<br>_t.groupUsers.<span class="hljs-built_in">all</span>()<br>get_object_or_404<br>正常如果我们要去数据库里搜索某一条数据时，通常使用下边的方法：<br><br>_t = User.objects.get(<span class="hljs-built_in">id</span>=<span class="hljs-number">734</span>)<br><br>但当<span class="hljs-built_in">id</span>=<span class="hljs-number">724</span>的数据不存在时，程序将会抛出一个错误<br><br>abcer.models.DoesNotExist: User matching query does <span class="hljs-keyword">not</span> exist.<br><br>为了程序兼容和异常判断，我们可以使用下边两种方式:<br><br>方式一：get改为<span class="hljs-built_in">filter</span><br>_t = User.objects.<span class="hljs-built_in">filter</span>(<span class="hljs-built_in">id</span>=<span class="hljs-number">724</span>)<br><span class="hljs-comment"># 取出_t之后再去判断_t是否存在</span><br>方式二：使用get_object_or_404<br><span class="hljs-keyword">from</span> django.shortcuts <span class="hljs-keyword">import</span> get_object_or_404<br><br>_t = get_object_or_404(User, <span class="hljs-built_in">id</span>=<span class="hljs-number">724</span>)<br><span class="hljs-comment"># get_object_or_404方法，它会先调用django的get方法，如果查询的对象不存在的话，则抛出一个Http404的异常</span><br>实现方法类似于下边这样：<br><br><span class="hljs-keyword">from</span> django.http <span class="hljs-keyword">import</span> Http404<br><br><span class="hljs-keyword">try</span>:<br> _t = User.objects.get(<span class="hljs-built_in">id</span>=<span class="hljs-number">724</span>)<br><span class="hljs-keyword">except</span> User.DoesNotExist:<br> <span class="hljs-keyword">raise</span> Http404<br>get_or_create<br>顾名思义，查找一个对象如果不存在则创建，如下：<br><br><span class="hljs-built_in">object</span>, created = User.objects.get_or_create(username=<span class="hljs-string">'Uzi'</span>)<br>返回一个由<span class="hljs-built_in">object</span>和created组成的元组，其中<span class="hljs-built_in">object</span>就是一个查询到的或者是被创建的对象，created是一个表示是否创建了新对象的布尔值<br><br>实现方式类似于下边这样：<br><br><span class="hljs-keyword">try</span>:<br> <span class="hljs-built_in">object</span> = User.objects.get(username=<span class="hljs-string">'Uzi'</span>)<br><br> created = <span class="hljs-literal">False</span><br>exception User.DoesNoExist:<br> <span class="hljs-built_in">object</span> = User(username=<span class="hljs-string">'Uzi'</span>)<br> <span class="hljs-built_in">object</span>.save()<br><br> created = <span class="hljs-literal">True</span><br><br>returen <span class="hljs-built_in">object</span>, created<br><br>执行原生SQL<br>Django中能用ORM的就用它ORM吧，不建议执行原生SQL，可能会有一些安全问题，如果实在是SQL太复杂ORM实现不了，那就看看下边执行原生SQL的方法，跟直接使用pymysql基本一致了<br><br><span class="hljs-keyword">from</span> django.db <span class="hljs-keyword">import</span> connection<br><br><span class="hljs-keyword">with</span> connection.cursor() <span class="hljs-keyword">as</span> cursor:<br> cursor.execute(<span class="hljs-string">'select * from accounts_User'</span>)<br> row = cursor.fetchall()<br><br><span class="hljs-keyword">return</span> row<br><br>注意这里表名字要用app名+下划线+model名的方式<br></code></pre></td></tr></table></figure>]]></content>
<categories>
<category>Django</category>
</categories>
<tags>
<tag>orm python</tag>
</tags>
</entry>
<entry>
<title>python epoll socket实例</title>
<link href="/2020/06/23/python%20epoll%20socket%E5%AE%9E%E4%BE%8B/"/>
<url>/2020/06/23/python%20epoll%20socket%E5%AE%9E%E4%BE%8B/</url>
<content type="html"><![CDATA[<span id="more"></span><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br><span class="line">97</span><br><span class="line">98</span><br><span class="line">99</span><br><span class="line">100</span><br><span class="line">101</span><br><span class="line">102</span><br><span class="line">103</span><br><span class="line">104</span><br><span class="line">105</span><br><span class="line">106</span><br><span class="line">107</span><br><span class="line">108</span><br><span class="line">109</span><br><span class="line">110</span><br><span class="line">111</span><br><span class="line">112</span><br><span class="line">113</span><br><span class="line">114</span><br><span class="line">115</span><br><span class="line">116</span><br><span class="line">117</span><br><span class="line">118</span><br><span class="line">119</span><br><span class="line">120</span><br><span class="line">121</span><br><span class="line">122</span><br><span class="line">123</span><br><span class="line">124</span><br><span class="line">125</span><br><span class="line">126</span><br><span class="line">127</span><br><span class="line">128</span><br><span class="line">129</span><br><span class="line">130</span><br><span class="line">131</span><br><span class="line">132</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-keyword">import</span> socket, logging<br><span class="hljs-keyword">import</span> select, errno<br><br>logger = logging.getLogger(<span class="hljs-string">"network-server"</span>)<br><br><br><span class="hljs-keyword">def</span> <span class="hljs-title function_">InitLog</span>():<br> logger.setLevel(logging.DEBUG)<br><br> fh = logging.FileHandler(<span class="hljs-string">"network-server.log"</span>)<br> fh.setLevel(logging.DEBUG)<br> ch = logging.StreamHandler()<br> ch.setLevel(logging.ERROR)<br><br> formatter = logging.Formatter(<span class="hljs-string">"%(asctime)s - %(name)s - %(levelname)s - %(message)s"</span>)<br> ch.setFormatter(formatter)<br> fh.setFormatter(formatter)<br><br> logger.addHandler(fh)<br> logger.addHandler(ch)<br><br><br><span class="hljs-keyword">if</span> __name__ == <span class="hljs-string">"__main__"</span>:<br><br> InitLog()<br><br> <span class="hljs-keyword">try</span>:<br> <span class="hljs-comment"># 创建 TCP socket 作为监听 socket</span><br> listen_fd = socket.socket(socket.AF_INET, socket.SOCK_STREAM, <span class="hljs-number">0</span>)<br> <span class="hljs-keyword">except</span> socket.error <span class="hljs-keyword">as</span> msg:<br> logger.error(<span class="hljs-string">"create socket failed"</span>)<br><br> <span class="hljs-keyword">try</span>:<br> <span class="hljs-comment"># 设置 SO_REUSEADDR 选项</span><br> listen_fd.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, <span class="hljs-number">1</span>)<br> <span class="hljs-keyword">except</span> socket.error <span class="hljs-keyword">as</span> msg:<br> logger.error(<span class="hljs-string">"setsocketopt SO_REUSEADDR failed"</span>)<br><br> <span class="hljs-keyword">try</span>:<br> <span class="hljs-comment"># 进行 bind -- 此处未指定 ip 地址，即 bind 了全部网卡 ip 上</span><br> listen_fd.bind((<span class="hljs-string">'127.0.0.1'</span>, <span class="hljs-number">8008</span>))<br> <span class="hljs-keyword">except</span> socket.error <span class="hljs-keyword">as</span> msg:<br> logger.error(<span class="hljs-string">"bind failed"</span>)<br><br> <span class="hljs-keyword">try</span>:<br> <span class="hljs-comment"># 设置 listen 的 backlog 数</span><br> listen_fd.listen(<span class="hljs-number">100</span>)<br> <span class="hljs-keyword">except</span> socket.error <span class="hljs-keyword">as</span> msg:<br> logger.error(msg)<br><br> <span class="hljs-keyword">try</span>:<br> <span class="hljs-comment"># 创建 epoll 句柄</span><br> epoll_fd = select.epoll()<br> <span class="hljs-comment"># 向 epoll 句柄中注册 监听 socket 的 可读 事件</span><br> epoll_fd.register(listen_fd.fileno(), select.EPOLLIN)<br> <span class="hljs-keyword">except</span> select.error <span class="hljs-keyword">as</span> msg:<br> logger.error(msg)<br><br> connections = {}<br> addresses = {}<br> datalist = {}<br> <span class="hljs-keyword">while</span> <span class="hljs-literal">True</span>:<br> <span class="hljs-comment"># epoll 进行 fd 扫描的地方 -- 未指定超时时间则为阻塞等待</span><br> epoll_list = epoll_fd.poll()<br><br> <span class="hljs-keyword">for</span> fd, events <span class="hljs-keyword">in</span> epoll_list:<br> <span class="hljs-comment"># 若为监听 fd 被激活</span><br> <span class="hljs-keyword">if</span> fd == listen_fd.fileno():<br> <span class="hljs-comment"># 进行 accept -- 获得连接上来 client 的 ip 和 port，以及 socket 句柄</span><br> conn, addr = listen_fd.accept()<br> logger.debug(<span class="hljs-string">"accept connection from %s, %d, fd = %d"</span> % (addr[<span class="hljs-number">0</span>], addr[<span class="hljs-number">1</span>], conn.fileno()))<br> <span class="hljs-comment"># 将连接 socket 设置为 非阻塞</span><br> conn.setblocking(<span class="hljs-number">0</span>)<br> <span class="hljs-comment"># 向 epoll 句柄中注册 连接 socket 的 可读 事件</span><br> epoll_fd.register(conn.fileno(), select.EPOLLIN | select.EPOLLET)<br> <span class="hljs-comment"># 将 conn 和 addr 信息分别保存起来</span><br> connections[conn.fileno()] = conn<br> addresses[conn.fileno()] = addr<br> <span class="hljs-keyword">elif</span> select.EPOLLIN & events:<br> <span class="hljs-comment"># 有 可读 事件激活</span><br> datas = <span class="hljs-string">''</span><br> <span class="hljs-keyword">while</span> <span class="hljs-literal">True</span>:<br> <span class="hljs-keyword">try</span>:<br> <span class="hljs-comment"># 从激活 fd 上 recv 10 字节数据</span><br> data = connections[fd].recv(<span class="hljs-number">10</span>)<br> <span class="hljs-comment"># 若当前没有接收到数据，并且之前的累计数据也没有</span><br> <span class="hljs-keyword">if</span> <span class="hljs-keyword">not</span> data <span class="hljs-keyword">and</span> <span class="hljs-keyword">not</span> datas:<br> <span class="hljs-comment"># 从 epoll 句柄中移除该 连接 fd</span><br> epoll_fd.unregister(fd)<br> <span class="hljs-comment"># server 侧主动关闭该 连接 fd</span><br> connections[fd].close()<br> logger.debug(<span class="hljs-string">"%s, %d closed"</span> % (addresses[fd][<span class="hljs-number">0</span>], addresses[fd][<span class="hljs-number">1</span>]))<br> <span class="hljs-keyword">break</span><br> <span class="hljs-keyword">else</span>:<br> <span class="hljs-comment"># 将接收到的数据拼接保存在 datas 中</span><br> datas += data<br> <span class="hljs-keyword">except</span> socket.error <span class="hljs-keyword">as</span> msg:<br> <span class="hljs-comment"># 在 非阻塞 socket 上进行 recv 需要处理 读穿 的情况</span><br> <span class="hljs-comment"># 这里实际上是利用 读穿 出 异常 的方式跳到这里进行后续处理</span><br> <span class="hljs-keyword">if</span> msg.errno == errno.EAGAIN:<br> logger.debug(<span class="hljs-string">"%s receive %s"</span> % (fd, datas))<br> <span class="hljs-comment"># 将已接收数据保存起来</span><br> datalist[fd] = datas<br> <span class="hljs-comment"># 更新 epoll 句柄中连接d 注册事件为 可写</span><br> epoll_fd.modify(fd, select.EPOLLET | select.EPOLLOUT)<br> <span class="hljs-keyword">break</span><br> <span class="hljs-keyword">else</span>:<br> <span class="hljs-comment"># 出错处理</span><br> epoll_fd.unregister(fd)<br> connections[fd].close()<br> logger.error(msg)<br> <span class="hljs-keyword">break</span><br> <span class="hljs-keyword">elif</span> select.EPOLLHUP & events:<br> <span class="hljs-comment"># 有 HUP 事件激活</span><br> epoll_fd.unregister(fd)<br> connections[fd].close()<br> logger.debug(<span class="hljs-string">"%s, %d closed"</span> % (addresses[fd][<span class="hljs-number">0</span>], addresses[fd][<span class="hljs-number">1</span>]))<br> <span class="hljs-keyword">elif</span> select.EPOLLOUT & events:<br> <span class="hljs-comment"># 有 可写 事件激活</span><br> sendLen = <span class="hljs-number">0</span><br> <span class="hljs-comment"># 通过 while 循环确保将 buf 中的数据全部发送出去</span><br> <span class="hljs-keyword">while</span> <span class="hljs-literal">True</span>:<br> <span class="hljs-comment"># 将之前收到的数据发回 client -- 通过 sendLen 来控制发送位置</span><br> sendLen += connections[fd].send(datalist[fd][sendLen:])<br> <span class="hljs-comment"># 在全部发送完毕后退出 while 循环</span><br> <span class="hljs-keyword">if</span> sendLen == <span class="hljs-built_in">len</span>(datalist[fd]):<br> <span class="hljs-keyword">break</span><br> <span class="hljs-comment"># 更新 epoll 句柄中连接 fd 注册事件为 可读</span><br> epoll_fd.modify(fd, select.EPOLLIN | select.EPOLLET)<br> <span class="hljs-keyword">else</span>:<br> <span class="hljs-comment"># 其他 epoll 事件不进行处理</span><br> <span class="hljs-keyword">continue</span><br></code></pre></td></tr></table></figure>]]></content>
<categories>
<category>python加油鸭</category>
</categories>
<tags>
<tag>epoll socket</tag>
</tags>
</entry>
<entry>
<title>进程间通信-Queue</title>
<link href="/2020/06/19/%E8%BF%9B%E7%A8%8B%E9%97%B4%E9%80%9A%E4%BF%A1-Queue/"/>
<url>/2020/06/19/%E8%BF%9B%E7%A8%8B%E9%97%B4%E9%80%9A%E4%BF%A1-Queue/</url>
<content type="html"><![CDATA[<span id="more"></span><h2 id="1-Queue的使用"><a href="#1-Queue的使用" class="headerlink" title="1. Queue的使用"></a>1. Queue的使用</h2><p>可以使用multiprocessing模块的Queue实现多进程之间的数据传递，Queue本身是一个消息列队程序，首先用一个小实例来演示一下Queue的工作原理：</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-keyword">import</span> multiprocessing<br><span class="hljs-keyword">import</span> time<br><br><span class="hljs-keyword">if</span> __name__ == <span class="hljs-string">'__main__'</span>:<br> <span class="hljs-comment"># 创建消息队列, 3:表示队列中最大消息个数</span><br> queue = multiprocessing.Queue(<span class="hljs-number">3</span>)<br> <span class="hljs-comment"># 放入数据</span><br> queue.put(<span class="hljs-number">1</span>)<br> queue.put(<span class="hljs-string">"hello"</span>)<br> queue.put([<span class="hljs-number">3</span>,<span class="hljs-number">5</span>])<br> <span class="hljs-comment"># 总结: 队列可以放入任意数据类型</span><br> <span class="hljs-comment"># 提示： 如果队列满了，需要等待队列有空闲位置才能放入数据，否则一直等待</span><br> <span class="hljs-comment"># queue.put((5,6))</span><br> <span class="hljs-comment"># 提示： 如果队列满了，不等待队列有空闲位置，如果放入不成功直接崩溃</span><br> <span class="hljs-comment"># queue.put_nowait((5,6))</span><br> <span class="hljs-comment"># 建议： 向队列放入数据统一使用put</span><br><br> <span class="hljs-comment"># 查看队列是否满了</span><br> <span class="hljs-comment"># print(queue.full())</span><br><br> <span class="hljs-comment"># 注意点：queue.empty()判断队列是否空了不可靠</span><br> <span class="hljs-comment"># 查看队列是否空了</span><br> <span class="hljs-comment"># print(queue.empty())</span><br><br> <span class="hljs-comment"># 解决办法: 1. 加延时操作 2. 使用判断队列的个数,不使用empty</span><br> <span class="hljs-comment"># time.sleep(0.01)</span><br> <span class="hljs-keyword">if</span> queue.qsize() == <span class="hljs-number">0</span>:<br> <span class="hljs-built_in">print</span>(<span class="hljs-string">"队列为空"</span>)<br> <span class="hljs-keyword">else</span>:<br> <span class="hljs-built_in">print</span>(<span class="hljs-string">"队列不为空"</span>)<br><br> <span class="hljs-comment"># 获取队列的个数</span><br> size = queue.qsize()<br> <span class="hljs-built_in">print</span>(size)<br><br> <span class="hljs-comment"># 获取数据</span><br> value = queue.get()<br> <span class="hljs-built_in">print</span>(value)<br> <span class="hljs-comment"># 获取队列的个数</span><br> size = queue.qsize()<br> <span class="hljs-built_in">print</span>(size)<br> <span class="hljs-comment"># 获取数据</span><br> value = queue.get()<br> <span class="hljs-built_in">print</span>(value)<br> <span class="hljs-comment"># 获取数据</span><br> value = queue.get()<br> <span class="hljs-built_in">print</span>(value)<br><br> <span class="hljs-comment"># 获取队列的个数</span><br> size = queue.qsize()<br> <span class="hljs-built_in">print</span>(size)<br><br> <span class="hljs-comment"># 提示：如果队列空了，再取值需要等待，只有队列有值以后才能获取队列中数据</span><br> <span class="hljs-comment"># value = queue.get()</span><br> <span class="hljs-comment"># print(value)</span><br> <span class="hljs-comment"># 提示： 如果队列空了 ，不需要等待队列有值，但是如果取值的时候发现队列空了直接崩溃</span><br> <span class="hljs-comment"># 建议大家: 向队列取值使用get</span><br> <span class="hljs-comment"># value = queue.get_nowait()</span><br> <span class="hljs-comment"># print(value)</span><br></code></pre></td></tr></table></figure><p>运行结果:</p><figure class="highlight angelscript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><code class="hljs angelscript">队列不为空<br><span class="hljs-number">3</span><br><span class="hljs-number">1</span><br><span class="hljs-number">2</span><br>hello<br><span class="hljs-string">[3, 5]</span><br><span class="hljs-number">0</span><br></code></pre></td></tr></table></figure><p>说明</p><p>初始化Queue()对象时（例如：q=Queue()），若括号中没有指定最大可接收的消息数量，或数量为负值，那么就代表可接受的消息数量没有上限（直到内存的尽头）；</p><ul><li><p>Queue.qsize()：返回当前队列包含的消息数量；</p></li><li><p>Queue.empty()：如果队列为空，返回True，反之False , 注意这个操作是不可靠的。</p></li><li><p>Queue.full()：如果队列满了，返回True,反之False；</p></li><li><p>Queue.get([block[, timeout]])：获取队列中的一条消息，然后将其从列队中移除，block默认值为True；</p></li></ul><p>1）如果block使用默认值，且没有设置timeout（单位秒），消息列队如果为空，此时程序将被阻塞（停在读取状态），直到从消息列队读到消息为止，如果设置了timeout，则会等待timeout秒，若还没读取到任何消息，则抛出”Queue.Empty”异常；</p><p>2）如果block值为False，消息列队如果为空，则会立刻抛出”Queue.Empty”异常；</p><ul><li><p>Queue.get_nowait()：相当Queue.get(False)；</p></li><li><p>Queue.put(item,[block[, timeout]])：将item消息写入队列，block默认值为True；</p></li></ul><p>1）如果block使用默认值，且没有设置timeout（单位秒），消息列队如果已经没有空间可写入，此时程序将被阻塞（停在写入状态），直到从消息列队腾出空间为止，如果设置了timeout，则会等待timeout秒，若还没空间，则抛出”Queue.Full”异常；</p><p>2）如果block值为False，消息列队如果没有空间可写入，则会立刻抛出”Queue.Full”异常；</p><ul><li>Queue.put_nowait(item)：相当Queue.put(item, False)；</li></ul><h2 id="2-消息队列Queue完成进程间通信的演练"><a href="#2-消息队列Queue完成进程间通信的演练" class="headerlink" title="2. 消息队列Queue完成进程间通信的演练"></a>2. 消息队列Queue完成进程间通信的演练</h2><p>我们以Queue为例，在父进程中创建两个子进程，一个往Queue里写数据，一个从Queue里读数据：</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-keyword">import</span> multiprocessing<br><span class="hljs-keyword">import</span> time<br><br><br><span class="hljs-comment"># 写入数据</span><br><span class="hljs-keyword">def</span> <span class="hljs-title function_">write_data</span>(<span class="hljs-params">queue</span>):<br> <span class="hljs-keyword">for</span> i <span class="hljs-keyword">in</span> <span class="hljs-built_in">range</span>(<span class="hljs-number">10</span>):<br> <span class="hljs-keyword">if</span> queue.full():<br> <span class="hljs-built_in">print</span>(<span class="hljs-string">"队列满了"</span>)<br> <span class="hljs-keyword">break</span><br> queue.put(i)<br> time.sleep(<span class="hljs-number">0.2</span>)<br> <span class="hljs-built_in">print</span>(i)<br><br><br><span class="hljs-comment"># 读取数据</span><br><span class="hljs-keyword">def</span> <span class="hljs-title function_">read_data</span>(<span class="hljs-params">queue</span>):<br> <span class="hljs-keyword">while</span> <span class="hljs-literal">True</span>:<br> <span class="hljs-comment"># 加入数据从队列取完了，那么跳出循环</span><br> <span class="hljs-keyword">if</span> queue.qsize() == <span class="hljs-number">0</span>:<br> <span class="hljs-built_in">print</span>(<span class="hljs-string">"队列空了"</span>)<br> <span class="hljs-keyword">break</span><br> value = queue.get()<br> <span class="hljs-built_in">print</span>(value)<br><br><br><span class="hljs-keyword">if</span> __name__ == <span class="hljs-string">'__main__'</span>:<br> <span class="hljs-comment"># 创建消息队列</span><br> queue = multiprocessing.Queue(<span class="hljs-number">5</span>)<br><br> <span class="hljs-comment"># 创建写入数据的进程</span><br> write_process = multiprocessing.Process(target=write_data, args=(queue,))<br> <span class="hljs-comment"># 创建读取数据的进程</span><br> read_process = multiprocessing.Process(target=read_data, args=(queue,))<br><br> <span class="hljs-comment"># 启动进程</span><br> write_process.start()<br> <span class="hljs-comment"># 主进程等待写入进程执行完成以后代码再继续往下执行</span><br> write_process.join()<br> read_process.start()<br></code></pre></td></tr></table></figure><p>运行结果：</p><figure class="highlight"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><code class="hljs">0<br>1<br>2<br>3<br>4<br>队列满了<br>0<br>1<br>2<br>3<br>4<br>队列空了<br></code></pre></td></tr></table></figure>]]></content>
<categories>
<category>python加油鸭</category>
</categories>
<tags>
<tag>python 队列 多进程</tag>
</tags>
</entry>
<entry>
<title>Ajax的使用</title>
<link href="/2020/06/19/Ajax%E7%9A%84%E4%BD%BF%E7%94%A8/"/>
<url>/2020/06/19/Ajax%E7%9A%84%E4%BD%BF%E7%94%A8/</url>
<content type="html"><![CDATA[<span id="more"></span><h2 id="ajax"><a href="#ajax" class="headerlink" title="ajax"></a>ajax</h2><p>ajax一个前后台配合的技术，它可以让javascript发送http请求，与后台通信，获取数据和信息。ajax技术的原理是实例化xmlhttp对象，使用此对象与后台通信。jquery将它封装成了一个函数$.ajax()，我们可以直接用这个函数来执行ajax请求。</p><p>ajax需要在服务器环境下运行。</p><p>$.ajax使用方法</p><p>常用参数：<br>1、url 请求地址<br>2、type 请求方式，默认是’GET’，常用的还有’POST’<br>3、dataType 设置返回的数据格式，常用的是’json’格式，也可以设置为’text’或者’html’<br>4、data 设置发送给服务器的数据<br>5、success 设置请求成功后的回调函数<br>6、error 设置请求失败后的回调函数<br>7、async 设置是否异步，默认值是’true’，表示异步</p><p>以前的写法：</p><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><code class="hljs javascript">$.<span class="hljs-title function_">ajax</span>({<br> <span class="hljs-attr">url</span>: <span class="hljs-string">'/change_data'</span>,<br> <span class="hljs-attr">type</span>: <span class="hljs-string">'GET'</span>,<br> <span class="hljs-attr">dataType</span>: <span class="hljs-string">'json'</span>,<br> <span class="hljs-attr">data</span>:{<span class="hljs-string">'code'</span>:<span class="hljs-number">10086</span>}<br> <span class="hljs-attr">success</span>:<span class="hljs-keyword">function</span>(<span class="hljs-params">dat</span>){<br> <span class="hljs-title function_">alert</span>(dat.<span class="hljs-property">name</span>);<br> },<br> <span class="hljs-attr">error</span>:<span class="hljs-keyword">function</span>(<span class="hljs-params"></span>){<br> <span class="hljs-title function_">alert</span>(<span class="hljs-string">'服务器超时，请重试！'</span>);<br> }<br>});<br></code></pre></td></tr></table></figure><p>新的写法(推荐)：</p><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><code class="hljs javascript">$.<span class="hljs-title function_">ajax</span>({<br> <span class="hljs-attr">url</span>: <span class="hljs-string">'/change_data'</span>,<br> <span class="hljs-attr">type</span>: <span class="hljs-string">'GET'</span>,<br> <span class="hljs-attr">dataType</span>: <span class="hljs-string">'json'</span>,<br> <span class="hljs-attr">data</span>:{<span class="hljs-string">'code'</span>:<span class="hljs-number">10086</span>}<br>})<br>.<span class="hljs-title function_">done</span>(<span class="hljs-keyword">function</span>(<span class="hljs-params">dat</span>) {<br> <span class="hljs-title function_">alert</span>(dat.<span class="hljs-property">name</span>);<br>})<br>.<span class="hljs-title function_">fail</span>(<span class="hljs-keyword">function</span>(<span class="hljs-params"></span>) {<br> <span class="hljs-title function_">alert</span>(<span class="hljs-string">'服务器超时，请重试！'</span>);<br>});<br></code></pre></td></tr></table></figure><p>$.ajax的简写方式</p><p>$.ajax按照请求方式可以简写成$.get或者$.post方式</p><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><code class="hljs javascript">$.<span class="hljs-title function_">get</span>(<span class="hljs-string">"/change_data"</span>, {<span class="hljs-string">'code'</span>:<span class="hljs-number">10086</span>},<br> <span class="hljs-keyword">function</span>(<span class="hljs-params">dat</span>){<br> <span class="hljs-title function_">alert</span>(dat.<span class="hljs-property">name</span>);<br>});<br><br>$.<span class="hljs-title function_">post</span>(<span class="hljs-string">"/change_data"</span>, {<span class="hljs-string">'code'</span>:<span class="hljs-number">10086</span>},<br> <span class="hljs-keyword">function</span>(<span class="hljs-params">dat</span>){<br> <span class="hljs-title function_">alert</span>(dat.<span class="hljs-property">name</span>);<br>});<br></code></pre></td></tr></table></figure><p> </p><h2 id="数据接口"><a href="#数据接口" class="headerlink" title="数据接口"></a><strong>数据接口</strong></h2><p>数据接口是后台程序提供的，它是一个url地址，访问这个地址，会对数据进行增、删、改、查的操作，最终会返回json格式的数据或者操作信息，格式也可以是text、xml等。</p><h2 id="同步和异步"><a href="#同步和异步" class="headerlink" title="同步和异步"></a><strong>同步和异步</strong></h2><p>现实生活中，同步指的是同时做几件事情，异步指的是做完一件事后再做另外一件事，程序中的同步和异步是把现实生活中的概念对调，也就是程序中的异步指的是现实生活中的同步，程序中的同步指的是现实生活中的异步。</p><h2 id="局部刷新和无刷新"><a href="#局部刷新和无刷新" class="headerlink" title="局部刷新和无刷新"></a><strong>局部刷新和无刷新</strong></h2><p>ajax可以实现局部刷新，也叫做无刷新，无刷新指的是整个页面不刷新，只是局部刷新，ajax可以自己发送http请求，不用通过浏览器的地址栏，所以页面整体不会刷新，ajax获取到后台数据，更新页面显示数据的部分，就做到了页面局部刷新。</p>]]></content>
<categories>
<category>前端</category>
</categories>
<tags>
<tag>ajax js</tag>
</tags>
</entry>
<entry>
<title>Django开发神奇的第三方包</title>
<link href="/2020/06/19/Django%E5%BC%80%E5%8F%91%E7%A5%9E%E5%A5%87%E7%9A%84%E7%AC%AC%E4%B8%89%E6%96%B9%E5%8C%85/"/>
<url>/2020/06/19/Django%E5%BC%80%E5%8F%91%E7%A5%9E%E5%A5%87%E7%9A%84%E7%AC%AC%E4%B8%89%E6%96%B9%E5%8C%85/</url>
<content type="html"><![CDATA[<span id="more"></span><p> </p><h2 id="1-Python-social-auth"><a href="#1-Python-social-auth" class="headerlink" title="1. Python social auth"></a>1. Python social auth</h2><p>一款社交账号认证/注册机制，支持Django、Flask、Webpy等在内的多个开发框架，提供了约50多个服务商的授权认证支持，如Google、Twitter、新浪微博等站点，配置简单。</p><p>GitHub 地址：<a href="https://link.zhihu.com/?target=https://github.com/pennersr/django-allauth">pennersr/django-allauth</a></p><p>文档地址：<a href="https://link.zhihu.com/?target=https://django-allauth.readthedocs.io/en/latest/">Welcome to django-allauth!</a></p><p>点评：增强 Django 内置的 django.contrib.auth 模块，提供登录、注册、邮件验证、找回密码等一切用户验证相关的功能。另外还提供 OAuth 第三方登录功能，例如国内的微博、微信登录，国外的 GitHub、Google、facebook 登录等，几乎囊括了大部分热门的第三方账户登录。配置简单，开箱即用。</p><figure class="highlight cmake"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs cmake">pip <span class="hljs-keyword">install</span> python-social-auth<br></code></pre></td></tr></table></figure><h2 id="2-Django-Guardian"><a href="#2-Django-Guardian" class="headerlink" title="2. Django Guardian"></a>2. Django Guardian</h2><p>Django默认没有提供对象（Object）级别的权限控制，我们可以通过该扩展来帮助Django实现对象级别的权限控制。</p><figure class="highlight cmake"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs cmake">pip <span class="hljs-keyword">install</span> django-guardian<br></code></pre></td></tr></table></figure><h2 id="3-Django-OAuth-Toolkit"><a href="#3-Django-OAuth-Toolkit" class="headerlink" title="3. Django OAuth Toolkit"></a>3. Django OAuth Toolkit</h2><p>可以帮助Django项目实现数据、逻辑的OAuth2功能，可与Django REST框架完美整合起来。</p><figure class="highlight cmake"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs cmake">pip <span class="hljs-keyword">install</span> django-oauth-toolkit<br></code></pre></td></tr></table></figure><h2 id="4-django-allauth"><a href="#4-django-allauth" class="headerlink" title="4. django-allauth"></a>4. django-allauth</h2><p>可用于账号注册、管理和第三方社交账号的认证。</p><p>django-allauth 是一个能够解决你的注册和认证需求的、可重用的 Django 应用。无论你需要构建本地注册系统还是社交账户注册系统，django-allauth 都能够帮你做到。</p><p>这个应用支持多种认证体系，比如用户名或电子邮件。一旦用户注册成功，它还可以提供从无需认证到电子邮件认证的多种账户验证的策略。同时，它也支持多种社交账户和电子邮件账户。它还支持插拔式注册表单，可让用户在注册时回答一些附加问题。</p><p>django-allauth 支持多于 20 种认证提供者，包括 Facebook、Google、微博 和 微信。如果你发现了一个它不支持的社交网站，很有可能通过第三方插件提供该网站的接入支持。这个项目还支持自定义后端，可以支持自定义的认证方式，对每个有定制认证需求的人来说这都很棒。</p><p>django-allauth 易于配置，且有完善的文档。该项目通过了很多测试，所以你可以相信它的所有部件都会正常运作。</p><figure class="highlight cmake"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs cmake">pip <span class="hljs-keyword">install</span> django-allauth<br></code></pre></td></tr></table></figure><h2 id="5-Celery"><a href="#5-Celery" class="headerlink" title="5. Celery"></a>5. Celery</h2><p>用来管理异步、分布式的消息作业队列，可用于生产系统来处理百万级别的任务。</p><p>django-celery是django web开发中执行异步任务或定时任务的最佳选择。它的应用场景包括:</p><ul><li>异步任务: 当用户触发一个动作需要较长时间来执行完成时，可以把它作为任务交给celery异步执行，执行完再返回给用户。这点和你在前端使用ajax实现异步加载有异曲同工之妙。</li><li>定时任务。假设有多台服务器，多个任务，定时任务的管理是很困难的，你要在不同电脑上写不同的crontab，而且还不好管理。Celery可以帮助我们快速在不同的机器设定不同任务。</li><li>其他可以异步执行的任务。比如发送短信，邮件，推送消息，清理/设置缓存等。这点还是比较有用的。</li></ul><figure class="highlight cmake"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs cmake">pip <span class="hljs-keyword">install</span> Celery<br></code></pre></td></tr></table></figure><h2 id="6-Django-REST-框架"><a href="#6-Django-REST-框架" class="headerlink" title="6. Django REST 框架"></a>6. Django REST 框架</h2><p>构建REST API的优秀框架，可管理内容协商、序列化、分页等，开发者可以在浏览器中浏览构建的API。</p><p>REST API 正在迅速成为现代 Web 应用的标准功能。 API 就是简单的使用 JSON 对话而不是 HTML，当然你可以只用 Django 做到这些。你可以制作自己的视图，设置合适的 Content-Type，然后返回 JSON 而不是渲染后的 HTML 响应。这是在像 Django Rest Framework（下称 DRF）这样的 API 框架发布之前，大多数人所做的。</p><p>如果你对 Django 的视图类很熟悉，你会觉得使用 DRF 构建 REST API 与使用它们很相似，不过 DRF 只针对特定 API 使用场景而设计。一般的 API 设置只需要一点代码，所以我们没有提供一份让你兴奋的示例代码，而是强调了一些可以让你生活的更舒适的 DRF 特性：</p><ul><li>可自动预览的 API 可以使你的开发和人工测试轻而易举。你可以查看 DRF 的示例代码。你可以查看 API 响应，并且不需要你做任何事就可以支持 POST/PUT/DELETE 类型的操作。</li><li>便于集成各种认证方式，如 OAuth, Basic Auth, 或API Tokens。</li><li>内建请求速率限制。</li><li>当与 django-rest-swagger 组合使用时，API 文档几乎可以自动生成。</li><li>广泛的第三方库生态。</li></ul><figure class="highlight cmake"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs cmake">pip <span class="hljs-keyword">install</span> djangorestframework<br></code></pre></td></tr></table></figure><h2 id="7-Django-stored-messages"><a href="#7-Django-stored-messages" class="headerlink" title="7. Django stored messages"></a>7. Django stored messages</h2><p>可以很好地集成在Django的消息框架中（django.contrib.messages）并让用户决定会话过程中存储在数据库中的消息。</p><h2 id="8-django-cors-headers"><a href="#8-django-cors-headers" class="headerlink" title="8. django-cors-headers"></a>8. django-cors-headers</h2><p>一款设置CORS（Cross-Origin Resource Sharing）标头的应用，基于XmlHttpRequest，对管理Django应用中的跨域请求非常有帮助。</p><figure class="highlight cmake"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs cmake">pip <span class="hljs-keyword">install</span> django-cors-headers<br></code></pre></td></tr></table></figure><h2 id="9-Debug-toolbar"><a href="#9-Debug-toolbar" class="headerlink" title="9. Debug toolbar"></a>9. Debug toolbar</h2><p>可在设置面板显示当前请求/响应的各种调试信息。除了本身提供的操作面板外，还有来自社区的多个第三方面板。</p><p>该工具给django web开发提供了强大的调试功能，包括查看执行的sql语句，db查询次数，request，headers，调试概览等。 通过安装插件Pympler，你还可以了解内存使用情况。</p><figure class="highlight mipsasm"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs mipsasm">pip <span class="hljs-keyword">install </span>django-<span class="hljs-built_in">debug</span>-toolbar<br></code></pre></td></tr></table></figure><p><strong>静态资源</strong></p><h2 id="10-Django-Storages"><a href="#10-Django-Storages" class="headerlink" title="10. Django Storages"></a>10. Django Storages</h2><p>可使静态资源方便地存储在外部服务上。安装后只需运行“python <a href="https://link.zhihu.com/?target=http://manage.py">manage.py</a> collectstatic”命令就可以将全部改动的静态文件复制到选定的后端。可结合库“python-boto”一起使用，将静态文件存储到Amazon S3上。</p><figure class="highlight cmake"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs cmake">pip <span class="hljs-keyword">install</span> django-storages<br></code></pre></td></tr></table></figure><h2 id="11-Django-Pipeline"><a href="#11-Django-Pipeline" class="headerlink" title="11. Django Pipeline"></a>11. Django Pipeline</h2><p>静态资源管理应用，支持连接和压缩CSS/Javascript文件、支持CSS和Javascript的多种编译器、内嵌JavaScript模板，可充分允许自定义。</p><figure class="highlight cmake"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs cmake">pip <span class="hljs-keyword">install</span> django-pipeline<br></code></pre></td></tr></table></figure><h2 id="12-Django-Compressor"><a href="#12-Django-Compressor" class="headerlink" title="12. Django Compressor"></a>12. Django Compressor</h2><p>可将页面中链接的以及直接编写的JavaScript和CSS打包到一个单一的缓存文件中，以减少页面对服务器的请求数，加快页面的加载速度。</p><figure class="highlight cmake"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs cmake">pip <span class="hljs-keyword">install</span> django_compressor<br></code></pre></td></tr></table></figure><h2 id="13-Reversion"><a href="#13-Reversion" class="headerlink" title="13. Reversion"></a>13. Reversion</h2><p>为模型提供版本控制功能，稍微配置后，就可以恢复已经删除的模型或回滚到模型历史中的任何一点。最新版本支持Django 1.6。</p><figure class="highlight cmake"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs cmake">pip <span class="hljs-keyword">install</span> django-reversion<br></code></pre></td></tr></table></figure><h2 id="14-Django-extensions"><a href="#14-Django-extensions" class="headerlink" title="14. Django extensions"></a>14. Django extensions</h2><p>Django框架的扩展功能集合，包括management命令扩展、数据库字段扩展、admin后台扩展等。</p><figure class="highlight cmake"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs cmake">pip <span class="hljs-keyword">install</span> django-extensions<br></code></pre></td></tr></table></figure><h2 id="15-Django-braces"><a href="#15-Django-braces" class="headerlink" title="15. Django braces"></a>15. Django braces</h2><p>是一系列可复用的行为、视图模型、表格和其他组件的合集。</p><figure class="highlight cmake"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs cmake">pip <span class="hljs-keyword">install</span> django-braces<br></code></pre></td></tr></table></figure><h2 id="16-django-haystack-全文检索引擎"><a href="#16-django-haystack-全文检索引擎" class="headerlink" title="16.django-haystack - 全文检索引擎"></a>16.django-haystack - 全文检索引擎</h2><p>全文检索不同于标题的简单匹配，是一件技术难度比较高的活。当文章很长时，你很难找到精确的匹配，同时搜索全文需要消耗大量的计算资源。有了haystack，你可以直接django中直接添加搜索功能，像搜索标题一样搜索全文，而无需关注索引建立、搜索解析等技术问题。haystack支持多种搜索引擎，不仅仅是whoosh，使用solr、elastic search等搜索，也可通过haystack，而且直接切换引擎即可，甚至无需修改搜索代码。</p><p>GitHub 地址：<a href="https://link.zhihu.com/?target=https://django-haystack.readthedocs.io/en/master/">Welcome to Haystack!</a></p><p>文档地址：<a href="https://link.zhihu.com/?target=https://github.com/django-haystack/django-haystack">django-haystack/django-haystack</a></p><h2 id="17-django-ckeditor-富文本编辑器"><a href="#17-django-ckeditor-富文本编辑器" class="headerlink" title="17.django-ckeditor - 富文本编辑器"></a>17.django-ckeditor - 富文本编辑器</h2><p>django没有提供官方的富文本编辑器，而ckeditor恰好是内容型网站后台管理中不可或缺的控件。ckeditor是一款基于javascript，使用非常广泛的开源网页编辑器。它允许用户直接编写图文，插入列表和表格，并支持文本和HTML格式代码输入。</p><p>GitHub 地址：<a href="https://link.zhihu.com/?target=https://github.com/django-ckeditor/django-ckeditor">django-ckeditor/django-ckeditor</a></p><h2 id="18-django-imagekit-自动化处理图像"><a href="#18-django-imagekit-自动化处理图像" class="headerlink" title="18.django-imagekit - 自动化处理图像"></a>18.django-imagekit - 自动化处理图像</h2><p>现代网站开发一般免不了处理一些图片，例如头像、用户上传的图片等内容。django-imagekit 帮你配合 django 的 model 模块自动完成图片的裁剪、压缩、生成缩略图、加水印等一系列图片相关的操作。</p><p>GitHub 地址：<a href="https://link.zhihu.com/?target=https://github.com/matthewwithanm/django-imagekit">matthewwithanm/django-imagekit</a></p><p>文档地址：<a href="https://link.zhihu.com/?target=http://django-imagekit.rtfd.org/">Installation - ImageKit 3.2.6 documentation</a></p><h2 id="19-django-xadmin-更美观更强大的后台"><a href="#19-django-xadmin-更美观更强大的后台" class="headerlink" title="19.django-xadmin - 更美观更强大的后台"></a>19.django-xadmin - 更美观更强大的后台</h2><p>如果你不喜欢django自带后台admin简陋的样式，你可以使用xadmin。xadmin是基于bootstrap和admin的一个更强大的后台管理系统。应该会给有强迫症的你带来惊喜。</p><p>GitHub 地址：<a href="https://link.zhihu.com/?target=https://github.com/sshwsfc/xadmin">sshwsfc/xadmin</a></p><p>文档地址：<a href="https://link.zhihu.com/?target=https://xadmin.readthedocs.io/en/docs-chinese/">Welcome to Django Xadmin’s Documentation!</a></p><h2 id="20-django-constance-常量管理"><a href="#20-django-constance-常量管理" class="headerlink" title="20.django-constance - 常量管理"></a>20.django-constance - 常量管理</h2><p>有时我们会在 django 的 settings 中设置一些常量，但是有可能会进行变更。利用这个包，只需简单的配置就可以自动生成 admin 管理后台可以修改管理常量。</p><p>Django 的好处就是大而全，不仅内置了 ORM、表单、模板引擎、用户系统等，而且第三方应用的生态也是十分完善，开发中大部分常见的功能都能找到对应的第三方实现。在这里给大家推荐 10 个十分优秀的 Django 第三方库（GitHub 星星数基本都在 1000 以上，而且都在持续维护与更新中）。虽然这些库很适合用于社交网站的开发，但也有很大一部分是通用的，可以用于任何用 Django 开发的项目。使用这些库将大大提高开发效率和生产力。</p><h2 id="21-django-model-utils"><a href="#21-django-model-utils" class="headerlink" title="21.django-model-utils"></a>21.django-model-utils</h2><p>简介：增强 Django 的 model 模块。内置了一些通用的 model Mixin，例如 TimeStampedModel 为模型提供一个创建时间和修改时间的字段，还有一些有用的 Field，几乎每个 Django 项目都能用得上。</p><p>GitHub 地址：<a href="https://link.zhihu.com/?target=https://github.com/jazzband/django-model-utils">jazzband/django-model-utils</a></p><p>文档地址：<a href="https://link.zhihu.com/?target=http://django-model-utils.readthedocs.io/en/latest/">django-model-utils - django-model-utils 3.2.0 documentation</a></p><h2 id="22-django-crispy-forms"><a href="#22-django-crispy-forms" class="headerlink" title="22.django-crispy-forms"></a>22.django-crispy-forms</h2><p>简介：大大增强 Django 内置的表单功能，Django 内置的表单生成原生的 HTML 表单代码还可以，但为其设置样式是一个麻烦的事情。django-crispy-forms 帮助你使用一行代码渲染一个 Bootstrap 样式的表单，当然它还支持其它一些热门的 CSS 框架样式的渲染。</p><p>GitHub 地址：<a href="https://link.zhihu.com/?target=https://github.com/django-crispy-forms/django-crispy-forms">django-crispy-forms/django-crispy-forms</a></p><p>文档地址：<a href="https://link.zhihu.com/?target=http://django-crispy-forms.rtfd.org/">Forms have never been this crispy</a></p><h2 id="23-django-mptt"><a href="#23-django-mptt" class="headerlink" title="23.django-mptt"></a>23.django-mptt</h2><p>简介：配合 Django 的 ORM 系统，为数据库的记录生成树形结构，并提供便捷的操作树型记录的 API。例如可以使用它实现一个多级的评论系统。总之，只要你的数据结构可能需要使用树来表示，django-mptt 将大大提高你的开发效率。</p><p>GitHub 地址：<a href="https://link.zhihu.com/?target=https://github.com/django-mptt/django-mptt">django-mptt/django-mptt</a></p><p>文档地址：<a href="https://link.zhihu.com/?target=https://django-mptt.readthedocs.io/">Django MPTT documentation</a></p><h2 id="24-django-contrib-comments"><a href="#24-django-contrib-comments" class="headerlink" title="24.django-contrib-comments"></a>24.django-contrib-comments</h2><p>简介：用于提供评论功能，最先集成在 django 的 contrib 内置库里，后来被移出来单独维护。这个评论库提供了基本的评论功能，但是只支持单级评论。好在这个库具有很好的拓展性，基于上边提到的 django-mptt，就可以构建一个支持层级评论的评论库。</p><p>GitHub 地址：<a href="https://link.zhihu.com/?target=https://github.com/django/django-contrib-comments">django/django-contrib-comments</a></p><p>文档地址：<a href="https://link.zhihu.com/?target=https://django-contrib-comments.readthedocs.io/">Django “excontrib” Comments</a></p><h2 id="25-django-brace"><a href="#25-django-brace" class="headerlink" title="25.django-brace"></a>25.django-brace</h2><p>简介：django 内置的 class based view 很 awesome，但还有一些通用的类视图没有包含在 django 源码中，这个库补充了更多常用的类视图。类视图是 django 的一个很重要也很优雅的特性，使用类视图可以减少视图函数的代码编写量、提高视图函数的代码复用性等。</p><p>GitHub 地址：<a href="https://link.zhihu.com/?target=https://github.com/brack3t/django-braces">brack3t/django-braces</a></p><p>文档地址：<a href="https://link.zhihu.com/?target=http://django-braces.readthedocs.io/en/latest/index.html">Welcome to django-braces’s documentation!</a></p><p>点评：深入学习类视图可以看Django类视图源码分析。</p><h2 id="26-django-notifications-hq"><a href="#26-django-notifications-hq" class="headerlink" title="26.django-notifications-hq"></a>26.django-notifications-hq</h2><p>简介：为你的网站提供类似于 GitHub 这样的通知功能。未读通知数、通知列表、标为已读等等。</p><p>GitHub 地址：<a href="https://link.zhihu.com/?target=https://github.com/django-notifications/django-notifications">django-notifications/django-notifications</a></p><p>文档地址：<a href="https://link.zhihu.com/?target=https://pypi.python.org/pypi/django-notifications-hq/">django-notifications-hq</a></p><h2 id="27-django-simple-captcha"><a href="#27-django-simple-captcha" class="headerlink" title="27.django-simple-captcha"></a>27.django-simple-captcha</h2><p>简介：配合 django 的表单模块，方便地为表单添加一个验证码字段。对验证性要求不高的需求，例如注册表单防止机器人自动注册等使用起来非常方便。</p><p>GitHub 地址：<a href="https://link.zhihu.com/?target=https://github.com/mbi/django-simple-captcha">mbi/django-simple-captcha</a></p><p>文档地址：<a href="https://link.zhihu.com/?target=http://django-simple-captcha.readthedocs.io/en/latest/">Django Simple Captcha</a></p><h2 id="28-django-anymail"><a href="#28-django-anymail" class="headerlink" title="28.django-anymail"></a>28.django-anymail</h2><p>简介：配合 django 的 email 模块，只需简单配置，就可以使用 Mailgun、SendGrid 等发送邮件。</p><p>GitHub 地址：<a href="https://link.zhihu.com/?target=https://github.com/anymail/django-anymail">anymail/django-anymail</a></p><p>文档地址：<a href="https://link.zhihu.com/?target=https://anymail.readthedocs.io/">Anymail: Django email integration for transactional ESPs</a></p><h2 id="29-django-activity-stream"><a href="#29-django-activity-stream" class="headerlink" title="29.django-activity-stream"></a>29.django-activity-stream</h2><p>简介：社交类网站免不了关注、收藏、点赞、用户动态等功能，这一个 app 全搞定。甚至用它实现一个朋友圈也不是不可能。</p><p>GitHub 地址：<a href="https://link.zhihu.com/?target=https://github.com/justquick/django-activity-stream">justquick/django-activity-stream</a></p><p>文档地址：<a href="https://link.zhihu.com/?target=http://django-activity-stream.rtfd.io/en/latest/">Django Activity Stream Documentation</a></p><h2 id="30-Datatables"><a href="#30-Datatables" class="headerlink" title="30.Datatables"></a>30.Datatables</h2><p>是一款jquery表格插件。它是一个高度灵活的工具，可以将任何HTML表格添加高级的交互功能。</p><p>官网：<a href="https://link.zhihu.com/?target=https://datatables.net/">Table plug-in for jQuery</a> 中文网站：<a href="https://link.zhihu.com/?target=http://datatables.club/">Datatables 中文网</a></p><p> </p><p>链接：<a href="https://zhuanlan.zhihu.com/p/76255269">https://zhuanlan.zhihu.com/p/76255269</a><br>来源：知乎</p>]]></content>
<categories>
<category>Django</category>
</categories>
<tags>
<tag>python</tag>
</tags>
</entry>
<entry>
<title>Git 常用命令速查表</title>
<link href="/2020/06/17/Git%20%E5%B8%B8%E7%94%A8%E5%91%BD%E4%BB%A4%E9%80%9F%E6%9F%A5%E8%A1%A8/"/>
<url>/2020/06/17/Git%20%E5%B8%B8%E7%94%A8%E5%91%BD%E4%BB%A4%E9%80%9F%E6%9F%A5%E8%A1%A8/</url>
<content type="html"><![CDATA[<span id="more"></span><h2 id="一、-Git-常用命令速查"><a href="#一、-Git-常用命令速查" class="headerlink" title="一、 Git 常用命令速查"></a>一、 Git 常用命令速查</h2><p><img src="https://img-blog.csdnimg.cn/20200617151856288.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MjQ3NDU0MA==,size_16,color_FFFFFF,t_70"></p><ul><li>git branch 查看本地所有分支</li><li>git status 查看当前状态</li><li>git commit 提交</li><li>git branch -a 查看所有的分支</li><li>git branch -r 查看远程所有分支</li><li>git commit -am “init” 提交并且加注释</li><li>git remote add origin [email protected]:ndshow</li><li>git push origin master 将文件给推到服务器上</li><li>git remote show origin 显示远程库origin里的资源</li><li>git push origin master:develop</li><li>git push origin master:hb-dev 将本地库与服务器上的库进行关联</li><li>git checkout –track origin/dev 切换到远程dev分支</li><li>git branch -D master develop 删除本地库develop</li><li>git checkout -b dev 建立一个新的本地分支dev</li><li>git merge origin/dev 将分支dev与当前分支进行合并</li><li>git checkout dev 切换到本地dev分支</li><li>git remote show 查看远程库</li><li>git add .</li><li>git rm 文件名(包括路径) 从git中删除指定文件</li><li>git clone git://github.com/schacon/grit.git 从服务器上将代码给拉下来</li><li>git config –list 看所有用户</li><li>git ls-files 看已经被提交的</li><li>git rm [file name] 删除一个文件</li><li>git commit -a 提交当前repos的所有的改变</li><li>git add [file name] 添加一个文件到git index</li><li>git commit -v 当你用－v参数的时候可以看commit的差异</li><li>git commit -m “This is the message describing the commit” 添加commit信息</li><li>git commit -a -a是代表add，把所有的change加到git index里然后再commit</li><li>git commit -a -v 一般提交命令</li><li>git log 看你commit的日志</li><li>git diff 查看尚未暂存的更新</li><li>git rm a.a 移除文件(从暂存区和工作区中删除)</li><li>git rm –cached a.a 移除文件(只从暂存区中删除)</li><li>git commit -m “remove” 移除文件(从Git中删除)</li><li>git rm -f a.a 强行移除修改后文件(从暂存区和工作区中删除)</li><li>git diff –cached 或 $ git diff –staged 查看尚未提交的更新</li><li>git stash push 将文件给push到一个临时空间中</li><li>git stash pop 将文件从临时空间pop下来</li></ul><hr><ul><li>git remote add origin [email protected]:username/Hello-World.git</li><li>git push origin master 将本地项目给提交到服务器中</li></ul><hr><ul><li>git pull 本地与服务器端同步</li></ul><hr><ul><li>git push (远程仓库名) (分支名) 将本地分支推送到服务器上去。</li><li>git push origin serverfix:awesomebranch</li></ul><hr><ul><li>git fetch 相当于是从远程获取最新版本到本地，不会自动merge</li><li>git commit -a -m “log_message” (-a是提交所有改动，-m是加入log信息)</li></ul><hr><p>初始化版本库，并提交到远程服务器端</p><p>mkdir WebApp<br>cd WebApp<br>git init 本地初始化<br>touch README<br>git add README 添加文件<br>git commit -m ‘first commit’<br>git remote add origin [email protected]:daixu/WebApp.git</p><p>上面的命令会增加URL地址为’[email protected]:daixu/WebApp.git’，名称为origin的远程服务器库，以后提交代码的时候只需要使用 origin别名即可</p><h2 id="二、-Git-命令速查表"><a href="#二、-Git-命令速查表" class="headerlink" title="二、 Git 命令速查表"></a>二、 Git 命令速查表</h2><h3 id="1、常用的Git命令"><a href="#1、常用的Git命令" class="headerlink" title="1、常用的Git命令"></a>1、常用的Git命令</h3><table><thead><tr><th>命令</th><th>简要说明</th></tr></thead><tbody><tr><td>git add</td><td>添加至暂存区</td></tr><tr><td>git add–interactive</td><td>交互式添加</td></tr><tr><td>git apply</td><td>应用补丁</td></tr><tr><td>git am</td><td>应用邮件格式补丁</td></tr><tr><td>git annotate</td><td>同义词，等同于 git blame</td></tr><tr><td>git archive</td><td>文件归档打包</td></tr><tr><td>git bisect</td><td>二分查找</td></tr><tr><td>git blame</td><td>文件逐行追溯</td></tr><tr><td>git branch</td><td>分支管理</td></tr><tr><td>git cat-file</td><td>版本库对象研究工具</td></tr><tr><td>git checkout</td><td>检出到工作区、切换或创建分支</td></tr><tr><td>git cherry-pick</td><td>提交拣选</td></tr><tr><td>git citool</td><td>图形化提交，相当于 git gui 命令</td></tr><tr><td>git clean</td><td>清除工作区未跟踪文件</td></tr><tr><td>git clone</td><td>克隆版本库</td></tr><tr><td>git commit</td><td>提交</td></tr><tr><td>git config</td><td>查询和修改配置</td></tr><tr><td>git describe</td><td>通过里程碑直观地显示提交ID</td></tr><tr><td>git diff</td><td>差异比较</td></tr><tr><td>git difftool</td><td>调用图形化差异比较工具</td></tr><tr><td>git fetch</td><td>获取远程版本库的提交</td></tr><tr><td>git format-patch</td><td>创建邮件格式的补丁文件。参见 git am 命令</td></tr><tr><td>git grep</td><td>文件内容搜索定位工具</td></tr><tr><td>git gui</td><td>基于Tcl/Tk的图形化工具，侧重提交等操作</td></tr><tr><td>git help</td><td>帮助</td></tr><tr><td>git init</td><td>版本库初始化</td></tr><tr><td>git init-db*</td><td>同义词，等同于 git init</td></tr><tr><td>git log</td><td>显示提交日志</td></tr><tr><td>git merge</td><td>分支合并</td></tr><tr><td>git mergetool</td><td>图形化冲突解决</td></tr><tr><td>git mv</td><td>重命名</td></tr><tr><td>git pull</td><td>拉回远程版本库的提交</td></tr><tr><td>git push</td><td>推送至远程版本库</td></tr><tr><td>git rebase</td><td>分支变基</td></tr><tr><td>git rebase–interactive</td><td>交互式分支变基</td></tr><tr><td>git reflog</td><td>分支等引用变更记录管理</td></tr><tr><td>git remote</td><td>远程版本库管理</td></tr><tr><td>git repo-config*</td><td>同义词，等同于 git config</td></tr><tr><td>git reset</td><td>重置改变分支“游标”指向</td></tr><tr><td>git rev-parse</td><td>将各种引用表示法转换为哈希值等</td></tr><tr><td>git revert</td><td>反转提交</td></tr><tr><td>git rm</td><td>删除文件</td></tr><tr><td>git show</td><td>显示各种类型的对象</td></tr><tr><td>git stage*</td><td>同义词，等同于 git add</td></tr><tr><td>git stash</td><td>保存和恢复进度</td></tr><tr><td>git status</td><td>显示工作区文件状态</td></tr><tr><td>git tag</td><td>里程碑管理</td></tr></tbody></table><h3 id="2、对象库操作相关命令"><a href="#2、对象库操作相关命令" class="headerlink" title="2、对象库操作相关命令"></a>2、对象库操作相关命令</h3><table><thead><tr><th>命令</th><th>简要说明</th></tr></thead><tbody><tr><td>git commit-tree</td><td>从树对象创建提交</td></tr><tr><td>git hash-object</td><td>从标准输入或文件计算哈希值或创建对象</td></tr><tr><td>git ls-files</td><td>显示工作区和暂存区文件</td></tr><tr><td>git ls-tree</td><td>显示树对象包含的文件</td></tr><tr><td>git mktag</td><td>读取标准输入创建一个里程碑对象</td></tr><tr><td>git mktree</td><td>读取标准输入创建一个树对象</td></tr><tr><td>git read-tree</td><td>读取树对象到暂存区</td></tr><tr><td>git update-index</td><td>工作区内容注册到暂存区及暂存区管理</td></tr><tr><td>git unpack-file</td><td>创建临时文件包含指定 blob 的内容</td></tr><tr><td>git write-tree</td><td>从暂存区创建一个树对象</td></tr></tbody></table><h3 id="3、引用操作相关命令"><a href="#3、引用操作相关命令" class="headerlink" title="3、引用操作相关命令"></a>3、引用操作相关命令</h3><table><thead><tr><th>命令</th><th>简要说明</th></tr></thead><tbody><tr><td>git check-ref-format</td><td>检查引用名称是否符合规范</td></tr><tr><td>git for-each-ref</td><td>引用迭代器，用于shell编程</td></tr><tr><td>git ls-remote</td><td>显示远程版本库的引用</td></tr><tr><td>git name-rev</td><td>将提交ID显示为友好名称</td></tr><tr><td>git peek-remote*</td><td>过时命令，请使用 git ls-remote</td></tr><tr><td>git rev-list</td><td>显示版本范围</td></tr><tr><td>git show-branch</td><td>显示分支列表及拓扑关系</td></tr><tr><td>git show-ref</td><td>显示本地引用</td></tr><tr><td>git symbolic-ref</td><td>显示或者设置符号引用</td></tr><tr><td>git update-ref</td><td>更新引用的指向</td></tr><tr><td>git verify-tag</td><td>校验 GPG 签名的Tag</td></tr></tbody></table><h3 id="4、版本库管理相关命令"><a href="#4、版本库管理相关命令" class="headerlink" title="4、版本库管理相关命令"></a>4、版本库管理相关命令</h3><table><thead><tr><th>命令</th><th>简要说明</th></tr></thead><tbody><tr><td>git count-objects</td><td>显示松散对象的数量和磁盘占用</td></tr><tr><td>git filter-branch</td><td>版本库重构</td></tr><tr><td>git fsck</td><td>对象库完整性检查</td></tr><tr><td>git fsck-objects*</td><td>同义词，等同于 git fsck</td></tr><tr><td>git gc</td><td>版本库存储优化</td></tr><tr><td>git index-pack</td><td>从打包文件创建对应的索引文件</td></tr><tr><td>git lost-found*</td><td>过时，请使用 git fsck –lost-found 命令</td></tr><tr><td>git pack-objects</td><td>从标准输入读入对象ID，打包到文件</td></tr><tr><td>git pack-redundant</td><td>查找多余的 pack 文件</td></tr><tr><td>git pack-refs</td><td>将引用打包到 .git/packed-refs 文件中</td></tr><tr><td>git prune</td><td>从对象库删除过期对象</td></tr><tr><td>git prune-packed</td><td>将已经打包的松散对象删除</td></tr><tr><td>git relink</td><td>为本地版本库中相同的对象建立硬连接</td></tr><tr><td>git repack</td><td>将版本库未打包的松散对象打包</td></tr><tr><td>git show-index</td><td>读取包的索引文件，显示打包文件中的内容</td></tr><tr><td>git unpack-objects</td><td>从打包文件释放文件</td></tr><tr><td>git verify-pack</td><td>校验对象库打包文件</td></tr></tbody></table><h3 id="5、数据传输相关命令"><a href="#5、数据传输相关命令" class="headerlink" title="5、数据传输相关命令"></a>5、数据传输相关命令</h3><table><thead><tr><th>命令</th><th>简要说明</th></tr></thead><tbody><tr><td>git fetch-pack</td><td>执行 git fetch 或 git pull</td></tr><tr><td>git receive-pack</td><td>执行 git push 命令时在远程执行的命令，用于接受推送的数据</td></tr><tr><td>git send-pack</td><td>执行 git push 命令时在本地执行的命令，用于向其他版本库推送数据</td></tr><tr><td>git upload-archive</td><td>执行 git archive –remote</td></tr><tr><td>git upload-pack</td><td>执行 git fetch 或 git pull 命令时在远程执行此命令，将对象打包、上传</td></tr></tbody></table><h3 id="6、邮件相关命令"><a href="#6、邮件相关命令" class="headerlink" title="6、邮件相关命令"></a>6、邮件相关命令</h3><table><thead><tr><th>命令</th><th>简要说明</th></tr></thead><tbody><tr><td>git imap-send</td><td>将补丁通过 IMAP 发送</td></tr><tr><td>git mailinfo</td><td>从邮件导出提交说明和补丁</td></tr><tr><td>git mailsplit</td><td>将 mbox 或 Maildir 格式邮箱中邮件逐一提取为文件</td></tr><tr><td>git request-pull</td><td>创建包含提交间差异和执行PULL操作地址的信息</td></tr><tr><td>git send-email</td><td>发送邮件</td></tr></tbody></table><h3 id="7、协议相关命令"><a href="#7、协议相关命令" class="headerlink" title="7、协议相关命令"></a>7、协议相关命令</h3><table><thead><tr><th>命令</th><th>简要说明</th></tr></thead><tbody><tr><td>git daemon</td><td>实现Git协议</td></tr><tr><td>git http-backend</td><td>实现HTTP协议的CGI程序，支持智能HTTP协议</td></tr><tr><td>git instaweb</td><td>即时启动浏览器通过 gitweb 浏览当前版本库</td></tr><tr><td>git shell</td><td>受限制的shell，提供仅执行Git命令的SSH访问</td></tr><tr><td>git update-server-info</td><td>更新哑协议需要的辅助文件</td></tr><tr><td>git http-fetch</td><td>通过HTTP协议获取版本库</td></tr><tr><td>git http-push</td><td>通过HTTP/DAV协议推送</td></tr><tr><td>git remote-ext</td><td>由Git命令调用，通过外部命令提供扩展协议支持</td></tr><tr><td>git remote-fd</td><td>由Git命令调用，使用文件描述符作为协议接口</td></tr><tr><td>git remote-ftp</td><td>由Git命令调用，提供对FTP协议的支持</td></tr><tr><td>git remote-ftps</td><td>由Git命令调用，提供对FTPS协议的支持</td></tr><tr><td>git remote-http</td><td>由Git命令调用，提供对HTTP协议的支持</td></tr><tr><td>git remote-https</td><td>由Git命令调用，提供对HTTPS协议的支持</td></tr><tr><td>git remote-testgit</td><td>协议扩展示例脚本</td></tr></tbody></table><h3 id="8、版本库转换和交互相关命令"><a href="#8、版本库转换和交互相关命令" class="headerlink" title="8、版本库转换和交互相关命令"></a>8、版本库转换和交互相关命令</h3><table><thead><tr><th>命令</th><th>简要说明</th></tr></thead><tbody><tr><td>git archimport</td><td>导入Arch版本库到Git</td></tr><tr><td>git bundle</td><td>提交打包和解包，以便在不同版本库间传递</td></tr><tr><td>git cvsexportcommit</td><td>将Git的一个提交作为一个CVS检出</td></tr><tr><td>git cvsimport</td><td>导入CVS版本库到Git。或者使用 cvs2git</td></tr><tr><td>git cvsserver</td><td>Git的CVS协议模拟器，可供CVS命令访问Git版本库</td></tr><tr><td>git fast-export</td><td>将提交导出为 git-fast-import 格式</td></tr><tr><td>git fast-import</td><td>其他版本库迁移至Git的通用工具</td></tr><tr><td>git svn</td><td>Git 作为前端操作 Subversion</td></tr></tbody></table><h3 id="9、合并相关的辅助命令"><a href="#9、合并相关的辅助命令" class="headerlink" title="9、合并相关的辅助命令"></a>9、合并相关的辅助命令</h3><table><thead><tr><th>命令</th><th>简要说明</th></tr></thead><tbody><tr><td>git merge-base</td><td>供其他脚本调用，找到两个或多个提交最近的共同祖先</td></tr><tr><td>git merge-file</td><td>针对文件的两个不同版本执行三向文件合并</td></tr><tr><td>git merge-index</td><td>对index中的冲突文件调用指定的冲突解决工具</td></tr><tr><td>git merge-octopus</td><td>合并两个以上分支。参见 git merge 的octopus合并策略</td></tr><tr><td>git merge-one-file</td><td>由 git merge-index 调用的标准辅助程序</td></tr><tr><td>git merge-ours</td><td>合并使用本地版本，抛弃他人版本。参见 git merge 的ours合并策略</td></tr><tr><td>git merge-recursive</td><td>针对两个分支的三向合并。参见 git merge 的recursive合并策略</td></tr><tr><td>git merge-resolve</td><td>针对两个分支的三向合并。参见 git merge 的resolve合并策略</td></tr><tr><td>git merge-subtree</td><td>子树合并。参见 git merge 的 subtree 合并策略</td></tr><tr><td>git merge-tree</td><td>显式三向合并结果，不改变暂存区</td></tr><tr><td>git fmt-merge-msg</td><td>供执行合并操作的脚本调用，用于创建一个合并提交说明</td></tr><tr><td>git rerere</td><td>重用所记录的冲突解决方案</td></tr></tbody></table><h3 id="10、-杂项"><a href="#10、-杂项" class="headerlink" title="10、 杂项"></a>10、 杂项</h3><table><thead><tr><th>命令</th><th>简要说明</th></tr></thead><tbody><tr><td>git bisect–helper</td><td>由 git bisect 命令调用，确认二分查找进度</td></tr><tr><td>git check-attr</td><td>显示某个文件是否设置了某个属性</td></tr><tr><td>git checkout-index</td><td>从暂存区拷贝文件至工作区</td></tr><tr><td>git cherry</td><td>查找没有合并到上游的提交</td></tr><tr><td>git diff-files</td><td>比较暂存区和工作区，相当于 git diff –raw</td></tr><tr><td>git diff-index</td><td>比较暂存区和版本库，相当于 git diff –cached –raw</td></tr><tr><td>git diff-tree</td><td>比较两个树对象，相当于 git diff –raw A B</td></tr><tr><td>git difftool–helper</td><td>由 git difftool 命令调用，默认要使用的差异比较工具</td></tr><tr><td>git get-tar-commit-id</td><td>从 git archive 创建的 tar 包中提取提交ID</td></tr><tr><td>git gui–askpass</td><td>命令 git gui 的获取用户口令输入界面</td></tr><tr><td>git notes</td><td>提交评论管理</td></tr><tr><td>git patch-id</td><td>补丁过滤行号和空白字符后生成补丁唯一ID</td></tr><tr><td>git quiltimport</td><td>将Quilt补丁列表应用到当前分支</td></tr><tr><td>git replace</td><td>提交替换</td></tr><tr><td>git shortlog</td><td>对 git log 的汇总输出，适合于产品发布说明</td></tr><tr><td>git stripspace</td><td>删除空行，供其他脚本调用</td></tr><tr><td>git submodule</td><td>子模组管理</td></tr><tr><td>git tar-tree</td><td>过时命令，请使用 git archive</td></tr><tr><td>git var</td><td>显示 Git 环境变量</td></tr><tr><td>git web–browse</td><td>启动浏览器以查看目录或文件</td></tr><tr><td>git whatchanged</td><td>显示提交历史及每次提交的改动</td></tr><tr><td>git-mergetool–lib</td><td>包含于其他脚本中，提供合并/差异比较工具的选择和执行</td></tr><tr><td>git-parse-remote</td><td>包含于其他脚本中，提供操作远程版本库的函数</td></tr><tr><td>git-sh-setup</td><td>包含于其他脚本中，提供 shell 编程的函数库</td></tr></tbody></table><h2 id="三、一些简单的使用例子"><a href="#三、一些简单的使用例子" class="headerlink" title="三、一些简单的使用例子"></a>三、一些简单的使用例子</h2><p>git init # 初始化本地git仓库（创建新仓库）<br>git config –global user.name “xxx” # 配置用户名<br>git config –global user.email “[email protected]” # 配置邮件<br>git config –global color.ui true # git status等命令自动着色<br>git config –global color.status auto<br>git config –global color.diff auto<br>git config –global color.branch auto<br>git config –global color.interactive auto<br>git clone git+ssh://[email protected]/VT.git # clone远程仓库<br>git status # 查看当前版本状态（是否修改）<br>git add xyz # 添加xyz文件至index<br>git add . # 增加当前子目录下所有更改过的文件至index<br>git commit -m ‘xxx’ # 提交<br>git commit –amend -m ‘xxx’ # 合并上一次提交（用于反复修改）<br>git commit -am ‘xxx’ # 将add和commit合为一步<br>git rm xxx # 删除index中的文件<br>git rm -r * # 递归删除<br>git log # 显示提交日志<br>git log -1 # 显示1行日志 -n为n行<br>git log -5<br>git log –stat # 显示提交日志及相关变动文件<br>git log -p -m<br>git show dfb02e6e4f2f7b573337763e5c0013802e392818 # 显示某个提交的详细内容<br>git show dfb02 # 可只用commitid的前几位<br>git show HEAD # 显示HEAD提交日志<br>git show HEAD^ # 显示HEAD的父（上一个版本）的提交日志 ^^为上两个版本 ^5为上5个版本<br>git tag # 显示已存在的tag<br>git tag -a v2.0 -m ‘xxx’ # 增加v2.0的tag<br>git show v2.0 # 显示v2.0的日志及详细内容<br>git log v2.0 # 显示v2.0的日志<br>git diff # 显示所有未添加至index的变更<br>git diff –cached # 显示所有已添加index但还未commit的变更<br>git diff HEAD^ # 比较与上一个版本的差异<br>git diff HEAD – ./lib # 比较与HEAD版本lib目录的差异<br>git diff origin/master..master # 比较远程分支master上有本地分支master上没有的<br>git diff origin/master..master –stat # 只显示差异的文件，不显示具体内容<br>git remote add origin git+ssh://[email protected]/VT.git # 增加远程定义（用于push/pull/fetch）<br>git branch # 显示本地分支<br>git branch –contains 50089 # 显示包含提交50089的分支<br>git branch -a # 显示所有分支<br>git branch -r # 显示所有原创分支<br>git branch –merged # 显示所有已合并到当前分支的分支<br>git branch –no-merged # 显示所有未合并到当前分支的分支<br>git branch -m master master_copy # 本地分支改名<br>git checkout -b master_copy # 从当前分支创建新分支master_copy并检出<br>git checkout -b master master_copy # 上面的完整版<br>git checkout features/performance # 检出已存在的features/performance分支<br>git checkout –track hotfixes/BJVEP933 # 检出远程分支hotfixes/BJVEP933并创建本地跟踪分支<br>git checkout v2.0 # 检出版本v2.0<br>git checkout -b devel origin/develop # 从远程分支develop创建新本地分支devel并检出<br>git checkout – README # 检出head版本的README文件（可用于修改错误回退）<br>git merge origin/master # 合并远程master分支至当前分支<br>git cherry-pick ff44785404a8e # 合并提交ff44785404a8e的修改<br>git push origin master # 将当前分支push到远程master分支<br>git push origin :hotfixes/BJVEP933 # 删除远程仓库的hotfixes/BJVEP933分支<br>git push –tags # 把所有tag推送到远程仓库<br>git fetch # 获取所有远程分支（不更新本地分支，另需merge）<br>git fetch –prune # 获取所有原创分支并清除服务器上已删掉的分支<br>git pull origin master # 获取远程分支master并merge到当前分支<br>git mv README README2 # 重命名文件README为README2<br>git reset –hard HEAD # 将当前版本重置为HEAD（通常用于merge失败回退）<br>git rebase<br>git branch -d hotfixes/BJVEP933 # 删除分支hotfixes/BJVEP933（本分支修改已合并到其他分支）<br>git branch -D hotfixes/BJVEP933 # 强制删除分支hotfixes/BJVEP933<br>git ls-files # 列出git index包含的文件<br>git show-branch # 图示当前分支历史<br>git show-branch –all # 图示所有分支历史<br>git whatchanged # 显示提交历史对应的文件修改<br>git revert dfb02e6e4f2f7b573337763e5c0013802e392818 # 撤销提交dfb02e6e4f2f7b573337763e5c0013802e392818<br>git ls-tree HEAD # 内部命令：显示某个git对象<br>git rev-parse v2.0 # 内部命令：显示某个ref对于的SHA1 HASH<br>git reflog # 显示所有提交，包括孤立节点<br>git show HEAD@{5}<br>git show master@{yesterday} # 显示master分支昨天的状态<br>git log –pretty=format:’%h %s’ –graph # 图示提交日志<br>git show HEAD~3<br>git show -s –pretty=raw 2be7fcb476<br>git stash # 暂存当前修改，将所有至为HEAD状态<br>git stash list # 查看所有暂存<br>git stash show -p stash@{0} # 参考第一次暂存<br>git stash apply stash@{0} # 应用第一次暂存<br>git grep “delete from” # 文件中搜索文本“delete from”<br>git grep -e ‘#define’ –and -e SORT_DIRENT<br>git gc<br>git fsck</p><p>更多….</p><p>Git常用操作命令：<br>1) 远程仓库相关命令<br>检出仓库：$ git clone git://github.com/jquery/jquery.git<br>查看远程仓库：$ git remote -v<br>添加远程仓库：$ git remote add [name] [url]<br>删除远程仓库：$ git remote rm [name]<br>修改远程仓库：$ git remote set-url –push [name] [newUrl]<br>拉取远程仓库：$ git pull [remoteName] [localBranchName]<br>推送远程仓库：$ git push [remoteName] [localBranchName]</p><p>*如果想把本地的某个分支test提交到远程仓库，并作为远程仓库的master分支，或者作为另外一个名叫test的分支，如下：<br>$git push origin test:master // 提交本地test分支作为远程的master分支<br>$git push origin test:test // 提交本地test分支作为远程的test分支</p><p>2）分支(branch)操作相关命令<br>查看本地分支：$ git branch<br>查看远程分支：$ git branch -r<br>创建本地分支：$ git branch [name] —-注意新分支创建后不会自动切换为当前分支<br>切换分支：$ git checkout [name]<br>创建新分支并立即切换到新分支：$ git checkout -b [name]<br>删除分支：$ git branch -d [name] —- -d选项只能删除已经参与了合并的分支，对于未有合并的分支是无法删除的。如果想强制删除一个分支，可以使用-D选项<br>合并分支：$ git merge [name] —-将名称为[name]的分支与当前分支合并<br>创建远程分支(本地分支push到远程)：$ git push origin [name]<br>删除远程分支：$ git push origin :heads/[name] 或 $ gitpush origin :[name] </p><p>*创建空的分支：(执行命令之前记得先提交你当前分支的修改，否则会被强制删干净没得后悔)<br>$git symbolic-ref HEAD refs/heads/[name]<br>$rm .git/index<br>$git clean -fdx</p><p>3）版本(tag)操作相关命令<br>查看版本：$ git tag<br>创建版本：$ git tag [name]<br>删除版本：$ git tag -d [name]<br>查看远程版本：$ git tag -r<br>创建远程版本(本地版本push到远程)：$ git push origin [name]<br>删除远程版本：$ git push origin :refs/tags/[name]<br>合并远程仓库的tag到本地：$ git pull origin –tags<br>上传本地tag到远程仓库：$ git push origin –tags<br>创建带注释的tag：$ git tag -a [name] -m ‘yourMessage’</p><p>4) 子模块(submodule)相关操作命令<br>添加子模块：$ git submodule add [url] [path]<br> 如：$git submodule add git://github.com/soberh/ui-libs.git src/main/webapp/ui-libs<br>初始化子模块：$ git submodule init —-只在首次检出仓库时运行一次就行<br>更新子模块：$ git submodule update —-每次更新或切换分支后都需要运行一下<br>删除子模块：（分4步走哦）<br> 1) $ git rm –cached [path]<br> 2) 编辑“.gitmodules”文件，将子模块的相关配置节点删除掉<br> 3) 编辑“ .git/config”文件，将子模块的相关配置节点删除掉<br> 4) 手动删除子模块残留的目录</p><p>5）忽略一些文件、文件夹不提交<br>在仓库根目录下创建名称为“.gitignore”的文件，写入不需要的文件夹名或文件，每个元素占一行即可，如<br>target<br>bin<br>*.db</p>]]></content>
<categories>
<category>Git</category>
</categories>
<tags>
<tag>github linux git</tag>
</tags>
</entry>
<entry>
<title>websocket 与 socket 非阻塞通信</title>
<link href="/2020/06/12/websocket%20%E4%B8%8E%20socket%20%E9%9D%9E%E9%98%BB%E5%A1%9E%E9%80%9A%E4%BF%A1/"/>
<url>/2020/06/12/websocket%20%E4%B8%8E%20socket%20%E9%9D%9E%E9%98%BB%E5%A1%9E%E9%80%9A%E4%BF%A1/</url>
<content type="html"><![CDATA[<span id="more"></span><p> 记录一下</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br><span class="line">97</span><br><span class="line">98</span><br><span class="line">99</span><br><span class="line">100</span><br><span class="line">101</span><br><span class="line">102</span><br><span class="line">103</span><br><span class="line">104</span><br><span class="line">105</span><br><span class="line">106</span><br><span class="line">107</span><br><span class="line">108</span><br><span class="line">109</span><br><span class="line">110</span><br><span class="line">111</span><br><span class="line">112</span><br><span class="line">113</span><br><span class="line">114</span><br><span class="line">115</span><br><span class="line">116</span><br><span class="line">117</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-keyword">import</span> select<br><span class="hljs-keyword">import</span> socket<br><span class="hljs-keyword">import</span> threading<br><br><span class="hljs-keyword">from</span> flask <span class="hljs-keyword">import</span> Flask<br><span class="hljs-keyword">from</span> flask_sockets <span class="hljs-keyword">import</span> Sockets<br><span class="hljs-keyword">from</span> gevent <span class="hljs-keyword">import</span> pywsgi<br><span class="hljs-keyword">from</span> geventwebsocket.handler <span class="hljs-keyword">import</span> WebSocketHandler<br><br><br>app = Flask(__name__)<br>sockets = Sockets(app)<br><br><br><span class="hljs-keyword">class</span> <span class="hljs-title class_">Config</span>(<span class="hljs-title class_ inherited__">object</span>):<br> SOCKET_HOST = <span class="hljs-string">'127.0.0.1'</span><br> SOCKET_PORT = <span class="hljs-number">2020</span><br><br><br><span class="hljs-keyword">def</span> <span class="hljs-title function_">read_thread_method</span>(<span class="hljs-params">sock,socket_lock,ws</span>):<br> <span class="hljs-keyword">while</span> <span class="hljs-literal">True</span>:<br> <span class="hljs-keyword">if</span> <span class="hljs-keyword">not</span> sock: <span class="hljs-comment"># 如果socket关闭，退出</span><br> <span class="hljs-keyword">break</span><br> <span class="hljs-comment"># 使用select监听客户端（这里客户端需要不停接收服务端的数据，所以监听客户端）</span><br> <span class="hljs-comment"># 第一个参数是要监听读事件列表，因为是客户端，我们只监听创建的一个socket就ok</span><br> <span class="hljs-comment"># 第二个参数是要监听写事件列表，</span><br> <span class="hljs-comment"># 第三个参数是要监听异常事件列表，</span><br> <span class="hljs-comment"># 最后一个参数是监听超时时间，默认永不超时。如果设置了超时时间，过了超时时间线程就不会阻塞在select方法上，会继续向下执行</span><br> <span class="hljs-comment"># 返回参数 分别对应监听到的读事件列表，写事件列表，异常事件列表</span><br> rs, _, _ = select.select([sock], [], [], <span class="hljs-number">10</span>)<br> <span class="hljs-keyword">for</span> r <span class="hljs-keyword">in</span> rs: <span class="hljs-comment"># 我们这里只监听读事件，所以只管读的返回句柄数组</span><br> socket_lock.acquire() <span class="hljs-comment"># 在读取之前先加锁，锁定socket对象（sock是主线程和子线程的共享资源，锁定了sock就能保证子线程在使用sock时，主线程无法对sock进行操作）</span><br><br> <span class="hljs-keyword">if</span> <span class="hljs-keyword">not</span> sock: <span class="hljs-comment"># 这里需要判断下，因为有可能在select后到加锁之间socket被关闭了</span><br> socket_lock.release()<br> <span class="hljs-keyword">break</span><br> <span class="hljs-keyword">try</span>:<br> data = r.recv(<span class="hljs-number">1024</span>) <span class="hljs-comment"># 读数据，按自己的方式读</span><br> <span class="hljs-keyword">except</span> Exception <span class="hljs-keyword">as</span> e:<br> data = <span class="hljs-literal">None</span><br><br> socket_lock.release() <span class="hljs-comment"># 读取完成之后解锁，释放资源</span><br><br> <span class="hljs-keyword">if</span> <span class="hljs-keyword">not</span> data:<br> <span class="hljs-built_in">print</span>(<span class="hljs-string">'server close'</span>)<br> <span class="hljs-keyword">else</span>:<br> <span class="hljs-built_in">print</span>(data)<br> ws.send(data.decode())<br><br><br><span class="hljs-keyword">def</span> <span class="hljs-title function_">start_client</span>(<span class="hljs-params">ws</span>):<br><br> sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)<br> sock.connect((Config.SOCKET_HOST, Config.SOCKET_PORT))<br> <span class="hljs-comment"># 创建线程锁，防止主线程socket被close了，子线程还在recv而引发的异常</span><br> socket_lock = threading.Lock()<br><br> <span class="hljs-comment"># 创建一个线程去读取数据</span><br> read_thread = threading.Thread(target=read_thread_method, args=(sock,socket_lock,ws))<br> read_thread.setDaemon(<span class="hljs-literal">True</span>)<br> read_thread.start()<br><br><br> <span class="hljs-keyword">return</span> sock,socket_lock,read_thread<br><br> <span class="hljs-comment"># # 测试不断写数据</span><br> <span class="hljs-comment"># for x in range(5):</span><br> <span class="hljs-comment"># print(x)</span><br> <span class="hljs-comment"># sock.send(str(x).encode())</span><br> <span class="hljs-comment"># sleep(1) # 交出CPU时间，否则其他线程只能看着</span><br><br> <span class="hljs-comment"># 清理socket，同样道理，这里需要锁定和解锁</span><br> <span class="hljs-comment"># socket_lock.acquire()</span><br> <span class="hljs-comment"># sock.close()</span><br> <span class="hljs-comment"># socket_lock.release()</span><br><br><br><span class="hljs-comment"># socket 路由</span><br><span class="hljs-meta">@sockets.route(<span class="hljs-params"><span class="hljs-string">'/'</span></span>)</span><br><span class="hljs-keyword">def</span> <span class="hljs-title function_">echo_socket</span>(<span class="hljs-params">ws</span>):<br><br> <span class="hljs-built_in">print</span>(<span class="hljs-string">'当前/:'</span>, ws)<br> sock,socket_lock,read_thread = start_client(ws)<br> <span class="hljs-built_in">print</span>(<span class="hljs-string">"连接了poll"</span>)<br> <span class="hljs-keyword">while</span> <span class="hljs-keyword">not</span> ws.closed:<br><br><br> message = ws.receive()<br> <span class="hljs-built_in">print</span>(<span class="hljs-string">"\n message:"</span>,message,<span class="hljs-built_in">type</span>(message))<br> <span class="hljs-built_in">print</span>(<span class="hljs-string">"ws.closed:"</span>, ws.closed)<br> <span class="hljs-comment"># print("线程id={},\n线程名称={},\n正在执行的线程列表:{},\n正在执行的线程数量={},\n当前激活线程={}".format(</span><br> <span class="hljs-comment"># read_thread.ident, read_thread.getName(), threading.enumerate(), threading.active_count(),</span><br> <span class="hljs-comment"># read_thread.isAlive)</span><br> <span class="hljs-comment"># )</span><br> <span class="hljs-keyword">if</span> message:<br> ws.send(<span class="hljs-string">"callback"</span>)<br> sock.send(message.encode())<br> <span class="hljs-keyword">if</span> ws.closed:<br> <span class="hljs-built_in">print</span>(<span class="hljs-string">"关闭了 sock "</span><br> socket_lock.acquire()<br> sock.close()<br> socket_lock.release()<br><br><br><br><br><span class="hljs-keyword">if</span> __name__ == <span class="hljs-string">"__main__"</span>:<br><br> <span class="hljs-keyword">try</span>:<br> server = pywsgi.WSGIServer((<span class="hljs-string">'0.0.0.0'</span>, <span class="hljs-number">2021</span>), application=app, handler_class=WebSocketHandler)<br> <span class="hljs-built_in">print</span>(<span class="hljs-string">"websocket server start ... "</span>)<br> server.serve_forever()<br> <span class="hljs-keyword">except</span> Exception <span class="hljs-keyword">as</span> e:<br> <span class="hljs-built_in">print</span>(<span class="hljs-string">'error:'</span>,e)<br><br><br><br></code></pre></td></tr></table></figure>]]></content>
<categories>
<category>python加油鸭</category>
</categories>
<tags>
<tag>socket python epoll</tag>
</tags>
</entry>
<entry>
<title>常见几种加密算法的Python实现</title>
<link href="/2020/05/29/%E5%B8%B8%E8%A7%81%E5%87%A0%E7%A7%8D%E5%8A%A0%E5%AF%86%E7%AE%97%E6%B3%95%E7%9A%84Python%E5%AE%9E%E7%8E%B0/"/>
<url>/2020/05/29/%E5%B8%B8%E8%A7%81%E5%87%A0%E7%A7%8D%E5%8A%A0%E5%AF%86%E7%AE%97%E6%B3%95%E7%9A%84Python%E5%AE%9E%E7%8E%B0/</url>
<content type="html"><![CDATA[<span id="more"></span><p>生活中我们经常会遇到一些加密算法，今天我们就聊聊这些加密算法的Python实现。部分常用的加密方法基本都有对应的Python库，基本不再需要我们用代码实现具体算法。</p><p> </p><p><strong>一、MD5加密</strong></p><p>全称：MD5消息摘要算法（英语：MD5 Message-Digest Algorithm），一种被广泛使用的密码散列函数，可以产生出一个128位（16字节）的散列值（hash value），用于确保信息传输完整一致。md5加密算法是不可逆的，所以解密一般都是通过暴力穷举方法，通过网站的接口实现解密。</p><p><strong>Python代码：</strong></p><figure class="highlight stylus"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><code class="hljs stylus">import hashlib<br>m = hashlib<span class="hljs-selector-class">.md5</span>()<br>m<span class="hljs-selector-class">.update</span>(str<span class="hljs-selector-class">.encode</span>(<span class="hljs-string">"utf8"</span>))<br><span class="hljs-function"><span class="hljs-title">print</span><span class="hljs-params">(m.hexdigest()</span></span>)<br></code></pre></td></tr></table></figure><p> </p><p><strong>二、SHA1加密</strong></p><p>全称：安全哈希算法（Secure Hash Algorithm）主要适用于数字签名标准（Digital Signature Standard DSS）里面定义的数字签名算法（Digital Signature Algorithm DSA），SHA1比MD5的安全性更强。对于长度小于2^ 64位的消息，SHA1会产生一个160位的消息摘要。</p><p><strong>Python代码：</strong></p><figure class="highlight stylus"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><code class="hljs stylus">import hashlib<br>sha1 = hashlib<span class="hljs-selector-class">.sha1</span>()<br>data = <span class="hljs-string">'2333333'</span><br>sha1<span class="hljs-selector-class">.update</span>(data<span class="hljs-selector-class">.encode</span>(<span class="hljs-string">'utf-8'</span>))<br>sha1_data = sha1<span class="hljs-selector-class">.hexdigest</span>()<br><span class="hljs-function"><span class="hljs-title">print</span><span class="hljs-params">(sha1_data)</span></span><br></code></pre></td></tr></table></figure><p> </p><p><strong>三、HMAC加密</strong></p><p>全称：散列消息鉴别码（Hash Message Authentication Code）， HMAC加密算法是一种安全的基于加密hash函数和共享密钥的消息认证协议。实现原理是用公开函数和密钥产生一个固定长度的值作为认证标识，用这个标识鉴别消息的完整性。使用一个密钥生成一个固定大小的小数据块，即 MAC，并将其加入到消息中，然后传输。接收方利用与发送方共享的密钥进行鉴别认证等。</p><p><strong>Python代码：</strong></p><figure class="highlight haxe"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><code class="hljs haxe"><span class="hljs-keyword">import</span> hmac<br><span class="hljs-keyword">import</span> hashlib<br><span class="hljs-meta"># 第一个参数是密钥key，第二个参数是待加密的字符串，第三个参数是hash函数</span><br>mac = hmac.<span class="hljs-keyword">new</span><span class="hljs-type"></span>(<span class="hljs-string">'key'</span>,<span class="hljs-string">'hello'</span>,hashlib.md5)<br>mac.digest() <span class="hljs-meta"># 字符串的ascii格式</span><br>mac.hexdigest() <span class="hljs-meta"># 加密后字符串的十六进制格式</span><br></code></pre></td></tr></table></figure><p> </p><p><strong>四、DES加密</strong></p><p>全称：数据加密标准（Data Encryption Standard），属于对称加密算法。DES是一个分组加密算法，典型的DES以64位为分组对数据加密，加密和解密用的是同一个算法。它的密钥长度是56位（因为每个第8 位都用作奇偶校验），密钥可以是任意的56位的数，而且可以任意时候改变。</p><p><strong>Python代码：</strong></p><figure class="highlight routeros"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br></pre></td><td class="code"><pre><code class="hljs routeros">import binascii<br><span class="hljs-keyword">from</span> pyDes import des, CBC, PAD_PKCS5<br><span class="hljs-comment"># 需要安装 pip install pyDes</span><br><br>def des_encrypt(secret_key, s):<br> iv = secret_key<br> k = des(secret_key, CBC, iv, <span class="hljs-attribute">pad</span>=None, <span class="hljs-attribute">padmode</span>=PAD_PKCS5)<br> en = k.encrypt(s, <span class="hljs-attribute">padmode</span>=PAD_PKCS5)<br> return binascii.b2a_hex(en)<br><br>def des_decrypt(secret_key, s):<br> iv = secret_key<br> k = des(secret_key, CBC, iv, <span class="hljs-attribute">pad</span>=None, <span class="hljs-attribute">padmode</span>=PAD_PKCS5)<br> de = k.decrypt(binascii.a2b_hex(s), <span class="hljs-attribute">padmode</span>=PAD_PKCS5)<br> return de<br><br>secret_str = des_encrypt(<span class="hljs-string">'12345678'</span>, <span class="hljs-string">'I love YOU~'</span>)<br><span class="hljs-built_in">print</span>(secret_str)<br>clear_str = des_decrypt(<span class="hljs-string">'12345678'</span>, secret_str)<br><span class="hljs-built_in">print</span>(clear_str)<br></code></pre></td></tr></table></figure><p> </p><p><strong>五、AES加密</strong></p><p>全称：高级加密标准（英语：Advanced Encryption Standard），在密码学中又称Rijndael加密法，是美国联邦政府采用的一种区块加密标准。这个标准用来替代原先的DES，已经被多方分析且广为全世界所使用。</p><p><strong>Python代码：</strong></p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-keyword">import</span> base64<br><span class="hljs-keyword">from</span> Crypto.Cipher <span class="hljs-keyword">import</span> AES<br><br><span class="hljs-string">'''</span><br><span class="hljs-string">AES对称加密算法</span><br><span class="hljs-string">'''</span><br><span class="hljs-comment"># 需要补位，str不是16的倍数那就补足为16的倍数</span><br><span class="hljs-keyword">def</span> <span class="hljs-title function_">add_to_16</span>(<span class="hljs-params">value</span>):<br> <span class="hljs-keyword">while</span> <span class="hljs-built_in">len</span>(value) % <span class="hljs-number">16</span> != <span class="hljs-number">0</span>:<br> value += <span class="hljs-string">'\0'</span><br> <span class="hljs-keyword">return</span> <span class="hljs-built_in">str</span>.encode(value) <span class="hljs-comment"># 返回bytes</span><br><span class="hljs-comment"># 加密方法</span><br><span class="hljs-keyword">def</span> <span class="hljs-title function_">encrypt</span>(<span class="hljs-params">key, text</span>):<br> aes = AES.new(add_to_16(key), AES.MODE_ECB) <span class="hljs-comment"># 初始化加密器</span><br> encrypt_aes = aes.encrypt(add_to_16(text)) <span class="hljs-comment"># 先进行aes加密</span><br> encrypted_text = <span class="hljs-built_in">str</span>(base64.encodebytes(encrypt_aes), encoding=<span class="hljs-string">'utf-8'</span>) <span class="hljs-comment"># 执行加密并转码返回bytes</span><br> <span class="hljs-keyword">return</span> encrypted_text<br><span class="hljs-comment"># 解密方法</span><br><span class="hljs-keyword">def</span> <span class="hljs-title function_">decrypt</span>(<span class="hljs-params">key, text</span>):<br> aes = AES.new(add_to_16(key), AES.MODE_ECB) <span class="hljs-comment"># 初始化加密器</span><br> base64_decrypted = base64.decodebytes(text.encode(encoding=<span class="hljs-string">'utf-8'</span>)) <span class="hljs-comment"># 优先逆向解密base64成bytes</span><br> decrypted_text = <span class="hljs-built_in">str</span>(aes.decrypt(base64_decrypted), encoding=<span class="hljs-string">'utf-8'</span>).replace(<span class="hljs-string">'\0'</span>, <span class="hljs-string">''</span>) <span class="hljs-comment"># 执行解密密并转码返回str</span><br> <span class="hljs-keyword">return</span> decrypted_text<br></code></pre></td></tr></table></figure><p> </p><p><strong>六、RSA加密</strong></p><p>全称：Rivest-Shamir-Adleman，RSA加密算法是一种非对称加密算法。在公开密钥加密和电子商业中RSA被广泛使用。它被普遍认为是目前最优秀的公钥方案之一。RSA是第一个能同时用于加密和数字签名的算法，它能够抵抗到目前为止已知的所有密码攻击。</p><p><strong>Python代码：</strong></p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-comment"># -*- coding: UTF-8 -*-</span><br><span class="hljs-comment"># reference codes: https://www.jianshu.com/p/7a4645691c68</span><br><br><span class="hljs-keyword">import</span> base64<br><span class="hljs-keyword">import</span> rsa<br><span class="hljs-keyword">from</span> rsa <span class="hljs-keyword">import</span> common<br><br><span class="hljs-comment"># 使用 rsa库进行RSA签名和加解密</span><br><span class="hljs-keyword">class</span> <span class="hljs-title class_">RsaUtil</span>(<span class="hljs-title class_ inherited__">object</span>):<br> PUBLIC_KEY_PATH = <span class="hljs-string">'xxxxpublic_key.pem'</span> <span class="hljs-comment"># 公钥</span><br> PRIVATE_KEY_PATH = <span class="hljs-string">'xxxxxprivate_key.pem'</span> <span class="hljs-comment"># 私钥</span><br><br> <span class="hljs-comment"># 初始化key</span><br> <span class="hljs-keyword">def</span> <span class="hljs-title function_">__init__</span>(<span class="hljs-params">self,</span><br><span class="hljs-params"> company_pub_file=PUBLIC_KEY_PATH,</span><br><span class="hljs-params"> company_pri_file=PRIVATE_KEY_PATH</span>):<br><br> <span class="hljs-keyword">if</span> company_pub_file:<br> self.company_public_key = rsa.PublicKey.load_pkcs1_openssl_pem(<span class="hljs-built_in">open</span>(company_pub_file).read())<br> <span class="hljs-keyword">if</span> company_pri_file:<br> self.company_private_key = rsa.PrivateKey.load_pkcs1(<span class="hljs-built_in">open</span>(company_pri_file).read())<br><br> <span class="hljs-keyword">def</span> <span class="hljs-title function_">get_max_length</span>(<span class="hljs-params">self, rsa_key, encrypt=<span class="hljs-literal">True</span></span>):<br> <span class="hljs-string">"""加密内容过长时 需要分段加密 换算每一段的长度.</span><br><span class="hljs-string"> :param rsa_key: 钥匙.</span><br><span class="hljs-string"> :param encrypt: 是否是加密.</span><br><span class="hljs-string"> """</span><br> blocksize = common.byte_size(rsa_key.n)<br> reserve_size = <span class="hljs-number">11</span> <span class="hljs-comment"># 预留位为11</span><br> <span class="hljs-keyword">if</span> <span class="hljs-keyword">not</span> encrypt: <span class="hljs-comment"># 解密时不需要考虑预留位</span><br> reserve_size = <span class="hljs-number">0</span><br> maxlength = blocksize - reserve_size<br> <span class="hljs-keyword">return</span> maxlength<br><br> <span class="hljs-comment"># 加密 支付方公钥</span><br> <span class="hljs-keyword">def</span> <span class="hljs-title function_">encrypt_by_public_key</span>(<span class="hljs-params">self, message</span>):<br> <span class="hljs-string">"""使用公钥加密.</span><br><span class="hljs-string"> :param message: 需要加密的内容.</span><br><span class="hljs-string"> 加密之后需要对接过进行base64转码</span><br><span class="hljs-string"> """</span><br> encrypt_result = <span class="hljs-string">b''</span><br> max_length = self.get_max_length(self.company_public_key)<br> <span class="hljs-keyword">while</span> message:<br> <span class="hljs-built_in">input</span> = message[:max_length]<br> message = message[max_length:]<br> out = rsa.encrypt(<span class="hljs-built_in">input</span>, self.company_public_key)<br> encrypt_result += out<br> encrypt_result = base64.b64encode(encrypt_result)<br> <span class="hljs-keyword">return</span> encrypt_result<br><br> <span class="hljs-keyword">def</span> <span class="hljs-title function_">decrypt_by_private_key</span>(<span class="hljs-params">self, message</span>):<br> <span class="hljs-string">"""使用私钥解密.</span><br><span class="hljs-string"> :param message: 需要加密的内容.</span><br><span class="hljs-string"> 解密之后的内容直接是字符串，不需要在进行转义</span><br><span class="hljs-string"> """</span><br> decrypt_result = <span class="hljs-string">b""</span><br><br> max_length = self.get_max_length(self.company_private_key, <span class="hljs-literal">False</span>)<br> decrypt_message = base64.b64decode(message)<br> <span class="hljs-keyword">while</span> decrypt_message:<br> <span class="hljs-built_in">input</span> = decrypt_message[:max_length]<br> decrypt_message = decrypt_message[max_length:]<br> out = rsa.decrypt(<span class="hljs-built_in">input</span>, self.company_private_key)<br> decrypt_result += out<br> <span class="hljs-keyword">return</span> decrypt_result<br><br> <span class="hljs-comment"># 签名 商户私钥 base64转码</span><br> <span class="hljs-keyword">def</span> <span class="hljs-title function_">sign_by_private_key</span>(<span class="hljs-params">self, data</span>):<br> <span class="hljs-string">"""私钥签名.</span><br><span class="hljs-string"> :param data: 需要签名的内容.</span><br><span class="hljs-string"> 使用SHA-1 方法进行签名（也可以使用MD5）</span><br><span class="hljs-string"> 签名之后，需要转义后输出</span><br><span class="hljs-string"> """</span><br> signature = rsa.sign(<span class="hljs-built_in">str</span>(data), priv_key=self.company_private_key, <span class="hljs-built_in">hash</span>=<span class="hljs-string">'SHA-1'</span>)<br> <span class="hljs-keyword">return</span> base64.b64encode(signature)<br><br> <span class="hljs-keyword">def</span> <span class="hljs-title function_">verify_by_public_key</span>(<span class="hljs-params">self, message, signature</span>):<br> <span class="hljs-string">"""公钥验签.</span><br><span class="hljs-string"> :param message: 验签的内容.</span><br><span class="hljs-string"> :param signature: 对验签内容签名的值（签名之后，会进行b64encode转码，所以验签前也需转码）.</span><br><span class="hljs-string"> """</span><br> signature = base64.b64decode(signature)<br> <span class="hljs-keyword">return</span> rsa.verify(message, signature, self.company_public_key)<br></code></pre></td></tr></table></figure><p> </p><p><strong>七、ECC加密</strong></p><p>全称：椭圆曲线加密（Elliptic Curve Cryptography），ECC加密算法是一种公钥加密技术，以椭圆曲线理论为基础。利用有限域上椭圆曲线的点构成的Abel群离散对数难解性，实现加密、解密和数字签名。将椭圆曲线中的加法运算与离散对数中的模乘运算相对应，就可以建立基于椭圆曲线的对应密码体制。</p><p><strong>Python代码：</strong></p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br><span class="line">97</span><br><span class="line">98</span><br><span class="line">99</span><br><span class="line">100</span><br><span class="line">101</span><br><span class="line">102</span><br><span class="line">103</span><br><span class="line">104</span><br><span class="line">105</span><br><span class="line">106</span><br><span class="line">107</span><br><span class="line">108</span><br><span class="line">109</span><br><span class="line">110</span><br><span class="line">111</span><br><span class="line">112</span><br><span class="line">113</span><br><span class="line">114</span><br><span class="line">115</span><br><span class="line">116</span><br><span class="line">117</span><br><span class="line">118</span><br><span class="line">119</span><br><span class="line">120</span><br><span class="line">121</span><br><span class="line">122</span><br><span class="line">123</span><br><span class="line">124</span><br><span class="line">125</span><br><span class="line">126</span><br><span class="line">127</span><br><span class="line">128</span><br><span class="line">129</span><br><span class="line">130</span><br><span class="line">131</span><br><span class="line">132</span><br><span class="line">133</span><br><span class="line">134</span><br><span class="line">135</span><br><span class="line">136</span><br><span class="line">137</span><br><span class="line">138</span><br><span class="line">139</span><br><span class="line">140</span><br><span class="line">141</span><br><span class="line">142</span><br><span class="line">143</span><br><span class="line">144</span><br><span class="line">145</span><br><span class="line">146</span><br><span class="line">147</span><br><span class="line">148</span><br><span class="line">149</span><br><span class="line">150</span><br><span class="line">151</span><br><span class="line">152</span><br><span class="line">153</span><br><span class="line">154</span><br><span class="line">155</span><br><span class="line">156</span><br><span class="line">157</span><br><span class="line">158</span><br><span class="line">159</span><br><span class="line">160</span><br><span class="line">161</span><br><span class="line">162</span><br><span class="line">163</span><br><span class="line">164</span><br><span class="line">165</span><br><span class="line">166</span><br><span class="line">167</span><br><span class="line">168</span><br><span class="line">169</span><br><span class="line">170</span><br><span class="line">171</span><br><span class="line">172</span><br><span class="line">173</span><br><span class="line">174</span><br><span class="line">175</span><br><span class="line">176</span><br><span class="line">177</span><br><span class="line">178</span><br><span class="line">179</span><br><span class="line">180</span><br><span class="line">181</span><br><span class="line">182</span><br><span class="line">183</span><br><span class="line">184</span><br><span class="line">185</span><br><span class="line">186</span><br><span class="line">187</span><br><span class="line">188</span><br><span class="line">189</span><br><span class="line">190</span><br><span class="line">191</span><br><span class="line">192</span><br><span class="line">193</span><br><span class="line">194</span><br><span class="line">195</span><br><span class="line">196</span><br><span class="line">197</span><br><span class="line">198</span><br><span class="line">199</span><br><span class="line">200</span><br><span class="line">201</span><br><span class="line">202</span><br><span class="line">203</span><br><span class="line">204</span><br><span class="line">205</span><br><span class="line">206</span><br><span class="line">207</span><br><span class="line">208</span><br><span class="line">209</span><br><span class="line">210</span><br><span class="line">211</span><br><span class="line">212</span><br><span class="line">213</span><br><span class="line">214</span><br><span class="line">215</span><br><span class="line">216</span><br><span class="line">217</span><br><span class="line">218</span><br><span class="line">219</span><br><span class="line">220</span><br><span class="line">221</span><br><span class="line">222</span><br><span class="line">223</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-comment"># -*- coding:utf-8 *-</span><br><span class="hljs-comment"># author: DYBOY</span><br><span class="hljs-comment"># reference codes: https://blog.dyboy.cn/websecurity/121.html</span><br><span class="hljs-comment"># description: ECC椭圆曲线加密算法实现</span><br><span class="hljs-string">"""</span><br><span class="hljs-string"> 考虑K=kG ，其中K、G为椭圆曲线Ep(a,b)上的点，n为G的阶（nG=O∞ ），k为小于n的整数。</span><br><span class="hljs-string"> 则给定k和G，根据加法法则，计算K很容易但反过来，给定K和G，求k就非常困难。</span><br><span class="hljs-string"> 因为实际使用中的ECC原则上把p取得相当大，n也相当大，要把n个解点逐一算出来列成上表是不可能的。</span><br><span class="hljs-string"> 这就是椭圆曲线加密算法的数学依据</span><br><span class="hljs-string"> 点G称为基点（base point）</span><br><span class="hljs-string"> k（k<n）为私有密钥（privte key）</span><br><span class="hljs-string"> K为公开密钥（public key)</span><br><span class="hljs-string">"""</span><br><br><span class="hljs-keyword">def</span> <span class="hljs-title function_">get_inverse</span>(<span class="hljs-params">mu, p</span>):<br> <span class="hljs-string">"""</span><br><span class="hljs-string"> 获取y的负元</span><br><span class="hljs-string"> """</span><br> <span class="hljs-keyword">for</span> i <span class="hljs-keyword">in</span> <span class="hljs-built_in">range</span>(<span class="hljs-number">1</span>, p):<br> <span class="hljs-keyword">if</span> (i*mu)%p == <span class="hljs-number">1</span>:<br> <span class="hljs-keyword">return</span> i<br> <span class="hljs-keyword">return</span> -<span class="hljs-number">1</span><br><br><span class="hljs-keyword">def</span> <span class="hljs-title function_">get_gcd</span>(<span class="hljs-params">zi, mu</span>):<br> <span class="hljs-string">"""</span><br><span class="hljs-string"> 获取最大公约数</span><br><span class="hljs-string"> """</span><br> <span class="hljs-keyword">if</span> mu:<br> <span class="hljs-keyword">return</span> get_gcd(mu, zi%mu)<br> <span class="hljs-keyword">else</span>:<br> <span class="hljs-keyword">return</span> zi<br><br><span class="hljs-keyword">def</span> <span class="hljs-title function_">get_np</span>(<span class="hljs-params">x1, y1, x2, y2, a, p</span>):<br> <span class="hljs-string">"""</span><br><span class="hljs-string"> 获取n*p，每次+p，直到求解阶数np=-p</span><br><span class="hljs-string"> """</span><br> flag = <span class="hljs-number">1</span> <span class="hljs-comment"># 定义符号位（+/-）</span><br><br> <span class="hljs-comment"># 如果 p=q k=(3x2+a)/2y1mod p</span><br> <span class="hljs-keyword">if</span> x1 == x2 <span class="hljs-keyword">and</span> y1 == y2:<br> zi = <span class="hljs-number">3</span> * (x1 ** <span class="hljs-number">2</span>) + a <span class="hljs-comment"># 计算分子 【求导】</span><br> mu = <span class="hljs-number">2</span> * y1 <span class="hljs-comment"># 计算分母</span><br><br> <span class="hljs-comment"># 若P≠Q，则k=(y2-y1)/(x2-x1) mod p</span><br> <span class="hljs-keyword">else</span>:<br> zi = y2 - y1<br> mu = x2 - x1<br> <span class="hljs-keyword">if</span> zi* mu < <span class="hljs-number">0</span>:<br> flag = <span class="hljs-number">0</span> <span class="hljs-comment"># 符号0为-（负数）</span><br> zi = <span class="hljs-built_in">abs</span>(zi)<br> mu = <span class="hljs-built_in">abs</span>(mu)<br><br> <span class="hljs-comment"># 将分子和分母化为最简</span><br> gcd_value = get_gcd(zi, mu) <span class="hljs-comment"># 最大公約數</span><br> zi = zi // gcd_value <span class="hljs-comment"># 整除</span><br> mu = mu // gcd_value<br> <span class="hljs-comment"># 求分母的逆元 逆元： ∀a ∈G ，ョb∈G 使得 ab = ba = e</span><br> <span class="hljs-comment"># P(x,y)的负元是 (x,-y mod p)= (x,p-y) ，有P+(-P)= O∞</span><br> inverse_value = get_inverse(mu, p)<br> k = (zi * inverse_value)<br><br> <span class="hljs-keyword">if</span> flag == <span class="hljs-number">0</span>: <span class="hljs-comment"># 斜率负数 flag==0</span><br> k = -k<br> k = k % p<br> <span class="hljs-comment"># 计算x3,y3 P+Q</span><br> <span class="hljs-string">"""</span><br><span class="hljs-string"> x3≡k2-x1-x2(mod p)</span><br><span class="hljs-string"> y3≡k(x1-x3)-y1(mod p)</span><br><span class="hljs-string"> """</span><br> x3 = (k ** <span class="hljs-number">2</span> - x1 - x2) % p<br> y3 = (k * (x1 - x3) - y1) % p<br> <span class="hljs-keyword">return</span> x3,y3<br><br><span class="hljs-keyword">def</span> <span class="hljs-title function_">get_rank</span>(<span class="hljs-params">x0, y0, a, b, p</span>):<br> <span class="hljs-string">"""</span><br><span class="hljs-string"> 获取椭圆曲线的阶</span><br><span class="hljs-string"> """</span><br> x1 = x0 <span class="hljs-comment">#-p的x坐标</span><br> y1 = (-<span class="hljs-number">1</span>*y0)%p <span class="hljs-comment">#-p的y坐标</span><br> tempX = x0<br> tempY = y0<br> n = <span class="hljs-number">1</span><br> <span class="hljs-keyword">while</span> <span class="hljs-literal">True</span>:<br> n += <span class="hljs-number">1</span><br> <span class="hljs-comment"># 求p+q的和，得到n*p，直到求出阶</span><br> p_x,p_y = get_np(tempX, tempY, x0, y0, a, p)<br> <span class="hljs-comment"># 如果 == -p,那么阶数+1，返回</span><br> <span class="hljs-keyword">if</span> p_x == x1 <span class="hljs-keyword">and</span> p_y == y1:<br> <span class="hljs-keyword">return</span> n+<span class="hljs-number">1</span><br> tempX = p_x<br> tempY = p_y<br><br><span class="hljs-keyword">def</span> <span class="hljs-title function_">get_param</span>(<span class="hljs-params">x0, a, b, p</span>):<br> <span class="hljs-string">"""</span><br><span class="hljs-string"> 计算p与-p</span><br><span class="hljs-string"> """</span><br> y0 = -<span class="hljs-number">1</span><br> <span class="hljs-keyword">for</span> i <span class="hljs-keyword">in</span> <span class="hljs-built_in">range</span>(p):<br> <span class="hljs-comment"># 满足取模约束条件，椭圆曲线Ep(a,b)，p为质数，x,y∈[0,p-1]</span><br> <span class="hljs-keyword">if</span> i**<span class="hljs-number">2</span>%p == (x0**<span class="hljs-number">3</span> + a*x0 + b)%p:<br> y0 = i<br> <span class="hljs-keyword">break</span><br><br> <span class="hljs-comment"># 如果y0没有，返回false</span><br> <span class="hljs-keyword">if</span> y0 == -<span class="hljs-number">1</span>:<br> <span class="hljs-keyword">return</span> <span class="hljs-literal">False</span><br><br> <span class="hljs-comment"># 计算-y（负数取模）</span><br> x1 = x0<br> y1 = (-<span class="hljs-number">1</span>*y0) % p<br> <span class="hljs-keyword">return</span> x0,y0,x1,y1<br><br><span class="hljs-keyword">def</span> <span class="hljs-title function_">get_graph</span>(<span class="hljs-params">a, b, p</span>):<br> <span class="hljs-string">"""</span><br><span class="hljs-string"> 输出椭圆曲线散点图</span><br><span class="hljs-string"> """</span><br> x_y = []<br> <span class="hljs-comment"># 初始化二维数组</span><br> <span class="hljs-keyword">for</span> i <span class="hljs-keyword">in</span> <span class="hljs-built_in">range</span>(p):<br> x_y.append([<span class="hljs-string">'-'</span> <span class="hljs-keyword">for</span> i <span class="hljs-keyword">in</span> <span class="hljs-built_in">range</span>(p)])<br><br> <span class="hljs-keyword">for</span> i <span class="hljs-keyword">in</span> <span class="hljs-built_in">range</span>(p):<br> val =get_param(i, a, b, p) <span class="hljs-comment"># 椭圆曲线上的点</span><br> <span class="hljs-keyword">if</span>(val != <span class="hljs-literal">False</span>):<br> x0,y0,x1,y1 = val<br> x_y[x0][y0] = <span class="hljs-number">1</span><br> x_y[x1][y1] = <span class="hljs-number">1</span><br><br> <span class="hljs-built_in">print</span>(<span class="hljs-string">"椭圆曲线的散列图为："</span>)<br> <span class="hljs-keyword">for</span> i <span class="hljs-keyword">in</span> <span class="hljs-built_in">range</span>(p): <span class="hljs-comment"># i= 0-> p-1</span><br> temp = p-<span class="hljs-number">1</span>-i <span class="hljs-comment"># 倒序</span><br><br> <span class="hljs-comment"># 格式化输出1/2位数，y坐标轴</span><br> <span class="hljs-keyword">if</span> temp >= <span class="hljs-number">10</span>:<br> <span class="hljs-built_in">print</span>(temp, end=<span class="hljs-string">" "</span>)<br> <span class="hljs-keyword">else</span>:<br> <span class="hljs-built_in">print</span>(temp, end=<span class="hljs-string">" "</span>)<br><br> <span class="hljs-comment"># 输出具体坐标的值，一行</span><br> <span class="hljs-keyword">for</span> j <span class="hljs-keyword">in</span> <span class="hljs-built_in">range</span>(p):<br> <span class="hljs-built_in">print</span>(x_y[j][temp], end=<span class="hljs-string">" "</span>)<br> <span class="hljs-built_in">print</span>(<span class="hljs-string">""</span>) <span class="hljs-comment">#换行</span><br><br> <span class="hljs-comment"># 输出 x 坐标轴</span><br> <span class="hljs-built_in">print</span>(<span class="hljs-string">" "</span>, end=<span class="hljs-string">""</span>)<br> <span class="hljs-keyword">for</span> i <span class="hljs-keyword">in</span> <span class="hljs-built_in">range</span>(p):<br> <span class="hljs-keyword">if</span> i >=<span class="hljs-number">10</span>:<br> <span class="hljs-built_in">print</span>(i, end=<span class="hljs-string">" "</span>)<br> <span class="hljs-keyword">else</span>:<br> <span class="hljs-built_in">print</span>(i, end=<span class="hljs-string">" "</span>)<br> <span class="hljs-built_in">print</span>(<span class="hljs-string">'\n'</span>)<br><br><span class="hljs-keyword">def</span> <span class="hljs-title function_">get_ng</span>(<span class="hljs-params">G_x, G_y, key, a, p</span>):<br> <span class="hljs-string">"""</span><br><span class="hljs-string"> 计算nG</span><br><span class="hljs-string"> """</span><br> temp_x = G_x<br> temp_y = G_y<br> <span class="hljs-keyword">while</span> key != <span class="hljs-number">1</span>:<br> temp_x,temp_y = get_np(temp_x,temp_y, G_x, G_y, a, p)<br> key -= <span class="hljs-number">1</span><br> <span class="hljs-keyword">return</span> temp_x,temp_y<br><br><span class="hljs-keyword">def</span> <span class="hljs-title function_">ecc_main</span>():<br> <span class="hljs-keyword">while</span> <span class="hljs-literal">True</span>:<br> a = <span class="hljs-built_in">int</span>(<span class="hljs-built_in">input</span>(<span class="hljs-string">"请输入椭圆曲线参数a(a>0)的值："</span>))<br> b = <span class="hljs-built_in">int</span>(<span class="hljs-built_in">input</span>(<span class="hljs-string">"请输入椭圆曲线参数b(b>0)的值："</span>))<br> p = <span class="hljs-built_in">int</span>(<span class="hljs-built_in">input</span>(<span class="hljs-string">"请输入椭圆曲线参数p(p为素数)的值："</span>)) <span class="hljs-comment">#用作模运算</span><br><br> <span class="hljs-comment"># 条件满足判断</span><br> <span class="hljs-keyword">if</span> (<span class="hljs-number">4</span>*(a**<span class="hljs-number">3</span>)+<span class="hljs-number">27</span>*(b**<span class="hljs-number">2</span>))%p == <span class="hljs-number">0</span>:<br> <span class="hljs-built_in">print</span>(<span class="hljs-string">"您输入的参数有误，请重新输入！！！\n"</span>)<br> <span class="hljs-keyword">else</span>:<br> <span class="hljs-keyword">break</span><br><br> <span class="hljs-comment"># 输出椭圆曲线散点图</span><br> get_graph(a, b, p)<br><br> <span class="hljs-comment"># 选点作为G点</span><br> <span class="hljs-built_in">print</span>(<span class="hljs-string">"user1：在如上坐标系中选一个值为G的坐标"</span>)<br> G_x = <span class="hljs-built_in">int</span>(<span class="hljs-built_in">input</span>(<span class="hljs-string">"user1：请输入选取的x坐标值："</span>))<br> G_y = <span class="hljs-built_in">int</span>(<span class="hljs-built_in">input</span>(<span class="hljs-string">"user1：请输入选取的y坐标值："</span>))<br><br> <span class="hljs-comment"># 获取椭圆曲线的阶</span><br> n = get_rank(G_x, G_y, a, b, p)<br><br> <span class="hljs-comment"># user1生成私钥，小key</span><br> key = <span class="hljs-built_in">int</span>(<span class="hljs-built_in">input</span>(<span class="hljs-string">"user1：请输入私钥小key（<{}）："</span>.<span class="hljs-built_in">format</span>(n)))<br><br> <span class="hljs-comment"># user1生成公钥，大KEY</span><br> KEY_x,kEY_y = get_ng(G_x, G_y, key, a, p)<br><br> <span class="hljs-comment"># user2阶段</span><br> <span class="hljs-comment"># user2拿到user1的公钥KEY，Ep(a,b)阶n，加密需要加密的明文数据</span><br> <span class="hljs-comment"># 加密准备</span><br> k = <span class="hljs-built_in">int</span>(<span class="hljs-built_in">input</span>(<span class="hljs-string">"user2：请输入一个整数k（<{}）用于求kG和kQ："</span>.<span class="hljs-built_in">format</span>(n)))<br> k_G_x,k_G_y = get_ng(G_x, G_y, k, a, p) <span class="hljs-comment"># kG</span><br> k_Q_x,k_Q_y = get_ng(KEY_x, kEY_y, k, a, p) <span class="hljs-comment"># kQ</span><br><br> <span class="hljs-comment"># 加密</span><br> plain_text = <span class="hljs-built_in">input</span>(<span class="hljs-string">"user2：请输入需要加密的字符串:"</span>)<br> plain_text = plain_text.strip()<br> <span class="hljs-comment">#plain_text = int(input("user1：请输入需要加密的密文："))</span><br> c = []<br> <span class="hljs-built_in">print</span>(<span class="hljs-string">"密文为："</span>,end=<span class="hljs-string">""</span>)<br> <span class="hljs-keyword">for</span> char <span class="hljs-keyword">in</span> plain_text:<br> intchar = <span class="hljs-built_in">ord</span>(char)<br> cipher_text = intchar*k_Q_x<br> c.append([k_G_x, k_G_y, cipher_text])<br> <span class="hljs-built_in">print</span>(<span class="hljs-string">"({},{}),{}"</span>.<span class="hljs-built_in">format</span>(k_G_x, k_G_y, cipher_text),end=<span class="hljs-string">"-"</span>)<br><br><br> <span class="hljs-comment"># user1阶段</span><br> <span class="hljs-comment"># 拿到user2加密的数据进行解密</span><br> <span class="hljs-comment"># 知道 k_G_x,k_G_y，key情况下，求解k_Q_x,k_Q_y是容易的，然后plain_text = cipher_text/k_Q_x</span><br> <span class="hljs-built_in">print</span>(<span class="hljs-string">"\nuser1解密得到明文："</span>,end=<span class="hljs-string">""</span>)<br> <span class="hljs-keyword">for</span> charArr <span class="hljs-keyword">in</span> c:<br> decrypto_text_x,decrypto_text_y = get_ng(charArr[<span class="hljs-number">0</span>], charArr[<span class="hljs-number">1</span>], key, a, p)<br> <span class="hljs-built_in">print</span>(<span class="hljs-built_in">chr</span>(charArr[<span class="hljs-number">2</span>]//decrypto_text_x),end=<span class="hljs-string">""</span>)<br><br><span class="hljs-keyword">if</span> __name__ == <span class="hljs-string">"__main__"</span>:<br> <span class="hljs-built_in">print</span>(<span class="hljs-string">"*************ECC椭圆曲线加密*************"</span>)<br> ecc_main()<br></code></pre></td></tr></table></figure><p>- 来源于Python乱炖 ，作者 - 我被狗咬了</p>]]></content>
<categories>
<category>python加油鸭</category>
</categories>
<tags>
<tag>python 加密解密 算法</tag>
</tags>
</entry>
<entry>
<title>selector Python selectors模块 I/O多路复用</title>
<link href="/2020/04/29/selector%20Python%20selectors%E6%A8%A1%E5%9D%97%20I!O%E5%A4%9A%E8%B7%AF%E5%A4%8D%E7%94%A8/"/>
<url>/2020/04/29/selector%20Python%20selectors%E6%A8%A1%E5%9D%97%20I!O%E5%A4%9A%E8%B7%AF%E5%A4%8D%E7%94%A8/</url>
<content type="html"><![CDATA[<span id="more"></span><h3 id="selectors模块"><a href="#selectors模块" class="headerlink" title="selectors模块"></a><strong>selectors模块</strong></h3><p>此模块允许高级和高效的I / O多路复用，构建在select模块原语上。鼓励用户使用此模块，除非他们需要精确控制所使用的操作系统级原语。（ 默认使用epoll，但由于Windows不支持epoll，如果在你的Windows上找不到epoll的话，就会用select） 它定义了一个抽象基类，有几个具体的实现工具(KqueueSelector, EpollSelector…),可以用于等待多个文件对象的I / O就绪通知。在下文中，file object” 指的是任何fileno() method，或一个原始文件的描述符。请参阅文件对象。BaseSelectorKqueueSelectorEpollSelectorfileno()<br> <strong>DefaultSelector</strong> 是别名到当前平台上可用的最有效的实现：这应该是大多数用户的默认选择<br> 注意支持的文件对象类型取决于平台：在Windows上，支持套接字，但不支持管道，而在Unix上，支持两者（也可以支持其他类型，例如fifos或特殊文件设备）</p><h3 id="实例-server端"><a href="#实例-server端" class="headerlink" title="实例- server端"></a>实例- server端</h3><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-keyword">import</span> selectors<br><span class="hljs-keyword">from</span> socket <span class="hljs-keyword">import</span> *<br><br><span class="hljs-keyword">def</span> <span class="hljs-title function_">accept</span>(<span class="hljs-params">sk,mask</span>):<br> conn,addr=sk.accept()<br> sel.register(conn,selectors.EVENT_READ,read)<br><br><span class="hljs-keyword">def</span> <span class="hljs-title function_">read</span>(<span class="hljs-params">conn,mask</span>):<br> <span class="hljs-keyword">try</span>:<br> data=conn.recv(<span class="hljs-number">1024</span>)<br> <span class="hljs-keyword">if</span> <span class="hljs-keyword">not</span> data:<br> <span class="hljs-built_in">print</span>(<span class="hljs-string">'closing'</span>,conn)<br> sel.unregister(conn)<br> conn.close()<br> <span class="hljs-keyword">return</span><br> conn.send(data.upper()+<span class="hljs-string">b'_SB'</span>)<br> <span class="hljs-keyword">except</span> Exception:<br> <span class="hljs-built_in">print</span>(<span class="hljs-string">'closing'</span>, conn)<br> sel.unregister(conn)<br> conn.close()<br><br>sk=socket()<br>sk.setsockopt(SOL_SOCKET,SO_REUSEADDR,<span class="hljs-number">1</span>)<br>sk.bind((<span class="hljs-string">'127.0.0.1'</span>,<span class="hljs-number">8008</span>))<br>sk.listen(<span class="hljs-number">5</span>)<br>sk.setblocking(<span class="hljs-literal">False</span>) <span class="hljs-comment">#设置socket的接口为非阻塞</span><br>sel=selectors.DefaultSelector() <span class="hljs-comment"># 选择一个适合我的IO多路复用的机制</span><br>sel.register(sk,selectors.EVENT_READ,accept)<br><span class="hljs-comment">#相当于网select的读列表里append了一个sk对象,并且绑定了一个回调函数accept</span><br><span class="hljs-comment"># 说白了就是 如果有人请求连接sk,就调用accept方法</span><br><br><span class="hljs-keyword">while</span> <span class="hljs-literal">True</span>:<br> events=sel.select() <span class="hljs-comment">#检测所有的sk,conn，是否有完成wait data阶段</span><br> <span class="hljs-keyword">for</span> sel_obj,mask <span class="hljs-keyword">in</span> events: <span class="hljs-comment"># [conn]</span><br> callback=sel_obj.data <span class="hljs-comment">#callback=read</span><br> callback(sel_obj.fileobj,mask) <span class="hljs-comment">#read(conn,1)</span><br></code></pre></td></tr></table></figure><h3 id="实例-client端"><a href="#实例-client端" class="headerlink" title="实例-client端"></a>实例-client端</h3><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-keyword">import</span> time<br><span class="hljs-keyword">import</span> socket<br><span class="hljs-keyword">import</span> threading<br><span class="hljs-keyword">def</span> <span class="hljs-title function_">func</span>():<br> sk = socket.socket()<br> sk.connect((<span class="hljs-string">'127.0.0.1'</span>,<span class="hljs-number">8008</span>))<br> sk.send(<span class="hljs-string">b'hello'</span>)<br> time.sleep(<span class="hljs-number">3</span>)<br> <span class="hljs-built_in">print</span>(sk.recv(<span class="hljs-number">1024</span>))<br> sk.close()<br><br><span class="hljs-keyword">for</span> i <span class="hljs-keyword">in</span> <span class="hljs-built_in">range</span>(<span class="hljs-number">20</span>):<br> threading.Thread(target=func).start()<br></code></pre></td></tr></table></figure>]]></content>
<categories>
<category>python加油鸭</category>
</categories>
<tags>
<tag>python socket epoll</tag>
</tags>
</entry>
<entry>
<title>Python IO模型（阻塞、非阻塞、多路复用与异步）</title>
<link href="/2020/04/29/Python%20IO%E6%A8%A1%E5%9E%8B%EF%BC%88%E9%98%BB%E5%A1%9E%E3%80%81%E9%9D%9E%E9%98%BB%E5%A1%9E%E3%80%81%E5%A4%9A%E8%B7%AF%E5%A4%8D%E7%94%A8%E4%B8%8E%E5%BC%82%E6%AD%A5%EF%BC%89/"/>
<url>/2020/04/29/Python%20IO%E6%A8%A1%E5%9E%8B%EF%BC%88%E9%98%BB%E5%A1%9E%E3%80%81%E9%9D%9E%E9%98%BB%E5%A1%9E%E3%80%81%E5%A4%9A%E8%B7%AF%E5%A4%8D%E7%94%A8%E4%B8%8E%E5%BC%82%E6%AD%A5%EF%BC%89/</url>
<content type="html"><![CDATA[<span id="more"></span><h1 id="IO模型"><a href="#IO模型" class="headerlink" title="IO模型"></a>IO模型</h1><p> 同步IO和异步IO，阻塞IO和非阻塞IO分别是什么，到底有什么区别？不同环境下给出的答案也是不一的。所以先限定一下上下文是非常有必要的。</p><p><em><code>本文讨论的背景是Linux环境下的network IO。</code></em></p><p><strong>在深入了解之前，我们应先了解几个概念：</strong></p><p> 用户空间和内核空间<br> - 进程切换<br> - 进程的阻塞<br> - 文件描述符<br> - 缓存 I/O</p><p><strong>用户空间与内核空间</strong></p><p> 现在操作系统都是采用虚拟存储器，那么对32位操作系统而言，它的寻址空间（虚拟存储空间）为4G（2的32次方）。操作系统的核心是内核，独立于普通的应用程序，可以访问受保护的内存空间，也有访问底层硬件设备的所有权限。为了保证用户进程不能直接操作内核（kernel），保证内核的安全，操心系统将虚拟空间划分为两部分，一部分为内核空间，一部分为用户空间。针对linux操作系统而言，将最高的1G字节（从虚拟地址0xC0000000到0xFFFFFFFF），供内核使用，称为内核空间，而将较低的3G字节（从虚拟地址0x00000000到0xBFFFFFFF），供各个进程使用，称为用户空间。</p><p> </p><p><strong>进程切换</strong></p><p> 为了控制进程的执行，内核必须有能力挂起正在CPU上运行的进程，并恢复以前挂起的某个进程的执行。这种行为被称为进程切换。因此可以说，任何进程都是在操作系统内核的支持下运行的，是与内核紧密相关的。</p><p>从一个进程的运行转到另一个进程上运行，这个过程中经过下面这些变化：<br> 1. 保存处理机上下文，包括程序计数器和其他寄存器。<br> 2. 更新PCB信息。</p><p> 3. 把进程的PCB移入相应的队列，如就绪、在某事件阻塞等队列。<br> 4. 选择另一个进程执行，并更新其PCB。<br> 5. 更新内存管理的数据结构。<br> 6. 恢复处理机上下文。</p><p>总而言之就是很耗资源，具体的可以参考这篇文章：<a href="http://guojing.me/linux-kernel-architecture/posts/process-switch/">进程切换</a></p><p>_注：进程控制块（Processing Control Block），是<a href="http://baike.baidu.com/view/880.htm">操作系统</a><a href="http://baike.baidu.com/view/22680.htm">核心</a>中一种数据结构，主要表示<a href="http://baike.baidu.com/view/19746.htm">进程</a>状态。其作用是使一个在<a href="http://baike.baidu.com/view/1189611.htm">多道程序</a>环境下不能独立运行的程序（含数据），成为一个能独立运行的基本单位或与其它进程并发执行的进程。或者说，OS是根据PCB来对并发执行的进程进行控制和管理的。 PCB通常是系统内存占用区中的一个连续存区，它存放着<a href="http://baike.baidu.com/view/880.htm">操作系统</a>用于描述进程情况及控制进程运行所需的全部信息 _</p><p><strong>进程的阻塞</strong></p><p> 正在执行的进程，由于期待的某些事件未发生，如请求系统资源失败、等待某种操作的完成、新数据尚未到达或无新工作做等，则由系统自动执行阻塞原语(Block)，使自己由运行状态变为阻塞状态。可见，进程的阻塞是进程自身的一种主动行为，也因此只有处于运行态的进程（获得CPU），才可能将其转为阻塞状态。<code>当进程进入阻塞状态，是不占用CPU资源的</code>。</p><p><strong>文件描述符fd</strong></p><p> 文件描述符（File descriptor）是计算机科学中的一个术语，是一个用于表述指向文件的引用的抽象化概念。</p><p> 文件描述符在形式上是一个非负整数。实际上，它是一个索引值，指向内核为每一个进程所维护的该进程打开文件的记录表。当程序打开一个现有文件或者创建一个新文件时，内核向进程返回一个文件描述符。在程序设计中，一些涉及底层的程序编写往往会围绕着文件描述符展开。但是文件描述符这一概念往往只适用于UNIX、Linux这样的操作系统。</p><p> </p><p><strong>缓存 I/O</strong></p><p> 缓存 I/O 又被称作标准 I/O，大多数文件系统的默认 I/O 操作都是缓存 I/O。在 Linux 的缓存 I/O 机制中，操作系统会将 I/O 的数据缓存在文件系统的页缓存（ page cache ）中，也就是说，数据会先被拷贝到操作系统内核的缓冲区中，然后才会从操作系统内核的缓冲区拷贝到应用程序的地址空间。</p><p>缓存 I/O 的缺点：<br> 数据在传输过程中需要在应用程序地址空间和内核进行多次数据拷贝操作，这些数据拷贝操作所带来的 CPU 以及内存开销是非常大的。</p><h2 id="IO模式"><a href="#IO模式" class="headerlink" title="IO模式"></a>IO模式</h2><p>对于一次IO访问（以read举例），数据会先被拷贝到操作系统内核的缓冲区中，然后才会从操作系统内核的缓冲区拷贝到应用程序的地址空间。所以说，当一个read操作发生时，它会经历两个阶段：<br> 1. 等待数据准备 (Waiting for the data to be ready)<br> 2. 将数据从内核拷贝到进程中 (Copying the data from the kernel to the process)</p><p><img src="https://imgconvert.csdnimg.cn/aHR0cHM6Ly9pbWFnZXMyMDE1LmNuYmxvZ3MuY29tL2Jsb2cvMTA1MDM5My8yMDE3MDQvMTA1MDM5My0yMDE3MDQxNTIxMTkxNzg0NS04NzI2MjYyNzEucG5n?x-oss-process=image/format,png"></p><p><em>详细：<a href="http://blog.csdn.net/haiross/article/details/39078853">http://blog.csdn.net/haiross/article/details/39078853</a></em></p><p> </p><p>正式因为这两个阶段，linux系统产生了下面五种网络模式的方案。<br> - 阻塞 I/O（blocking IO）<br> - 非阻塞 I/O（nonblocking IO）<br> - I/O 多路复用（ IO multiplexing）<br> - 信号驱动 I/O（ signal driven IO）<br> - 异步 I/O（asynchronous IO）</p><p>注：由于signal driven IO在实际中并不常用，所以我这只提及剩下的四种IO Model。</p><p> </p><p><strong>阻塞 I/O（blocking IO）</strong></p><p>阻塞I/O模型是最广泛的模型，在linux中，默认情况下所有的socket都是blocking，一个典型的读操作流程大概是这样：</p><p><img src="https://imgconvert.csdnimg.cn/aHR0cHM6Ly9pbWFnZXMyMDE1LmNuYmxvZ3MuY29tL2Jsb2cvMTA1MDM5My8yMDE3MDMvMTA1MDM5My0yMDE3MDMwODE5MDAxNTI5Ny0xNDIwNDg3Nzc5LnBuZw?x-oss-process=image/format,png"></p><p> 当用户进程调用了recvfrom这个系统调用，kernel就开始了IO的第一个阶段：准备数据（对于网络IO来说，很多时候数据在一开始还没有到达。比如，还没有收到一个完整的UDP包。这个时候kernel就要等待足够的数据到来）。这个过程需要等待，也就是说数据被拷贝到操作系统内核的缓冲区中是需要一个过程的。而在用户进程这边，整个进程会被阻塞（当然，是进程自己选择的阻塞）。当kernel一直等到数据准备好了，它就会将数据从kernel中拷贝到用户内存，然后kernel返回结果，用户进程才解除block的状态，重新运行起来。</p><p> ** 所以，blocking IO的特点就是在IO执行的两个阶段都被block了。**</p><p>** 注意：进程处于阻塞模式时，让出CPU，进入休眠状态。**</p><p> </p><p><strong>非阻塞 I/O（nonblocking IO）</strong></p><p>linux下，可以通过设置socket使其变为non-blocking。当对一个non-blocking socket执行读操作时，流程是这个样子：</p><p> </p><p><img src="https://imgconvert.csdnimg.cn/aHR0cHM6Ly9pbWFnZXMyMDE1LmNuYmxvZ3MuY29tL2Jsb2cvMTA1MDM5My8yMDE3MDMvMTA1MDM5My0yMDE3MDMwODE5MDIxNDg3NS0xNzA4MDgwMTk2LnBuZw?x-oss-process=image/format,png"></p><p> 当用户进程发出read操作时，如果kernel中的数据还没有准备好，那么它并不会block用户进程，而是立刻返回一个error。从用户进程角度讲 ，它发起一个read操作后，并不需要等待，而是马上就得到了一个结果。用户进程判断结果是一个error时，它就知道数据还没有准备好，于是它可以再次发送read操作。一旦kernel中的数据准备好了，并且又再次收到了用户进程的system call，那么它马上就将数据拷贝到了用户内存，然后返回。</p><p> <strong>所以，nonblocking IO的特点是用户进程需要不断的主动询问kernel数据好了没有。</strong></p><p>阻塞IO与非阻塞IO的性能区别</p><p> 在阻塞模式下，若从网络流中读取不到指定大小的数据量，阻塞IO就在那里阻塞着。比如，已知后面会有10个字节的数据发过来，但是我现在只收到8个字节，那么当前线程就在那傻傻地等到下一个字节的到来，对，就在那等着，啥事也不做，直到把这10个字节读取完，这才将阻塞放开通行。</p><p> 在非阻塞模式下，若从网络流中读取不到指定大小的数据量，非阻塞IO就立即通行。比如，已知后面会有10个字节的数据发过来，但是我现在只收到8个字节，那么当前线程就读取这8个字节的数据，读完后就立即返回，等另外两个字节再来的时候再去读取。</p><p> 从上面可以看出，阻塞IO在性能方面是很低下的，如果要使用阻塞IO完成一个Web服务器的话，那么对于每一个请求都必须启用一个线程进行处理。而使用非阻塞IO的话，一到两个线程基本上就够了，因为线程不会产生阻塞，好比一下接收A请求的数据，另一下接收B请求的数据，等等，就是不停地东奔西跑，直接到把数据接收完了。</p><p><strong>阻塞IO和非阻塞IO的区别就在于：应用程序的调用是否立即返回！</strong></p><p><strong><strong>注意；非阻塞模式的使用并不普遍，因为非阻塞模式会浪费大量的CPU资源。</strong></strong></p><p> </p><p><strong>I/O 多路复用（ IO multiplexing）</strong></p><p> 多路复用的本质是同步非阻塞I/O，多路复用的优势并不是单个连接处理的更快，而是在于能处理更多的连接。</p><p> I/O编程过程中，需要同时处理多个客户端接入请求时，可以利用多线程或者I/O多路复用技术进行处理。 I/O多路复用技术通过把多个I/O的阻塞复用到同一个select阻塞上，一个进程监视多个描述符，一旦某个描述符就位， 能够通知程序进行读写操作。因为多路复用本质上是同步I/O，都需要应用程序在读写事件就绪后自己负责读写。 与传统的多线程/多进程模型比，I/O多路复用的最大优势是系统开销小，系统不需要创建新的额外进程或者线程。</p><ul><li>应用场景<ul><li>服务器需要同时处理多个处于监听状态或者多个连接状态的套接字</li><li>需要同时处理多种网络协议的套接字</li><li>一个服务器处理多个服务或协议</li></ul></li></ul><p>目前支持多路复用的系统调用有 <code>select</code> , <code>poll</code> , <code>epoll</code> 。</p><p> IO multiplexing就是我们说的select，poll，epoll，有些地方也称这种IO方式为event driven IO。select/epoll的好处就在于单个process就可以同时处理多个网络连接的IO。它的基本原理就是select，poll，epoll这个function会不断的轮询所负责的所有socket，当某个socket有数据到达了，就通知用户进程。</p><p><img src="https://imgconvert.csdnimg.cn/aHR0cHM6Ly9pbWFnZXMyMDE1LmNuYmxvZ3MuY29tL2Jsb2cvMTA1MDM5My8yMDE3MDMvMTA1MDM5My0yMDE3MDMwODE5MDMzMTQ2OS0xMzk0OTM5NzY2LnBuZw?x-oss-process=image/format,png"></p><p><img src="https://imgconvert.csdnimg.cn/aHR0cHM6Ly9pbWFnZXMyMDE1LmNuYmxvZ3MuY29tL2Jsb2cvMTA1MDM5My8yMDE3MDMvMTA1MDM5My0yMDE3MDMwODE5MDMzMTQ2OS0xMzk0OTM5NzY2LnBuZw?x-oss-process=image/format,png"></p><p>**<code> 当用户进程调用了select，那么整个进程会被block</code>**，而同时，kernel会“监视”所有select负责的socket，当任何一个socket中的数据准备好了，select就会返回。这个时候用户进程再调用read操作，将数据从kernel拷贝到用户进程。</p><p> <strong>所以，I/O 多路复用的特点是通过一种机制一个进程能同时等待多个文件描述符，而这些文件描述符（套接字描述符）其中的任意一个进入读就绪状态，select()函数就可以返回。</strong></p><p> 这个图和blocking IO的图其实并没有太大的不同，事实上，还更差一些。因为这里需要使用两个system call (select 和 recvfrom)，而blocking IO只调用了一个system call (recvfrom)。但是，用select的优势在于它可以同时处理多个connection。</p><p> 所以，如果处理的连接数不是很高的话，使用select/epoll的web server不一定比使用multi-threading + blocking IO的web server性能更好，可能延迟还更大。select/epoll的优势并不是对于单个连接能处理得更快，而是在于能处理更多的连接。）</p><p> 在IO multiplexing Model中，实际中，对于每一个socket，一般都设置成为non-blocking，但是，如上图所示，整个用户的process其实是一直被block的。只不过process是被select这个函数block，而不是被socket IO给block。</p><p> </p><p><strong>异步 I/O（asynchronous IO）</strong></p><p> 相比于IO多路复用模型，异步IO并不十分常用，不少高性能并发服务程序使用IO多路复用模型+多线程任务处理的<a href="http://lib.csdn.net/base/architecture">架构</a>基本可以满足需求</p><p>linux下的asynchronous IO其实用得很少。先看一下它的流程：</p><p><img src="https://imgconvert.csdnimg.cn/aHR0cHM6Ly9pbWFnZXMyMDE1LmNuYmxvZ3MuY29tL2Jsb2cvMTA1MDM5My8yMDE3MDMvMTA1MDM5My0yMDE3MDMwODE5MDUxOTczNC0yNjMyNDAxNDkucG5n?x-oss-process=image/format,png"></p><p> 用户进程发起read操作之后，立刻就可以开始去做其它的事。而另一方面，从kernel的角度，当它受到一个asynchronous read之后，首先它会立刻返回，所以不会对用户进程产生任何block。然后，kernel会等待数据准备完成，然后将数据拷贝到用户内存，当这一切都完成之后，kernel会给用户进程发送一个signal，告诉它read操作完成了。</p><h2 id="总结："><a href="#总结：" class="headerlink" title="总结："></a>总结：</h2><p><em>浅析 I/O 模型及其设计模式<a href="http://blog.jobbole.com/104638/">http://blog.jobbole.com/104638/</a></em></p><p>blocking和non-blocking的区别</p><p>调用blocking IO会一直block住对应的进程直到操作完成，而non-blocking IO在kernel还准备数据的情况下会立刻返回。</p><p>synchronous IO和asynchronous IO的区别</p><p><strong>同步IO和异步IO的区别就在于：数据访问的时候进程是否阻塞！</strong></p><p>在说明synchronous IO和asynchronous IO的区别之前，需要先给出两者的定义。POSIX的定义是这样子的：<br> - A synchronous I/O operation causes the requesting process to be blocked until that I/O operation completes;<br> - An asynchronous I/O operation does not cause the requesting process to be blocked;</p><p> 两者的区别就在于synchronous IO做”IO operation”的时候会将process阻塞。按照这个定义，之前所述的blocking IO，non-blocking IO，IO multiplexing都属于synchronous IO。</p><p> 有人会说，non-blocking IO并没有被block啊。这里有个非常“狡猾”的地方，定义中所指的”IO operation”是指真实的IO操作，就是例子中的recvfrom这个system call。non-blocking IO在执行recvfrom这个system call的时候，如果kernel的数据没有准备好，这时候不会block进程。但是，当kernel中数据准备好的时候，recvfrom会将数据从kernel拷贝到用户内存中，这个时候进程是被block了，在这段时间内，进程是被block的。</p><p> 而asynchronous IO则不一样，当进程发起IO 操作之后，就直接返回再也不理睬了，直到kernel发送一个信号，告诉进程说IO完成。在这整个过程中，进程完全没有被block。</p><p> </p><p>各个IO Model的比较如图所示：</p><p><img src="https://imgconvert.csdnimg.cn/aHR0cHM6Ly9pbWFnZXMyMDE1LmNuYmxvZ3MuY29tL2Jsb2cvMTA1MDM5My8yMDE3MDMvMTA1MDM5My0yMDE3MDMwODE5MDcyMjg5MS0xMjc5Nzk2NzE5LnBuZw?x-oss-process=image/format,png"></p><p> 通过上面的图片，可以发现non-blocking IO和asynchronous IO的区别还是很明显的。在non-blocking IO中，虽然进程大部分时间都不会被block，但是它仍然要求进程去主动的check，并且当数据准备完成以后，也需要进程主动的再次调用recvfrom来将数据拷贝到用户内存。而asynchronous IO则完全不同。它就像是用户进程将整个IO操作交给了他人（kernel）完成，然后他人做完后发信号通知。在此期间，用户进程不需要去检查IO操作的状态，也不需要主动的去拷贝数据。</p><h2 id="IO多路复用之select、poll、epoll"><a href="#IO多路复用之select、poll、epoll" class="headerlink" title="IO多路复用之select、poll、epoll"></a>IO多路复用之select、poll、epoll</h2><p> select，poll，epoll都是IO多路复用的机制。I/O多路复用就是通过一种机制，一个进程可以监视多个描述符，一旦某个描述符就绪（一般是读就绪或者写就绪），能够通知程序进行相应的读写操作。但select，poll，epoll本质上都是同步I/O，因为他们都需要在读写事件就绪后自己负责进行读写，也就是说这个读写过程是阻塞的，而异步I/O则无需自己负责进行读写，异步I/O的实现会负责把数据从内核拷贝到用户空间。</p><p><strong>select</strong></p><figure class="highlight html"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs html">select(rlist, wlist, xlist, timeout=None)<br></code></pre></td></tr></table></figure><p> select 函数监视的文件描述符分3类，分别是writefds、readfds、和exceptfds。调用后select函数会阻塞，直到有描述副就绪（有数据 可读、可写、或者有except），或者超时（timeout指定等待时间，如果立即返回设为null即可），函数返回。当select函数返回后，可以 通过遍历fdset，来找到就绪的描述符。</p><p> select目前几乎在所有的平台上支持，其良好跨平台支持也是它的一个优点。select的一 个缺点在于单个进程能够监视的文件描述符的数量存在最大限制，在Linux上一般为1024，可以通过修改宏定义甚至重新编译内核的方式提升这一限制，但 是这样也会造成效率的降低。</p><figure class="highlight html"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><code class="hljs html"> 使用select库的一般步骤：创建所关注事件的描述集合。对于一个描述符，可以关注其上面的读事件、写事件以及异常发生事件，所以要创建三类事件描述符集合，分别用来收集读事件的描述符、写事件的描述符和异常事件的描述符。<br> 其次，调用底层提供的select（）函数，等待事件的发生。select的阻塞与是否设置非阻塞的IO是没有关系的。<br> 然后，轮询所有事件描述符集合中的每一个事件描述符，检查是否有响应的时间发生，如果有，则进行处理。<br> nginx服务器在编译过程中如果没有为其指定其他高性能事件驱动模型库，它将自动编译该库。<br> 可以使用--with-select_module和--without-select_module两个参数，强制nginx是否编译该库。<br><br>实例：<br></code></pre></td></tr></table></figure><figure class="highlight html"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs html">利用select通过单进程实现同时处理多个非阻塞的socket连接:<br></code></pre></td></tr></table></figure><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-comment"># _*_coding:utf-8_*_</span><br><br><span class="hljs-keyword">import</span> select<br><span class="hljs-keyword">import</span> socket<br><span class="hljs-keyword">import</span> sys<br><span class="hljs-keyword">import</span> queue<br><br><span class="hljs-comment"># Create a TCP/IP socket</span><br>server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)<br>server.setblocking(<span class="hljs-literal">False</span>)<br><br><span class="hljs-comment"># Bind the socket to the port</span><br>server_address = (<span class="hljs-string">'localhost'</span>, <span class="hljs-number">6666</span>)<br><span class="hljs-built_in">print</span>(sys.stderr, <span class="hljs-string">'starting up on %s port %s'</span> % server_address)<br>server.bind(server_address)<br><br><span class="hljs-comment"># Listen for incoming connections</span><br>server.listen(<span class="hljs-number">5</span>)<br><br><span class="hljs-comment"># Sockets from which we expect to read</span><br>inputs = [server]<br><br><span class="hljs-comment"># Sockets to which we expect to write</span><br>outputs = []<br><br>message_queues = {}<br><span class="hljs-keyword">while</span> inputs:<br> <span class="hljs-comment"># Wait for at least one of the sockets to be ready for processing</span><br> <span class="hljs-built_in">print</span>(<span class="hljs-string">'\nwaiting for the next event'</span>)<br> readable, writable, exceptional = select.select(inputs, outputs, inputs)<br> <span class="hljs-comment"># Handle inputs</span><br> <span class="hljs-keyword">for</span> s <span class="hljs-keyword">in</span> readable:<br> <span class="hljs-keyword">if</span> s <span class="hljs-keyword">is</span> server:<br> <span class="hljs-comment"># A "readable" server socket is ready to accept a connection</span><br> connection, client_address = s.accept()<br> <span class="hljs-built_in">print</span>(<span class="hljs-string">'new connection from'</span>, client_address)<br> connection.setblocking(<span class="hljs-literal">False</span>)<br> inputs.append(connection)<br><br> <span class="hljs-comment"># Give the connection a queue for data we want to send</span><br> message_queues[connection] = queue.Queue()<br> <span class="hljs-keyword">else</span>:<br> data = s.recv(<span class="hljs-number">1024</span>)<br> <span class="hljs-keyword">if</span> data:<br> <span class="hljs-comment"># A readable client socket has data</span><br> <span class="hljs-built_in">print</span>(sys.stderr, <span class="hljs-string">'received "%s" from %s'</span> % (data, s.getpeername()))<br> message_queues[s].put(data)<br> <span class="hljs-comment"># Add output channel for response</span><br> <span class="hljs-keyword">if</span> s <span class="hljs-keyword">not</span> <span class="hljs-keyword">in</span> outputs:<br> outputs.append(s)<br> <span class="hljs-keyword">else</span>:<br> <span class="hljs-comment"># Interpret empty result as closed connection</span><br> <span class="hljs-built_in">print</span>(<span class="hljs-string">'closing'</span>, client_address, <span class="hljs-string">'after reading no data'</span>)<br> <span class="hljs-comment"># Stop listening for input on the connection</span><br> <span class="hljs-keyword">if</span> s <span class="hljs-keyword">in</span> outputs:<br> outputs.remove(s) <span class="hljs-comment"># 既然客户端都断开了，我就不用再给它返回数据了，所以这时候如果这个客户端的连接对象还在outputs列表中，就把它删掉</span><br> inputs.remove(s) <span class="hljs-comment"># inputs中也删除掉</span><br> s.close() <span class="hljs-comment"># 把这个连接关闭掉</span><br><br> <span class="hljs-comment"># Remove message queue</span><br> <span class="hljs-keyword">del</span> message_queues[s]<br> <span class="hljs-comment"># Handle outputs</span><br> <span class="hljs-keyword">for</span> s <span class="hljs-keyword">in</span> writable:<br> <span class="hljs-keyword">try</span>:<br> next_msg = message_queues[s].get_nowait()<br> <span class="hljs-keyword">except</span> queue.Empty:<br> <span class="hljs-comment"># No messages waiting so stop checking for writability.</span><br> <span class="hljs-built_in">print</span>(<span class="hljs-string">'output queue for'</span>, s.getpeername(), <span class="hljs-string">'is empty'</span>)<br> outputs.remove(s)<br> <span class="hljs-keyword">else</span>:<br> <span class="hljs-built_in">print</span>(<span class="hljs-string">'sending "%s" to %s'</span> % (next_msg, s.getpeername()))<br> s.send(next_msg)<br> <span class="hljs-comment"># Handle "exceptional conditions"</span><br> <span class="hljs-keyword">for</span> s <span class="hljs-keyword">in</span> exceptional:<br> <span class="hljs-built_in">print</span>(<span class="hljs-string">'handling exceptional condition for'</span>, s.getpeername())<br> <span class="hljs-comment"># Stop listening for input on the connection</span><br> inputs.remove(s)<br> <span class="hljs-keyword">if</span> s <span class="hljs-keyword">in</span> outputs:<br> outputs.remove(s)<br> s.close()<br><br> <span class="hljs-comment"># Remove message queue</span><br> <span class="hljs-keyword">del</span> message_queues[s]<br></code></pre></td></tr></table></figure><p> </p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-comment">#!/usr/bin/env/ python</span><br><span class="hljs-comment"># -*-coding:utf-8 -*-</span><br><span class="hljs-keyword">import</span> socket,select,queue<br><span class="hljs-keyword">import</span> sys<br><br>server=socket.socket()<br>server.bind((<span class="hljs-string">'localhost'</span>,<span class="hljs-number">8080</span>))<br><br>server.listen(<span class="hljs-number">1000</span>)<br><br>server.setblocking(<span class="hljs-literal">False</span>)<span class="hljs-comment">#设置为非阻塞模式，socket必须在非阻塞情况下才能实现IO多路复用。</span><br><br><span class="hljs-comment">#需要监听的可读对象</span><br>inputs=[server,]<span class="hljs-comment">#存放selcct要监测的链接,一开始监测自己，有活动就表示有链接进来</span><br>outputs=[]<span class="hljs-comment">#这里存放的是内核返回的活跃的客户端连接，就是服务器需给send data的客户端连接</span><br><br>message_queues={}<br><span class="hljs-keyword">while</span> inputs:<br> <span class="hljs-comment"># Wait for at least one of the sockets to be ready for processing</span><br> <span class="hljs-built_in">print</span>(<span class="hljs-string">'\nwaiting for the next event'</span>)<br> readable,writeable,exceptionable=select.select(inputs,outputs,inputs) <span class="hljs-comment">#此处会被select模块阻塞，只有当监听的三个参数发生变化时，select才会返,</span><br><br> <span class="hljs-comment"># print(readable,writeable,exceptionable)</span><br> <span class="hljs-keyword">for</span> r <span class="hljs-keyword">in</span> readable:<br> <span class="hljs-keyword">if</span> r <span class="hljs-keyword">is</span> server:<span class="hljs-comment">#代表来了一个新连接</span><br> conn,addr=server.accept()<br> <span class="hljs-built_in">print</span>(<span class="hljs-string">"来了一个新连接"</span>,addr)<br> inputs.append(conn)<span class="hljs-comment">#因为这个新建立的连接还没发数据过来，现在就接收的话会报错，所以要想客户端发数据过来时server端能知道，就要select监测这个连接。</span><br> message_queues[conn]=queue.Queue()<span class="hljs-comment">#接收到客户端的数据后,不立刻返回 ,暂存在队列里,以后发送</span><br> <span class="hljs-keyword">else</span>:<br> data = r.recv(<span class="hljs-number">1024</span>) <span class="hljs-comment"># 注意这里是r，而不是conn，多个连接的情况</span><br> <span class="hljs-keyword">if</span> data:<br> <span class="hljs-comment"># A readable client socket has data</span><br> <span class="hljs-built_in">print</span>(sys.stderr, <span class="hljs-string">'received "%s" from %s'</span> % (data, r.getpeername()))<br> <span class="hljs-comment"># r.send(data) # 先不发，放到一个对列里，之后再发</span><br> message_queues[r].put(data)<span class="hljs-comment"># 往里面放数据</span><br> <span class="hljs-comment"># Add output channel for response</span><br> <span class="hljs-keyword">if</span> r <span class="hljs-keyword">not</span> <span class="hljs-keyword">in</span> outputs:<br> outputs.append(r) <span class="hljs-comment">#放入返回的连接队列里</span><br> <span class="hljs-keyword">else</span>:<br> <span class="hljs-comment"># Interpret empty result as closed connection</span><br> <span class="hljs-built_in">print</span>(<span class="hljs-string">'closing'</span>, addr, <span class="hljs-string">'after reading no data'</span>)<br> <span class="hljs-comment"># Stop listening for input on the connection</span><br> <span class="hljs-keyword">if</span> r <span class="hljs-keyword">in</span> outputs:<br> outputs.remove(r) <span class="hljs-comment"># 既然客户端都断开了，我就不用再给它返回数据了，所以这时候如果这个客户端的连接对象还在outputs列表中，就把它删掉</span><br> inputs.remove(r) <span class="hljs-comment"># inputs中也删除掉</span><br> r.close() <span class="hljs-comment"># 把这个连接关闭掉</span><br><br> <span class="hljs-comment"># Remove message queue</span><br> <span class="hljs-keyword">del</span> message_queues[r]<br><br> <span class="hljs-keyword">for</span> w <span class="hljs-keyword">in</span> writeable: <span class="hljs-comment"># 要返回给客户端的连接列表</span><br> data_to_client = message_queues[w].get() <span class="hljs-comment"># 在字典里取数据</span><br> w.send(data_to_client) <span class="hljs-comment"># 返回给客户端</span><br> outputs.remove(w) <span class="hljs-comment"># 删除这个数据，确保下次循环的时候不返回这个已经处理完的连接了。</span><br><br> <span class="hljs-keyword">for</span> e <span class="hljs-keyword">in</span> exceptionable: <span class="hljs-comment"># 如果连接断开，删除连接相关数据</span><br> <span class="hljs-keyword">if</span> e <span class="hljs-keyword">in</span> outputs:<br> outputs.remove(e)<br> inputs.remove(e)<br> <span class="hljs-keyword">del</span> message_queues[e]<br></code></pre></td></tr></table></figure><p><strong>poll</strong></p><figure class="highlight html"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs html"> poll库，作为linux平台上的基本事件驱动模型，Windows平台不支持poll库。<br></code></pre></td></tr></table></figure><figure class="highlight html"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><code class="hljs html"> 使用poll库的一般过程是：与select的基本工作方式是相同的，都是先创建一个关注事件的描述符集合，再去等待这些事件的发生，然后在轮询描述符集合，检查有没有事件发生，如果有，就进行处理。<br> 与select的主要区别是select需要为读事件、写事件、异常事件分别创建一个描述符的集合，因此在轮询的时候，需要分别轮询这三个集合。而poll库只需创建一个集合，在每个描述符对应的结构上分别设置读事件，写事件和异常事件，最后轮询的时候可以同时检查这三种事件是否发生。是select库优化的实现。<br>nginx服务器在编译过程中如果没有为其指定其他高性能事件驱动模型库，它将自动编译该库。可以使用--with-poll_module和--without-poll_module两个参数，强制nginx是否编译该库。<br></code></pre></td></tr></table></figure><figure class="highlight html"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs html">int poll (struct pollfd *fds, unsigned int nfds, int timeout);<br></code></pre></td></tr></table></figure><p> 不同与select使用三个位图来表示三个fdset的方式，poll使用一个 pollfd的指针实现。 </p><figure class="highlight html"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><code class="hljs html">struct pollfd {<br> int fd; /* file descriptor */<br> short events; /* requested events to watch */<br> short revents; /* returned events witnessed */<br>};<br></code></pre></td></tr></table></figure><p> pollfd结构包含了要监视的event和发生的event，不再使用select“参数-值”传递的方式。同时，pollfd并没有最大数量限制（但是数量过大后性能也是会下降）。 和select函数一样，poll返回后，需要轮询pollfd来获取就绪的描述符。</p><p> 从上面看，select和poll都需要在返回后，<code>通过遍历文件描述符来获取已经就绪的socket</code>。事实上，同时连接的大量客户端在一时刻可能只有很少的处于就绪状态，因此随着监视的描述符数量的增长，其效率也会线性下降。</p><p><strong>epoll</strong></p><figure class="highlight html"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><code class="hljs html"> epoll是在2.6内核中提出的，是之前的select和poll的增强版本。相对于select和poll来说，epoll更加灵活，没有描述符限制。epoll使用一个文件描述符管理多个描述符，将用户关系的文件描述符的事件存放到内核的一个事件表中，这样在用户空间和内核空间的copy只需一次。 <br> epoll库是Nginx服务器支持的高性能事件之一，它是公认的非常优秀的时间驱动模型，和poll和select有很大的不同，属于poll库的一个变种，他们的处理方式都是创建一个待处理事件列表，然后把这个事件列表发送给内核，返回的时候，再去轮询检查这个列表，以判断事件是否发生。如果这样的描述符在比较多的应用中，效率就显得低下了，epoll是描述符列表的管理交给内核负责，一旦某种事件发生，内核会把发生事件的描述符列表通知给进程，这样就避免了轮询整个描述符列表，epoll库得到事件列表，就开始进行事件处理了。<br> 注意：select和epoll最大的区别就是：select只是告诉你一定数目的流有事件了，至于哪个流有事件，还得你一个一个地去轮询，而 epoll会把发生的事件告诉你，通过发生的事件，就自然而然定位到哪个流了<br></code></pre></td></tr></table></figure><p>一 epoll操作过程</p><p>epoll操作过程需要三个接口，分别如下：</p><figure class="highlight html"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><code class="hljs html">int epoll_create(int size)；//创建一个epoll的句柄，size用来告诉内核这个监听的数目一共有多大<br>int epoll_ctl(int epfd, int op, int fd, struct epoll_event *event)；<br>int epoll_wait(int epfd, struct epoll_event * events, int maxevents, int timeout);<br></code></pre></td></tr></table></figure><p>1. int epoll_create(int size);<br> 创建一个epoll的句柄，size用来告诉内核这个监听的数目一共有多大，这个参数不同于select()中的第一个参数，给出最大监听的fd+1的值，<code>参数size并不是限制了epoll所能监听的描述符最大个数，只是对内核初始分配内部数据结构的一个建议</code>。<br>当创建好epoll句柄后，它就会占用一个fd值，在linux下如果查看/proc/进程id/fd/，是能够看到这个fd的，所以在使用完epoll后，必须调用close()关闭，否则可能导致fd被耗尽。</p><p>2. int epoll_ctl(int epfd, int op, int fd, struct epoll_event *event)；<br> 函数是对指定描述符fd执行op操作。<br> - epfd：是epoll_create()的返回值。<br> - op：表示op操作，用三个宏来表示：添加EPOLL_CTL_ADD，删除EPOLL_CTL_DEL，修改EPOLL_CTL_MOD。分别添加、删除和修改对fd的监听事件。<br> - fd：是需要监听的fd（文件描述符）<br> - epoll_event：是告诉内核需要监听什么事</p><p>3. int epoll_wait(int epfd, struct epoll_event * events, int maxevents, int timeout);<br> 等待epfd上的io事件，最多返回maxevents个事件。<br>参数events用来从内核得到事件的集合，maxevents告之内核这个events有多大，这个maxevents的值不能大于创建epoll_create()时的size，参数timeout是超时时间（毫秒，0会立即返回，-1将不确定，也有说法说是永久阻塞）。该函数返回需要处理的事件数目，如返回0表示已超时。</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br><span class="line">97</span><br><span class="line">98</span><br><span class="line">99</span><br><span class="line">100</span><br><span class="line">101</span><br><span class="line">102</span><br><span class="line">103</span><br><span class="line">104</span><br><span class="line">105</span><br><span class="line">106</span><br><span class="line">107</span><br><span class="line">108</span><br><span class="line">109</span><br><span class="line">110</span><br><span class="line">111</span><br><span class="line">112</span><br><span class="line">113</span><br><span class="line">114</span><br><span class="line">115</span><br><span class="line">116</span><br><span class="line">117</span><br><span class="line">118</span><br><span class="line">119</span><br><span class="line">120</span><br><span class="line">121</span><br><span class="line">122</span><br><span class="line">123</span><br><span class="line">124</span><br><span class="line">125</span><br><span class="line">126</span><br><span class="line">127</span><br><span class="line">128</span><br><span class="line">129</span><br><span class="line">130</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-keyword">import</span> socket, logging<br><span class="hljs-keyword">import</span> select, errno<br><br>logger = logging.getLogger(<span class="hljs-string">"network-server"</span>)<br><br><span class="hljs-keyword">def</span> <span class="hljs-title function_">InitLog</span>():<br> logger.setLevel(logging.DEBUG)<br><br> fh = logging.FileHandler(<span class="hljs-string">"network-server.log"</span>)<br> fh.setLevel(logging.DEBUG)<br> ch = logging.StreamHandler()<br> ch.setLevel(logging.ERROR)<br><br> formatter = logging.Formatter(<span class="hljs-string">"%(asctime)s - %(name)s - %(levelname)s - %(message)s"</span>)<br> ch.setFormatter(formatter)<br> fh.setFormatter(formatter)<br><br> logger.addHandler(fh)<br> logger.addHandler(ch)<br><br><br><span class="hljs-keyword">if</span> __name__ == <span class="hljs-string">"__main__"</span>:<br> InitLog()<br><br> <span class="hljs-keyword">try</span>:<br> <span class="hljs-comment"># 创建 TCP socket 作为监听 socket</span><br> listen_fd = socket.socket(socket.AF_INET, socket.SOCK_STREAM, <span class="hljs-number">0</span>)<br> <span class="hljs-keyword">except</span> socket.error <span class="hljs-keyword">as</span> msg:<br> logger.error(<span class="hljs-string">"create socket failed"</span>)<br><br> <span class="hljs-keyword">try</span>:<br> <span class="hljs-comment"># 设置 SO_REUSEADDR 选项</span><br> listen_fd.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, <span class="hljs-number">1</span>)<br> <span class="hljs-keyword">except</span> socket.error <span class="hljs-keyword">as</span> msg:<br> logger.error(<span class="hljs-string">"setsocketopt SO_REUSEADDR failed"</span>)<br><br> <span class="hljs-keyword">try</span>:<br> <span class="hljs-comment"># 进行 bind -- 此处未指定 ip 地址，即 bind 了全部网卡 ip 上</span><br> listen_fd.bind((<span class="hljs-string">''</span>, <span class="hljs-number">2003</span>))<br> <span class="hljs-keyword">except</span> socket.error <span class="hljs-keyword">as</span> msg:<br> logger.error(<span class="hljs-string">"bind failed"</span>)<br><br> <span class="hljs-keyword">try</span>:<br> <span class="hljs-comment"># 设置 listen 的 backlog 数</span><br> listen_fd.listen(<span class="hljs-number">10</span>)<br> <span class="hljs-keyword">except</span> socket.error <span class="hljs-keyword">as</span> msg:<br> logger.error(msg)<br><br> <span class="hljs-keyword">try</span>:<br> <span class="hljs-comment"># 创建 epoll 句柄</span><br> epoll_fd = select.epoll()<br> <span class="hljs-comment"># 向 epoll 句柄中注册 监听 socket 的 可读 事件</span><br> epoll_fd.register(listen_fd.fileno(), select.EPOLLIN)<br> <span class="hljs-keyword">except</span> select.error <span class="hljs-keyword">as</span> msg:<br> logger.error(msg)<br><br> connections = {}<br> addresses = {}<br> datalist = {}<br> <span class="hljs-keyword">while</span> <span class="hljs-literal">True</span>:<br> <span class="hljs-comment"># epoll 进行 fd 扫描的地方 -- 未指定超时时间则为阻塞等待</span><br> epoll_list = epoll_fd.poll()<br><br> <span class="hljs-keyword">for</span> fd, events <span class="hljs-keyword">in</span> epoll_list:<br> <span class="hljs-comment"># 若为监听 fd 被激活</span><br> <span class="hljs-keyword">if</span> fd == listen_fd.fileno():<br> <span class="hljs-comment"># 进行 accept -- 获得连接上来 client 的 ip 和 port，以及 socket 句柄</span><br> conn, addr = listen_fd.accept()<br> logger.debug(<span class="hljs-string">"accept connection from %s, %d, fd = %d"</span> % (addr[<span class="hljs-number">0</span>], addr[<span class="hljs-number">1</span>], conn.fileno()))<br> <span class="hljs-comment"># 将连接 socket 设置为 非阻塞</span><br> conn.setblocking(<span class="hljs-number">0</span>)<br> <span class="hljs-comment"># 向 epoll 句柄中注册 连接 socket 的 可读 事件</span><br> epoll_fd.register(conn.fileno(), select.EPOLLIN | select.EPOLLET)<br> <span class="hljs-comment"># 将 conn 和 addr 信息分别保存起来</span><br> connections[conn.fileno()] = conn<br> addresses[conn.fileno()] = addr<br> <span class="hljs-keyword">elif</span> select.EPOLLIN & events:<br> <span class="hljs-comment"># 有 可读 事件激活</span><br> datas = <span class="hljs-string">''</span><br> <span class="hljs-keyword">while</span> <span class="hljs-literal">True</span>:<br> <span class="hljs-keyword">try</span>:<br> <span class="hljs-comment"># 从激活 fd 上 recv 10 字节数据</span><br> data = connections[fd].recv(<span class="hljs-number">10</span>)<br> <span class="hljs-comment"># 若当前没有接收到数据，并且之前的累计数据也没有</span><br> <span class="hljs-keyword">if</span> <span class="hljs-keyword">not</span> data <span class="hljs-keyword">and</span> <span class="hljs-keyword">not</span> datas:<br> <span class="hljs-comment"># 从 epoll 句柄中移除该 连接 fd</span><br> epoll_fd.unregister(fd)<br> <span class="hljs-comment"># server 侧主动关闭该 连接 fd</span><br> connections[fd].close()<br> logger.debug(<span class="hljs-string">"%s, %d closed"</span> % (addresses[fd][<span class="hljs-number">0</span>], addresses[fd][<span class="hljs-number">1</span>]))<br> <span class="hljs-keyword">break</span><br> <span class="hljs-keyword">else</span>:<br> <span class="hljs-comment"># 将接收到的数据拼接保存在 datas 中</span><br> datas += data<br> <span class="hljs-keyword">except</span> socket.error <span class="hljs-keyword">as</span> msg:<br> <span class="hljs-comment"># 在 非阻塞 socket 上进行 recv 需要处理 读穿 的情况</span><br> <span class="hljs-comment"># 这里实际上是利用 读穿 出 异常 的方式跳到这里进行后续处理</span><br> <span class="hljs-keyword">if</span> msg.errno == errno.EAGAIN:<br> logger.debug(<span class="hljs-string">"%s receive %s"</span> % (fd, datas))<br> <span class="hljs-comment"># 将已接收数据保存起来</span><br> datalist[fd] = datas<br> <span class="hljs-comment"># 更新 epoll 句柄中连接d 注册事件为 可写</span><br> epoll_fd.modify(fd, select.EPOLLET | select.EPOLLOUT)<br> <span class="hljs-keyword">break</span><br> <span class="hljs-keyword">else</span>:<br> <span class="hljs-comment"># 出错处理</span><br> epoll_fd.unregister(fd)<br> connections[fd].close()<br> logger.error(msg)<br> <span class="hljs-keyword">break</span><br> <span class="hljs-keyword">elif</span> select.EPOLLHUP & events:<br> <span class="hljs-comment"># 有 HUP 事件激活</span><br> epoll_fd.unregister(fd)<br> connections[fd].close()<br> logger.debug(<span class="hljs-string">"%s, %d closed"</span> % (addresses[fd][<span class="hljs-number">0</span>], addresses[fd][<span class="hljs-number">1</span>]))<br> <span class="hljs-keyword">elif</span> select.EPOLLOUT & events:<br> <span class="hljs-comment"># 有 可写 事件激活</span><br> sendLen = <span class="hljs-number">0</span><br> <span class="hljs-comment"># 通过 while 循环确保将 buf 中的数据全部发送出去</span><br> <span class="hljs-keyword">while</span> <span class="hljs-literal">True</span>:<br> <span class="hljs-comment"># 将之前收到的数据发回 client -- 通过 sendLen 来控制发送位置</span><br> sendLen += connections[fd].send(datalist[fd][sendLen:])<br> <span class="hljs-comment"># 在全部发送完毕后退出 while 循环</span><br> <span class="hljs-keyword">if</span> sendLen == <span class="hljs-built_in">len</span>(datalist[fd]):<br> <span class="hljs-keyword">break</span><br> <span class="hljs-comment"># 更新 epoll 句柄中连接 fd 注册事件为 可读</span><br> epoll_fd.modify(fd, select.EPOLLIN | select.EPOLLET)<br> <span class="hljs-keyword">else</span>:<br> <span class="hljs-comment"># 其他 epoll 事件不进行处理</span><br> <span class="hljs-keyword">continue</span><br></code></pre></td></tr></table></figure><hr><h3 id="select-amp-poll-amp-epoll比较"><a href="#select-amp-poll-amp-epoll比较" class="headerlink" title="select & poll & epoll比较"></a>select & poll & epoll比较</h3><ul><li>每次调用 <code>select</code> 都需要把所有要监听的文件描述符拷贝到内核空间一次，fd很大时开销会很大。 <code>epoll</code> 会在epoll_ctl()中注册，只需要将所有的fd拷贝到内核事件表一次，不用再每次epoll_wait()时重复拷贝</li><li>每次 <code>select</code> 需要在内核中遍历所有监听的fd，直到设备就绪； <code>epoll</code> 通过 <code>epoll_ctl</code> 注册回调函数，也需要不断调用 <code>epoll_wait</code> 轮询就绪链表，当fd或者事件就绪时，会调用回调函数，将就绪结果加入到就绪链表。</li><li><code>select</code> 能监听的文件描述符数量有限，默认是1024； <code>epoll</code> 能支持的fd数量是最大可以打开文件的数目，具体数目可以在/proc/sys/fs/file-max查看</li><li><code>select</code> , <code>poll</code> 在函数返回后需要查看所有监听的fd，看哪些就绪，而epoll只返回就绪的描述符，所以应用程序只需要就绪fd的命中率是百分百。</li></ul><p>表面上看epoll的性能最好，但是在连接数少并且链接都十分活跃的情况下，select和poll的性能可能比epoll好，毕竟epoll的通知机制需要很多函数回调。</p><p>select效率低是一位每次都需要轮询，但效率低也是相对的，也可通过良好的设计改善</p><p>参考：<a href="https://www.cnblogs.com/freely/p/6522432.html">https://www.cnblogs.com/freely/p/6522432.html</a></p>]]></content>
<categories>
<category>python加油鸭</category>
</categories>
<tags>
<tag>epoll python linux</tag>
</tags>
</entry>
<entry>
<title>Ubuntu16.04 关闭防火墙</title>
<link href="/2020/04/17/Ubuntu16.04%20%E5%85%B3%E9%97%AD%E9%98%B2%E7%81%AB%E5%A2%99/"/>
<url>/2020/04/17/Ubuntu16.04%20%E5%85%B3%E9%97%AD%E9%98%B2%E7%81%AB%E5%A2%99/</url>
<content type="html"><![CDATA[<span id="more"></span><p>Ubuntu16.04 关闭防火墙](<a href="https://blog.csdn.net/weixin_42474540">https://blog.csdn.net/weixin_42474540</a>)</p><p>一、关闭防火墙</p><p>1. 先查看防火墙状态</p><table border="0" cellpadding="0" cellspacing="0"><tbody><tr><td><p>1</p></td><td><p><code>systemctl status firewalld</code></p></td></tr></tbody></table><figure class="highlight awk"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre></td><td class="code"><pre><code class="hljs awk">firewalld.service - firewalld - dynamic firewall daemon<br><br> Loaded: loaded (<span class="hljs-regexp">/usr/</span>lib<span class="hljs-regexp">/systemd/</span>system/firewalld.service; enabled)<br><br> Active: active (running) since<br><br> Main PID: <span class="hljs-number">979</span> (firewalld)<br><br> CGroup: <span class="hljs-regexp">/system.slice/</span>firewalld.service<br><br> └─<span class="hljs-number">979</span> <span class="hljs-regexp">/usr/</span>bin<span class="hljs-regexp">/python -Es /u</span>sr<span class="hljs-regexp">/sbin/</span>firewalld --nofork --nopid<br><br><span class="hljs-number">5</span>月 <span class="hljs-number">25</span> <span class="hljs-number">22</span>:<span class="hljs-number">53</span>:<span class="hljs-number">54</span> localhost.localdomain systemd[<span class="hljs-number">1</span>]: Started firewalld - dynami...<br><br>Hint: Some lines were ellipsized, use -l to show <span class="hljs-keyword">in</span> full.<br></code></pre></td></tr></table></figure><p>2. 关闭防火墙</p><p> </p><table border="0" cellpadding="0" cellspacing="0"><tbody><tr><td><p>1</p></td><td><p><code>systemctl stop firewalld</code></p></td></tr></tbody></table><table border="0" cellpadding="0" cellspacing="0"><tbody><tr><td><p>1</p></td><td><p><code>systemctl status firewalld</code></p></td></tr></tbody></table><p>3. 查看防火墙服务是否开机启动</p><p> </p><table border="0" cellpadding="0" cellspacing="0"><tbody><tr><td><p>1</p></td><td><p><code>systemctl is-enabled firewalld</code></p></td></tr></tbody></table><p>enabled #开启</p><p>4. 关闭防火墙开机启动</p><p> </p><table border="0" cellpadding="0" cellspacing="0"><tbody><tr><td><p>1</p></td><td><p><code>systemctl disable firewalld</code></p></td></tr></tbody></table><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><code class="hljs bash"><span class="hljs-built_in">rm</span> <span class="hljs-string">'/etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service'</span><br><br><span class="hljs-built_in">rm</span> <span class="hljs-string">'/etc/systemd/system/basic.target.wants/firewalld.service'</span><br></code></pre></td></tr></table></figure><p> </p><table border="0" cellpadding="0" cellspacing="0"><tbody><tr><td><p>1</p></td><td><p><code>systemctl is-enabled firewalld</code></p></td></tr></tbody></table><p>disabled</p>]]></content>
<categories>
<category>Linux</category>
</categories>
<tags>
<tag>systemd</tag>
</tags>
</entry>
<entry>
<title>基于Python Flask 详解 OAuth 2.0 (以github为例)</title>
<link href="/2020/04/16/%E5%9F%BA%E4%BA%8EPython%20Flask%20%E8%AF%A6%E8%A7%A3%20OAuth%202.0%20(%E4%BB%A5github%E4%B8%BA%E4%BE%8B)/"/>
<url>/2020/04/16/%E5%9F%BA%E4%BA%8EPython%20Flask%20%E8%AF%A6%E8%A7%A3%20OAuth%202.0%20(%E4%BB%A5github%E4%B8%BA%E4%BE%8B)/</url>
<content type="html"><![CDATA[<span id="more"></span><p><img src="https://imgconvert.csdnimg.cn/aHR0cHM6Ly9pbWcubWVpd2VuLmNvbS5jbi9pMTcwODkxMS8zNTBlODg5ZDQxYTc0YmM1LmpwZWc?x-oss-process=image/format,png"></p><p><em>OAuth2流程图</em></p><p>OAuth2 对于我来说是一个神秘的东西，我想初步的弄懂中间的整个流程，于是就去google搜索相关的文档资料。</p><p>在浏览了参差不齐的各种文章后，<a href="https://www.barretlee.com/blog/2016/01/10/oauth2-introduce/">简述 OAuth 2.0 的运作流程</a> 基本对于小白来说是最浅显明了的。</p><p>这篇文章以用户使用 github 登录网站留言为例，详述 OAuth 2.0 的运作流程。</p><p>整个OAuth2 的流程分为三个阶段：</p><ol><li>网站和 Github 之间的协商</li><li>用户和 Github 之间的协商</li><li>网站和 Github 用户数据之间的协商</li></ol><p>由于这篇文章是简述，所以并不涉及代码相关的东西，我在原来的文章基础上添加了代码相关的具体实现和一些关键网络交互截图说明方便理解。对于一些文字，由于原文已经写的很流畅严谨，我直接就从原来的博文中复制过来了。</p><hr><p>假如我有一个网站，你是我网站上的访客，看了文章想留言表示「朕已阅」，留言时发现有这个网站的帐号才能够留言，此时给了你两个选择：一个是在我的网站上注册拥有一个新账户，然后用注册的用户名来留言；一个是使用 github 帐号登录，使用你的 github 用户名来留言。前者你觉得过于繁琐，于是惯性地点击了 github 登录按钮，此时 OAuth 认证流程就开始了。</p><p>需要明确的是，即使用户刚登录过 github，我的网站也不可能向 github 发一个什么请求便能够拿到访客信息，这显然是不安全的。就算用户允许你获取他在 github 上的信息，github 为了保障用户信息安全，也不会让你随意获取。所以操作之前，我的网站与 github 之间需要要有一个协商。</p><p>1. 网站和 Github 之间的协商</p><p>Github 会对用户的权限做分类，比如读取仓库信息的权限、写入仓库的权限、读取用户信息的权限、修改用户信息的权限等等。如果我想获取用户的信息，Github 会要求我，先在它的平台上注册一个应用，在申请的时候标明需要获取用户信息的哪些权限，用多少就申请多少，并且在申请的时候填写你的网站域名，Github 只允许在这个域名中获取用户信息。</p><p>此时我的网站已经和 Github 之间达成了共识，Github 也给我发了两张门票，一张门票叫做 Client Id，另一张门票叫做 Client Secret。</p><p>我先去阅读了一下github上相关OAuth2的资料，然后在<a href="https://github.com/settings/developers">这里</a>注册了一个应用。</p><p><img src="https://imgconvert.csdnimg.cn/aHR0cHM6Ly9pbWcubWVpd2VuLmNvbS5jbi9pMTcwODkxMS8wYWI5ZTRjMWYxMDBlZmI4LnBuZw?x-oss-process=image/format,png"><img src="https://img-blog.csdnimg.cn/20200416170217566.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MjQ3NDU0MA==,size_16,color_FFFFFF,t_70"></p><p>其中最后一个callback URL表示用户授权之后github默认要跳转的url地址，在代码中需要添加一个路由来处理针对这个地址的请求。</p><p>创建好之后就会显示在OAuth Apps的列表中。</p><p><img src="https://imgconvert.csdnimg.cn/aHR0cHM6Ly9pbWcubWVpd2VuLmNvbS5jbi9pMTcwODkxMS80ZTA4MWY5OTkxYzZhNzNhLnBuZw?x-oss-process=image/format,png"></p><p>这一步非常简单，github生成了两个钥匙，Client ID和Client Secret。现在我的网站就可以使用合法的使用github提供的OAuth登陆机制了。</p><p>2. 用户和 Github 之间的协商</p><p>用户进入我的网站，点击 github 登录按钮的时候，我的网站会把上面拿到的 Client Id 交给用户，让他进入到 Github 的授权页面，Github 看到了用户手中的门票，就知道这是我的网站让他过来的，于是它就把我的网站想要获取的权限摆出来，并询问用户是否允许我获取这些权限。</p><p>如果用户觉得我的网站要的权限太多，或者压根就不想我知道他这些信息，选择了拒绝的话，整个 OAuth 2.0 的认证就结束了，认证也以失败告终。如果用户觉得 OK，在授权页面点击了确认授权后，页面会跳转到我预先设定的 <code>redirect_uri</code> 并附带一个盖了章的门票 code。</p><p>这个时候，用户和 Github 之间的协商就已经完成，Github 也会在自己的系统中记录这次协商，表示该用户已经允许在我的网站访问上直接操作和使用他的部分资源。</p><p>这个中间会涉及到非常多的流程，我选择使用python基于flask来演示整个流程。</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-comment"># github生成的两把钥匙</span><br>client_id = <span class="hljs-string">'1f93ab8ba338b032b8e7'</span><br>client_secret = <span class="hljs-string">'f0cf5600d2749d1651f2d5f7225c81f562******'</span><br><br><br><span class="hljs-meta">@app.route(<span class="hljs-params"><span class="hljs-string">'/'</span>, methods=[<span class="hljs-string">'GET'</span>, <span class="hljs-string">'POST'</span>]</span>)</span><br><span class="hljs-keyword">def</span> <span class="hljs-title function_">index</span>():<br> url = <span class="hljs-string">'https://github.com/login/oauth/authorize'</span><br> params = {<br> <span class="hljs-string">'client_id'</span>: client_id,<br> <span class="hljs-comment"># 如果不填写redirect_uri那么默认跳转到oauth中配置的callback url。</span><br> <span class="hljs-comment"># 'redirect_uri': 'http://dig404.com/oauth2/github/callback',</span><br> <span class="hljs-string">'scope'</span>: <span class="hljs-string">'read:user'</span>,<br> <span class="hljs-comment"># 随机字符串，防止csrf攻击</span><br> <span class="hljs-string">'state'</span>: <span class="hljs-string">'An unguessable random string.'</span>,<br> <span class="hljs-string">'allow_signup'</span>: <span class="hljs-string">'true'</span><br> }<br> url = furl(url).<span class="hljs-built_in">set</span>(params)<br> <span class="hljs-keyword">return</span> redirect(url, <span class="hljs-number">302</span>)<br></code></pre></td></tr></table></figure><p>当用户在浏览器中访问127.0.0.1:5000的时候，flask会将请求重定向到github的oauth服务页面，重定向的url会携带上两个主要的参数，一个是client_id，一个是scope，这两个参数可以让github知道这个请求是从哪里过来的，并且想要获取的权限。</p><p><img src="https://imgconvert.csdnimg.cn/aHR0cHM6Ly9pbWcubWVpd2VuLmNvbS5jbi9pMTcwODkxMS9lZWRjNzZmMjY0ODE1OWU0LnBuZw?x-oss-process=image/format,png"></p><p>访问125.0.0.1:5000后flask重定向到github授权页面<img src="https://imgconvert.csdnimg.cn/aHR0cHM6Ly9pbWcubWVpd2VuLmNvbS5jbi9pMTcwODkxMS84ZjM3NWVkN2VkZmNiNzJiLnBuZw?x-oss-process=image/format,png"></p><p>如果没有登陆github那么首先github会先跳转到用户的登陆页面<img src="https://imgconvert.csdnimg.cn/aHR0cHM6Ly9pbWcubWVpd2VuLmNvbS5jbi9pMTcwODkxMS8yYTExNDhjNjc0N2JmNDA4LnBuZw?x-oss-process=image/format,png"></p><p>用户登陆自己的github账号</p><p><img src="https://img-blog.csdnimg.cn/20200416165941162.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MjQ3NDU0MA==,size_16,color_FFFFFF,t_70"></p><p>登陆成功后跳转到授权页面</p><p><img src="https://img-blog.csdnimg.cn/20200416165956689.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MjQ3NDU0MA==,size_16,color_FFFFFF,t_70"></p><p>点击授权后github跳转到之前设置的callback页面</p><p>其中最终重定向url中的code参数就是github分配的针对当前登陆用户的授权码，也就是一张门票。在github的后台，这个code和client_id，user是对应的。</p><p>到这里用户和github之间的协商就完成了，剩下的事情就是网站和github之间的事情了。</p><p>3. 从 Github 获取用户的信息</p><p>第二步中，已经拿到了盖过章的门票 code，但这个 code 只能表明，用户允许我的网站从 github 上获取该用户的数据，如果我直接拿这个 code 去 github 访问数据一定会被拒绝，因为任何人都可以持有 code，github 并不知道 code 持有方就是我本人。</p><p>还记得之前申请应用的时候 github 给我的两张门票么，Client Id 在上一步中已经用过了，接下来轮到另一张门票 Client Secret。</p><p>创建一个处理callback路由的处理函数，首先是获取github返回的code。</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-meta">@app.route(<span class="hljs-params"><span class="hljs-string">'/oauth2/<service>/callback'</span></span>)</span><br><span class="hljs-keyword">def</span> <span class="hljs-title function_">oauth2_callback</span>(<span class="hljs-params">service</span>):<br> <span class="hljs-built_in">print</span>(service)<br><br> code = request.args.get(<span class="hljs-string">'code'</span>)<br> <span class="hljs-comment"># 根据返回的code获取access token</span><br> access_token_url = <span class="hljs-string">'https://github.com/login/oauth/access_token'</span><br> payload = {<br> <span class="hljs-string">'client_id'</span>: client_id,<br> <span class="hljs-string">'client_secret'</span>: client_secret,<br> <span class="hljs-string">'code'</span>: code,<br> <span class="hljs-comment"># 'redirect_uri':</span><br> <span class="hljs-string">'state'</span>: <span class="hljs-string">'An unguessable random string.'</span><br> }<br> r = requests.post(access_token_url, json=payload, headers={<span class="hljs-string">'Accept'</span>: <span class="hljs-string">'application/json'</span>})<br> access_token = json.loads(r.text).get(<span class="hljs-string">'access_token'</span>)<br> <span class="hljs-comment"># 拿到access token之后就可以去读取用户的信息了</span><br> access_user_url = <span class="hljs-string">'https://api.github.com/user'</span><br> r = requests.get(access_user_url, headers={<span class="hljs-string">'Authorization'</span>: <span class="hljs-string">'token '</span> + access_token})<br> <span class="hljs-keyword">return</span> jsonify({<br> <span class="hljs-string">'status'</span>: <span class="hljs-string">'success'</span>,<br> <span class="hljs-string">'data'</span>: json.loads(r.text)<br> })<br></code></pre></td></tr></table></figure><p>拿着用户盖过章的 code 和能够标识个人身份的 client_id、client_secret 去拜访 github，拿到最后的绿卡 access_token。</p><p>有了access_token之后就可以读取用户授权的信息了，最后为了演示我把读取到的信息回显到了网页上。<br>这其中的过程对于用户来说不可见的，用户最终在浏览器中的url还是第二步重定向的url。</p><p><img src="https://imgconvert.csdnimg.cn/aHR0cHM6Ly9pbWcubWVpd2VuLmNvbS5jbi9pMTcwODkxMS80NzlhZTllYzRiMGZkMTY4LnBuZw?x-oss-process=image/format,png"></p><p>读取到的用户信息</p><p>拿到用户信息后其实就相当于用户已经登陆了，下一步就可以基于获取到的用户信息对用户做一些业务相关的处理了。</p><hr><p>整个 OAuth2 流程在这里也基本完成了，文章中的表述很粗糙，比如 access_token 这个绿卡是有过期时间的，如果过期了需要使用 refresh_token 重新签证。重点是让读者理解整个流程，细节部分可以阅读 <a href="http://www.rfcreader.com/#rfc6749">RFC6749 文档</a>。</p><p>希望对你理解 OAuth 2.0 有帮助。</p><p>转自：<a href="https://www.meiwen.com.cn/subject/iqncpftx.html">https://www.meiwen.com.cn/subject/iqncpftx.html</a></p>]]></content>
<categories>
<category>python加油鸭</category>
</categories>
<tags>
<tag>python jwt</tag>
</tags>
</entry>
<entry>
<title>Python 常用库 模块 插件 大全</title>
<link href="/2020/04/16/Python%20%E5%B8%B8%E7%94%A8%E5%BA%93%20%E6%A8%A1%E5%9D%97%20%E6%8F%92%E4%BB%B6%20%E5%A4%A7%E5%85%A8/"/>
<url>/2020/04/16/Python%20%E5%B8%B8%E7%94%A8%E5%BA%93%20%E6%A8%A1%E5%9D%97%20%E6%8F%92%E4%BB%B6%20%E5%A4%A7%E5%85%A8/</url>
<content type="html"><![CDATA[<span id="more"></span><p><strong><a href="https://www.jianshu.com/p/3a299e7beefd">python 常用库大全 整理版</a></strong></p>]]></content>
<categories>
<category>python加油鸭</category>
</categories>
<tags>
<tag>python 编程语言</tag>
</tags>
</entry>
<entry>
<title>微信小程序[object,object]这样的数据如何打印出来 方便调试~</title>
<link href="/2020/04/15/%E5%BE%AE%E4%BF%A1%E5%B0%8F%E7%A8%8B%E5%BA%8F%5Bobject,object%5D%E8%BF%99%E6%A0%B7%E7%9A%84%E6%95%B0%E6%8D%AE%E5%A6%82%E4%BD%95%E6%89%93%E5%8D%B0%E5%87%BA%E6%9D%A5%20%E6%96%B9%E4%BE%BF%E8%B0%83%E8%AF%95~/"/>
<url>/2020/04/15/%E5%BE%AE%E4%BF%A1%E5%B0%8F%E7%A8%8B%E5%BA%8F%5Bobject,object%5D%E8%BF%99%E6%A0%B7%E7%9A%84%E6%95%B0%E6%8D%AE%E5%A6%82%E4%BD%95%E6%89%93%E5%8D%B0%E5%87%BA%E6%9D%A5%20%E6%96%B9%E4%BE%BF%E8%B0%83%E8%AF%95~/</url>
<content type="html"><![CDATA[<span id="more"></span><blockquote><p>你肯定会遇到过打印json数据或者object类型的数据的时候，看不到数据内容的情况，那么你可以往下看。</p></blockquote><p>先上接口获取数据的相关代码</p><figure class="highlight arcade"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br></pre></td><td class="code"><pre><code class="hljs arcade"><span class="hljs-comment">// 获取社保缴费年份列表. 参数为被查询人的id</span><br> insurance_YearInfo(userId) {<br> <span class="hljs-keyword">var</span> that = this<br> api.post({<br> <span class="hljs-attr">url</span>: <span class="hljs-string">'wxapp/chaxun/get_detail_year'</span>,<br> <span class="hljs-attr">data</span>: {<br> <span class="hljs-attr">userid</span>: userId,<br> <span class="hljs-attr">account</span>:that.data.idCard,<br> },<br> <span class="hljs-attr">success</span>: <span class="hljs-function"><span class="hljs-params">data</span> =></span> {<br> <span class="hljs-keyword">if</span> (data.code == <span class="hljs-number">1</span>) { <br> <span class="hljs-built_in">console</span>.<span class="hljs-built_in">log</span>(data);<br> <span class="hljs-built_in">console</span>.<span class="hljs-built_in">log</span>(<span class="hljs-string">'get_detail_year接口中data数据如下'</span>+data);<br> <span class="hljs-built_in">console</span>.<span class="hljs-built_in">log</span>(<span class="hljs-string">'get_detail_year接口中data数据如下'</span>,data);<br> <br> <span class="hljs-keyword">for</span>(<span class="hljs-keyword">var</span> item in data.data){<br> data.data[item].isHidden = <span class="hljs-literal">true</span>;<br> data.data[item].yearisClick = <span class="hljs-literal">false</span>;<br> }<br> <span class="hljs-built_in">console</span>.<span class="hljs-built_in">log</span>(data);<br><br> this.setData({ <span class="hljs-comment">// 设置数据到UI上</span><br> insurance_yearList: data.data<br> });<br><br><br> } <span class="hljs-keyword">else</span> {<br> <span class="hljs-comment">// 隐藏加载框</span><br> <span class="hljs-comment">// wx.hideLoading();</span><br> api.show_toast(data.msg, <span class="hljs-number">2000</span>);<br> }<br><br> },<br> <span class="hljs-attr">fail</span>: <span class="hljs-function"><span class="hljs-params">err</span> =></span> {<br><br> }<br> });<br> },<br></code></pre></td></tr></table></figure><p>打印结果如下</p><p><img src="https://imgconvert.csdnimg.cn/aHR0cHM6Ly91cGxvYWQtaW1hZ2VzLmppYW5zaHUuaW8vdXBsb2FkX2ltYWdlcy8yMzY0OTQwLTZhNzUxMzVkY2IxZDAzOTAucG5nP2ltYWdlTW9ncjIvYXV0by1vcmllbnQvc3RyaXB8aW1hZ2VWaWV3Mi8yL3cvMTIwMC9mb3JtYXQvd2VicA?x-oss-process=image/format,png"></p><p> </p><p>第1个log的弊端:虽然能看到数据的内容，但是前面没有文字描述，当有很多log要打印时候，哪个log是自己需要的log需要找很长时间。</p><p>第2个log的弊端:虽然前面有文字描述具体是哪个接口的log，但是数据的内容展示不出来。<code>（+ 的方式）</code></p><p>第3个log：既可以看到是哪个接口的log，而且也看到数据的内容。<code>（,的方式）</code></p><h2 id="总结：小程序开发过程中，完全可以使用第3个log的方式-打印你的log"><a href="#总结：小程序开发过程中，完全可以使用第3个log的方式-打印你的log" class="headerlink" title="总结：小程序开发过程中，完全可以使用第3个log的方式 打印你的log"></a>总结：小程序开发过程中，完全可以使用第3个log的方式 打印你的log</h2><p>作者：CoderZb<br>链接：<a href="https://www.jianshu.com/p/342478e4bf54">https://www.jianshu.com/p/342478e4bf54</a><br>来源：简书</p>]]></content>
<categories>
<category>wechat 小程序</category>
</categories>
<tags>
<tag>javascript 微信游戏 小程序</tag>
</tags>
</entry>
<entry>
<title>Flask+ nginx + gunicorn + supervisor 部署项目</title>
<link href="/2020/04/07/Flask+%20nginx%20+%20gunicorn%20+%20supervisor%20%E9%83%A8%E7%BD%B2%E9%A1%B9%E7%9B%AE/"/>
<url>/2020/04/07/Flask+%20nginx%20+%20gunicorn%20+%20supervisor%20%E9%83%A8%E7%BD%B2%E9%A1%B9%E7%9B%AE/</url>
<content type="html"><![CDATA[<span id="more"></span><h5 id="编辑manage-py-文件-作为启动文件来管理Flask-app"><a href="#编辑manage-py-文件-作为启动文件来管理Flask-app" class="headerlink" title="编辑manage.py 文件 作为启动文件来管理Flask app"></a>编辑manage.py 文件 作为启动文件来管理Flask app</h5><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-keyword">from</span> flask_script <span class="hljs-keyword">import</span> Manager<br><span class="hljs-keyword">from</span> flask_migrate <span class="hljs-keyword">import</span> Migrate, MigrateCommand<br><br>app = Flask(__name__)<br>manager = Manager(app)<br><span class="hljs-comment"># 数据库迁移初始化</span><br>Migrate(app, db)<br><span class="hljs-comment"># 添加迁移命令</span><br>manager.add_command(<span class="hljs-string">"db"</span>, MigrateCommand)<br><br><br><span class="hljs-keyword">if</span> __name__==<span class="hljs-string">"__main__"</span>:<br> <span class="hljs-comment"># 使用manager对象启动flask项目，代替app.run()</span><br> manager.run()<br><br></code></pre></td></tr></table></figure><h5 id="在supervisor-conf-文件中增加-以下program"><a href="#在supervisor-conf-文件中增加-以下program" class="headerlink" title="在supervisor.conf 文件中增加 以下program"></a>在supervisor.conf 文件中增加 以下program</h5><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><code class="hljs python">[program:projects]<br>command=/usr/local/<span class="hljs-built_in">bin</span>/gunicorn -w <span class="hljs-number">4</span> -b <span class="hljs-number">0.0</span><span class="hljs-number">.0</span><span class="hljs-number">.0</span>:<span class="hljs-number">2020</span> -k gevent manage:app<br>directory=/home/zzz/projects/<br>autostart=true<br>redirect_stderr=true<br>stdout_logfile=/home/server_log/projects.log<br></code></pre></td></tr></table></figure><blockquote><p>directory 这里填写项目的路径</p></blockquote><p>如果没有写manage.py文件的话 command中 后面也可以直接写创建了 flask app 的文件<br>即 初始化了这句话的文件 app = Flask(<strong>name</strong>)</p><h4 id="这是gunicorn-的一些常用参数"><a href="#这是gunicorn-的一些常用参数" class="headerlink" title="这是gunicorn 的一些常用参数"></a>这是gunicorn 的一些常用参数</h4><blockquote><p>-w: 指定worker的数量（根据实际情况设定）<br>-b：指定绑定的地址和端口号<br>-k: 指定worker-class模式，默认为sync，这里用gevent使之变为异步协程，提高性能。<br>最后指定app的位置。</p></blockquote><h4 id="nginx的简单配置"><a href="#nginx的简单配置" class="headerlink" title="nginx的简单配置"></a>nginx的简单配置</h4><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><code class="hljs python">server {<br> listen <span class="hljs-number">80</span>(监听的端口);<br> server_name ip地址或者域名;<br> <br> location / {<br> proxy_pass http://<span class="hljs-number">127.0</span><span class="hljs-number">.0</span><span class="hljs-number">.1</span>:<span class="hljs-number">2020</span>(转发至的端口);<br> }<br>}<br></code></pre></td></tr></table></figure><p><strong>nginx 的详细配置可以参考：</strong><br><a href="https://blog.csdn.net/qq_40036754/article/details/102463099">nginx 详解</a></p>]]></content>
<categories>
<category>Linux</category>
</categories>
<tags>
<tag>nginx linux python</tag>
</tags>
</entry>
<entry>
<title>SQL 语句大全(收藏)</title>
<link href="/2020/04/07/SQL%20%E8%AF%AD%E5%8F%A5%E5%A4%A7%E5%85%A8(%E6%94%B6%E8%97%8F)/"/>
<url>/2020/04/07/SQL%20%E8%AF%AD%E5%8F%A5%E5%A4%A7%E5%85%A8(%E6%94%B6%E8%97%8F)/</url>
<content type="html"><![CDATA[<span id="more"></span><h3 id="一、基础"><a href="#一、基础" class="headerlink" title="一、基础"></a>一<strong>、基础</strong></h3><p><strong>1、说明：创建数据库<br>CREATE DATABASE database-name CHARACTER SET utf8 COLLATE utf8_general_ci</strong><br><strong>2****、说明：删除数据库<br>drop database dbname</strong><br><strong>3、说明：创建新表<br>create table tabname(col1 type1 [not null] [primary key],col2 type2 [not null],..)</strong></p><p><strong>根据已有的表创建新表： **<br><strong>A：create table tab_new like tab_old (使用旧表创建新表)</strong><br><strong>B<strong><strong>：create table tab_new as select col1,col2… from tab_old definition only</strong></strong>5、说明：删除新表<br>drop table tabname</strong> <br><strong>6、说明：增加一个列<br>Alter table tabname add column col type****注</strong>：列增加后将不能删除。DB2中列加上后数据类型也不能改变，唯一能改变的是增加varchar类型的长度。<br><strong>7、说明：添加主键</strong>： <strong>Alter table tabname add primary key(col) **<br>说明：删除主键：</strong> Alter table tabname drop primary key(col) *<em><strong>8、说明：创建索引</strong>：</em>*create [unique] index idxname on tabname(col….) **<br>删除索引：</strong>drop index idxname**<br>注：索引是不可更改的，想更改必须删除重新建。<br><strong>9****、说明：创建视图</strong>：create view viewname as select statement <br><strong>删除视图</strong>：drop view viewname<br><strong>10****、说明：几个简单的基本的sql语句<br>选择：</strong>select * from table1 where 范围<br><strong>插入：</strong>insert into table1(field1,field2) values(value1,value2)<br><strong>删除：</strong>delete from table1 where 范围<strong>更新</strong>：update table1 set field1=value1 where 范围<br><strong>查找</strong>：select * from table1 where field1 like ’%value1%’ -–like的语法很精妙，查资料!<br><strong>排序</strong>：select * from table1 order by field1,field2 [desc]<br><strong>总数</strong>：select count as totalcount from table1<br><strong>求和</strong>：select sum(field1) as sumvalue from table1<br><strong>平均</strong>：select avg(field1) as avgvalue from table1<br><strong>最大</strong>：select max(field1) as maxvalue from table1<br><strong>最小</strong>：select min(field1) as minvalue from table1<br><strong>11、说明：几个高级查询运算词<br>A： UNION 运算符</strong> <br>UNION 运算符通过组合其他两个结果表（例如 TABLE1 和 TABLE2）并消去表中任何重复行而派生出一个结果表。当 ALL 随 UNION 一起使用时（即 UNION ALL），不消除重复行。两种情况下，派生表的每一行不是来自 TABLE1 就是来自 TABLE2。 <br><strong>B： EXCEPT 运算符 <br>EXCEPT</strong>运算符通过包括所有在 TABLE1 中但不在 TABLE2 中的行并消除所有重复行而派生出一个结果表。当 ALL 随 EXCEPT 一起使用时 (EXCEPT ALL)，不消除重复行。<br><strong>C： INTERSECT 运算符<br>INTERSECT</strong>运算符通过只包括 TABLE1 和 TABLE2 中都有的行并消除所有重复行而派生出一个结果表。当 <strong>ALL</strong>随 INTERSECT 一起使用时 (INTERSECT ALL)，不消除重复行。 <br><strong>注：</strong>使用运算词的几个查询结果行必须是一致的。 <br><strong>12****、说明：使用外连接 **<br>A、</strong>left （outer） join<strong>： <br>左外连接（左连接）：结果集几包括连接表的匹配行，也包括左连接表的所有行。 <br>SQL: select a.a, a.b, a.c, b.c, b.d, b.f from a LEFT OUT JOIN b ON a.a = b.c<br><strong>B：right （outer） join: **<br>右外连接(右连接)：结果集既包括连接表的匹配连接行，也包括右连接表的所有行。 <br><strong>C：full/cross （outer） join</strong>： <br>全外连接：不仅包括符号连接表的匹配行，还包括两个连接表中的所有记录。<br><strong>12****、分组:Group by:</strong><br> 一张表，一旦分组 完成后，查询后只能得到组相关的信息。<br> 组相关的信息：（统计信息） count,sum,max,min,avg 分组的标准)<br> 在SQLServer中分组时：不能以text,ntext,image类型的字段作为分组依据<br> 在selecte统计函数中的字段，不能和普通的字段放在一起</strong>；</strong></p><p><strong>13、对数据库进行操作：<br> 分离数据库</strong>：** sp_detach_db;附加数据库<strong>：</strong>sp_attach_db **后接表明，附加需要完整的路径名<br>**14.**<strong>如何修改数据库的名称:</strong><br> sp_renamedb ‘old_name’, ‘new_name’</p><h3 id="二、提升"><a href="#二、提升" class="headerlink" title="二、提升"></a><strong>二、提升</strong></h3><p><strong>1、说明：复制表(只复制结构,源表名：a 新表名：b) (Access可用)<br>法一：</strong>select * into b from a where 1<>1（仅用于SQlServer）<strong>法二：</strong>select top 0 * into b from a<br><strong>2、说明：拷贝表(拷贝数据,源表名：a 目标表名：b) (Access可用)</strong><br>insert into b(a, b, c) select d,e,f from b;</p><p><strong>3、说明：跨数据库之间表的拷贝(具体数据使用绝对路径) (Access可用)</strong><br>insert into b(a, b, c) select d,e,f from b in ‘具体数据库’ where 条件<br>例子：..from b in ‘“&Server.MapPath(“.”)&“\data.mdb” &“‘ where..</p><p><strong>4、说明：子查询(表名1：a 表名2：b)</strong><br>select a,b,c from a where a IN (select d from b ) 或者: select a,b,c from a where a IN (1,2,3)</p><p><strong>5、说明：显示文章、提交人和最后回复时间</strong><br>select a.title,a.username,b.adddate from table a,(select max(adddate) adddate from table where table.title=a.title) b</p><p><strong>6、说明：外连接查询(表名1：a 表名2：b)</strong><br>select a.a, a.b, a.c, b.c, b.d, b.f from a LEFT OUT JOIN b ON a.a = b.c</p><p><strong>7、说明：在线视图查询(表名1：a )</strong><br>select * from (SELECT a,b,c FROM a) T where t.a > 1;</p><p><strong>8、说明：between的用法,between限制查询数据范围时包括了边界值,not between不包括</strong><br>select * from table1 where time between time1 and time2<br>select a,b,c, from table1 where a not between 数值1 and 数值2</p><p><strong>9、说明：in 的使用方法</strong><br>select * from table1 where a [not] in (‘值1’,’值2’,’值4’,’值6’)</p><p>**10、说明：两张关联表，删除主表中已经在副表中没有的信息 **<br>delete from table1 where not exists ( select * from table2 where table1.field1=table2.field1 )</p><p><strong>11、说明：四表联查问题：</strong><br>select * from a left inner join b on a.a=b.b right inner join c on a.a=c.c inner join d on a.a=d.d where …..</p><p>**12、说明：日程安排提前五分钟提醒 **<br>SQL: select * from 日程安排 where datediff(‘minute’,f开始时间,getdate())>5</p><p><strong>13<strong><strong>、说明：一条</strong></strong>sql *<em><strong>语句搞定数据库分页</strong>select top 10 b.* from (select top 20 主键字段,排序字段 from 表名 order by 排序字段 desc) a,表名 b where b.主键字段 = a.主键字段 order by a.排序字段</em>*具体实现：</strong>关于数据库分页：</p><p> declare @start int,@end int</p><p> @sql nvarchar(600)</p><p> set @sql=’select top’+str(@end-@start+1)+’+from T where rid not in(select top’+str(@str-1)+’Rid from T where Rid>-1)’</p><p> exec sp_executesql @sql</p><p><strong>注意：在<strong><strong>top</strong></strong>后不能直接跟一个变量，所以在实际应用中只有这样的进行特殊的处理。<strong><strong>Rid</strong></strong>为一个标识列，如果<strong><strong>top</strong></strong>后还有具体的字段，这样做是非常有好处的。因为这样可以避免**</strong> top<strong><strong>的字段如果是逻辑索引的，查询的结果后实际表中的不一致（</strong></strong>逻辑索引中的数据有可能和数据表中的不一致，而查询时如果处在索引则首先查询索引**<strong>）</strong></p><p><strong>14、说明：前10条记录</strong><br>select top 10 * form table1 where 范围</p><p><strong>15、说明：选择在每一组b值相同的数据中对应的a最大的记录的所有信息(类似这样的用法可以用于论坛每月排行榜,每月热销产品分析,按科目成绩排名,等等.)</strong><br>select a,b,c from tablename ta where a=(select max(a) from tablename tb where tb.b=ta.b)</p><p><strong>16、说明：包括所有在 TableA中但不在 TableB和TableC中的行并消除所有重复行而派生出一个结果表</strong><br>(select a from tableA ) except (select a from tableB) except (select a from tableC)</p><p><strong>17、说明：随机取出10条数据</strong><br>select top 10 * from tablename order by <strong>newid()</strong></p><p><strong>18、说明：随机选择记录</strong><br>select newid()</p><p>**19、说明：删除重复记录<br>1),**delete from tablename where id not in (select max(id) from tablename group by col1,col2,…)<br><strong>2)</strong>,select distinct * into temp from tablename<br> delete from tablename<br> insert into tablename select * from temp<br><strong>评价： 这种操作牵连大量的数据的移动，这种做法不适合大容量但数据操作3),例如：在一个外部表中导入数据，由于某些原因第一次只导入了一部分，但很难判断具体位置，这样只有在下一次全部导入，这样也就产生好多重复的字段，怎样删除重复字段</strong></p><p>alter table tablename<br>--添加一个自增列<br>add column_b int identity(1,1)<br> delete from tablename where column_b not in(<br>select max(column_b) from tablename group by column1,column2,…)<br>alter table tablename drop column column_b</p><p><strong>20、说明：列出数据库里所有的表名</strong><br>select name from sysobjects where type=’U’ // U代表用户</p><p><strong>21、说明：列出表里的所有的列名</strong><br>select name from syscolumns where id=object_id(‘TableName’)</p><p><strong>22、说明：列示type、vender、pcs字段，以type字段排列，case可以方便地实现多重选择，类似select 中的case。</strong><br>select type,sum(case vender when ‘A’ then pcs else 0 end),sum(case vender when ‘C’ then pcs else 0 end),sum(case vender when ‘B’ then pcs else 0 end) FROM tablename group by type<br><strong>显示结果：<br>type vender pcs</strong><br>电脑 A 1<br>电脑 A 1<br>光盘 B 2<br>光盘 A 2<br>手机 B 3<br>手机 C 3</p><p><strong>23、说明：初始化表table1</strong></p><p>TRUNCATE TABLE table1</p><p><strong>24、说明：选择从10到15的记录</strong><br>select top 5 * from (select top 15 * from table order by id asc) table_别名 order by id desc</p>]]></content>
<categories>
<category>SQL</category>
</categories>
</entry>
<entry>
<title>No application found. Either work inside a view function or push an application context. （Flask报错解决）</title>
<link href="/2020/04/03/No%20application%20found.%20Either%20work%20inside%20a%20view%20function%20or%20push%20an%20application%20context.%20%EF%BC%88Flask%E6%8A%A5%E9%94%99%E8%A7%A3%E5%86%B3%EF%BC%89/"/>
<url>/2020/04/03/No%20application%20found.%20Either%20work%20inside%20a%20view%20function%20or%20push%20an%20application%20context.%20%EF%BC%88Flask%E6%8A%A5%E9%94%99%E8%A7%A3%E5%86%B3%EF%BC%89/</url>
<content type="html"><![CDATA[<span id="more"></span><p><strong>No application found. Either work inside a view function or push an application context.</strong></p><p>flask 报了这个错,字面意思是说没有应用上下文,字面给的解决意见是要么放置在一个视图内,要么提供一个应用(flask)上下文.</p><p>这是采用crate_app()来创建Flask app 的一种错误</p><p>可以采用装饰器来解决，通过app.app_context().push()来推入一个上下文</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><code class="hljs python"><span class="hljs-keyword">def</span> <span class="hljs-title function_">sqlalchemy_context</span>(<span class="hljs-params">app</span>):<br> <span class="hljs-keyword">def</span> <span class="hljs-title function_">add_context</span>(<span class="hljs-params">func</span>):<br><span class="hljs-meta"> @wraps(<span class="hljs-params">func</span>)</span><br> <span class="hljs-keyword">def</span> <span class="hljs-title function_">do_job</span>(<span class="hljs-params">*args, **kwargs</span>):<br> app.app_context().push()<br> result = func(*args,**kwargs)<br> <span class="hljs-keyword">return</span> result<br> <span class="hljs-keyword">return</span> do_job<br> <span class="hljs-keyword">return</span> add_context<br></code></pre></td></tr></table></figure><p><strong>官方解释</strong><br>应用上下文目的<br>应用上下文存在的主要原因是，在过去，没有更好的方式来在请求上下文中附加一堆函数， 因为 Flask 设计的支柱之一是你可以在一个 Python 进程中拥有多个应用。</p><p>那么代码如何找到“正确的”应用？在过去，我们推荐显式地到处传递应用，但是这导致没有用这种想法设计的库的问题，因为让库实现这种想法太不方便。</p><p>解决上述问题的常用方法是使用后面将会提到的 current_app 代理，它被限制在当前请求的应用引用。 既然无论如何在没有请求时创建一个这样的请求上下文是一个没有必要的昂贵操作，那么就引入了应用上下文。</p>]]></content>
<categories>
<category>python加油鸭</category>
</categories>
<tags>
<tag>python 设计模式</tag>
</tags>
</entry>
<entry>