More than 3 years have passed since last update.

Pythonにおける並列処理について

Last updated at 2020-03-10Posted at 2020-03-10

　Pythonにおけるconcurrent.futuresとmultiprocessingの挙動についてテストしてみたのでメモ代わりに投稿します。
　目的はPythonによる仕様を確認することではなく、あくまでも「どう動くか」ということです。

CPU thread数と引数max_workersについて

そもそもmax_workersをCPUのthread数と一致させる必要はあるのか？

という疑問がふと沸いたので検証しました。
特に対照実験や統計学的検証は行っていません。ご了承ください。

###検証環境

CPU core i7 6800k (6core,12thread base clock=3.40GHz)
Memory 32GB(DDR4-2133)
OS Windows 10 Pro

###検証コード 1

concurrent-futures.py

from concurrent import futures
from time import time as t


def primes(x):
    primelist_ = [2]
    for i in range(3, x + 1):
        for j in primelist_:
            if j ** 2 > i:
                primelist_.append(i)
                break
            elif i % j == 0:
                break
    return primelist_

if __name__ == "__main__":
    
    list_ = [1000] * 60
    for i in range(1,61):
        s = t()
        with futures.ProcessPoolExecutor(max_workers=i) as executor:
            executor.map(primes,list_)
        e = t()
        print("{0} workers time is {1}".format(i,e-s))

###実行結果 1

1 workers time is 0.5020229816436768
2 workers time is 0.352001428604126
3 workers time is 0.36597752571105957
4 workers time is 0.4050002098083496
5 workers time is 0.5020010471343994
6 workers time is 0.600996732711792
7 workers time is 0.8990006446838379
8 workers time is 1.0159986019134521
9 workers time is 1.1840019226074219
10 workers time is 1.265998363494873
11 workers time is 1.3460001945495605
12 workers time is 1.5809991359710693
13 workers time is 1.7050011157989502
14 workers time is 1.8249998092651367
15 workers time is 1.9100017547607422
16 workers time is 2.215999126434326
17 workers time is 2.0129990577697754
18 workers time is 1.860001564025879
19 workers time is 1.98099946975708
20 workers time is 2.031001091003418
21 workers time is 2.225998878479004
22 workers time is 2.263998508453369
23 workers time is 3.3330018520355225
24 workers time is 3.2790024280548096
25 workers time is 3.2919981479644775
26 workers time is 3.398635149002075
27 workers time is 3.5269999504089355
28 workers time is 3.714276075363159
29 workers time is 3.823046922683716
30 workers time is 3.9260809421539307
31 workers time is 4.0271430015563965
32 workers time is 4.247858047485352
33 workers time is 4.3211283683776855
34 workers time is 4.448809385299683
35 workers time is 4.6109983921051025
36 workers time is 5.038041353225708
37 workers time is 4.97698187828064
38 workers time is 5.122000694274902
39 workers time is 5.114000082015991
40 workers time is 5.294032096862793
41 workers time is 5.48790717124939
42 workers time is 5.696187257766724
43 workers time is 5.781583309173584
44 workers time is 5.892255783081055
45 workers time is 6.041832447052002
46 workers time is 6.1898744106292725
47 workers time is 5.498929738998413
48 workers time is 1.6420001983642578
49 workers time is 3.067247152328491
50 workers time is 2.9033234119415283
51 workers time is 6.299036264419556
52 workers time is 7.5803093910217285
53 workers time is 7.545795917510986
54 workers time is 7.648757219314575
55 workers time is 7.437737703323364
56 workers time is 7.803879022598267
57 workers time is 7.753534317016602
58 workers time is 8.295880317687988
59 workers time is 8.039872407913208
60 workers time is 8.545195579528809

グラフにするとこんな感じ。

###結果 1
多くすればいいってものでもないし、thread数に合わせれば良いってものでもないらしい。

決まり悪いのでもう一つ検証してみる
###検証コード 2

multiprocessing.py

from multiprocessing import Pool
from time import time as t


def primes(x=1000000000):
    primelist_ = [2]
    for i in range(3, x + 1):
        for j in primelist_:
            if j ** 2 > i:
                primelist_.append(i)
                break
            elif i % j == 0:
                break
    return primelist_


if __name__ == "__main__":
    list_ = [1000] * 60
    for i in range(1,61):
        p = Pool(i)
        s = t()
        p.map(primes, list_)
        e = t()
        print("{0} workers time is {1}".format(i, e - s))

###実行結果 2

1 workers time is 0.2633683681488037
2 workers time is 0.22099924087524414
3 workers time is 0.19600248336791992
4 workers time is 0.18899846076965332
5 workers time is 0.194000244140625
6 workers time is 0.20499968528747559
7 workers time is 0.22199773788452148
8 workers time is 0.22899842262268066
9 workers time is 0.25099897384643555
10 workers time is 0.27899861335754395
11 workers time is 0.2799994945526123
12 workers time is 0.29799890518188477
13 workers time is 0.3249998092651367
14 workers time is 0.328000545501709
15 workers time is 0.3409996032714844
16 workers time is 0.37000060081481934
17 workers time is 0.38399839401245117
18 workers time is 0.354999303817749
19 workers time is 0.17999982833862305
20 workers time is 0.453000545501709
21 workers time is 0.3829984664916992
22 workers time is 1.06699800491333
23 workers time is 4.132883548736572
24 workers time is 2.9350016117095947
25 workers time is 3.341001033782959
26 workers time is 3.19700026512146
27 workers time is 3.712002754211426
28 workers time is 3.354022741317749
29 workers time is 4.1504881381988525
30 workers time is 0.10300064086914062
31 workers time is 8.025314092636108
32 workers time is 4.032264471054077
33 workers time is 1.3879990577697754
34 workers time is 0.38199782371520996
35 workers time is 0.8919985294342041
36 workers time is 0.44299936294555664
37 workers time is 0.9930007457733154
38 workers time is 0.5849990844726562
39 workers time is 1.2019994258880615
40 workers time is 3.2748472690582275
41 workers time is 8.506859064102173
42 workers time is 5.890998840332031
43 workers time is 6.897999286651611
44 workers time is 0.09200048446655273
45 workers time is 47.79252886772156
46 workers time is 107.53967475891113
47 workers time is 85.88229584693909
48 workers time is 82.07777094841003
49 workers time is 86.7360258102417
50 workers time is 89.32221388816833
51 workers time is 87.51184272766113
52 workers time is 65.91199898719788
53 workers time is 120.46363306045532
54 workers time is 79.80397748947144
55 workers time is 55.082069873809814
56 workers time is 84.51597738265991
57 workers time is 114.39745855331421
58 workers time is 75.04559135437012
59 workers time is 109.53564620018005
60 workers time is 81.29796814918518

グラフにするとこんな感じ。

これだと見づらいので1-24に限定してみる

###結果 2
multiprocessingを用いる場合はCPUのthread数以下にとどめるのが丸く収まるっぽい。

You get articles that match your needs
You can efficiently read back useful information
You can use dark theme

What you can do with signing up