Python multiprocessing

Python multiprocessing で CPU 論理コア総動員で処理させる

環境

• Ubuntu 18.04.4 LTS / Python 3.6.8
• macOS Monterey 12.0.1 / Python 3.8.5

multiprocessing は Python に最初から入っているパッケージなので、別途 install 不要。

論理コア数 確認

import multiprocessing
multiprocessing.cpu_count()

時間のかかる処理を並列で実行するテンプレ

論理コア数12の場合

import multiprocessing

def run(process_count, process):

    if process == 0:
        print('completed', process)
    elif process == 1:
        print('completed', process)
    elif process == 2:
        print('completed', process)
    elif process == 3:
        print('completed', process)
    elif process == 4:
        print('completed', process)
    elif process == 5:
        print('completed', process)
    elif process == 6:
        print('completed', process)
    elif process == 7:
        print('completed', process)
    elif process == 8:
        print('completed', process)
    elif process == 9:
        print('completed', process)
    elif process == 10:
        print('completed', process)
    elif process == 11:
        print('completed', process)

def wrapper(args):
    return run(*args)

def multi_process(argument_list):
    pool = multiprocessing.Pool(multiprocessing.cpu_count())
    pool.map(wrapper, argument_list)
    pool.close()

if __name__ == '__main__':
    process_count = multiprocessing.cpu_count()
    argument_list = [(process_count, i) for i in range(process_count)]
    multi_process(argument_list)

大量データに総動員で FFT を実行した時
$ htop

CPU の温度に注意
$ watch sensors