在多处理模块中,ThreadPool 和 Pool 有什么区别?

multiprocessing模块中 ThreadPoolPool的区别是什么。当我试用我的代码时,我看到的主要区别是:

from multiprocessing import Pool
import os, time


print("hi outside of main()")


def hello(x):
print("inside hello()")
print("Proccess id: ", os.getpid())
time.sleep(3)
return x*x


if __name__ == "__main__":
p = Pool(5)
pool_output = p.map(hello, range(3))


print(pool_output)

我看到以下输出:

hi outside of main()
hi outside of main()
hi outside of main()
hi outside of main()
hi outside of main()
hi outside of main()
inside hello()
Proccess id:  13268
inside hello()
Proccess id:  11104
inside hello()
Proccess id:  13064
[0, 1, 4]

使用“线程池”:

from multiprocessing.pool import ThreadPool
import os, time


print("hi outside of main()")


def hello(x):
print("inside hello()")
print("Proccess id: ", os.getpid())
time.sleep(3)
return x*x


if __name__ == "__main__":
p = ThreadPool(5)
pool_output = p.map(hello, range(3))


print(pool_output)

我看到以下输出:

hi outside of main()
inside hello()
inside hello()
Proccess id:  15204
Proccess id:  15204
inside hello()
Proccess id:  15204
[0, 1, 4]

我的问题是:

  • 为什么在 Pool中每次都运行“ outside _ _ main _ _ ()”?

  • multiprocessing.pool.ThreadPool不产生新的进程? 它只是产生新的线程?

  • 如果是这样的话,使用 multiprocessing.pool.ThreadPool和仅仅使用 threading模块有什么区别?

我没有看到任何关于 ThreadPool的官方文件,有人能帮我在哪里找到它吗?

78958 次浏览

The multiprocessing.pool.ThreadPool behaves the same as the multiprocessing.Pool with the only difference that uses threads instead of processes to run the workers logic.

The reason you see

hi outside of main()

being printed multiple times with the multiprocessing.Pool is due to the fact that the pool will spawn 5 independent processes. Each process will initialize its own Python interpreter and load the module resulting in the top level print being executed again.

Note that this happens only if the spawn process creation method is used (only method available on Windows). If you use the fork one (Unix), you will see the message printed only once as for the threads.

The multiprocessing.pool.ThreadPool is not documented as its implementation has never been completed. It lacks tests and documentation. You can see its implementation in the source code.

I believe the next natural question is: when to use a thread based pool and when to use a process based one?

The rule of thumb is:

  • IO bound jobs -> multiprocessing.pool.ThreadPool
  • CPU bound jobs -> multiprocessing.Pool
  • Hybrid jobs -> depends on the workload, I usually prefer the multiprocessing.Pool due to the advantage process isolation brings

On Python 3 you might want to take a look at the concurrent.future.Executor pool implementations.