問題描述

我有 Python 代碼:

from multiprocessing import Process

def f(name):
    print 'hello', name

if __name__ == '__main__':
    for i in range(0, MAX_PROCESSES):
        p = Process(target=f, args=(i,))
        p.start()

運行良好.但是，MAX_PROCESSES 是可變的，可以是 1 和 512 之間的任何值.由于我只在具有 8 內核的機器上運行此代碼，因此我需要確定是否可以限制允許同時運行的進程數.我查看了 multiprocessing.Queue，但它看起來不像我需要的 - 或者我可能錯誤地解釋了文檔.

which runs well. However, MAX_PROCESSES is variable and can be any value between 1 and 512. Since I'm only running this code on a machine with 8 cores, I need to find out if it is possible to limit the number of processes allowed to run at the same time. I've looked into multiprocessing.Queue, but it doesn't look like what I need - or perhaps I'm interpreting the docs incorrectly.

有沒有辦法限制同時運行的 multiprocessing.Process 的數量?

Is there a way to limit the number of simultaneous multiprocessing.Processs running?

推薦答案

使用 multiprocessing.Pool 可能是最明智的，它根據可用的最大內核數生成工作進程池您的系統，然后基本上在內核可用時提供任務.

It might be most sensible to use multiprocessing.Pool which produces a pool of worker processes based on the max number of cores available on your system, and then basically feeds tasks in as the cores become available.

標準文檔中的示例 (http://docs.python.org/2/library/multiprocessing.html#using-a-pool-of-workers)顯示也可以手動設置核心數:

The example from the standard docs (http://docs.python.org/2/library/multiprocessing.html#using-a-pool-of-workers) shows that you can also manually set the number of cores:

from multiprocessing import Pool

def f(x):
    return x*x

if __name__ == '__main__':
    pool = Pool(processes=4)              # start 4 worker processes
    result = pool.apply_async(f, [10])    # evaluate "f(10)" asynchronously
    print result.get(timeout=1)           # prints "100" unless your computer is *very* slow
    print pool.map(f, range(10))          # prints "[0, 1, 4,..., 81]"

如果您的代碼中需要，知道有 multiprocessing.cpu_count() 方法來計算給定系統上的內核數量也很方便.

And it's also handy to know that there is the multiprocessing.cpu_count() method to count the number of cores on a given system, if needed in your code.

這是一些似乎適用于您的特定情況的代碼草案:

Here's some draft code that seems to work for your specific case:

import multiprocessing

def f(name):
    print 'hello', name

if __name__ == '__main__':
    pool = multiprocessing.Pool() #use all available cores, otherwise specify the number you want as an argument
    for i in xrange(0, 512):
        pool.apply_async(f, args=(i,))
    pool.close()
    pool.join()

這篇關于使用具有最大同時進程數的 multiprocessing.Process的文章就介紹到這了，希望我們推薦的答案對大家有所幫助，也希望大家多多支持html5模板網！