問題描述

已解決:問題出在 Wingware Python IDE 上.我想現在很自然的問題是這怎么可能以及如何解決.

SOLVED: The problem was Wingware Python IDE. I guess the natural question now is how it is possible and how this could be fixed.

我昨天問了一個問題(Python 中 multiprocessing.Pool 的問題) 這個問題幾乎是一樣的，但我發現它似乎可以在 Windows 計算機上運行，??而不是在我的 ubuntu 中運行.在這篇文章的最后，我將發布一個略有不同的代碼版本.

I asked a question yesterday ( Problem with multiprocessing.Pool in Python ) and this question is almost the same but I have figured out that it seems to work on a Windows computer and not in my ubuntu. At the end of this post I will post a slightly different version of the code that does the same thing.

我的問題的簡短摘要:在 Python 中使用 multiprocessing.Pool 時，我并不總是能夠獲得我所要求的工人數量.發生這種情況時，程序就會停止.

Short summary of my problem: When using multiprocessing.Pool in Python I am not always able to get the amount of workers that I am asking for. When this happens, the program just stalls.

我整天都在尋找解決方案，然后我開始思考 Noahs 對我之前的問題的評論.他說它可以在他的機器上運行，所以我把代碼提供給了我的同事，他運行的是一臺帶有 Enthoughts 64 位 Python 2.7.1 發行版的 Windows 機器.我與我在 ubuntu 上運行的巨大差異相同.我還提到我們都有 Wingware Python IDE，但我懷疑這是否重要?

I have been working for a solution all day, and then I came to think about Noahs' comment on my previous question. He said that it worked on his machine so I gave the code to my colleague who runs a Windows machine with Enthoughts 64-bit Python 2.7.1 distribution. I have the same with the big difference that mine runs on ubuntu. I also mention that we both have Wingware Python IDE, but I doubt that this is of any importance?

當我的同事在他的機器上運行代碼時，我的代碼不會出現兩個問題.

There are two problems with my code that don't arise when my colleague runs the code on his machine.

1. 我并不總是能夠得到我要求的四個工人(盡管我的機器有 12 個工人).發生這種情況時，該過程會停止并且不會繼續.沒有引發異常或錯誤.

1. I am not always able to get the four workers I am asking for (Although my machine has 12 workers). When this happens, the process just stalls and does not continue. No exception or Error is raised.

當我能夠得到我要求的四名工人時(大約 5 次左右出現 1 次)，所產生的數字(普通隨機數)對于所有四張圖片都是完全相同的.我的同事不是這樣.

When I am able to get the four workers I ask for (which happens approximately 1 out 5 times or so), the figures that are produced (plain random numbers) are EXACTLY the same for all four pictures. This is not the case for my colleague.

有些事情很可疑，我非常感謝你們提供的任何幫助.

Something is very fishy and I am very thankful for any kind of help you guys can offer.

代碼:

import multiprocessing as mp
import scipy as sp
import scipy.stats as spstat
import pylab

def testfunc(x0, N):
    print 'working with x0 = %s' % x0
    x = [x0]
    for i in xrange(1,N):
        x.append(spstat.norm.rvs(size = 1)) # stupid appending to make it slower
        if i % 10000 == 0:
            print 'x0 = %s, i = %s' % (x0, i)
    return sp.array(x)

def testfuncParallel(fargs):
    return testfunc(*fargs)


# Define Number of tasks.
nTasks = 4
N = 100000

if __name__ == '__main__':

    """
    Try number 1. Using multiprocessing.Pool together with Pool.map_async
    """
    pool = mp.Pool(processes = nTasks) # I have 12 threads (six cores) available so I am suprised that it does not get access to nTasks = 4 amount of workers

    # Define tasks:
    tasks = [(x, n) for x, n in enumerate(nTasks*[N])] # nTasks different tasks

    # Compute parallel: async - asynchronically, i.e. not necessary in order.
    result = pool.map_async(testfuncParallel, tasks)

    pool.close() # These are needed if map_async is used
    pool.join()

    # Get results:
    sim = sp.zeros((N, nTasks)) 

    for nn, res in enumerate(result.get()):    
        sim[:, nn] = res

    pylab.figure()
    for i in xrange(nTasks):
        pylab.subplot(nTasks,1, i + 1)
        pylab.plot(sim[:, i])

    pylab.show()

提前致謝.

真誠地，馬蒂亞斯

推薦答案

更新: 原來這與 matplotlib 或后端無關，而是與一般多處理相關的錯誤有關.我們已經為 Wing 版本 4.0.4+ 修復了這個問題.解決方法是不在子進程中執行的代碼中設置斷點.

Update: Turns out this had nothing to do with matplotlib or the backends but rather with a bug associated with multiprocessing in general. We've fixed this for Wing version 4.0.4+. The work-around is not to set breakpoints in the code that is executed in the sub-processes.

這似乎是 Wing IDE 的 matplotlib 支持 Tkinter 后端與多處理交互不良.當我嘗試這個示例時，它會在 TCL/Tk 代碼中崩潰.我懷疑在 Windows 上工作的人正在使用不同的 matplotlib 后端.

It seems to be Wing IDE's matplotlib support for the Tkinter backend interacting badly with multiprocessing. When I try this example it crashes in TCL/Tk code. I suspect the person working on Windows was using a different matplotlib backend.

在擴展"選項卡下的項目屬性"中關閉matplotlib 事件循環支持"似乎可以解決這個問題.

Turning off the "matplotlib event loop support" in Project Properties under the Extensions tab seems to work around it.

或者，當matplotlib 事件循環支持"打開時，添加以下內容似乎可以為我修復它.

Or, adding the following seems to fix it for me when the "matplotlib event loop support" is turned on.

導入matplotlibmatplotlib.use('WXAgg')

import matplotlib matplotlib.use('WXAgg')

這僅在您擁有 WXAgg 后端時才有效.Wing IDE 支持的其他后端(即使調試過程暫停，繪圖也能保持交互)是 GTKAgg 和 Qt4Agg，但我還沒有嘗試過.

This will only work if you have the WXAgg backend. Other backends supported by Wing IDE (in such a way that plots remain interactive even if the debug process is paused) are GTKAgg and Qt4Agg but I didn't try those yet.

我會看看我是否能找到并修復這個錯誤.我懷疑我們需要在進程 ID 更改時禁用我們的事件循環支持.感謝您報告此事.

I'll see if I can find and fix the bug. I suspect we need to disable our event loop support when the process ID changes. Thanks for reporting this.

這篇關于multiprocessing.Pool 似乎可以在 Windows 中工作，但不能在 ubuntu 中工作?的文章就介紹到這了，希望我們推薦的答案對大家有所幫助，也希望大家多多支持html5模板網！