問題描述

注意.我見過 multiprocessing.Process 的日志輸出 - 不幸的是，它沒有回答這個問題問題.

NB. I have seen Log output of multiprocessing.Process - unfortunately, it doesn't answer this question.

我正在通過多處理創(chuàng)建一個子進程(在 Windows 上).我希望將子進程的 stdout 和 stderr 輸出的 all 重定向到日志文件，而不是出現(xiàn)在控制臺上.我看到的唯一建議是讓子進程將 sys.stdout 設(shè)置為文件.但是，由于 Windows 上的標準輸出重定向行為，這并不能有效地重定向所有標準輸出輸出.

I am creating a child process (on windows) via multiprocessing. I want all of the child process's stdout and stderr output to be redirected to a log file, rather than appearing at the console. The only suggestion I have seen is for the child process to set sys.stdout to a file. However, this does not effectively redirect all stdout output, due to the behaviour of stdout redirection on Windows.

為了說明問題，使用以下代碼構(gòu)建一個 Windows DLL

To illustrate the problem, build a Windows DLL with the following code

#include <iostream>

extern "C"
{
    __declspec(dllexport) void writeToStdOut()
    {
        std::cout << "Writing to STDOUT from test DLL" << std::endl;
    }
}

然后創(chuàng)建并運行如下 python 腳本，它會導(dǎo)入這個 DLL 并調(diào)用函數(shù):

Then create and run a python script like the following, which imports this DLL and calls the function:

from ctypes import *
import sys

print
print "Writing to STDOUT from python, before redirect"
print
sys.stdout = open("stdout_redirect_log.txt", "w")
print "Writing to STDOUT from python, after redirect"

testdll = CDLL("Release/stdout_test.dll")
testdll.writeToStdOut()

為了看到與我相同的行為，可能需要針對不同于 Python 使用的 C 運行時構(gòu)建 DLL.在我的例子中，python 是用 Visual Studio 2010 構(gòu)建的，但我的 DLL 是用 VS 2005 構(gòu)建的.

In order to see the same behaviour as me, it is probably necessary for the DLL to be built against a different C runtime than than the one Python uses. In my case, python is built with Visual Studio 2010, but my DLL is built with VS 2005.

我看到的行為是控制臺顯示:

The behaviour I see is that the console shows:

> stdout_test.py

Writing to STDOUT from python, before redirect

Writing to STDOUT from test DLL

雖然文件 stdout_redirect_log.txt 最終包含:

While the file stdout_redirect_log.txt ends up containing:

Writing to STDOUT from python, after redirect

換句話說，設(shè)置 sys.stdout 無法重定向 DLL 生成的 stdout 輸出.鑒于 Windows 中用于標準輸出重定向的底層 API 的性質(zhì)，這不足為奇.我之前在本機/C++ 級別遇到過這個問題，但從未找到一種方法來可靠地從進程中重定向標準輸出.它必須在外部完成.

In other words, setting sys.stdout failed to redirect the stdout output generated by the DLL. This is unsurprising given the nature of the underlying APIs for stdout redirection in Windows. I have encountered this problem at the native/C++ level before and never found a way to reliably redirect stdout from within a process. It has to be done externally.

這實際上是我啟動子進程的原因 - 這樣我就可以從外部連接到它的管道，從而保證我正在攔截它的所有輸出.我絕對可以通過使用 pywin32 手動啟動進程來做到這一點，但我非常希望能夠使用多處理的功能，特別是通過多處理 Pipe 對象與子進程通信的能力，以獲得進展更新.問題是是否有任何方法既可以為其 IPC 設(shè)施使用多處理和可靠地將所有孩子的 stdout 和 stderr 輸出重定向到一個文件.

This is actually the very reason I am launching a child process - it's so that I can connect externally to its pipes and thus guarantee that I am intercepting all of its output. I can definitely do this by launching the process manually with pywin32, but I would very much like to be able to use the facilities of multiprocessing, in particular the ability to communicate with the child process via a multiprocessing Pipe object, in order to get progress updates. The question is whether there is any way to both use multiprocessing for its IPC facilities and to reliably redirect all of the child's stdout and stderr output to a file.

更新:查看 multiprocessing.Processs 的源代碼，它有一個靜態(tài)成員 _Popen，看起來它可以用來覆蓋用于創(chuàng)建進程的類.如果設(shè)置為 None (默認)，它使用一個 multiprocessing.forking._Popen，但它看起來像說

UPDATE: Looking at the source code for multiprocessing.Processs, it has a static member, _Popen, which looks like it can be used to override the class used to create the process. If it's set to None (default), it uses a multiprocessing.forking._Popen, but it looks like by saying

multiprocessing.Process._Popen = MyPopenClass

我可以覆蓋進程創(chuàng)建.然而，雖然我可以從 multiprocessing.forking._Popen 中得到它，但看起來我必須將一堆內(nèi)部的東西復(fù)制到我的實現(xiàn)中，這聽起來很不穩(wěn)定，而且不是很面向未來.如果這是唯一的選擇，我想我可能會更愿意用 pywin32 手動完成整個工作.

I could override the process creation. However, although I could derive this from multiprocessing.forking._Popen, it looks like I would have to copy a bunch of internal stuff into my implementation, which sounds flaky and not very future-proof. If that's the only choice I think I'd probably plump for doing the whole thing manually with pywin32 instead.

推薦答案

您建議的解決方案是一個不錯的解決方案:手動創(chuàng)建您的進程，以便您可以顯式訪問它們的 stdout/stderr 文件句柄.然后，您可以創(chuàng)建一個套接字與子進程通信，并在該套接字上使用 multiprocessing.connection(multiprocessing.Pipe 創(chuàng)建相同類型的連接對象，因此這應(yīng)該為您提供所有相同的 IPC 功能).

The solution you suggest is a good one: create your processes manually such that you have explicit access to their stdout/stderr file handles. You can then create a socket to communicate with the sub-process and use multiprocessing.connection over that socket (multiprocessing.Pipe creates the same type of connection object, so this should give you all the same IPC functionality).

這是一個包含兩個文件的示例.

Here's a two-file example.

ma??ster.py:

import multiprocessing.connection
import subprocess
import socket
import sys, os

## Listen for connection from remote process (and find free port number)
port = 10000
while True:
    try:
        l = multiprocessing.connection.Listener(('localhost', int(port)), authkey="secret")
        break
    except socket.error as ex:
        if ex.errno != 98:
            raise
        port += 1  ## if errno==98, then port is not available.

proc = subprocess.Popen((sys.executable, "subproc.py", str(port)), stdout=subprocess.PIPE, stderr=subprocess.PIPE)

## open connection for remote process
conn = l.accept()
conn.send([1, "asd", None])
print(proc.stdout.readline())

subproc.py:

import multiprocessing.connection
import subprocess
import sys, os, time

port = int(sys.argv[1])
conn = multiprocessing.connection.Client(('localhost', port), authkey="secret")

while True:
    try:
        obj = conn.recv()
        print("received: %s
" % str(obj))
        sys.stdout.flush()
    except EOFError:  ## connection closed
        break

您可能還想查看 this問題 從子進程中獲取非阻塞讀取.

You may also want to see the first answer to this question to get non-blocking reads from the subprocess.

這篇關(guān)于多處理:我怎樣才能 ???????? 從子進程重定向標準輸出?的文章就介紹到這了，希望我們推薦的答案對大家有所幫助，也希望大家多多支持html5模板網(wǎng)！