問題描述

我正在嘗試讓解析器使用 beautifulSoup 和多處理.我有一個(gè)錯(cuò)誤:

I'm trying to make a parser use beautifulSoup and multiprocessing. I have an error:

RecursionError: 超出最大遞歸深度

RecursionError: maximum recursion depth exceeded

我的代碼是:

import bs4, requests, time
from multiprocessing.pool import Pool

html = requests.get('https://www.avito.ru/moskva/avtomobili/bmw/x6?sgtd=5&radius=0')
soup = bs4.BeautifulSoup(html.text, "html.parser")

divList = soup.find_all("div", {'class': 'item_table-header'})


def new_check():
    with Pool() as pool:
        pool.map(get_info, divList)

def get_info(each):
   pass

if __name__ == '__main__':
    new_check()

為什么會(huì)出現(xiàn)此錯(cuò)誤以及如何解決?

Why I get this error and how I can fix it?

更新:所有錯(cuò)誤文本都是

Traceback (most recent call last):
  File "C:/Users/eugen/PycharmProjects/avito/main.py", line 73, in <module> new_check()
  File "C:/Users/eugen/PycharmProjects/avito/main.py", line 67, in new_check
    pool.map(get_info, divList)
  File "C:UserseugenAppDataLocalProgramsPythonPython36libmultiprocessingpool.py", line 266, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "C:UserseugenAppDataLocalProgramsPythonPython36libmultiprocessingpool.py", line 644, in get
    raise self._value
  File "C:UserseugenAppDataLocalProgramsPythonPython36libmultiprocessingpool.py", line 424, in _handle_tasks
    put(task)
  File "C:UserseugenAppDataLocalProgramsPythonPython36libmultiprocessingconnection.py", line 206, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "C:UserseugenAppDataLocalProgramsPythonPython36libmultiprocessing
eduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
RecursionError: maximum recursion depth exceeded

推薦答案

當(dāng)你使用 multiprocessing 時(shí)，你傳遞給 worker 的所有東西都必須是 腌制.

When you use multiprocessing, everything you pass to a worker has to be pickled.

很遺憾，很多 BeautifulSoup 樹無法腌制.

Unfortunately, many BeautifulSoup trees can't be pickled.

這有幾個(gè)不同的原因.其中一些是已修復(fù)的錯(cuò)誤，因此您可以嘗試確保您擁有最新的 bs4 版本，而有些則特定于不同的解析器或樹構(gòu)建器......但很有可能沒有這樣的事情會(huì)有幫助的.

There are a few different reasons for this. Some of them are bugs that have since been fixed, so you could try making sure you have the latest bs4 version, and some are specific to different parsers or tree builders… but there's a good chance nothing like this will help.

但根本問題是樹中的許多元素都包含對(duì)樹其余部分的引用.

But the fundamental problem is that many elements in the tree contain references to the rest of the tree.

有時(shí)，這會(huì)導(dǎo)致實(shí)際的無限循環(huán)，因?yàn)檠h(huán)引用對(duì)于其循環(huán)引用檢測來說過于間接.但這通常是一個(gè)可以修復(fù)的錯(cuò)誤.

Occasionally, this leads to an actual infinite loop, because the circular references are too indirect for its circular reference detection. But that's usually a bug that gets fixed.

但是，更重要的是，即使循環(huán)不是無限，它仍然可以從樹的其余部分拖入 1000 多個(gè)元素，這已經(jīng)足以導(dǎo)致遞歸錯(cuò)誤.

But, even more importantly, even when the loop isn't infinite, it can still drag in more than 1000 elements from all over the rest of the tree, and that's already enough to cause a RecursionError.

我認(rèn)為后者就是這里發(fā)生的事情.如果我使用您的代碼并嘗試腌制 divList[0]，它會(huì)失敗.(如果我提高遞歸限制并計(jì)算幀數(shù)，它需要 23080 的深度，這遠(yuǎn)遠(yuǎn)超過默認(rèn)的 1000.)但是如果我采用完全相同的 div 并解析分開，它成功沒有問題.

And I think the latter is what's happening here. If I take your code and try to pickle divList[0], it fails. (If I bump the recursion limit way up and count the frames, it needs a depth of 23080, which is way, way past the default of 1000.) But if I take that exact same div and parse it separately, it succeeds with no problem.

所以，一種可能性是只做 sys.setrecursionlimit(25000).這將解決這個(gè)確切頁面的問題，但是稍微不同的頁面可能需要更多.(另外，將遞歸限制設(shè)置得那么高通常不是一個(gè)好主意——不是因?yàn)槔速M(fèi)了內(nèi)存，而是因?yàn)檫@意味著實(shí)際的無限遞歸需要 25 倍的時(shí)間和 25 倍的資源浪費(fèi)來檢測.)

So, one possibility is to just do sys.setrecursionlimit(25000). That will solve the problem for this exact page, but a slightly different page might need even more than that. (Plus, it's usually not a great idea to set the recursion limit that high—not so much because of the wasted memory, but because it means actual infinite recursion takes 25x as long, and 25x as much wasted resources, to detect.)

另一個(gè)技巧是編寫修剪樹"的代碼，在您腌制之前/之后消除 div 中的任何向上鏈接.這是一個(gè)很好的解決方案，但它可能需要大量工作，并且需要深入了解 BeautifulSoup 的工作原理，我懷疑你是否愿意這樣做.

Another trick is to write code that "prunes the tree", eliminating any upward links from the div before/as you pickle it. This is a great solution, except that it might be a lot of work, and requires diving into the internals of how BeautifulSoup works, which I doubt you want to do.

最簡單的解決方法有點(diǎn)笨拙，但是……您可以將湯轉(zhuǎn)換為字符串，將其傳遞給孩子，然后讓孩子重新解析它:

The easiest workaround is a bit clunky, but… you can convert the soup to a string, pass that to the child, and have the child re-parse it:

def new_check():
    divTexts = [str(div) for div in divList]
    with Pool() as pool:
        pool.map(get_info, divTexts)

def get_info(each):
    div = BeautifulSoup(each, 'html.parser')

if __name__ == '__main__':
    new_check()

這樣做的性能成本可能無關(guān)緊要；更大的擔(dān)憂是，如果您的 HTML 不完美，則轉(zhuǎn)換為字符串并重新解析它可能不是完美的往返.所以，我建議你先做一些沒有多處理的測試，以確保這不會(huì)影響結(jié)果.

The performance cost for doing this is probably not going to matter; the bigger worry is that if you had imperfect HTML, converting to a string and re-parsing it might not be a perfect round trip. So, I'd suggest that you do some tests without multiprocessing first to make sure this doesn't affect the results.

這篇關(guān)于超出最大遞歸深度.多處理和 bs4的文章就介紹到這了，希望我們推薦的答案對(duì)大家有所幫助，也希望大家多多支持html5模板網(wǎng)！