問題描述

這是幾個小時抓取后的錯誤回溯:

This is the error traceback after several hours of scraping:

The process started from chrome location /usr/bin/google-chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.

這是我的 selenium python 設置:

This is my setup of selenium python:

#scrape.py
from selenium.common.exceptions import *
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.chrome.options import Options

def run_scrape(link):
    chrome_options = Options()
    chrome_options.add_argument('--no-sandbox')
    chrome_options.add_argument("--headless")
    chrome_options.add_argument('--disable-dev-shm-usage')
    chrome_options.add_argument("--lang=en")
    chrome_options.add_argument("--start-maximized")
    chrome_options.add_experimental_option("excludeSwitches", ["enable-automation"])
    chrome_options.add_experimental_option('useAutomationExtension', False)
    chrome_options.add_argument("user-agent=Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36")
    chrome_options.binary_location = "/usr/bin/google-chrome"
    browser = webdriver.Chrome(executable_path=r'/usr/local/bin/chromedriver', options=chrome_options)
    browser.get(<link passed here>)
    try:
        #scrape process
    except:
        #other stuffs
    browser.quit()

#multiprocess.py
import time,
from multiprocessing import Pool
from scrape import *

if __name__ == '__main__':
    start_time = time.time()
    #links = list of links to be scraped
    pool = Pool(20)
    results = pool.map(run_scrape, links)
    pool.close()
    print("Total Time Processed: "+"--- %s seconds ---" % (time.time() - start_time))

Chrome、ChromeDriver 設置、Selenium 版本

Chrome, ChromeDriver Setup, Selenium Version

ChromeDriver 79.0.3945.36 (3582db32b33893869b8c1339e8f4d9ed1816f143-refs/branch-heads/3945@{#614})
Google Chrome 79.0.3945.79
Selenium Version: 4.0.0a3

我想知道為什么 chrome 正在關閉但其他進程正在運行?

Im wondering why is the chrome is closing but other processes are working?

推薦答案

我拿了你的代碼，稍微修改了一下以適應我的測試環境，下面是執行結果:

I took your code, modified it a bit to suit to my Test Environment and here is the execution results:

• 代碼塊:

• Code Block:

• multiprocess.py:

import time
from multiprocessing import Pool
from multiprocessingPool.scrape import run_scrape

if __name__ == '__main__':
    start_time = time.time()
    links = ["https://selenium.dev/downloads/", "https://selenium.dev/documentation/en/"] 
    pool = Pool(2)
    results = pool.map(run_scrape, links)
    pool.close()
    print("Total Time Processed: "+"--- %s seconds ---" % (time.time() - start_time)) 

from selenium import webdriver
from selenium.common.exceptions import NoSuchElementException, TimeoutException
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.options import Options

def run_scrape(link):
    chrome_options = Options()
    chrome_options.add_argument('--no-sandbox')
    chrome_options.add_argument("--headless")
    chrome_options.add_argument('--disable-dev-shm-usage')
    chrome_options.add_argument("--lang=en")
    chrome_options.add_argument("--start-maximized")
    chrome_options.add_experimental_option("excludeSwitches", ["enable-automation"])
    chrome_options.add_experimental_option('useAutomationExtension', False)
    chrome_options.add_argument("user-agent=Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36")
    chrome_options.binary_location=r'C:Program Files (x86)GoogleChromeApplicationchrome.exe'
    browser = webdriver.Chrome(executable_path=r'C:UtilityBrowserDriverschromedriver.exe', options=chrome_options)
    browser.get(link)
    try:
        print(browser.title)
    except (NoSuchElementException, TimeoutException):
        print("Error")
    browser.quit()

控制臺輸出:

Downloads
The Selenium Browser Automation Project :: Documentation for Selenium
Total Time Processed: --- 10.248600006103516 seconds ---

很明顯你的程序在邏輯上完美無缺.

It is pretty much evident your program is logically flawless and just perfect.

正如您在幾個小時的抓取后提到的這個錯誤，我懷疑這是因為 WebDriver 不是線程安全的.話雖如此，如果您可以序列化對底層驅動程序實例的訪問，則可以在多個線程中共享一個引用.這是不可取的.但是你總是可以實例化一個 WebDriver 每個線程的實例.

As you mentioned this error surfaces after several hours of scraping, I suspect this due to the fact that WebDriver is not thread-safe. Having said that, if you can serialize access to the underlying driver instance, you can share a reference in more than one thread. This is not advisable. But you can always instantiate one WebDriver instance for each thread.

理想情況下，線程安全的問題不在于您的代碼，而在于實際的瀏覽器綁定.他們都假設一次只有一個命令(例如，像真實用戶一樣).但另一方面，您始終可以為每個將啟動多個瀏覽選項卡/窗口的線程實例化一個 WebDriver 實例.到目前為止，您的程序似乎很完美.

Ideally the issue of thread-safety isn't in your code but in the actual browser bindings. They all assume there will only be one command at a time (e.g. like a real user). But on the other hand you can always instantiate one WebDriver instance for each thread which will launch multiple browsing tabs/windows. Till this point it seems your program is perfect.

現在，不同的線程 可以在同一個Webdriver 上運行，但是測試的結果不會是你所期望的.背后的原因是，當您使用多線程在不同的選項卡/窗口上運行不同的測試時，需要一點線程安全編碼，否則您將執行的操作如 click() 或 send_keys() 將轉到當前具有焦點 的打開的選項卡/窗口，而不管您希望運行的線程.這實質上意味著所有測試將在具有焦點但不在預期選項卡/窗口上的同一選項卡/窗口上同時運行.

Now, different threads can be run on same Webdriver, but then the results of the tests would not be what you expect. The reason behind is, when you use multi-threading to run different tests on different tabs/windows a little bit of thread safety coding is required or else the actions you will perform like click() or send_keys() will go to the opened tab/window that is currently having the focus regardless of the thread you expect to be running. Which essentially means all the test will run simultaneously on the same tab/window that has focus but not on the intended tab/window.

這篇關于通過 Python 使用 Selenium 進行多處理時，Chrome 在幾個小時后崩潰的文章就介紹到這了，希望我們推薦的答案對大家有所幫助，也希望大家多多支持html5模板網！