OS: Ubuntu Trusty on travis ci
Selenium Version: 3.8.1
Browser: Chromedriver 2.34.0 with Chrome
I can do the patch-level update from 3.8.0 to 3.8.1 without exeriencing any differences.
With 3.8.0 everything works fine, while 3.8.1 non-deterministically fails with ConnectionResetError: [Errno 104] Connection reset by peer when trying to instantiate chrome.
The problem occurs in my django tests, only on travis ci and even on travis ci not reliably, so I unfortunately I can't provide a minimized example. The following is the reduced test boilerplate where the failure occurs.
from django.contrib.staticfiles.testing import StaticLiveServerTestCase
class ChromeDriverTestCase(StaticLiveServerTestCase):
browser = None
@classmethod
def setUpClass(cls):
cls.browser = Browser('chrome', headless=True,
executable_path="node_modules/.bin/chromedriver")
super(ChromeDriverTestCase, cls).setUpClass()
The chromedriver npm package is locked to 2.34.0.
This yields the following traceback in case of an failure (full travis ci log for reference):
Traceback (most recent call last):
File "/home/travis/build/meine-stadt-transparent/meine-stadt-transparent/mainapp/tests/live/chromedriver_test_case.py", line 29, in setUpClass
options=options)
File "/home/travis/virtualenv/python3.6.3/lib/python3.6/site-packages/splinter/browser.py", line 63, in Browser
return driver(*args, **kwargs)
File "/home/travis/virtualenv/python3.6.3/lib/python3.6/site-packages/splinter/driver/webdriver/chrome.py", line 35, in __init__
self.driver = Chrome(chrome_options=options, **kwargs)
File "/home/travis/virtualenv/python3.6.3/lib/python3.6/site-packages/selenium/webdriver/chrome/webdriver.py", line 75, in __init__
desired_capabilities=desired_capabilities)
File "/home/travis/virtualenv/python3.6.3/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 154, in __init__
self.start_session(desired_capabilities, browser_profile)
File "/home/travis/virtualenv/python3.6.3/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 243, in start_session
response = self.execute(Command.NEW_SESSION, parameters)
File "/home/travis/virtualenv/python3.6.3/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 310, in execute
response = self.command_executor.execute(driver_command, params)
File "/home/travis/virtualenv/python3.6.3/lib/python3.6/site-packages/selenium/webdriver/remote/remote_connection.py", line 466, in execute
return self._request(command_info[0], url, body=data)
File "/home/travis/virtualenv/python3.6.3/lib/python3.6/site-packages/selenium/webdriver/remote/remote_connection.py", line 490, in _request
resp = self._conn.getresponse()
File "/opt/python/3.6.3/lib/python3.6/http/client.py", line 1331, in getresponse
response.begin()
File "/opt/python/3.6.3/lib/python3.6/http/client.py", line 297, in begin
version, status, reason = self._read_status()
File "/opt/python/3.6.3/lib/python3.6/http/client.py", line 258, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/opt/python/3.6.3/lib/python3.6/socket.py", line 586, in readinto
return self._sock.recv_into(b)
ConnectionResetError: [Errno 104] Connection reset by peer
I'm seeing this as well, just spent a couple days tracking it down. Definitely caused by 3.8.1
Happens to me on 3.9.0 as well
Occasionally (1/3 of the times) receiving this with 3.9.0
I am running selenium 3.10.0 with python 3.6.4 on debian and I get the same erorr, is it possible it is still not fixed?
driver = webdriver.Chrome(chrome_options=chrome_options, executable_path=chrome_driver)
Traceback (most recent call last):
File "", line 1, in
File "/home/stefan/venv/lib/python3.6/site-packages/selenium/webdriver/chrome/webdriver.py", line 75, in __init__
desired_capabilities=desired_capabilities)
File "/home/stefan/venv/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 154, in __init__
self.start_session(desired_capabilities, browser_profile)
File "/home/stefan/venv/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 243, in start_session
response = self.execute(Command.NEW_SESSION, parameters)
File "/home/stefan/venv/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 310, in execute
response = self.command_executor.execute(driver_command, params)
File "/home/stefan/venv/lib/python3.6/site-packages/selenium/webdriver/remote/remote_connection.py", line 466, in execute
return self._request(command_info[0], url, body=data)
File "/home/stefan/venv/lib/python3.6/site-packages/selenium/webdriver/remote/remote_connection.py", line 490, in _request
resp = self._conn.getresponse()
File "/usr/local/lib/python3.6/http/client.py", line 1331, in getresponse
response.begin()
File "/usr/local/lib/python3.6/http/client.py", line 297, in begin
version, status, reason = self._read_status()
File "/usr/local/lib/python3.6/http/client.py", line 258, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/local/lib/python3.6/socket.py", line 586, in readinto
return self._sock.recv_into(b)
ConnectionResetError: [Errno 104] Connection reset by peer
Selenium: 3.9.0
Chromium: 61.0.3163.100
ChromeDriver: 2.30 (30087a73289cd80b9611f658ed81448badaf549e)
Same here.
SELENIUM_RETRIES = 10
SELENIUM_DELAY = 3 # seconds
def _instanciateChrome():
now = datetime.utcnow()
chromeOptions = webdriver.ChromeOptions()
chromeOptions.add_argument("--headless")
chromeOptions.add_argument("--no-sandbox")
chromeOptions.add_argument("--disable-gpu")
for attempt in range(SELENIUM_RETRIES):
try:
chrome_driver = webdriver.Chrome(chrome_options=chromeOptions)
seconds = round((datetime.utcnow() - now).total_seconds(), 2)
logger.info(f'Chrome instance took {seconds}s and {attempt+1} attempts to start.')
return chrome_driver
except ConnectionResetError: # https://github.com/SeleniumHQ/selenium/issues/5296
sleep((attempt+1) * SELENIUM_DELAY)
raise SeleniumError(f'Cannot connect to Chrome, giving up after {SELENIUM_RETRIES} attempts.')
is it possible it is still not fixed?
considering nobody has even mentioned the version of Chrome they are attempting to use, I don't see how this could even possibly be diagnosed.
@cgoldberg A quick look in the travis log I linked in the initial report shows that this build used the google-chrome-stable package in version 63.0.3239.132-1.
same problem for:
for a code:
from selenium import webdriver
BEHAVE_DEBUG_ON_ERROR = False
def before_all(context):
options = webdriver.ChromeOptions()
options.add_argument('--headless')
options.add_argument('--no-sandbox')
context.browser = webdriver.Chrome(chrome_options=options)
here are the logs:
Exception ConnectionResetError: [Errno 104] Connection reset by peer
Traceback (most recent call last):
File "/usr/bin/behave", line 11, in <module>
sys.exit(main())
File "/usr/lib/python3.6/site-packages/behave/__main__.py", line 109, in main
failed = runner.run()
File "/usr/lib/python3.6/site-packages/behave/runner.py", line 672, in run
return self.run_with_paths()
File "/usr/lib/python3.6/site-packages/behave/runner.py", line 693, in run_with_paths
return self.run_model()
File "/usr/lib/python3.6/site-packages/behave/runner.py", line 471, in run_model
self.run_hook('before_all', context)
File "/usr/lib/python3.6/site-packages/behave/runner.py", line 405, in run_hook
self.hooks[name](context, *args)
File "features/environment.py", line 10, in before_all
context.browser = webdriver.Chrome(chrome_options=options)
File "/usr/lib/python3.6/site-packages/selenium/webdriver/chrome/webdriver.py", line 75, in __init__
desired_capabilities=desired_capabilities)
File "/usr/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 154, in __init__
self.start_session(desired_capabilities, browser_profile)
File "/usr/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 243, in start_session
response = self.execute(Command.NEW_SESSION, parameters)
File "/usr/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 310, in execute
response = self.command_executor.execute(driver_command, params)
File "/usr/lib/python3.6/site-packages/selenium/webdriver/remote/remote_connection.py", line 466, in execute
return self._request(command_info[0], url, body=data)
File "/usr/lib/python3.6/site-packages/selenium/webdriver/remote/remote_connection.py", line 490, in _request
resp = self._conn.getresponse()
File "/usr/lib/python3.6/http/client.py", line 1331, in getresponse
response.begin()
File "/usr/lib/python3.6/http/client.py", line 297, in begin
version, status, reason = self._read_status()
File "/usr/lib/python3.6/http/client.py", line 258, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/lib/python3.6/socket.py", line 586, in readinto
return self._sock.recv_into(b)
ConnectionResetError: [Errno 104] Connection reset by peer
Same
Same as @paterit
Happens to me on 3.11.0 as well
can't reproduce... you need to provide more info
what information could help diagnosing this issue?
Not sure if this is the same issue, but I have similar problem on Travis CI
https://travis-ci.org/andreymal/mini_fiction/jobs/381141177#L873
I've set up a repo to stress test the Python driver to reproduce it:
https://github.com/kevlened/python-webdriver-stress-test
I used ChromeDriver 2.38.552518 and Chrome 66.0.3359.139, but I think the error is seen due to a change in the python client.
Between 3.8.0 and 3.8.1, this change went in. I haven't dug into why this causes the issue (it should only open a socket briefly), but reverting it fixes the problem.
Reverting it also fixes these other errors I was seeing in chromedriver:
Starting ChromeDriver 2.38.552518 (183d19265345f54ce39cbb94cf81ba5f15905011) on port 9515
Only local connections are allowed.
[1526949447.637][SEVERE]: CreatePlatformSocket() returned an error, errno=24: Too many open files (24)
[1526949447.637][SEVERE]: CreatePlatformSocket() returned an error, errno=24: Too many open files (24)
[1526949447.637][SEVERE]: CreatePlatformSocket() returned an error, errno=24: Too many open files (24)
[1526949447.637][SEVERE]: CreatePlatformSocket() returned an error, errno=24: Too many open files (24)
[1526949447.637][SEVERE]: CreatePlatformSocket() returned an error, errno=24: Too many open files (24)
That makes me think that common_utils.is_connectable() isn't properly closing sockets.
We're experiencing the same issue. Adding a small delay either before or after the socket close() in webdriver/common/utils.py seems to prevent the issue in our case.
Turns out that under this stress test, the errors are also raised with 3.8.0, so I think the errors I'm seeing may be a different issue.
The Node webdriver implementation retries requests 3 times on these types of connection errors. I can open a PR to handle the errors in a similar way, but it would simply mask the error.
@chadawagner I'm curious what happens if instead of a delay, you add socket_.shutdown(socket.SHUT_RDWR) before it. The docs here say:
Note: close() releases the resource associated with a connection but does not necessarily close the connection immediately. If you want to close the connection in a timely fashion, call shutdown() before close().
Yeah, I saw that too and tried shutdown() but it didn't change anything :-(
We monkeypatched Selenium to remove the change that @kevlened pointed out (https://github.com/SeleniumHQ/selenium/commit/66f77a0c3623d0761f35f82ad620b3cd1cba66c7#diff-60d463a3d3684a87bf6a994fd436bc43) and that seems to have fixed it for us.
I see nothing wrong with the is_connectable() code or the socket.py create_connection() code for that matter, so I think there's a pretty good chance the bug could be in chromedriver... but it's just a guess.
I've been running into this issue as well.
I've found that, oddly enough, I was getting the same issue when running Selenium with Firefox. I went in and did some AB testing with a Linux VM and could recreate it when my memory was capped at a gig, however doubling the memory to two gigs seems to have resolved this for me. Has anyone else had similar results? It's odd to me that the error message has absolutely nothing to do with memory
I also get this issue but with Firefox. Main stack trace at the bottom
macOS Sierra 10.12.6 16GB memory
geckodriver 0.21.0
Firefox 61.0.1 installed on system
selenium 3.13.0
Python 3.6.5
EDIT: Does not occur with ChromeDriver 2.40.565386 and 67.0.3396.99 (Official Build) (64-bit)
I get this without fail when running my script.
Script runs less than 10 minutes
I also tried rebooting and not running Firefox at the same time, no change.
I also set a breakpoint in remote_connections.py ln 498 to see if I figure anything else out, and expanded the exception with as e and e was Errno 32 Broken Pipe
Main Stack trace
Traceback (most recent call last):
File "automate.py", line 136, in <module>
driver.find_element_by_css_selector(link_css).click()
File "/REALTIMEENV/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 597, in find_element_by_css_selector
return self.find_element(by=By.CSS_SELECTOR, value=css_selector)
File "/REALTIMEENV/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 966, in find_element
'value': value})['value']
File "/REALTIMEENV/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 318, in execute
response = self.command_executor.execute(driver_command, params)
File "/REALTIMEENV/lib/python3.6/site-packages/selenium/webdriver/remote/remote_connection.py", line 472, in execute
return self._request(command_info[0], url, body=data)
File "/REALTIMEENV/lib/python3.6/site-packages/selenium/webdriver/remote/remote_connection.py", line 496, in _request
resp = self._conn.getresponse()
File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1331, in getresponse
response.begin()
File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 297, in begin
version, status, reason = self._read_status()
File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 258, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/socket.py", line 586, in readinto
return self._sock.recv_into(b)
ConnectionResetError: [Errno 54] Connection reset by peer
Memory usage at time of failure on macOS via Activity Monitor
Firefox - Memory 501.7MB - Real mem 1.34GB (Spiked up to 1.6) - Private mem 209.4MB - Shared Mem 329.0MB - Sandbox No - Restricted No
Firefox CP W... - Memory 598.0MB - Real Mem 697.4MB - Private Mem 469.0MB - Shared Mem 329.0MB - Sandbox yes - Restricted No
Selenium actions taken
driver = webdriver.Firefox(executable_path=driver_path)
driver.maximize_window()
driver.get(login_url)
username = driver.find_element_by_id("email")
password = driver.find_element_by_id("password")
username.send_keys(user_name)
password.send_keys(user_password)
driver.find_element_by_class_name("submit").click()
driver.get(URL)
Repeated until failure occurs
driver.find_element_by_css_selector(sticky_menu).click()
driver.find_element_by_css_selector(sticky_menu).click()
driver.find_element_by_css_selector(link_css).click()
With Selenium 3.13.0 and chromedriver its working fine on macos host but fail inside a docker container.
Downgrading Selenium to 3.8.0 in docker seems its working
I've got exactly same issue as @bySabi , using Selenium 3.13.0 with chromedriver seems to work fine on my windows machine, but when I use docker it has [error 104]connection reset by peer problem, and downgrading to 3.8.0 resolves this problem. Thanks to the solution, since I got stuck on this problem for hours.
Here is my dockerfile of installing chromedriver
FROM python:3.6-alpine
# update apk repo
RUN echo "http://dl-4.alpinelinux.org/alpine/v3.7/main" >> /etc/apk/repositories && \
echo "http://dl-4.alpinelinux.org/alpine/v3.7/community" >> /etc/apk/repositories
# install chromedriver
RUN apk update
RUN apk add chromium chromium-chromedriver
and complete error code
Traceback (most recent call last):
File "/usr/local/bin/celery", line 11, in <module>
sys.exit(main())
File "/usr/local/lib/python3.6/site-packages/celery/__main__.py", line 16, in main
_main()
File "/usr/local/lib/python3.6/site-packages/celery/bin/celery.py", line 322, in main
cmd.execute_from_commandline(argv)
File "/usr/local/lib/python3.6/site-packages/celery/bin/celery.py", line 496, in execute_from_commandline
super(CeleryCommand, self).execute_from_commandline(argv)))
File "/usr/local/lib/python3.6/site-packages/celery/bin/base.py", line 273, in execute_from_commandline
argv = self.setup_app_from_commandline(argv)
File "/usr/local/lib/python3.6/site-packages/celery/bin/base.py", line 479, in setup_app_from_commandline
self.app = self.find_app(app)
File "/usr/local/lib/python3.6/site-packages/celery/bin/base.py", line 501, in find_app
return find_app(app, symbol_by_name=self.symbol_by_name)
File "/usr/local/lib/python3.6/site-packages/celery/app/utils.py", line 359, in find_app
sym = symbol_by_name(app, imp=imp)
File "/usr/local/lib/python3.6/site-packages/celery/bin/base.py", line 504, in symbol_by_name
return imports.symbol_by_name(name, imp=imp)
File "/usr/local/lib/python3.6/site-packages/kombu/utils/imports.py", line 56, in symbol_by_name
module = imp(module_name, package=package, **kwargs)
File "/usr/local/lib/python3.6/site-packages/celery/utils/imports.py", line 104, in import_from_cwd
return imp(module, package=package)
File "/usr/local/lib/python3.6/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 994, in _gcd_import
File "<frozen importlib._bootstrap>", line 971, in _find_and_load
File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 678, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/app/xiaohongshu_sel.py", line 9, in <module>
browser = webdriver.Chrome(options=options)
File "/usr/local/lib/python3.6/site-packages/selenium/webdriver/chrome/webdriver.py", line 75, in __init__
desired_capabilities=desired_capabilities)
File "/usr/local/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 156, in __init__
self.start_session(capabilities, browser_profile)
File "/usr/local/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 251, in start_session
response = self.execute(Command.NEW_SESSION, parameters)
File "/usr/local/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 318, in execute
response = self.command_executor.execute(driver_command, params)
File "/usr/local/lib/python3.6/site-packages/selenium/webdriver/remote/remote_connection.py", line 472, in execute
return self._request(command_info[0], url, body=data)
File "/usr/local/lib/python3.6/site-packages/selenium/webdriver/remote/remote_connection.py", line 496, in _request
resp = self._conn.getresponse()
File "/usr/local/lib/python3.6/http/client.py", line 1331, in getresponse
response.begin()
File "/usr/local/lib/python3.6/http/client.py", line 297, in begin
version, status, reason = self._read_status()
File "/usr/local/lib/python3.6/http/client.py", line 258, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/local/lib/python3.6/socket.py", line 586, in readinto
return self._sock.recv_into(b)
ConnectionResetError: [Errno 104] Connection reset by peer
@KaitoHH don't trust at all on chromedriver on this alpine container, this also bit me. I moved to a debian base container and it is working flawless
This a fragment of my current working setup
FROM python:3.6-stretch
## For chromedriver installation: curl/wget/libgconf/unzip
RUN \
apt-get update -y && apt-get install -y --no-install-recommends \
wget curl unzip libgconf-2-4 \
gcc libffi-dev libssl-dev libxml2-dev libxslt1-dev zlib1g-dev \
chromium xvfb xauth \
sqlite \
msmtp &&\
# cleanup
apt-get clean &&\
rm -rf /var/cache/apt/archives/* /var/lib/apt/lists/* /tmp/* /var/tmp/*
# Download, unzip, and install chromedriver
RUN wget -O /tmp/chromedriver.zip http://chromedriver.storage.googleapis.com/`curl -sS chromedriver.storage.googleapis.com/LATEST_RELEASE`/chromedriver_linux64.zip
RUN unzip /tmp/chromedriver.zip chromedriver -d /usr/local/bin/
....
@bayandin are you now using latest selenium? I have also tried to directly connect to the container using docker run -it <my-image> /bin/ash , open python, and run several scripts of starting up selenium browser, which works well, but using docker-compose up combined with some other services got [error 104]connection reset by peer.
I will try your dockerfile solution, thanks a lot.
@KaitoHH I don't try Selenium 3.13 on debian due a lack of time.
I don't now for sure but maybe this alpine 3.7 repo has a chromium and chromium-drive a little outdated: https://pkgs.alpinelinux.org/packages?name=chromium&branch=v3.7
and maybe Selenium package is a bit sensitive on this.
I use too pyvirtualdisplay inside the container together with jaraco.docker(isdocker function)
Seeing this on TravisCI: https://travis-ci.org/bokeh/bokeh/jobs/408670050
chromium-chromedriver_65.0.3325.181-0ubuntu0.14.04.1_amd64.deb
selenium 3.12.0
Edit: can confirm that downgrading to 3.8.0 makes the intermittent failures stop
Perhaps I found a pattern
Selenium always fails if "Runtime kernel version: 4.4.0-112-generic" or "4.9.6-040906-generic"
Selenium always works if "Runtime kernel version: 4.14.12-041412-generic"
Maybe someone can test selenium with different kernel versions
UPD: okay, I installed Ubuntu 16.04 (4.4.0-131-generic) and everything works here
@andreymal I saw the issue with 4.17.12 (OpenSUSE Tumbleweed).
I got same error with Selenium 3.13.1. I had downgrade Selenium to 3.8.0 and it has been fixed.
Same issue for me as well, using Selenium 3.13.0 with chromedriver seems to work fine on my Mac, but when I use it on a Debian server the [error 104]connection reset by peer problem randomly occurs.
Had to downgrade to v3.8.0 to resolves this problem.
note: "same issue" != "similar error message".
This issue has degraded into a catch-all for anything close to the original description... it is not clear under which conditions this occurs or how to reproduce. Rather than gathering more "me too" replies, I am going to close this.
Most helpful comment
I've set up a repo to stress test the Python driver to reproduce it:
https://github.com/kevlened/python-webdriver-stress-test
I used ChromeDriver 2.38.552518 and Chrome 66.0.3359.139, but I think the error is seen due to a change in the python client.
Between 3.8.0 and 3.8.1, this change went in. I haven't dug into why this causes the issue (it should only open a socket briefly), but reverting it fixes the problem.
Reverting it also fixes these other errors I was seeing in chromedriver:
That makes me think that
common_utils.is_connectable()isn't properly closing sockets.