Docker-selenium: Chrome cannot download files to a nfs mounted download directory

[ybizeul]

Meta -

Image(s):
standalone-chrome-debug
Docker-Selenium Image Version(s):
latest (3989b10ec28e)
Selenium bindings version :
3.14.0
Docker Version:
18.03.1-ce
OS:
rancheros v1.4.0

My driver is configured using python as follows :

  driver = webdriver.Remote(remote,{
                                    "browserName":"chrome",
                                    "trustAllSSLCertificates":True,
                                    "acceptSslCerts":True,
                                    "goog:chromeOptions": {  "args": [],
                                                        "prefs": {
                                                                    "download.default_directory": downloadDirectory,
                                                                    "download.prompt_for_download": False,
                                                                    "directory_upgrade": True,
                                                                    "profile.default_content_settings.popups": 0
                                                                 }
                                                     }
                                    });

When I check in the launched chrome instance, I see the options are indeed passed to chrome (in preferences, the download location is good and the prompt setting is correctly set).

The configured directory is rwxrwxrwx and if I go ahead and take the proposed window to point it to this directory and save, the file is written (so selenium has privileges)

I see suggestions to use --headless but Chrome wouldn't launch at all in that case

Expected Behavior -

File should directly download

Actual Behavior -

A window is opened to offer directory selection for download

[ybizeul]

Sample code :

#!/usr/bin/python -u

from selenium import webdriver
from selenium.webdriver.common.by import By

driver = webdriver.Remote("http://selenium.nabox-build.lab.tynsoe.org:4445/wd/hub",{
                                "browserName":"chrome",
                                "trustAllSSLCertificates":True,
                                "acceptSslCerts":True,
                                'goog:chromeOptions': {  'args': ['--test-type'],
                                                    'prefs': {
                                                                'download.default_directory': "/downloads",
                                                                'download.prompt_for_download': False,
                                                                'directory_upgrade': True,
                                                                'profile.default_content_settings.popups': 0
                                                                }
                                                    }
                                })

driver.get("http://ftp.hosteurope.de/mirror/centos.org/")

driver.find_element_by_css_selector("A[href=\"dir_sizes\"]").click()

[ybizeul]

Changing the volume mounted in the container to a seluser accessible mount works, so it looks like a NFS (v4?) issue with Chrome

[ddavison]

i couldn't imagine it being an NFS problem with Chrome since Chrome will just abide by the system. (but i suppose i can't completely discount it)

i'm confused though - did you get this resolved? if this is indeed a problem with Chrome, then we should bring this up with the Chrome team.

[ybizeul]

Thanks @ddavison, yes it took me a while to realize.

No it's not resolved, now I'm trying to think of a way to troubleshoot Chrome not being able to directly download without prompting the window. My best bet is Chrome is doing some filesystem call that doesn't work well with NFSv4 to determine the destination is writable or not.

[diemol]

Hi @ybizeul, is there something we could do to help? I mean, my first impression is that this is not a docker-selenium issue but more one related to the environment where the containers are running.
Just let us know.

[ybizeul]

Hi @diemol, I think the problem is more on the selenium docker side, if you consider that's something that should be working.

The fact that Chrome works fine on a local mount and not on a NFS docker-volume mount is disturbing and will be hard to troubleshoot without getting the Chrome people involved. I tried looking at the source code without much success, and not many people are actually running browsers in container that would have the experience to troubleshoot.

NFS mounts in docker are quite standard and becoming mainstream, it's something that selenium users expect to be able to rely on especially for web test downloads and uploads and at this point that's not functional.

[diemol]

@ybizeul does it happen also with Firefox?

[ybizeul]

I don't think so, but I couldn't setup a firefox container that doesn't prompt, even when not using mounts.

[cgoldberg]

Chrome seems to change the options for doing a direct download with nearly every release, so this is going to be highly dependent on the version of Chrome in use.

If you just need to download something, rather than fiddling with Chrome options... a much easier workaround is to just download the content directly with an HTTP request and forget about trying to force your browser to do it.

[diemol]

Hi @ybizeul

What are the next steps here? Could you provide detailed instructions on how to reproduce this issue? So we can help to get it solved? Right now we have only a set of assumptions.

[ybizeul]

I think I have, running the provided code on NFS persistent volume breaks the download.

I think lots of people need persistent volumes for these tests, nevertheless I think it's an expectation that persistant volume shouldn't interfere with features.

Now I realize it's a Chrome issue but it's bundled as part of the container.

[diemol]

@ybizeul that does not really answer the question, in order to help a bit, it would be nice to have detailed instructions showing how to reproduce the issue.

[ybizeul]

Not sure what I can you more.

So there is the script up there, and the download location has to be a docker volume from a nfs server, I use rancher but it could be any orchestrator probably

[diemol]

How are the containers started? Or how do you start the services?

Additional question, have you tried to write a file into the container when the volume is mounted? Do you get an error?

[ybizeul]

Yes, if I go in the container and create a new file as the selenium user it works fine. Actually, if use the debug container and answer the save panel from VNC, the file saves and the execution continues.

The container is started with no particular options other than the volume mapping but I’ll double check

[ybizeul]

So I'm just mapping port 4444 and 5900 and a persistent volume on /files

[memclutter]

Hi! Is there any progress with this task? I have a similar problem.

Python code

desired_capabilities = {
                'browserName': 'chrome',
                'platform': 'linux',
                'goog:chromeOptions': {
                    "useAutomationExtension": False,
                    "forceDevToolsScreenshot": True,
                    "excludeSwitches": ["enable-automation"],
                    'prefs': {
                        'download.default_directory': os.path.join('/home/seluser/Downloads', str(self.app_id)),
                        'download.directory_upgrade': True,
                        "download.prompt_for_download": False,
                        'intl.accept_languages': 'ru,ru_RU',
                        'profile.default_content_settings.popups': 0
                    },
                    'extensions': [],
                    'args': [
                        '--start-maximized',
                        '--disable-infobars',
                        '--lang=ru',
                        '--user-agent=Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36'
                    ]
                }
            }
driver = webdriver.Remote(command_executor=self.seleniun_url,
                                            desired_capabilities=desired_capabilities)

[gladykov]

Download of file works for me on node-chrome-debug in headless and non-headless mode to a mounted folder. It does not work on node-chrome in headless. Works in non-headless

Only difference I found is /home/seluser is not 777 on node-chrome while it is 777 on node-chrome-debug Even after changing rights to 777 on node-chrome it didn't work either.

Config (Ruby):

``` "goog:chromeOptions":
{
prefs: {
download: {
default_directory: file_export_dir,
directory_upgrade: true,
prompt_for_download: false,
extensions_to_open: ''
},
browser: {
set_download_behavior: {
behavior: 'allow',
downloadPath: @file_export_dir
}
}
},

Extra command invoked (altough I read it may not be needed anymore)

  if @browser == :chrome
    # Following is necessary for file download to work in headless mode but for chrome only!
    # Credit: https://bugs.chromium.org/p/chromium/issues/detail?id=696481
    bridge = driver.browser.send(:bridge)
    path = '/session/:session_id/chromium/send_command'
    path = path.gsub(':session_id', bridge.session_id)
    bridge.http.call(:post,
                     path,
                     cmd: 'Page.setDownloadBehavior',
                     params: {
                       behavior: 'allow',
                       downloadPath: @file_export_dir
                     })
  end

There is also bug in Chrome https://bugs.chromium.org/p/chromium/issues/detail?id=696481

yml config

selenium-chrome:
image: "selenium/node-chrome"
shm_size: '2gb'
links:
- selenium-hub:hub
volumes:
# Used for visibility of downloaded files in test (Capybara) container
- ./output/file_export:/home/seluser/file_export # Do not use seluser directly as it will prevent node-chrome-debug to start
```

[petrus-v]

I'm getting the same kind of issue on a server (not sure it's the same matter). Downloading files are working on my laptop. Both are debian 9 docker hosts.

on the firefox node (docker container) I'm getting this kind of log:

firefox_1           | (firefox:132): dconf-WARNING **: 21:02:29.965: failed to commit changes to dconf: Could not connect: No such file or directory

There are no log on the chrome node.

I'm using the same image node-chrome/node-firefox container on both hosts.

@ybizeul FYI my firefox setting to avoid open any prompt is (it may related to the python library):

        "moz:firefoxOptions":
          prefs:
            "browser.download.folderList": 2
            "browser.helperApps.neverAsk.saveToDisk": "application/octet-stream"
            "browser.download.lastDir": "/downloads/"
            "browser.download.dir": "/downloads/"

I'm wondering if it could be related to dbus setting on the docker host but it doesn't make that much sense to me.

[jdrew99]

I am searching for what could be related issue..? When the home directory of the user is NFS mounted, google-chrome-stable fails with "Trace/breakpoint trap (core dumped)". Firefox works fine; when the home directory is local mount then all is fine. I was assuming there were some NFS mount options (tried nolock).

Any thoughts? (Version 78.0.3904.108 (Official Build) (64-bit))

[dhirajforyou]

any workaround for this ?

[petrus-v]

@dhirajforyou a workaround is to download directly without selenium...

I've created a fallback for CI where I know it's not working

    # When it's working the browser download file to ``export_path``
    export_path = os.path.join(DOWNLOAD_DIR, a_filename)
    if not os.path.exists(export_path):
        # I haven't figured out why it's working on my laptop but it's
        # not working on CI or a server (without desktop)
        # https://github.com/SeleniumHQ/docker-selenium/issues/765#issuecomment-556408483
        # so fallback to request if file is not present which
        if os.getenv("ALLOW_DOWNLOAD_FALLBACK", "0").lower() in ["true", "1"]:
            W = "\033[0m"  # white (normal)
            R = "\033[31m"  # red
            print("")
            print(
                f"{R}** WARNING **: fallback to python download from bdd-tests scenario "
                f"instead direct download by the browser{W}"
            )
            with open(export_path, "wb") as export:
                a_search.click_display_export_form()
                req = request.Request(a_search.download_url_el)
                with request.urlopen(req) as response:
                    export.write(response.read())
        else:
            assert False, (
                f"Downloaded file wasn't found. You may want "
                f"to set ALLOW_DOWNLOAD_FALLBACK to download file "
                f"from bdd-tests scenario."
            )

Hope it helps !

[dhirajforyou]

@petrus-v great, it helped.

[diemol]

Re-reading all comments again, this happens in Chrome, does not happen in Firefox. Is it a Chrome related issue? Would someone want to create a bug in the Chromium repo?

[wtanaka]

I encountered this in Chrome Version 86.0.4240.183 (Official Build) (64-bit) completely outside of the context of Selenium.

[diemol]

Thanks for the comment, as it seems to be a Chrome related issue, I will close this here and I would encourage you to submit a Chrome/ChromeDriver issue to the Chromium team, either at https://bugs.chromium.org/p/chromium or https://bugs.chromium.org/p/chromedriver.