Securedrop: Default to Tor Browser for functional tests, keep Firefox as fallback

[psivesely]

Since we're expecting sources to be using Tor Browser, it makes the most sense to be using it in our functional tests as well. When this project was started, there wasn't much in the way of tools to do this, but we've been using https://github.com/webfp/tor-browser-selenium for at least 6 months in https://github.com/freedomofpress/fingerprint-securedrop w/o issues. In fact, due to the slower moving nature of TB, it has been much more stable to drive than using Selenium with Firefox.

My plan is to keep Firefox as a fallback browser (just in case), especially because while tor-browser-selenium has been pretty well maintained over the last year, it is less likely to have the longevity of the Selenium project. It just seems good to have a backup in case of kinks. I'll wait until #1527 gets merged because I want the fallback browser to be callable like so: ./manage.py functional --all --fallback-browser -- --pdb -x.

One thing to note is that tbselenium depends on selenium<3. So this potential conflicts with https://github.com/freedomofpress/securedrop/issues/1502 unless we can find a non-hacky way install and use two versions of Selenium. I think it's more important that we run TB for tests than use Selenium 3, as long as things are stable.

[psivesely]

Updated tentative plans:

• Let the add-selenium-3-support branch sit for the time being.
• Install tbselenium (0.2.1) and let that pull in the correct version of selenium.
• Install TBB 6.5 (latest stable)--we can steal the Ansible from https://github.com/freedomofpress/fingerprint-securedrop.
• Install FF 45.7.0esr (on which TBB 6.5 is based) as a fallback (this should work with the selenium that tbselenium pulls in if tbselenium is not driving TBB correctly for w/e reason).
• Implement the switch into the manage module as above.
• If/when tbselenium supports selenium>=3 (there's a branch there working on this with and I might just do it myself if it hasn't happened in another month or two), we fix up/ cannibalize add-selenium-3-support (mostly we want the functional test fixes, assuming things will break during this transition).

Thoughts on this plan?

[tkell]

I like this idea, and agree that tests should match the desired environment. I can take a run at this?

[psivesely]

@tkell Sure! So the scope of your PR would be the second through fifth bullet points? I'll go ahead and put an in-progress label here. Do let us know if work stalls out for w/e reason (no pressure), or you have any questions.

If you're not familiar with pip-tools, read http://nvie.com/posts/better-package-management/.

[psivesely]

@tkell Since review on https://github.com/freedomofpress/securedrop/pull/1527 is lagging don't worry about the 4th bullet point.

[tkell]

@fowlslegs, so I'm clear: because the manage module is being refactored, I should wait on adding the switch to it. Correct?

[psivesely]

@tkell Yes.

[psivesely]

Some investigation into using was done in https://github.com/webfp/tor-browser-selenium/issues/58. I just tried the tbselenium test suite w/ the latest stable Selenium, TBB, and geckodriver, and did get 8/45 errors, but a lot of the issues seem like they might be resolved pretty easily.

[tkell]

Update: Got a working set of changes to Ansible. Have begun to modify the functional tests to use tbselenium, but it appears that I am loading a generic error page instead of any real page. More as it breaks.

[tkell]

Hm. So I can read from the test server using wget, and my tbselenium can read from www.google.com, but it refuses to read from the local test server.

[garrettr]

Hm. So I can read from the test server using wget, and my tbselenium can read from www.google.com, but it refuses to read from the local test server.

@tkell I have a hunch that this is due to Tor Browser's default behavior of forcing every address (including loopback) through the network proxy. See this note from the developer documentation for a tip on accessing loopback addresses with Tor Browser.

Of course, for your purposes, you will need to somehow apply those configuration changes to the Tor Browser instance started by tbselenium. I'm not sure, but think you might be able to find a way to make this work by leveraging the pref_dict argument to the TorBrowserDriver constructor.

[tkell]

Update: cookies are not being set on tbselenium. Working on finding the correct config tweak to get around this.

[tkell]

Currently setting the following, which are about all I can see that Tor Browser has to do with cookies, and still not getting any cookies set.

'network.proxy.no_proxies_on': '127.0.0.1'
'privacy.clearOnShutdown.cookies': False
'network.cookie.lifetimePolicy': 0
'network.cookie.cookieBehavior': 0
'network.cookie.alwaysAcceptSessionCookies': True
'extensions.torbutton.cookie_auto_protect': False
'privacy.items.cookies': False
'noscript.secureCookies': False
'extensions.torbutton.cookie_protections': False
'network.cookie.thirdparty.sessionOnly': False

[tkell]

Hm. Tor Browser itself will set cookies, with or without that config. (Tested on SecureDrop and Twitter). Worried that there is a problem with tbselenium now. =

[psivesely]

Are all the tbselenium tests passing w/ the version of TBB you are using and is there one that checks setting cookies?

[tkell]

@fowlslegs, _journalist_downloads_message reads cookies to get the message sent. All tests to that point are passing.

[psivesely]

I'm asking about tbselenium's own test suite. See convo in https://github.com/webfp/tor-browser-selenium/issues/78.

[psivesely]

I can help look into it too when I get the chance.

[tkell]

Ah, got it. All tbselenium tests pass, except for tbselenium/test/test_tor.py::Test::test_running_with_system_tor, which is SKIPPED, which seems like it should be OK?

Also moved to TBB 6.5.1, no change.

[tkell]

Update: tbselenium can't read cookies (using self.driver.get_cookies()) even when I see the Cookie header in tcpdump, and the cookie can be read with the Firefox driver.

However, the tcpdump logs are also different. There's a call to /col/<code>/<code-name> that has a cookie header on the develop branch, but has a _blank_ cookie header on my branch that is using TBB and tbselenium.

I'm going to open an issue with tbselenium and see what they say – there may be two things going on here though.

[tkell]

tbselenium issue is here: https://github.com/webfp/tor-browser-selenium/issues/79

I can't get it to set a cookie when logging into Twitter.

[tkell]

Fixed by a fine gent at `tbselenium. To quote:

"If that's compatible with what you're doing, you can disable PBM to access cookies.
Just add "browser.privatebrowsing.autostart": False to the pref_dict you have.
(Idea from robotframework/Selenium2Library#177 (comment))

Disabling PBM may have side effects like storing things on the disk, please make sure it's OK for your case."

@fowlslegs, is that an acceptable bit of config to use in tests?

[psivesely]

PBM makes all cookies session cookies, however, we also explicitly set our cookies as session cookies. Therefore, they should still not be written to disk, and should not persist beyond the lifetime of the browser process in which they are set.

It might be good to look into what else PBM does in 45.8.0esr, but in-all it's definitely better still to be using TBB than FF, even if we do have to change this one setting.

[psivesely]

Anyway, yes, go for it! And good work debugging that one. Thanks also @gunesacar.

[tkell]

@gunesacar deserves all the credit for the config fix, for sure.

PR is #1629 - let me know if it looks OK.

[gunesacar]

You're welcome. I'm happy to help.

[garrettr]

Note that once a PR for this issue is ready to merge, we should reexamine the change to source_navigation_steps.py made in #1985. Since we're not using Tor Browser for the functional tests right now, it is appropriate for #1985 to remove the assertion that checks for the flashed logout message, but once we start using Tor Browser for the tests we should make sure such an assertion is re-added, since the presence or absence of the flashed logout message upon logout should be tested.

[redshiftzero]

If someone is interested in this ticket, you can check out branch tkell-test-with-tor-browser from PR #1655 and resume that work.

One thought: we now have code paths that are Firefox only (e.g. the warning to use Tor Browser) and other code paths that are Tor Browser only (e.g. the warning to turn the security slider to high) and we should have both code paths tested.

[eloquence]

I'm closing this issue in favor of a more cleanly organized epic (will cross-link shortly, and will update all referenced issues as well). This doesn't mean that the work is done - it's still WIP.