Mybinder.org-deploy: CORS issue causing event stream error on gesis

[minrk]

Builds on gesis are failing with "Failed to connect to event stream" due to a CORS issue. The request to open the event stream is failing with:

[Error] Failed to load resource: Cross-origin redirection to https://notebooks.gesis.org/binder/build/gh/TomAugspurger/idp-results/master?binder_launch_host=https%3A%2F%2Fmybinder.org%2F denied by Cross-Origin Resource Sharing policy: Cannot use wildcard in Access-Control-Allow-Origin when credentials flag is true. (master, line 0)
[Error] EventSource cannot load https://gesis.mybinder.org/build/gh/TomAugspurger/idp-results/master?binder_launch_host=https%3A%2F%2Fmybinder.org%2F due to access control checks.
[Error] Failed to construct event stream – Event {isTrusted: true, type: "error", target: EventSource, …}
Event {isTrusted: true, type: "error", target: EventSource, currentTarget: EventSource, eventPhase: 2, …}Event
(anonymous function) (bundle.js:21152)

I'm not sure what setting is different on gesis that causes this. We should probably disable gesis until this works out, since all builds will appear to fail (the builds and launches do not appear to actually be affected, only the event stream)

[bitnik]

hi @minrk , can you help me here? I am not sure if I really understand the error. How can I produce it? So I can investigate what is wrong in our deployment.

[minrk]

I'm trying to understand the root cause myself. Binder deployments have Access-Control-Allow-Origin: * to allow anyone to connect. However, there is a conflict between this option and Access-Control-Allow-Credentials: true and/or sending requests withCredentials.

It is
There should be tw

[minrk]

What I don't understand is what is different about gesis that is triggering this, why the credentials flag would be different.

I think the best way to test this would be to add gesis to the staging federation so that you can try builds there. It's hard to reproduce when gesis is temporarily out of rotation.

[manics]

@bitnik Did anything change with your authenticated binderhub that might have inadvertently affected the public one?

[minrk]

You can reproduce this in the js console in a browser with:

evt = new EventSource("https://mybinder.org/build/gh/TomAugspurger/idp-results/master", {withCredentials: true})
evt.close()

what's weird is that we don't pass withCredentials: true, and without specifying that, I can't reproduce the issue from the console, even if I specify the exact URL that doesn't work from the same page that doesn't work

[betatim]

One thing that is different for Gesis compared to our other clusters is that there is a nginx which performs a redirect (30x) on requests we make

[minrk]

Ah, that has a very high probability of being it

[minrk]

since CORS queries tend to be made with OPTIONS requests, and may not be redirected the same

[minrk]

what URLs get those redirects?

(edit: apologies for using GitHub comments like a chat app)

[betatim]

https://github.com/gesiscss/orc/blob/master/load_balancer/sites-available/gesis_mybinder is the nginx config running "at" gesis.mybinder.org

[betatim]

Maybe we need to add OPTIONS to https://github.com/gesiscss/orc/blob/f657f2213e5f4d90762ed5311cfb304f280208f0/load_balancer/sites-available/gesis_mybinder#L29?

[minrk]

Yeah, worth a try. I'm not sure if OPTIONS can follow redirects, we may need to write the OPTIONS reply ourselves (it should just set the CORS headers and that's it)

[bitnik]

hey, I am sorry that I couldn't really follow you in this conversation. I even still cant reproduce it myself. According to what you wrote I understand that builds are failing at GESIS when it is done through federation, but I don't experience this neither.

I am adding now OPTIONS as Tim suggested.

[arnim]

Hi, we have just compared @minrk s example (TomAugspurger/idp-results => GKE):

evt = new EventSource("https://staging.mybinder.org/build/gh/TomAugspurger/idp-results/master", {withCredentials: true})
EventSource {url: "https://staging.mybinder.org/build/gh/TomAugspurger/idp-results/master", withCredentials: true, readyState: 0, onopen: null, onmessage: null, …}url: "https://staging.mybinder.org/build/gh/TomAugspurger/idp-results/master"withCredentials: truereadyState: 2onopen: nullonmessage: nullonerror: null__proto__: EventSource
(index):1 Access to resource at 'https://gke.staging.mybinder.org/build/gh/TomAugspurger/idp-results/master?binder_launch_host=https%3A%2F%2Fmybinder.org%2F' (redirected from 'https://staging.mybinder.org/build/gh/TomAugspurger/idp-results/master') from origin 'https://staging.mybinder.org' has been blocked by CORS policy: The value of the 'Access-Control-Allow-Origin' header in the response must not be the wildcard '*' when the request's credentials mode is 'include'.
gke.staging.mybinder.org/build/gh/TomAugspurger/idp-results/master?binder_launch_host=https%3A%2F%2Fmybinder.org%2F:1 GET https://gke.staging.mybinder.org/build/gh/TomAugspurger/idp-results/master?binder_launch_host=https%3A%2F%2Fmybinder.org%2F net::ERR_FAILED

and bitnik/simple-binder-repo-2 => GESIS

evt = new EventSource("https://staging.mybinder.org/build/gh/bitnik/simple-binder-repo-2/master", {withCredentials: true})
EventSource {url: "https://staging.mybinder.org/build/gh/bitnik/simple-binder-repo-2/master", withCredentials: true, readyState: 0, onopen: null, onmessage: null, …}
(index):1 Access to resource at 'https://gesis.mybinder.org/build/gh/bitnik/simple-binder-repo-2/master?binder_launch_host=https%3A%2F%2Fmybinder.org%2F' (redirected from 'https://staging.mybinder.org/build/gh/bitnik/simple-binder-repo-2/master') from origin 'https://staging.mybinder.org' has been blocked by CORS policy: The value of the 'Access-Control-Allow-Origin' header in the response must not be the wildcard '*' when the request's credentials mode is 'include'.
gesis.mybinder.org/build/gh/bitnik/simple-binder-repo-2/master?binder_launch_host=https%3A%2F%2Fmybinder.org%2F:1 GET https://gesis.mybinder.org/build/gh/bitnik/simple-binder-repo-2/master?binder_launch_host=https%3A%2F%2Fmybinder.org%2F net::ERR_FAILED
evt
EventSource {url: "https://staging.mybinder.org/build/gh/bitnik/simple-binder-repo-2/master", withCredentials: true, readyState: 2, onopen: null, onmessage: null, …}

on the first look the seem to behave the same.

[betatim]

Not sure I understand what is going wrong yet. As I was poking around things I noticed that we have https://gke.staging.mybinder.org/build/gh/binder-examples/requirements/master?binder_launch_host=https%3A%2F%2Fmybinder.org%2F. Does someone know why the binder_launch_host query parameter gets added (and where)?

[manics]

Sounds like it's related to this feature added to support Gesis:

• https://github.com/jupyterhub/binderhub/pull/1031
• https://github.com/jupyterhub/mybinder.org-deploy/pull/1284#issuecomment-560273078

[bitnik]

https://github.com/jupyterhub/mybinder.org-deploy/pull/1307:

https://github.com/jupyterhub/mybinder.org-deploy/blob/9cba7bf6f282db3ed2939bd3a47770f3fc5720b5/images/federation-redirect/app.py#L195

It is used to set BINDER_LAUNCH_HOST env variable correctly. If user is redirected from federation BINDER_LAUNCH_HOST is "https://mybinder.org/" or if user is launching a repo directly through GESIS Binder, it is "https://notebooks.gesis.org/binder/".

This env variable is used for the value of "Copy Binder Link" button. And also in https://github.com/manics/jupyter-offlinenotebook extention. That's all I remember.

[bitnik]

Not sure I understand what is going wrong yet.

@betatim does it mean that you can reproduce the error?

[betatim]

What I meant is that I didn't even get close to trying to reproduce it because I ended up with other questions first. I do get the error Min reported if I set withCredentials to true.

I will try again today with a locally running federation redirector which only contains Gesis. Otherwise trying to find a repo that gets redirected to the place you want is a bit tedious :)

Another thing I am unsure about: does the result you get from typing things into the console depend on which website you opened the console for? For example "open console on mybinder.org" vs "open console on gesis.mybinder.org" vs "..."?

[minrk]

FWIW, I'm able to reproduce this currently at https://staging.mybinder.org/v2/gh/binderhub-ci-repos/requirements/master which is currently binned to gesis. I can see it with Chrome and Safari, but not Firefox. Safari gives informative error messages about withCredentials, while Chrome only says "Failed to construct event stream".

When I run the federation-redirector on its own locally (on localhost), there do not appear to be issues. The main difference I can think of is that it's on http not https.

[minrk]

It doesn't make sense to set the binder_launch_host param on all redirected URLs (mostly build URLs), so I'm going to try reverting that part of #1307 to see if it helps.

[betatim]

On Firefox (74.0b4) and Chrome (79.0.3945.130 (Official Build) (64-bit)) I don't see any errors for https://staging.mybinder.org/v2/gh/betatim/first-binder/master.

[minrk]

@betatim I didn't see errors on my very first test with Chrome 79 this morning, but after restarting to update to 80 I did!

[bitnik]

On firefox (68.5.0esr (64-bit)) and chrome (Version 80.0.3987.106 (Official Build) (64-bit)) I don't see any errors neither. Does error occur when only building or also when launching?

[minrk]

Build or launch doesn't matter. Attempting to establish the eventstream connection is failing, so even if it's an invalid ref, it fails.

[minrk]

Removing the url param didn't appear to change things, nor did reverting the redirect to 302.

I am going to note that gesis is the only one that is indeed outside the mybinder.org domain, so maybe that's the difference. This also means it's the only one with two redirects: mybinder.org -> gesis.mybinder.org (inside gke) -> notebooks.gesis.org/binder (happens at gesis). It is always that second redirect that fails, not the first, so I guess the issue resides in the nginx redirect on the gesis side? Or maybe it's purely in the browser policy applying to this one and only truly cross-domain request?

Maybe withCredentials is implicitly true because the initial request is not cross-origin and the default credentials mode is same-origin? I'm flailing a bit here.

[betatim]

Now on Chrome 80 https://staging.mybinder.org/v2/gh/betatim/first-binder/master still works for me after restarting.

OVH also has its own domain .mybinder.ovh, but I think the difference is that that only gets used when we launch the pod?

[minrk]

Based on the error message, this appears to be an issue with the CORS headers on the redirect handler. @bitnik could you change the header on the /build/ redirector to:

add_header "Access-Control-Allow-Origin"  "https://mybinder.org";
add_header "Access-Control-Allow-Credentials"  "false";

I don't think this should affect the real service, just the redirection. But maybe it will affect things like spaCy and thebelab...

[minrk]

@betatim yeah, the redirect to the Hub is not an issue. Turing Hub is also another domain. Redirect for a new page isn't a CORS request like the event stream is.

[bitnik]

@minrk done.

[manics]

There's a warning of breaking changes in Chrome 80 relating to the SameSite attribute https://www.theregister.co.uk/2020/01/30/google_chrome_80_cookies/

[manics]

Note at the end of the article:

Though Chrome 80 is still slated to ship on February 4, 2020, Google now says, "The SameSite-by-default and SameSite=None-requires-Secure behaviors will begin rolling out to Chrome 80 Stable for an initial limited population starting the week of February 17, 2020."

sounds like not all Chrome 80 users will get the new feature immediately, this might explain why it's working for some.

[minrk]

@bitnik silly me, can you make it staging.mybinder.org for now while we are testing there?

[bitnik]

@minrk I also just realized it and changed it

[minrk]

Hunch while looking at the steps: we set the host cookie during the redirect process. I wonder if that has an effect on why it's behaving like the credentials flag is true.

[minrk]

ok, getting new and reasonable(ish) error:

Cross-origin redirection to https://notebooks.gesis.org/binder/build/gh/binderhub-ci-repos/requirements/whatevs denied by Cross-Origin Resource Sharing policy: Credentials flag is true, but Access-Control-Allow-Credentials is not "true".

can you now toggle the Allow-Credentials to "true"?

[minrk]

ok, it works now. Can you finally switch back to Access-Control-Allow-Origin: *, leaving credentials true?

[minrk]

okay, that's back to denied by Cross-Origin Resource Sharing policy: Cannot use wildcard in Access-Control-Allow-Origin when credentials flag is true.

Try with this:

add_header Access-Control-Allow-Origin $http_origin;
add_header Vary Origin;

This is equivalent to Access-Control-Allow-Origin *, but evaluated with a single match per request. Should get the permissiveness we had before, but satisfying the policy issues per-request.

[minrk]

Still a mystery: why are some browsers sometimes sending requests with the credentials flag set? My guess is that it's a cross-origin redirect and the initial request is sent same-origin. Maybe it's relevant that we are setting a header in the middle? The exact circumstances are unclear to me.

[minrk]

It's working on staging, thanks @bitnik! Restoring gesis to rotation in #1356

[minrk]

Not sure about closing this issue or not: we still have a mystery of why requests are being made with credentials to some endpoints and why that's offensive to some browsers (Safari) and other browsers some of the time (Chrome) and not others (Firefox). But we've at least worked out how to keep things working.

[bitnik]

thanks a lot @minrk for fixing this!

[betatim]

A mega debugging thread! I'd suggest closing this and opening a new issue which starts with a summary like this and points here.

[meeseeksmachine]

This issue has been mentioned on Jupyter Community Forum. There might be relevant details there:

https://discourse.jupyter.org/t/access-control-allow-origin-header-is-missing-on-mybinder-org/3367/5

[betatim]

I just came across https://mybinder.org/v2/gh/geomar-tm/python-intro-201804/v2.0.0?urlpath=lab which launches on gesis (at the moment) and reliably and instantly says "can't connect to event stream" for me with Firefox 74.0b4 (64-bit). This is what I see in the console:

A few minutes later and on Chrome it "just works"

Though it now gets stuck on this:

Neither "clear workspace" no "keep waiting" do anything other than take me back to the error.

[betatim]

The weird thing is that the launch via Chrome works right now, but even 5min later on Firefox it continues to be broken.

[betatim]

For even more fun: https://mybinder.org/v2/gh/geomar-tm/python-intro-201804/v2.0.0 doesn't work but https://mybinder.org/v2/gh/geomar-tm/python-intro-201804/master works.

The headers look right in both cases.

[meeseeksmachine]

This issue has been mentioned on Jupyter Community Forum. There might be relevant details there:

https://discourse.jupyter.org/t/access-control-allow-origin-header-is-missing-on-mybinder-org/3367/9

[betatim]

https://mybinder.org/v2/gh/ogrisel/notebooks/master?filepath=sklearn_demos/ames_housing.ipynb is another link that consistently doesn't work for me :-/

[consideRatio]

I've never really digged down into CORS stuff, so I decided to read up and learn HTTP matters and CORS matters thoroughly. I'm certainly not done yet, but I figured I'd link two resources of relevance to me already!

1. A longer thorough but readable text about HTTP in general: https://developer.mozilla.org/en-US/docs/Web/HTTP
2. Nice video about CORS specifically: https://www.youtube.com/watch?v=Ka8vG5miErk

I've already started to become aware of a lot of complexity, such as requests from javascript sometimes goes directly to the server, but sometimes first does a preflight request to verify if the actual request is of a kind that will be considered valid, and that the web-server needs to also respond positively to that. Ah... I feel happy to set out to learn about this thoroughly instead of remaining confused whenever this comes up! :D

[consideRatio]

@betatim

Another thing I am unsure about: does the result you get from typing things into the console depend on which website you opened the console for? For example "open console on mybinder.org" vs "open console on gesis.mybinder.org" vs "..."?

I think the browser is aware of this, and will act differently, while it may look the same for the web server which isn't passed this information about the client side. Perhaps the browser may or may not make an OPTIONS request depending on this also. Hmm...

This was exemplified and pointed out specifically in the video i linked above btw :)

[consideRatio]

UPDATE: Ooops this was already suggested by @betatim: https://github.com/jupyterhub/mybinder.org-deploy/issues/1344#issuecomment-585810935

---

Is it really okay that gesis limits itself to GET requests? Then any OPTIONS request will be denied I assume, and those could be relevant I guess?

https://github.com/gesiscss/orc/blob/c20f98828afe30173c4f8685d13be37ea5745648/load_balancer/sites-available/gesis_mybinder#L27-L39

    location / {
        # https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/301
        limit_except GET {
            deny  all;
        }

...

[consideRatio]

If you make an OPTIONS request to gesis.mybinder.org, and then get 301-ed somewhere along with some headers, will those headers be combined somehow with the next request? What will the browser to with an OPTIONS response with relevant headers that suggest one goes to another page right after using a 301?

I guess in general I want to understand how response headers with a 301 status will be interpreted and used by browsers that then also may make additional web requests by following the redirection. What if that response has the same headers, are they overridden, ignored, merged, etc.

Perhaps the difference between 301 and 308 is of relevance? I think we should use 308 instead of 301 redirects, and also accept both GET and OPTIONS here: https://github.com/gesiscss/orc/blame/master/load_balancer/sites-available/gesis_mybinder.

[minrk]

@consideRatio I thought that as well, since we switched to 307 around the time the problems started happening, but switching back to 302 didn't help. 307 is still right, even though GET requests appear to be the only ones made, so there's no real difference.

[bitnik]

@betatim

• https://staging.mybinder.org/v2/gh/geomar-tm/python-intro-201804/v2.0.0?urlpath=lab

For this one you said that you got this error: Reason: CORS request did not succeed

When I search for the error (ref):

The HTTP request which makes use of CORS failed because the HTTP connection failed at either the network or protocol level. The error is not directly related to CORS, but is a fundamental network error of some kind.
In many cases, it is caused by a browser plugin (e.g. an ad blocker or privacy protector) blocking the request.

Btw this url is for me now launched at https://hub.gke.staging.mybinder.org and there I get stuck at "Keep Waiting" and "Clear Workspace"

• https://staging.mybinder.org/v2/gh/ogrisel/notebooks/master?filepath=sklearn_demos/ames_housing.ipynb

This one for me launches at GESIS without problem. Did you get the same error here?

[bitnik]

@consideRatio thanks a lot for your inputs! I will try your suggestions out, when I am really able to debug this.

[betatim]

The 'Keep waiting' thing is (I think) an unrelated problem with the JupyterLab workspace.

https://staging.mybinder.org/v2/gh/binder-examples/requirements/master and https://staging.mybinder.org/v2/gh/ogrisel/notebooks/master?filepath=sklearn_demos/ames_housing.ipynb both give me a instant "error to connect to event source"

[betatim]

You are right! Privacy badger (a browser extension I use) is blocking the request to notebooks.gesis.org because it looks like it is a tracker. Privacy badger has a bunch of heuristics it uses to decide if requests look like "trackers" or not. Need to do some digging why/what the rules are. The last time we had a problem with appearing like a tracker it was related to setting cookies and loading an image. This isn't the case here but it will be something similar (I bet).

Nice find of the reason for the error message!

Just to confirm: if I disable all "anti tracking" extensions and settings in Firefox I "just works again" :)

[bitnik]

Does anyone have any recommendation what to do here as next step?

[bitnik]

I realized that in chrome I disabled privacy badger for mybinder.org some time ago for some reason I don't remember now. I enabled it again and had the same problem (notebooks.gesis.org is blocked on staging.mybinder.org, ex URL: https://staging.mybinder.org/v2/gh/ogrisel/notebooks/mastersss) as @betatim has. Then I installed privacy badger in Firefox, just to see if it behaves the same, and it didn't behave the same, it didn't block notebooks.gesis.org on staging.mybinder.org with the same URL.

Then today I was reading this post (https://github.com/EFForg/privacybadger/wiki/Find-out-why-Privacy-Badger-is-blocking-a-domain) to understand why privacy badger is blocking notebooks.gesis.org in chrome and here is the output for notebooks.gesis.org:

**** ACTION_MAP for gesis.org
gesis.org {
  "dnt": false,
  "heuristicAction": "block",
  "nextUpdateTime": 0,
  "userAction": ""
}
notebooks-test.gesis.org {
  "dnt": false,
  "heuristicAction": "",
  "nextUpdateTime": 1565035835885,
  "userAction": "user_cookieblock"
}
notebooks.gesis.org {
  "dnt": false,
  "heuristicAction": "block",
  "nextUpdateTime": 1582943208262,
  "userAction": "user_block"
}
**** SNITCH_MAP for gesis.org
gesis.org [
  "127.0.0.1",
  "0.0.0.0",
  "github.com"
]

To be honest this didn't helped me really. Later I found this issue and because of

To follow up on the last comment: if you've had PB installed since before we added the MDFP list, your badger might have learned to block wikimedia domains before that time, and never "un-learned" to block them if you didn't reset Privacy Badger's local storage. We can fix that problem by adding a migration to Privacy Badger's startup procedure, as we have with the yellowlist.

I decided to uninstall privacy badger in chrome and install it again and see what happens. And now it doesn't block notebooks.gesis.org anymore on staging.mybinder.org.

Now I am thinking about creating an issue on https://github.com/EFForg/privacybadger/.

[bitnik]

Here is the link to the issue: https://github.com/EFForg/privacybadger/issues/2558

[arnim]

We've got a response on the Privacy badger issue, however, I'm not certain if / how fare we should go down debugging PB. Doing so slows us (in the best case) down to make the persistent BH code available in a usable fashion. In the worst case, it hinders us to do this all together since we have only so much time for this. I agree that we should put in 20x the effort so [we] can reach 1000x fewer people. Yet I wonder:

• how widespread this PB issue is,
• if there are other CORS or related issues blocking us and
• if jupyterhub/binderhub#1048 could (potentially) help without debugging PB.

Maybe this could help us to stay efficient.

[bitnik]

Btw @betatim if you still have the problem, could you also post the output of https://github.com/EFForg/privacybadger/wiki/Find-out-why-Privacy-Badger-is-blocking-a-domain into the issue? It might be helpful.

[bitnik]

I realized that in chrome I disabled privacy badger for mybinder.org some time ago for some reason I don't remember now.

Now I remember. One time I had problem with adblocker on mybinder.org, I didn't see any build/launch logs. Then I disabled blocker extensions (including Privacy Badger) for mybinder.org and turned adblocker off completely on the browser that I do tests.

And now just to test, I turned adblocker on and I don't see any launch/build logs on mybinder.org (e.g. https://mybinder.org/v2/gh/binder-examples/requirements/master). The weird thing is that if I disable adblocker only for mybinder.org, it doesn't help, I don't see any logs.

There is one more thing :) Because now I have Privacy Badger enabled for mybinder.org, I see that it is also blocking nbviewer.jupyter.org on loading page and I don't see any preview. Again I just seen it with https://mybinder.org/v2/gh/binder-examples/requirements/master

[betatim]

I think trying to understand this is worth it. Not so much because of the PB use-case (install base is probably small). However I think more and more browsers are shipping with (aggressive) ad/tracker/privacy settings on by default. At the last few updates Firefox informed me about doing more to block stuff automatically and I think Safari is also pretty aggressive. I saw a headline somewhere that Google was going to make Chrome so that it blocks most ads (except for Google whitelisted ads).

Over the last few months it has felt like the number of people complaining about "can't connect to eventstream" errors has increased. This was a mystery to me until we discovered this. Now I am wondering if the increased reports are related to the more aggressive ad blocker settings that browsers ship with by default.

I am wondering if it makes a difference if the "tracker domain" (in our case nbviewer or gesis) sets a cookie or not. The way the Facebook like button was used to track people is by embedding it in many many pages and having a unique cookie set on the domain that the like button was served from. This way evil-button.com could track you visiting fun.com and happyplace.de (if they both embedded the button).

I think if we fall into this pattern we need to reconsider how we do things because it is a losing battle. My prediction is that browsers will get ever more aggressive at blocking requests that match this pattern because they can't tell the difference to trackers.

Right now we serve the images and HTML from mybinder.org (via a proxy) but open an event stream from another domain (notebooks.gesis.org, gke.mybinder.org, etc). When the pod has launched we redirect the user to a different domain.

My hunch is that the fact that a website served from mybinder.org is making requests to notebooks.gesis.org (potentially with cookies?) is what triggers the tracker detection. There is no way to know that notebooks.gesis.org isn't track-all-notebooks.evil.com.

Three ideas:

1. figure out if there is a way to open the event stream without sending "any" information (no cookies, referrer, etc) and see if that (doesn't) trigger the tracker detection
2. proxy the event stream through the federation proxy (traffic goes from browser <-> fed proxy <-> actual site). More traffic on the proxy but everything appears to be on mybinder.org
3. not redirect gesis.mybinder.org to notebooks.gesis.org for the event stream endpoint but only for the other paths. It seems like that using a subdomain doesn't trigger the tracker detection (yet?)

[psychemedia]

In terms of browser extensions causing issues with MyBinder, using Ghostery Chrome extension (on a Mac), and mybinder.org as an untrusted site, builds often(?; not sure if it's always; will keep a close watch...) don't actually launch for me and build log remains blank.

Ghostery only reports one tracker blocked, and that's Google Analytics. (I'm on a free plan; not sure if I can get more detailed reports on what Ghostery does / sees if I sign up, subscribe etc.)

Works fine if I trust the site. Also note that this means that try.jupyter.org can fail as it tries to launch using MyBinder. (Another issue is I regularly see Too many users running https://github.com/ipython/ipython-in-depth! Try again soon. when testing from the try.jupyter.org link.)

[psychemedia]

With Ghostery enabled and site not trusted, if i try:

• https://mybinder.org/v2/gh/ipython/ipython-in-depth/master?filepath=binder/Index.ipynb: blank log, doesn't work;
• https://ovh.mybinder.org/v2/gh/ipython/ipython-in-depth/master?filepath=binder/Index.ipynb: works;
• https://gesis.mybinder.org/v2/gh/ipython/ipython-in-depth/master?filepath=binder/Index.ipynb: works;
• https://gke.mybinder.org/v2/gh/ipython/ipython-in-depth/master?filepath=binder/Index.ipynb: works;

[betatim]

Right now https://mybinder.org/v2/gh/ipython/ipython-in-depth/master?filepath=binder/Index.ipynb connects to gke.mybinder.org (for me).

[manics]

It's going to hub.mybinder.turing.ac.uk for me.

Edit: Tried in a second browser window and this time it went to hub.gke.mybinder.org

[manics]

To make testing cross-domain request easier is there any chance of adding an external domain as an alias for staging.mybinder.org, e.g. 104.197.11.66.xip.io ? Then every visit to the BinderHub at 104.197.11.66.xip.io would go to a different domain.

[psychemedia]

So even if I trust MyBinder site with Ghostery now, the top level mybinder.org site is broken. Launching from X.mybinder.org is fine.

[betatim]

What do you mean with "adding an alias"? We have notebooks.gesis.org in the staging federation so every second request should go to a "not subdomain" host.

[manics]

Gesis has a weight of 1, compared to 4 for gke.staging.
https://github.com/jupyterhub/mybinder.org-deploy/blob/29f4b2100018818134ba732641791156c1b5d7a4/config/staging.yaml#L128.
Doesn't that imply only 20% of requests will go to Gesis?

[bitnik]

@manics I was also confused (https://github.com/jupyterhub/mybinder.org-deploy/pull/1363)

The size of the weight doesn't do anything (as long as it is above zero) when we are using rendezvous routing.

https://github.com/jupyterhub/mybinder.org-deploy/blob/29f4b2100018818134ba732641791156c1b5d7a4/images/federation-redirect/app.py#L189-L193

[bitnik]

https://github.com/jupyterhub/mybinder.org-deploy/issues/1344#issuecomment-593385623:

not redirect gesis.mybinder.org to notebooks.gesis.org for the event stream endpoint but only for the other paths. It seems like that using a subdomain doesn't trigger the tracker detection (yet?)

I have just made changes for this, so /build/ endpoint is not redirect anymore (e.g. https://gesis.mybinder.org/build/gh/binder-examples/requirements/master), so redirection happens on the same domain. And now I don't get "Failed to connect to event stream" error anymore, when Privacy Badger is enabled (my Privacy Badger recently learned to block notebooks.gesis.org on staging.mybinder.org). @betatim could you confirm this?

I know this doesn't resolve problems regarding privacy extentions in general, but at least brings GESIS Binder at the same level with other members in federation. So maybe we can bring GESIS Binder back to mybinder.org?

Edit: in order to test this change don't forget to use https://staging.mybinder.org/

[betatim]

I think we can close this now. The immediate problem is solved and the issue has gotten very long. I propose we open a new issue when we want to work on other privacy extension issues.

Thanks a lot for all the brains and hours spent on this everyone!