Binderhub: RStudio error "500: Internal Server Error" with rocker image

[januz]

When trying to launch RStudio in binder, I get the error "500: Internal Server Error". I thought it was related to this issue, but while this one seems to be resolved, my problems persist.

The additional problem I lined out in the other issue (i.e., browser crashing when trying to open an R Jupyter notebook) seems to be related to my browser (Vivaldi). When I try it in Chrome, this problem doesn't persist. The "500: Internal Server Error" happens in both browsers.

Here's my repository:

https://mybinder.org/v2/gh/idea-labs/comsldpsy/master?urlpath=rstudio

As mentioned in the other issue, everything seemed to work perfectly last week. The problems occurred only today. Did you do some changes to the server over the weekend? How high is the probability that those changes happen to make a binder image not run anymore? (I'm asking because I want to post the link for other researchers to inspect our analysis, and I'm worried that things won't run anymore, without me knowing).

Thanks for your help!

[choldgraf]

Hmmm - could you try pushing a change to that repo so that repo2docker re-builds it? Perhaps that's the issue...

[januz]

unfortunately, rebuilding after push doesn't change anything

[betatim]

mybinder.org/v2/gh/idea-labs/comsldpsy/master?urlpath=rstudio

Your repository has a Dockerfile in the top level directory. This means that will be used to build the image used on mybinder.org. We recommend that you only use one if you have tried all other ways as we can not provide a lot of support when it breaks. This is because you could be doing "anything" in the Dockerfile. To help debug the issue I'd check your Dockerfile installs all the required bits and build it with the latest version of repo2docker locally.

When I launch your repo without requesting the RStudio frontend I see that it is empty, this means something at a lower level of preparing the image is going wrong :-/

[januz]

We recommend that you only use one if you have tried all other ways

The repository is for a reproducible research compendium, so I want to / have to use the Dockerfile to reproduce the exact computational environment.

When I launch your repo without requesting the RStudio frontend I see that it is empty, this means something at a lower level of preparing the image is going wrong :-/

The user folder is empty, but all the code and data are contained in an R package, which is installed using the Dockerfile. When one opens a terminal instead of RStudio and runs R in it, one can load the package and re-run the analysis. But something goes wrong when trying to use RStudio.

To help debug the issue I'd check your Dockerfile installs all the required bits and build it with the latest version of repo2docker locally.

I'll do that tomorrow morning. What I just tried, was to build the Docker image locally or to download it from DockerHub where I also build it from the same GitHub repository. Indeed, downloading it from DockerHub gives the same error, so you're right that it should be something related to preparing the container. I just don't get what it could be as everything (building the container locally, running it from DockerHub, running it on mybinder.org) was working at the end of last week and I didn't change anything with regards to the Dockerfile.

I'll test more tomorrow morning and let you know!

[januz]

Hm, unfortunately, I don't have conclusive results yet. To be honest, I'm quite puzzled over what is going on. Here what I did so far:

1. build container locally from directory that contains cloned Github repository --> RStudio works

2. build container using repo2docker with the same URL I use on mybinder.org --> RStudio works

   jupyter-repo2docker https://github.com/idea-labs/comsldpsy

3. build container from BinderHub image that is build from the same repo --> RStudio DOES NOT work

   docker run -p 8888:8888 idealabsffm/comsldpsy

I don't really know what to make out of this. Especially that 2 works is confusing to me as this is the process that should be happening when running on mybinder.org, right?

[januz]

A further test I just did was to fork the repo and reset it to the first commit to see whether changes I introduced afterwards are at the core of the repo not being run successfully on mybinder.org. Unfortunately, I get the same error with this repository:

https://github.com/januz/comsldpsy-1/tree/binder-check

I'm really clueless what to make out of this. If you have any ideas/suggestions, please let me know. Thanks!

[betatim]

I just inspected the build logs from launching https://github.com/idea-labs/comsldpsy on mybinder.org

This is it:

[E 22:11:36.057 NotebookApp] Uncaught exception GET /user/idea-labs-comsldpsy-2ju41dxf/rstudio/ (31.10.137.178)
    HTTPServerRequest(protocol='https', host='hub.mybinder.org', method='GET', uri='/user/idea-labs-comsldpsy-2ju41dxf/rstudio/', version='HTTP/1.1', remote_ip='31.10.137.178')
    Traceback (most recent call last):
      File "/srv/venv/lib/python3.5/site-packages/tornado/web.py", line 1592, in _execute
        result = yield result
      File "/srv/venv/lib/python3.5/site-packages/tornado/gen.py", line 1133, in run
        value = future.result()
      File "/usr/lib/python3.5/asyncio/futures.py", line 293, in result
        raise self._exception
      File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step
        result = coro.send(None)
      File "/srv/venv/lib/python3.5/site-packages/nbserverproxy/handlers.py", line 95, in get
        return await self.http_get(*args, **kwargs)
      File "/srv/venv/lib/python3.5/site-packages/nbserverproxy/handlers.py", line 461, in http_get
        return await self.proxy(self.port, path)
      File "/srv/venv/lib/python3.5/site-packages/nbserverproxy/handlers.py", line 440, in proxy
        return await super().proxy(self.port, path)
      File "/srv/venv/lib/python3.5/site-packages/nbserverproxy/handlers.py", line 242, in proxy
        url_path_join(self.base_url, 'proxy', port)
      File "/srv/venv/lib/python3.5/site-packages/notebook/utils.py", line 46, in url_path_join
        final = pieces[-1].endswith('/')
    AttributeError: 'int' object has no attribute 'endswith'

this is the same error we got from nbserverproxy we got with the old version of repo2docker. We then fixed nbserverproxy and released a new version of it and nbrsessionproxy (this is what repo2docker actually installs).

I am not quite sure why you end up with the broken version of nbrsessionproxy on mybinder.org but a good one when building locally. Especially as in your Dockerfile you do not even mention nbrsessionproxy, which means it is something installed in the rocker/binder image. However you are using a tagged version of it so that image shouldn't be different between your machine and mybinder.org either.

It might be worth following up with the rocker image creators to find out what version of nbrsessionproxy and nbserverproxy they have in their image. I'd assume that the one you are using with a tag from before the fix was made contains the version with the bug. Maybe releasing a new tag of the rocker image is what is needed.

[betatim]

1. build container from BinderHub image that is build from the same repo --> RStudio DOES NOT work

What exactly do you mean with this? There should be no way for you to obtain the image that mybinder.org created for you when you build a repository.

[januz]

It might be worth following up with the rocker image creators to find out what version of nbrsessionproxy and nbserverproxy they have in their image. I'd assume that the one you are using with a tag from before the fix was made contains the version with the bug. Maybe releasing a new tag of the rocker image is what is needed.

Thank you so much for looking into this! I will experiment a bit more with using different rocker images. Maybe I can pin down where the problem is introduced.

What exactly do you mean with this? There should be no way for you to obtain the image that mybinder.org created for you when you build a repository

Sorry, I meant to write "DockerHub", not "BinderHub". So when using Docker with an image that is automatically build from my repo on DockerHub, I get the same error.

[januz]

Okay, so I now tried out using other rocker containers as the base image and copying the instructions from the rocker/binder Dockerfile (https://github.com/rocker-org/binder/blob/master/3.5.0/Dockerfile) to my Dockerfile. That works!

This is my test repository: https://github.com/januz/comsldpsy-1

So, as you assumed, only when I use the rocker/binder image, I have the described problem with using RStudio in Binder. The weird thing is that even when I use the exact Dockerfile from the docker/binder image (i.e., basing off of rocker/geospatial and the exact same instructions), plus my few lines, it works. Only when building with rocker/binder as base image, it does not.

I don't understand all the dependencies well enough, but I noticed that there was a commit in the rocker/binder repository that was done on the weekend, i.e., in between the time Binder worked with my original Dockerfile (based on rocker/binder) and it not working anymore: https://github.com/rocker-org/binder/commit/42b4779855ca8517f1561192820cb801fac6fa0d

Maybe this commit adding post push hooks has something to do with it?!

[betatim]

Unfortunately the only thing I can say about rocker is: I've never used it and have no idea how they do stuff :-/

Without digging into it my guess would be that they updated a docker image on dockerhub but did not update the tag? Say a week ago you pulled rocker/binder:3.4 and today you pull rocker/binder:3.4 on a different machine and because of the update they aren't actually the same image. Not sure how to confirm/check this though.

Maybe @cboettig has an idea/time to dig into this. Otherwise I'd open an issue on the rocker/binder repo and see if someone there has an idea.

[januz]

@betatim Thank you so much for your help! I will take this over to the rocker/binder repo...

[cboettig]

@januz Thanks for the bug report, and @betatim, thanks for the help!

When I either build locally or just fork @januz 's repo, my-binder builds fine. I'm still not clear where an old nbproxysession could have been coming from. @betatim is there a simple way to break the cache to trigger my-binder to rebuild?

Just a note re:

my guess would be that they updated a docker image on dockerhub but did not update the tag?

All Rocker images from Docker Hub are set on automated builds, so what you see in the Dockerfiles on GitHub should match what you get from DockerHub. We can't update the image without updating the tag.

Say a week ago you pulled rocker/binder:3.4 and today you pull rocker/binder:3.4 on a different machine and because of the update they aren't actually the same image. Not sure how to confirm/check this though.

You can check if an image you pulled a week ago and an image you pulled today are binary-identical by checking the image ID (hash) (e.g. docker images, note that rebuilding the identical Dockerfile in Docker does not generally create an identical hash, since upstream sources may change). Of course this isn't rocker-specific, we try and be as vanilla-docker as we can.

[betatim]

@betatim is there a simple way to break the cache to trigger my-binder to rebuild?

Make a new commit in the repo you want to build.

However, I think/my guess is that some docker layer in the process is being cached sometimes and sometimes it is getting executed (and the result of executing that statement of the Dockerfile results in different layers at different times. The cached ones get the old version of nbserverproxy and the ones being executed now get the current version.

Overall I don't know where to look for further clues and will point to the "if you are using your own Dockerfile you void your repo2docker warranty" to become a lurker in this murder mystery :)

[cboettig]

@betatim Thanks, good to know any commit breaks binder cache, and good point that binder's docker command may still be caching. In Binder's Docker build call, do you build with docker build --pull? In this case, that should fix things I believe.

And yeah, I need to play with the vanilla binder more and see if I can swap over to that; previously had been unable to resolve finicky LaTeX and font issues.