Repo2docker: "start" file doesn't play nicely with "-E" if executable permissions aren't set

[neuromusic]

trying to get a gallery of notebooks working well with repo2docker

fwiw, the repo is here, currently on the branch "reorg"

https://github.com/AllenInstitute/brain_observatory_examples/tree/reorg

I need to specify an environment variable that is stable for all of the notebooks, and I've defined this using the "start" file:

#!/bin/bash
export BRAIN_OBSERVATORY_MANIFEST=~/brain_observatory_data/manifest.json

exec "$@"

This works beautifully when running repo2docker . & I can confirm that the environment variable is set properly.

However....

I'm not 100% sure that all of these notebooks work together (same set of dependencies, etc). I would like to spin them up in repo2docker in editable mode, make sure they all run, make any changes necessary, shut it down, then commit any changes back to the repo.

However, when I run this in editable mode repo2docker -E ., I get an error and the docker container doesn't run:

Step 33/35 : RUN chmod +x "./start"
 ---> Running in 44557a8fa7e5
Removing intermediate container 44557a8fa7e5
 ---> 9b35a30889e3
Step 34/35 : ENTRYPOINT ["./start"]
 ---> Running in 5ab226e6dc83
Removing intermediate container 5ab226e6dc83
 ---> 8e17f62aace0
Step 35/35 : CMD ["jupyter", "notebook", "--ip", "0.0.0.0"]
 ---> Running in af6f7ad1d36f
Removing intermediate container af6f7ad1d36f
 ---> 356a160c50a3
{"aux": {"ID": "sha256:356a160c50a3c985d6a24e96f5c4e56201ec1105e7b471ca57a1629ba62224e9"}}Successfully built 356a160c50a3
Successfully tagged r2d-2e1542325997:latest
Traceback (most recent call last):
  File "/home/justink/anaconda3/lib/python3.6/site-packages/docker/api/client.py", line 229, in _raise_for_status
    response.raise_for_status()
  File "/home/justink/anaconda3/lib/python3.6/site-packages/requests/models.py", line 935, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: http+docker://localhost/v1.38/containers/b26d31fbb36e7b3e761fe86ae95ecfa1006840e9fc5b8d18be2f16b587f5543c/start

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/justink/anaconda3/bin/repo2docker", line 11, in <module>
    load_entry_point('jupyter-repo2docker', 'console_scripts', 'repo2docker')()
  File "/home/justink/Code/repo2docker/repo2docker/__main__.py", line 7, in main
    f.start()
  File "/home/justink/Code/repo2docker/repo2docker/app.py", line 756, in start
    self.run_image()
  File "/home/justink/Code/repo2docker/repo2docker/app.py", line 572, in run_image
    container = self.start_container()
  File "/home/justink/Code/repo2docker/repo2docker/app.py", line 634, in start_container
    environment=self.environment
  File "/home/justink/anaconda3/lib/python3.6/site-packages/docker/models/containers.py", line 772, in run
    container.start()
  File "/home/justink/anaconda3/lib/python3.6/site-packages/docker/models/containers.py", line 374, in start
    return self.client.api.start(self.id, **kwargs)
  File "/home/justink/anaconda3/lib/python3.6/site-packages/docker/utils/decorators.py", line 19, in wrapped
    return f(self, resource_id, *args, **kwargs)
  File "/home/justink/anaconda3/lib/python3.6/site-packages/docker/api/container.py", line 1064, in start
    self._raise_for_status(res)
  File "/home/justink/anaconda3/lib/python3.6/site-packages/docker/api/client.py", line 231, in _raise_for_status
    raise create_api_error_from_http_exception(e)
  File "/home/justink/anaconda3/lib/python3.6/site-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation)
docker.errors.APIError: 400 Client Error: Bad Request ("OCI runtime create failed: container_linux.go:348: starting container process caused "exec: \"./start\": permission denied": unknown")

I'm running repo2docker on master.

Any thoughts?

[neuromusic]

oh! and repo2docker -E . works perfectly without the "start" file

[choldgraf]

hmmmm, I know that the presence of a start file basically creates an ENTRYPOINT line in the dockerfile that builds the image: https://github.com/jupyter/repo2docker/blob/58cbe1797dcd318f79244767497b5b757ea7a181/repo2docker/buildpacks/base.py#L138 but I dunno why this would cause this error.

Out of curiosity, does this error happen if you give any start file? E.g., one that just echos something?

[neuromusic]

Yeah, it threw the error without the echo. I added it trying to debug since
the docs indicated it was needed.

On Thu, Nov 15, 2018, 18:22 Chris Holdgraf [e-mail] wrote:

hmmmm, I know that the presence of a start file basically creates an
ENTRYPOINT line in the dockerfile that builds the image:
https://github.com/jupyter/repo2docker/blob/58cbe1797dcd318f79244767497b5b757ea7a181/repo2docker/buildpacks/base.py#L138
but I dunno why this would cause this error.

Out of curiosity, does this error happen if you give any start file?
E.g., one that just echos something?

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/jupyter/repo2docker/issues/473#issuecomment-439259663,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABMBrwrmYYdE5eEF_sGlFgEPaiPsbG53ks5uviFUgaJpZM4Yj5dD
.

[betatim]

What are the permissions of the start file set to?

The "permission denied" in the (unfortunately gigantic) error message suggests that it does not have the executable bit set. Try chmod u+x start to see if that fixes it. Hopefully that fixes it.

For start and postBuild we try to ensure that the executable bit is set, no matter what the permissions really are. Apparently we are missing that somewhere.

[neuromusic]

ok, running chmod u+x start before repo2docker -E . does seem to work, but it's not going to persist through git

There is (apparently) a step where repo2docker tries to run the chmod on the start file ...

Step 33/35 : RUN chmod +x "binder/./start"
 ---> Running in 49c51fc30f99

Is this happening from within the Docker container? I only have a cursory understanding of docker.... can you set the permissions of a mounted directory from the within the docker container?

[betatim]

but it's not going to persist through git

Why not? I think git (now) records the executable flag in some special case kinda way.

When creating the image repo2docker first copies the contents of the repo into the build context, then as part of the build it is setting the executable flag inside the container. Once that is done the start file is executable. However then we get ready to start the container and mount the local directory over the copy of the repo contents and run stuff. But because the executable flag was only set on the copy inside the image (which is now hidden because of the mount) we are back to the start script not being marked as executable any more.

We added the "make executable" step a while back to help people who use GitHub UI to create files. They can do everything except mark files as executable so we decided we'd be helpful and do it for you. Now that we let people mount directories in we get this inconsistent behaviour. my suggestion is that we perform a check before building the image to see if start and postBuild are executable. If not we can warn the user. Potentially if they use the -E flag we even error out if they aren't executable. What do you think of that as a way forward?

[neuromusic]

oh, I see. you're right. Gitkraken simply didn't notice the permissions change, but I could commit it with vanilla git. that's what I get for using a GUI.

checking and erroring out makes sense to me.

[betatim]

Do you want to give implementing the warning part a go (or take on the whole thing)?

[betatim]

Somewhere around https://github.com/jupyter/repo2docker/blob/1cef296b00478000f431b53fbee55d8281235de9/repo2docker/app.py#L715 is where I'd implement the checks and emit the warnings. It is just after we fetched the contents of the repository.

[neuromusic]

awesome. I'll try to dive in the next week or two, before my new job starts

[yuvipanda]

Have you found any time to work on this since, @neuromusic? Just checking in, no pressure!