Hydrogen: Docker proposal

[max-sixty]

Follow up to https://github.com/nteract/hydrogen/pull/524

I think the simplest way is a dockerfile:

FROM python:2
ARG TINI_VERSION=v0.10.0
ADD https://github.com/krallin/tini/releases/download/${TINI_VERSION}/tini /tini
RUN chmod +x /tini
RUN pip install jupyter_kernel_gateway
ENV KG_LIST_KERNELS=True

A docker-compose:

version: '2'
services:
  jupyter:
    context: .
    entrypoint: ["/tini", "--"]
    command: ["jupyter", "kernelgateway", "--ip=0.0.0.0", "--port=8888"]

And a kernel settings default:

[{"name": "docker", "options": {"baseUrl": "http://localhost:8888"}}]

I'd propose this replaces basically all the README from L250-580. But probably I'm missing something and there's useful stuff there?

Questions:

• Do we need a docker-compose file, or should we just the commands in the dockerfile?
• Should we map the current path by default? This would allow you to run the code you're working on
• What else are we missing?

[lgeiger]

Thanks for opening the issue. Unfortunately I don't know much about Docker.

@tiangolo May be able to answer your questions.

[tiangolo]

I think that would be a cool addition to the README.md as the simplest case, to just have immediate external execution, in a case that the user can give up other Docker features or configurations.

...But in that case, wouldn't it be easier to use a Jupyter Docker stack?

They already have tini and everything (including most of jupyter_kernel_gateway's dependencies, so the build and installation would be quite fast). For example, with the minimal-notebook the setup would be something like:

A Dockerfile with:

FROM jupyter/minimal-notebook

RUN pip install jupyter_kernel_gateway

EXPOSE 8888
CMD ["jupyter", "kernelgateway", "--KernelGatewayApp.ip=0.0.0.0", "--KernelGatewayApp.port=8888"]

And then build and run with:

docker build -t my/kernel-gateway .
docker run -it --rm -p 8888:8888 my/kernel-gateway

Or the equivalent docker-compose.yml, probably both options. I think that having both options helps covering more use cases, and users can more easily copy - paste and adapt what they need (omiting what they don't need). I think this would answer question 1.

Note: Taken and modified from the official Jupyter Kernel Gateway docs.

---

The kernel settings wouldn't necessarily be like you specified above, as many users won't run that from a Linux machine but rather from Windows or Mac, and in those cases it will mostly be a Virtual Machine with a local IP (as 192.168.99.100).

I think it's better to explain how it works so that everyone can use and adapt it to their needs. It is currently explained here.

I'm not sure if I understood the question, but I think this would answer question 2.

---

I think something like what you propose would be a good addition to the README.md for the simplest cases, just having a containerized environment, but not a replacement for everything between lines 250 to 280 as there are many different use cases not covered:

• In 250, the instructions are general, not just for Docker. Those instructions would work in local machine on the same network, for example a Raspberry Pi (I've done precisely that).

• In 268 is an explanation of how it all works, so that users can understand and adapt to their needs.

• In 278 is a general purpose simple Dockerfile that could be easily adapted to everyone's needs. It doesn't assume a specific Docker Entrypoint, init (PID 0) process, etc. It could be adapted for production systems (not just a local environment) using, e.g. Supervisor. For example, see this base Flask image. In the case of tini, that image would have conflicts.

• I won't describe each following section, but then, after that, there are commands and instructions for pure Docker commands and docker-compose commands (to help all the users).

• The next big section: Custom kernel connection, although won't probably be the most common case now that Hydrogen supports Kernel Gateways, is still a feature of Hydrogen. And there could be people depending on that feature for their current workflow. So it should probably be kept for backward compatibility.

...this should answer question 3.

---

I just got curious and now I want to test the simple approach with the Jupyter Docker Stacks (that is just a "simplification" of your proposal).

@MaximilianR Do you think that adding instructions to the README.md with a Jupyter Docker Stack would document your current use case? I think I can easily add that to the README.md (it's almost already written in this issue :laughing: ). Or do you want to take a stab at it and create a PR?

[max-sixty]

Thanks for the response @tiangolo !

...But in that case, wouldn't it be easier to use a Jupyter Docker stack?

Yes, good idea.

Or the equivalent docker-compose.yml, probably both options.

Agreed

The kernel settings wouldn't necessarily be like you specified above, as many users won't run that from a Linux machine but rather from Windows or Mac, and in those cases it will mostly be a Virtual Machine with a local IP (as 192.168.99.100).

I use Docker for Mac and localhost worked for me. What makes you think you need the IP?

So it should probably be kept for backward compatibility.

Your call, but there is a lot there! Might be simpler as a link to a 'Further detail' / 'Previous implementation' page. For a first time user, there's a lot of detail that's not really needed, and makes it look arduous to set this up when it's surprisingly simple. I'm sure there are several good approaches for making it simpler, though.

I think I can easily add that to the README.md (it's almost already written in this issue 😆 ). Or do you want to take a stab at it and create a PR?

If you want to add it I would be honored! If you prefer me to, I'm happy to take a stab in the near future.

[tiangolo]

Excellent @MaximilianR !

Let's try with a Jupyter Docker stack.

I use Docker for Mac and localhost worked for me. What makes you think you need the IP?

Interesting that it just works in Mac! I didn't know. In Docker for Windows, and in Docker Toolbox (at least in Windows) it doesn't work like that, the VM gets an IP in the local (VM) network, so you have to use that IP (it is not bound to the localhost).

On the other side, if the user is connecting to a Docker running in a different machine in the same network, the IP would also be different.

Your call, but there is a lot there! Might be simpler as a link to a 'Further detail' ...

You're right. It is a lot indeed. Maybe we can split it in a separate file KERNEL_CONNECTION.md and link to it from the main README.md as with the plug-ins.

---

So, the steps to do would be:

• Move everything from the section Custom Kernel Connection to a file KERNEL_CONNECTION.md and link to it from that section with a short note explaining what it is and that the easier method would probably be the Kernel Gateway "documented above".

• In the Example Kernel Gateway section add a sub-section with instructions on how to set it up with a Jupyter Docker Stack. Including:

  • A Dockerfile
  • How to build and run it with docker commands
  • How to build and run it with docker-compose
  • A note with a link to jump to the Connect Atom section as those instructions would also apply unmodified.

• The section Testing it gets a note to generalize it to both Dockerfiles.

• In the section Terminate the connection and container add a note explaining how to do it with the Jupyter Docker Stack (that would be a lot simpler).

Is that right @MaximilianR ?

Would that be OK @lgeiger ?

[max-sixty]

Beautiful, @tiangolo

[max-sixty]

In the section Terminate the connection and container add a note explaining how to do it with the Jupyter Docker Stack (that would be a lot simpler).

That's just Ctrl-C now 💃

[lgeiger]

@tiangolo @MaximilianR Thank you a lot for those excellent explanations! I'm sure that this will help a lot of users 🎉

I love the idea of a separate KERNEL_CONNECTION.md file!

Would that be OK @lgeiger ?

That sounds awesome!

[tiangolo]

Excellent! I'll test the setup and work on a PR.

[tiangolo]

As the PR is already merged, I think we can close this issue.