Repo2docker: Julia builds

[choldgraf]

We should support dependencies for Julia.

Major to-do items

• [x] Allow users to specify requirements with a REQUIRE file
• [x] Allow users to pre-compile packages before building the docker image

Julia handles dependencies in a text file similar to how Python does this.

Thoughts on that? @yuvipanda

[tkelman]

If you run julia -e 'using Gadfly' etc once while building the docker image it should precompile and reduce the initial load time

[sje30]

Hi,
I'm struggling to get started with Julia and binder. I've built a simple repo at: https://github.com/sje30/simplejl with my own dockerfile to get Julia 0.6.0 (rather than 0.5.2), but I'm getting a dead kernel when I put it into beta.mybinder.org. Any clues? Or should I go via the REQUIRE mechanism?

[dpsanders]

I tried adding a REQUIRE file with
julia 0.6, but the notebook launched with a Julia 0.5.2 kernel.

[sje30]

I copied/stole the Dockerfile from a source in the file; but perhaps I need more of the invocations from this Dockerfile instead?
https://github.com/jupyter/docker-stacks/blob/master/datascience-notebook/Dockerfile

[choldgraf]

hmm - are you building on your own machine? And if so try running from master...we bumped the julia version recently: https://github.com/jupyter/repo2docker/pull/30

[yuvipanda]

We haven't deployed 0.6 to beta.mybinder.org for REQUIRE support yet. Shall
do shortly!

On Jun 30, 2017 8:35 AM, "Chris Holdgraf" [e-mail] wrote:

hmm - are you building on your own machine? And if so try running from
master...we bumped the julia version recently: #30
https://github.com/jupyter/repo2docker/pull/30

—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/jupyter/repo2docker/issues/23#issuecomment-312300161,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAB23ppUd9fw1k1omLdLj560IxH5yVuQks5sJRXNgaJpZM4Nulja
.

[choldgraf]

but to your broader point about setting up your own Dockerfile with Julia, see here for the "base" dockerfile etc that are used in the case of Julia builds...you could base your own dockerfile off of this.

[dpsanders]

Thanks. Does that Dockerfile live somewhere to use as a base in a FROM field?

[yuvipanda]

@sje30 @dpsanders so https://github.com/yuvipanda/simplejl works now (I did a bunch more deployments) on beta.mybinder.org.

I don't think the docker images we have are a good source of FROM, because they are subject to change pretty quickly + are meant to be used explicitly with s2i. We're working on making it easy to do other things (such as installing apt packages, running arbitrary scripts, etc) easy (see #34).

For a plain dockerfile to work with Julia, am not sure what the best approach is. We haven't quite documented the requirements for getting a plain Dockerfile to work with binder yet. It's mostly just making sure that:

1. the notebook python package is installed, and available in $PATH
2. An appropriate version of the jupyterhub python package is installed, and jupyterhub-singleuser is available in $PATH

Do you think a Julia enabled image would be a good addition to https://github.com/jupyer/docker-stacks? Or could it live somewhere where it's maintained by the Julia community instead?

We'd also want to add support for specifying versions of julia in the REQUIRE file - #34 should lead to a refactor that should help with this!

Finally, I must admit I know very little Julia! Any and all help welcome and gladly accepted!

thank you!

[tkelman]

We don't even maintain https://github.com/docker-library/julia/blob/master/Dockerfile, but if there were some way to paste that together with your jupyter dockerfiles (plus a Pkg.add("IJulia")) that would be neat.

If it helps, I live in Berkeley and would be happy to drop by BIDS for an hour or two any day if it would be useful to hack something out in person with some Julia expertise in the room.

[sje30]

I'd appreciate some help getting a Dockerfile that had Julia 0.6.0 + pyplot (or similar) working. My efforts so far floundering with the plotting!

[yuvipanda]

Did you try the repo I linked to with binder? It works for me...

On Jul 1, 2017 10:33 AM, "Stephen Eglen" [e-mail] wrote:

I'd appreciate some help getting a Dockerfile that had Julia 0.6.0 +
pyplot (or similar) working. My efforts so far floundering with the
plotting!

—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/jupyter/repo2docker/issues/23#issuecomment-312445235,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAB23tM9hrlF4RCZZi2uPw79ipFbPbzoks5sJoLagaJpZM4Nulja
.

[yuvipanda]

Specifically, https://beta.mybinder.org/v2/gh/yuvipanda/simplejl/master

On Sat, Jul 1, 2017 at 10:55 AM, Yuvi Panda [e-mail] wrote:

Did you try the repo I linked to with binder? It works for me...

On Jul 1, 2017 10:33 AM, "Stephen Eglen" [e-mail] wrote:

I'd appreciate some help getting a Dockerfile that had Julia 0.6.0 +
pyplot (or similar) working. My efforts so far floundering with the
plotting!

—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/jupyter/repo2docker/issues/23#issuecomment-312445235,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAB23tM9hrlF4RCZZi2uPw79ipFbPbzoks5sJoLagaJpZM4Nulja
.

--
Yuvi Panda T
http://yuvi.in/blog

[sje30]

thanks. I'd not seen the require file. Your version makes progress, but it needs a plotting engine to work still. e.g. PyPlot

thanks for adding 0.6.0 of Julia!

[sje30]

e.g. a minimal example for plotting

Using Plots
pyplot()
plot(rand(100))

I'll add it to the notebook when I get chance.

[yuvipanda]

To clarify, it requires python libraries installed too to work?

On Jul 1, 2017 11:39 AM, "Stephen Eglen" [e-mail] wrote:

e.g. a minimal example for plotting

Using Plots
pyplot()
plot(rand(100))

I'll add it to the notebook when I get chance.

—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/jupyter/repo2docker/issues/23#issuecomment-312448735,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAB23qZRXYqV-KDHPPHVhTGk-cTy510cks5sJpJDgaJpZM4Nulja
.

[sje30]

This is where some advice from @tkelman or @dpsanders might help.

pyplot() is part of the PyPlot package, which I could only install this morning on Dockerfile by

RUN apt-get install -y python-matplotlib

(see https://github.com/sje30/simplejl/blob/master/Dockerfile)

but then I think I got an error within the notebook saying that it could not find a suitable frontend.

@dpsanders : which plotting frontend would you recommend be added?

[yuvipanda]

Right, so if you were using pip it would be 'matplotlib' that'll need to be installed (we generally prefer to install python dependenceis with pip or conda rather than apt).

I'll work a bit on #34 today, which should help this situation a bit.

[dpsanders]

Pkg.add("PyPlot") should install everything automatically (using its own installation of Conda).

I recommend the GR and PlotlyJS backends.

[tkelman]

Though if we're going to use a jupyter backend that's set up already in the dockerfile through some other means, we wouldn't want to let Conda.jl bring in a whole separate instance.

[yuvipanda]

@tkelman we should totally hang out at Berkeley! I'll start an email thread to co-ordinate (anyone else in the area and wants to join is also welcome)

[yuvipanda]

Progress!

https://binder.binder-staging.omgwtf.in/v2/gh/yuvipanda/simpljl/master

You can see in https://github.com/yuvipanda/simplejl/ I now have a requirements.txt and a REQUIRE file - so multiple runtimes are supported! if you use a environment.yml file instead of requirements.txt you'll get a conda environment instead of venv. You can also install arbitrary apt packages by listing them in apt.txt but remember that since the python running is running from a virtualenv python packages (such as python3-matplotlib) installed via apt won't be available to the python runtime. Those should be installed via requirements.txt or environment.yml instead!

Try it out now?

There are still some rough edges in this, but I hope to have it running on the main binder beta site by end of the week.

[sje30]

Thanks Yuvi, this sounds great, BUT when I visit
https://binder.binder-staging.omgwtf.in/v2/gh/yuvipanda/simpljl/master
and press Launch, nothing happens for me.

Am I missing something, or is something down right now?

On Tue, Jul 04 2017, Yuvi Panda wrote:

Progress!

https://binder.binder-staging.omgwtf.in/v2/gh/yuvipanda/simpljl/master

You can see in https://github.com/yuvipanda/simplejl/ I now have a requirements.txt and a REQUIRE file - so multiple runtimes are supported! if you use a environment.yml file instead of requirements.txt you'll get a conda environment instead of venv. You can also install arbitrary apt packages by listing them in apt.txt but remember that since the python running is running from a virtualenv python packages (such as python3-matplotlib) installed via apt won't be available to the python runtime. Those should be installed via requirements.txt or environment.yml instead!

Try it out now?

There are still some rough edges in this, but I hope to have it running on the main binder beta site by end of the week.

[yuvipanda]

I made a typo! Can you correct it to simplejl rather than simpljl in the
url? On mobile now... Sorry!

On Jul 4, 2017 2:17 PM, "Stephen Eglen" [e-mail] wrote:

Thanks Yuvi, this sounds great, BUT when I visit
https://binder.binder-staging.omgwtf.in/v2/gh/yuvipanda/simpljl/master
and press Launch, nothing happens for me.

Am I missing something, or is something down right now?

On Tue, Jul 04 2017, Yuvi Panda wrote:

Progress!

https://binder.binder-staging.omgwtf.in/v2/gh/yuvipanda/simpljl/master

You can see in https://github.com/yuvipanda/simplejl/ I now have a
requirements.txt and a REQUIRE file - so multiple runtimes are supported!
if you use a environment.yml file instead of requirements.txt you'll get a
conda environment instead of venv. You can also install arbitrary apt
packages by listing them in apt.txt but remember that since the python
running is running from a virtualenv python packages (such as
python3-matplotlib) installed via apt won't be available to the python
runtime. Those should be installed via requirements.txt or environment.yml
instead!

Try it out now?

There are still some rough edges in this, but I hope to have it running
on the main binder beta site by end of the week.

—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/jupyter/repo2docker/issues/23#issuecomment-312956454,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAB23hHhv5lIYrXeNfLh2rMeu2LpNaIhks5sKqvNgaJpZM4Nulja
.

[sje30]

Thank you for continuing to work on this. Details below. It looks like we need a Julia guru ( @tkelman @dpsanders ?) to help with getting the plotting packages into the notebooks, as I'm stumped by the errors.

https://binder.binder-staging.omgwtf.in/v2/gh/yuvipanda/simplejl/master looks good thanks. I've updated my version accordingly:
https://binder.binder-staging.omgwtf.in/v2/gh/sje30/simplejl/master

[my files at: https://github.com/sje30/simplejl ]
What works:
It starts Julia 0.6.0 kernel okay.

what doesn't work

in gr.ipynb the

using GR
inline()
histogram(randn(1000))

generates the following error

INFO: Precompiling module GR.
error compiling inline: error compiling inline: could not load library "/srv/julia/pkg/v0.6/GR/src/../deps/gr/lib/libGR.so"
libXt.so.6: cannot open shared object file: No such file or directory

SO, back to simplejl.ipynb, which use Plots and PyPlot, I get the error:

INFO: Precompiling module PyPlot.
Fatal Python error: Py_Initialize: Unable to get the locale encoding
ImportError: No module named 'encodings'

Current thread 0x00007fc8592f4c40 (most recent call first):

signal (6): Aborted
while loading /srv/julia/pkg/v0.6/PyPlot/src/PyPlot.jl, in expression starting on line 5
gsignal at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
abort at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
Py_FatalError at /usr/lib/x86_64-linux-gnu/libpython3.5m.so (unknown line)
unknown function (ip: 0x7fc83bf4c419)

[tkelman]

The GR problem looks like there's no global X library that it can find on the load path? Does the python virtualenv from requirements.txt get activated before julia runs? I imagine if the matplotlib wheel (it would be installed from a wheel with requirements.txt, right? would that come with its own local X libs?) needs some environment variables set up when the venv gets activated, then julia would need to have those same env vars set to be able to see the matplotlib wheel's copies of those libraries (the GR binary may try to link against some of them). For PyCall / PyPlot, the Julia PyCall package will need to see the virtualenv's python (or maybe try with conda instead and set the PYTHON env var to point at that before adding the Julia packages?) at the time it runs Pkg.add / Pkg.build. That PyPlot error about the encodings module looks like it's trying to use the system global python, which probably is a minimal debbuntu packaged version that only has a subset of the standard library. There are apt packages you could probably also install to flesh that out more, but I don't know if the binder requirements.txt or environment.yml will see the same set of packages for hooking the jupyter notebook and kernels all together.

[sje30]

Thanks for the diagnosis Tony. This is a bit out of my area of expertise to debug, but let me know @yuvipanda if I can help test anything. I think the binder service is wonderful, and it already works for scientific notebooks with plotting in R. Now we just need a case study in Julia!

I don't personally mind whether it is the GR package or the PyPlot (or plotly...) but I think it would be good to have one plotting environment working.

[choldgraf]

it sounds like this stuff would be best-handled with an afternoon hack session. @sje30 is it OK if this functionality gets hashed out over the course of, say, a couple weeks? Or do you need this to work ASAP?

[sje30]

hi @choldgraf -- I'd be delighted for this to work, no matter how long it takes. So, if you think you can fix it best by an afternoon hack, great. Just ping me when you need some external testing. As I said before, having decent plotting for Julia in a notebook (whichever framework is adopted) would be a great step forward.

[sje30]

p.s. I am working on some local Julia notebooks in the meantime, so we should have quite a few examples to work though later this summer.

[yuvipanda]

@tkelman does Julia need x client libraries or X itself? We probably don't want to support running an X server itself inside the containers - IIRC matplotlib doesn't need X and supports other backends?

I think we just need to figure out which environment variables we need to set for Julia to find the right Python. Currently I'm only setting PATH, but might need PYTHONPATH and/or PYTHONHOME as well?

[tkelman]

Julia doesn't need X directly, but some graphics libraries might. The GR.jl package downloads a libGR.so binary that apparently is trying to find libXt.

ref https://github.com/JuliaPy/PyCall.jl/#specifying-the-python-version

[yuvipanda]

Me and @tkelman tracked some of this down to a bug in PyCall: https://github.com/JuliaPy/PyCall.jl/issues/410

We have it working with conda right now, so we recommend using environment.yml for python packages for now.

[tkelman]

And looks like the GR issue with missing libXt.so.6 has been reported at https://github.com/jheinen/GR.jl/issues/35. Adding libxt with apt-get changes the error to missing libGL.so which we're trying to hunt down a good candidate for (see below). Probably a better idea would be ask the GR maintainers to build a no-X copy of the library, check for existence of libXt at Pkg.add / Pkg.build time, and download the no-X copy if it's not present.

[yuvipanda]

Installing (via apt.txt) the following libraries: libxt-dev libgl1-mesa-dev gets GR to work. However, that's setting up an X we don't use at all, and GR can be run / built without X support (which is what is the long term solution to this, IMO)

[yuvipanda]

Me and @tkelman hacked on this today at BIDS!

1. pre-compilation now works! We pre-compile everything that's opted into pre-compilation :)
2. We tracked down the issue with PyPlot to PyCall, and with GR to missing apt packages. We sent a PR to PyCall (https://github.com/JuliaPy/PyCall.jl/pull/411), and getting that merged would let Julia's PyCall work with virtualenv. Until then, you can use an environment.yml to get Python / Julia combination stuff working.

Note that I haven't deployed pre-compilation anywhere yet.

@sje30 https://binder.binder-staging.omgwtf.in/v2/gh/yuvipanda/simplejl/master now works fine (I've updated my repo!) and I can do using PyPlot and it works ok :D

@choldgraf: @tkelman mentioned there's a Julia Users discourse mailing list, perhaps we could publicize this there too? (Once we deploy this support to beta, than just to staging)

[choldgraf]

Me and @tkelman hacked on this today at BIDS!

woooot!

pre-compilation now works! We pre-compile everything that's opted into pre-compilation :)
We tracked down the issue with PyPlot to PyCall, and with GR to missing apt packages.

wooooooooooot!

We sent a PR to PyCall (JuliaPy/PyCall.jl#411), and getting that merged would let Julia's PyCall work with virtualenv. Until then, you can use an environment.yml to get Python / Julia combination stuff working.

wooooooooooooooooooot!

Note that I haven't deployed pre-compilation anywhere yet.

sounds good - do we have a good demo repo set up somewhere?

@choldgraf: @tkelman mentioned there's a Julia Users discourse mailing list, perhaps we could publicize this there too? (Once we deploy this support to beta, than just to staging)

+1 to that...let's set up a demo repo or two first and then for sure we should spread the word

[sje30]

Thanks guys! Amazing work. I'm happy to help with a minimal working example using PyPlots.

If its easy to fix GR, that would be great to get working too. (@jheinen - have you seen this thread? It relates to getting jupyter notebooks running Julia with plotting working on binder.)

[jheinen]

GR can bee build without X11, but with limited output support. For IJulia, this should be no problem, as long as SVG is the default output format. SVG does not have any package requirements (same for PDF and PS).

[yuvipanda]

@sje30 @tkelman I finally got time to merge the work we did into master (along with a ton of other cleanups!). I've added integration tests now to this repo running on Travis, including a simple PyPlot based Julia one at https://github.com/jupyter/repo2docker/tree/master/tests/julia/pyplot.

Thank you for all your work!

[yuvipanda]

Note that this isn't deployed on binder yet. I'm going to try to make a full release of repo2docker first before deploying on Binder.

[sje30]

Great! Thanks for your work.

[tkelman]

Excellent. Guess the PYTHONHOME issue with PyCall would still need to be addressed to make virtualenv work, but conda working without hacks is good enough for most people. Is there a test log somewhere to check that PyCall and the binder notebook are using the same conda install, and PyCall isn't downloading its own private one?

[yuvipanda]

@tkelman I just turned on full log capturing for all tests (and not just private ones), and you can see the logs for the Julia test here: https://travis-ci.org/jupyter/repo2docker/jobs/259053546 (it's just building).

I'd also love to have at least a couple other Julia integration tests here :D

[tkelman]

PyCall is using python (Python 3.6.1) at /srv/conda/bin/python is a very good sign. I think you can get that from the value of PyCall.python

[yuvipanda]

w00t. Should I add it to the verify script at https://github.com/jupyter/repo2docker/blob/master/tests/julia/pyplot/verify?

[yuvipanda]

@tkelman I've made a PR incorporating that - https://github.com/jupyter/repo2docker/pull/50. I'll merge once tests go green!

[yuvipanda]

@tkelman so PyCall.python did not work - https://github.com/jupyter/repo2docker/pull/50/commits/92d0c5e6d6ee72fdca131d9d5a58632b847bab82 did however!

@choldgraf @tkelman @sje30 Do you think we can now officially call Julia 'supported' in repo2docker? :)

[choldgraf]

Awesome! Nicely done everybody!

I think we should call it "officially" supported but with a "this has not been totally vetted yet" caveat ;-) I guess if we're still in beta technically that's kinda assumed

[sje30]

Thanks everyone, this is great.

[yuvipanda]

Awesome! I'm going to close this one then! Tyvm everyone!

[choldgraf]

FYI we've now got examples of this here: https://github.com/binder-examples/julia_python

[tkelman]

Cool, did the reason for the hack of needing to copy the conda folder get resolved somewhere?

[sje30]

FYI we've now got examples of this here: https://github.com/
binder-examples/julia_python

this is great! We have two serious coding examples at:
https://github.com/dkeitley/Map-Formation-Models

with models in python and julia. @dkeitley used two branches of the
repos so that we could have binder running -- but it looks like now we
can put them into the same binder?

Stephen

[yuvipanda]

@sje30 yeah, the 'julia' branch there seems to work for me. You can remove the 'conda' folder in there too

[choldgraf]

as a neuroscientist, just wanna say that repo is super cool :-)

[sje30]

hi Chris,

I've got another notebook in progress...

https://github.com/sje30/cnw/blob/master/Introduction%20to%20Computational%20Neuroscience.ipynb

I've got it working on juliabox.com, but the user needs to manually do
Pkg.add("OrdinaryDiffEq") in the console.

I'd like to try in on mybinder... how do I solve the package dependency
in Julia on binder?

On Mon, Oct 23 2017, Chris Holdgraf wrote:

as a neuroscientist, just wanna say that repo is super cool :-)

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.*

[dpsanders]

I believe you can make a REQUIRE file which lists the dependencies (one per
line).

On Tue, Nov 14, 2017 at 6:11 PM, Stephen Eglen [e-mail]
wrote:

hi Chris,

I've got another notebook in progress...

https://github.com/sje30/cnw/blob/master/Introduction%20to%
20Computational%20Neuroscience.ipynb

I've got it working on juliabox.com, but the user needs to manually do
Pkg.add("OrdinaryDiffEq") in the console.

I'd like to try in on mybinder... how do I solve the package dependency
in Julia on binder?

On Mon, Oct 23 2017, Chris Holdgraf wrote:

as a neuroscientist, just wanna say that repo is super cool :-)

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.*

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/jupyter/repo2docker/issues/23#issuecomment-344443264,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AALtTsVLca2bkrptOGj38IS5ydMBI0Tdks5s2iw_gaJpZM4Nulja
.

--
Dr. David P. Sanders

Profesor Titular "B" / Associate Professor
Departamento de Física, Facultad de Ciencias
Universidad Nacional Autónoma de México (UNAM)

dpsanders@g [e-mail] / Twitter: @DavidPSanders
https://twitter.com/DavidPSanders
http://sistemas.fciencias.unam.mx/~dsanders / GitHub: dpsanders
https://github.com/dpsanders

Cubículo / office: #414, 4o. piso del Depto. de Física

Tel.: [phone]

[sje30]

Where is the julia 0.6 kernel? Just added a REQUIRE, but not finding julia 0.6 on http://beta.mybinder.org (which now redirects)

[sje30]

Thanks @dpsanders -- all working now! http://sje30.github.io/post/neuro-binder/