Docker-stacks: how to use custom module?

[ryantuck]

I've got a custom module I'd like to use in my notebooks. I've read that all I need to do is update my PATH and conda should pick up on changes. This doesn't seem to be working as expected.

Dockerfile:

from jupyter/scipy-notebook

ADD ./my_module /home/jovyan/work/my_module

ENV PATH=$PATH:/home/jovyan/work/my_module

I build the docker image and run the container and am unable to import my_module.

However, if I shell into the running container and fire up python from the command line, I can import the module.

my-machine $ docker exec -it my_container bash
$ python
>>> import my_module
>>>

I'm assuming this is some conda weirdness - how can I tell the running conda to recognize my module?

[parente]

I build the docker image and run the container and am unable to import my_module.

From where are you trying to import the module? A notebook in the same directory?

[ryantuck]

@parente ideally I should be able to install this in any arbitrary location. I understand that if the module was in the same directory as the notebook I was working in, I'd be good to go.

I was able to find a workaround in the following way, by explicitly copying over a file custom.pth that contains the destination location of my_module into the site-packages directories of both python installs, which updates the PYTHONPATH for both.

# custom.pth
/home/jovyan/work

and my Dockerfile:

FROM jupyter/scipy-notebook

# add custom module to python 2.7 and 3 paths
COPY ./custom.pth /opt/conda/envs/python2/lib/python2.7/site-packages/custom.pth
COPY ./custom.pth /opt/conda/lib/python3.5/site-packages/custom.pth

# bring over custom modules
COPY ./my_module /home/jovyan/work/my_module

But it seems like this is convoluted, and that I should be able to do that at the conda level.

[parente]

If I were tackling this problem, I'd first look to use conda or pip to install my module as part of the docker build process like all the other packages. If I couldn't package my module properly, I'd next look at setting the PYTHONPATH environment variable in my Dockerfile to a directory where I could dump my module for use in both Python2 and 3. If that failed, I'd fall back on copying it into both the python2 and python3 site-packages, unless I was interested in developing that module within the container.

[ryantuck]

@parente totally - my confusion is around how to install a custom module via conda. That would certainly be the cleanest implementation but I couldn't find anything online about how to do that.

[parente]

These are the steps for building a conda package that you can install with conda install mypackage:

http://conda.pydata.org/docs/building/build.html

I don't know of a shortcut around these steps if you have just one simple .py module. That's why my next thought would be to Dockerfile COPY and set PYTHONPATH if the work is overkill for the use case.

[parente]

There's been no activity on this issue for a few months now so I'm going to close it out. @ryantuck, feel free to reopen if you have more questions. If you found a nice way of doing what you want, please add it to the recipes page on the wiki for others to follow.