Msprime: create official docker image

[agladstein]

Hi, seems useful to have a msprime "official" Docker image. I can submit a pull request with the Dockerfile and documentation.

I think it would be good to link Dockerhub to the Github repo - that way it will automatically build the image on dockerhub, triggered by new commit on Github. This also includes automated integrated testing. Repo admin would have to set this up.
https://docs.docker.com/docker-hub/builds/
https://docs.docker.com/docker-hub/builds/link-source/

I can make a Dockerfile that installs msprime from pip - which seems better if any users wanted to create an image. Or I can make a Dockerfile that compiles the entire repo from source. Which would you prefer?

The other option is skip Docker and just make a Singularity image. But, I think it is more robust to start with a Docker image and then run it as a Singularity container. I could also include the equivalent Singularity recipe in case any users/developers want to play with Singularity images. You need root to build Singularity images, but you don't need root to run Singularity containers. So, I don't think building Singularity images isn't any more accessible than building Docker images.

[jeromekelleher]

This sounds good, but I wonder if it's better to spin it out into a different repo. Maybe tskit-containers or something which we update on significant releases? This could then be linked up with Dockerhub, etc.

Any thoughts @tskit-dev/all?

[andrewkern]

I agree this should be in a different repo. Strikes me that these container images will have their own lifecycles, different from the code they carry.

[molpopgen]

Yes, a different repo is a good idea. And the base image has to be thought
through, as there's a big difference between an image you can work in vs
one you'd want to work in.

On Mon, Dec 17, 2018, 7:55 AM Andrew Kern <[email protected] wrote:

I agree this should be in a different repo. Strikes me that these
container images will have their own lifecycles, different from the code
they carry.

—
You are receiving this because you are on a team that was mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tskit-dev/msprime/issues/660#issuecomment-447894663,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AGHnHyQQl8LRkV4ySFUILXGVU3tsgnW0ks5u575kgaJpZM4ZWcYj
.

[agladstein]

If you do that you won't have the benefit of docker automatic builds with linked tags to the git repo.
I think you'll have to remember to push new builds to Dockerhub for each new release, or figure out how to trigger new builds not based on commits because there will likely be new code releases without changes to Dockerfile.

But, maybe that's less important than having a variety of images with different useful environments.

[molpopgen]

That's a good point, but perhaps not a deal breaker?

I'm happy to talk more, but I'll be offline. I'm teaching a one day
workshop on msprime tomorrow! (Surprise, @jeromekelleher!)

On Mon, Dec 17, 2018, 8:02 AM Ariella Gladstein <[email protected]
wrote:

If you do that you won't have the benefit of docker automatic builds with
linked tags to the git repo.
I think you'll have remember to push to builds to Dockerhub for each new
release, or figure out how to trigger new builds not based on commits
because there will likely be new code releases without changes to
Dockerfile.

But, maybe that's less important than having a variety of images with
different useful environments.

—
You are receiving this because you are on a team that was mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tskit-dev/msprime/issues/660#issuecomment-447896708,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AGHnHwZjt6KfQhWV88FueELtzRbxpWxWks5u57-dgaJpZM4ZWcYj
.

[agladstein]

I do think it's more standard practice to have the Dockerfile in the same repo as the code and build from there.
You could have an "official" msprime docker image that compiles from source and has continuous integration, and a separate repo for a variety of containers that may be useful for users (e.g. sims + deep learning, sims + stats...)

[jeromekelleher]

I'm happy to talk more, but I'll be offline. I'm teaching a one day workshop on msprime tomorrow! (Surprise, @jeromekelleher!)

Cool; good luck with it @molpopgen!

If you do that you won't have the benefit of docker automatic builds with linked tags to the git repo.

So if we can automatically push an image when a release is tagged, then yes, I agree it'd be really good to have a minimal image that gets automatically pushed to Dockerhub. @agladstein, would you mind setting up a tskit-dev account on Dockerhub so it all looks 'official'?

Regarding the separate repo, I think this would also be useful. For example, I built an image the other day with msprime, tsinfer and SLiM for a workshop using JupyterHub here. Perhaps we could keep things like this in tskit-containers? @agladstein, would you be willing to lead on managing this repo?

[agladstein]

Sure thing. I'll set up the tskit-dev account on Dockerhub.

Yes. I can also head a tskit-containers repo for useful images.

[jeromekelleher]

Sure thing. I'll set up the tskit-dev account on Dockerhub.

Excellent, thanks!

Yes. I can also head a tskit-containers repo for useful images.

OK, cool. I've added a new repo https://github.com/tskit-dev/containers, and made a team @tskit-dev/containers-admin consisting of @agladstein, @molpopgen and me which has admin access to this repo. (If you're not interested in being involved @molpopgen just let me know and I'll remove you).

So, I guess the next step here would be to create the minimal Docker image that's pushed to Dockerhub. Once that's done we can close this issue. Any wider container-y stuff can be discussed over in https://github.com/tskit-dev/containers

[agladstein]

I will work on the minimal Docker image that's integrated with Dockerhub for tskit-dev/msprime.

[molpopgen]

Happy to do stuff. I'll probably base fwdpy11 images on these!

On Mon, Dec 17, 2018, 10:55 AM Jerome Kelleher <[email protected]
wrote:

Sure thing. I'll set up the tskit-dev account on Dockerhub.

Excellent, thanks!

Yes. I can also head a tskit-containers repo for useful images.

OK, cool. I've added a new repo https://github.com/tskit-dev/containers,
and made a team @tskit-dev/containers-admin
https://github.com/orgs/tskit-dev/teams/containers-admin consisting of
@agladstein https://github.com/agladstein, @molpopgen
https://github.com/molpopgen and me which has admin access to this
repo. (If you're not interested in being involved @molpopgen
https://github.com/molpopgen just let me know and I'll remove you).

So, I guess the next step here would be to create the minimal Docker image
that's pushed to Dockerhub. Once that's done we can close this issue. Any
wider container-y stuff can be discussed over in
https://github.com/tskit-dev/containers

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tskit-dev/msprime/issues/660#issuecomment-447956957,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AGHnH_4C4mObej9Q5-l4Gg_X-qytwkAgks5u5-g3gaJpZM4ZWcYj
.

[agladstein]

So... there has been some struggle https://github.com/tskit-dev/msprime/pull/640 https://github.com/tskit-dev/msprime/issues/639

Does the repo compile successfully? Is docs up to date? I've been trying to compile for the image, but haven't been successful yet.

On Ubuntu 18.04 - starting with an ubuntu image:

FROM ubuntu:18.04

# Set the working directory to /app
WORKDIR /app

# Copy the current directory contents into the container at /app
COPY . /app

Build image:

docker build -t test_msprime .

Run container:

docker run -it test_msprime:latest

Then, in the container I've gotten through:

apt-get update
apt-get install -y --no-install-recommends \
    python3 \
    python3-dev \
    python3-pip \
    build-essential \
    libgsl0-dev \
    libssl-dev \
    libffi-dev \
    libxml2-dev \
    libxslt1-dev \
    zlib1g-dev

pip3 install --upgrade setuptools
pip3 install -r requirements/development.txt

Then, when I do make ext3
I get the error:

python3 setup.py build_ext --inplace
your setuptools is too old (<12)
setuptools_scm functionality is degraded
Traceback (most recent call last):
  File "setup.py", line 220, in <module>
    cmdclass={"build_ext": local_build_ext},
  File "/usr/local/lib/python3.6/dist-packages/setuptools/__init__.py", line 143, in setup
    return distutils.core.setup(**attrs)
  File "/usr/lib/python3.6/distutils/core.py", line 108, in setup
    _setup_distribution = dist = klass(attrs)
  File "/usr/local/lib/python3.6/dist-packages/setuptools/dist.py", line 442, in __init__
    k: v for k, v in attrs.items()
  File "/usr/lib/python3.6/distutils/dist.py", line 281, in __init__
    self.finalize_options()
  File "/usr/local/lib/python3.6/dist-packages/setuptools/dist.py", line 601, in finalize_options
    ep.load()(self, ep.name, value)
  File "/usr/local/lib/python3.6/dist-packages/setuptools_scm/integration.py", line 19, in version_keyword
    dist.metadata.version = get_version(**value)
  File "/usr/local/lib/python3.6/dist-packages/setuptools_scm/__init__.py", line 104, in get_version
    version = _do_parse(root, parse)
  File "/usr/local/lib/python3.6/dist-packages/setuptools_scm/__init__.py", line 69, in _do_parse
    version = version_from_scm(root)
  File "/usr/local/lib/python3.6/dist-packages/setuptools_scm/__init__.py", line 30, in version_from_scm
    return _version_from_entrypoint(root, 'setuptools_scm.parse_scm')
  File "/usr/local/lib/python3.6/dist-packages/setuptools_scm/__init__.py", line 36, in _version_from_entrypoint
    return ep.load()(root)
  File "/usr/local/lib/python3.6/dist-packages/setuptools_scm/git.py", line 11, in parse
    real_root, _, ret = do_ex('git rev-parse --show-toplevel', root)
  File "/usr/local/lib/python3.6/dist-packages/setuptools_scm/utils.py", line 58, in do_ex
    HGPLAIN='1',
  File "/usr/lib/python3.6/subprocess.py", line 709, in __init__
    restore_signals, start_new_session)
  File "/usr/lib/python3.6/subprocess.py", line 1344, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'git': 'git'
Makefile:6: recipe for target 'ext3' failed
make: *** [ext3] Error 1

[jeromekelleher]

Things are in a bit of flux at the moment, but everything should work OK. The error you're getting here is because git is missing probably.

I don't think we should install the development dependencies though, as this will bloat things out a lot. I would just run pip3 install . and let setup.py sort out the run-time dependencies.

Maybe open a PR with the Dockerfile so we can discuss the diff directly?

[agladstein]

Official Docker Image can be found here: https://hub.docker.com/r/tskit/msprime