Repo2docker: Replace `freeze.py` with `conda-lock`

[yuvipanda]

Proposed change

We use freeze.py to generate fully specified environment.yml files out of our environment.yml files. It creates a docker image, installs everything, and then lists out the installed packages.

Instead, we could use conda-lock. This resolves environment.yml files to fully specified lock files we can use. It doesn't install the packages either, so should be much faster.

Alternative options

Do nothing! Our current solution works well.

Who would use this feature?

Developers of repo2docker who modify environment.yml files. Currently the freezing process can take a while - this would make it faster.

How much effort will adding it take?

We need to investigate if conda-lock will actually do what we want. After that, implementing it shouldn't be too difficult - a couple hours of work for someone very familiar with the codebase, maybe 8-12h for someone who isn't.

Who can do this work?

1. Decent python skills
2. A vague understanding of how conda's environment.yml files work

[A-Fisk]

Hello. First time contributer, saw you mention this on twitter so thought I'd have a look. Feel like I've got no idea what I'm doing so help appreciated.

I've had a look around freeze.py and conda lock and I think I understand what they both do.
This is my current understanding:

• freeze.py - takes environment.yml file and generates environment.*.frozen.yml files
• app.py - then at main runtime when it uses CondaBuildPack class it selects the right environment.*.frozen.yml file and runs this as part of install-miniforge.bash
• conda-lock generates conda-system.lock file which can then be used to create an environment using conda create -f conda-system.lock
• to switch to using conda-lock we need to replace where conda env create -f environment.yml is run with conda create -f conda-system.lock
• from having a bit of a dig this seems to be done in install-miniforge.bash line 42-44:

echo "installing notebook env:"
cat /tmp/environment.yml
conda env create -p ${NB_PYTHON_PREFIX} -f /tmp/environment.yml

• I'm stuck trying to figure out the system for selecting the right environment file and how that is then passed as an argument into install-miniforge.bash and then ends up as /tmp/environment.yml
  
  
  
  • I think it's something in the get_build_scripts function in CondaBuildPack but I'm not sure.
  
  

Extra question:

1. Workflow for using freeze.py? I'm trying to figure out where this fits in. If I've got it right it's run whenever the base environment.yml file is updated and then that updates all the frozen files.

Anyway, that's all I can come up with for now. I hope I'm on the right track and haven't got it all completely wrong. Not sure how useful this is. Not sure if I've described things in too much detail or not enough, but that's what I've found for now. Hope to come back to this tomorrow.

[yuvipanda]

Welcome to the project, @A-Fisk! You're very much on the right track - that's a very impressive deep dive!

Workflow for using freeze.py? I'm trying to figure out where this fits in. If I've got it right it's run whenever the base environment.yml file is updated and then that updates all the frozen files.

https://repo2docker.readthedocs.io/en/latest/contributing/tasks.html#update-and-freeze-buildpack-dependencies has a bit more information on when it is used. Does that help?

I'm stuck trying to figure out the system for selecting the right environment file and how that is then passed as an argument into install-miniforge.bash and then ends up as /tmp/environment.yml

The python_version property figures out what version of python is specified in the environment.yml file in the user's git repository. After that, as you say, we use get_build_scripts to copy the appropriate file into /tmp/environment.yml.

https://repo2docker.readthedocs.io/en/latest/architecture.html might be useful to see the various stages repo2docker runs through to build the final docker image. The actual dockerfile template used might also help.

Hope this is helpful! Excited to have you contributing :)

[betatim]

Commentary from the side lines: I think we should investigate (and I am very impressed with your work A-fisk, especially as a first time contributor!!) conda-lock, however our freeze.py script "works" and doesn't cause all that much pain. It is somewhat comprehend-able to the maintainers and won't change unless we change it.

If conda-lock was a widely used, constrained to not "change" by having lots of users, etc I'd be less nervous about switching. The problem won't be switching over or even in the next few months. It will be at some point "far" in the future when freezing an environment starts failing for reasons. It feels (to me) like we have acquired quite a few tools in the "binder universe" that we use rarely. This "rare usage" makes it hard to debug when things change behaviour. The fix here is either to use stuff more often so that the changes we need to look at when things change behaviour is smaller or become more conservative with changing our rarely used tools. I don't know what the right trade-off is :-/ (sorry for being a grumpy old man, I promise on the inside I am just as excited as you ;) )

[ocefpaf]

If conda-lock was a widely used, constrained to not "change" by having lots of users, etc I'd be less nervous about switching.

@betatim conda-lock is quite new but the change necessary here is to support a conda explicit artifact install file, see https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#building-identical-conda-environments.

That is stable and is in conda for a while now. Users will use conda-lock to create that file and, if that changes, it won't affect anything here as long as that file format remains unchanged. It is the same thing as adopting the environment.yaml file, which repo2docker already did.

TL;DR while I like to see conda-lock mentioned in there the actual feature request here is "support conda explicit file."

[minrk]

@ocefpaf we add the conda-forge broken label as a lowest priority to our frozen environments (after freezing them, so no broken packages are used at freeze-time) to ensure that a package being marked as broken doesn't cause our frozen envs to stop installing. Does the explicit file also work in that case when a package has been moved to a new label? We did this because it was not uncommon that our frozen environments stopped working because a past package that actually worked fine was marked as broken, typically due to something like insufficient dependency pinning.

[ocefpaf]

@ocefpaf we add the conda-forge broken label as a lowest priority to our frozen environments (after freezing them, so no broken packages are used at freeze-time) to ensure that a package being marked as broken doesn't cause our frozen envs to stop installing. Does the explicit file also work in that case when a package has been moved to a new label? We did this because it was not uncommon that our frozen environments stopped working because a past package that actually worked fine was marked as broken, typically due to something like insufficient dependency pinning.

Yep. Labels are just channels, everything should work.

However, I would revisit that policy. Since we started doing repodata patches instead of moving to broken things the pkg solutions got more stable. Also, the chances of actually getting a broken env when using broken is really high now, not to mention the longer solve times. I guess the former you get away with by just keeping it low priority but the latter probably hits the performance here.

[xhochy]

Does the explicit file also work in that case when a package has been moved to a new label?

Being more explicit: the explicit URLs work even when a package is marked as broken in the conda-forge setup (broken packages keep their main URL).

Additionally, with the frozen explicit file, the solver is skipped completely making the installation even with conda superfast. The parallel downloading of mamba still improves on conda though.

[ocefpaf]

Being more explicit: the explicit URLs

:laughing: Thanks! Yeah, the "explict" URL is a direct path to the artifact, no matter where it is.

[xhochy]

Being more explicit: the explicit URLs

😆 Thanks! Yeah, the "explict" URL is a direct path to the artifact, no matter where it is.

Actually: no :P It still includes the label but nowadays, we don't remove the main label anymore from a package, we just patch it out of the repodata. Thus this trick really only works with the conda-forge channel.

[ocefpaf]

Actually: no :P It still includes the label but nowadays, we don't remove the main label anymore from a package, we just patch it out of the repodata. Thus this trick really only works with the conda-forge channel.

@xhochy if one adds the broken label to the env file list of channels, and run conda-lock, it should work even without the repodata trick as far as I know. But that is not too relevant to the discussion here :-)