Openff-toolkit: AmberTools and parmed infrastructure issues

[jchodera]

I wanted to collect a few AmberTools and parmed infrastructure issues in one thread:

The ambertools 18 conda package for AmberTools is currently being built as a special "universal" package that bundles several versions of Python (2.7, 3.4, 3.5, 3.6) inside the conda package tarball without flagging the package as depending on a conda version. The build infrastructure is discussed in this thread, and can be found here, though it is only minimally documented since it is meant to be used by the Amber devs to generate these packages.

The current release of ambertools that we mirror in the omnia channel has two issues:

• It contains an older version of parmed that gets overwritten if a newer parmed is installed after it (but overwrites newer parmed if the order is reversed): https://github.com/ParmEd/ParmEd/issues/1015
• It doesn't contain support for Python 3.7 (only 2.7, 3.4, 3.5, 3.6), though we don't use any of the Python tools at the moment (aside from ParmEd). conda-forge currently only supports 2.7, 3.6, and 3.6.

I'm working on whether there is a way to remove parmed from the prebuilt package for the omnia version we post, as suggested by @hainm in https://github.com/ParmEd/ParmEd/issues/1015#issuecomment-437651728, and it sounds like @hainm may be able to rebuild the package to include python 3.7 support in a few weeks.

Note that there is not a current official AmberTools package maintainer aside from @hainm (who moved to a Senior Scientist role at Schrödinger ~2 years ago, but is still a volunteer maintainer). It would be great if we could pitch in and help (maybe with some help from @lnaden) since we expect to make use of ambertools as a dependency for some time.

Note also that the whole ambertools build is difficult to automate on travis due to timeouts, as @hainm explains in https://github.com/ParmEd/ParmEd/issues/1015#issuecomment-449911089, but other tools like circle-ci or appveyor may offer alternatives that could work.

[jchodera]

Similarly, ParmEd seems to have no official (full-time) maintainer since @swails moved on, and there are a few bugs in the backlog as a result. I think @swails still hopes to make his way through those at some point (but is generally appreciative of help!), but it's just something we want to keep our eye on as we currently depend on ParmEd for conversions to other packages.

[multiplemonomials]

By the way, I'm a third volunteer maintainer of Amber build stuff (who saw this pop up in his GitHub notifications). I'm the author of Amber's new CMake build system that quietly debuted in AmberTools 18. I've worked a fair bit on ways to automate the Amber build through CMake, and am currently able to generate installer packages for multiple platforms, and create _pan-configuration_ builds with MPI and serial executables made in the same build run.

This has made Amber a lot easier to work with, and about a year ago we set up our own internal Jenkins server that runs builds nightly on a couple different platforms.

One of the last things that still needs to be done though is add support for Conda builds. To be honest I don't have a whole lot of Conda experience so I still have a lot of research to do, but I'm here if you guys need advice on Amber stuff. And if you've got any input on how I can go about generating COnda builds from CMake, that's welcome too.

[hainm]

it sounds like @hainm may be able to rebuild the package to include python 3.7 support in a few weeks.

sorry @jchodera. I have not done this yet. It's due to my long vacation in Vietnam and to some technical issues I don't have time to resolve (job + baby :D). Basically ambertools's using centos-5 based docker image but there's no python 3.7 available for this distro (I bet people move on to centos >= 6). It takes time for me to rebuild and test the centos-6 based docker image.

By the way, since you don't need any python programs (besides parmed that exists in omnia channel) in AmberTools, and AmberTools 19 will be released soon, I think I will skip python 3.7 for AT 18.

But let me know if you have different opinions. Cheers and sorry again.

[jchodera]

sorry @jchodera. I have not done this yet.

No worries! I have had this floating on my TODO list as well, but haven't had a chance to get to the new ambermini package either.

(job + baby :D)

Congratulations!!!

Basically ambertools's using centos-5 based docker image but there's no python 3.7 available for this distro (I bet people move on to centos >= 6). It takes time for me to rebuild and test the centos-6 based docker image.

Would it be too difficult to build on conda-forge? It seems like this might be the best strategy for maximum compatibility with everything else.

By the way, since you don't need any python programs (besides parmed that exists in omnia channel) in AmberTools, and AmberTools 19 will be released soon, I think I will skip python 3.7 for AT 18.

Is there an ETA on this? And would AT19 include parmed as a separate dependency rather than bundling its own parmed (which would rapidly get out of sync)?

Tagging @j-wags and @Lnaden here to keep them in the loop.

[hainm]

Would it be too difficult to build on conda-forge? It seems like this might be the best strategy for maximum compatibility with everything else.

I am thinking about using cond-forge docker image to build AT. conda-forge workflow is not suitable for AT since we bundle all python versions in a single build.

Is there an ETA on this?

It's up to @dacase but it's normally on April.

And would AT19 include parmed as a separate dependency rather than bundling its own parmed (which would rapidly get out of sync)?

No, AT19 still bundle its own parmed. We also use the conda built package to generate a non-conda package for non-conda user (e.g: for phenix people). I can help to write the script to remove parmed and update the meta and let omnia to maintain it.

[dacase]

What Hai proposed above is not what I have in mind. My proposal is this:

1. I (and other amber developers) will create an amber_omnia package, that has just the parts of AmberTools needed for omnia. This would presumably not include parmed, since the omnia folks seem to prefer to get parmed straight from Jason's repo (which I think is fine).
2. I can make "ordinary" conda packages for Mac and Linux already. I don't think it should be hard to build on conda-forge instead of natively on my laptop.

3. Key point is to avoid the "let omnia maintain it" part: Amber developers would take responsibility for periodically syncing "amber_omnia" with AmberTools. I don't see what else makes sense: there will always be a develoment branch for AmberTools, but Amber developers can't worry about doing something there that breaks omnia --they have neither the expertise not access to the test suites needed to do that. Rather, we could (at intervals) sync amber_omnia with the development version of AmberTools, and at that point (after running the omnia tests) create a updated amber_omnia release. This means that amber_omnia will not see the bleeding edge of AmberTools development, but that seems like a feature, not a bug.

4. With this "subset" scheme, omnia is also not vulnerable to bugs arising from cross-talk with parts of AmberTools that omnia will never use.

5. The omnia developers essentially have control over when and how new releases of amber_omnia get made, but the task of tracking Amber development would be a joint effort. In particular, an updated amber_omnia release could be done at any time, and not need to be coordinated with the generally yearly releases of AmberTools.

I'm open to counter arguments here. But I will point out that the AmberTools source tarball is about 400 MB, whereas the amber_omnia tarball (containing fewer test cases) should be less than 10 MB.
And the configure script for AmberTools is a 4000-line shell script with several dozen options; the configure script for amber_omnia is 300 lines with a single option (--verbose, to control how much info gets sent to the screen during the build.)

end of sales-pitch....sorry for the longish post

[hainm]

It doesn't contain support for Python 3.7 (only 2.7, 3.4, 3.5, 3.6)

@jchodera should be done for AT19 now. cheers.