Msprime: [Docs] When, why, how use containers vs conda vs pip

[castedo]

In reviewing PR #1136, @agladstein brought up many good points on what might be
confusing or missing in the installation instructions.

I see the overall issue is that the installation page could do a better job
helping users figure out when they should choose one option over another and
perhaps some guidance on how. The installation page already seems to be
extremely high quality and helpful. So to me this GH issue seems relatively low
priority to be honest. But I'm recording these 'loose ends' here because they
would make the installation documentation even better.

One piece of information that seems missing, and I didn't realize until today,
is how easily, I think, Python modules and scripts outside the container, can
call the msprime Python API inside the official docker image. I even wonder
whether this container scenario is a better recommendation than using conda
(when possible). It certainly seems better than calling the CLI.

@agladstein started communicating about how Python scripts outside a container
can call the msprime Python API inside the container, but we have not written
down the exact details yet.

Some addition guidance @agladstein made for the instructions are:

• "This means, you do not have to install msprime or any of its dependencies
  locally. Instead, you pull the msprime docker image and run the container."

• "Containers are great when you don't want to or can't install msprime on
  multiple systems, for example on HPC clusters or distributed cloud"

However, I'm not sure about saying docker images are not local. And containers
seem great to me even for single installations, not just clusters and clouds. I
haven't given conda a fair shake, but I think podman users might prefer using
containers rather than conda.

[jeromekelleher]

Sounds good to me @castedo - I think there's a pretty broad userbase at this point, so we should expect a mixture of conda, containers and pip installing. It's worth keeping in mind that we do have binary wheels now, and this might solve a lot of the installation problems that containers would help with. In particular, it should make it a lot easier to install on HPC systems.

[castedo]

Copying over a suggestion by @jeromekelleher originally made in #994:

For HPC users, we might mention the easybuild option. We'd probably need to learn how to update this recipe also (but I know one of the maintainers, so that shouldn't be too hard)

I have near-zero experience with HPC users, but I have to ask: Are most HPC users either using or planning to use containers?

[jeromekelleher]

Are most HPC users either using or planning to use containers?

"No", would be my guess. Most HPC users are inexperienced and working on very, very old linux distros with very complicated software setups. There is support in a lot of places for running singularity, but it's pretty recent I think. You have to expect HPC setups to be at least 5 years behind the cutting edge, I think.

[agladstein]

the installation page could do a better job helping users figure out when they should choose one option over another and perhaps some guidance on how.

I agree with this.
I think most users (scientists) will be most familiar with conda and automatically opt for conda. Then, I think people who come more from the developer side of things choose pip. Most users are not already familiar with containers. I also think that once people start using containers, they realize their awesomeness and will continue to use containers - but, I bet for the time being that is a very small fraction of the msprime user-base. I suggested putting those sentences in about containers to give people the background they might need to choose whether to use them.

I'm not sure about saying docker images are not local.

The image does exist locally on your computer, but you do not need to install it. You skip any sort of instillation and dealing with any dependencies, but pulling an image and using the container.

And containers seem great to me even for single installations, not just clusters and clouds.

This is definitely true. But, I think for most users, if they are using conda (especially with conda environments) there isn't much benefit to using a container. I think the main benefit of containers for msprime users is when you want to distribute your simulations to many different systems and you don't necessarily know what's already installed on them and you don't want to go through installation process for every unicorn system. At least this is when I have personally found containers useful.

Are most HPC users either using or planning to use containers?

I agree with @jeromekelleher on this. But, this is more because scientists do not really know about containers yet, not because it is hard. Singularity makes this incredibly easy on HPC clusters.

[jeromekelleher]

@castedo - would you like to write a short paragraph on this for the installation page, or should we close the issue?

[castedo]

I will either write a short paragraph or merely review the installation doc page by end of this weekend. I've been a bit distracted this week here in the US. :astonished: