Conan: Add option to download packages in parallel

[tru]

When running conan install on a file that depends on a lot of packages those packages should be able to be downloaded in parallel. This should speed up that process a lot. Especially if unpacking would be done in parallel as well. This should of course be an option of some sorts I think. But it would be a huge win for some CI deployments.

[GamePad64]

We are using Conan to download modularized boost and others. Lack of parallel download makes downloading a very time-consuming process (~6 mins)

[tru]

yeah I would like this to get a bit more traction - maybe I write the code myself. I think with a multiprocess.pool it should be fairly easy to fix.

[memsharded]

I think parallelism is always tricky :)
For example: how to ouput the parallel downloads?

I am not against this feature, but I think this could be way more complicated than you think.
To propagate downstream the package_info() method information from dependencies, you need to have already downloaded and unzipped it (some package_info() methods actually read from the package folder). Also, some packages might be built from sources, and they need to wait until the dependencies have been already retrieved. The propagation is executed just after packages are retrieved. So it would be necessary to do big changes of the ConanInstaller to accomodate this functionality.

Please have a look to the ConanInstaller flow, and tell me, if you still think it is easy, then a proof of concept of the functionality would be great, thanks!

[tru]

I think it makes sense to download the package_info stuff serially - but then downloading conan_package files in parallel. I think in terms of output yarn does the best job for this. I will have a look when I have some time (haha)

[memsharded]

You can't, package_info() can be only evaluated when the package folder is already there, and for that the conan_package.tgz files should have already been downloaded and unzipped. So first download, then execute package_info().

I don't think yarn needs to look into package artifacts for propagating downstream build information (like the name of the libraries), but sure, it would be interesting to have a look, I will also will (when I have time too :) :) :)

[memsharded]

We will be checking how to approach this in 1.3, then if green light, implement it for 1.4 or 1.5

[helmesjo]

How are things looking regarding this? Is the 1.4 milestone a possibility?

[memsharded]

1.4 is today or tomorrow so, impossible.
However, we are working on it, and we have done huge changes to the internals of the graph computation (latest ongoing work https://github.com/conan-io/conan/pull/2891, but we did many other changes in 1.4 too for this) and how the things are installed, in order to make possible to implement this in future releases.

[helmesjo]

Just a quick check-in:
Any progress on this particular feature (AKA "bulk download")? Most concrete use-case at the moment would be bincrafters modular boost packages.

[memsharded]

Hi @helmesjo

Well, Conan 1.5 did a huge step forward this, decoupling the binary requirements computation from the actual process of retrieving them. Not that the feature is there, it needs still a bunch of work, but that was probably about 50-60% of the required work.

[helmesjo]

Just out of curiosity, how are things looking? :)

[Lawrencemm]

Adding a boost requirement to my project makes the install process incredibly slow to the point of being absurd. Please add this feature.

[DoDoENT]

Now that #3452 is closed, we are looking forward for this one too, 😛

[ofiriluz]

Yes please
Would be much needed to our environment with process parallelism ;3

[memsharded]

There is some preliminary ongoing work here:

• https://github.com/memsharded/conan/tree/feature/installer_reorder
• https://github.com/memsharded/conan/tree/feature/installer_parallel

Likely that 1.22 will also get experimental download cache, which can also speed up much installations, especially in CI

[dmn-star]

Will conan try to compile the downloaded recipes in parallel too?

[memsharded]

No, this only applies to downloading pre-compiled binaries.
Compiling recipes in parallel would be a different story, not sure about the value. Parallel download of binaries can benefit all conan users. But building in parallel, not that clear, many users already build in parallel, but at the process level or even better, at a CI job level. And if building in the same machine and compilation is slow, typically most of the computing power is put parallelizing each build, as compiler and build systems do also benefit a lot from parallelism. Plus building in the same machine is also a way more complex issue, regarding for example the build outputs cannot longer be streamed to stdout, so where to put them for inspection if something fail? There would be many questions here, so better focus now just on the parallel download only.

[DoDoENT]

I agree with @memsharded - the process should first analyze which binaries can be downloaded and which need building. Then, all downloadable packages should be downloaded in parallel and only then the building of packages that need building should be started (sequentially, as most build recipes already use all available CPUs for building the code). However, if some of those packages have some build requirements that could be downloaded, I would argue that those should be downloaded in parallel download step (prior building). Is that feasible?

[memsharded]

@DoDoENT Yes, absolutely all packages that need to be downloaded will be downloaded before building, including build requirements