Conan: Performance issue & multithreaded builds support

Created on 2 May 2016 · 20Comments · Source: conan-io/conan

I have 521 Visual Studio projects and among those projects there are 282 projects referencing at least one Conan package. When I run the pre-build step I created for installing the Conan packages on all the projects (non multi-threaded) it takes 6 hours. If a project references one or several packages it takes at least one minute to perform the installation even if all the packages are already installed, here is an example from the build log:

[exec] Compiling   510/  521 ...\JsPrseWS.vcxproj (Debug/x64)
[exec]     Requirements
[exec]         boost/1.55.0@demo/testing from local
[exec]     Packages
[exec]         boost/1.55.0@demo/testing:2c0843cc59ff2d07e33c808b3398bc624e6b54e4
[exec]
[exec]     boost/1.55.0@demo/testing: Already installed!
[exec]     PROJECT: Generated conaninfo.txt
[exec]     PROJECT: Generated visual_studio created conanbuildinfo.props
[exec]     PROJECT imports(): Copied 26 '.pdb' files
[exec]     PROJECT imports(): Copied 31 '.dll' files
[exec]
[exec]   Compilation time: 80,7803049s
[exec]

If I run a multi-threaded build I get errors because the "imports()" step sometimes tries to copy the same files to the same destination.
There are two issues, or one issue and one question:
1) Why is it that when I run "conan install ..." it takes at least 20 seconds to respond (most of the time it is more around 60 seconds).
2) How can I handle the "imports()" step so that it correctly handles concurrent copy to the same output folder?

obsolete

Source

youcefl

Most helpful comment

I've finally been able to do some tests using 0.9.4 version and the situation is much better, now it takes 10 minutes instead of 6 hours. Thanks!

youcefl on 28 Jun 2016

👍3 😄1

All 20 comments

Hi Youcef,

Conan is not thread-safe, it is not designed to run concurrently (it can work in parallel for different projects, what it cannot handle is concurrent installation (including import) of the same package.

Those times are too high, indeed, lets do some profiling and see what can we optimize. From the output, I'd say that most of the time could be due to the "imports" step, as, for example, .pdb files are typically large. Another source could be the checks for upstream updates of packages, maybe we can make it opt-in or opt-out, that would alleviate the initial delay. I will setup some performance tests, and keep you updated.

Maybe, in the meanwhile, for such a large project, the best could be to do a multi-configuration install, so it has to be done only once while setting-up the project or if requirements change, but not doing it at every build in a pre-build step. You can check: http://docs.conan.io/en/latest/manage_deps/workflows.html#multi-configuration. Have you tried this approach? It not, please try it, while we investigate these performance issues and try to improve it. Thanks for reporting!

memsharded on 2 May 2016

Hi,
thanks for the quick answer, I arranged things so that the "conan install ..." command is run only if options related to packages have changed (or if it is a first time build). I think what I did is equivalent to what is described in the multi-configuration page you're pointing to, the issue is that the first build where I only run packages installation takes lots of time.

youcefl on 2 May 2016

Hi,

Did my first "performance check" session. The first "bottleneck" for such a large project is the check for updates. At each install command, each package recipe and binary looks for upstream updates, which multiply x 500 gives the large initial delay. The typical project size is in the order or tenths of packages, not hundreds, that is why it wasn't noticed. I think we can easily make it opt-in or opt-out:

opt-in: Fast for everybody. Cons: Updates are not noticed automatically.
opt-out. The opposite.

Maybe the install is doing too much, and the check for updates should be explicit (i.i opt-in), please @lasote, tell us your toughts.

The fix for this is not difficult, it can be ready soon. Release 0.9 is ready, so probably it will not make it into it, but we will find some way, so you can test the improvements with your large project soon, so we can move forward one bottleneck at a time. Are you using conan installed with pip?

memsharded on 2 May 2016

I agree, the install command is doing to much things. The Conan info command is already showing the available updates so the install -u should be enough. the behavior would be similar to pip, you are not notified about updates with an install command.

lasote on 2 May 2016

Yes, probably could be cached if necessary in the future. Lets do:

Remove the check for updates in "conan install". No network calls if all packages locally
Make it opt-in in "conan info"? Something like "conan info --update (-u)" could make sense, in line with the conan install, and will make conan info fast by default.

memsharded on 2 May 2016

memsharded wrote: "[...] Are you using conan installed with pip?"
The answer is no, I ran the build using the conan installed by the Windows installer (version 0.8.6).

youcefl on 3 May 2016

Ok, no problem, we can also generate custom win installers, so you can try things if you want, before the release. Thanks again.

memsharded on 3 May 2016

Hi Youcef,
I have already improved the first part, and submitted a PR: https://github.com/conan-io/conan/pull/252
If approved, I will generate a win installer for you, so you can try it too. Cheers

memsharded on 5 May 2016

It's merged now, sorry for the delay.

lasote on 6 May 2016

Hi @youcefl

Could you be able to try this installer? https://s3-eu-west-1.amazonaws.com/conanio-staging/conan0_10_pre1.exe

It optimizes only the first part, computing the deps, should be much faster. The "imports" still the same.
I am doing big refactors in the model to speed up things, will check the imports issue asap.

Also, I am very interested in your project, why do you have such a large (521! projects) project, what kind of project, how many people/devs... knowing more about use cases, companies, etc. helps us a lot to focus development on conan itself. If possible, you could also write me to james at conan dot io (as this is a bit off-topic from this thread). Thanks very much, cheers!

memsharded on 6 May 2016

~~Windows (defender) is complaining a bit about the installer, but I have checked it, should be safe, please tell me if you have any problem.~~
@lasote has regenerated the package and now defender is ok with the new one. Hope it helps!

memsharded on 6 May 2016

I have submitted a new PR, with a small but effective optimization: do not compute SHA1 ids again and again: https://github.com/conan-io/conan/pull/259. Can reduce 15% for 250 packages projects, and 5x in processing empty builds if packages are depending one on the next one.

memsharded on 11 May 2016

Hi @youcefl ,

We have released a minor, 0.9.1, and included these performance improvements in it (http://docs.conan.io/en/latest/changelog.html), so you might be able to try it from the latest installer. All included but the latest one, cause the PR is still open. Still working on improving performance, next step, the imports.

Please upgrade (https://www.conan.io/downloads) and give feedback, thanks!

memsharded on 11 May 2016

Hi again @youcefl,

I am working in improving the slow "imports" now. I would need some feedback on the patterns, as there are several options, like multithreading the imports, or maybe doing the filename matching and then bulk copying. So things like:

Number of files and total size, that are being copied to your project in the imports (that is the reference size, and the limit for speed)
What files are you copying, the matching patterns. All your packages have things to copy? All shared libraries? Would it be better if the package name could be specify in the imports pattern, so it optimizes and only looks in that package?
Why copying .pdb? These are typically large, but could they be loaded from the store (the path is provided), so the debugging information is loaded without needing to copy them? The proposed imports solution is for the DLLs, as we dont want to modify the path, and provide a developer-solution that do not depend on it. But adding those paths to the system path will also work, and in this case, it will be extremely fast, no imports at all.

Willing to move this forward, please, any information that you could provide would be very useful, thanks!

memsharded on 13 May 2016

Hi!
Sorry for the delay, I've been on holidays. I'll upgrade and see how it performs then I'll give you some feedback. Thanks a lot and once again sorry for the delay.

youcefl on 17 May 2016

No problem at all! And no apologies, your project is unusually large, so it is helping a lot to optimize things. Take your time, don't get stressed too quickly after your holidays :) Just tell me when you are able to, and I will continue with it. Cheers, and thanks to you.

memsharded on 17 May 2016

I've finally been able to do some tests using 0.9.4 version and the situation is much better, now it takes 10 minutes instead of 6 hours. Thanks!

youcefl on 28 Jun 2016

👍3 😄1

Glad it improved!
We might be able to further improve speed, if you could give some feedback about the questions above, that would be very useful. I guess now most of the time goes to "import" things, is that right? Thank you!

memsharded on 29 Jun 2016

I have 524 projects among those 286 depend on at least one conan package,
running installation of all needed packages takes 711 seconds when none of
the packages is installed when the build starts.
Here is a list of the packages with useful info for each one of them:

name            size of package folder
                (MB/num. of files/num. of folders)
boost           457/10071/852
googletest      14.3/32/14
icu             28.8/192/5
libxml2         13.9/54/4
libxlst         3.1/42/5
llvm            906/1293/76
mockpp          6.3/263/29
protobuf        110/54/12
python          28.6/149/3
xerces          18.9/501/39

Example of output for a project referencing several packages:

Compiling     1/  525 E:\***************.vcxproj (Release/x64)
  Build started 29/06/2016 13:10:30.
  Project "E:\***************.vcxproj" on node 1 (InstallCppPackages target(s)).
  InstallCppPackages:
    conan  install -f "************\x64\Release\cpp_pkg_ref.txt" -s "compiler=Visual Studio" -s "compiler.version=12" -s "build_type=Release" -s "arch=x86_64" -r conlan 
    boost/1.55.0@y******/stable: Not found, retrieving from server 'conlan' 
    boost/1.55.0@y******/stable: Trying with 'conlan'...
    Downloading conanmanifest.txt

    [                                                  ]                  
    [=                                                 ]                  
    [==                                                ]                  
    <... snipped ...>
    [==================================================]                  
    icu/50.1.0@y******/stable: Not found, retrieving from server 'conlan' 
    icu/50.1.0@y******/stable: Trying with 'conlan'...
    Downloading conanmanifest.txt

    [===                                               ]                  
    <... snipped ...>
    [==================================================]                  
    libxml2/2.9.1@y******/stable: Not found, retrieving from server 'conlan' 
    libxml2/2.9.1@y******/stable: Trying with 'conlan'...
    Downloading conanmanifest.txt
    <... snipped ...>
    [==================================================]                  
    libxslt/1.1.28@y******/stable: Not found, retrieving from server 'conlan' 
    libxslt/1.1.28@y******/stable: Trying with 'conlan'...
    Downloading conanmanifest.txt
    <... snipped ...>
    [==================================================]                  
    xerces/3.1.1@y******/stable: Not found, retrieving from server 'conlan' 
    xerces/3.1.1@y******/stable: Trying with 'conlan'...
    Downloading conanmanifest.txt
    <... snipped ...>
    [==================================================]                  
    Requirements
        boost/1.55.0@y******/stable from conlan
        icu/50.1.0@y******/stable from conlan
        libxml2/2.9.1@y******/stable from conlan
        libxslt/1.1.28@y******/stable from conlan
        xerces/3.1.1@y******/stable from conlan
    Packages
        boost/1.55.0@y******/stable:e0a02d496bbb652b6295152dfce0d3937acc0b56
        icu/50.1.0@y******/stable:e0a02d496bbb652b6295152dfce0d3937acc0b56
        libxml2/2.9.1@y******/stable:a4501f33ae09df332b76b4d6f0e5cebffbe83874
        libxslt/1.1.28@y******/stable:a4501f33ae09df332b76b4d6f0e5cebffbe83874
        xerces/3.1.1@y******/stable:e0a02d496bbb652b6295152dfce0d3937acc0b56

    boost/1.55.0@y******/stable: Installing package e0a02d496bbb652b6295152dfce0d3937acc0b56
    boost/1.55.0@y******/stable: Looking for package e0a02d496bbb652b6295152dfce0d3937acc0b56 in remote 'conlan' 
    Downloading conan_package.tgz

    <... snipped ...>

    [==================================================]                  
    boost/1.55.0@y******/stable: Package installed e0a02d496bbb652b6295152dfce0d3937acc0b56
    icu/50.1.0@y******/stable: Installing package e0a02d496bbb652b6295152dfce0d3937acc0b56
    icu/50.1.0@y******/stable: Looking for package e0a02d496bbb652b6295152dfce0d3937acc0b56 in remote 'conlan' 
    Downloading conan_package.tgz

    <... snipped ...>

    [==================================================]                  
    icu/50.1.0@y******/stable: Package installed e0a02d496bbb652b6295152dfce0d3937acc0b56
    libxml2/2.9.1@y******/stable: Installing package a4501f33ae09df332b76b4d6f0e5cebffbe83874
    libxml2/2.9.1@y******/stable: Looking for package a4501f33ae09df332b76b4d6f0e5cebffbe83874 in remote 'conlan' 
    Downloading conan_package.tgz


    <... snipped ...>

    [==================================================]                  
    libxml2/2.9.1@y******/stable: Package installed a4501f33ae09df332b76b4d6f0e5cebffbe83874
    libxslt/1.1.28@y******/stable: Installing package a4501f33ae09df332b76b4d6f0e5cebffbe83874
    libxslt/1.1.28@y******/stable: Looking for package a4501f33ae09df332b76b4d6f0e5cebffbe83874 in remote 'conlan' 
    Downloading conan_package.tgz

    <... snipped ...>

    [==================================================]                  
    libxslt/1.1.28@y******/stable: Package installed a4501f33ae09df332b76b4d6f0e5cebffbe83874
    xerces/3.1.1@y******/stable: Installing package e0a02d496bbb652b6295152dfce0d3937acc0b56
    xerces/3.1.1@y******/stable: Looking for package e0a02d496bbb652b6295152dfce0d3937acc0b56 in remote 'conlan' 
    Downloading conan_package.tgz

    <... snipped ...>

    [==================================================]                  
    xerces/3.1.1@y******/stable: Package installed e0a02d496bbb652b6295152dfce0d3937acc0b56
    PROJECT: Generated conaninfo.txt
    PROJECT: Generated visual_studio created conanbuildinfo.props
    PROJECT imports(): Copied 31 '.pdb' files
    PROJECT imports(): Copied 43 '.dll' files
    PROJECT imports(): Copied 1 '.exe' files: xsltproc.exe
  Done Building Project "E:\***************************.vcxproj" (InstallCppPackages target(s)).

  Build succeeded.
      0 Warning(s)
      0 Error(s)

  Time Elapsed 00:02:24.50

  Compilation time: 144,6454207s

Another example:

Compiling     3/  525 E:\******************************Tests.vcxproj (Release/x64)
  Build started 29/06/2016 13:12:56.
  Project "E:\******************************Tests.vcxproj" on node 1 (InstallCppPackages target(s)).
  InstallCppPackages:
    conan  install -f "E:\******************************\x64\Release\cpp_pkg_ref.txt" -s "compiler=Visual Studio" -s "compiler.version=12" -s "build_type=Release" -s "arch=x86_64" -r conlan 
    googletest/1.6.0@y******/stable: Not found, retrieving from server 'conlan' 
    googletest/1.6.0@y******/stable: Trying with 'conlan'...
    Downloading conanmanifest.txt

    [=============                                     ]                  
    <... snipped ...>
    [==================================================]                  
    Requirements
        googletest/1.6.0@y******/stable from conlan
    Packages
        googletest/1.6.0@y******/stable:e0a02d496bbb652b6295152dfce0d3937acc0b56

    googletest/1.6.0@y******/stable: Installing package e0a02d496bbb652b6295152dfce0d3937acc0b56
    googletest/1.6.0@y******/stable: Looking for package e0a02d496bbb652b6295152dfce0d3937acc0b56 in remote 'conlan' 
    Downloading conan_package.tgz

    [                                                  ]                  
    <... snipped ...>
    [==================================================]                  
    Downloading conanmanifest.txt

    [========================                          ]                  
    [================================================  ]                  
    [==================================================]                  
    Downloading conaninfo.txt

    [==================================================]                  
    googletest/1.6.0@y******/stable: Package installed e0a02d496bbb652b6295152dfce0d3937acc0b56
    PROJECT: Generated conaninfo.txt
    PROJECT: Generated visual_studio created conanbuildinfo.props
    PROJECT imports(): Copied 2 '.pdb' files: gtest_unittest.pdb, gtest_prod_test.pdb
    PROJECT imports(): Copied 2 '.exe' files: gtest_unittest.exe, gtest_prod_test.exe
  Done Building Project "E:\******************************************Tests.vcxproj" (InstallCppPackages target(s)).

  Build succeeded.
      0 Warning(s)
      0 Error(s)

  Time Elapsed 00:00:03.24

  Compilation time: 3,351257s

Compiling     4/  525 E:\*************************************Tests.vcxproj (Release/x64)
  Build started 29/06/2016 13:12:59.
  Project "E:\*************************************Tests.vcxproj" on node 1 (InstallCppPackages target(s)).
  InstallCppPackages:
    conan  install -f "E:\*************************************\x64\Release\cpp_pkg_ref.txt" -s "compiler=Visual Studio" -s "compiler.version=12" -s "build_type=Release" -s "arch=x86_64" -r conlan 
    Requirements
        googletest/1.6.0@y******/stable from conlan
    Packages
        googletest/1.6.0@y******/stable:e0a02d496bbb652b6295152dfce0d3937acc0b56

    googletest/1.6.0@y******/stable: Already installed!
    PROJECT: Generated conaninfo.txt
    PROJECT: Generated visual_studio created conanbuildinfo.props
    PROJECT imports(): Copied 2 '.pdb' files: gtest_unittest.pdb, gtest_prod_test.pdb
    PROJECT imports(): Copied 2 '.exe' files: gtest_unittest.exe, gtest_prod_test.exe
  Done Building Project "*************************************Tests.vcxproj" (InstallCppPackages target(s)).

  Build succeeded.
      0 Warning(s)
      0 Error(s)

  Time Elapsed 00:00:00.61

  Compilation time: 0,7014987s


<.... snipped ....>


Compiling    21/  525 E:\*************************************.vcxproj (Release/x64)
  Build started 29/06/2016 13:13:04.
  Project "E:\*************************************.vcxproj" on node 1 (InstallCppPackages target(s)).
  InstallCppPackages:
    conan  install -f "E:\*************************************\x64\Release\cpp_pkg_ref.txt" -s "compiler=Visual Studio" -s "compiler.version=12" -s "build_type=Release" -s "arch=x86_64" -r conlan 
    protobuf/2.5.0@y******/stable: Not found, retrieving from server 'conlan' 
    protobuf/2.5.0@y******/stable: Trying with 'conlan'...
    Downloading conanmanifest.txt

    [=======                                           ]                  
    <... snipped ...>
    [==================================================]                  
    Downloading conanfile.py

    [=============================================     ]                  
    [==================================================]                  
    Downloading conan_export.tgz

    [                                                  ]                  
    [=                                                 ]                  
    <... snipped ...>
    [==================================================]                  
    Requirements
        protobuf/2.5.0@y******/stable from conlan
    Packages
        protobuf/2.5.0@y******/stable:e0a02d496bbb652b6295152dfce0d3937acc0b56

    protobuf/2.5.0@y******/stable: Installing package e0a02d496bbb652b6295152dfce0d3937acc0b56
    protobuf/2.5.0@y******/stable: Looking for package e0a02d496bbb652b6295152dfce0d3937acc0b56 in remote 'conlan' 
    Downloading conan_package.tgz

    [                                                  ]                  
    <... snipped ...>
    [==================================================]                  
    Downloading conanmanifest.txt

    [=============                                     ]                  
    <... snipped ...>
    [==================================================]                  
    Downloading conaninfo.txt

    [==================================================]                  
    protobuf/2.5.0@y******/stable: Package installed e0a02d496bbb652b6295152dfce0d3937acc0b56
    PROJECT: Generated conaninfo.txt
    PROJECT: Generated visual_studio created conanbuildinfo.props
    PROJECT imports(): Copied 5 '.pdb' files
    PROJECT imports(): Copied 4 '.exe' files: test_plugin.exe, lite-test.exe, tests.exe, protoc.exe
  Done Building Project "E:\*************************************.vcxproj" (InstallCppPackages target(s)).

  Build succeeded.
      0 Warning(s)
      0 Error(s)

  Time Elapsed 00:00:09.13

  Compilation time: 9,2371077s

<.... snipped ....>

Compiling    31/  525 E:\*****************************************.vcxproj (Release/x64)
  Build started 29/06/2016 13:13:15.
  Project "E:\*****************************************.vcxproj" on node 1 (InstallCppPackages target(s)).
  InstallCppPackages:
    conan  install -f "E:\*****************************************\x64\Release\cpp_pkg_ref.txt" -s "compiler=Visual Studio" -s "compiler.version=12" -s "build_type=Release" -s "arch=x86_64" -r conlan 
    Requirements
        boost/1.55.0@y******/stable from conlan
    Packages
        boost/1.55.0@y******/stable:e0a02d496bbb652b6295152dfce0d3937acc0b56

    boost/1.55.0@y******/stable: Already installed!
    PROJECT: Generated conaninfo.txt
    PROJECT: Generated visual_studio created conanbuildinfo.props
    PROJECT imports(): Copied 26 '.pdb' files
    PROJECT imports(): Copied 31 '.dll' files
  Done Building Project "E:\*****************************************.vcxproj" (InstallCppPackages target(s)).

  Build succeeded.
      0 Warning(s)
      0 Error(s)

  Time Elapsed 00:00:01.29

Hope this helps. Let me know if you want more info.

youcefl on 29 Jun 2016

The original performance issue was solved time ago.

The binary performance issue, with the latest features: