Conan: Re-run system_requirements()

[bilke]

I am using the following scenario:

• I am running builds of our software on CI (Jenkins) which rely on some Conan packages
• The packages feature system_requirements() to install some system packages
• The builds run inside Docker container
• The Docker container mount a volume to which all Conan data is cached between CI builds (to save bandwidth and speed things up in general)

This is what happens:

• The first CI build is fine
  
  
  
  • Conan package is not installed
  
  
  • it gets downloaded
  
  
  • system_requirements() install system packages inside the Docker container
  
  
  • whole build runs fine
  
  
• On subsequent builds it fails
  
  
  
  • Conan package is already installed (because it is inside the mounted volume)
  
  
  • Conan does not invoke system_requirements()
  
  
  • System packages are missing in the Docker container
  
  
  • the build fails
  
  

Is there any way to re-run system_requirements() for an already installed package?

[lasote]

No, system_requirements() is run when it's needed to build the package or when it's downloaded, but if it's already there it's not run. this is a very special case where the cache is shared between different systems (docker and host).

[lasote]

Maybe we could think of a special config to force it, but it feels like a corner case, or an explicit way to call it, like local commands.

[bilke]

Thanks for your answer! Yes it is indeed a corner case. For the moment I will rethink my setup..

[memsharded]

Well, I can provide you an easy workaround, that might work for your CI: Just remove system_reqs.txt files from your conan local cache. They are the ones representing that those system requirements have been installed.

[bilke]

Please note that currently (Conan 1.7.4) there are directories called system_reqs which have to be removed (not files system_reqs.txt) as a workaround.

But even when those directories are removed system_requirements() is not run again. @memsharded Is there anything more I have to do?

[lasote]

Let's take a look for the following release.

[bilke]

Thanks @lasote. I guess _handle_system_requirements should be called in _handle_node_cache too, maybe here: https://github.com/conan-io/conan/blob/719d66d1843a5180b665e81249fb34316844a34e/conans/client/installer.py#L306-L309

[yangcha]

Please note that currently (Conan 1.7.4) there are directories called system_reqs which have to be removed (not files system_reqs.txt) as a workaround.

But even when those directories are removed system_requirements() is not run again. @memsharded Is there anything more I have to do?

@bilke , how did you remove the system_reqs folder in recipe?

[bilke]

@yangcha I don't remove the folder from Conan. I remove it during our CI run just before the conan install call.

[yangcha]

@yangcha I don't remove the folder from Conan. I remove it during our CI run just before the conan install call.

@bilke Just be nitpicky, is it better to remove them in the cleanup stage? It is kind of like garbage collecting in destructor() or in the try-finally clause. Also globally searching in CONAN_USER_HOME could be time consuming. Maybe I can write a plugin which will delete them without global searching.
I proposed in #3812 to store those files under the user's home directory. Unfortunately it was not accepted.