Conan: How to import dependencies only for a project not global on pc

[gnobbi]

Hi conan team,
thanks for this great tool again :)

We have the scenario that we want use conan as package manager on a build server that build and provides the binaries with conan_server.
Than we want to use the import mechanism in the conanfile.txt to copy the binaries to the project folder.
This works fine but it first copy it to the conan home directory of the local pc and then it copy the binaries to project.

My question is there a direct way from the conan_server to the the project folder if the binaries are available on the conan_server?

We want to keep the local pc clean and our workaround is to remove the conan package after project setup.

Thank you

[memsharded]

Not sure what you want to achieve, but for sure you can set your CONAN_USER_HOME to your current dirs (or better, to your <cwd>/deps or something like that), then everything will be downloaded there, and linked from there. The folder for installing those dependencies, as well as the settings.yml, the remotes, the user logins, everything, will be specific for that project, and isolated to other projects. Is this what you are looking for?

[lasote]

I think they don't want to copy the artifacts first to the cache, they want directly from the server to the user space. It's not possible. You could change the cache folder as memsharded said or you could clean the local cache before the installation maybe using a wrapper script 'conan_imports.sh/bat' that first call 'conan install $' and then 'conan remove $ -f' or something similar.

[gnobbi]

Hey guys,
thank you for your fast reply.

We want to achive a behavior like the npm package manager where you can choose a local or a global installation on your local pc.

I have now a working solution that only change the the storage.path of the global config and set it back to the old one after conan install.

I'm not a python profi but this works for me on windows

from subprocess import Popen, PIPE, call
import os
import sys

# get old path
get_path_cmd = ["conan", "config", "get", "storage.path"]
p = Popen(get_path_cmd, stdin=PIPE, stdout=PIPE, stderr=PIPE)
output, err = p.communicate(b"input data that is passed to subprocess' stdin")
rc = p.returncode

old_path = output.decode("utf-8")
print("old path : {0}".format(old_path))

# get new path
temp_path = os.path.join(os.getcwd(), "deps")
os.makedirs(temp_path, exist_ok=True)
new_storage_path = ("{0}".format(temp_path).replace("\\", "/"))
print("new path : {0}".format(new_storage_path))

#set new path
set_new_path_cmd = ["conan", "config", "set", "storage.path={0}".format(new_storage_path)]
call(set_new_path_cmd)

#install local
install_cmd = ["conan", "install", "..\\source", "--build=missing"] #--build=missing should be not set if we want only the builded binaries
call(install_cmd)

#set back to old path
set_old_path_cmd = ["conan", "config", "set", "storage.path={0}".format(old_path)]
call(set_old_path_cmd)

that would be a nice feature for us but i don't know if this a common use case.

[memsharded]

This could have some issues: the conan local cache stores in the registry.txt files the origins of each package (the remote from which they were installed). Temporarily moving the storage to different locations would leave this in unconsistent state.

Why setting CONAN_USER_HOME is not valid? It will put the complete cache, including the metadata in the deps folder, and everything will be consistent.

Also, could you explain a bit further what is your use case? You are trying to avoid a copy (with the imports() approach), but if you do that per-project, you will actually end with multiple copies of the same binaries spread in many different user projects anyway.

[gnobbi]

Ok this is be a problem that i'm not noticed.
Before this i tried to set the CONAN_USER_HOME to the local but then my global config with the remotes in the registry is lost.
And i thought the other approach would be easier.
Maybe this is the best solution and i can copy that from the real CONAN_USER_HOME or load it with conan config but i don't know now how should i do that.

Nevertheless I will explain our use case or idea how to use conan:

First of all we are in the state that we not use conan for our complete build process we want check if we can use conan to handle our third party dependcies.

• Let's assume we have build server that provides conan packages and binaries of the third party projects .
• The local pc has not to build a dependency because every configuration we need is provided by the build server.
  
  
  
  • If the buid server has not the binaries for this configuration this could be an error.
  
  
• Then we want to add only the "binaries" (conan package folder) of dependencies to our project without the local cache in the real CONAN_USER_HOME.
• We also want the generated cmake file provides links to the local deps folder.

For example

conanfile.txt
[requires]
opencv/x.x.x@build_server/stable

[generators]
cmake

#get_depencies.bat/.py
conan install . --local --local_folder=./deps -r build-server -s COMPLETE_CONFIGURATION

The result might be the
./deps/opencv/x.x.x/build_server/stable/conanbuildinfo.cmake
./deps/opencv/x.x.x/build_server/stable/package/<with all files from the server for this configuration>

#conanbuildinfo.cmake
set(CONAN_OPENCV_ROOT "..deps/opencv/x.x.x/build_server/stable/package/5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9")
set(CONAN_INCLUDE_DIRS_OPENCV"..deps/opencv/x.x.x/build_server/stable/package/5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9/include")
...

I'm not sure with or with out the configuration hash ... but may be in a way like this. So we are independent and have no links to the real CONAN_USER_HOME.

But your approach do this isn't it?

[memsharded]

Thanks very much for the detailed info!
But there is still a piece of information that I am missing. It is why you want the binaries in your local folder for those third-party libraries, without all the rest. What are you doing with your project after that. Just building?

The big issue with that approach is that you lose all information and management of those binaries. If you switch the build from debug to release, then you will be using the wrong binaries maybe. Or if having to switch compiler version. When the binaries are in a local conan cache, then, everything is kept consistent, and the correct binaries will be used. If you extract them, you lose the way conan manages them. Why do you really mind that the binaries for third party libs that your project is linking are in one place or other?

In any case, it seems the biggest issue with the CONAN_USER_HOME was that you lost the configuration of the remotes. It is strongly recommended to also automate the project or team configuration, and conan config install could be the perfect way to achieve this (it can use remote git/zip-url config, or even local zip files or git repos). Such command will be able to set the correct remotes for you, and also profiles and other config that you want to have for yor project. Maybe this helps? With that I think the process could be quite smooth:

$ export CONAN_USER_HOME=<cwd>/deps
$ conan config install /path/or/url/to/project/config
$ conan install . -r build_server -pr=profile_installed_with_config

Everything will be in your projects deps folder, the conanbuildinfo.cmake will point to it. There will be hashes, but I strongly suggest keeping them, in case you want to manage multiple configurations of binaries. What do you think?

[gnobbi]

Thank you I think the way with the zipped configuration is the best solution.

For the question what we do after this. We want to use this to develop/debug our code that depends on 3rdParty binaries on a local pc.
We want to check out our code and then call get dependencies that is managed by Conan. And then develop/debug it. I hope this answers your question.

In some older projects we added the 3rdParty dependencies to the repository. And we want to avoid it.

Thank you for your great support and the working solution.

[memsharded]

Ok, great! :)

So, if you think the zipped configuration helps, and you are ok with that approach and this issue doesn't require further action from us, then you might close it.

Regarding the develop/debug your code, that can also be done without copying the artifacts to the local folder, but just linking and debugging using the artifacts in the conan local cache (even the per-user global one, not the per-project one). If you want to join us on slack to be able to chat and discuss this and other things, please drop us an email to [email protected]. Thanks!