Conan: Enhancement: add '--deploy-folder' parameter to `conan install` and related commands

[solvingj]

Community Feedback Requested

The deploy() method of conanfile.py intends to meet one of the needs we have in our environment, but it does not work for us for a few reasons. We suggest the addition of a new command conan deploy, along with some related enhancements to the deploy() method. This new command would also include the implementation of a default behavior for recipes which do not have deploy() methods defined, which are actually the primary focus of this feature request.

Description of Need

What we need is to extract files out of Conan packages into user-space for use in software-pipeline which are do not involve Conan. This includes creating MSI/RPM/DEB packages containing DLL and executable files and licenses from open-source packages, as well as from our own libraries.

Existing Feature

Conan already has the deploy() method of conanfile to copy files out of Conan cache into user space. Currently, in recipes that have chosen to define this method, it is always run implicitly as part of the conan install command. We find the current design of the deploy() method impractical for all of our cases for the reasons listed below.

1. We need to extract files out of OSS packages, most of which haven't defined deploy()
2. We want to define the destination directory at consumption-time, not package-time.
3. We only want to run deploy() on-demand, not implicitly

Prior Art

The basic functionality to copy files out of a "private" directory into user space is similar in nature to CMake's cmake install command. For the same reasons that the cmake install is typically run with a separate command and independently of cmake build , we feel a dedicated deploy command makes sense for Conan.

Note: Objectively, it might have made more sense for this proposed command to be conan install to align it with cmake install. However since that command is already taken we are suggesting conan deploy instead.

Also with CMake, we have cmake configure, which lets the end-user control the destinations to which CMake will install the files. This feature makes sense because in most cases, it's impractical for the author of the CMake recipe to define precise destination paths that will work on all machines where the recipe is consumed.

Proposed Characteristics

We propose this new feature with the characteristics below because we feel the design is sufficiently generic while also being reasonable to implement. It does not seem to require any extensive re-architecture, new code, or even much new logic to document or maintain. Also, it's behavior is consistent with existing features.

Parameters:
The conan deploy command is requires a prior execution of conan install <packageref>, and thus should run it implicitly if necessary (much like conan create does). It must therefor accept all the same arguments as conan install (profile/settings/options/build-policy/etc), plus the new optional arguments of --deploy-folder and --deploy-override.

Behavior:
For recipe's which have a deploy() method defined:

• The default behavior would be to run this method
• The --deploy-folder should be made available inside the function
• The --deploy-override would skip deploy(), and use behavior below

For recipes which do not have a deploy() method defined:

• The default behavior would be to copy the entire contents of the package folder to CWD
• If the --deploy-folder is provided, it is used instead of CWD
• Intentionally has no support for dependencies.

Future Features:
The most likely functionality not included in this proposal is integration with cmake install. It seems likely that some users will want the default behavior for recipes using CMake to pass the --deploy-folder as the CMAKE_INSTALL_PREFIX or something like this. While reasonable, this could significantly complicate implementation. If so, the suggestion is to wait some time to implement that functionality.

Also, Conan already knows how to put all files for a variant in a .tgz archive when uploading to a remote, an optional parameter of --archive is also proposed to deploy the files as a .tgz.

Demand

Over the past year, there have been numerous requests in slack discussions and Github issues requesting functionality around using Conan binaries outside of Conan. Many have suggested more advanced native features, such as automatically making RPM's and DEB's out of Conan packages, etc. While we feel those features would be novel and useful, they would be difficult to implement and maintain. Conversely, we feel this proposal is a more achievable and supportable feature that makes post-conan steps like making RPM's and DEB's from the Conan binaries much easier.

Feedback

In early discussion with Conan team so far, the point has been raised that a naive implementation of this functionality without dependency support would likely raise many questions, not address all use-cases, and likely result in more feature requests to extend it. While this is almost certainly all true, it does not seem to make it any less valid or valuable as a feature. Conversely, such discussion would demonstrate the demand for features in this area. Regarding the lack of support for dependencies by the default behavior, we find a great many use cases of this feature are for bundling executables and dynamic libraries for delivery to the field. Very often in these cases, the dependencies are not bundled because they are deployed to runtime systems via other mechanisms such as YUM, APT, etc.

• [x] I've read the CONTRIBUTING guide.
• [x] I've specified the Conan version, operating system version and any tool that can be relevant.
• [x] I've explained the steps to reproduce the error or the motivation/use case of the question/suggestion.

[lasote]

It must therefor accept all the same arguments as conan install (profile/settings/options/build-policy/etc), plus the new optional arguments of --deploy-folder and --deploy-override.

Well, a new command deploy doing the same but with new options... not sure. Also, we would need to keep the conan install ref behavior until 2.0, otherwise it will be breaking.

I would start by adding --deploy-folder and --skip-deploy (Doesn't it sound better than --deploy-override?) (and --deploy_all???)

Let's discuss it.

[solvingj]

There's no reason for a breaking change of conan install ref. perhaps I mis-communicated that part. All existing behavior of conan install is intended to stay the same. New behavior for conan install would be processing the --deploy-override flag when specified, but that would not be breaking.

The purpose of --deploy-override was to tell Conan to use the "default deploy behavior" as described, INSTEAD of the recipe-defined deploy() method. --skip-deploy to skip the step altogether would be a separate but indeed useful flag for recipes which have deploy() methods defined.

However, both the items above seem like minor details. Can you please comment on the fundamental needs expressed:

1. Getting files out of packages which have no deploy() method
2. The need for the user to control the destination where files go

We build and package executable's and dll's, and there are many OSS packages with executables and dll's which are typically deployed alone. Very often, we just want the freedom to extract the contents of a specific package and we cant. Does any of that not make sense?

[solvingj]

In retrospect, I could see this working as an addition to conan install ref , but with adding options that effectively enable the deploy behavior for recipes with no deploy() method.

[lasote]

Getting files out of packages which have no deploy() method
The need for the user to control the destination where files go

Sorry I wasn't clear, I like and agree about both features. Just wanted to open the discussion about how/where to implement them.

[solvingj]

So, the minimal step forward appears to be adding two parameters to conan install ref:
--deploy
--deploy-folder

When a recipe has no deploy() method defined, the new behavior when passing --deploy is the equivalent of a deploy() method with a body of self.copy(pattern=*, dst=".").

When a recipe has no deploy() method defined, the new behavior when passing --deploy along with a --deploy-folder="abc" is the equivalent of a deploy() method with a body of self.copy(pattern=*, dst='abc').

When a recipe has a deploy() method defined, the behavior is unchanged.

Are there any objections to the feature with this scope?

[grisumbras]

I played around with deploy method, and I found it to be mostly satisfying. For example, conan install <ref> -if install_root copied files into install_root as I wanted it to. The only 2 things I would have preferred is

• default to copying the contents of package_folder if no the conanfile has no deploy method defined
• have the ability to not create deploy_manifest.txt.

So, it seems --deploy-folder is already handled by --install-folder. And I would suggest renaming --deploy to --force-deploy, because --deploy implies that deployment is only done when this flag is provided. Finally, I suggest adding --no-deploy-manifest argument, that will disable creation of deploy_manifest.txt.

As a result, basically any package could be installed into the system with conan install <ref> -if /usr/local --force-deploy --no-deploy-manifest.

[solvingj]

Thanks for taking the time to experiment. I believe --install-folder is specifically intended to define where the conanbuildinfo files are generated for consumption by other Conan projects. It seems almost incidental that the package contents were also copied there in your case. On the surface, the co-mingling of that path argument for two purposes seems bad.

To me, behaviors relating --deploy feature should be focused on getting files out of Conan cache for use outside of Conan. At least, this is how I interpret the word deploy in this context, which might be subjective. In any case, I've opened another ticket to add features to provide another avenue for getting ahold of cached files for use outside of Conan: https://github.com/conan-io/conan/issues/3648

While I would still like deploy features to be more decoupled from the recipe and from other features like --install-folder, if the conan info and conan install commands can provide json ouptuts, external tools will be more efficient and capable of integrating than trying to use the deploy features as integration points.

To be clear, i still want to have a simple command to copy all package contents into a provided destination.

[solvingj]

Changing title to reflect latest state of this request.

[jgsogo]

Given the plugin feature, does it make sense to think this issue in terms of plugins instead of commands?

[solvingj]

WRT the --deploy-folder, I don't think so. I think the --deploy-folder is analgous to cmake install without having CMAKE_INSTALL_PREFIX. It's just incomplete. Maybe this feature should be called --deploy-prefix.

WRT the default deploy, probably yes, but we'll carry that discussion on in #3457

[luiscarox]

Came across something similar the other day.

Perhaps this can help:

conan info libpng/version@user/channel -r remote | grep -B1 ' ID:'

It will print something like:

libpng/ver@user/channel
    ID: 4486f1c64a5b4a3ae46e5809140d6243348211b8
--
zlib/ver@user/channel
    ID: de56269f90f07cc61427d9012acf098d4354f43d

(Which is more or less parseable by a different tool).

Then maybe use conan get to retrieve the package directly:

conan get -p de56269f90f07cc61427d9012acf098d4354f43d -r remote -raw zlib/ver@user/channel  conan_package.tgz >> zlib.tgz

(I didnt get that far and the downloaded .tgz seemed to be corrupted, but it did appear to be downloading something!)

That way you could get around the lack of deploy() for a given package. That said, there are a number of conan install generators that will give you all the relevant root paths in a way that you can parse that and copy them somewhere via scripting. Perhaps not ideal, but I suppose you could also write up a new generator that mimics what "deploy" would have done anyway.
E.g.
conan install ../conanfile.py -g copy_all_files_to_the_install_folder

[solvingj]

Actually, the best alternative we've found is:
conan install zlib/1.2.11@conan/stable -pr default

conan info zlib/1.2.11@conan/stable --paths -n package_folder --package-filter zlib/1.2.11@conan/stable -pr default

Replace -pr default with any profile you like, or the combination of settings and options.

From that point forward, you can easily parse out the package_folder, and then enumerate and/or copy whatever files we want to whatever destination we want.

It will be better if/when this conan_info output supports json output, but it's reasonable right now.

[jgsogo]

As the conan info with json output is in issue https://github.com/conan-io/conan/issues/3694, does it makes sense to close this one and focus on that one?

[monsdar]

Getting files out of packages which have no deploy() method

Let's say I want to deploy the binaries that the Boost package from conan-center is providing. Could I solve this with a proxy package?

from conans import ConanFile

class BoostDeployConan(ConanFile):
    name = "BoostDeploy"
    version = "1.0"
    settings = "os", "compiler", "build_type", "arch"

    def requirements(self):
        self.requires("boost/1.69.0@conan/stable")

    def deploy(self):
        #Code that deploys the artifacts from boost/1.69.0@conan/stable
        ...

That solution would work right out of the box, does not change any existing recipes and gives you a lot of control about what to do with the files.

[monsdar]

The need for the user to control the destination where files go

UPDATE: Learned that the following approach won't work easily with the current version of Conan.

Of course a standardized Conan-wide solution would be way better, but if you have control over all your packages could this be implemented with options?

Something like the following:

from conans import ConanFile

class FooConan(ConanFile):
    name = "foo"
    version = "1.0.0"
    settings = "os", "compiler", "build_type", "arch"
    options = {"deploypath": "ANY"}
    default_options = {"deploypath": "c:\\temp\\"}

    def deploy(self):
        self.copy("*", dst=self.options.deploypath)

Would result in conan install foo/1.0.0@user/channel -o deploypath="~/mydestination".

I'm having a few open questions in that regards though (sorry for not taking more time and trying these out before posting):

• Is deploy() being called recursively for required packages as well? Then using an option would install everything into the same destination. Could be a neat side-effect. (UPDATE: I just checked, deploy is not called recursively. There was an issue for that: https://github.com/conan-io/conan/issues/3360)
• Are prebuilt binaries of a package tightly coupled to the options they've been built with? That would be a showstopper, as changing the deploypath would trigger a recompilation of all packages, surely not what is intended in this case. (UPDATE: Checked this as well... changing an option needs the whole package to be rebuilt. So the approach I'm describing here isn't really working out. Added this feature request: https://github.com/conan-io/conan/issues/4654)

[monsdar]

And last but not least:

We only want to run deploy() on-demand, not implicitly

This would be awesome and certainly a feature that we'd need as well. There are usecases where it is not necessary/wanted to deploy right after install. Scenarios I could think of:

• When developing I need boost/1.69.0@conan/stable to compile and link against Boost-functionality. From a devs perspective boost does not need to be deployed onto my system, I just need Conan to find the right files during build. On my target system though I need to have the runtime libraries available, so I'd want to copy them into a specific directory.
• Sometimes my target system is not connected to the Conan Server. Imagine having your software deployed on-premise on a customers site. When splitting install and deploy the following would be possible:
  
  
  
  • On my dev computer do conan install product/version@user/channel. All gets built and installed into the local conan cache
  
  
  • Put that cache onto a removable drive, have a portable conan version on it as well
  
  
  • Go to the customer, plug in that removable drive
  
  
  • Do conan deploy product/version@user/channel at the customers site
  
  
  • Everything gets installed right out of the conan cache
  
  

[jgsogo]

In Conan v1.15 it was introduced a deploy generator (https://github.com/conan-io/conan/pull/4972 | documentation) that could be enough for the requested functionality: it retrieves the files from the packages to a user folder. After this action, a local script can pick the required files and build the tarball/installer/... or whatever is needed for distribution.

We are also investigating if it makes sense to implement some _packagers_ as generators or if we should provide scripts into Conan or just some docs on how to do it; but for the general use case, IMO, the deploy-generator + user-script should be enough.

[solvingj]

i agree that the deploy generator is close enough to close this issue. I'll continue to watch for future development in this area.