conan package_files preserve folders/dirs used in --package_folder= option?

[Twoflower2]

Using conan package_files, how can this folder/dir be preserved in the conan cache packageshaxxx, used in the --package_folder= option in package_files?

[lasote]

Sorry, I don't understand the question. What command are you trying to run/what are you expecting to get?

[Twoflower2]

I have a dir dist I want to package. Here is a lib test1 lay-out:

├───bin
├───dist
│       └───default
│                └───inc
│                └───bin
├───inc
├───lib
├───src

So, I use
conan package_files test1/0.1.0@user/testing --package_folder=dist/default to package.
Then in the conan cache package\shaxxx these dirs inc and bin resides.

BUT I want to preserve the dir structure in conan cache location packageshaxxx, so that inc and bin resides in dir default under cache packageshaxxx.

So my question is how can I use the conan package_files cmd to put inc and bin when using the --package_folder= option
1) under a specific dir in conan cache location packageshaxxx or
2) preserve the --package_folder= location in the conan cache packageshaxxx

[lasote]

If you want to preserve the default folder specify conan package_files test1/0.1.0@user/testing --package_folder=dist. So default will be copied.
Sorry, I feel I'm missing something. In case you are asking about specifying a non-existent custom directory as a destination directory, it's not possible. package_files command will just copy the specified --package_folder directory to the local cache package/shaxxxx.

[Twoflower2]

Sure, but I left out to add the dist dir lay-out are more complex and looks like this:

├───dist
│       └───default
│                └───inc
│                └───bin
│       └───oss_default
│                └───inc
│                └───bin
│       └───test_default
│                └───inc
│                └───bin

and I want to package default , oss_default and test_default separately and always preserve these dirs.

If I now specify conan package_files test1/0.1.0@user/testing --package_folder=dist all 3x highlevel dirs = everything are copied.

[lasote]

Thanks, now I understand. I think currently it's not possible. I'm labeling the issue and will analyze how to improve it. One possibility could add a new parameter with an (optional) destination folder, in your case, it could be something like:

conan package_files test1/0.1@user/testing --package_folder=dist/default --dest=default

[Twoflower2]

@lasote Also something like #1549

[lasote]

Yes, you are right. With #1549 feature you could benefit from the copy command in the package() method. Could it fit your needs? Take into account that the files layout from your local folders has to be the same than the build folder in the local cache, or at least the package() method has to be able to manage both layouts without collisions.

[Twoflower2]

Unfortunately it won't fit my needs, hence opening this new ticket. I therefore are still interested in your improvement:

One possibility could add a new parameter with an (optional) destination folder, in your case, it could be something like:
conan package_files test1/0.1@user/testing --package_folder=dist/default --dest=default

[memsharded]

The pattern of having a package with the intermediate folder sounds a bit weird to me. Can you please explain why the usage of this intermediate folder, and how are you consuming these packages? Because that makes necessary to use #include "default/inc/header.h", and change it when using the other package binaries, thus requiring changing the consumer source code.

Are you packaging different binaries of the same package (like Pkg/0.1@user/channel, many binaries), or are you packaging different packages (like Pkg1/..., Pkg2/..., etc)?

Please clarify, I would need to get the whole picture to understand this issue. Thanks!

[Twoflower2]

@memsharded read from here https://github.com/conan-io/conan/issues/1646#issuecomment-324011454

[Twoflower2]

The pattern of having a package with the intermediate folder sounds a bit weird to me. Can you please explain why the usage of this intermediate folder, and how are you consuming these packages? Because that makes necessary to use #include "default/inc/header.h", and change it when using the other package binaries, thus requiring changing the consumer source code.

In Eclipse: You would just use #include "header.h" and configure default/inc/ in the Eclipse project settings. (That is if all file names are different 😁 )

Are you packaging different binaries of the same package (like Pkg/0.1@user/channel, many binaries), or are you packaging different packages (like Pkg1/..., Pkg2/..., etc)?

different binaries of the same package with different Conan options.

[lasote]

We are going to remove the support for package_files in favor of export-pkg in the 0.28, where it will be mandatory to use a recipe with a package() method. So probably using the package() method you are able to introduce some logic in python to cover this issue. I'm assigning the issue to 0.29 waiting for feedback using 0.28 when released.

[Twoflower2]

@lasote This is creating a deep concern now. We heavily use package_files and rely on it to work like it does. Will export-pkg work similar ie is it a Conan cmd also?

Please pass all info on about using export-pkg

Yes I would need to try 0.28, which would mean many Conan config changes and many scripts changes to many of our libraries :-(

I would guess we could also stick with version 0.27, because it is working for our needs now.

[Twoflower2]

@memsharded Any reply on https://github.com/conan-io/conan/issues/1646#issuecomment-337826695 please?

[memsharded]

Yes, it will be the command conan export-pkg. It will change some of the existing package_files functionality, but it shouldn't be very difficult to upgrade:

• Regarding the package recipes, the only necessary change might be adding a method:

def package(self):
     self.copy("*")

If you are interested in the exact copy of the layout, the same as the previos package_files did if not provided build-folder information

• The command will be very similar, just require the conanfile.py path. But it will be more complete, like not having to specify the profile or settings if you did a conan install first, which is very convenient for local development. It will also execute the conan export of the conanfile.py, so it will save one step.

The updated docs will follow the release as quick as possible.

Sorry for the deprecation, but it is becoming quite important to do some house-keeping and codebase cleaning, it is the only way we can manage to achieve stability in a reasonable time to be able to get out of beta soon.

[memsharded]

The new export-pkg command has been already released in 0.28. The docs are here: http://docs.conan.io/en/latest/reference/commands/creator/export-pkg.html

With this feature, and adequate modeling of the package, the desired flow should be possible. Something like:

class Pkg(ConanFile):
    ...
    options = {"mode": ["default", "oss", "test"]}
    default_options = "mode=default"

    def package(self):
             #Note I am using self.options.mode as the src for copying
             #can be used as "dst" too
             self.copy("*", src=str(self.options.mode), ...)

Then called in the command line:

$ cd dist
$ conan export-pkg . Pkg/0.1@user/stable -o Pkg:mode=default
$ conan export-pkg . Pkg/0.1@user/stable -o Pkg:mode=oss
$ conan export-pkg . Pkg/0.1@user/stable -o Pkg:mode=test

Could you please have a look and test it? It seems that there is nothing to do on our side, so I would consider this issue fixed at the moment.

[Twoflower2]

@memsharded

I will and get back to you on the results.

[memsharded]

This has been released in 0.29. I'll tentatively close the issue, but please try it and report if necessary. Thanks!