Conan: exports not copied when inheriting from another package with python_requires

[liberforce]

I tried inheriting from the bzip2/1.0.6@conan/stable package. I wanted to generate an additional file at the end of the build. So here's what I did:

import conans

base = conans.python_requires('bzip2/1.0.6@conan/stable')

class Bzip2QiToolchain(base.Bzip2Conan):
    def build(self):
        print("Calling parent class")
        super(Bzip2QiToolchain, self).build()
        self.generate_metadata()

    def generate_metadata(self):
        pass

The problem here is that the bzip2 package has a CMakeLists.txt listed in its exports, that it copies using:

shutil.move("CMakeLists.txt", "%s/CMakeLists.txt" % self.zip_folder_name)

The problem is that the source CMakeLists.txt it want to copy doesn't exist, so the copy fails.

Using Conan 1.7.3 on Linux.

• [X] I've read the CONTRIBUTING guide.
• [X] I've specified the Conan version, operating system version and any tool that can be relevant.
• [X] I've explained the steps to reproduce the error or the motivation/use case of the question/suggestion.

[memsharded]

Just a quick check: are you sure that you also have a CMakeLists.txt in your current package that inherits from bzip2? It is not that it is not exported because the declaration of exports is not inherited, is that the file is not there to be exported. Could be?

[danimtb]

Python requires reuse just the python base class, not the packaging and export sources. Here the CMakelitst.txt is missing next to your Bzip2QiToolchain recipe.

[liberforce]

No, I don't have a CMakeLists.txt in my package, as I had no plans to change the one in the upstream bzip2 package. I don't want to change the way it's built, I just want to generate and install a new file.

Without knowing how it's implemented, my expectations were subclassing a package would work like subclassing a class. I.e., that I would get the whole upstream package, be able to add new data and behavior. But I expected this to work using overlays, so that I could replace upstream files if wanted, but not have to copy them, which adds maintenance burden.

[danimtb]

Maybe we have to clarify its purpose here https://docs.conan.io/en/latest/mastering/python_requires.html
but in the first sentence it says:

The python_requires() feature allows to reuse python from other conanfile.py

[liberforce]

Ok, so it seems I misunderstood the aim of the thing. It's useful when you control the code you inherit from, and want to factorize some code, but that's not the same use case as mine.

I'd like to create a very small variation of a package, but without forking the whole recipe which wouldn't scale on a maintenance level. I thought inheriting from the recipe would give me the ability to inherit from the whole package. There's a bit of a confusion on the docs too, as python_requires is given a package name, and that's what we derive from afterwards.

In the example:

from conans import python_requires

base = python_requires("MyBase/0.1@user/channel")

class PkgTest(base.MyBase):
    pass

When I tried to adapt that example, I first had an issue as I used base.bzip2 (the package name) instead of base.Bzip2Conan (the class name) as a base class, thinking there was some machinery below to detect the ConanFile class used.

So as I have clearly misunderstood the goal of this feature, is there a way for me to derive a whole package ? Forking lots of recipe repositories just to add a very small modification there is cumbersome.

[memsharded]

I am afraid that won't be possible, extending a whole package behavior would be equivalent to trying to extend a filesystem. The "data" files, (i.e. the source files), cannot be extended or inherited.

Depending on what exactly what you want to do, the possibilities would be:

• Forking the recipes. Even for small modifications, I really think it is the conceptual correct mode
• Using the plugins (WIP) to inject those modifications in an aspect-oriented approach, to all recipes you want. We are aiming to release a first preview of plugins in 1.8. Given your "Inject metadata at the end of the build" use case, I'd say a plugin "post_build" would be probably the perfect solution for your use case.

[liberforce]

The "data" files, (i.e. the source files), cannot be extended or inherited.

Right, but they could be overlaid. I.e., if they're provided in the derived package, use those, otherwise use the base package ones.

[memsharded]

@liberforce this has been implemented, will be released in conan 1.9. Files exported in python-requires exports_sources can be automatically merged to the consumers of such python-requires, accordingly to their exports_sources attribute. If they inherit and extend from a base ConanFile, they will inherit the exports_sources and get them automatically! I will be updating the docs for 1.9 accordingly.

[sourcedelica]

Could you explain how this works? Is it supposed to copy the files from the export_sources directory in the python_requires package to the subclass package?

I have a base class with:

    exports_sources = '*.cpp', '*.h', '*.hpp', 'CMakeLists.txt*', '!test/*', '!test_package/*', '../../cmake/*'

The files in ../../cmake/* are copied to the base class package export_sources directory when the base class package is exported.

Exporting package recipe
project-base/0.2@idbdevbuild/stable exports_sources: Copied 4 '.cmake' files: copy_resource.cmake, coverage.cmake, project-base.cmake, CodeCoverage.cmake

The other items are sources of the subclass itself.

When the consuming package is exported, it copies the *.cpp, etc. files to its export_sources

Exporting package recipe
core-ull/1.80@idbdevbuild/testing exports_sources: Copied 13 '.cpp' files:
core-ull/1.80@idbdevbuild/testing exports_sources: Copied 24 '.h' files:
core-ull/1.80@idbdevbuild/testing exports_sources: Copied 1 '.txt' file: CMakeLists.txt

but it does not get the files from the base class package.

As a workaround is there a way for me to get the export_source directory name from the base class so I can get access to the files there?

[danimtb]

Sorry, we have this piece of the documentation pending conan-io/docs#916

If I recall it correctly, in case you want to use exports of the python_require and exports in the child conanfile you would have to state so in the attribute. Like this:

from conans import ConanFile, python_requires

base = python_requires("MyBase/0.1@user/channel")

class TestConan(base.Mybase):
    ...
    exports_sources = base.MyBase.exports_sources + "*.cpp"

Please give it a try and let us know if that works as expected.

[sourcedelica]

I see - thanks. Actually that won't help for us. Our subclass recipes don't have an export_sources attribute because they inherit it from the base class. The idea is to completely remove all redundant code. We don't want to have

class TestConan(base.Mybase):
    exports_sources = "*.cpp"

because every recipe will have "*.cpp" in their exports sources.

Our base class recipe looks like this:

def all_exports_sources():
    if EXPORTING_BASE_ENV in os.environ:
        exports = os.path.join(os.environ[EXPORTING_CMAKE_DIR], '*'),
    else:
        exports = '*.cpp', '*.h', '*.hpp', 'CMakeLists.txt*', '!test/*', '!test_package/*', 
    return exports

class BaseConan(ConanFile):
    ...
    exports_sources = all_exports_sources()

We have a script that exports/uploads the base class recipe. That script sets an environment variable so the recipe knows its being exported as a base class, not as a subclass. Because of this our subclass recipes can be super minimal, like:

from conans import python_requires
base = python_requires('my-base/0.2@idbdevbuild/stable')

class MyConan(base.BaseConan):
    name = 'common'
    description = 'My common library'

Since it doesn't look like this feature (#3538) applies to us, what's the best way to find the export_source directory of the base package? One option is to parse the output of

conan info --paths falcon-base/0.2@idbdevbuild/stable

though the output is for humans, and the export_source path is not in the output, but it's close enough :) Is there any way to get it directly in Python code without spawning a new process?

[sourcedelica]

Is the subclass supposed to have access to the actual files that the base class exported, or just the tuple of file patterns from the base class?

When our base class recipe is exported it copies the matching files from its exports_sources to the export_source directory in the cache. For example, the base class recipe copies a bunch of cmake modules to the base class export_source. Should my subclass then have access to those copied files? Or just the tuple of filename patterns?