Conan: [question] recommendation for export_sources during package development

[ericjameszimmerman]

To help us debug your issue please explain:

• [x] I've read the CONTRIBUTING guide.
• [x] I've specified the Conan version, operating system version and any tool that can be relevant.
• [x] I've explained the steps to reproduce the error or the motivation/use case of the question/suggestion.

Conan version = 1.11.1
OS = Win10

My end goal for a recipe is to use the source() method when the package is uploaded to Artifactory.

During package development, I have been struggling with how the proper way...

• I want package to be built with my local changes / not necessarily require that I check-in / change the source() to specify my branch
• I have been playing with self.develop to conditionally turn off source(), but I find that I have to use export_sources = "*" in this case...
• If I used export_sources = "*", the source code gets tar.gz into artifactory
• fyi: I tend to just use 'conan create' when working with my libraries, rather than 'conan install / conan build'

Do you have a recommendation workflow for 'package development'?

[Johnnyxy]

hi there,
I have two tips for you how you could implement your use case. The following is Python 3 (Path -> Py 3.6+) code.

SCM attribute

If you use the SCM attribute you can completely omit the source(self) function.

When you call conan create Conan automatically copies your local sources to its source folder.
It does detect the root of your repository and copies everything from there. This is nessecary to sync with the folder layout in the destination folder like it would be after a git clone.

If you install the recipe as dependency then Conan would automatically clone the given repository in cache and work from there like conan create would do.

Be aware that conan create has a bug when it comes to package / package-dependencies options: https://github.com/conan-io/conan/issues/4784
conan install is running fine. But when the bug is fixed you should use conan create.

Manually

If you do not wish to use the SCM feature then you could do it yourself by implementing a source(self) function like this:

...
scm_url = "YOUR_SCM_URL"
scm_revision = "YOUR_SCM_BRANCH"
def source(self):
    if (not self.in_local_cache):
        self.output.info(f'{self.name}: local source request')
        src_path = Path(__file__).parent.resolve() # current conanfile path
        dst_path = Path().resolve()
        self.output.info(f'{self.name}: retrieving source ({{source: filesystem}}, {{src: {src_path}}}, {{dst: {dst_path}}})')
        for x in Path(src_path).iterdir():
            if (x.is_file()): shutil.copy2(x, dst_path)
            elif (x.is_dir()): shutil.copytree(x, Path(dst_path, x.name))
            else: raise TypeError(f'copying node type not supported ({{node: {x}}})')
    else:
        self.output.info(f'{self.name}: retrieving source ({{source: git}}, {{src: {self.scm_url}}}, {{revision: {self.scm_revision}}}, {{dst: {Path().resolve()}}})')
        git = tools.Git()
        git.clone(self.scm_url, self.scm_revision)
...

conan install ...
conan build ...
conan package ...
conan export-pkg ...

The attribute in_local_cache gets set in the case your conanfile is not in the cache. Then you have to detect the correct path and what to copy by yourself (see src_path = ...).

[Johnnyxy]

I have a follow up question for the devs.

I have a recipe that does not have the SCM feature. Now I want to use only conan create but with the local sources and no export_sources attribute as this would export the source in any case.

To detect that the recipe has been called manually through conan create one can use the self.develop attribute. But .

If I have to implement a source(self) function that copies the source manually I would need the path where the original conanfile was exported from. Sadly I do not know how to retrieve this path. As the current path has been switched in the source(self) to the source folder in the conan repository and conan create exports the recipe and executes it from there (no local src path from conanfile "___file___").

In the end the information where the conanfile was exported from is lost and I cannot copy the source into the cache.

Question

Is there any way except using conan install / conan source / conan build / ... to get the sources from my local source path (origin of exported conanfile) with conan create?

Example

The following is an example of a source function I came up with. But when using conan create it always stops at NotImplementedError("copying non local source not implemented in developer mode").

class Recipe(ConanFile):
    name            = "test"
    version         = "0.1.0"
    settings        = "os", "arch", "build_type", "compiler", "arch_build"
    generators      = ["cmake_find_package", "virtualrunenv", "virtualenv"]
    options         = {"shared": [True, False]}
    default_options = {"shared": True}

    def source(self):
        if (self.develop):
            if (not self.in_local_cache):
                src = Path(__file__).parent
                dst = Path(self.source_folder)
                for node in src.iterdir():
                    if (node.is_file()):  shutil.copy2(node, dst)
                    elif (node.is_dir()): shutil.copytree(node, Path(dst, node.name))
                    else: raise TypeError("source node type not supported for copying")
            else: raise NotImplementedError("copying non local source not implemented in developer mode")
        else: raise NotImplementedError("copying only supported in developer mode")

    def build(self):
        cmake = self.__create_cmake()
        cmake.configure()
        cmake.build()
        cmake.install()

    def package_id(self):
        del self.info.settings.compiler
        del self.info.settings.arch_build

    def __create_cmake(self):
        if (self.settings.compiler == "Visual Studio"):
            toolset = None
            if (self.settings.arch_build == "x86_64"): 
                toolset = "host=x64"
            return CMake(self, toolset=toolset)
        else:
            return CMake(self)

[ericjameszimmerman]

Aaargh... I still have difficulty.
At this time, I am developing both library packages and applications simultaneously... eventually the packages will stabilize / be less frequently added to.

Case 1: application with conan dependencies, not working on libraries

• No problem: conan install, conan build

Case 2: application with conan dependencies, changes to libraries

• I change the application to depend on an 'experimental' package
• I use 'conan create' when I am ready to build / test the application
  
  
  
  • I have to commit / push the files to my remote repository due to the source() / scm functionality, which is annoying if it doesn't build successfully for whatever reason.
  
  
  • Alternatively, I can change the recipe to export_sources = * for this package, but have to remember to flip back and forth
  
  

Case 3: library changes

• Partially due to Git / scm interface, dependending if I use 'conan install / conan build' or 'conan create', I need to use of project subfolder or not.

Ideally, I'm looking for a solution so I don't have to keep changing the conanfile.py...
Any guidance would be appreciated!
Have you had similar workflows? ... what do you typically do?

[Johnnyxy]

As I understood you work on a lib that should be a dependency to an application.
You can work simultaneously on both that is no problem.

SCM feature

I think you misunderstood my points with the SCM feature. I will give you an example. For simplicity I will use Windows paths. Everything (recipe, commands) except the paths formats can be used with Linux/Apple too without any changes.

In this example both the "mylib" and "myapp" are in the same git repository. But it does not make any difference as you are going to see.
Suppose you have checked out your code into "c:\tmp\repo".
With the following repository folder structure:
root

• mylib
  
  
  
  • conanfile.py
  
  
  • CMakeLists.txt
  
  
• myapp
  
  
  
  • conanfile.py
  
  
  • CMakeLists.txt
  
  

class LibraryRecipe(ConanFile):
    name = "MyLib"
    version = "0.0.0"
    scm = {
        type = "git"
        url = "auto"
        revision = "auto"
    }

    def build(self):
        cmake = CMake(self)
        # As the CMakeLists.txt is in a subfolder one has to specify 
        # the path of it.
        cmake.configure(source_folder="mylib")
        cmake.build()
        cmake.install() 
        # no "def package(self)" function needed as CMake installs
        # into self.package_folder automatically

conan create  "C:\tmp\repo\mylib" "test/testing" --update --build=outdated --settings build_type="Debug"



md5-b614502715f958af9314697b2df94098



mylib/0.0.0@test/testing: Repo origin deduced by 'auto': http://GIT_URL/
mylib/0.0.0@test/testing: WARN: Repo status is not pristine: there might be modified files
mylib/0.0.0@test/testing: Revision deduced by 'auto': GIT_HASH
mylib/0.0.0@test/testing: A new conanfile.py version was exported
mylib/0.0.0@test/testing: Folder: C:\Users\TEST_USER\.conan\data\mylib\0.0.0\test\testing\export
mylib/0.0.0@test/testing: Package recipe modified in export, forcing source folder removal
mylib/0.0.0@test/testing: Use the --keep-source, -k option to skip it
Configuration:
[settings]
arch=x86_64
arch_build=x86_64
build_type=Debug
compiler=Visual Studio
compiler.runtime=MDd
compiler.version=15
os=Windows
os_build=Windows
[options]
[build_requires]
[env]

gtest/1.8.1@bincrafters/stable: WARN: Current package is newer than remote upstream one
gtest/1.8.1@bincrafters/stable: Package is up to date
icu/63.1@bincrafters/stable: Package is up to date
libiconv/1.15@bincrafters/stable: Package is up to date
zlib/1.2.11@conan/stable: Package is up to date
libxml2/2.9.9@bincrafters/stable: Package is up to date
Version ranges solved
    Version range '~=63.1' required by 'mylib/0.0.0@test/testing' resolved to 'icu/63.1@bincrafters/stable'
    Version range '~=2.9' required by 'mylib/0.0.0@test/testing' resolved to 'libxml2/2.9.9@bincrafters/stable'

Installing package: mylib/0.0.0@test/testing
Requirements
    mylib/0.0.0@test/testing from local cache - No remote
Packages
    mylib/0.0.0@test/testing:c8d17a319932c46c49c0763cb82fa22732a2a76f - Build
Build requirements
    gtest/1.8.1@bincrafters/stable from 'conan-center' - Cache
    icu/63.1@bincrafters/stable from 'bincrafters' - Cache
    libiconv/1.15@bincrafters/stable from 'conan-center' - Cache
    libxml2/2.9.9@bincrafters/stable from 'conan-center' - Cache
    zlib/1.2.11@conan/stable from 'conan-center' - Cache
Build requirements packages
    gtest/1.8.1@bincrafters/stable:b929ae4772cdf3ed57b41579b4cb08e4712a2350 - Cache
    icu/63.1@bincrafters/stable:8aa249060e72d47b73461fede150e53dd6b2d3e8 - Cache
    libiconv/1.15@bincrafters/stable:8cf01e2f50fcd6b63525e70584df0326550364e1 - Cache
    libxml2/2.9.9@bincrafters/stable:5d80b55957b5095e6fc61e36b741067ca2d822ec - Cache
    zlib/1.2.11@conan/stable:8cf01e2f50fcd6b63525e70584df0326550364e1 - Cache

gtest/1.8.1@bincrafters/stable: Already installed!
icu/63.1@bincrafters/stable: Already installed!
libiconv/1.15@bincrafters/stable: Already installed!
zlib/1.2.11@conan/stable: Already installed!
libxml2/2.9.9@bincrafters/stable: Applying build-requirement: zlib/1.2.11@conan/stable
libxml2/2.9.9@bincrafters/stable: Applying build-requirement: libiconv/1.15@bincrafters/stable
libxml2/2.9.9@bincrafters/stable: Already installed!
mylib/0.0.0@test/testing: Applying build-requirement: icu/63.1@bincrafters/stable
mylib/0.0.0@test/testing: Applying build-requirement: libxml2/2.9.9@bincrafters/stable
mylib/0.0.0@test/testing: Applying build-requirement: gtest/1.8.1@bincrafters/stable
mylib/0.0.0@test/testing: Applying build-requirement: zlib/1.2.11@conan/stable
mylib/0.0.0@test/testing: Applying build-requirement: libiconv/1.15@bincrafters/stable
mylib/0.0.0@test/testing: WARN: Detected 'scm' auto in conanfile, trying to remove source folder
mylib/0.0.0@test/testing: WARN: This can take a while for big packages
mylib/0.0.0@test/testing: Configuring sources in C:\Users\TEST_USER\.conan\data\mylib\0.0.0\test\testing\source
mylib/0.0.0@test/testing: Getting sources from folder: C:/tmp/repo
mylib/0.0.0@test/testing: Copying sources to build folder
mylib/0.0.0@test/testing: Building your package in C:\Users\TEST_USER\.conan\data\mylib\0.0.0\test\testing\build\c8d17a319932c46c49c0763cb82fa22732a2a76f
mylib/0.0.0@test/testing: Generator cmake_find_package created Findicu.cmake
mylib/0.0.0@test/testing: Generator cmake_find_package created Findlibxml2.cmake
mylib/0.0.0@test/testing: Generator cmake_find_package created Findgtest.cmake
mylib/0.0.0@test/testing: Generator cmake_find_package created Findzlib.cmake
mylib/0.0.0@test/testing: Generator cmake_find_package created Findlibiconv.cmake



md5-06d38830c09c26aa81644555e2e39782



conan install "mylib/0.0.0@test/testing" --build=outdated ...

In time you will master Conan and can explore other ways to achieve the same outcome (like editable packages or workspaces). But for now I advise you to stick with the normal "proven" way. As only this guarantees that everything will work as expected when you switch over to distribute your package to other Conan users. In the beginning it is crucial to get it working and move on from there.

[ericjameszimmerman]

Am I making this more complicated than necessary?
I am currently making changes to multiple packages at once and can't seem to get this right... seem to have to purge the conan cache to get it to pull in correctly.

I'd like to:

• make changes to one of my dependencies and install/build or conan create
• install/build this package, pulling in the updated headers / library

from conans import ConanFile, CMake, tools
import json


def get_version():
    try:
        with open('ProjectSettings.json') as json_file:  
            data = json.load(json_file)

        version = '{}.{}.{}'.format(data['version_major'], data['version_minor'], data['version_subminor'])
        return version

    except Exception as e:
        return None


def get_scm(is_develop, is_in_local_cache):
    if is_develop:
        if is_in_local_cache:
            scm = {
                "type": "git",
                "url": "auto",
                "revision": "auto",
                "subfolder": "lightfoot-iprc"
            }
        else:
            scm = {
                "type": "git",
                "url": "auto",
                "revision": "auto"
            }
    else:
        scm = {
            "type": "git",
            "subfolder": "lightfoot-iprc",
            "url": "ssh://<omitted>",
            "revision": "feature/prototype"
        }
    return scm


def get_project_folder(is_develop, is_in_local_cache):
    if is_develop:
        if is_in_local_cache:
            return 'lightfoot-iprc'
        else:
            return ''
    else:
        return 'lightfoot-iprc'


class LightfootiprcConan(ConanFile):
    name = "lightfoot-iprc"
    version = get_version()
    scm = get_scm(ConanFile.develop, ConanFile.in_local_cache)
    description = "Inter-processor communication classes"
    topics = ("IPRC")
    settings = "os", "compiler", "build_type", "arch"
    options = {"shared": [True, False],
               "build_tests": [True, False]}
    default_options = {"shared": False,
                       "build_tests": False}
    generators = "cmake_find_package"
    requires = "blackstone-platform/0.4.0@drs-nps/testing", "backbone-framework/0.4.0@drs-nps/experimental"

    def requirements(self):
        if self.options.build_tests:
            self.requires("anvil-test-framework/0.2.0@drs-nps/stable")

    def build(self):
        self.projectfolder = get_project_folder(self.develop, self.in_local_cache)

        cmake = CMake(self)
        cmake.definitions["BUILD_TESTS"] = self.options.build_tests
        cmake.configure(source_folder=self.projectfolder)
        cmake.build()

        if self.options.build_tests:
            cmake.test()

    def project_path(self, folder):
        if self.projectfolder:
            return "{}/{}".format(self.projectfolder, folder)
        else:
            return folder

    def package(self):
        self.projectfolder = get_project_folder(self.develop, self.in_local_cache)

        self.copy("*.h", dst="include", src=self.project_path("include"))
        self.copy("ProjectSettings.json", dst="info", src=self.project_path(""))

        excludePaths = "projects/*"
        self.copy("*lightfootiprc.lib", dst="lib", keep_path=False, excludes=excludePaths)
        self.copy("*.dll", dst="bin", keep_path=False, excludes=excludePaths)
        self.copy("*.so", dst="lib", keep_path=False, excludes=excludePaths)
        self.copy("*.dylib", dst="lib", keep_path=False, excludes=excludePaths)
        self.copy("*.a", dst="lib", keep_path=False, excludes=excludePaths)

    def package_info(self):
        self.cpp_info.libs = ["lightfootiprc"]

    def imports(self):
        self.copy("*.dll", dst="bin", src="bin") # From bin to bin
        self.copy("*.dylib*", dst="bin", src="lib") # From lib to bin
        self.copy("*.so*", dst="bin", src="lib") # From bin to bin

[jgsogo]

Yes, I think your recipe is much more complicated than it should be, only in rare cases you should be using the .develop and .in_local_cache... let's see if we can simplify your recipe. I've created a repository trying to reproduce your project: https://github.com/jgsogo/issue-4795 (check proposed conanfile.py in the repository)

Let's work on it, let's add relevant pieces to reproduce your needs. Local and cache workflows should work as expected:

Local workflow

$> git clone https://github.com/jgsogo/issue-4795.git
$> cd issue-4795
$> mkdir _build
$> cd _build
$> conan install ..
$> conan source ..
$> conan build ..
$> ./iprc

If you try to run this steps again, the conan source .. one will
fail because you cannot clone a repository over an existing one.

Cache build

$> git clone https://github.com/jgsogo/issue-4795.git
$> cd issue-4795
$> conan create . issue/testing

[ericjameszimmerman]

Thank-you for taking time on this..
first, could you clarify how the revision "auto" portion works..?

What happens if --build missing is set and this package needs to be re-build due to dependency resolution differences or specific options not in the package within Artifactory? Which branch / tag/ commit does it pull source from?

[jgsogo]

The revision = "auto" tells Conan to take the commit sha from the git repository and use that exact commit to retrieve the sources (no need for branch/tag). If you take a look at the output of the create command it says:

Exporting package recipe
lightfoot-iprc/1.2.3@issue/testing exports: Copied 1 '.json' file: ProjectSettings.json
lightfoot-iprc/1.2.3@issue/testing: Revision deduced by 'auto': 11c702ead53b36b18f125c2f4fd07e1a1330cd2a
...

which is the last commit:

$> git log -1
commit 11c702ead53b36b18f125c2f4fd07e1a1330cd2a

And this is stored in the exported recipe:

⇒  conan get lightfoot-iprc/1.2.3@issue/testing conanfile.py
....
class MyLibrary(ConanFile):
    ...

    scm = {"revision": "11c702ead53b36b18f125c2f4fd07e1a1330cd2a",
           "type": "git",
           "url": "https://github.com/jgsogo/issue-4795.git"}
...

So, when the creation of the package is performed in the cache, the recipe is pointing to an exact commit, to a snapshot of your repository, it will build your package from that exact-sources* and the binary is totally reproducible.

*there is an optimization for the conan create command and Conan will copy the sources instead of cloning the repo, so you can test with a dirty repo (with modified files) and test the recipe without committing each change (Conan will print a WARNING saying that the _"repository is not pristine"_). Again, in the output:

...
lightfoot-iprc/1.2.3@issue/testing: Getting sources from folder: /Users/jgsogo/dev/conan/issues/4795
..

This optimization is not used when installing a package or when its build is being triggered from another package (or using an install command), in this case, Conan will clone the repo:

$> conan install lightfoot-iprc/1.2.3@issue/testing --build
...
lightfoot-iprc/1.2.3@issue/testing: Getting sources from url: 'https://github.com/jgsogo/issue-4795.git'
...

[ericjameszimmerman]

thank-you.. so far, so good.
Let me revise my other libraries and see if I run into snags.

[ericjameszimmerman]

This has worked well.