Conan: [question] ERROR: Attempt to modify locked lib when trying to create recipe revision lockfile

Created on 14 Dec 2020 · 19Comments · Source: conan-io/conan

We are following through the examples in https://docs.conan.io/en/latest/versioning/lockfiles/ci.html#package-pipeline and we get to a point where we have already run the "package" lockfile creation steps:

$ cd libb

# Derive one lockfile per profile/configuration
$ conan lock create conanfile.py --name=libb --version=0.2
  --user=user --channel=testing --lockfile=../locks/libb_deps_base.lock
  --lockfile-out=../locks/libb_deps_debug.lock -s build_type=Debug
$ conan lock create conanfile.py --name=libb --version=0.2
  --user=user --channel=testing --lockfile=../locks/libb_deps_base.lock
  --lockfile-out=../locks/libb_deps_release.lock

# Create the package binaries, one with each lockfile
$ conan create . libb/0.2@user/testing --lockfile=../locks/libb_deps_release.lock
$ conan create . libb/0.2@user/testing --lockfile=../locks/libb_deps_debug.lock

and now we were looking at this optional step:

When using revisions (not this example), it is important to capture the recipe revision, and lock it too. We can capture the recipe revision doing an export, creating a new libb_base.lock lockfile:

So we ran in our CI after the steps above:

 conan export . libb/0.2@user/testing --lockfile=../locks/libb_deps_base.lock
  --lockfile-out=../locks/libb_base.lock

But we get this output (note: we used slightly different lockfile output names to be clearer about package versus recipe lockfiles):

2020-12-14 05:31:31 |  [Conan_pipeline_master] $ cmd.exe /C "conan export . libb/0.2@conan/stable --lockfile=package_deps_base.lock --lockfile-out=recipe_base.lock && exit %%ERRORLEVEL%%"
2020-12-14 05:31:33 |  [HOOK - attribute_checker.py] pre_export(): WARN: Conanfile doesn't have 'url'. It is recommended to add it as attribute
2020-12-14 05:31:33 |  [HOOK - attribute_checker.py] pre_export(): WARN: Conanfile doesn't have 'license'. It is recommended to add it as attribute
2020-12-14 05:31:33 |  [HOOK - attribute_checker.py] pre_export(): WARN: Conanfile doesn't have 'description'. It is recommended to add it as attribute
2020-12-14 05:31:33 |  ERROR: Attempt to modify locked libb/0.2 to libb/0.2@conan/stable#e7e83409e3735f8c2a464f151e66f011

Question: Can I ask what we've done wrong?

[x] I've read the CONTRIBUTING guide.

question

Source

michaelmaguire

All 19 comments

Hi @michaelmaguire

I think the docs are not clear enough, in these cases, please refer to the code in the examples repo, which is complete and functional. What is being ran there is:

run("conan create liba liba/0.2@user/testing")
    with chdir("libb"):
        # This will be useful to capture the revision
        run("conan export . libb/0.2@user/testing --lockfile=../locks/libb_deps_base.lock "
            "--lockfile-out=../locks/libb_base.lock")
        print(load("../locks/libb_base.lock"))
        # Capture the configuration lockfiles, one per configuration
        run("conan lock create conanfile.py --name=libb --version=0.2 "
            "--user=user --channel=testing --lockfile=../locks/libb_base.lock --lockfile-out=../locks/libb_deps_debug.lock -s build_type=Debug")
        run("conan lock create conanfile.py --name=libb --version=0.2 "
            "--user=user --channel=testing --lockfile=../locks/libb_base.lock --lockfile-out=../locks/libb_deps_release.lock")
        # Now build libb
        run("conan create . libb/0.2@user/testing --lockfile=../locks/libb_deps_release.lock")
        run("conan create . libb/0.2@user/testing --lockfile=../locks/libb_deps_debug.lock")

You can see the export needs to be done before the create. If you execute the create, then the package binary package_id and package revision will be computed and fully locked. Once they are locked, it is impossible to export it again.

If it makes sense and it works, please let me know and I will move this to the docs repo to be fixed in the documentation.

memsharded on 14 Dec 2020

Thank you.

I see the example https://github.com/conan-io/examples/blob/master/features/lockfiles/ci/build.py#L80 .

In fact, we were already doing base lock file creation before we did the per-configuration lockfile creation:

                        conanClient.run(buildInfo: buildInfo, command: "lock create ./conanfile.py  ${conan_lock_create_package_base_additional_parameters}".toString())

                        buildProfiles.each {
                            echo "buildProfile[${it}]"

                            try {
                                conanClient.run(buildInfo: buildInfo, command: "lock create ./conanfile.py --user=${conan_package_user} --channel=${conan_package_channel} --lockfile-out=package_deps_${it}.lock --profile ${it}   ${conan_lock_create_package_per_profile_additional_parameters}".toString())

                                conanClient.run(buildInfo: buildInfo, command: "create . ${conan_package_version}@${conan_package_user}/${conan_package_channel} --json conanCreateOutput_${it}.json ${conan_additional_create_parameters} --lockfile=package_deps_${it}.lock".toString())
                } catch( err ) {
                                if( promote_intermediates_on_failure ) {
                                    // Catch err here so that even in case of some failures the build continues 
                                    // and any profiles or intermediate packages we can manage to build will get uploaded 
                                    // to Artifactory so that that build time is not wasted.
                                    // This is very handy, e.g. when building for a new compiler for the first time.
                                    // We will still consider it a failed build, however.
                                    currentBuild.result = 'FAILURE'
                                } else {
                                    // Fail out and stop this build now.
                                    // This is safest, in that if a particular modification to a build config
                                    // (e.g. a new BDE version) breaks an overall build, should you really
                                    // consider intermediate packages that managed to build "successfully" 
                                    // built?  In the dpkg buildall world you would reject the modification outright.
                                    throw err
                                }
                            }                            
                        }

                        // Commented out for now as we saw issues: 
                        // https://github.com/conan-io/conan/issues/8205 [question] ERROR: Attempt to modify locked lib when trying to create recipe revision lockfile
                        //conanClient.run(buildInfo: buildInfo, command: "export .  ${conan_lock_create_recipe_base_additional_parameters}".toString())

but then in the docs just before https://docs.conan.io/en/latest/versioning/lockfiles/ci.html#products-pipeline we saw the part I mentioned above, namely:

When using revisions (not this example), it is important to capture the recipe revision, and lock it too. We can capture the recipe revision doing an export, creating a new libb_base.lock lockfile:

$ conan export . libb/0.2@user/testing --lockfile=../locks/libb_deps_base.lock
  --lockfile-out=../locks/libb_base.lock

and were trying to understand how that fit in relative to the lock create ./conanfile.py --base call we were told to already do at the start:

As we want to make sure that all different configurations are built with the same versions of the dependencies, the first thing is to capture a “base” lockfile of the dependencies of libb:

$ cd libb
$ conan lock create conanfile.py --name=libb --version=0.2 --user=user --channel=testing
  --lockfile-out=../locks/libb_deps_base.lock --base

michaelmaguire on 15 Dec 2020

Dear @memsharded, can you help us update what the difference between the initial conan lock create --base mentioned in the current https://docs.conan.io/en/latest/versioning/lockfiles/ci.html#products-pipeline versus the conan export . <package> --lockfile?

From the https://github.com/conan-io/examples/blob/master/features/lockfiles/ci/build.py example, I don't understand yet why we first need to perform a --base package lock action in https://github.com/conan-io/examples/blob/master/features/lockfiles/ci/build.py#L73 then a recipe lock action with the export in https://github.com/conan-io/examples/blob/master/features/lockfiles/ci/build.py#L80. Then in https://github.com/conan-io/examples/blob/master/features/lockfiles/ci/build.py#L84 we do a per-profile package lockfile creation.

Why is it package -> recipe -> per-profile package?

Shouldn't it be recipe -> package -> per-profile package?

michaelmaguire on 4 Jan 2021

@memsharded, we tried your suggestion above in: https://github.com/conan-io/conan/issues/8205#issuecomment-744752830 and we still got an error:

2021-01-04 05:29:39 |  [Conan_pipeline_master] $ cmd.exe /C "conan lock create ./conanfile.py --lockfile-out=package_deps_base.lock --base && exit %%ERRORLEVEL%%"
2021-01-04 05:29:41 |  Requirements
2021-01-04 05:29:41 |      liba/0.1@conan/stable from '6b0a4d91-9a84-472d-b42e-63f0b0037cd5' - Cache
2021-01-04 05:29:41 |  Packages
2021-01-04 05:29:41 |      liba/0.1@conan/stable:c32596dcd26b8c708dc3d19cb73738d2b48f12a8 - Missing
2021-01-04 05:29:41 |  Build requirements
2021-01-04 05:29:41 |      cmake/3.18.2@bloomberg/stable from '6b0a4d91-9a84-472d-b42e-63f0b0037cd5' - Cache
2021-01-04 05:29:41 |      strawberryperl/5.30.0.1@bloomberg/stable from '6b0a4d91-9a84-472d-b42e-63f0b0037cd5' - Cache
2021-01-04 05:29:41 |  Build requirements packages
2021-01-04 05:29:41 |      cmake/3.18.2@bloomberg/stable:ca33edce272a279b24f87dc0d4cf5bbdcffbc187 - Download
2021-01-04 05:29:41 |      strawberryperl/5.30.0.1@bloomberg/stable:456f15897172eef340fcbac8a70811f2beb26a93 - Download
2021-01-04 05:29:41 |  
2021-01-04 05:29:41 |  Generated lockfile: D:\J\workspace\Conan_pipeline_master\package_deps_base.lock
2021-01-04 05:29:41 |  [Conan_pipeline_master] $ cmd.exe /C "conan_build_info D:/J/workspace/Conan_pipeline_master\uh\conan_log.log --output D:\J\workspace\Conan_pipeline_master@tmp\artifactory\conan4304545018508377101build-info && exit %%ERRORLEVEL%%"
2021-01-04 05:29:42 |  [Pipeline] runConanCommand
2021-01-04 05:29:42 |  [Conan_pipeline_master] $ cmd.exe /C "conan export . libb/0.2@conan/stable --lockfile=package_deps_base.lock --lockfile-out=recipe_base.lock && exit %%ERRORLEVEL%%"
2021-01-04 05:29:43 |  [HOOK - attribute_checker.py] pre_export(): WARN: Conanfile doesn't have 'url'. It is recommended to add it as attribute
2021-01-04 05:29:43 |  [HOOK - attribute_checker.py] pre_export(): WARN: Conanfile doesn't have 'license'. It is recommended to add it as attribute
2021-01-04 05:29:43 |  [HOOK - attribute_checker.py] pre_export(): WARN: Conanfile doesn't have 'description'. It is recommended to add it as attribute
2021-01-04 05:29:43 |  ERROR: Attempt to modify locked libb/0.2 to libb/0.2@conan/stable#e7e83409e3735f8c2a464f151e66f011
2021-01-04 05:29:43 |  Using lockfile: 'D:\J\workspace\Conan_pipeline_master\package_deps_base.lock'
2021-01-04 05:29:43 |  Exporting package recipe
2021-01-04 05:29:43 |  libb/0.2@conan/stable: A new conanfile.py version was exported
2021-01-04 05:29:43 |  libb/0.2@conan/stable: Folder: D:\J\workspace\Conan_pipeline_master\uh\.conan\data\libb\0.2\conan\stable\export
2021-01-04 05:29:43 |  libb/0.2@conan/stable: Using the exported files summary hash as the recipe revision: e7e83409e3735f8c2a464f151e66f011 
2021-01-04 05:29:43 |  libb/0.2@conan/stable: Exported revision: e7e83409e3735f8c2a464f151e66f011

Here's the current Jenkinsfile recipe snippet:


                        conanClient.run(buildInfo: buildInfo, command: "lock create ./conanfile.py  ${conan_lock_create_package_base_additional_parameters}".toString())
                        // TODO: Be smarter or more constrained about filename?
                        //conan_lock_content = readFile('package_deps_base.lock')
                        //echo "conan_lock_content[${conan_lock_content}]"


                        conanClient.run(buildInfo: buildInfo, command: "export .  ${conan_lock_create_recipe_base_additional_parameters}".toString())


                        buildProfiles.each {
                            echo "buildProfile[${it}]"
                            env.MY_FAILURE_STAGE = STAGE_NAME + "__conan_create_loop_profile_${it}"

                            try {
                                conanClient.run(buildInfo: buildInfo, command: "lock create ./conanfile.py --user=${conan_package_user} --channel=${conan_package_channel} --lockfile-out=package_deps_${it}.lock --profile ${it}   ${conan_lock_create_package_per_profile_additional_parameters}".toString())

                                conanClient.run(buildInfo: buildInfo, command: "create . ${conan_package_version}@${conan_package_user}/${conan_package_channel} --json conanCreateOutput_${it}.json ${conan_additional_create_parameters} --lockfile=package_deps_${it}.lock".toString())
                } catch( err ) {

michaelmaguire on 4 Jan 2021

Hi @michaelmaguire

I think the error comes from the difference in user/channel when you capture the lockfile and when you try to use it. Note that a different user/channel could completely change the dependency graph, so the user/channel used at create should match the one at later step.

So use something like:

$ conan lock create ./conanfile.py --user=conan --channel=stable ...

Then inspect the lockfile json file. You will see that conan/stable is part of the reference.
Then when you run

$ conan export . libb/0.2@conan/stable --lockfile=...
# or
$ conan create . libb/0.2@conan/stable --lockfile=...

Please let me know if this makes sense and it works as expected. Thanks!

memsharded on 13 Jan 2021

❤1

This was the issue, thanks!

Now, with:

[...]
                        // Package names don't always match github repo names.
                        // Programmatically obtain package name and version, by using conan inspect
                        // on the conanfile.py within the checked out repo.
                        def buildInfo = conanClient.run(command: "inspect . -j conanInspectOutput.json")
                        def conanInspectOutputJson = readJSON file: "conanInspectOutput.json"
                        def packageName = package_name_from_inspect(conanInspectOutputJson)
                        def packageVersion = package_version_from_inspect(conanInspectOutputJson)
                        echo "package packageName[${packageName}] packageVersion[${packageVersion}]"

                        def buildProfiles = conan_build_profiles.tokenize(",")

                        // conan info also bases decisions on supplied profile.
                        // Don't default here, use the first of the supplied profiles.
                        // Also, create the first buildInfo object to be passed into
                        // all subsequent calls, and eventually published up to 
                        // Artifactory's https://artprod.dev.bloomberg.com/artifactory/webapp/#/builds/
                        conanClient.run(buildInfo: buildInfo, command: "info . --graph=dependency_graph.html --profile ${buildProfiles.first()}".toString())

                        conanClient.run(buildInfo: buildInfo, command: "lock create ./conanfile.py  ${conan_lock_create_package_base_additional_parameters} --user=${conan_package_user} --channel=${conan_package_channel}".toString())
                        // TODO: Be smarter or more constrained about filename?
                        //conan_lock_content = readFile('package_deps_base.lock')
                        //echo "conan_lock_content[${conan_lock_content}]"

                        conanClient.run(buildInfo: buildInfo, command: "export . ${packageName}/${packageVersion}@${conan_package_user}/${conan_package_channel}  ${conan_lock_create_recipe_base_additional_parameters}".toString())

                        buildProfiles.each {
                            echo "buildProfile[${it}]"
                            env.MY_FAILURE_STAGE = STAGE_NAME + "__conan_create_loop_profile_${it}"

                            try {
                                conanClient.run(buildInfo: buildInfo, command: "lock create ./conanfile.py --user=${conan_package_user} --channel=${conan_package_channel} --lockfile-out=package_deps_${it}.lock --profile ${it}   ${conan_lock_create_package_per_profile_additional_parameters}".toString())

                                conanClient.run(buildInfo: buildInfo, command: "create . ${conan_package_version}@${conan_package_user}/${conan_package_channel} --json conanCreateOutput_${it}.json ${conan_additional_create_parameters} --lockfile=package_deps_${it}.lock".toString())
[...]

We see output like:

[...]
2021-01-13 15:12:03 |  [Conan_pipeline_master] $ cmd.exe /C "conan lock create ./conanfile.py --lockfile-out=package_deps_base.lock --base --user=conan --channel=stable && exit %%ERRORLEVEL%%"
2021-01-13 15:12:05 |  Requirements
2021-01-13 15:12:05 |      liba/0.1@conan/stable from '6f8a5f6a-cfb5-40ee-bfa3-f6de382049e4' - Cache
2021-01-13 15:12:05 |  Packages
2021-01-13 15:12:05 |      liba/0.1@conan/stable:c32596dcd26b8c708dc3d19cb73738d2b48f12a8 - Missing
2021-01-13 15:12:05 |  Build requirements
2021-01-13 15:12:05 |      cmake/3.18.2@bloomberg/stable from '6f8a5f6a-cfb5-40ee-bfa3-f6de382049e4' - Cache
2021-01-13 15:12:05 |      strawberryperl/5.30.0.1@bloomberg/stable from '6f8a5f6a-cfb5-40ee-bfa3-f6de382049e4' - Cache
2021-01-13 15:12:05 |  Build requirements packages
2021-01-13 15:12:05 |      cmake/3.18.2@bloomberg/stable:ca33edce272a279b24f87dc0d4cf5bbdcffbc187 - Download
2021-01-13 15:12:05 |      strawberryperl/5.30.0.1@bloomberg/stable:456f15897172eef340fcbac8a70811f2beb26a93 - Download
2021-01-13 15:12:05 |  
2021-01-13 15:12:05 |  Generated lockfile: D:\J\workspace\Conan_pipeline_master\package_deps_base.lock
2021-01-13 15:12:05 |  [Conan_pipeline_master] $ cmd.exe /C "conan_build_info D:/J/workspace/Conan_pipeline_master\uh\conan_log.log --output D:\J\workspace\Conan_pipeline_master@tmp\artifactory\conan284047456412700489build-info && exit %%ERRORLEVEL%%"
2021-01-13 15:12:06 |  [Pipeline] runConanCommand
2021-01-13 15:12:06 |  [Conan_pipeline_master] $ cmd.exe /C "conan export . libb/0.2@conan/stable --lockfile=package_deps_base.lock --lockfile-out=recipe_base.lock && exit %%ERRORLEVEL%%"
2021-01-13 15:12:07 |  [HOOK - attribute_checker.py] pre_export(): WARN: Conanfile doesn't have 'url'. It is recommended to add it as attribute
2021-01-13 15:12:07 |  [HOOK - attribute_checker.py] pre_export(): WARN: Conanfile doesn't have 'license'. It is recommended to add it as attribute
2021-01-13 15:12:07 |  [HOOK - attribute_checker.py] pre_export(): WARN: Conanfile doesn't have 'description'. It is recommended to add it as attribute
2021-01-13 15:12:07 |  Using lockfile: 'D:\J\workspace\Conan_pipeline_master\package_deps_base.lock'
2021-01-13 15:12:07 |  Exporting package recipe
2021-01-13 15:12:07 |  libb/0.2@conan/stable: A new conanfile.py version was exported
2021-01-13 15:12:07 |  libb/0.2@conan/stable: Folder: D:\J\workspace\Conan_pipeline_master\uh\.conan\data\libb\0.2\conan\stable\export
2021-01-13 15:12:07 |  libb/0.2@conan/stable: Using the exported files summary hash as the recipe revision: e7e83409e3735f8c2a464f151e66f011 
2021-01-13 15:12:07 |  libb/0.2@conan/stable: Exported revision: e7e83409e3735f8c2a464f151e66f011
2021-01-13 15:12:07 |  [Conan_pipeline_master] $ cmd.exe /C "conan_build_info D:/J/workspace/Conan_pipeline_master\uh\conan_log.log --output D:\J\workspace\Conan_pipeline_master@tmp\artifactory\conan3303733873776135770build-info && exit %%ERRORLEVEL%%"
2021-01-13 15:12:08 |  [Pipeline] echo
2021-01-13 15:12:08 |  buildProfile[vs2019_16_v142_Debug]
2021-01-13 15:12:09 |  [Pipeline] runConanCommand
2021-01-13 15:12:09 |  [Conan_pipeline_master] $ cmd.exe /C "conan lock create ./conanfile.py --user=conan --channel=stable --lockfile-out=package_deps_vs2019_16_v142_Debug.lock --profile vs2019_16_v142_Debug --lockfile=package_deps_base.lock && exit %%ERRORLEVEL%%"
2021-01-13 15:12:10 |  Requirements
2021-01-13 15:12:10 |      liba/0.1@conan/stable from '6f8a5f6a-cfb5-40ee-bfa3-f6de382049e4' - Cache
2021-01-13 15:12:10 |  Packages
2021-01-13 15:12:10 |      liba/0.1@conan/stable:d057732059ea44a47760900cb5e4855d2bea8714 - Download
2021-01-13 15:12:10 |  Build requirements
2021-01-13 15:12:10 |      cmake/3.18.2@bloomberg/stable from '6f8a5f6a-cfb5-40ee-bfa3-f6de382049e4' - Cache
2021-01-13 15:12:10 |      strawberryperl/5.30.0.1@bloomberg/stable from '6f8a5f6a-cfb5-40ee-bfa3-f6de382049e4' - Cache
2021-01-13 15:12:10 |  Build requirements packages
2021-01-13 15:12:10 |      cmake/3.18.2@bloomberg/stable:ca33edce272a279b24f87dc0d4cf5bbdcffbc187 - Download
2021-01-13 15:12:10 |      strawberryperl/5.30.0.1@bloomberg/stable:456f15897172eef340fcbac8a70811f2beb26a93 - Download
2021-01-13 15:12:10 |  
2021-01-13 15:12:10 |  Generated lockfile: D:\J\workspace\Conan_pipeline_master\package_deps_vs2019_16_v142_Debug.lock
2021-01-13 15:12:10 |  [Conan_pipeline_master] $ cmd.exe /C "conan_build_info D:/J/workspace/Conan_pipeline_master\uh\conan_log.log --output D:\J\workspace\Conan_pipeline_master@tmp\artifactory\conan7001147612714941425build-info && exit %%ERRORLEVEL%%"
2021-01-13 15:12:11 |  [Pipeline] runConanCommand
2021-01-13 15:12:11 |  [Conan_pipeline_master] $ cmd.exe /C "conan create . @conan/stable --json conanCreateOutput_vs2019_16_v142_Debug.json --lockfile=package_deps_vs2019_16_v142_Debug.lock && exit %%ERRORLEVEL%%"
2021-01-13 15:12:13 |  Using lockfile: 'D:\J\workspace\Conan_pipeline_master\package_deps_vs2019_16_v142_Debug.lock'
2021-01-13 15:12:13 |  Exporting package recipe
[...]

michaelmaguire on 13 Jan 2021

To close out this ticket, can I just ask for a little more clarification about my question above in https://github.com/conan-io/conan/issues/8205#issuecomment-753892176 ?

Dear @memsharded, can you help us update what the difference between the initial conan lock create --base mentioned in the current https://docs.conan.io/en/latest/versioning/lockfiles/ci.html#products-pipeline versus the conan export . <package> --lockfile?

From the https://github.com/conan-io/examples/blob/master/features/lockfiles/ci/build.py example, I don't understand yet why we first need to perform a --base package lock action in https://github.com/conan-io/examples/blob/master/features/lockfiles/ci/build.py#L73 then a recipe lock action with the export in https://github.com/conan-io/examples/blob/master/features/lockfiles/ci/build.py#L80. Then in https://github.com/conan-io/examples/blob/master/features/lockfiles/ci/build.py#L84 we do a per-profile package lockfile creation.

Why is it package -> recipe -> per-profile package?

Shouldn't it be recipe -> package -> per-profile package?

michaelmaguire on 13 Jan 2021

Hi @michaelmaguire

Let me try to clarify:

https://github.com/conan-io/examples/blob/master/features/lockfiles/ci/build.py#L73 is doing:
```
conan lock create conanfile.py --name=libb --version=0.2 "
        "--user=user --channel=testing --lockfile-out=../locks/libb_deps_base.lock --base
```
That is capturing a "recipes information only" lockfile of the dependencies of libb (in this case liba/0.1)
Next step is doing an export of the changes done to lib, under a new 0.2 version:
```
conan export . libb/0.2@user/testing --lockfile=../locks/libb_deps_base.lock "
        "--lockfile-out=../locks/libb_base.lock
```
The export command does not compute the graph, the dependencies or the packages. The output libb_base.lock is still a recipes-only lockfile, with the main change compared to libb_deps_base.lock of having the full reference to libb/0.2@user/testing, including the recipe revision, in the lockfile

Next step is to create full configuration lockfiles with:

conan lock create conanfile.py --name=libb --version=0.2 "
        "--user=user --channel=testing --lockfile=../locks/libb_base.lock --lockfile-out=../locks/libb_deps_debug.lock -s build_type=Debug

This will be creating the complete lockfile, including package-ids, package revisions, locking the binaries that will be used for liba/0.1

So the presented flow is "recipe (local path)" -> "recipe (exported lib/0.2)" -> "package (one per config)"

The example is illustrative of the things that can be done and how lockfiles work. You might skip the first step, export the libb/0.2 changes, then capture the initial "base" lockfile just before capturing the 2 configuration lockfiles from the base one. Just make sure that the lockfile is captured before the unwanted liba/0.2 changes.

Does it make sense now? Thanks!

memsharded on 13 Jan 2021

❤1

Thank you.

The export command does not compute the graph, the dependencies or the packages. The output libb_base.lock is still a recipes-only lockfile, with the main change compared to libb_deps_base.lock of having the full reference to libb/0.2@user/testing, including the recipe revision, in the lockfile

I guess I'm still a little foggy on the role of the conan export'ed libb_base.lock here. I thought:

conan lock create conanfile.py --name=libb --version=0.2 "
          "--user=user --channel=testing --lockfile-out=../locks/libb_deps_base.lock --base

was already incorporating the diffs needed to build libb/0.2.

michaelmaguire on 13 Jan 2021

Yes, that lockfile can be enough to derive full configuration lockfiles, locking the dependencies of libb (locking liba/0.1), and be used to create the different binaries for libb/0.2.

The exported libb_base.lock goes one step further, adding to the lock of the dependencies (liba/0.1), the locked recipe revision of libb/0.2 including that modification, and then an extra level of protection. The full configuration lockfiles derived from it, will all contain exactly the same locked recipe revision of libb/0.2, that belong to the same modification. As it is very likely that different configurations goes to different agents to be built, passing that lockfile to those agents will guarantee that there will be no mistakes, those agents need to build exactly that recipe revision, and nothing different. If they do any mistake checking out something differently, or doing some local modifications to sources for a platform, that would try to generate a different recipe revision and the lockfile will block it, raising an error.

memsharded on 14 Jan 2021

Thanks. Sorry for all the questions. Why is that 'enhanced' lockfile with the extra level of protection an export command, rather than a different kind of conan lock create --extra-strong? What does exporting do here that's important?

michaelmaguire on 14 Jan 2021

No prob @michaelmaguire!

The thing is that if you do the conan lock create conanfile.py, that is a local conanfile, not a recipe in the cache. As such, it does not define a recipe revision, it is a pure consumer of dependencies. The process of exporting the recipe into the Conan cache is when the recipe revision is defined, and at that moment that revision can be updated in the lockfile.

It would be equivalent to:

$ conan export . libb/0.2@user/channel
$ conan lock create --reference=libb/0.2@user/channel --base

Assuming that nothing else happens between the first line conan export and the second line conan lock create. The second one can only work over something that is already exported in the cache. So it is not a matter of "strong", it is mostly about the origin. When the origin, or the root of the graph and lockfile is the local file (a path as an argument to conan lock create), it does not contain revision at all, but when the argument to conan lock create --reference=xxxx, the root of the graph and lockfile is the recipe in the cache, that already defines a recipe revision.

memsharded on 14 Jan 2021

Gah.

We tried this out, we tested, and then we landed the change and started using this in our "real" builds.

Now we're getting project 'pract' failures with error:

bzip2/1.0.6@PORT/stable: WARN: config() has been deprecated. Use config_options() and configure()
ERROR: Build-require 'gtest' cannot be found in lockfile
[adem] Loaded debian\changelog to parse version[0.3.0] commitAtVersion[60401e4]

or (from a different 'erep' build):

bzip2/1.0.6@PORT/stable: WARN: config() has been deprecated. Use config_options() and configure()
ERROR: Build-require 'cmake' cannot be found in lockfile
Loaded debian\changelog to parse version[1.9.1.0] commitAtVersion[11936c6]

I don't know if it's relevant, but cmake is a build requirement via our shared profiles:

[build_requires]
cmake/3.18.2@bloomberg/stable
strawberryperl/5.30.0.1@bloomberg/stable

as seen in the log:

Build requirements
    cmake/3.18.2@bloomberg/stable from '6f9d1314-8180-4725-9c2b-bde9e13cce3a' - Cache
    strawberryperl/5.30.0.1@bloomberg/stable from '6f9d1314-8180-4725-9c2b-bde9e13cce3a' - Cache

while gtest was neither a shared profile build requirement nor a direct build-require in the actual 'erep' package we were building (although might be in some dependent package):

    build_requires = (
            )
    requires =  (
            ("gtest/1.8.0@PORT/stable"),
            ("bde-monolith/3.55.0.0@PORT/stable"),
            ("pal/[>=2.17.0]@PORT/stable"),
            ("fmt/0.0.1e@PORT/stable"),
            ("sqlite3/3.29.0@bloomberg/stable"),
            )

michaelmaguire on 14 Jan 2021

We have now reproduced this problem in our 'sandbox' environment with a build DAG consisting only of liba, libb, libc, libd and app1 and app2 to match the test libs in https://docs.conan.io/en/latest/versioning/lockfiles/ci.html

It would appear the error clearly has to do with an impedance mismatch between shared profiles [build_requires] and lockfiles:

2021-02-05 15:09:27 |  [build_requires]
2021-02-05 15:09:27 |  *: cmake/3.19.3@bloomberg/stable, strawberryperl/5.30.0.1@bloomberg/stable, cmake/3.19.3@bloomberg/stable, strawberryperl/5.30.0.1@bloomberg/stable, cmake/3.19.3@bloomberg/stable, strawberryperl/5.30.0.1@bloomberg/stable
2021-02-05 15:09:27 |  [env]

2021-02-05 15:12:07 |  [Conan_pipeline_master] $ cmd.exe /C "conan install libd/0.1@conan/stable#7400ebfada85b91f49ad7f4bbb5c0244 --build=libd/0.1@conan/stable#7400ebfada85b91f49ad7f4bbb5c0244 --lockfile=distribution_per_profile_deps_vs2019_16_v142_Debug.lock --lockfile-out=distribution_per_profile_deps_vs2019_16_v142_Debug_updated.lock && exit %%ERRORLEVEL%%"
2021-02-05 15:12:09 |  ERROR: Build-require 'cmake' cannot be found in lockfile
2021-02-05 15:12:08 |  Using lockfile: 'D:\J\workspace\Conan_pipeline_master\distribution_per_profile_deps_vs2019_16_v142_Debug.lock'

Note: no bzip2 anywhere to be seen in this example.

Can we confirm -- are lockfiles compatible with shared profile config build_requires?

michaelmaguire on 5 Feb 2021

Hi @michaelmaguire

I am having a look at this. I am a bit confused about the line:

*: cmake/3.19.3@bloomberg/stable, strawberryperl/5.30.0.1@bloomberg/stable, cmake/3.19.3@bloomberg/stable, strawberryperl/5.30.0.1@bloomberg/stable, cmake/3.19.3@bloomberg/stable, strawberryperl/5.30.0.1@bloomberg/stable

Is this the line printed by Conan in the input profile? How is it possible that it is repeating cmake and strawberryperl 3 times?
In the example https://docs.conan.io/en/latest/versioning/lockfiles/ci.html, there are no build_requires. You said you reproduced with liba, libb, libc, libd and app1 and app2 only, but then I guess there are other packages, the ones that will be used as build_requires.

Lockfiles should work with build_requires: https://github.com/conan-io/conan/blob/develop/conans/test/integration/graph_lock/graph_lock_build_requires_test.py, and there are also tests covering a CI example with a build_requires defined in a profile, check: https://github.com/conan-io/conan/blob/2ccc79c48ce42972a19c7d3587e4ab8b585f220a/conans/test/integration/graph_lock/graph_lock_ci_test.py#L618

So it should be some other detail that we are skipping or missing. Is there any chance that you could share the reduced example with liba, libb, libc, libd and app1 and app2 only?

memsharded on 7 Feb 2021

About config include hieararchies

I am having a look at this. I am a bit confused about the line:
*: cmake/3.19.3@bloomberg/stable, strawberryperl/5.30.0.1@bloomberg/stable, cmake/3.19.3@bloomberg/stable, strawberryperl/5.30.0.1@bloomberg/stable, cmake/3.19.3@bloomberg/stable, strawberryperl/5.30.0.1@bloomberg/stable
Is this the line printed by Conan in the input profile? How is it possible that it is repeating cmake and strawberryperl 3 times?

Yes, that's the line of output. from our current Release profile:

C:\dev\src>conan profile show vs2015_14_v140_Release
Configuration for profile vs2015_14_v140_Release:

[settings]
os=Windows
os_build=Windows
compiler=Visual Studio
compiler.version=14
compiler.toolset=v140
arch=x86_64
arch_build=x86_64
compiler.runtime=MD
build_type=Release
[options]
[build_requires]
*: cmake/3.19.3@bloomberg/stable, strawberryperl/5.30.0.1@bloomberg/stable, cmake/3.19.3@bloomberg/stable, strawberryperl/5.30.0.1@bloomberg/stable, cmake/3.19.3@bloomberg/stable, strawberryperl/5.30.0.1@bloomberg/stable
[env]

It may be possible we're getting the duplicates because of our profile hierarchy.

We are building against conan_common_config_install/profiles/vs2015_14_v140_Release:

include(fragments/visual_studio_2015)
include(fragments/windows_64)
include(fragments/windows_dynamic_crt_release)

[build_requires]
[settings]
[options]
[env]

This includes, in order, the following:

conan_common_config_install/profiles/fragments/visual_studio_2015:

include(visual_studio)
[settings]
compiler.version=14
compiler.toolset=v140

which includes conan_common_config_install/profiles/fragments/visual_studio:

include(windows)
[settings]
compiler=Visual Studio

then back up we have conan_common_config_install/profiles/fragments/windows_64:

include(windows)
include(x86_64)

which mentions conan_common_config_install/profiles/fragments/x86_64

include(base)

[settings]
arch=x86_64
arch_build=x86_64

and conan_common_config_install/profiles/fragments/windows_dynamic_crt_release:

include(windows)

[settings]
compiler.runtime=MD
build_type=Release

three of which include conan_common_config_install/blob/master/profiles/fragments/windows:

include(base)

[build_requires]
cmake/3.19.3@bloomberg/stable
strawberryperl/5.30.0.1@bloomberg/stable

[settings]
os=Windows
os_build=Windows

Maybe that's our fault for causing undefined behaviour -- we were trying to "code defensively" and ensure correctness in case some other in-house user came and created a new profile picking one or two fragments -- should include repetition actually be a problem here?

michaelmaguire on 8 Feb 2021

Let me have a look to that profiles hierarchy, maybe the composition can avoid duplicated build_requires.
The profile inclusion mechanism is not designed to handle diamonds, that could be the cause of accumulated repeated build_requires. I will check it, but if in the meantime you want to do a test, and move those [build_requires] to a different place, so they are not repeated, that could be useful.

memsharded on 8 Feb 2021

Let me have a look to that profiles hierarchy, maybe the composition can avoid duplicated build_requires.
The profile inclusion mechanism is not designed to handle diamonds, that could be the cause of accumulated repeated build_requires. I will check it, but if in the meantime you want to do a test, and move those [build_requires] to a different place, so they are not repeated, that could be useful.

We have now tried this with a new profile: conan_common_config_install/profiles/vs2019_16_v142_Debug_all_in_one:

[settings]
os=Windows
os_build=Windows
compiler=Visual Studio
compiler.version=16
compiler.toolset=v142
arch=x86_64
arch_build=x86_64
compiler.runtime=MDd
build_type=Debug

[build_requires]
cmake/3.19.3@bloomberg/stable
strawberryperl/5.30.0.1@bloomberg/stable

[options]
[env]

Retrying the build, we see a cleaned up build_requires:

2021-02-08 11:47:29 |  [build_requires]
2021-02-08 11:47:29 |  *: cmake/3.19.3@bloomberg/stable, strawberryperl/5.30.0.1@bloomberg/stable
2021-02-08 11:47:29 |  [env]

but the error still persists:

2021-02-08 11:49:28 |  Building downstream packages for buildProfile[vs2019_16_v142_Debug_all_in_one] using build-order file build_order_vs2019_16_v142_Debug_all_in_one.json
2021-02-08 11:49:28 |  [Pipeline] readJSON
2021-02-08 11:49:28 |  [Pipeline] echo
2021-02-08 11:49:28 |  Building build-order level
2021-02-08 11:49:28 |  [Pipeline] echo
2021-02-08 11:49:28 |  Building build-order item: package_ref[libd/0.1@conan/stable#7400ebfada85b91f49ad7f4bbb5c0244] package_id[0d4135ad3773880e8ae63ed2e7e1581b8358c30d] context[host] id[6]
2021-02-08 11:49:28 |  [Pipeline] runConanCommand
2021-02-08 11:49:28 |  [Conan_pipeline_master] $ cmd.exe /C "conan install libd/0.1@conan/stable#7400ebfada85b91f49ad7f4bbb5c0244 --build=libd/0.1@conan/stable#7400ebfada85b91f49ad7f4bbb5c0244 --lockfile=distribution_per_profile_deps_vs2019_16_v142_Debug_all_in_one.lock --lockfile-out=distribution_per_profile_deps_vs2019_16_v142_Debug_all_in_one_updated.lock && exit %%ERRORLEVEL%%"
2021-02-08 11:49:30 |  ERROR: Build-require 'cmake' cannot be found in lockfile
2021-02-08 11:49:30 |  Using lockfile: 'D:\J\workspace\Conan_pipeline_master\distribution_per_profile_deps_vs2019_16_v142_Debug_all_in_one.lock'

michaelmaguire on 8 Feb 2021

The fix to our remaining issues seems to have been including --build=missing in our conan lock create calls as indicated in https://github.com/conan-io/examples/pull/81#pullrequestreview-586357817

Notes from our slack discussion in case they help others:

drodri 10:55
10:56
The --build=missing should be done everytime you plan to use that lockfile for building and the thing that is going to be built has build_requires

I would say it is safe to pass it in the general case if you are using the lockfiles in CI for building, and at least is necessary when you are computing a "full" lockfile with configuration that will be used in a install|create that builds packages

Michael 16:43
So that I understand, we say --build=missing, but e.g. if later I chose to actually do --build of all it would be the same? i.e. the --build=missing in the conan lock create calls is effectively just a hint to conan lock to tell it to include build_requires?

drodri 16:49
No, it will not be the same, that might also fail. The --build=missing is related to the graph expansion when saving the lockfile
16:49
--build will build everything, will require every build_require for every package
16:50
At some point you might be trying to build a given package, even if it had a binary, and it will not have build_requires there because the binary was not missing when you created the lockfile

Michael 16:54
OK, so if, for example, we wanted to run a job which used lockfiles to perform a rebuild of the whole DAG from source (perhaps as a once a week job just to avoid things possibly going stale) then in all the lockfile create calls we should use --build to ensure the lockfile creation includes build_requires. Correct?

drodri 18:51
Yes, exactly