Conan: [question] how to store package lockfiles in Artifactory?

[GordonJess]

I am interested in creating a lockfile for each package that is built in CI and storing it on Artifactory so that it can be used to exactly reproduce the package again in the future.

conan lock create --reference=${ref} --lockfile=${product_lockfile} --lockfile-out=${package_lockfile}

So far I have been storing these lockfiles in a generic repo under name/version/user/channel/rrev/package_id/prev thinking that this would be unique for each package but I'm now running into problems when building multiple configurations...

I have found that in some cases it is possible to have different lockfiles for the same full package reference. For example, in the ci example, if libd doesn't use the build_type setting I get this lockfile when integrating into a debug app1:

{
 "graph_lock": {
  "nodes": {
   "0": {
    "ref": "libb/0.2@user/testing",
    "options": "",
    "package_id": "281a7e3eaa861ca65ad8f566ded8fc8b48206913",
    "prev": "0",
    "requires": [
     "1"
    ],
    "context": "host"
   },
   "1": {
    "ref": "liba/0.1@user/testing",
    "options": "",
    "package_id": "5a67a79dbc25fd0fa149a0eb7a20715189a0d988",
    "prev": "0",
    "context": "host"
   },
   "4": {
    "ref": "libd/0.1@user/testing",
    "options": "",
    "package_id": "3dbaa18e9c58ee1730a3c70ec88b190afd7354f4",
    "prev": "0",
    "modified": true,
    "requires": [
     "0",
     "5"
    ],
    "context": "host"
   },
   "5": {
    "ref": "libc/0.1@user/testing",
    "options": "",
    "package_id": "281a7e3eaa861ca65ad8f566ded8fc8b48206913",
    "prev": "0",
    "requires": [
     "1"
    ],
    "context": "host"
   }
  },
  "revisions_enabled": false
 },
 "version": "0.4",
 "profile_host": "[settings]\narch=x86_64\narch_build=x86_64\nbuild_type=Debug\nos=Windows\nos_build=Windows\n[options]\n[build_requires]\n[env]\n"
}

and this lockfile when integrating into a release app1:

{
 "graph_lock": {
  "nodes": {
   "0": {
    "ref": "libb/0.2@user/testing",
    "options": "",
    "package_id": "39b0b5fad32aaf2a462d27d4282d12c08bd6c818",
    "prev": "0",
    "requires": [
     "1"
    ],
    "context": "host"
   },
   "1": {
    "ref": "liba/0.1@user/testing",
    "options": "",
    "package_id": "4024617540c4f240a6a5e8911b0de9ef38a11a72",
    "prev": "0",
    "context": "host"
   },
   "4": {
    "ref": "libd/0.1@user/testing",
    "options": "",
    "package_id": "3dbaa18e9c58ee1730a3c70ec88b190afd7354f4",
    "prev": "0",
    "modified": true,
    "requires": [
     "0",
     "5"
    ],
    "context": "host"
   },
   "5": {
    "ref": "libc/0.1@user/testing",
    "options": "",
    "package_id": "39b0b5fad32aaf2a462d27d4282d12c08bd6c818",
    "prev": "0",
    "requires": [
     "1"
    ],
    "context": "host"
   }
  },
  "revisions_enabled": false
 },
 "version": "0.4",
 "profile_host": "[settings]\narch=x86_64\narch_build=x86_64\nbuild_type=Release\nos=Windows\nos_build=Windows\n[options]\n[build_requires]\n[env]\n"
}

Following my repo structure, both are stored under the same path in the lockfile repo, and one of them overwrites the other.

I understand that either file should create the same binary package based on the 'package ID mode' and that this shouldn't prevent me from recreating the package at all. But the two lockfiles still do resolve different packages so in one of the cases I'm still losing my exact lock.

I also lose buildinfo tracability due to the checksums of the lockfile changing. The lockfile artifacts (which I add to the buildinfo) cannot be resolved properly in Artifactory once they are overwritten - and this becomes an even bigger issue for me because I collect the lockfile artifacts from a buildinfo to use in the promotion.

So I'd just like to check if you had any other ideas for storing these lockfiles?

[jgsogo]

Hi, @GordonJess

I will pass this question to @solvingj . He is working on a new course/workshop about how to integrate Conan into a successful CI. He's been doing a lot of work around it, so I'm sure he will be able to point you in the right direction.

---

(Probably he won't be able to answer before next week, I hope all of us will enjoy these Christmas days)

[solvingj]

@GordonJess I'm glad you're reaching out to ask for guidance here. When we first started training with lockfiles, we started by leveraging Artifactory Generic repos for lockfile storage. In the past 6 months, we've found several major difficulties with this approach. As a result, we've begun storing lockfiles in a GIT repository for our internal development, testing, and training. We've found this to offer a variety of advantages over any other storage strategy we've seen, so we will begin promoting this strategy in the near future. We're planning to release an additional Conan course in the JFrog Academy focused on CI/CD, and there we will use and demonstrate this strategy in action.

Hopefully this is helpful enough advice to get you moving forward for now. However, if you can keep an eye on the Conan blog for the CI/CD course release announcement, there will be a much more extensive discussion about the strategy there.

[GordonJess]

Hello @solvingj,

Many thanks for the info! Looking forward to hearing more about it and what your recommendations are for structuring git repositories for lockfiles, as I think the issue still exists no matter what the repo type (artifactory, git, etc.).

In the meantime as a quick workaround to avoid any of the overwriting mentioned above, I have added the checksum to the artifactory storage path.

<artifactory url>/<lockfile repo>/<name>/<version>/<user>/<channel>/<rrev>/<package_id>/<prev>/<lockfile_sha1>/conan.lock