Conan: Downloading Packages in CI Environment

[kolrami]

Hi, I currently use conan for managing my tool dependencies and that works quite well :) A developer only need to do a conan install and get a completely working enviornment.

Of course, I would also use that mechanism in a CI environment. This runs Zuul in an open stack cloud, where for every build a VM is created from a predefined image. I would like to already download all packages of a remote (for a specific profile) into this predefined image, so that I do not have to download all tools for every build again and again, but only updated ones not yet in the image.

What I did for now was to use conan download for every package found in the remote repository. Unfortunately, this is only working partially. After having downloaded a package via conan download a following conan install finds all packages in the local cache and do not download them again. However, the packages are not connected to the remote any longer (in registry.json) and a conan install --update does not work, i.e. does not ask on the remote if the package was updated.

Do you already thought of such a use-case? What I would need is basically doing a conan install for all packages in a remote repository (including all different options, ...), i.e. without a specific conanfile.txt just install all available packages into the cache. I could build a complicated script which queries the remote for all packages and its options and execute conan install for all of them but I think it would helpful to have a command in conan directly supporting such a use-case. What do you think? I am looking forward to hear your opinions and ideas :)

[lasote]

Hi! Thanks for opening this, sounds interesting.
It is the first time I hear that use case of "caching" the packages in the CI server, but together with the --update idea, I think that is a legit idea!

I would like to know more about what is the reason you want to cache the packages, transfer time? or maybe only download many packages (being the network fast) is slow?

At first sight, I wouldn't be opposed to updating the registry when a package is downloaded using conan download so you would be able later to do a conan install --update.

[kolrami]

HI @lasote, thank you for the answer. The main reason I want to cache the packages is really transfer time and also reducing the network load. You can imagine if hundreds of jobs are running in parallel and they all download the tools (currently around 3GBs), this results in increased build times and increased load. With a cache in the image we could reduce this overhead significantly.

[lasote]

Thanks for the answer.
Feature:

• Make the "download" command to create the registry entries when downloading packages, so the "install --update" can work correctly.

[memsharded]

This surprises me a bit, I think it is already the current behavior. The following test passes OK:

    def download_set_remote_test(self):
        # https://github.com/conan-io/conan/issues/4373
        client = TestClient(servers={"default": TestServer()},
                            users={"default": [("lasote", "mypass")]})
        conanfile = """from conans import ConanFile
class Pkg(ConanFile):
    pass
"""
        client.save({"conanfile.py": conanfile})
        client.run("create . pkg/0.1@lasote/stable")

        ref = ConanFileReference.loads("pkg/0.1@lasote/stable")
        self.assertTrue(os.path.exists(client.cache.conanfile(ref)))

        client.run("upload pkg/0.1@lasote/stable --all")
        client.run("remove pkg/0.1@lasote/stable -f")
        self.assertFalse(os.path.exists(client.cache.export(ref)))
        client.run("remote list_ref")
        self.assertEqual("", client.out)
        self.assertNotIn("pkg/0.1@lasote/stable", client.out)

        # Now download, and update
        client.run("download pkg/0.1@lasote/stable")
        client.run("remote list_ref")
        self.assertIn("pkg/0.1@lasote/stable: default", client.out)
        client.run("install pkg/0.1@lasote/stable --update")

[kolrami]

Hi @memsharded, thank you very much for your feedback. It's interesting, I now also tried a unit test for my case and it also passes, even with 1.9.0 release which I used for testing "in the real world". However, I was quite sure that I had trouble with this in my setup. I will investigate and report back :)

[memsharded]

@kolrami sure, please let us know :)

[kolrami]

Hi @memsharded and @lasote, I now tried it again with version 1.9.0 and it did not work, i.e. after conan download .../...@.../... it downloaded the package but a conan install --update did not check on the remote for a newer version.

However, I also tried 1.12.0 and that seems to work, so exactly the same commands work as expected. So I will try to roll out 1.12.0 to our CI and hopefully my issue is resolved with that :)

Thank you again very much for your help. If I encounter another issue with this I will just write here again.

[lasote]

Thanks, glad to hear it is working.I'm closing this. Write again if needed.