Conan: Have conan info --build_order respect --update option.

[tru]

When meeting up with my colleagues @16k and @stsimmons during our current conference we discussed through the CI integration. We just realised that we could simplify some of our code if we could have the following feature implemented:

Goal:

Get a list of binary packages (for current settings) that is not available on the remote in the build_order
format.

Suggestion:

conan info --build_info ALL --update should just include a list of packages that are not available
in binary format on the conan server.

Implementation:

We imagine that this would work just a conan install would do if you do conan install --build=never.
Compute the hash for the binary and check them against the remote. If they are not available the package
and all it's dependencies should be outputted in the normal build_order format.

This would allow our CI to build all changed conanfiles on random slaves and upload them without jumping through hoops.

Let us know if this is clear or not and if it's something you guys would consider to add or point us in the right direction for getting it in.

[memsharded]

I think I understand, doesn't seem simple, but possible. As a general guideline, we are not opposed to add many information functions, even for special cases, we have to fully understand it and check about the approach, but yes, we will consider to add it.

I am just wondering about your workflow, how will you guarantee that there won't be built packages in the server from a previous build?

[tru]

I am just wondering about your workflow, how will you guarantee that there won't be built packages in the server from a previous build?

Just to make it clear, when we are talking about server I mean conan_server. So we just want to "fill in" the packages that don't have binaries already uploaded. Just as conan install does it builds the ones that are not available as a binary package. We want to use that list and just go back run test_package and then upload on them.

We are not concerned about the cases where the source changes but not the version number, we want to avoid that by way of policy.

[abuszta]

Maybe easiest solution for this would be implementing what @Phibedy requested in https://github.com/conan-io/conan/issues/500#issuecomment-250552950 and what I suggested in https://github.com/conan-io/conan/issues/447#issuecomment-251958076? Simply getting information from conan server which packages are out-of-sync/should-be-updated (could be in build order)?

[memsharded]

Hi! @lasote has come with a proposal for this, in https://github.com/conan-io/conan/pull/611/files

I would like to summarize the proposal approach, and check that it addresses the need:

• It uses a recipe hash, that it will change if the recipe changes. Generated binaries will store in their conaninfo.txt the hash of the recipe used to generate them.
• $ conan install --build=outdated will perform the task at once. It will check the dependency graph, check which binaries exists, and those that do not exist or are outdated wrt the package recipe, will be re-built. Of course they will be re-built in the correct order.
• After install they can be uploaded to the remote.

With this, the implementation on CI would be trivial, just change recipes, upload recipes, fire the install of the base/downstream project with --build=outdated to take charge of rebuilds of the changed package. As usual, if it is needed to rebuild the dependents, the typical build-order can be used.

Any suggestion, feedback, thought...? Thanks!

[tru]

So yeah this seems to be what I wanted. Except for one little thing. I want to be able to run test_package on the outdated packages. That means that either I want it to output the packages that are outdated so that I can use the build-order to build them and update them. Also you say it can be uploaded after install but how do I know what packages to upload? Do I need to parse the full output to get that info?

[lasote]

@tru do you have multiple packages generated for each recipe, right? But test_package will run with a specified set of settings, How do you know what settings are needed to match the outdated packages? About the upload, I would suggest to upload --all, the upload won't upload again an already uploaded package.

[tru]

Not sure what you mean?

Some recipes have multiple options that can generate different packages, I do this by dropping a txt file in the directories that need multiple options and then my wrapper script just invokes test_package for each line in that file to generate all different options.

So for upload you would suggest to just run upload on _all_ packages even if they have not been executed and tested? As long as that's a pretty fast operation I don't see a huge problem with that, even if I think that my ultimate solution would be something that just lists all the outdated packages in a build_order style output and I can run test_package and then upload on each one of them. I'll handle options building outside of conan for now.

To me it's just important that the CI have the opportunity to run the tests as well to catch any problems early.

[lasote]

Some recipes have multiple options that can generate different packages, I do this by dropping a txt file in the directories that need multiple options and then my wrapper script just invokes test_package for each line in that file to generate all different options.

I understand. So you will always generate all the packages for a given outdated recipe, right?

So for upload you would suggest to just run upload on _all_ packages even if they have not been executed and tested? As long as that's a pretty fast operation I don't see a huge problem with that, even if I think that my ultimate solution would be something that just lists all the outdated packages in a build_order style output and I can run test_package and then upload on each one of them. I'll handle options building outside of conan for now.

I was just trying to understand why do you need a list of recipe references that have one or more packages outdated, but I think I get it now.

In the opened PR the command conan search offers information about the outdated packages.
So, if you run conan search Boost/1.60.0@lasote/stable you will get a long list, if any package is outdated will print "oudated from recipe: True".

Could be suitable to iterate the conan info --build_order output and check with a search if there are outdated?
Something like:

    for each conan info --build_order reference:
        conan search Boost/1.60.0@lasote/stable | grep "outdated from recipe: True"

If you can use some wrapper script to do that maybe you can upload only the packages belonging to an outdated recipe (with --all, that's what I meant with --all).

[tru]

Yeah that sounds very possible, I will try that search command out.

[lasote]

Feel free to write at #1895, I'm trying to describe how the conan info should be in Conan 2.0 (and start deprecating current mess)

[jgsogo]

I think that the discussion on this issue should be moved to the one referenced above where the redesign of the conan info command takes place. I will mark it as obsolete so it gets closed in the next few days.