Conan: Question about manifests

[zomeck]

Hi,
I have been looking at the --manifests and --verify options which are very interesting.
I have noticed that it will check for both the package and the export. Giving the following:

Manifests
Manifest for 'A/2.0-0@testUser/stable': OK
Manifest for 'A/2.0-0@testUser/stable:7dde9e5721ee2560c1c5255a11181381c0721f8a': OK

In my case I am actually not interested in verifying the package as we have several compilers and it will make the workflow more complicated to keep track of the package.

Is there any way to check just the export?

Thank you very much!

[memsharded]

The structure of the manifests storage is designed to take into account the existence of multiple compilers. Each compiler will get a different binary, then a different ID, and a different manifest, but that will be stored and checked consistently, using the package ID as subfolder into the captured manifests folder.

Regarding your question, I am afraid it is not possible. The issue is that the manifests functionality is attached to conan install and conan install is intended to install both the recipe and the binary package, so no easy way to decouple that. I guess that if you are not interested in the manifests of the binary packages, you can just remove the "packages" folder inside the manifests folder, that would be easy.

[zomeck]

Hi,
Thanks for the quick response.
I tried removing the package folder. However then I get the following when I run --verify:
ERROR: Modified or new manifest 'A/2.0-1@testUser/stable:500a5737cfb7bee13d4a0039e72446892ca242ab' detected.
Remote: local cache
Project manifest doesn't match installed one

[memsharded]

Yes, that is true, it is not prepared for such folder being remove, sorry for the wrong tip.
In any case, what is the problem of having the package manifests? They work well even for multiple compilers, so not sure about the use case. Could you please explain a bit further?

[zomeck]

Yes I can explain further.
In my situation we have several OS and compilers. When I generate manifests of the recipes I will generate them only for my configuration in the package folder.

If I understood correctly when you generate the manifest you generate them always for the export and the package one. When being in a different OS I could actually have a recipe with the same version number but different content (could be because I forgot to update it when I was doing tests etc.) in this case I will override the manifest of the recipe to the wrong one (which is what I want to prevent).
After that the verify will be ok. Since the verify is ok I will upload the packages to the server overriding the recipe which will make all the rest have a verification failure .

In my case I want only to know when the recipe itself has changed and then warn the developer that maybe has done the wrong thing as you usually do not want to modify something that is already uploaded (The most likely case is that one does some changes in the recipe and forgets to update the recipe revision).
Maybe there is a simpler way to do this since there is already a conanmanifest in the export folder of the recipe.

Thanks again for your time.

[memsharded]

No worries :)

• When you are in a different OS, as the package is different, it will have a different ID, and the manifests for the package will be stored in a different folder under .conan_manifests, so it will not be overwritten.
• I understood that you had a problem with the package manifest, but it doesn't seem the case, but instead an issue with the recipe ("export") manifest.
• The issue when creating a package (for same name, version, user, channel) with a different recipe (because it is outdated) from a different OS has a different problem: other package binaries will be marked as "outdated" (check conan info). Each package binary contains the hash of the package recipe, to ensure consistency (this package was generated from the same recipe), and even if we allow the behavior, it would be preferred that all packages in all OS will be generated with the same recipe.

I'd say that we could be looking for a different use case. Manifests were intended to achieve security, in the sense that no package could be modified upstream without the consumer knowing, not to avoid consumers overwriting existing recipes. From my experience, I'd say that this is also mostly a process issue: developers shouldn't be able to write upstream packages from their machines, but CI jobs should do the task, precisely to ensure repeatibility and avoiding errors.

We will consider it as a possible enhancement for next 0.23, can't guarantee that it will be implemented, but we will analyze the issue and possible solutions.

Thanks for your feedback!

[zomeck]

Hi thanks for the clarifications.
There is some details I have a bit unclear.
I understand the package has a different id, (e.g.:.conan_manifests/A/2.0-1/testUser/stable/package/500a5737cfb7bee13d4a0039e72446892ca242ab/conanmanifest.txt) that one will not change as it will create a folder in parallel when I run --manfiests

But the following manifest (.conan_manifests/A/2.0-1/testUser/stable/export/conanmanifest.txt) will it be overridden when I run --manifests when I generate the second package? I recall from the tests I did it does (I could be wrong).

I think the feature of being outdated can be useful. However when I was doing tests. I would do the following:

• generate recipe manifests
• modify the code of the recipe (the binaries generated are the same in this case)
• run test_package
• run conan info /path/to/conanfile --update, it says I am in the latest version (None).
• run conan info /path/to/conanfile --build=outdated, it says package up to date.
• run conan info /path/to/conanfile, Updates: Version not checked
• run install with --verify fails (as expected)

Should I run conan info with different arguments?

About the process. I agree that the CI jobs would help a lot (not there yet). But still someone has to write the recipes and has to make sure they work well in the different platforms and that the packages generated are relocatable. Not all members of the team have the same level of expertise on build systems or on conan itself and several iterations are usually needed for the same revision.
I am trying to have a process where it is easy to try out new things and hard to mess up. This way the team can learn by trying out things without being scared of breaking anything.

I think maybe in my case it would be enough to be able to compare the content of my conanmanifest with the one that is in the local cache. Since that already has the hash of all files in the export. Maybe conan info provides that information? I see it provides a path to the export folder.

Thanks for considering the enhancement.

[memsharded]

Hi,

What do you mean with the first point generate recipe manifests? With an export?
Just in case, the conan test_package can also receive manifests args. If you are generating a package, it makes sense to capture the manifest for the recently create package, so you could try:

$ conan test_package --manifests

Doing that, the last conan install --verify shouldn't fail.

About the enhancement, it is not difficult, as there are already some checks in place (check conan messages when you do a conan export of a modified recipe vs an unmodified recipe), but the main thing is that it should be an opt-in behavior, because otherwise we would break existing processes for many existing users. Then it is not very protective against user mistakes, because users have to remember to do a conan export --check, if they do the standard conan export it will not prevent the overwrite of the existing recipe in the cache. So I am not sure about the value of this feature, I would say that it won't be used at all. What do you think?

[zomeck]

Hi,
Sorry for being unclear. What I meant is conan install --manifests (the manifests of the recipe stored in the repo).
I have seen that the test_package has a manifests options. But then the workflow complicated more.
Maybe it could something like install --verify all --verify export. However verify already takes a path as an argument. I agree that the feature would not make much sense and also as you mentioned verify has other purposes.

I have seen that if I run
conan info ../ --paths -n export_folder
I get the path to the export folder.
From there I can compare the 2 conanmanifests. Which should not be much work.

Thanks for all the information

[memsharded]

No worries.

Maybe you might be interested in the --no-export command line argument:

$ conan test_package --no-export  #-ne

This will use the already installed recipe instead of overwriting it. I have thought about it, and it is the same behavior you would get with such requested feature:

• If there is no package recipe in the cache, it will try to retrieve it from the remote (so it will do the conan install for you)
• If no existing recipe, it will fail

Could this arg help?

[zomeck]

hi,
yes I think this is a piece I was missing :)
Just to make sure I understand. Running with the --no-export with option with --manifests would never override the manifests of the export to the wrong ones (as it is not doing the export) thus creating just the package manifests safely.

Thanks a lot

[memsharded]

Great! Yes, exactly that, it will not copy ("export") the recipe in the user folder to the local conan cache, but use the one already existing in conan, so it will not change the recipe, nor the recipe manifest, and it will only create the package (and the package manifest) for the respective configuration. If you don't want to re-build the package if it is already existing, you should add conan test_package --build=missing. Cheers!

[memsharded]

Hi, @zomeck,

I was reviewing this issue, that is scheduled for 0.23, but seems it was already addressed with -no-export. Anything else required, any further feedback, or could this issue be closed? Thanks!

[zomeck]

Hi,
It could be closed. thanks to you guys!