Conan: How to implement an "append only" conan workflow without the need for delete/overwrite remote access? [question]

[gesellc]

Our motivation

We would like to minimize the risk of accidentally overwriting conan packages that have been uploaded to our artifactory repos. There are technical reasons for that (technologies with non-deterministic build results) and also reasons due to the size of our program (big multi-team effort).

Our use case

• We want to implement an "append only" conan workflow for all automated uploads that are triggered by our build users (Jenkins, GitLabRunner, etc.).
• We want to allow these users to create files but not to change content of existing files in our artifactory repos.
• We restrict overwrite and delete access to our conan repos using the artifactory access control.
• We also need to parallelize our builds. Which means that - for one package version - we need to upload different package variations from different build machines. We use recipes with identical content for each of these uploads.

We are aware of this trade off

• We know that we loose the ability to change recipes of existing packages.
• But we can do without this feature because we can create and upload new package versions in any situation where changing a recipe would normally be an option.

Our current challenge

• While uploading additional package variations (for example windows vs linux) conan tries to overwrite the conanmanifest.txt file next to the already existing conanfile.py

Example

• Create and upload windows package variants.
• Create and upload linux packages variants for the same version but from a different build machine.
• each time using conan upoad ... --all ...
• => error

Requesting upload permissions...Done!
Uploading conanmanifest.txt
ERROR:
Error uploading file: conanmanifest.txt, '{
  "errors" : [ {
    "status" : 500,
    "message" : "Not enough permissions to overwrite artifact <artefact and user name removed> needs DELETE permission)."
  } ]
}'
Uploading conaninfo.txt
Uploading conan_package.tgz
ERROR: Execute upload again to retry upload the failed files: conanmanifest.txt. [Remote: <remote name removed>

• this is the conanmanifest.txt next to the conanfile.py
• conanfile.py has the same content locally and remotely
• conanmanifest.txt has a different content locally and remotely. It's the time stamp in the first line.

Our question (and assumed solution)

• How can we prevent conan from trying to overwrite files during an upload if the local and remote files have the same content but different time stamps?
• A "don't upload if only the time stamp differs" flag comes to mind. Or is there an elegant way to implement our use case with the existing conan release?

Related Issues

• https://github.com/conan-io/conan/issues/2407 How to upload additional artifacts to an existing package?

  • The mentioned resolution is not what we want ("an argument that explicitly allows to upload only package binaries, but not the recipe")
  • We want to upload the recipe in case it has not been uploaded yet

• https://github.com/conan-io/conan/issues/1381 Upload without force overwrites server with local package

  • This issue is only remotely related to our request.
  • We want to give our CI users the right to upload filed, but not to change files.
  • We want to enforce this rule independently of the flags that are used when calling conan.

• https://github.com/conan-io/conan/issues/679 Feature request: ovewrite_permissions for conan server

  • The discussed ability to set permissions for certain user/channel combinations does not solve our use case.
  • Our CI users should not be able to delete or overwrite existing files.

To help us debug your issue please explain:

• [x] I've read the CONTRIBUTING guide.
• [x] I've specified the Conan version, operating system version and any tool that can be relevant.
  
  
  
  • Conan version 1.1.0 (but we have the same issue with older versions)
  
  
  • Windows 7 (for most of our builds)
  
  
  • Artifactory Version 5.5.1
  
  
• [x] I've explained the steps to reproduce the error or the motivation/use case of the question/suggestion.

[memsharded]

This is a very good point. Thanks very much for such detailed and clear explanation, it really helps!

How can we prevent conan from trying to overwrite files during an upload if the local and remote files have the same content but different time stamps?

I thought conan did already that, if the only thing that changes is the timestamp, it won't upload at all. I am going to run a couple of checks, but if in the meantime you could double check that the manifest is indeed identical both in the client and in Artifactory, except the timestamp, that could help too. A possible cause could be checking out the recipes from VCS like git and changing the line-endings, which will end in a different manifest. Many thanks.

[memsharded]

I think your use case might be covered by this test: https://github.com/conan-io/conan/blob/9fc0cf0f0389f637b7fa60ce78c2b677ea619291/conans/test/command/upload_test.py#L125

So not sure what could be failing. Looking forward your feedback on the full manifests.txt, to see if it brings some light. Thanks!

[mmatrosov]

In our company we have very similar setup and we face the very same problem. The only difference is that we actually want to be able to generate new packages for existing recipe using an _updated_ recipe, but we still don't want to overwrite the recipe on the remote.

Here is the workflow:

1. Prepare recipe and generate Windows packages
2. Upload Windows packages and the recipe to remote
3. Later, realize that this recipe cannot produce proper Linux packages. Update the recipe so it now can produce proper Linux packages.
4. Generate Linux packages
5. Upload Linux packages

On step 5, I want conan to merely upload new package, but not touch remote recipe or manifest. And now comes the crazy part. It actually works. But only if I have permissions to overwrite. While conan _does not actually overwrite_ neither recipe, nor manifest! So writing permissions are required, but nothing is overwritten actually!

Here are detailed steps how to reproduce my test (on Windows):

1. Write first version of recipe, name is "temp"
2. conan create temp mmatrosov/testing
3. conan upload -r align --all temp/1.0.0-1@mmatrosov/testing
4. Update recipe to second version (I simply changed "description" field)
5. conan create temp mmatrosov/testing -s os=Linux
6. conan upload -r align -p <hash_for_linux_package> temp/1.0.0-1@mmatrosov/testing
7. conan search temp/1.0.0-1@mmatrosov/testing - this shows Windows package as outdated from recipe, which makes sense, since Linux package is generated with seconds version of the recipe
8. conan search -r align temp/1.0.0-1@mmatrosov/testing - this shows Linux package as outdated from recipe (!), which also makes sense, since recipe on remote _was note overwritten_, and corresponds to the first version, used to generate Windows package
9. I also logged into Artifactory and manually checked, that both conanfile.py and conanmanifest.txt correspond to the first version.

Now, I was able to perform this test, but I have overwrite permissions for Artifactory. When my colleague (who doesn't have permissions) repeated this test, he got error on step 6. Exactly the same error as OP posted.

Obviously, most of our developers don't have overwrite permissions. So the question is, could you please fix this behavior, so that conan doesn't check permissions for what it actually doesn't need?

[danimtb]

@mmatrosov Which version of conan did you use for the steps indicated?

IIRC, behavior described in step 6 was changed to avoid different status from actions done in remote (upload) vs actions done in local cache (export/create). Now every time you upload a package it also uploads the recipe it was generated with.

Please check it before this behavior fix interferes with your current flow.

[mmatrosov]

@danimtb

Ok, seems like we had a little mess with the versions, sorry about that. The scenario that I described to you was actually tested with version 1.2.3. Now I tried to do the same with version 1.4.5, and it works just as you described :( The recipe and manifest were updated on remote after step 6.

But I guess you understood what I need. Is there a way I can do it? Seems really convenient to me. You see, when I don't overwrite recipes, I can guarantee _builds reproducibility_! I.e. whichever was uploaded, is not going to change. Which is very nice.

[danimtb]

@mmatrosov I think I dont really get your idea of build reproducibility. If you have packages generated with one version of the recipe (step 2 in your workflow) then change the recipe to add new changes for other configuration (step 3 in your workflow) and upload ONLY new packages (step 5) you are losing the connection between the packages and the recipe they were generated with.

• What if the new modified recipe is not able to generate previous packages?
• Where do you save the changes of the modified recipe?
• How can you reproduce (build from sources) the new generated packages?

I understand the need to constrain permissions in Artifactory but I think there are good reasons to keep recipe sync'ed with packages.

As a side note, an upload of a new packages WITHOUT changing the recipe will not overwrite the recipe and might not require overwrite permissions.

[mmatrosov]

@danimtb

Where do you save the changes of the modified recipe?

In Git. We have dedicated repository for recipes called ThirdpartyRecipes.

How can you reproduce (build from sources) the new generated packages?

By taking latest recipe for this library from ThirdpartyRecipes.

What if the new modified recipe is not able to generate previous packages?

Usually we don't need to do this, since packages are never deleted from Artifactory. But if we need to, we can use previous version of the recipe from ThirdpartyRecipes.

So the idea is to use Git to version recipes (since Git is, you know, version control system), and use Artifactory to merely store packages.

This setup guarantees builds reproducibility in a sense if you have some CI build plans using certain packages, they won't break, since recipe and existing packages do not change. However, the same recipe might match new packages, if they are uploaded. And it is not really important, how they were build, what's important is their package_id, which should be correctly identified by the existing recipe.

Consider another, yet related problem: a typo in a recipe. In the description, for example. Or wrong link to homepage. Or problem with code formatting. Do I really need to generate new version of existing packages if I change this kind of stuff in the recipe? I think not. I want to fix these typos/problems as I see them, but I don't want this change to prevent me from generating new packages for effectively the same recipe.

[mmatrosov]

I did a couple of more experiments using version 1.2.3. Concerning this paragraph that I wrote before:

Now, I was able to perform this test, but I have overwrite permissions for Artifactory. When my colleague (who doesn't have permissions) repeated this test, he got error on step 6. Exactly the same error as OP posted.

It turned out that my colleague was actually running version 1.3.1. But if we repeat exactly the same steps with 1.2.3, it works! We can add new packages without updating the recipe, and overwrite permissions are not required! Exactly what we need!

Now the question is, how do we preserve this nice behavior, when we update to the latest version of conan? We checked that behavior in 1.5.1 is the same as in 1.3.1, so seems like it is not going to change.

[danimtb]

@mmatrosov As I told you, the behaviour was changed recently to keep it consistent.

Regarding the overwrite permissions, we are considering to introduce "revisions" to the remotes, so you can upload a new revision for the same reference (no need to bump the package version) and you won't be overwriting anything. The idea is to get always by default the latest revision unless it is specified somehow.

This mechanism is it still under investigation and planned in our roadmap.

[mmatrosov]

@danimtb The revision mechanism is nice, however, IIUC, I still will need to regenerate all the packages for the new revision, which is not convenient. This is according to @lasote's comment: https://github.com/conan-io/conan/issues/3158#issuecomment-402106782:

About generating again the binary packages for the recipe, we don't have a fix for that, you would need to do it even with the revisions mechanism.

[danimtb]

@mmatrosov you are right, revisions will not achieve exactly the "append only" flow. However, I was trying to say that you could benefit from revisions to have new versions of the recipe tracked and still don't overwrite anything. Old packages will still be accessible in old revisions.

I have to disagree with you when you say that rebuilding is not convenient, as I think it is the only way to make sure recipe works every configuration and packages and recipe are always synced in the remote. As said before, uploading new packages without changing the recipe should be possible without overwrite permissions.

btw, I would love to see @gesellc joining the discussion to see what they think about the original issue and if revisions could be a better approach to solve it

[wittmeie]

I also stumbled over two similar issues in our environment. One was due to the line endings as and the other due to a change that was introduced in Conan 1.3.2. Here, the write permissions for "group" and "other" are removed from the files when the tarballs are generated: https://github.com/conan-io/conan/commit/833b9ef288230311865c23025777c41268fbc15d#diff-9e273c58f528ec321db5954d73096c3a

This leads to different tarballs and hence different checksums for packages deployed with Conan <= 1.3.1 and Conan >= 1.3.2. Hence, we are not able to deploy additional binary packages for recipe packages that were deployed with Conan <=1.3.1 when the delete-rights are not granted.

Maybe it might be a possibilty to store the Conan version that was used for the creation of the recipe package as part of the package in the future in order to provide means to handle such inconsistencies that arise from different Conan versions?