Conan: [bug] New revision uploads although --no-overwrite all flag is used

[RadMap]

I have a remote artifactory server configured for revisions. Creating and uploading a package twice with just introducing an empty line in the recipe for the second build, creates two revisions on the remote server although the --no-overwrite all flag was used

Environment Details

• OS: Windows 10:
• Conan version: 1.20.3
• Python version: 3.6.5

Steps to reproduce

First I run
conan upload package/1.2.3@user/channel --all -r my_remote --no-overwrite all
This correctly uplads the package with recipe.
Then I change the recipe of package locally by introducing a new empty line and recreate the package.
After this I run
conan upload package/1.2.3@user/channel --all -r my_remote --no-overwrite all
again. I would expect the last operation to fail since I effectively invaidate the exisitng package on the server and the --no-overwrite should prevent that.

[Minimonium]

I'd consider it a bug.
The behaviour of --no-overwrite shouldn't change when dealing with revisions. Revisions are a tool to not lose information on overwriting. The current behaviour seemingly just ignores the option, which is clearly not correct, since users are getting an overwritten version.

[czoido]

Hi @RadMap,
Thanks a lot for reporting.
I think, that if you have revisions enabled, from the moment you are updating the recipe you are creating a new revision, so no recipe exists for that revision in the remote, that is why --no-overwrite is not producing the result you would expect... Maybe we should change the documentation about this to make it more clear in the case you are using revisions because in that case, it does not make so much sense to use this option by the behavior you are telling.

[RadMap]

@czoido
I agree that technically nothing is overwritten. I am however effectively overwriting the package for all users. If this is intended It would be good to document it as you say. I still think it would make sence to implement the behavior i seek because I could not think of any one using this flag with any other intention. Possibly implement a new flag --no-revision for my case. But this just seems convoluted. Thanks.

[RadMap]

With risk of wasting peoples time, I have looked into the code. On line 365 in uploader.py (method _recipe_files_to_upload) we find remote_snapshot = self._remote_manager.get_recipe_snapshot(ref, remote). After executing this line remote_snapshot is an empty list in my case and conan subsequently uploads the package. I would not expect this list to be empty since there is a package with my reference already on the server. The get_recipe_snapshot method even requires revisions to be enabled so something does not make sence here. Is it the server that does not respond properly to the get_recipe_snapshot request (Im told the server is a commersial artifactory server)?

[czoido]

Hi,
I think that the empty list is the expected result as there is no snapshot of the new calculated revision in the remote. In ref we are passing the revision and checking if the new revision is in the remote. As the conanfile.py is modified, the revision changed so there's nothing in the remote with that revision.

[RadMap]

Hi and thanks for answering. It looks like ref is the reference and not the revision or am I wrong?

[czoido]

Sorry, I did not explain properly, ref is the object we pass with the revision property, and the revision is checked later.

[RadMap]

Ah! Ok. thanks for clarifying! How do I request the functionality I want (with --no-overwrite flag or other flag)?

[abhi1010]

I have the same issue - I would like uploads to respect --no-overwrite flag even if revisions are allowed through revisions_enabled=1 in ~/.conan/conan.conf. I had a discussion about it here:
https://cpplang.slack.com/archives/C41CWV9HA/p1583570176014300

[memsharded]

Just to summarize about the rationale here:

• Revisions are automatic versions. A new revision of a package is a new version.
• A new version does not overwrite a previous one.
• Overwrite, as defined in Conan, is intended to mean that some files are being overwritten in the server, and thus, their content is lost forever. This doesn't happen with a new revision.

Now, there is a feature request: to restrict uploading new revisions of packages. I am not sure what does it mean. Wouldn't that be the same as disabling revisions? If you can't upload new revisions, why revisions at all? If you could please clarify your use case. Thanks!

[Minimonium]

I assume that the use-case of --no-overwrite for a non-revision setup and the lack of understanding of revisions forces this discussion into a misunderstanding.

The flag evidently brings up some confusion for its [lack of] behaviour on revision-enabled clients, which means that while the purely server-centric explanation is technically correct - it's less obvious to a client user.

I think that revisions and overwriting are different things. From a client perspective, when a new revision is created - the recipe/configuration is overridden [despite passing the flag not to override it]. Revisions here are just a tool to get the history. Is the main concern with the flag for revisions - a semantic meaning?

Now let's take a look into a similar use-case to one brought up by @abhi1010 in Slack:
We need a more flexible approach for our CI pipelines. To create correct graphs - we need to rebuild some configurations which should have different hashes depending on their options/requirements and their options too. It's very inconvenient to create an additional job for each additional configuration (configuration explosion). So in order to make the CI more "reactive" - we build rare missing configurations on the spot instead of modifying an upstream pipeline and to not throw away built artifacts - we upload them to the remote.

Correct me if I'm wrong here since I've yet to integrate revisions into my workflow - our task is to upload only new revisions that were created on that client, meaning revisions that were only created if configurations were lacking for a downloaded dependency recipe revision. When we do a conan upload --all - it will upload only new revisions that were created locally, it'll not override existing revisions if they're fit to use, correct?

[memsharded]

When we do a conan upload --all - it will upload only new revisions that were created locally, it'll not override existing revisions if they're fit to use, correct?

If I understood correctly, no. The thing is that a revision is also a hash of the contents. Trying to upload that revision again is not overwriting anything, as the de-duplication mechanism will skip the upload completely. Only in the very weird case when the revision is the same but the contents have changed, the -no-overwrite will kick in. Take into account that -no-overwrite is not a security measure in any case, just a little helper. If you really want to avoid overwriting you should set up your Artifactory repo, removing the "delete" permission to those users. Things that cannot be deleted cannot be overwritten.
So if the contents changed but the revision is still the same, it is actually ok that it will raise an error.