Conan: [bug] Auth with Artifactory from revisions to no-revisions

[RadMap]

As previously reported in #6082, passing the --no-overwrite flag when uploading a conan package does not take revisions into account. This means that if "revisions" is enabaled, a new package will be uploaded, invalidating the previous revision. Although nothing is technically overwritten (since we get a new revision), the flag works in a counter intuitive way. I request a feature where there is a Conan way of preventing upoading a new revision if a revision already exists. This could be done in two ways:

1. Make the no-overwrite flag take revisions into account and document this.
2. Document that --no-overwrite does not have any effect working with revisions and implement a new flag --no-revision that prevents uploading new revisions.

I think the first alternative would make most sence.

[Minimonium]

If we're dead set on no-overwrite doing nothing in the current Conan version with revisions consider the behaviour change for the Conan 2.0.

[lasote]

invalidating the previous revision

Technically speaking, the previous revision is not invalidated, but the latest uploaded when no revision is specified explicitly or with a lock file in a consumer, Conan resolves to the latest.

Could I ask why do you need this feature? Maybe a better flow with different repositories/permission management could work.

[RadMap]

@lasote
Hi and thanks for answering.
We work in a highly regulated industri (med-tech) where, when we have 3d party dependencies, we create our own conan recipes and build all packages our selves. We also have abslolutely fixed dependencies. This pushes us into a situation where we have all our recipes in the same repo as the product code. This makes it possible to easily update the whole dependency chain and build in one go. We want to be able to automatically update all our dependencies from the ci pipeline if a dependency buid was succesfull but only if there is not already a revision. We also do not have control over the permissions and the setup of the artifactory server (managed by devops and takes a week to get response). The safest way would be to be able to manage when we upload and not by conan commands. The simplest way to go about would be to have the --no-overwrite flag do the intuitive thing.

[lasote]

This makes it possible to easily update the whole dependency chain and build in one go. We want to be able to automatically update all our dependencies from the ci pipeline if a dependency buid was succesfull but only if there is not already a revision

So, you want to make sure that only 1 revision is generated for the same name/version@user/channel to make sure you bump the version of your packages, right? My question is, do you need the revisions feature?

[RadMap]

do you need the revisions feature?

No. But the problem is that you are not always in control of the server side. For larger companies, this is often the case. What is so great about Conan is that it is flexible and yet comprehensible. We can happily manage all the package creation in the team (unlike when we had to squeeze native code in nuget packages with CoApp which was a nightmare). To be able to have full control from the client side is one of Conan's strength (I think). (I would probably not ask for this feature if there was not already a flag --no-overwrite.) Thanks!

[lasote]

But the problem is that you are not always in control of the server side

But the revisions can be enabled/disabled in the client-side, so with the revisions disabled only a "0" revision will be generated on the server and then the --no-overwrite flag will work.

[RadMap]

My experience with our commersial artifactory server is that I cannot upload packages unless I have enabeled revisions on the client side. If I try without revisions enabled, I get an error. If it was possible to disable revisions on the server from the client side it would be great!

[lasote]

It should work perfectly. The server have 2 apis, the v1 (without revisions) and v2 (with revisions). Depending if the client have revisions mechanism enabled, it will speak with the server with the correct API. So, when using v1, all the revisions are stored in the server as "0" transparently. BUT if you have virtual or remote repositories (not local directly) configured in the server side, only v2 will work. Maybe that's the case.

[RadMap]

Thanks for feed-back.
The devops people assure me that all repos are local on the server.
The error I get when trying to upload a package without revisions enabled is

ERROR: 404 Client Error: Not Found for url: https://<SERVER>/artifactory/api/conan/<Remote alias>/v1/files/<PACKAGE>/0/export/conanfile.py

where the <> are introduced by me to obfuscate the secret parts of the message.
With revisions turned on it works wthout problem.

I have tried to look in the Artifactory documentation if the server is configured to only respond to APIv2 calls but I cannot find anything about that. Any clue of whats going on?
Artifactory version is 6.13.1
Repository layout is: conan-default

[lasote]

That is definitely not normal. Try to export CONAN_TRACE_FILE=/tmp/some_file.txt and run the command again. Maybe the traces in the communication help. You can also try to authenticate first on the server doing conan user <USER> -p -r <REMOTENAME> and repeat the command to check if it could be an issue with the authentication flow in the apiv1.

There is no way to tell Artifactory to use only the v2, (unless remotes/virtuals) it supports both protocols.

[RadMap]

Thanks a lot for input.
I attache the conan trace file for failed an successful upload and corrseponding screenshots. I dont really know how to interpret the log.

Can it be a problem that there are already packages (not with the same reference as the one failing upload) that are uploaded with revisions on the server?

Conserning the Feature request. I still believe it could be worth implementing since certain server configurations are forced to use APIv2.

conan_trace.txt

[lasote]

Thanks so much for the traces. But nothing obvious :(

Can it be a problem that there are already packages (not with the same reference as the one failing upload) that are uploaded with revisions on the server?

Shouldn't be a problem unless something is corrupted. Have you tried uploading different packages?

[uilianries]

Yesterday I tried with Artifactory 6.15.0 and it's working.

@RadMap Please, prefer Conan issues than Stackoverflow to get support. Here you will find Conan Devs and more people from the community.

[RadMap]

@lasote It is a general problem for all packages and at least two more users (if not all) trying to upload to the same Artifactory server so probably a server-side problem?

@uilianries I will keep this in mind. I was not sure though if this was the right forum for Artifactory issues. Hence the Stackoverflow question taged with [Artifactory].

Many thanks!

[lasote]

This is very very unusual. Would be possible to take a look at the artifactory logs just in case something weird appear during the upload attempt?

[uilianries]

@RadMap I see, thanks for the explanation

[RadMap]

@lasote If you would like to have a look at them, that would be fantastic. I will have to reserve some time with our devops team since I need their help to get the logs. Difficult to say when they will be available. Ill post back here as soon as I have anythng new to report.

Thanks!

[RadMap]

@lasote Do you have any clue of what log(s) we are talking about. Apparently Artifactory produces a very large number of logs so more info seems to be needed. Thanks!

[lasote]

I only use: Admin > Advanced > System logs.
https://URL/webapp/#/admin/advanced/system_logs

[RadMap]

I have collected the logs after a failed upoad when CONAN_REVISIONS_ENABLED=0 and attache them here.

request.log
access.log
artifactory.log

I cannot see any other problem than that the server responds with a 404. The contents of the artifactory.log shows no output related to my actions (only what i believe is normal artifactory work) and i guess that is in alignment to nothing being uploaded.

[lasote]

Sorry I've been OOO last week. Agree, nothing on the Artifactory logs. Which conan version are you using? Thanks

[RadMap]

Currently we are running Conan version: 1.20.3 on Windows 10 and a Linux distribution. We have had this problem ever since we started using conan+artifacory (since the summer) so have reasons to believe that this happens for earlier conan versions + Artifactory 6.13.1 aswell. Trying to make our devops guys try out artifactory 6.15 to see if that helps as mentioned earlier in this therad.

[lasote]

Currently we are running Conan version: 1.20.3 on Windows 10 and a Linux distribution.

Sorry, where are you running Artifactory? on Windows 10 on a Linux subsystem?

[RadMap]

Sorry for the confusion.
Artifactory is running on Linux CentOS 7. Conan clients running on Windows 10, Ubuntu and CentOS.
Many thanks for keeping an interest in this!

[lasote]

Could you try to:

• Make sure that revisions are not enabled: SET CONAN_REVISIONS_ENABLED=0
• Clear any user authentication data in the client running conan user -c
• Log again with the server with conan user foo -p bar -r myremote
• Try to upload again.

[RadMap]

I have tried the procedure described in the previous comment on a Windows 10 machine and I'm happy to say that it seems to work (no logs to present yet though)! What I think has happened is that we (unknkowingly and independenty) have used an anonymos user that has been accepted by the artifactory server only when we have had revisions turned ON. The 404 answer from the server when revisions is turned OFF is however difficult to understand. I will continue the investigation to make sure that it works from conan on Linux aswell and report back. Hopefully it will work. In the mean time: Many thanks for all the help! The quality feedback from the community is a strong argument when I try to spread the use of conan in our company.

Edit: Works on Linux CentOS 7 aswell.

[lasote]

Fantastic! I suspected something like that related to the authentication, that 404 made no sense. I think it might be related to a new token authentication that was introduced in 6.13, but still not sure how. We will try to reproduce. Thanks!

[memsharded]

Hi @RadMap

We haven't been able to reproduce this on our side. So if things are working good for you, I am closing this issue at the moment. Feel free to reopen it or file a new issue if something similar happens again.

Thanks to you for all the feedback, help and support! Cheers!