Conan: [bug] Remote searches for channel-promoted packages are broken since 1.30.0

[jasal82]

It seems that PR #7673 broke search queries which used to work in Conan < 1.30.0. We use the copy feature in Artifactory UI to promote packages from master to release channel. In previous versions Conan sent queries like

https://server/artifactory/api/conan/euler-conan-local/v2/conans/search?q=euler_threading&ignorecase=False

These returned all of the available packages with that name and filtered for channel on the client side. Now I see queries like

https://server/artifactory/api/conan/euler-conan-local/v2/conans/search?q=euler_threading%2F%2A%40sick%2Frelease&ignorecase=False

This works fine for packages which were uploaded with that exact user/channel combination. But when a package is promoted from channel master to release it is not part of the result for the query containing release in the expression. When I change the query to search for master (the original channel before promotion) the package is listed in the results but with the new (promoted) channel.

We're on Artifactory 6.21.0.

[memsharded]

Hi @jasal82

Just to clarify, are you using the private http api directly, or that is only the debugging information? Could you please clarify which Conan commands/recipes are you using that will trigger this issue?

Also, please take into account that there was a bug related to this change, that has been fixed in Conan 1.30.1.

[jasal82]

The queries are debugging traces from a simple conan install .. --build missing --update. It happens also with Conan 1.30.1.

[jasal82]

It should be possible to reproduce this easily with any recipe by promoting its channel via Artifactory UI or JFrog CLI and then searching with a query similar to the above.

[jasal82]

I think it's actually an Artifactory problem which was uncovered by the changes in the PR.

[memsharded]

If the command is conan install .. --build missing --update, then the search will be invoked by a version range. Such version range could be:

requires = "pkg/[>1.0 <2.0]@user/channel"
# or
requires = "otherpkg/[>1.0 <2.0]"

In the first case, the version range should only resolve to packages that have the exact user/channel, and in the second case it should only match packages without user/channel. It if was matching other packages, it was a bug that was fixed.

I am trying with Artifactory 6.8, it seems to be fine:

$  conan install "hello/[>0.0]@james/testing" -r=local 
# and
$ conan install "hello/[>0.0]@" -r=local

Both manage to install the correct package with and without user/channel.

[jasal82]

Did you try to promote to another channel in Artifactory? It's broken after promotion.

[jasal82]

Here are some simple steps to reproduce:

• Run conan create . user/A
• Upload to remote conan upload mypkg/[*]@user/A --all -r remote
• Copy to channel B on remote
• Run conan install mypkg/[*]@user/B -r remote <- fails in Conan >= 1.30.0

[memsharded]

Uhm, yes that could be a bug, but in this case, seems totally unrelated to the "search" issue, as no search is involved in those commands.

I am trying to do a copy directly in Artifactory 6.8, and I am getting:

Target path 'conan-local/james/hello/0.1/stable/export/conan_sources.tgz' for Conan files must match the source path'conan-local/james/hello/0.1/testing/export/conan_sources.tgz' (excluding repo)
.... (x 6 files)

This could be a "copy" bug in 6.8, I need to test with 6.10 then.

[jasal82]

I see, maybe the title was misleading. But the server query changed in the PR is also used for finding the packages to install, isn't it? So I think it is related to that other issue.

[jasal82]

See the updated steps above, that's what we're actually doing. With that version expression in place there should also be a search query involved.

[memsharded]

I have been able to reproduce in Artifactory 6.10.
It is true, doing a "copy" operation directly in Artifactory, which will succeed, but then it will fail to retrieve. But then it also happens when I upload to another channel from the client. I am investigating.

[memsharded]

This is still a bug in Artifactory 7.5

For some reason, packages that are promoted via a "copy" directly in Artifactory do not show in the search, and thus cannot show correctly in version-ranges resolution.

[memsharded]

Reported in https://www.jfrog.com/jira/browse/RTFACT-23580. @czoido please track it.

Seems problematic. What was fixed in Conan is totally correct, it was a bug, and fixing it is totally necessary to apply filters in Artifactory remote repositories (to ConanCenter) not only because of long delays, but it is also a security issue. But it has uncovered this Artifactory bug.

The main workaround I could suggest at this moment is to do the promotion via the client, would be a bit slower, but should work:

$ conan download pkg/version@user/testing -r=myremote
$ conan copy pkg/version@user/testing  user/stable
$ conan upload pkg/version@user/stable -r=myremote --all

[jasal82]

Thanks for your efforts. Promotion via the client is something that I'd like to avoid. It unpacks and repacks the archives which is likely to introduce unwanted changes to the packages. For example you must not use a Windows client to promote Linux packages because it would break symlinks and execution flags etc.

[memsharded]

This problem also happens when copying to a different Artifactory repo, not only for copies within the same repo (I am trying to find viable workarounds).

[czoido]

This problem also happens when copying to a different Artifactory repo, not only for copies within the same repo (I am trying to find viable workarounds).

It looks like at least if you don't change the channel, after copying to another repo it works. Maybe that could open a feasible solution...

[memsharded]

Thanks for your efforts. Promotion via the client is something that I'd like to avoid. It unpacks and repacks the archives which is likely to introduce unwanted changes to the packages. For example you must not use a Windows client to promote Linux packages because it would break symlinks and execution flags etc.

We have put into the roadmap fixing this, and letting clients download-upload without re-zipping at all, so this operation is idempotent irrespective of the platform that is executed. It will take a few weeks, so lets keep exploring other possible workaround (check slack when possible too please)

[memsharded]

Hi @jasal82

https://github.com/conan-io/conan/pull/7938 has finalized the implementation of the idempotent download + upload, will be available in Conan 1.31. Can be tested now from the develop branch (or PyPI testing). This will allow the promotions in the client side without risk of breaking the compressed .tgz. I hope this is a viable workaround until the Artifactory bug is fixed.

[memsharded]

This is still pending, waiting for https://www.jfrog.com/jira/browse/RTFACT-23580., so there is little we can do on our side, I am moving to next release.

[jasal82]

@memsharded The RTFACT issue seems to be resolved, but there's no fix version and nothing. Do you know if the fix is already available?

[memsharded]

https://www.jfrog.com/jira/browse/RTFACT-23580 has been resolved and closed, so I guess this will be released in next Artifactory version (7.13).

There is nothing more that we can do here in the client side, so I am going to close this issue now. Please create a new one if after upgrading to next Artifactory 7.13 the problem persists. Many thanks!