Conan-center-index: [bug] Conan search returns timeout

[uilianries]

To help us debug your issue please explain:

Hi!

As Conan Center is growing, the time requested to list all recipes is no longer enough:

conan search -r conan-center
ERROR: HTTPSConnectionPool(host='conan.bintray.com', port=443): Read timed out. (read timeout=60.0). [Remote: conan-center]

As you can see, 60 seconds is not enough to retrieve all information.

However, if I update the timeout request by CONAN_TIMEOUT_REQUEST, I can receive the answer:

CONAN_REQUEST_TIMEOUT=360 conan search -r conan-center
Existing package recipes:

7z_installer/1.0@conan/stable
Args/5.0.1.4@igormironchik/stable
...

Usingtime it took 1m39.

As Conan Center is the official repository, I think search should work without timeouts. So Or Bintray is slow, Or Conan needs to update its timeout limit.

Conan Version: 1.16.1

• [x] I've read the CONTRIBUTING guide.
• [x] I've specified the Conan version, operating system version and any tool that can be relevant.
• [x] I've explained the steps to reproduce the error or the motivation/use case of the question/suggestion.

[lasote]

Reported to bintray. I don't think that incrementing the default timeout on the client is a sane solution, a list of recipes should be retrieved in milliseconds.

[uilianries]

Thanks!

[uilianries]

UPDATE:

Running Conan 1.17.0:

real 1m47,145s
user 0m0,861s
sys 0m0,139s

[Minimonium]

Reported by the users again. 😅

[uilianries]

Do you have a reference to the comment? just in case ...

[uilianries]

Anyway, I can confirm such behavior

$ conan --version
Conan version 1.22.2
$ time conan search -r conan-center
DEBUG :conan_api.py   [163]: INIT: Using config '/home/uilian/.conan/conan.conf' [2020-02-21 18:05:31,835]
DEBUG :tracer.py      [156]: CONAN_API: search_recipes(remote_name=conan-center,case_sensitive=False) [2020-02-21 18:05:31,836]
DEBUG :rest_client_common.py[156]: REST: ping: https://conan.bintray.com/v1/ping [2020-02-21 18:05:31,838]
DEBUG :rest_client.py [58]: REST: Cached capabilities for the remote: ['complex_search', 'checksum_deploy', 'revisions'] [2020-02-21 18:05:32,749]
DEBUG :rest_client_common.py[183]: REST: get: https://conan.bintray.com/v2/conans/search [2020-02-21 18:05:32,749]
ERROR: HTTPSConnectionPool(host='conan.bintray.com', port=443): Max retries exceeded with url: /v2/conans/search (Caused by ReadTimeoutError("HTTPSConnectionPool(host='conan.bintray.com', port=443): Read timed out. (read timeout=60.0)"))

Unable to connect to conan-center=https://conan.bintray.com
1. Make sure the remote is reachable or,
2. Disable it by using conan remote disable,
Then try again.

real    4m3,414s
user    0m0,660s
sys 0m0,083s

[Minimonium]

It was from the #include Discord server.
They were trying to use the

conan search liii* -r=conan-center

And getting timeout errors (the default is 60 seconds after all)

[uilianries]

Discord? That's cool, thanks for sharing! haha

[memsharded]

Hi all,

yes, we are aware of this limitation. It is in our todo list, but as it is a complex thing that requires changes in the backend it will take some time. Thanks very much for your patience.

[memsharded]

Moved to conan-center-index, as this is mostly about the ConanCenter, not the client side.

[kaihowl]

Probably observing the problem again right now. Details: https://twitter.com/kaihowl/status/1267750305443262465

[hewigovens]

I just installed and tried searching boost, it took 2min50s to get the results, Is someone working on this?

$ time sudo conan search "boost" --remote=conan-center
Existing package recipes:

boost/1.64.0@conan/stable
boost/1.65.1@conan/stable
boost/1.66.0@conan/stable
boost/1.67.0@conan/stable
boost/1.68.0@conan/stable
boost/1.69.0
boost/1.69.0@conan/stable
boost/1.70.0
boost/1.70.0@conan/stable
boost/1.71.0
boost/1.71.0@conan/stable
boost/1.72.0
boost/1.73.0

real    2m50.146s
user    0m0.530s
sys 0m0.066s

BTW: we manage our c++ project internal dependencies (boost/json/gtest/protobuf) in a script, trying to find a usable package manager

[uilianries]

I can say https://conan.io/center is much much faster than the command line when looking for patterns.

[kaihowl]

@uilianries I observe this timeout during the resolution of version ranges as part of conan create. See https://gitlab.com/ptr-project/catch2_extended/-/jobs/589079001 for example.

 raise ConanConnectionError(("%s\n\nUnable to connect to %s=%s\n" +
 conans.errors.ConanConnectionError: HTTPSConnectionPool(host='conan.bintray.com', port=443): Max retries exceeded with url: /v1/conans/search?q=boost&ignorecase=False (Caused by ReadTimeoutError("HTTPSConnectionPool(host='conan.bintray.com', port=443): Read timed out. (read timeout=60.0)"))

The only way to remedy this seems increasing the timeout, pinning the version, or using lockfiles. I don't have any good logical reason for pinning or lockfiles as far as I can see. Increasing the timeout seems ok. Yet, the performance is surprisingly slow for a simple query. Is this something that should be reported here or is there a better place (or even an existing ticket) for this?

[uilianries]

@kaihowl Bintray reported an internal problem hours ago: https://status.bintray.com/

Please, try again, it should be solved now.

[kaihowl]

{"_action": "REST_API_CALL", "duration": 117.91095399856567, "headers": {"User-Agent": "Conan/1.25.2 (Python 3.7.7) python-requests/2.22.0", "X-Client-Anonymous-Id": "**********", "X-Client-Id": ""}, "met
hod": "GET", "time": 1591800609.898228, "url": "https://conan.bintray.com/v2/conans/search?q=boost&ignorecase=False"}

Still observing the issue. For "smaller" packages like sqlite3 I see the same problem, yet on a smaller scale:

{"_action": "REST_API_CALL", "duration": 20.84811496734619, "headers": {"User-Agent": "Conan/1.25.2 (Python 3.7.7) python-requests/2.22.0", "X-Client-Anonymous-Id": "**********", "X-Client-Id": ""}, "method": "GET", "time": 1591800488.0407581, "url": "https://conan.bintray.com/v2/conans/search?q=sqlite3&ignorecase=False"}

I don't think the reported outage correlates with this problem. I continually observe this problem since my original report, @uilianries.

[kuklinistvan]

I'm experiencing this too :(

[thejohnfreeman]

4 minutes seems like too long.

$ time conan search "boost/*" -r=conan-center
Existing package recipes:

boost/1.64.0@conan/stable
boost/1.65.1@conan/stable
boost/1.66.0@conan/stable
boost/1.67.0@conan/stable
boost/1.68.0@conan/stable
boost/1.69.0
boost/1.69.0@conan/stable
boost/1.70.0
boost/1.70.0@conan/stable
boost/1.71.0
boost/1.71.0@conan/stable
boost/1.72.0
boost/1.73.0

real    3m53.877s
user    0m0.483s
sys     0m0.060s

[DominikDeak]

Unfortunately, search performance is becoming a significant bottleneck and often fails in my case. Sample error message:

conan search "boost/*" --remote=all --raw

ERROR: HTTPSConnectionPool(host='conan.bintray.com', port=443): Max retries exceeded with url: /v1/conans/search?q=boost%2F%2A (Caused by ReadTimeoutError("HTTPSConnectionPool(host='conan.bintray.com', port=443): Read timed out. (read timeout=60.0)"))

Unable to connect to conan-center=https://conan.bintray.com
1. Make sure the remote is reachable or,
2. Disable it by using conan remote disable,
Then try again.

[sonman]

I am using a conan center on premise. Search takes sometimes minutes because local ConanCenter waits for bintray to answer. The conan search has therefore a not so good user experience.

[AndWass]

Also hit this issue.

It also happens if I have a range in conanfile.txt:

[requires]
boost/[>=1.73]

$> conan install -if build .
Configuration:
[settings]
arch=x86_64
arch_build=x86_64
build_type=Release
compiler=Visual Studio
compiler.runtime=MD
compiler.version=16
os=Windows
os_build=Windows
[options]
[build_requires]
[env]

ERROR: HTTPSConnectionPool(host='conan.bintray.com', port=443): Max retries exceeded with url: /v1/conans/search?q=boost%2F%2A&ignorecase=False (Caused by ReadTimeoutError("HTTPSConnectionPool(host='conan.bintray.com', port=443): Read timed out. (read timeout=60.0)"))

Unable to connect to conan-center=https://conan.bintray.com
1. Make sure the remote is reachable or,
2. Disable it by using conan remote disable,
Then try again.

Output of time conan search boost/* -r all:

ERROR: HTTPSConnectionPool(host='api.bintray.com', port=443): Max retries exceeded with url: /conan/conan/conan-center/v1/conans/search?q=boost%2F%2A (Caused by ReadTimeoutError("HTTPSConnectionPool(host='api.bintray.com', port=443): Read timed out. (read timeout=60.0)"))

Unable to connect to conan-center=https://api.bintray.com/conan/conan/conan-center
1. Make sure the remote is reachable or,
2. Disable it by using conan remote disable,
Then try again.

real    4m2.708s
user    0m0.015s
sys     0m0.000s