Conan: [feature] parallel upload - control thread count from env / parameter

[ofiriluz]

Hi,

Would be much easier if i could control the thread count of the uploader, instead of having it hard coded to max 8 threads (we are having at least 14-20 packages being uploaded at a time due to alot of different unix distributions)
On client/cmd/uploader.py
something like:

if parallel_upload:
   parallel_count =  int(os.environ['CONAN_PARALLEL_COUNT']) or 8
   self._upload_thread_pool = ThreadPool(parallel_count)

(Obviously this doesnt really check for any errors)
Would be much appreciated if it could be added in some manner to the next patch :)

Thanks

• [x] I've read the CONTRIBUTING guide.

[jgsogo]

Hi, @ofiriluz! I'm sure that @czoido will be very pleased to read that people are actually using this feature without problems and they are willing to use even more workers. This is great feedback for us 👍

[memsharded]

In any case, IIRC, @czoido chose the 8 threads, because he tested a few approaches, and adding more didn't really speeded up the upload. But yes, I understand that other machines with more core might be more efficient with more threads.

I would suggest @czoido that instead of using and env-var, start using conan.conf and config only configuration internally. It has some advantages, like it is more reproducible, because conan.conf can also be synchronized with conan config install and can be conveniently modified with conan config, no need to pollute the global environment with other Conan things, and the internal structure of the codebase it is also a bit more elegant, because you get the config instead of just a global variable which is what the environment is (and less elegant to test too).

I will be pushing in Conan 2.0 to use config object in Conan internals to pass configuration, and that config might be mainly defined in conan.conf and not in env-vars (probably will discuss this in ConanDays)

[memsharded]

Parallel upload is controlled now by the cpu_count configuration (which also have a CONAN_CPU_COUNT env-var) since https://github.com/conan-io/conan/pull/7781 (Conan 1.30), so I think this can be closed now. Thanks!