Conan: [bug] Conan is not able to download big packages

Created on 4 Sep 2019  路  26Comments  路  Source: conan-io/conan

To help us debug your issue please explain:

Conan Version 1.12 and 1.18.2

According "Christian Charles" from Conan channel on Slack:

Hi all,
I'm encountering a weird issue. With a big package (3.5GB), the download fails on my PC every time at the same size:

Downloading conan_sources.tgz
[===============                                   ] 1.07GB/3.56GB    ERROR: Download failed, check server, possibly try again
Transfer interrupted before complete: 1093543780 < 3556596602
Waiting 0 seconds to retry...
[===============                                   ] 1.07GB/3.56GB    ERROR: Download failed, check server, possibly try again
Transfer interrupted before complete: 1092076956 < 3556596602
Waiting 0 seconds to retry...
[===============                                   ] 1.07GB/3.56GB    ERROR: Download failed, check server, possibly try again
Transfer interrupted before complete: 1092712844 < 3556596602. [Remote: MYREMOTE]

More detailed log:

DEBUG :rest_client_common.py[136]: REST: get: http://artifactory.lyon.ts3d.lan/artifactory/api/conan/HolyGrail/v1/conans/ODA/2019/HE/stable/download_urls [2019-09-04 19:10:25,603]
DEBUG :uploader_downloader.py[209]: DOWNLOAD: http://artifactory.lyon.ts3d.lan/artifactory/api/conan/HolyGrail/v1/files/HE/ODA/2019/stable/export/conan_sources.tgz [2019-09-04 19:10:25,618]
Downloading conan_sources.tgz:  31%|###       | 1.02G/3.31G [01:33<03:50, 10.7MB/s]DEBUG :uploader_downloader.py[215]: <class 'conans.errors.ConanException'> [2019-09-04 19:11:59,454]
DEBUG :uploader_downloader.py[216]: Traceback (most recent call last):
                                      File "conan\conans\client\rest\uploader_downloader.py", line 210, in _download_file
                                      File "conan\conans\client\rest\uploader_downloader.py", line 286, in _download_data
                                    conans.errors.ConanException: Transfer interrupted before complete: 1093767356 < 3556596602
                                     [2019-09-04 19:11:59,454]
ERROR: Download failed, check server, possibly try again
Transfer interrupted before complete: 1093767356 < 3556596602
Waiting 0 seconds to retry...
DEBUG :uploader_downloader.py[209]: DOWNLOAD: http://artifactory.lyon.ts3d.lan/artifactory/api/conan/HolyGrail/v1/files/HE/ODA/2019/stable/export/conan_sources.tgz [2019-09-04 19:11:59,508]
Downloading conan_sources.tgz:  31%|###       | 1.02G/3.31G [01:50<03:50, 10.7MB/s]DEBUG :uploader_downloader.py[215]: <class 'conans.errors.ConanException'> [2019-09-04 19:13:31,887]
DEBUG :uploader_downloader.py[216]: Traceback (most recent call last):27, 11.9MB/s]
                                      File "conan\conans\client\rest\uploader_downloader.py", line 210, in _download_file
                                      File "conan\conans\client\rest\uploader_downloader.py", line 286, in _download_data
                                    conans.errors.ConanException: Transfer interrupted before complete: 1093793916 < 3556596602
                                     [2019-09-04 19:13:31,896]
ERROR: Download failed, check server, possibly try again
Transfer interrupted before complete: 1093793916 < 3556596602
Waiting 0 seconds to retry...
DEBUG :uploader_downloader.py[209]: DOWNLOAD: http://artifactory.lyon.ts3d.lan/artifactory/api/conan/HolyGrail/v1/files/HE/ODA/2019/stable/export/conan_sources.tgz [2019-09-04 19:13:31,910]
                                                                                   DEBUG :uploader_downloader.py[215]: <class 'conans.errors.ConanException'> [2019-09-04 19:15:04,181]
DEBUG :uploader_downloader.py[216]: Traceback (most recent call last):27, 11.9MB/s]
                                      File "conan\conans\client\rest\uploader_downloader.py", line 210, in _download_file
                                      File "conan\conans\client\rest\uploader_downloader.py", line 286, in _download_data
                                    conans.errors.ConanException: Transfer interrupted before complete: 1092720716 < 3556596602
                                     [2019-09-04 19:15:04,182]
ERROR: Download failed, check server, possibly try again
Transfer interrupted before complete: 1092720716 < 3556596602. [Remote: MYREMOTE]
Downloading conan_sources.tgz:  31%|###       | 1.02G/3.31G [04:38<10:27, 3.93MB/s]
Exception ignored in: <bound method tqdm.__del__ of Downloading conan_sources.tgz:  31%|###       | 1.02G/3.31G [03:04<03:27, 11.9MB/s]>
Traceback (most recent call last):%|###       | 1.02G/3.31G [01:32<03:28, 11.8MB/s]
  File "site-packages\tqdm\_tqdm.py", line 931, in __del__
  File "site-packages\tqdm\_tqdm.py", line 1133, in close
  File "site-packages\tqdm\_tqdm.py", line 496, in _decr_instances
  File "site-packages\tqdm\_monitor.py", line 52, in exit
  File "threading.py", line 1053, in join
RuntimeError: cannot join current thread

The server in this case is Artifactory CPP CE, not sure about the version.

Possible causes:

  • Artifactory limitation, the file to be download is 3.31.GB

  • tqdm bug or side-effect from connection interruption

  • [ ] I've read the CONTRIBUTING guide.

  • [x] I've specified the Conan version, operating system version and any tool that can be relevant.
  • [x] I've explained the steps to reproduce the error or the motivation/use case of the question/suggestion.
picture uilianries

All 26 comments

As I don't have a big package, I'll use https://speed.hetzner.de/ to download the 5GB file and create a dummy package.

uilianries picture uilianries on 4 Sep 2019

I created a small gist to reproduce the steps, no errors at first try:

https://gist.github.com/uilianries/a2698618c6367cce4943451156c6baa1

Even killing the server the result is not the same. Maybe it's related to tqdm version

I'm running Artifactory 6.12.1

uilianries picture uilianries on 4 Sep 2019

My version of tqdm

C:\Users\christian>pip show tqdm Name: tqdm Version: 4.35.0 Summary: Fast, Extensible Progress Meter Home-page: https://github.com/tqdm/tqdm Author: None Author-email: None License: MPLv2.0, MIT Licences Location: c:\users\christian\appdata\local\programs\python\python37-32\lib\site-packages Requires: Required-by: conan

le-kriss picture le-kriss on 5 Sep 2019

@uilianries could you give it another try with that version of tqdm?

Please @le-kriss indicate the version of Artifactory if possible so we can try to reproduce it better. Thanks!

danimtb picture danimtb on 5 Sep 2019

Hi, other colleagues tried again this morning and it worked nice for them.
Only my PC fails to get the package.

Server version:
Artifactory Version 6.4.1

le-kriss picture le-kriss on 5 Sep 2019

Could you please indicate if you are running windows and the python version too? We had similar and weird issues like this in the past (https://github.com/conan-io/conan/issues/5422) that were a bit difficult to reproduce

danimtb picture danimtb on 5 Sep 2019

Yes, sometimes things like memory limits for python apps in a given machine, that made things fail. Difficult to debug, indeed, not sure what could we do to investigate this. Which python version, with which architecture (32/64)? Was it a clean python environment or used for other things? What is the OS running in the client? same for your colleagues?

memsharded picture memsharded on 5 Sep 2019

Windows version 1903 build 18362.295
Python 3.7.4 (tags/v3.7.4:e09359112e, Jul 8 2019, 19:29:22) [MSC v.1916 32 bit (Intel)] on win32
pip 19.0.3
Conan version 1.18.2

le-kriss picture le-kriss on 5 Sep 2019

Good, I see 32 bit (Intel). Could you please try again with a 64 bits Python 3.7?

memsharded picture memsharded on 5 Sep 2019

Yes, sometimes things like memory limits for python apps in a given machine, that made things fail.

Oh, my machine has 32GB, I think I'll not be able to reproduce the same error. Maybe I could set some memory limitation.

uilianries picture uilianries on 5 Sep 2019

https://airbrake.io/blog/python-exception-handling/memoryerror

"The memory for 32bits python is very low". I have been reported that 64bits python have also some kind of issue with some dependency while installing Conan itself, this needs to be checked, please @danimtb or @uilianries have a look when possible.

memsharded picture memsharded on 5 Sep 2019
👍1

Sure, I'll try on my machine. Now the scenario is clear to be reproduced.

uilianries picture uilianries on 5 Sep 2019

I tried again using a different approach, but I wasn't able to reproduce.

Environment

  • Windows 10
  • Python 3.7.4 (tags/v3.7.4:e09359112e, Jul 8 2019, 19:29:22) [MSC v.1916 32 bit (Intel)] on win32
  • tqdm Version: 4.35.0
  • Conan version 1.18.2
  • Linux (Artifactory 6.11.1)

The idea is: Run Linux with Artifactory to store the big package (3.3GB) and download it using Windows.

However, my Windows machine has 32GB RAM and 6 cores, so I limited the process by software:

cprocgov --maxmem 2G --cpu 1 "conan install big/0.1.0@uilianries/testing -r artifactory --build"
Process Governor v2.3.19057.1 - sets limits on your processes
Copyright (C) 2019 Sebastian Solnica (lowleveldesign.org)

CPU affinity mask:                      0x1
Maximum committed memory (MB):          2,048
Process user-time execution limit (ms): (not set)
Job user-time execution limit (ms):     (not set)
Clock-time execution limit (ms):        (not set)

Press Ctrl-C to end execution without terminating the process.

Configuration:
[settings]
arch=x86_64
arch_build=x86_64
build_type=Release
compiler=Visual Studio
compiler.runtime=MD
compiler.version=16
os=Windows
os_build=Windows
[options]
[build_requires]
[env]

big/0.1.0@uilianries/testing: Retrieving from server 'artifactory'
big/0.1.0@uilianries/testing: Trying with 'artifactory'...
Downloading conanmanifest.txt: 100%|##########| 115/115 [00:00<00:00, 57.3kB/s]
Downloading conanfile.py: 100%|##########| 246/246 [00:00<00:00, 244kB/s]
big/0.1.0@uilianries/testing: Downloaded recipe revision 6484e7680042383144f93e8429e3af46
big/0.1.0@uilianries/testing: Forced build from source
Installing package: big/0.1.0@uilianries/testing
Requirements
    big/0.1.0@uilianries/testing from 'artifactory' - Downloaded
Packages
    big/0.1.0@uilianries/testing:5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9 - Build

Downloading conan_sources.tgz: 100%|##########| 3.22G/3.22G [35:06<00:00, 1.64MB/s]
Decompressing conan_sources.tgz: 100%|##########| 3.22G/3.22G [00:11<00:00, 298MB/s]
big/0.1.0@uilianries/testing: Configuring sources in C:\Users\uilia\.conan\data\big\0.1.0\uilianries\testing\source
big/0.1.0@uilianries/testing: Building your package in C:\Users\uilia\.conan\data\big\0.1.0\uilianries\testing\build\5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9
big/0.1.0@uilianries/testing: Generator txt created conanbuildinfo.txt
big/0.1.0@uilianries/testing: Calling build()
big/0.1.0@uilianries/testing: WARN: This conanfile has no build step
big/0.1.0@uilianries/testing: Package '5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9' built
big/0.1.0@uilianries/testing: Build folder C:\Users\uilia\.conan\data\big\0.1.0\uilianries\testing\build\5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9
big/0.1.0@uilianries/testing: Generated conaninfo.txt
big/0.1.0@uilianries/testing: Generated conanbuildinfo.txt
big/0.1.0@uilianries/testing: Generating the package
big/0.1.0@uilianries/testing: Package folder C:\Users\uilia\.conan\data\big\0.1.0\uilianries\testing\package\5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9
big/0.1.0@uilianries/testing: Calling package()
big/0.1.0@uilianries/testing package(): Packaged 1 '.txt' file: file.txt
big/0.1.0@uilianries/testing: Package '5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9' created
big/0.1.0@uilianries/testing: Created package revision da9e2d3fb470069d36235705dc0e7205

To check all the process, I watched the Windows task manager, the Python process consumed 17MB and 0.2% CPU level.

uilianries picture uilianries on 9 Sep 2019

I checked the python 64 bit installation of conan and worked without issue:

$ python
Python 3.7.4 (tags/v3.7.4:e09359112e, Jul  8 2019, 20:34:20) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> quit()

$ conan -v
Conan version 1.18.3
danimtb picture danimtb on 12 Sep 2019

@le-kriss could you try the python 64 bit version? maybe it improves as indicated above by @memsharded. Currently, we are not able to reproduce the issue 馃槙

danimtb picture danimtb on 12 Sep 2019

I've been trying to reproduce an issue with both python 32-bit and python 64-bit, but no luck as well

SSE4 picture SSE4 on 18 Sep 2019
👍1

So I re-install my python. My version is 3.7.4 :
Python 3.7.4 (tags/v3.7.4:e09359112e, Jul 8 2019, 20:34:20) [MSC v.1916 64 bit (AMD64)] on win32
This win32 is weird since I use the installer python-3.7.4-amd64.exe provided https://www.python.org/ftp/python/3.7.4/python-3.7.4.exe

We are deploying this 3rdparty package on Windows and Linux.
In my office I have 3 people having issue with the same error : failing to download it a 1GB/3.3GB both on Windows and Linux.
On the other hand, I have 15 jenkins slaves having no issue and same for 15 developers.
On my pc, I had 2 successes other 30+ trials.

le-kriss picture le-kriss on 18 Sep 2019
😕1

@le-kriss I'm curious about your Artifactoy. Is it installed and running directly on Linux OS, or is there an extra layer, like Docker or VM? Did you check for any network limitation?

I remember years ago in a past company, when I was not able to download big files due network proxy rules.

uilianries picture uilianries on 18 Sep 2019

So the server is a Docker instance. Its host is Ubuntu 16.4 server. The host is also a swarm manager, it may not be relevant.
Docker image: docker.bintray.io/jfrog/artifactory-cpp-ce:6.4.1
There is no proxy.
What else do you need ?

le-kriss picture le-kriss on 18 Sep 2019

On my PC (dell laptop), with the same custom Docker image we use on our build slaves, I reproduce the same issue that the one in Windows.
I tried to disable the firewall with no luck.

le-kriss picture le-kriss on 18 Sep 2019

Docker image: docker.bintray.io/jfrog/artifactory-cpp-ce:6.4.1

Interesting, I've used the same Docker image + Linux Mint (similar to Ubuntu).

uilianries picture uilianries on 18 Sep 2019

Hi, I'd like to solve this issue as more and more people encounter this problem here. It is still random, but most people here managed to download the package, even our build slaves, but few of them are still failing to get it. People start to share their cache to avoid the download, and that's not a valid solution.
So, what can I do to help you more ?
Artifactory logs ?
Conan error log ? Here is my last log with CONAN_LOGGING_LEVEL=10

C:\WORK>conan download ODA/2019@HE/stable -r HOLYGRAIL
DEBUG :conan_api.py   [154]: INIT: Using config 'C:\Users\christian\.conan\conan.conf' [2019-10-14 11:21:50,653]
DEBUG :tracer.py      [155]: CONAN_API: download(reference=ODA/2019@HE/stable,packages=None,remote_name=HOLYGRAIL,recipe=False) [2019-10-14 11:21:50,656]
DEBUG :rest_client_common.py[110]: REST: ping: http://artifactory.lyon.ts3d.lan/artifactory/api/conan/HolyGrail/v1/ping [2019-10-14 11:21:50,773]
DEBUG :rest_client.py [37]: REST: Cached capabilities for the remote: [] [2019-10-14 11:21:50,815]
DEBUG :rest_client_common.py[136]: REST: get: http://artifactory.lyon.ts3d.lan/artifactory/api/conan/HolyGrail/v1/conans/ODA/2019/HE/stable/download_urls [2019-10-14 11:21:50,816]
DEBUG :uploader_downloader.py[209]: DOWNLOAD: http://artifactory.lyon.ts3d.lan/artifactory/api/conan/HolyGrail/v1/files/HE/ODA/2019/stable/export/conanmanifest.txt [2019-10-14 11:21:50,849]
Downloading conanmanifest.txt: 100%|##########| 1.60M/1.60M [00:00<00:00, 2.89MB/s]
DEBUG :uploader_downloader.py[209]: DOWNLOAD: http://artifactory.lyon.ts3d.lan/artifactory/api/conan/HolyGrail/v1/files/HE/ODA/2019/stable/export/conanfile.py [2019-10-14 11:21:51,441]
Downloading conanfile.py: 100%|##########| 3.07k/3.07k [00:00<00:00, 3.12MB/s]
DEBUG :rest_client_common.py[136]: REST: get: http://artifactory.lyon.ts3d.lan/artifactory/api/conan/HolyGrail/v1/conans/ODA/2019/HE/stable/download_urls [2019-10-14 11:21:51,462]
DEBUG :uploader_downloader.py[209]: DOWNLOAD: http://artifactory.lyon.ts3d.lan/artifactory/api/conan/HolyGrail/v1/files/HE/ODA/2019/stable/export/conan_sources.tgz [2019-10-14 11:21:51,475]
Downloading conan_sources.tgz:  31%|###       | 1.01G/3.31G [01:37<03:27, 11.9MB/s]DEBUG :uploader_downloader.py[215]: <class 'conans.errors.ConanException'> [2019-10-14 11:23:29,157]
DEBUG :uploader_downloader.py[216]: Traceback (most recent call last):
                                      File "c:\users\christian\appdata\local\programs\python\python37\lib\site-packages\conans\client\rest\uploader_downloader.py", line 210, in _download_file
                                        data = self._download_data(response, file_path)
                                      File "c:\users\christian\appdata\local\programs\python\python37\lib\site-packages\conans\client\rest\uploader_downloader.py", line 286, in _download_data
                                        "complete: %s < %s" % (dl_size, total_length))
                                      conans.errors.ConanException: Transfer interrupted before complete: 1090156396 < 3556596602
                                     [2019-10-14 11:23:29,157]
ERROR: Download failed, check server, possibly try again
Transfer interrupted before complete: 1090156396 < 3556596602
Waiting 0 seconds to retry...
DEBUG :uploader_downloader.py[209]: DOWNLOAD: http://artifactory.lyon.ts3d.lan/artifactory/api/conan/HolyGrail/v1/files/HE/ODA/2019/stable/export/conan_sources.tgz [2019-10-14 11:23:29,176]
Downloading conan_sources.tgz:  31%|###       | 1.02G/3.31G [01:50<03:27, 11.9MB/s]DEBUG :uploader_downloader.py[215]: <class 'conans.errors.ConanException'> [2019-10-14 11:25:04,699]
DEBUG :uploader_downloader.py[216]: Traceback (most recent call last):28, 11.8MB/s]
                                      File "c:\users\christian\appdata\local\programs\python\python37\lib\site-packages\conans\client\rest\uploader_downloader.py", line 210, in _download_file
                                        data = self._download_data(response, file_path)
                                      File "c:\users\christian\appdata\local\programs\python\python37\lib\site-packages\conans\client\rest\uploader_downloader.py", line 286, in _download_data
                                        "complete: %s < %s" % (dl_size, total_length))
                                    conans.errors.ConanException: Transfer interrupted before complete: 1089457908 < 3556596602
                                     [2019-10-14 11:25:04,700]
ERROR: Download failed, check server, possibly try again
Transfer interrupted before complete: 1089457908 < 3556596602
Waiting 0 seconds to retry...
DEBUG :uploader_downloader.py[209]: DOWNLOAD: http://artifactory.lyon.ts3d.lan/artifactory/api/conan/HolyGrail/v1/files/HE/ODA/2019/stable/export/conan_sources.tgz [2019-10-14 11:25:04,723]
DEBUG :uploader_downloader.py[215]: <class 'conans.errors.ConanException'> [2019-10-14 11:26:41,178]
DEBUG :uploader_downloader.py[216]: Traceback (most recent call last):28, 11.8MB/s]
                                      File "c:\users\christian\appdata\local\programs\python\python37\lib\site-packages\conans\client\rest\uploader_downloader.py", line 210, in _download_file
                                        data = self._download_data(response, file_path)
                                      File "c:\users\christian\appdata\local\programs\python\python37\lib\site-packages\conans\client\rest\uploader_downloader.py", line 286, in _download_data
                                        "complete: %s < %s" % (dl_size, total_length))
                                    conans.errors.ConanException: Transfer interrupted before complete: 1088730684 < 3556596602
                                     [2019-10-14 11:26:41,179]
ERROR: Download failed, check server, possibly try again
Transfer interrupted before complete: 1088730684 < 3556596602. [Remote: HOLYGRAIL]
Downloading conan_sources.tgz:  31%|###       | 1.01G/3.31G [01:36<03:38, 11.3MB/s]
Downloading conan_sources.tgz:  31%|###       | 1.02G/3.31G [04:49<10:55, 3.76MB/s]
Downloading conan_sources.tgz:  31%|###       | 1.01G/3.31G [03:12<07:14, 5.67MB/s]

So as everything direct me to the server configuration rather than my client settings.
What can I provide you ? (in non devops speaking please, my knowledge is quite limited).
The artifactory logs are pretty lame and not helping, so where can I find the one telling to stop this download ?

Thanks

le-kriss picture le-kriss on 14 Oct 2019

Thanks for sharing some more info about this issue. We would like to take another look at it and see if we can manage to reproduce it consistently.

  • Are you experiencing this on different machines?
  • Does it happen only in Windows or in any other OS?
  • What is the smallest package size that triggers the error?
  • Are you launching simultaneous package downloads using the same Conan cache?
  • Does it happen always with the same package?

Thanks again

danimtb picture danimtb on 15 Oct 2019

In our company we experienced the same problem where conan is running on Python 3.6 (win32) downloading from our artifactory pro server.
Within our company network everything is fine. When we download the packages via VPN (slow) interrupts can happen at 1GB. So I think, this is network related.
I implemented resumable downloads in FileDownloader class which mitigates the problem. If you like I could work on it.

flashdagger picture flashdagger on 2 Apr 2020

Hi @flashdagger,

This issue has been inactive for a while but the information you are proving us is very helpful. I think the presumable downloads will be a very nice feature to mitigate this kind of issue. I have also experienced some latency using a VPN and it could be very annoying if de download of big packages is interrupted.

Please submit the changes to the FileDownloader in a Pull Request and we will try to work with you to include the feature in the next Conan releases 馃槃 Than you!!

danimtb picture danimtb on 3 Apr 2020

https://github.com/conan-io/conan/pull/6791 has implemented resumable downloads, so this should fix this issue. It will be released next 1.25.

memsharded picture memsharded on 28 Apr 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

Polibif picture Polibif  路  3Comments
theodelrieu picture theodelrieu  路  3Comments
Adnn picture Adnn  路  3Comments
uilianries picture uilianries  路  3Comments
claasd picture claasd  路  3Comments