Vagrant: Fetching box on Atlas is painfully slow

[dqlopez]

Is it just me or anyone having the same experience, but fetching a box hosted on Atlas is too slow. In my case, I get the ubuntu/trusty64 base box.

After doing some reads online, seems that there are already several reported claims like this. Now I wonder why this isn't seriously taken look at. Why not provide alternative mirrors to lessen this wait time.

[matti]

I have a feeling that the download is capped at 500k/s based on multiple days.

@mitchellh back in the days you closed https://github.com/mitchellh/vagrant/issues/5319 with "I really can't explain this. Vagrant doesn't do anything during the subprocess Ruby-wise: it subprocesses to curl. It doesn't even do the download in Ruby. Perhaps wget is using multiple connections to download multiple parts? I really don't know, but unless we get more information I have to assume that Vagrant is fine here."

It's slow. Every single day. I can not get more than 500k/s.

[chrisroberts]

Hi! The biggest issue is being able to reliably reproduce the behavior. For example, my speed downloading this box (ubuntu/trusty64) sits around 2700k/s. It may be a location issue, a network issue, a local system issue, or many other variables that increase the difficulty in reproducing slow download speeds.

Also something to note is that this box is not hosted on Atlas but is externally hosted by ubuntu within cloud-images. You might also try pulling the hashicorp/precise64 box and see if your download speeds are consistent.

As with #5319, I have to also assume that Vagrant is fine here. If you can provide more information on what may be the cause of the slowness, or something that's reproducible and points to Vagrant being the cause, I will be happy to investigate more.

Cheers!

[matti]

@chrisroberts to me it looks like speeds in US are fine, but the speeds to Finland (and assuming other EU countries) are capped to 500k/s

[chrisroberts]

Related: #8640

[Pythic]

Same issue in NL. starts out fine then drops to 0 . Opened up a VPN connection to US East. Same issue.

Downloaded the box from my server which is hosted @ transip . No problems there. My home/office internet connection is Ziggo. Maybe they have an issue with the connection/route path

[sfunkho]

Same issue in Central part of U.S.

This appears to perhaps to only occur if the provider is an "Externally Hosted" url? The url vagrant shows to be downloading from startswith app.vagrantup.com; instead of, the actual url (In my case it's should be downloading from an s3 url).

[chrisroberts]

Adding the --debug flag will display 302 results and show actual download URL

[nbayle]

Same issue in West US. Local bandwidth is not the problem. This happens 100% of the time. It will spike at 1000k/s every once in awhile, but mostly sits at 150k/s or less, often less than 1k/s.

[scriptspry]

On macOS Sierra, using Vagrant 1.9.7, with curl installed through homebrew

curl 7.51.0 (x86_64-apple-darwin16.0) libcurl/7.51.0 SecureTransport zlib/1.2.8
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: AsynchDNS IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz UnixSockets

I did:

cp /opt/vagrant/embedded/bin/curl ~/curl.backup.bin
sudo ln -s `which curl` /opt/vagrant/embedded/bin/curl

After wards:

sudo rm /opt/vagrant/embedded/bin/curl
sudo mv ~/curl.backup.bin /opt/vagrant/embedded/bin/curl

And everything worked fine. However, I can't assert that this is in fact the problem after running the following test script[1] like 5 times. I couldn't concretely assert that the embedded version is slower. This makes me question if my original fix was what caused the download to happen quickly or was it a random variable out of context of my process?

Best thing is probably to let people try the swapping suggested in this post and post their own experience on whether or not the download sped up?

---

[1] - Test Script

echo "# curl from homebrew"
curl -r 1-10000000 -q --fail --location --max-redirs 10 --user-agent "Vagrant/1.9.7 (+https://www.vagrantup.com; ruby2.3.4)" --continue-at - --output /dev/null https://vagrantcloud.com/ubuntu/boxes/xenial64/versions/20170803.0.0/providers/virtualbox.box

echo "# curl packaged with vagrant"
/opt/vagrant/embedded/bin/curl -r 1-10000000 -q --fail --location --max-redirs 10 --user-agent "Vagrant/1.9.7 (+https://www.vagrantup.com; ruby2.3.4)" --continue-at - --output /dev/null https://vagrantcloud.com/ubuntu/boxes/xenial64/versions/20170803.0.0/providers/virtualbox.box --cacert /opt/vagrant/embedded/cacert.pem

[cawka]

I am experiencing the same issue with http://cdn.boxcutter.io/ubuntu/virtualbox5.1.26/ubuntu1604-17.0907.1.box from east coast. After connecting VPN @ west cost the download got way better.

I tried to convince DNS to give me a different CDN box, but failed. I suspect that it is somehow related.

[abhimanyu003]

Same issue here, I'm not getting speed higher than 200kb/s on 3Mb/s connection.

[cacarr-pdxweb]

This is still an infuriating problem -- I'm under 200kb/s downloading from: https://vagrantcloud.com/ubuntu/boxes/trusty64/versions/20171128.0.0/providers/virtualbox.box

[nelsonq]

same pain in B.C. Canada
100 - 500 kb/s in office ( high speed internet access via fiber )
50 - 250 kb/s at home ( 100Mbps internet access )

[kiklop74]

I can confirm that replacing vagrant's curl with OS curl increases download speed.
I use Ubuntu 16.04 64bit with Vagrant 2.0.1
It is silly that there is no option of configuring which curl to use.

[marcoflorian]

Progress: 0% (Rate: 3984/s, Estimated time remaining: 58:41:06)

Vagrant should change from "Development Environments Made Easy" to "Development Environments Made Very Slow"

[narration-sd]

One more on the pile -- and surely there are actually many others.

Please fix this. It's as important as any features -- and especially for persons around the world, quite evidently.

Here in metro or so California, it's taking literally hours to download one box -- and it's gotten worse. It used to take a half hour or so, which was bad enough....

By contrast, my actual connection is consistently 10Mbps...

[gusdecool]

Tested from Southeast Asia and this issue still existed 😞
Is there any temporary fix for this?
Knowing this issue is important to fix and already exist for a year

[kiklop74]

To all who experience this issue - If you use Mac or Linux replace the curl binary as explained in one of the comments in this issue. That helps

[omandotkom]

5% (Rate: 49822/s, Estimated time remaining: 4:17:36, i mean... for real ? i'm in 30 mbps fiber optic network. i think it's not a bug, but the server is suck.

[chrisroberts]

Hi everyone,

This issue overall is very complicated in that it is not easily reproducible and is very dependent on location, network health, and the actual download location of the target box. In many cases the box being downloaded is located on a third party system. One feature that was introduced in Vagrant a couple releases ago was the notification of redirect to show when boxes are being downloaded from a third party server.

There have also been claims that the embedded curl is downloading files slower than the system curl. I have not been successful in validating this claim and do not see a difference in download speed between my system curl and the embedded curl. However, as of Vagrant 2.0.4, the VAGRANT_PREFER_SYSTEM_BINS environment variable is available for all platforms. When enabled, Vagrant will use local system binaries if available instead of the embedded binaries. If you observe faster download speeds with your local curl binary, this provides a switch to easily enable Vagrant to use the local binary.

My apologies for the frustration this issue has caused. It is something I have investigated multiple times, but the number of variables involved makes this extremely difficult to get a valid reproduction. I have updated Vagrant's functionality to make the underlying cause more easily understandable where available (redirect notifications) or easier to work around. I am continuing to investigate our options for box downloads from Vagrant Cloud in different regions of the world (which have occasionally presented with issues). However, with box downloads from third party locations, there is little I am able to do to control their available bandwidth or any kind of throttling they may impose.

Cheers!

[sfunkho]

I appreciate your efforts to resolve this, and understand how vast a problem scope it has.

I have one question about 3rd party downloads that was never answered from my original comment.

All of my base boxes are hosted on Amazon S3, but when I download the box the cli shows vagrantup.com/some-more-url/ as the download url. Is the download being redirected through your servers instead of just downloading from S3?

[chrisroberts]

@sfunkho No, the downloads are not being proxied. They are directly downloaded from the remote location. Starting with Vagrant 2.0.3 a redirect notification is included when the box is not being downloaded directly from Vagrant Cloud. The output looks like this:

> vagrant box add ubuntu/trusty64
==> box: Loading metadata for box 'ubuntu/trusty64'
    box: URL: https://vagrantcloud.com/ubuntu/trusty64
==> box: Adding box 'ubuntu/trusty64' (v20180419.0.0) for provider: virtualbox
    box: Downloading: https://vagrantcloud.com/ubuntu/boxes/trusty64/versions/20180419.0.0/providers/virtualbox.box
    box: Download redirected to host: cloud-images.ubuntu.com

Cheers!

[axeloz]

Sorry for bringing that up again but I must confirm this. 7 estimated hours to download a ubuntu server box. Bandwith is very bad to Atlas.
Thanks

[Reiner030]

Same also here from Germany - different providers

• 1 GBit at datacentre (with x TBit fibre connection),
• 1 GBit fibre at office and
• 50 MBit DSL at home

all with latest vagrant 2.2.2 (2.2.1, 2.2.0 before) on

• Debian 9.6 (actual)
• MacOS Mavic (uname -r show 18.2.0) and different previous versions
• Windows 10 (actual)

host systems and I/we get always a very slow download of generic/debian9 "Hosted by Vagrant Cloud" is very slow while downloading from some US AWS bucket instead of using a CDN like Cloudfront which would speed up delivery worldwide and should additionally save delivery costs from your side (even you would'nt use all regions) .

[baipaiha]

Same thing here, from France, with vagrant 2.2.1:

Downloading: https://vagrantcloud.com/generic/boxes/ubuntu1804/versions/1.9.6/providers/virtualbox.box
Download redirected to host: vagrantcloud-files-production.s3.amazonaws.com
Progress: 57% (Rate: 445k/s, Estimated time remaining: 0:09:19)

[baipaiha]

Downloading from system curl or chromium/firefox browser is even worst:
https://vagrantcloud.com/generic/boxes/ubuntu1804/versions/1.9.6/providers/virtualbox.box
350 KB/s - 32.6 MB of 941 MB, 44 mins left

[Aaron-ML]

Wow this is miserable, trying to download two boxes on a Gbit business fiber line:

Bringing machine 'default' up with 'virtualbox' provider...
==> default: Box 'generic/centos7' could not be found. Attempting to find and install...
    default: Box Provider: virtualbox
    default: Box Version: >= 0
==> default: Loading metadata for box 'generic/centos7'
    default: URL: https://vagrantcloud.com/generic/centos7
==> default: Adding box 'generic/centos7' (v1.9.6) for provider: virtualbox
    default: Downloading: https://vagrantcloud.com/generic/boxes/centos7/versions/1.9.6/providers/virtualbox.box
    default: Download redirected to host: vagrantcloud-files-production.s3.amazonaws.com
    default: Progress: 15% (Rate: 109k/s, Estimated time remaining: 0:31:55)

Bringing machine 'default' up with 'virtualbox' provider...
==> default: Box 'scalefactory/centos6' could not be found. Attempting to find and install...
    default: Box Provider: virtualbox
    default: Box Version: >= 0
==> default: Loading metadata for box 'scalefactory/centos6'
    default: URL: https://vagrantcloud.com/scalefactory/centos6
==> default: Adding box 'scalefactory/centos6' (v1.2.224) for provider: virtualbox
    default: Downloading: https://vagrantcloud.com/scalefactory/boxes/centos6/versions/1.2.224/providers/virtualbox.box
    default: Download redirected to host: sfvagrantng.s3.eu-west-1.amazonaws.com
    default: Progress: 9% (Rate: 78078/s, Estimated time remaining: 1:18:0

I have confirmed it's not the connection on our side, using macOS Mojave Version 10.14.2 and Vagrant 2.2.1

[Reiner030]

I have also this problem since last year when you switched to new cloud storage system in DE/germany with GBit fibre connection and VDSL connections at home.

I think that's the common problem of S3 buckets which weren't performant for large files even their request rate was improved (for small objects) if many people request them.

Perhaps it's possible to setup Cloudfront ad Edge location for it? So far I see from AWS pricing the Cloudfront traffic is also cheaper than direct S3 traffic and is also suggested by several sites like e.g.:

• https://www.infoq.com/news/2018/10/amazon-s3-performance-increase
• https://cloudacademy.com/blog/optimize-amazon-s3-performance/
• https://dzone.com/articles/how-to-optimize-amazon-s3-performance

The curl process itself is fine as written - here my output of above speed compare check:

MacBook-Pro-2:vagrant reiner$ echo "# curl from homebrew"
# curl from homebrew
MacBook-Pro-2:vagrant reiner$ curl -r 1-10000000 -q --fail --location --max-redirs 10 --user-agent "Vagrant/1.9.7 (+https://www.vagrantup.com; ruby2.3.4)" --continue-at - --output /dev/null https://vagrantcloud.com/ubuntu/boxes/xenial64/versions/20170803.0.0/providers/virtualbox.box
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   152    0   152    0     0    195      0 --:--:-- --:--:-- --:--:--   194
100 9765k  100 9765k    0     0  3996k      0  0:00:02  0:00:02 --:--:-- 7130k

MacBook-Pro-2:vagrant reiner$ echo "# curl packaged with vagrant"
# curl packaged with vagrant
MacBook-Pro-2:vagrant reiner$ /opt/vagrant/embedded/bin/curl -r 1-10000000 -q --fail --location --max-redirs 10 --user-agent "Vagrant/1.9.7 (+https://www.vagrantup.com; ruby2.3.4)" --continue-at - --output /dev/null https://vagrantcloud.com/ubuntu/boxes/xenial64/versions/20170803.0.0/providers/virtualbox.box --cacert /opt/vagrant/embedded/cacert.pem
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   152    0   152    0     0    136      0 --:--:--  0:00:01 --:--:--   136
100 9765k  100 9765k    0     0  3447k      0  0:00:02  0:00:02 --:--:-- 8294k

EDIT/forgot to append:
I might also have to do with latency EU <=> US with ~ 100 ms.
This is not the worst case (I got years ago to asia > 400ms) but it would effect commonly also the download speed.

MacBook-Pro-2:~ reiner$ ping -c 3 vagrantcloud-files-production.s3.amazonaws.com
PING s3-1-w.amazonaws.com (52.216.129.155): 56 data bytes
64 bytes from 52.216.129.155: icmp_seq=0 ttl=44 time=106.726 ms
64 bytes from 52.216.129.155: icmp_seq=1 ttl=44 time=106.880 ms
64 bytes from 52.216.129.155: icmp_seq=2 ttl=44 time=106.771 ms

--- s3-1-w.amazonaws.com ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 106.726/106.792/106.880/0.065 ms

which you can also check on e.g. https://www.speedtest.net/ with different foreign country providers.

Better is mathematics proof of this problems which can be checked on this > 15 years old helper tool which I like very much:
https://www.speedguide.net/analyzer.php

In Windows 10, unless "TCP/IP Auto-Tuning" is disabled, only the Current TCP Window is displayed. Use the latest TCP Optimizer for tweaking.
You seem to be using Google Chrome. Note that Chrome can modify the TCP Window for sockets it creates under some OSes, and therefore servers may not get your OS-assigned RWIN value. FAQ
RWIN is a multiple of MSS
Other RWIN values that might work well with your current MTU/MSS: 
64240  (up to 2 Mbit lines, depending on latency. MSS * 44) <-- current value
128480 (1-5 Mbit lines, depending on latency. MSS * 44 * 2) 
256960 (2-15 Mbit lines, depending on latency. MSS * 44 * 2^2) 
513920 (10-30 Mbit lines, depending on latency. MSS * 44 * 2^3) 
1027840 (30-100 Mbit lines depending on latency. MSS * 44 * 2^4) 
bandwidth * delay product (Note this is not a speed test):

Your current TCP Window limits you to: 2570 kbps (321 KBytes/s) @ 200ms latency
Your current TCP Window limits you to: 1028 kbps (128 KBytes/s) @ 500ms latency

[somq]

Same problem here with a vagrant cloud organisation account and a private box:

default: Download redirected to host: vagrantcloud-files-production.s3.amazonaws.com
default: Progress: 3% (Rate: 118k/s, Estimated time remaining: 1:49:38))

We decided to give a go to vagrant cloud instead of a custom on-premise solution to accelerate our dev process but sadly it's slowing it for now. :(

[mikelkew]

Using a US-based VPN server made a huge difference for me. Using my normal Australian connection had an ETA of 33 hours. Using the VPN it downloaded in just 7 minutes!

[vouza]

since 3 years this issue exists... wow

[Reiner030]

It's much faster in last months (for me in Germany) ...
but maybe there are countries/regions which may be still slow ?

I knew Saltstack bugs which exists over 4 years (in community version) and even good paid NewRelic monitoring with stupid bugs where one has next Saturday it's 5th birthday (I plan to gratulate there, especially because my fix suggestion which works for me nice is ignored since over 2,5 years)

[GurudattaNA]

Came here to know that same issue not resolved from 3 years!

[jeshtan]

Using a US-based VPN server made a huge difference for me. Using my normal Australian connection had an ETA of 33 hours. Using the VPN it downloaded in just 7 minutes!

This definitely works, get the download complete less than 5min

[cosmicvibes]

I still have this problem. It used to work setting VAGRANT_PREFER_SYSTEM_BIN to get full speed but since upgrading to ubuntu 19.10 that makes no difference. The download gets redirected to vagrantcloud-files-production.s3.amazonaws.com and to be limited to around 400k/s (from the UK). When trying to download the url manually both curl and wget get stuck at that speed. Using Axel (which uses multiple simultaneous downloads) it will download at around 20mbps but that could be limited by my connection.

To me this gels with @Reiner030 findings so perhaps adding a concurrent downloader such as Axel (https://github.com/axel-download-accelerator/axel) as an optional alternative to curl could be a workaround. Although, in playing around with axel I hit a 429 rate limit, so maybe that would not work with the cdn setup used here.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.