Vagrant: Vagrant box downloads extremely slow

[spkane]

When relying on vagrant to download a box I frequently see connection speeds like this:

default: Downloading: http://boxes.example.com/vagrant/boxes/c6/packer_c6_2.5.2_virtualbox.box
default: Progress: 20% (Rate: 179k/s, Estimated time remaining: 0:41:37)

(Rate: 179k/s)

Yet when I use wget to the same URL:

wget http://boxes.example.com/vagrant/boxes/c6/packer_c6_2.5.2_virtualbox.box
--2015-02-10 09:52:12--  http://boxes.example.com/vagrant/boxes/c6/packer_c6_2.5.2_virtualbox.box
Resolving boxes.example.com... 10.1.0.17
Connecting to boxes.example.com|10.1.0.17|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 830674320 (792M) [text/plain]
Saving to: 'packer_c6_2.5.2_virtualbox.box'

packer_c6_2.5.2_virtualbox.bo   0%[                                                         ]   7.12M   696KB/s   eta 19m 50s

(Rate: 696KB/s) or often higher.

This particular example was pulled when on Wifi and connected to an IPSEC VPN.

[sethvargo]

Hi @spkane

Some boxes are hosted on Atlas and sometimes Atlas is just acting as a proxy to a user-hosted box. If you give more information on the specific box(es) you're downloading, we can do some research.

[spkane]

@sethvargo This box is actually a box I built using packer and it is hosted on a remote server. I'm trying to understand why the download in significantly slower using vagrant then using wget to the exact same URL.

[sethvargo]

@spkane sorry - I misread your original issue.

I would suspect (and maybe @mitchellh could elaborate more) a few things:

1. Ruby is slow and somehow throttling the subprocess
2. Wget is faster than curl (which is what Vagrant is using)
3. Vagrant is also allocating time to unpack the box
4. Wget is allowing for some type of compressed download

It would be helpful if you could benchmark this with curl for reference.

[mitchellh]

I really can't explain this. Vagrant doesn't do anything during the subprocess Ruby-wise: it subprocesses to curl. It doesn't even do the download in Ruby. Perhaps wget is using multiple connections to download multiple parts? I really don't know, but unless we get more information I have to assume that Vagrant is fine here.

Is curl just as slow? Vagrant is just subprocessing to curl until it completes.

[keeprock]

I'm experience the same slow experience. Anyone can try aria - http://aria2.sourceforge.net/ and http://stackoverflow.com/questions/3430810/wget-download-with-multiple-simultaneous-connections

It's seems a little bit faster, but, man, you can set this up using default vagrant download mechanism and take a walk or make yourself a sandwich. Get way from screen for a little bit.

[karlkfi]

Having the same problem here:

1. Upload a box manually to atlas
2. Create a new Vagrantfile with just vm_cfg.vm.box_url = <user>/box-name
3. vagrant up - box downloads slowly
4. wget box url from atlas (see vagrant up output) - box downloads lightening fast

[clakeb]

I wish there was just a +1 for this. Me too. Same connection for all 3 attempts. VPN turned off.

• vagrant up took 25+ minutes.
• wget took 3 minutes.
• curl took 4 minutes.

[milhaus]

Ubuntu vivid64 is downloading at ~56kbps. I'm on a 100mbit symmetric connection.
edit: it timed out before it could finish.
edit2: I can confirm that https://atlas.hashicorp.com/ubuntu/boxes/vivid64/versions/20160128.0.0/providers/virtualbox.box downloads dramatically faster over wget than via "vagrant up".

[ghost]

I'm trying to download the scotch/box and current download speeds using vagrant are less than 10kbps.

default: Progress: 0% (Rate: 2603/s, Estimated time remaining: 33:17:38)

However just as bad using wget.

[tehmaspc]

ditto; some popular boxes are very slow to download - i'm updating ubuntu/trusty64 as we speak and it's dropping below 1Kb/s. Been seeing this for a couple wks now.

[JasonTheAdams]

+1 -- exact same as last comment

[winni2k]

Same here:

$ vagrant box add lazygray/heroku-cedar-14
==> box: Loading metadata for box 'lazygray/heroku-cedar-14'
    box: URL: https://atlas.hashicorp.com/lazygray/heroku-cedar-14
==> box: Adding box 'lazygray/heroku-cedar-14' (v1.0.6) for provider: virtualbox
    box: Downloading: https://atlas.hashicorp.com/lazygray/boxes/heroku-cedar-14/versions/1.0.6/providers/virtualbox.box
==> box: Box download is resuming from prior download progress
    box: Progress: 3% (Rate: 281k/s, ...

[k1ng440]

same here

vagrant box update
==> default: Checking for updates to 'laravel/homestead'
    default: Latest installed version: 0.4.1
    default: Version constraints: >= 0
    default: Provider: vmware_desktop
==> default: Updating 'laravel/homestead' with provider 'vmware_desktop' from version
==> default: '0.4.1' to '0.4.2'...
==> default: Loading metadata for box 'https://atlas.hashicorp.com/laravel/homestead'
==> default: Adding box 'laravel/homestead' (v0.4.2) for provider: vmware_desktop
    default: Downloading: https://atlas.hashicorp.com/laravel/boxes/homestead/versions/0.4.2/providers/vmware_desktop.box
    default: Progress: 0% (Rate: 42210/s, Estimated time remaining: 6:10:54))

[richard-scott]

Is there any way to use something like axel to stream downloads in quicker?

[winni2k]

I guess there's nothing preventing people from sharing boxes via torrent. For example, below is a magnet link for the heroku-cedar-14 box:

magnet:?xt=urn:btih:5bb1480d5316f229bb71be55b56b06278de41a67&dn=heroku-cedar-14.box&tr=http%3A%2F%2F9.rarbg.com%3A2710%2Fannounce&tr=http%3A%2F%2Fannounce.torrentsmd.com%3A6969%2Fannounce&tr=http%3A%2F%2Fbt.careland.com.cn%3A6969%2Fannounce&tr=http%3A%2F%2Fexplodie.org%3A6969%2Fannounce&tr=http%3A%2F%2Fmgtracker.org%3A2710%2Fannounce&tr=http%3A%2F%2Ftracker.tfile.me%2Fannounce&tr=http%3A%2F%2Ftracker.torrenty.org%3A6969%2Fannounce&tr=http%3A%2F%2Ftracker.trackerfix.com%2Fannounce&tr=http%3A%2F%2Fwww.mvgroup.org%3A2710%2Fannounce&tr=udp%3A%2F%2F9.rarbg.com%3A2710%2Fannounce&tr=udp%3A%2F%2F9.rarbg.me%3A2710%2Fannounce&tr=udp%3A%2F%2F9.rarbg.to%3A2710%2Fannounce&tr=udp%3A%2F%2Fcoppersurfer.tk%3A6969%2Fannounce&tr=udp%3A%2F%2Fexodus.desync.com%3A6969%2Fannounce&tr=udp%3A%2F%2Fglotorrents.pw%3A6969%2Fannounce&tr=udp%3A%2F%2Fopen.demonii.com%3A1337%2Fannounce&tr=udp%3A%2F%2Ftracker.coppersurfer.tk%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.glotorrents.com%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.leechers-paradise.org%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.openbittorrent.com%3A80%2Fannounce&tr=udp%3A%2F%2Ftracker.opentrackr.org%3A1337%2Fannounce&tr=udp%3A%2F%2Ftracker.publicbt.com%3A80%2Fannounce&tr=udp%3A%2F%2Ftracker4.piratux.com%3A6969%2Fannounce

Anyone know a good website where one can search for torrents of vagrant boxes?

[tehmaspc]

@wkretzsch - I personally don't know at the moment about any torrent sites - but for me I wouldn't want to trust torrent links as the source for my infrastructure testing. It's a possible option but security is also important. For me official vagrant boxes from folks like puppetlabs hosted on Atlas are so slow to download at times that I wish this issue could be resolved. For internal vagrant boxes that I build for my company we have the option to host on S3 or Artifactory or private Atlas org.

@mitchellh - yes - curl is just as slow (for me). I don't think it is a Vagrant issue - but a backed server hosting issue. Granted - not a Vagrant issue per se.

[faddat]

Yes, this is because curl can only use one of my 3 connections at the same time. No, that's not the connection's rated speed. The rated speed is 45mbps. Yes, bittorrent does perform better. Just sayin-- your rationale for not supporting bittorrent is kinda thin here.

[winni2k]

@tehmaspc surely there must be a way for a website to publish the hash of their box along with a torrent link?

[darkn3rd]

I wish in general, there was a way to have incremental images, like docker images, with vagrant boxes. For the provisioners, which bootstrap (cfengine, chef, salt, puppet, docker, etc) by downloading their platform, I wish there was a way to download a packaged up installer, so that other fresh images that use that provisioner, e.g. ubuntu + docker, would not need to download the goods again. Box updates and provisioner downloads were already painful, but recently, have been beyond notoriously slow.

[PorterBytes]

Just went to update my box for the first time (trusty64 - noticed the warning on my vagrant up command output), and it's going to take my 1.5 hours on a 150MBps connection - pathetic. It's 2016 - I don't know the specifics of what's going on here, but surely we can fix this, like, by the end of next week? The tech that goes into modern technologies like vagrant is amazing, something this basic should be overcome in mere hours.

[faddat]

Amen, Matt, Amen. This is about UX.

There should be a recognition that line speed != line speed and practical
steps can be taken to overcome the daunting issue of line speed != line
speed.

Jacob Gadikian
E-mail: [email protected]
SKYPE: faddat
Phone/SMS: [phone]

On Sun, Apr 10, 2016 at 6:32 AM, Matt Porter [email protected]
wrote:

Just went to update my box for the first time (noticed the warning on my
vagrant up command output), and it's going to take my 1.5 hours on a
150MBps connection - pathetic. It's 2016 - I don't know the specifics of
what's going on here, but surely we can fix this, like, by the end of next
week? The tech that goes into modern technologies like vagrant is amazing,
something this basic should be overcome in mere hours.

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
https://github.com/mitchellh/vagrant/issues/5319#issuecomment-207881297

[briancline]

I just tried asking Vagrant to download ubuntu/trusty64, and was getting speeds of <= 5 KiB/sec. I killed it and tried again using the exact same command, and got 29 MiB/sec.

I think @mitchellh is correct in that this doesn't really seem like a Vagrant issue. If anything, it seems more like an Atlas issue (so possibly the ELB and/or whatever's sitting behind it). I highly doubt it has anything to do with the routes or hops between end-users and the ELB VIPs -- you wouldn't typically see such a polarizing set of speeds in that case, especially considering both VIPs terminate in us-east-1.

If for no other reason, it'd be highly desirable to see these made available through a CDN rather than a centrally-located ELB. Then again, I'm just one guy (who isn't paying for this service), so take that for what it's worth. Pretty thankful it's there either way.

[karlkfi]

It's not just an Atlas issue. I have boxes and metadata.json on S3, with a Fastly CDN in front and regularly have the exact same issue: sometimes vagrant downloads at 100kbps and sometimes it downloads at > 5mbps. You can cancel a slow download and half the time a retry gets you the faster speeds.

[milhaus]

I contacted support about this around the same time I chimed in here initially. Their response is that Vagrant uses curl to download things so they don't see this as a Vagrant problem. IMO that's an unprofessional cop-out because they chose to use curl, know that there are problems and aren't considering swapping out with an alternative to eliminate the problem for their users.

[Haroenv]

I can confirm that this is still an issue. All my peers also report times of >1h, while the connection here for other connections is around 200MB/s.

vagrant up
Bringing machine 'default' up with 'virtualbox' provider...
==> default: Box 'ubuntu/trusty32' could not be found. Attempting to find and install...
    default: Box Provider: virtualbox
    default: Box Version: >= 0
==> default: Loading metadata for box 'ubuntu/trusty32'
    default: URL: https://atlas.hashicorp.com/ubuntu/trusty32
==> default: Adding box 'ubuntu/trusty32' (v20160406.0.0) for provider: virtualbox
    default: Downloading: https://atlas.hashicorp.com/ubuntu/boxes/trusty32/versions/20160406.0.0/providers/virtualbox.box
    default: Progress: 11% (Rate: 43801/s, Estimated time remaining: 1:36:50)

[faddat]

While I am unsure of the origin of the problem, I really do wish that Hashicorp would get back to its unrelenting focus on user experience with this one. Muli-hour downloads (that should take 1-10 minutes)==bad ux.

[dacodekid]

Currently downloading an image for the 5th time (@13Xk/s, even with wget). Keep disconnecting me while around 50-90%. But it ALWAYS downloads at full speed either early morning / late night EST. Assuming it is a traffic issue, but regardless very bad UX.

    box: Progress: 47% (Rate: 106k/s, Estimated time remaining: 0:14:50)

[brazitech]

I have been trying for 2 day's now and still can not get it to download... its a shame.. it is really not impressing new comers to laravel .. i can only get 34ks speed.........

[richard-scott]

Speeds ok from the UK:

Bringing machine 'default' up with 'virtualbox' provider...
==> default: Box 'bento/centos-7.2' could not be found. Attempting to find and install...
    default: Box Provider: virtualbox
    default: Box Version: >= 0
==> default: Loading metadata for box 'bento/centos-7.2'
    default: URL: https://atlas.hashicorp.com/bento/centos-7.2
==> default: Adding box 'bento/centos-7.2' (v2.2.6) for provider: virtualbox
    default: Downloading: https://atlas.hashicorp.com/bento/boxes/centos-7.2/versions/2.2.6/providers/virtualbox.box
    default: Progress: 11% (Rate: 7728k/s, Estimated time remaining: 0:01:19)

What is your location?

[richard-scott]

Also, https://atlas.hashicorp.com/ URL's are delivered from Amazon Web Services (atlas-frontend-atlas-230110478.us-east-1.elb.amazonaws.com) so I doubt they are tight for bandwidth ;-)

Are the slow downloads being made from locations a long distance away from the AWS us-east-1 DC, perhaps thats the root cause of the issue?

Maybe the AWS CDN could be used to cache files around the world?

[PorterBytes]

I'm located in Vermont, which is pretty us-east-1 last I checked :dart:

[brazitech]

I am in Brazil.... got it to download.... 10m connection here took 4.6 hours!!!!! my wife just gave birth to our 8th little girl.. It only took her 40 minutes !!!!! lol There is a big problem with there download!!!!!

[thokari]

Same here, I have a 50Mbps connection...

default: Progress: 44% (Rate: 102k/s, Estimated time remaining: 0:15:01))

[macnibblet]

Sign me upp here, 100mb symmetric connection (Fiber) sloooow as shit, doing 150kb/s

[brazitech]

after looking at the years of complaints of slow download with no effort of resolving the issue,,, i think its time to start emailing Laravel to stop endorsing homestead until the issue is resolved..... maybe that will get their attention!!!! this is a real problem... 15 retries and then 4.6 hours to download a file is irresponsible on their part........

[richard-scott]

I bet it gets closed, but if you don't ask you don't get:

https://github.com/mitchellh/vagrant/issues/7307

[bradisbell]

Just snagged a box at 85mbit. You all fix something recently? Much better than it used to be.

[voycey]

This is painful to do anything on any more - On 100mbps synchronous connection and getting 168kb, either overloaded servers or throttling

[mccorkle]

In looking to debug curl being slow -- I found a stackoverflow post that suggests that --trace-ascii /dev/null makes your curl go at the speed you'd expect. For me, I'm trying to download CentosOS 7 and here are my results:

NO --trace-ascii option:

$ curl http://cloud.centos.org/centos/nt/x86_64/images/CentOS-7-x86_64-Vagrant-1606_01.VirtualBox.box -o CentOS-7-x86_64-Vagrant-1606_01.VirtualBox.box
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0  483M    0  489k    0     0  77724      0  1:48:44  0:00:06  1:48:38 69721

With trace-ascii the first time:

$ curl http://cloud.centos.org/centos/7/vagrant/x86_64/images/CentOS-7-x86_64-Vagrant-1606_01.VirtualBox.box -o CentOS-7-x86_64-Vagrant-1606_01.VirtualBox.box --trace-ascii /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  5  483M    5 24.5M    0     0  4259k      0  0:01:56  0:00:05  0:01:51 4599k

Does anyone else see the same behavior?

[codehaiku]

The download is extremely slow on my end too. I'm trying vagrant for the very first time. Might ditch this software and go back to my native apache2 instead.

[daryn-k]

Help. I have same problem. I can't wait 3 hours! Very slow! Stupid!

[brazitech]

i gave up a year ago..download to slow.. problems after down load.. have to download for 3 hours again.... Vagrant will not fix the problem that has been there for several years now.. you would think that after 3 or 4 years of this problem they would address the issue.....

[daryn-k]

8 hours to download! I hate you all!

[winni2k]

Lol, I hate you too @daryn-k :)

[ameyaagashe]

Guys why is this issue closed? This is still an outstanding issue and needs to be addressed ASAP. I am experiencing the same issue.

[grassjedi]

Wow! Downloading boxes is painful please fix this. PLEASE?

[JasonTheAdams]

I think it really has to do with time of day, traffic, alignment of the planets, etc. I haven't had slow speeds in a while. It seems very hit or miss. In fact, if it is slow you and start and stop it with the chance of getting a better connection. I'm not sure this is really the fault of the vagrant framework as much as it is the nature of large bottlenecked downloads.

[ryanaslett]

@mitchellh These errant speed symptoms from hashicorp's servers could be indicative of bumping up against AWS's IOPS credits for GP2 filesystems. https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSVolumeTypes.html#IOcredit

We had some testing infrastructure on drupal.org that would run fine for a long time, then suddenly drop to a crawl because we had "spent" all of our IO credits. It could be possible that hashicorps' servers are bumping up against the same limit.

sar -b could give some insight as to whether or not this explains the random performance drops.

[richard-scott]

It could be an idea for Hashicorp to move the images for download into S3 and use that for downloads... that would save on running instances specifically for downloads.

[babadofar]

Downloading boxes used to be quick, now it's so slow it makes vagrant a no-go for quick and simple developer environments.

[DeadlySystem]

Trying to download ubuntu/xenial64. Download speed maxes out at 150 KB/s on a 1 Gbps symmetrical fiber connection. WTF. Remaining time 1 hour? I could probably download the ISO, read the guide on how to set up my own box, and finish earlier.

EDIT: Interestingly, speed went up by factor 10 when I tried to download the same box in the browser simultaneously.

[matti]

use latest devops tools to speed things up
spend days watching max 420k/s download speeds

[minhphuc429]

Same here, I have a 30Mbps connection

default: Adding box 'ubuntu/trusty64' (v20170313.0.5) for provider: virtualbox
default: Downloading: https://atlas.hashicorp.com/ubuntu/boxes/trusty64/versions/20170313.0.5/providers/virtualbox.box
default: Box download is resuming from prior download progress
default: Progress: 0% (Rate: 80568/s, Estimated time remaining: 1:26:22)

[tersmitten]

@DeadlySystem We have the same experience, when I download the same box (url) using curl from the commandline (during the vagrant up)

[dqlopez]

Any update on this, fetching box from Hashicorp is painfully slow.

[faddat]

Hey, quick thought:

If this uses curl (not libcurl) through some sort of ruby-controlled, bash-mediated process, why not just remove curl for one of:

• ipfs
• aria2

Both would do the job better than curl.

[voycey]

It honestly looks like they dont give a shit, rules this out as an option
for me!

On 3 Apr 2017 8:15 AM, "Jacob Gadikian" [e-mail] wrote:

Hey, quick thought:

If this uses curl (not libcurl) through some sort of ruby-controlled,
bash-mediated process, why not just remove curl for one of:

• ipfs
• aria2

Both would do the job better than curl.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/mitchellh/vagrant/issues/5319#issuecomment-291019093,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABBAihR9ng4t2Jq1XTmAjMyMCnlEFtxRks5rsB4WgaJpZM4Deq5d
.

[bobflorian]

I'm on a 150Mbps line.

vagrant up = HOURS
vagrant box add = HOURS
browser download /wget = HOURS

May not be a vagrant issue per se, BUT IT IS. If your infrastructure can't handle it then your product is broken.

BAD UX

[richard-scott]

I opened a ticket about customizing the download tool, but it got rejected as too complicated to impliment ;-(

That said, my recent download speeds have been ok from the UK for a while.
This example downloaded just now:

    default: Box Provider: virtualbox
    default: Box Version: >= 0
==> default: Loading metadata for box 'bento/ubuntu-16.04'
    default: URL: https://atlas.hashicorp.com/bento/ubuntu-16.04
==> default: Adding box 'bento/ubuntu-16.04' (v2.3.4) for provider: virtualbox
    default: Downloading: https://atlas.hashicorp.com/bento/boxes/ubuntu-16.04/versions/2.3.4/providers/virtualbox.box
    default: Progress: 12% (Rate: 8940k/s, Estimated time remaining: 0:01:06)

Maybe you guys are just too far from their AWS instances for a good download speed?
All their download servers look to be in New York with no CDN to distribute content.

$ dig atlas.hashicorp.com
; <<>> DiG 9.10.3-P4-Ubuntu <<>> atlas.hashicorp.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 44169
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4000
;; QUESTION SECTION:
;atlas.hashicorp.com.           IN      A

;; ANSWER SECTION:
atlas.hashicorp.com.    120     IN      CNAME   atlas.hashi.co.
atlas.hashi.co.         60      IN      A       52.206.86.0
atlas.hashi.co.         60      IN      A       52.200.255.5
atlas.hashi.co.         60      IN      A       52.55.203.197

;; Query time: 42 msec
;; SERVER: 10.1.1.14#53(10.1.1.14)
;; WHEN: Mon Apr 24 09:20:24 DST 2017
;; MSG SIZE  rcvd: 124

http://geoiplookup.net/ip/52.55.203.197
http://geoiplookup.net/ip/52.206.86.0
http://geoiplookup.net/ip/52.200.255.5

Maybe its worth the people getting slow downloads detailing what ISP they are using? Maybe you are all on an ISP with a high contention ratio?

[chrisroberts]

https://github.com/mitchellh/vagrant/issues/8434#issuecomment-291979521

[bradisbell]

Maybe its worth the people getting slow downloads detailing what ISP they are using?

Any ISP, in any state I've traveled to in the last couple years.

This problem is squarely on whatever Hashicorp is doing for hosting, and that's where it needs to be fixed. If they're unable or unwilling to fix it, a torrent solution would certainly help without taking their resources aside from development for adding a torrent client to the tool. If all this stuff is on S3 anyway, AWS provides torrent seeding out of the box.

[colin-marshall]

I was getting painfully slow speeds and so I decided to see if upgrading Vagrant would change anything. Before I updated I was getting speeds of around 50-200kbps. After the update I was using the full 70mbps of my connection.

So for those of you that have slow speeds, try updating to v1.9.4 if you aren't already running it.

[sXe79]

Hi, I'm on the latest Vagrant 1.9.4 and the download speed is so slow that keeps disconnect. No way to download latest laravel homestead 2.1.0 box...
When I downloaded the 2.0 with Vagrant 1.9.3 (or previous, can't remember) it was flawless

[da-kami]

same issue here, very slow download rates
Rate: 35033/s (keeps jumping between 20000 and 60000...)
Estimated time remaining: 9:26:13 :(
(on vagrant 1.9.5)

[da-kami]

Now it's fast again... really depends on time of day and luck (as stated above...).

[oncet]

I'm having this same problem when trying to download Bento boxes. I'm using Vagrant 1.9.5 with 5.1.22. I've tried several times, during various days. My normal download speed is 1.7mb~ but when dowloading boxes I can't get pass 40kb.

If I try downloading with wget I get the same low speed.

[richard-scott]

Maybe the move will help:

https://www.vagrantup.com/docs/vagrant-cloud/vagrant-cloud-migration.html

[oncet]

Gonna try again tonight to see if the migration helped. Anyway I noted that with a different ISP the download rates were fine, so I guess it's not a problem with Vagrant itself (like many users already reported).

[prabhu89wali]

Well, nothing has changed, it still downloads at a snail's pace given that i am on a 125 Mbps connection!

[acerest]

Sooooooooooooooo slowly！！！！:(

[matti]

create a new issue, this has been closed for ages

[iron-cherep]

i just downloaded latest scotch-box vagrant box, spent ~1hour for that, and then i found an issue which could be fixed only by downgrading to previous version of box, so i'm here again downloading second box for next hour of my working time. So.. it's good since i got per-time payment, but on the other hand probably i'll have some problems explaining why i spen last day downloading stuff from the internet

[otjutt]

I'm also facing the slow download issue.

[prabhu89wali]

Try downloading from a wired connection, it's exponentially faster!

On 4 September 2017 at 17:02, Omar Tariq [e-mail] wrote:

I'm also facing the slow download issue.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/mitchellh/vagrant/issues/5319#issuecomment-326939886,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABmbHxWHtAnCnEw34FB4oXX8lTQrOsNRks5se9-_gaJpZM4Deq5d
.

--
Best regards,
Shivaprabhu Wali.

[CaelanStewart]

Typical issue, been around for ages, everybody's moaning about it, and nothing is being done.

[PascalAOMS]

Came here via Google and this might help some people:

When I run vagrant init [box] && vagrant up, it loads super slow.
But when I run vagrant box add [box], it's faster.

Going from 120 kb/s to >1000 kb/s.

[lukewang1024]

@PascalAOMS Thanks for the tip. At least it worked for me.

[oleg-andreyev]

Today I've tried aria2 with 16 simultaneous connections to download laravel/homestead managed to get up to 9.6MiB

Download Results:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
80fa38|ERR |   7.8MiB/s|C:/development/homestead/47dce273-9892-4691-a746-c4f351ae44a5

[sandeshchaudhari]

From last two days i'm trying to download vigrant box but every time i try to download it gives me error and downloading spedd is very slow. I also tried to install it manually but it also didn't worked.For newbie to Laravel it pretty frustrating and i'm still not able to download it.Don't know what should i do now.

[stevenrombauts]

I'm seeing the same problem. Running latest Vagrant 2.0.0. When downloading a box (configured using external URL on Vagrant Cloud, so it's actually redirecting to Rackspace Cloud Files) it's painfully slow.

If I use cURL or Chrome to download the same URL (https://vagrantcloud.com/joomlatools/boxes/box/versions/1.5.0/providers/virtualbox.box) from the same machine it only takes a couple of seconds to complete the download. What could be the issue here?

[marcoflorian]

Using terminal to add a box is extremely slow, more than 1 hour with 100kb/s. If I use a browser to directly download the box it is much faster, just a few minutes with 600kb/s. What's the reason for having such a huge difference? Seems as adding the box through terminal won't use all the available bandwidth.

[panozzaj]

I am trying to download boxes more quickly, and following the advice of some of the above comments, I used aria2 through homebrew to download much faster in parallel. Sample command (url was from original box add attempt from Vagrant):

$ aria2c -x16 https://vagrantcloud.com/ubuntu/boxes/xenial64/versions/20171118.0.0/providers/virtualbox.box

My speed was around 100k/s with Vagrant's download, up to 1 MB/s with aria2.

Then when you finish downloading that, you can add the box using:

$ vagrant box add ubuntu/xenial64 ../xenial64.box

I think you can remove the downloaded box after this since it will be copied to the standard box storage path after adding it. Hope this helps save someone else some time.

[madmatt]

@panozzaj's comment above doesn't quite work if a Vagrantfile expects a specific version of a box (that command will add a box without a version). Instead, you can do the below.

Steps to get a specific box at a specific version without using vagrant to perform the download

I'm using laravel/homestead in this example because it's what I was trying to get.

1. Download the box using a better download client (e.g. your browser, curl, wget, whatever).

2. Create a new json file (anywhere), add this to it (note, tweak the name, version and url keys to match the box you want, don't worry about the checksum key yet. For url, use the path to the file you just downloaded, example below):

{
    "name": "laravel/homestead",
    "description": "Whatever you want",
    "versions": [{
        "version": "4.0.0",
        "providers": [{
                "name": "virtualbox",
                "url": "file://c:/users/madmatt/Downloads/47dce273-9892-4691-a746-c4f351ae44a5",
                "checksum_type": "sha1",
                "checksum": "abc123"
        }]
    }]
}

1. In your Vagrantfile, add the following lines vm.box and vm.box_url keys):

Vagrant.configure("2") do |config|
    config.vm.box = "laravel/homestead"
    config.vm.box_url = "file://c:/users/madmatt/path/to/the/json-file-you-created-in-step-2.json"
end

1. Run vagrant up as normal.

2. vagrant will complain that the sha hash doesn't match (The checksum of the downloaded box did not match...). Take the string that appears next to 'Expected', copy and paste that into the checksum key of the json file you created in step 2.

3. Run vagrant up again, this time it should load from the local file, store it as the correct name and version, and successfully run it.

Hopefully someone finds these steps useful... funnily enough I have done all this research (having never used vagrant before), have tested it a bunch of times, and the original vagrant box add laravel/homestead command that I started running 3 hours ago is still only 8% complete, even though in that time I've downloaded the box file 8 times outside of vagrant.

The rest of this is just my experience, no need to read further ;)

My experience (aka. why is vagrant so slow to download boxes)

Was getting 420b/sec (yes, that's bytes per second) on a gigabit connection downloading https://vagrantcloud.com/laravel/boxes/homestead/versions/4.0.0/providers/virtualbox.box.

I downloaded the same file via browser, curl and wget with speeds varying between 12 and 27MB/sec. I then tried doing both at the same time - I was able to download via Chrome, Firefox, curl and wget before vagrant box add laravel/homestead had downloaded 1%.

The URL I got in the browser was https://vagrantcloud-files-production.s3.amazonaws.com/archivist/boxes/47dce273-9892-4691-a746-c4f351ae44a5?X-Amz-Algorithm=<snip>&X-Amz-Credential=<snip>&X-Amz-Date=<snip>&X-Amz-Expires=<snip>&X-Amz-SignedHeaders=<snip>&X-Amz-Signature=<snip>

I don't know what the problem here is, but I can think of a couple:

• Whatever UA vagrant uses is bad, and AWS severely limits it
• Whatever vagrant uses to download isn't following redirects, or somehow never ends up downloading from AWS infrastructure, instead using some other terribly overloaded server/proxy

[TimOgilvy]

how is this closed? still a major issue

[bradisbell]

Perhaps it's time to fork. This project is core to a lot of development environments, leaving us all subject to the whims of HashiCorp... and on this issue we seemingly cannot even get a reasonable official response.

Vagrant is MIT-licensed. Boxes could easily be distributed via torrent, or we can even just specify URLs in our Vagrantfile. We don't need the Vagrant Cloud dependency if some basic enhancements around box handling are made. Are there any maintained forks already in existence?

Any thoughts from others?

[fredngo]

Definitely still an issue!

[atb00ker]

Are there any trusted torrent links available? I am trying to download bentoo/ubuntu16.04 and i am getting 12 kbps and upto 15kbps when i try to do wget, i am sure that my connection is fine!

[DenzoNL]

Still an issue here, have to update my box at ~150k/s, not able to do a lot of useful work in the meanwhile..

[dogweather]

Still an issue.

[dogweather]

@bradisbell The real solution is probably a docker-based development environment. Just waiting for that ecosystem to settle down .

[TimOgilvy]

🍺 🍺 🍺 🍺 🍺 🍺

I'm just going to leave these here for any poor sod waiting for this download.

[dogweather]

I found a root cause and a solution for my situation. I increased my download speed 100x:

Root cause: Slow speed of ubuntu.com cloud box servers
Solution: Switch to another trusted source for Ubuntu boxes, Puppet Labs.

I've been using ubuntu/xenial64 for developing apps that must deploy to Ubuntu 16. (Heroku) And download speeds have become ridiculously slow, or failed entirely. I used wget and watched it follow the redirects and saw that it's actually hitting ubuntu.com. I tried several ways to download from Ubuntu, but they all resulted in speeds around 50 KB/s.

So I went back to the Vagrant box listing to see who else might have a Xenial 64 box, and if I'd trust them. Turns out Puppet does, and I trust them as much as (or even more than) Ubuntu. My Vagrantfile now has the config line:

config.vm.box = "puppetlabs/ubuntu-16.04-64-nocm"

And it downloaded in just a few seconds, as opposed to possibly an hour or never.

[tristanmason]

@dogweather Thanks for this! My download time estimate went from 3 hours to 3 minutes when I switched to a box from the list at https://app.vagrantup.com/puppetlabs/

[dogweather]

@tristanmason You're welcome!

Ironically: In the past week I switched to Docker Compose, and it's AMAZING. I can't believe I've waited this long to use it. I used this Quickstart for Rails doc.

• Configuration is 1/10 the size and complexity of Vagrant for my typical Rails app
• Build and Boot times are much faster
• On my Linux desktop in particular, there is a 10x speed increase vs. running on Mac.
• There's a huge reduction in RAM required.
• I'm going to save a ton of money on development computers. I.e., 8GB RAM is plenty for a Docker-based dev machine. (Not so good for Vagrant+VirtualBox though.) And a top-tier i7 isn't necessary anymore either to get good speeds.

I was able to setup a Docker-based dev environment on four computers in a fraction of the time it takes me to create a single Vagrant setup from scratch.

I'm going to write up a blog with detailed metrics - I'll post it here.

[oncet]

@dogweather Vagrant and Docker seems similar but they are different things. Also, you can use a Docker provider within Vagrant.

[dogweather]

@oncet Yep, very different! But for my use case, they're equivalent and commodified: Ways I can create and launch an isolated dev environment with just one or two commands.

Docker Compose works great and is far simpler than Vagrant (note, this not just "Docker" per se, which only launches containers one by one) and I'd need to hear about a compelling reason to try a Vagrant+Docker solution, which sounds pretty damn complex. ;-)

[sXe79]

Pain. This is a pain. I'll never be able to download the 5.0.1 I guess

[jdunk]

On a 100mbps connection in Manila, with Vagrant 2.0.1, I was trying to download:

https://vagrantcloud.com/ubuntu/boxes/trusty64/versions/20180110.0.0/providers/virtualbox.box

...and I was getting speeds of anywhere from 1k-150k via vagrant up or even wget. Total download time: 12 hours!

Then I did one thing: I VPN'd to California.

Suddenly, my download took only 3 mins. So that's something worth trying possibly.

[racterub]

Same here, I thought this is a personal problem lol.
Even update box still get really poor download speed, hashicorp please fix :(

==> default: Updating 'ubuntu/xenial64' with provider 'virtualbox' from version
==> default: '20171221.0.0' to '20180122.0.0'...
==> default: Loading metadata for box 'https://vagrantcloud.com/ubuntu/xenial64'
==> default: Adding box 'ubuntu/xenial64' (v20180122.0.0) for provider: virtualbox
    default: Downloading: https://vagrantcloud.com/ubuntu/boxes/xenial64/versions/20180122.0.0/providers/virtualbox.box
==> default: Box download is resuming from prior download progress
    default: Progress: 13% (Rate: 52999/s, Estimated time remaining: 0:31:46)

[dqlopez]

Ignoring such persistent issue means something on Hashicorp's end, I believe.

[dogweather]

@dqlopez from the previous comments, this is a solved issue: it's purely the speed of ubuntu's servers. The solution is to use some other org's images.

[fredngo]

Confirming that when I used bento/ubuntu-16.04 instead of ubuntu/xenial64 (why?), I got pretty good download speeds. The box downloaded in less than a minute, on a 300 Mbps fibre optic connection. (Unfortunately I didn't think to copy and paste the log.)

[dogweather]

I promised I'd report back on my transition to Docker, so here goes. I spent a ton of time creating a config that delivers the one-liner vagrant up experience. It's docker-compose up. And there's only one dependency, Docker. Not two (VM _plus_ Vagrant).

My post about performance of a Docker dev environment on Mac and Linux: medium.com

Repos with my configuration, tailored for Ruby on Rails and Phoenix development. Very very similar. Can be tailored for any language, I'd imagine.

cc: @fredngo

[bradisbell]

@dogweather Frankly, I don't see how your commentary on docker has anything to do with the issue at hand. Please stop taking this thread off-topic.

[externl]

Docker is not an alternative to Vagrant. They work completely differently and are intended for different environments. While I'm all for a discussion on the merits of when each should be used, this thread is not really the place for it.

Edit: I love both and use them both in development and production environments.

[jeliasson]

Haven't read the full thread, but it was a significant difference between using PowerShell and Git Bash.

[mohclips]

Well, I cant download a single box tonite. I'm in the UK.

The box 'puppetlabs/ubuntu-16.04-64-puppet' could not be found or
could not be accessed in the remote catalog. If this is a private
box on HashiCorp's Atlas, please verify you're logged in via
`vagrant login`. Also, please double-check the name. The expanded
URL and error message are shown below:

URL: ["https://atlas.hashicorp.com/puppetlabs/ubuntu-16.04-64-puppet"]
Error: The requested URL returned error: 404 Not Found

config.vm.box = "puppetlabs/ubuntu-16.04-64-nocm"
config.vm.box = "bento/ubuntu-16.04"
config.vm.box = "ubuntu/xenial64"

none of those work.

BUT if you download the box manually via wget, all works fine...

nick@TX200-S5:~/workspaces/elk-vagrant$ wget https://app.vagrantup.com/puppetlabs/boxes/ubuntu-16.04-64-nocm/versions/1.0.0/providers/virtualbox.box
--2018-03-04 00:30:45--  https://app.vagrantup.com/puppetlabs/boxes/ubuntu-16.04-64-nocm/versions/1.0.0/providers/virtualbox.box
Resolving app.vagrantup.com (app.vagrantup.com)... 50.17.237.77, 54.221.226.80, 54.243.175.62, ...
Connecting to app.vagrantup.com (app.vagrantup.com)|50.17.237.77|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://s3.amazonaws.com/puppetlabs-vagrantcloud/ubuntu-16.04-x86_64-virtualbox-nocm-1.0.0.box [following]
--2018-03-04 00:30:46--  https://s3.amazonaws.com/puppetlabs-vagrantcloud/ubuntu-16.04-x86_64-virtualbox-nocm-1.0.0.box
Resolving s3.amazonaws.com (s3.amazonaws.com)... 54.231.32.82
Connecting to s3.amazonaws.com (s3.amazonaws.com)|54.231.32.82|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 666915336 (636M) [application/vnd.previewsystems.box]
Saving to: ‘virtualbox.box’

100%[==========================================================================================>] 666,915,336 2.23MB/s   in 4m 48s 

2018-03-04 00:35:35 (2.21 MB/s) - ‘virtualbox.box’ saved [666915336/666915336]

nick@TX200-S5:~/workspaces/elk-vagrant$ vagrant box add bento/ubuntu-16.04
The box 'bento/ubuntu-16.04' could not be found or
could not be accessed in the remote catalog. If this is a private
box on HashiCorp's Atlas, please verify you're logged in via
`vagrant login`. Also, please double-check the name. The expanded
URL and error message are shown below:
URL: ["https://atlas.hashicorp.com/bento/ubuntu-16.04"]
Error: The requested URL returned error: 404 Not Found


nick@TX200-S5:~/workspaces/elk-vagrant$ wget -O bento-ubuntu-16.04 https://app.vagrantup.com/bento/boxes/ubuntu-16.04/versions/201802.02.0/providers/virtualbox.box
--2018-03-04 00:37:41--  https://app.vagrantup.com/bento/boxes/ubuntu-16.04/versions/201802.02.0/providers/virtualbox.box
Resolving app.vagrantup.com (app.vagrantup.com)... 54.243.252.123, 50.19.252.69, 54.243.137.45, ...
Connecting to app.vagrantup.com (app.vagrantup.com)|54.243.252.123|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://archivist.vagrantup.com/v1/object/eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJrZXkiOiJib3hlcy80NzYzZDNiMy04Yzk0LTQ2YmMtYTQxNy02MDEwYjkxYzhlZjIiLCJtb2RlIjoiciIsImV4cGlyZSI6MTUyMDEyNDc2MX0.At50HVbqsvj9bfhDrbkzH7G5ON5RCcnYHwm5Xx1GXzA [following]
--2018-03-04 00:37:41--  https://archivist.vagrantup.com/v1/object/eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJrZXkiOiJib3hlcy80NzYzZDNiMy04Yzk0LTQ2YmMtYTQxNy02MDEwYjkxYzhlZjIiLCJtb2RlIjoiciIsImV4cGlyZSI6MTUyMDEyNDc2MX0.At50HVbqsvj9bfhDrbkzH7G5ON5RCcnYHwm5Xx1GXzA
Resolving archivist.vagrantup.com (archivist.vagrantup.com)... 50.16.237.173, 23.21.92.233, 54.221.212.171, ...
Connecting to archivist.vagrantup.com (archivist.vagrantup.com)|50.16.237.173|:443... connected.
HTTP request sent, awaiting response... 307 Temporary Redirect
Location: https://vagrantcloud-files-production.s3.amazonaws.com/archivist/boxes/4763d3b3-8c94-46bc-a417-6010b91c8ef2?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIA4WZ7ZDX3WM4HDQ%2F20180304%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20180304T003742Z&X-Amz-Expires=60&X-Amz-SignedHeaders=host&X-Amz-Signature=871143b6e5a7078c775fba1f262ad4b3cd31e065ecde0d4ff0c4da2081aec7e7 [following]
--2018-03-04 00:37:42--  https://vagrantcloud-files-production.s3.amazonaws.com/archivist/boxes/4763d3b3-8c94-46bc-a417-6010b91c8ef2?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIA4WZ7ZDX3WM4HDQ%2F20180304%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20180304T003742Z&X-Amz-Expires=60&X-Amz-SignedHeaders=host&X-Amz-Signature=871143b6e5a7078c775fba1f262ad4b3cd31e065ecde0d4ff0c4da2081aec7e7
Resolving vagrantcloud-files-production.s3.amazonaws.com (vagrantcloud-files-production.s3.amazonaws.com)... 52.216.164.43
Connecting to vagrantcloud-files-production.s3.amazonaws.com (vagrantcloud-files-production.s3.amazonaws.com)|52.216.164.43|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 428203828 (408M) [binary/octet-stream]
Saving to: ‘bento-ubuntu-16.04’

100%[==========================================================================================>] 428,203,828 2.09MB/s   in 3m 6s  

2018-03-04 00:40:48 (2.19 MB/s) - ‘bento-ubuntu-16.04’ saved [428203828/428203828]

A direct box add works too

nick@TX200-S5:~/workspaces/elk-vagrant$ vagrant box add "bento/ubuntu-16.04" https://app.vagrantup.com/bento/boxes/ubuntu-16.04/versions/201802.02.0/providers/virtualbox.box
==> box: Box file was not detected as metadata. Adding it directly...
==> box: Adding box 'bento/ubuntu-16.04' (v0) for provider: 
    box: Downloading: https://app.vagrantup.com/bento/boxes/ubuntu-16.04/versions/201802.02.0/providers/virtualbox.box
    box: Progress: 7% (Rate: 2247k/s, Estimated time remaining: 0:03:13)

[mahmoodn]

I would like to know how can I manually add the .box file in an offline manner. It is not clear in the documents on how to do that. I think the correct place for modification is Vagrantfile. However, I can not figure out which parameter should I change.

[externl]

@mahmoodn Your question is totally unrelated to this thread. The box documentation should help you with your question.

[Silic0nS0ldier]

https://vagrantcloud.com/laravel/boxes/homestead/versions/5.2.0/providers/virtualbox.box
    homestead-7: Progress: 36% (Rate: 95492/s, Estimated time remaining: 0:54:16)

99% of the time, this is what I'm dealing with. On a stable fibre connection. No WiFi. Every now and again I get a short burst of speed that pushes progress up around 5%, but overall, I'm left waiting several hours. This is insane. People have proposed work arounds and mitigations but those aren't solutions. When this issue was first closed, it was implied that curl might be to blame. Perhaps this is true, and if it is, then it should be switched out for a different solution.

[YousefSaber]

I suffer the same problem also

My connection on speedtest.net 6.6Mbps while the download takes only 0.6 Mbps of my bandwidth

[chrisroberts]

Hi everyone,

This issue overall is very complicated in that it is not easily reproducible and is very dependent on location, network health, and the actual download location of the target box. In many cases the box being downloaded is located on a third party system. One feature that was introduced in Vagrant a couple releases ago was the notification of redirect to show when boxes are being downloaded from a third party server.

There have also been claims that the embedded curl is downloading files slower than the system curl. I have not been successful in validating this claim and do not see a difference in download speed between my system curl and the embedded curl. However, as of Vagrant 2.0.4, the VAGRANT_PREFER_SYSTEM_BINS environment variable is available for all platforms. When enabled, Vagrant will use local system binaries if available instead of the embedded binaries. If you observe faster download speeds with your local curl binary, this provides a switch to easily enable Vagrant to use the local binary.

My apologies for the frustration this issue has caused. It is something I have investigated multiple times, but the number of variables involved makes this extremely difficult to get a valid reproduction. I have updated Vagrant's functionality to make the underlying cause more easily understandable where available (redirect notifications) or easier to work around. I am continuing to investigate our options for box downloads from Vagrant Cloud in different regions of the world (which have occasionally presented with issues). However, with box downloads from third party locations, there is little I am able to do to control their available bandwidth or any kind of throttling they may impose.

Cheers!