vagrant box update - Fails with 404 Not Found error

[remd]

Vagrant version

$ vagrant -v
Vagrant 2.0.2

Host operating system

Ubuntu 16.04.3 LTS

Guest operating system

Ubuntu 16.04.3 LTS (ubuntu/xenial64)

Vagrantfile

Vagrant.configure(2) do |config|
  config.vm.box = "ubuntu/xenial64"

  # insecure vagrant key
  config.ssh.insert_key = false
end

Debug output

$ vagrant --debug box update > debug.output 2>&1
https://gist.github.com/remd/8cc26e356fbb8b04c5f25df82ce077d1

Expected behavior

Should have downloaded and installed the new box version. 20170713.0.0 -> 20180126.0.0

Actual behavior

404 Not Found Error.

Steps to reproduce

1. Create Vagrantfile with ubuntu/xenial64
2. vagrant box update

Also, directly navigating to the URL listed in the debug output (https://atlas.hashicorp.com/ubuntu/xenial64) shows a 404 Not Found error in my browser. Is Atlas not working properly at the moment?

References

I could not find any associated with the latest version of vagrant.

EDIT

See https://github.com/hashicorp/vagrant/issues/9442#issuecomment-363646926 for a clearer summary of the root cause of this issue.

[Val]

Same with debian/stretch64:

> /usr/bin/curl -I -q --fail --location --max-redirs 10 --user-agent "Vagrant/1.9.1 \
(+https://www.vagrantup.com; ruby2.3.3)" --continue-at - \
-H "Accept: application/json" https://atlas.hashicorp.com/debian/stretch64
HTTP/1.1 200 Connection established

curl: (22) The requested URL returned error: 404 Not Found

At this time "_All Systems Operational_" https://status.hashicorp.com ...

[mal-risma]

Testet this with a lot of different boxes including ubuntu/xenial64, centos/7 and laravel/homestead. All gives 404.

[tjventurini]

Same here for ubuntu/xenial64 ... At least it doesn't seem to be our firewall anymore ^^

[wizonesolutions]

I'm wondering if this is related to Vagrant Cloud now being the source for Vagrant boxes. Perhaps HashiCorp Atlas removed its redirects recently? Going to experiment with this a bit.

[mal-risma]

Upgraded to Vagrant 2.0.2 and it started working.
Version 1.9.1 is latest in ubuntu 17.10, so anyone running ubuntu need to fetch the latest version from the vagrant homepage.

[Val]

A workarround to add in your Vagrantfile:

Vagrant::DEFAULT_SERVER_URL.replace('https://vagrantcloud.com')

[remd]

@mal-risma I am using Vagrant 2.0.2.

[mal-risma]

Can't explain why it started working on my machine then. Sorry.

[Val]

@mal-risma : perhaps you don't need authentication

The whole patch to read is: 5f955c3d38f27a036e3eb569ad17494afe293a21

So my workarround is not a valid solution for authenticated users...

[HenningCash]

I had the same issue. Updated to v2.0.2 but updates were still downloaded from atlas. New installs work fine.
Looks like this is a caching error: Vagrant stores in a file where it downloaded the box from.
So find all $VAGRANT_HOME/boxes/[box-slug]/metadata_url files and replace atlas.hashicorp.com with vagrantcloud.com. After that box updates should work fine again.

I'm not a find/xarg/sed pro but I bet someone will come up with a one-liner to replace all occurrences automatically

[remd]

That looks like the underlying cause @HenningCash. Thank you!

$ cat ~/.vagrant.d/boxes/ubuntu-VAGRANTSLASH-xenial64/metadata_url
https://atlas.hashicorp.com/ubuntu/xenial64

After fixing this I was able to successfully download the box update.

Since I had been using a version earlier than Vagrant 2.0.2 when I originally downloaded this box's metadata the metadata_url for each box was still cached against Atlas.

[nicwortel]

sed -i -- 's/atlas.hashicorp/vagrantcloud/g' ~/.vagrant.d/boxes/*/metadata_url

This seems to work for me (on an Ubuntu 16.04 host).

[laurentj]

A better solution would be to setup a redirection on the web server of atlas.hashicorp. My 2 cents...

[whatnickcodes]

This is effecting all boxes. Anyone clarify what's happening here exactly?

[Val]

@whatnickcodes : HashCorp's SysOps/DevOps/WhaterverOps removed previous Vagrant URL redirections from atlas.hashicorp.com to vagrantcloud.com too quickly.

[whatnickcodes]

What's a fix for people who aren't going to be knowledgable enough to modify Vagrant meta data?

Edit: I support a popular Vagrant box targeted towards absolute beginners

[Val]

@whatnickcodes : try the both

• https://github.com/hashicorp/vagrant/issues/9442#issuecomment-363080565
• https://github.com/hashicorp/vagrant/issues/9442#issuecomment-363126830

[remd]

@whatnickcodes Likely you will need to wait for the upstream fix. This issue is tagged with the 2.0.3 milestone, so that, presumably, will be the version that this fix lands in.

[samrocketman]

@whatnickcodes the fix is to install the latest version of Vagrant from the website. I also had to upgrade VirtualBox because it was freezing.

[remd]

@samrocketman That won't fix box metadata_urls that were downloaded using a version of Vagrant <1.9.6.

To be clear what the issue here is: 5f955c3d38f27a036e3eb569ad17494afe293a21 converted cached box metadata_urls from atlas.hashicorp.com to vagrantcloud.com. This was released in Vagrant 1.9.6.

If you were using a version of Vagrant <1.9.6 you were still able to download boxes, update box versions, etc., until very recently (I first encountered the 404 Not Found error on 2018-02-03). Because my box metadata was created with a version of Vagrant <1.9.6, it still pointed at https://atlas.hashicorp.com/ubuntu/xenial64 instead of https://vagrantcloud.com/ubuntu/xenial64 (see https://github.com/hashicorp/vagrant/issues/9442#issuecomment-363113660 and https://github.com/hashicorp/vagrant/issues/9442#issuecomment-363119645).

Even after I updated to Vagrant 2.0.2, the issue was still present, because Vagrant does not migrate the old box metadata_url stored in $VAGRANT_HOME/boxes/[box-slug]/metadata_url.

In order to be backwards compatible with box metadata_urls created prior to 1.9.6, a fix needs to be introduced to Vagrant.

[sromberg]

What's the ETA for 2.0.3? This issue broke a critical part in our test infrastructure so we are very much looking forward to a fix.

[samrocketman]

@remd I guess I didn't face a problem because I deleted all of my virtualbox VMs and ran

rm -rf ~/.vagrant.d

to clean up the settings. I have .vagrant in my global gitignore (core.excludesfile) file so it's not tracked in my repositories. .vagrant is created in the current working directory where vagrant up is run. That means I can run

# git clean -xfdn to dry run
git clean -xfd

to remove the vagrant metadata. It was an opportunity to clean up a year of vagrant cruft for me.

[Fuxy22]

i did vagrant box remove laravel/homestead --all and after vagrant up before i found out what exactly was the issue.

This seems to wor as well since it deleted everything :)

[VigRoco]

A more nuclear option that worked for me was to uninstall Vagrant and then reinstall it: https://www.vagrantup.com/docs/installation/uninstallation.html

[masudiiuc]

I was also getting the following error with running vagrant 1.9.1

Bringing machine 'redd' up with 'virtualbox' provider...
==> redd: Box 'bento/ubuntu-16.04' could not be found. Attempting to find and install...
    redd: Box Provider: virtualbox
    redd: Box Version: >= 0
The box 'bento/ubuntu-16.04' could not be found or
could not be accessed in the remote catalog. If this is a private
box on HashiCorp's Atlas, please verify you're logged in via
`vagrant login`. Also, please double-check the name. The expanded
URL and error message are shown below:

URL: ["https://atlas.hashicorp.com/bento/ubuntu-16.04"]
Error: The requested URL returned error: 404 Not Found

my Vagrantfile was like this:

# -*- mode: ruby -*-

dir = File.dirname(File.expand_path(__FILE__))

require 'yaml'
require "#{dir}/puphpet/ruby/deep_merge.rb"
require "#{dir}/puphpet/ruby/to_bool.rb"

configValues = YAML.load_file("#{dir}/puphpet/config.yaml")

provider = ENV['VAGRANT_DEFAULT_PROVIDER'] ? ENV['VAGRANT_DEFAULT_PROVIDER'] : 'local'
if File.file?("#{dir}/puphpet/config-#{provider}.yaml")
  custom = YAML.load_file("#{dir}/puphpet/config-#{provider}.yaml")
  configValues.deep_merge!(custom)
end

if File.file?("#{dir}/puphpet/config-custom.yaml")
  custom = YAML.load_file("#{dir}/puphpet/config-custom.yaml")
  configValues.deep_merge!(custom)
end

data = configValues['vagrantfile']

Vagrant.require_version '>= 1.8.1'

Vagrant.configure('2') do |config|
  eval File.read("#{dir}/puphpet/vagrant/Vagrantfile-#{data['target']}")
end

and I changed it to

Solution

# -*- mode: ruby -*-

dir = File.dirname(File.expand_path(__FILE__))

require 'yaml'
require "#{dir}/puphpet/ruby/deep_merge.rb"
require "#{dir}/puphpet/ruby/to_bool.rb"

configValues = YAML.load_file("#{dir}/puphpet/config.yaml")

provider = ENV['VAGRANT_DEFAULT_PROVIDER'] ? ENV['VAGRANT_DEFAULT_PROVIDER'] : 'local'
if File.file?("#{dir}/puphpet/config-#{provider}.yaml")
  custom = YAML.load_file("#{dir}/puphpet/config-#{provider}.yaml")
  configValues.deep_merge!(custom)
end

if File.file?("#{dir}/puphpet/config-custom.yaml")
  custom = YAML.load_file("#{dir}/puphpet/config-custom.yaml")
  configValues.deep_merge!(custom)
end

data = configValues['vagrantfile']

Vagrant.require_version '>= 1.8.1'
# add this Line as per @Val comments
Vagrant::DEFAULT_SERVER_URL.replace('https://vagrantcloud.com') 

Vagrant.configure('2') do |config|
  eval File.read("#{dir}/puphpet/vagrant/Vagrantfile-#{data['target']}")
end

And finally It works

[pkarjala]

Can confirm that this affects existing boxes without updating the remote URL. Also, editing the metadata_url file is causing Vagrant to throw illegal character warnings. Have used the sed command suggested, and also edited with vim.

[remd]

@pkarjala You might accidentally be introducing an EOL character to the end of the metadata_url file, which will throw an illegal character warning. Try these instructions and confirm with cat that there is no EOL character.

[pkarjala]

@remd Thank you; I was able to remove the EOL character from the metadata_url file, which resolved the issue. Hopefully Vagrant gets updated to fix the entirety of the issue in the first place.

[ilbonte]

I tried vagrant box add ubuntu/xenial64 and then vagrant box update and it worked

[shkkmo]

I just ran into this on a fresh install of ubuntu 16.04
it looks like the version in the official canonical repositories is still 2.0.0.2

Hashi corp should really just fix the redirects as this will continue to cause problems to people.

[ageis]

I know this issue is closed; I still love your products but it's clear people see this as a failure on HashiCorp's part. There should've been better communication and a long-term (more than 8 months) redirect from Atlas. I thought I was set with the new DEFAULT_SERVER_URL, didn't know there was metadata in ~/.vagrant.d which was still causing 404s for boxes until catching up on this issue. Good case study in migration we can all learn from in open source.

[shkkmo]

Yes, this will be an ongoing issue for all new installations of vagrant on Ubuntu LTS 14.04 and 16.04 since Canonical doesn't have the newer versions of vagrant in the official repositories for those distributions. Especially given that there is no official PPA for vagrant, this seems like a significant problem (https://github.com/hashicorp/vagrant-installers/issues/12).

Is there any reason not to maintain the old redirects from Atlas? That seems like less work than getting this change back-ported by Canonical into the official repositories or launching an official PPA for vagrant.

[remd]

@shkkmo For what it's worth, the documentation for installation on Vagrant's website specifically warns against using the system package manager for installing Vagrant for exactly the reason you described.

From https://www.vagrantup.com/docs/installation/:

Beware of system package managers! Some operating system distributions include a vagrant package in their upstream package repos. Please do not install Vagrant in this manner. Typically these packages are missing dependencies or include very outdated versions of Vagrant. If you install via your system's package manager, it is very likely that you will experience issues. Please use the official installers on the downloads page.

[shkkmo]

@remd If that is the official stance, why hasn't vagrant requested the removal of Vagrant from the repositories for up coming releases? http://news.softpedia.com/news/ownCloud-Asks-Canonical-to-Remove-Their-Software-From-Ubuntu-Repos-Sparks-Fly-462906.shtml

The current situation seems problematic to say the least:

• old versions of vagrant included in distro repositories,
• no support for (not so) old versions of vagrant,
• no official PPA so package managers can be setup to do their job correctly
• no plans to support automatic updates (https://github.com/hashicorp/vagrant/issues/1731)
• and no Redirect for the old box locations,

[samrocketman]

I'm not sure it's realistic to ask the vagrant team to go around to package maintainers who they may or may not know. The would probably waste a lot of time which could be better spent on improving Vagrant.

[margyly]

Is this the command to add the geerlingguy/ubuntu1604 box at its new address?

vagrant box add -name geerlingguy/ubuntu1604 https://vagrantcloud.com/geerlingguy/boxes/ubuntu1604/versions/1.1.7/providers/virtualbox.box

[samrocketman]

Not sure about your command but I'd like to note that I resolved all issues by installing vagrant from the vagrantup website (same with virtualbox).

[margyly]

Nope, that didn't do it. Try this:

vagrant init geerlingguy/ubuntu1604 --box-version 1.1.7
vagrant up

[jasonfoy21]

cd ~/.vagrant./boxes//

//Haven't Tested This Method
cat ~/.vagrant.d/boxes/ubuntu-VAGRANTSLASH-xenial64/metadata_url
https://atlas.hashicorp.com/ubuntu/xenial64

//Tested Method
nano metadata_url
replace url https://atlas.hashicorp.com/ubuntu/xenial64 with https://vagrantcloud.com/ubuntu/xenial64

//Fix EOL issue
touch metadata_url2
perl -pe 'chomp if eof' metadata_url > metadata_url2

//Backup and replace
mv metadata_url metadata_url.bak
mv metadata_url2 metadata_url

vagrant box update
enjoy

[remd]

One-liner fix described in https://github.com/hashicorp/vagrant/issues/9442#issuecomment-363126830:

sed -i -- 's/atlas.hashicorp/vagrantcloud/g' ~/.vagrant.d/boxes/*/metadata_url

[NSpencer4]

People still seeing this error before creating a box should try this:
vagrant plugin install vagrant-hostsupdater
Then run:
vagrant up

[ghost]

add Vagrant::DEFAULT_SERVER_URL.replace('https://vagrantcloud.com') in Vagrantfile
this works for me .

[theoinsight]

@jackiilong Nice!

[reedy]

Just installed Vagrant 2.0.3. With the workaround from https://github.com/hashicorp/vagrant/issues/9442#issuecomment-372102515 (we've actually had it in place a little while https://github.com/wikimedia/mediawiki-vagrant/commit/b9535ec8960cc6d3dabe48627c291904a634b730 )

Vagrant::DEFAULT_SERVER_URL.replace('https://vagrantcloud.com')

I get an error

There was an error loading a Vagrantfile. The file being loaded
and the error message are shown below. This is usually caused by
a syntax error.

Path: /Users/reedy/vagrant/Vagrantfile
Line number: 53
Message: RuntimeError: can't modify frozen String

Does that mean we need different handling for 2.0.3 (and newer) vs older versions?

[dentarg]

@reedy Looks like Vagrant 2.0.3 broke your workaround when fixing this:

• https://github.com/hashicorp/vagrant/compare/v2.0.2...v2.0.3#diff-94a61718fdc2808af81dff1661e806a8
• https://github.com/hashicorp/vagrant/commit/f2bf18e56be36ed4bfbfe42fe20ab9147910ab3f

I don't think you need it for 2.0.3 and later tho? Not sure if it is possible to the check current Vagrant version before calling #replace

One way to go about it is to require a minimum Vagrant version in the Vagrantfile: https://www.vagrantup.com/docs/vagrantfile/vagrant_version.html

[remd]

It seems DEFAULT_SERVER_URL is now immutable in 2.0.3. Whether or not that is ideal is probably something to be discussed in another issue.

@reedy As far as #9442 goes, It should not be necessary to do any workaround in 2.0.3.

[reedy]

Unless we can't avoid doing so, I think we will want to avoid limiting versions (to the newest releases) where possible. As many people will install vagrant from apt on debian for example, and won't want to install dpkg's from the vagrant website as it then becomes a maintenance burden for themselves.

So I guess we either need to do some sort of version comparison... And/or checking if the string is frozen.. And then apply the do the Vagrant::DEFAULT_SERVER_URL.replace as appropriate

[reedy]

I came up with this... Feels hacky as hell... :/

unless Vagrant::DEFAULT_SERVER_URL.frozen?
  Vagrant::DEFAULT_SERVER_URL.replace('https://vagrantcloud.com')
end

[alex1x]

 ~/vagrant/concourse  vagrant up                 
Bringing machine 'default' up with 'virtualbox' provider...
==> default: Box 'concourse/lite' could not be found. Attempting to find and install...
    default: Box Provider: virtualbox
    default: Box Version: >= 0
==> default: Loading metadata for box 'concourse/lite'
    default: URL: https://vagrantcloud.com/concourse/lite
==> default: Adding box 'concourse/lite' (v3.2.1) for provider: virtualbox
    default: Downloading: https://vagrantcloud.com/concourse/boxes/lite/versions/3.2.1/providers/virtualbox.box
    default: Download redirected to host: s3.amazonaws.com
An error occurred while downloading the remote file. The error
message, if any, is reproduced below. Please fix this error and try
again.

OpenSSL SSL_read: SSL_ERROR_SYSCALL, errno 54
 ✘  ~/vagrant/concourse  ls -a
.           ..          .vagrant    Vagrantfile
 ~/vagrant/concourse  rm -rf .vagrant 
 ~/vagrant/concourse  vagrant up     
Bringing machine 'default' up with 'virtualbox' provider...
==> default: Box 'concourse/lite' could not be found. Attempting to find and install...
    default: Box Provider: virtualbox
    default: Box Version: >= 0
==> default: Loading metadata for box 'concourse/lite'
    default: URL: https://vagrantcloud.com/concourse/lite
==> default: Adding box 'concourse/lite' (v3.2.1) for provider: virtualbox
    default: Downloading: https://vagrantcloud.com/concourse/boxes/lite/versions/3.2.1/providers/virtualbox.box
==> default: Box download is resuming from prior download progress
    default: Download redirected to host: s3.amazonaws.com
==> default: Successfully added box 'concourse/lite' (v3.2.1) for 'virtualbox'!

¯\_(ツ)_/¯

[vavilen84]

Vagrant 2.0.4 downloaded image successfull without 'not found' error

[samirspatel]

Not sure if this will help anyone but I just nixed the box from my system, which seemed to resolve the issue

vagrant destroy
vagrant box list
vagrant box remove  {boxName}
vagrant up

[Ayush97Didwaniya]

it will fix your error.
run command "vagrant destroy".
use version 5.18 of virtual box.
then install version 1.9.2 of vagrant.
restart your computer.
now go to the directory you want to work in.
"vagrant init ubuntu/trusty64" run command.
"vagrant up" run command.

[matthewpennell]

Update for anyone coming across this thread in the future - after downloading Vagrant version 2.1.2 from their website, the vagrant box update command ran with no complaints, and no issues updating existing boxes.

Specifically it located the metadata in the right place:

==> homestead-7: Loading metadata for box 'https://vagrantcloud.com/laravel/homestead'

Maybe they fixed the issue or added a redirect.

[ramprsad1433]

Hi Team,
I'm unable to install ubuntu os throgh vagrant,I 'm getting 404 error,Please find the below ,I tried 32 & 64 bit same error,Please check and reply ASAP..
prasad@prasad-PC MINGW64 /i/sample1
$ vagrant up
vagrant up
Bringing machine 'default' up with 'virtualbox' provider...
==> default: Box 'ubuntu/trusty32' could not be found. Attempting to find and install...
default: Box Provider: virtualbox
default: Box Version: >= 0
The box 'ubuntu/trusty32' could not be found or
could not be accessed in the remote catalog. If this is a private
box on HashiCorp's Atlas, please verify you're logged in via
vagrant login. Also, please double-check the name. The expanded
URL and error message are shown below:
URL: ["https://atlas.hashicorp.com/ubuntu/trusty32"]
Error: The requested URL returned error: 404 Not Found

[NSpencer4]

Hey @ramprsad1433, try adding the following to your VagrantFile:
config.vm.box = "ubuntu/trusty32"
config.vm.box_url = "https://cloud-images.ubuntu.com/vagrant/trusty/current/trusty-server-cloudimg-i386-vagrant-disk1.box"

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.