Rust: Rust source make install fails due to missing vendor folder

[itarato]

I was building Rust (branch master) from source. ./configure and make completed without any problems, however sudo make install immediately fails with the following problem:

sudo make install
info: looks like you are running this command under `sudo`
      and so in order to preserve your $HOME this will now
      use vendored sources by default. Note that if this
      does not work you should run a normal build first
      before running a command like `sudo make install`
error: failed to load source for a dependency on `toml`

Caused by:
  Unable to update registry https://github.com/rust-lang/crates.io-index

Caused by:
  failed to update replaced source `registry https://github.com/rust-lang/crates.io-index`

Caused by:
  failed to read root of directory source: /mnt/shared/Checkout/rust/src/vendor

To learn more, run the command again with --verbose.
Makefile:71: recipe for target 'install' failed
make: *** [install] Error 101

The error: failed to load source for a dependency on 'toml' line is complaining about seemingly rangom things, such as cmake, toml or getopts. However the last line (failed to read root of directory source: /mnt/shared/Checkout/rust/src/vendor) is odd as it doesn't have any src/vendor folder and this kind of error is more likely on Rust projects using cargo.toml.

For reference, my build env:

• os: Ubuntu 16.04
• clang 3.8
• python 2.7
• make 4.1
• cmake 3.5.1
• curl 7.47
• git 2.7

I also have already installed via the official rustup script.

So far no luck getting closer to what would cause this. Do you have any idea where to look for?

[retep998]

As that info message quite helpfully mentioned, because you're doing that command under sudo, it will not fetch sources from the internet but instead rely on vendored sources. In order to not clog up this repository with all those vendored sources however, those sources are only vendored into the folder when requested. So you'll just have to call the appropriate command to tell the build system to vendor the sources and then you can do your sudo make install. Alternatively, you could realize that globally installing a Rust toolchain when you're using Rustup is probably a bad idea.

[itarato]

Thanks @retep998 . I've uninstalled the default rust installation (from rustup) are restarted the whole process and unfortunately ended up with the same error. If I run make install without sudo it fails due to permission issues in /usr/local. With sudo make install I go back to my original problem.
I roughly see what you're saying but I don't get what you mean by

you'll just have to call the appropriate command to tell the build system to vendor the sources

What is this command you're referring to?

Appreciate the help.

[itarato]

Without sudo I could compile if I set the destination dir, however I'm still curious of that command in order to be able to do it with sudo.

[retep998]

I believe it might be x.py dist which vendors the stuff so that it can create a tarball.

[itarato]

Appreciate @retep998 found it however I need to to do some digging how to call it in order to generate the missing vendor files. Thanks.

[retep998]

cc @alexcrichton

[est31]

Note that if you get the source code from the official tarballs, the vendor directory is populated: https://static.rust-lang.org/dist/2017-02-26/rust-src-nightly.tar.gz

src/vendor is just not checked into git, to not pollute the history as @retep998 said.

Of course, one could argue whether this is a bug or not, whether configure is supposed to access the internet, or whether having tools rely on the internet is a good thing, but we could certainly have a better error message, and possibly have a option to rely on the internet via the configure/make interface as well.

[steveklabnik]

/cc @eddyb who implemented this behavior

[eddyb]

@steveklabnik The sudo behavior predates my changes, seems that make install needs more things built than make, even if it really shouldn't. cc @rust-lang/tools

@itarato Does ./x.py dist --install work (without sudo)?

[itarato]

@eddyb I did a few tries with make install and ./x.py dist --install, both without sudo, and it works if I'm explicitly selecting a path with configure --prefix= or make DESTDIR=.. install. I assume they'd fail if I keep them on the default install dir (/usr/local).
This is fine for me as I both have the source in git (for contribution) and can compile. As well as in the meantime I've found https://github.com/rust-lang/rust/blob/master/CONTRIBUTING.md that explaines a bit more about the build process.
However my ocd is still troubled by the sudo make install not working (due to the missing vendor folder - which wasn't generated in any cases I mentioned at the beginning). It would be nice to have some explanation on how that scenario would work out. I'm happy to run some tests if you have ideas. Thanks again.

[eddyb]

It's not supposed to build anything make didn't, AFAIK, in order to not trigger this exact issue.
The vendor folder is not supposed to be there, nor should it be needed.

[alexcrichton]

@eddyb the guards here are very much intentional to prevent Cargo from spraying root-owned files in CARGO_HOME.

Without these guards there's a very high risk that Cargo decides to download crates from crates.io and cache them in CARGO_HOME, messing with the permissions there. The original thinking was that the vendored source code was always available so we'd just switch to using that.

@itarato thanks for the report! This is definitely a bug that we should fix.

[eddyb]

Oh, so no precautions were taken for sudo make install to not need to run cargo if make succeeded?

[alexcrichton]

The "easiest" fix for this is to somehow just detect that a normal make happened first which means that we're unlikely (not guaranteed) to not download anything into CARGO_HOME.

@eddyb precautions have always been taking on sudo make install regardless of the build dir in the past.

[eddyb]

@alexcrichton Can't we trigger it on the first cargo command?

EDIT: nevermind, we run cargo build to confirm the targets are built, don't we :/
Still, can't we freeze or something? Even without a vendor directory?

[alexcrichton]

@eddyb oh yes that's true, we could pass --frozen. That's not a bulletproof solution either but it could get the job done.

[char16t]

As I seen, command

$ ./x.py build && sudo ./x.py dist --install

return error

Copying stage2 rustc from stage1 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu)
Build completed successfully in 1:34:34
[sudo] password for user: 
Sorry, try again.
[sudo] password for user: 
info: looks like you are running this command under `sudo`
      and so in order to preserve your $HOME this will now
      use vendored sources by default. Note that if this
      does not work you should run a normal build first
      before running a command like `sudo make install`
error: failed to load source for a dependency on `toml`

Caused by:
  Unable to update registry https://github.com/rust-lang/crates.io-index

Caused by:
  failed to update replaced source `registry https://github.com/rust-lang/crates.io-index`

Caused by:
  failed to read root of directory source: /home/user/Sources/rust/src/vendor

To learn more, run the command again with --verbose.
Build completed unsuccessfully in 0:00:00

And next attempts returns:

➜  rust git:(master) ./x.py build && sudo ./x.py dist --install
Traceback (most recent call last):
  File "./x.py", line 20, in <module>
    bootstrap.main()
  File "/home/user/Sources/rust/src/bootstrap/bootstrap.py", line 602, in main
    bootstrap()
  File "/home/user/Sources/rust/src/bootstrap/bootstrap.py", line 577, in bootstrap
    shutil.rmtree('.cargo')
  File "/usr/lib/python2.7/shutil.py", line 252, in rmtree
    onerror(os.remove, fullname, sys.exc_info())
  File "/usr/lib/python2.7/shutil.py", line 250, in rmtree
    os.remove(fullname)
OSError: [Errno 13] Permission denied: '.cargo/config'

Command (without sudo) works after manually changed user and group for .cargo directory

$ ./x.py build && ./x.py dist --install

[eddyb]

@alexcrichton Uh oh I think this issue slipped, it should've gotten tagged and assigned :(

[char16t]

If it need I can create a new issue.

[cbcrouch]

Is there an easy workaround for gathering the vendor directory that corresponds with a given revision?

[alexcrichton]

@cbcrouch one possible workaround is:

cd src
cargo install cargo-vendor
cargo vendor

[est31]

@cbcrouch you can try to do ./x.py dist src

[est31]

Hmm, is this really the officially endorsed way of building rust?

[whitequark]

@est31 as far as I can tell it is and it used to work, but it got broken during rustbuild migration and no one cared enough to fix it since then

[timmyjose]

Has this issue been fixed yet? I get the exact problem when following the build steps mentioned in the README. I am currently on a macOS Sierra box.

EDIT: Okay, so this is what worked for me in the end:

$ cd rust
$ sudo chown $USER .cargo
$ ./x.py build && ./x.py install

[WawerOS]

I got the same problem when I followed the README on xubuntu and @timmyjose 's solution worked.

[steveklabnik]

Tagging as T-doc because it seems to be a docs issue.

[virtuoushub]

I ran into this issue today on macOS. @timmyjose's workaround fixed it.

~ ❯❯❯ sw_vers
ProductName:    Mac OS X
ProductVersion: 10.11.6
BuildVersion:   15G1611

[aclarry]

I, too am suffering from this problem. Changing the ownership on the .cargo directory to non-root doesn't help (and as soon as I run sudo ./x.py install or sudo ./x.py build, it changes ownership back to root). I still get

me@myplace rust $ sudo ./x.py install
info: looks like you are running this command under `sudo`
      and so in order to preserve your $HOME this will now
      use vendored sources by default. Note that if this
      does not work you should run a normal build first
      before running a command like `sudo make install`
Updating submodules
error: failed to load source for a dependency on `cc`

Caused by:
  Unable to update registry `https://github.com/rust-lang/crates.io-index`

Caused by:
  failed to update replaced source registry `https://github.com/rust-lang/crates.io-index`

Caused by:
  failed to read root of directory source: /home/andrew/Projects/software-projects/systems/rust/src/vendor

Caused by:
  No such file or directory (os error 2)
failed to run: /home/andrew/Projects/software-projects/systems/rust/build/x86_64-unknown-linux-gnu/stage0/bin/cargo build --manifest-path /home/andrew/Projects/software-projects/systems/rust/src/bootstrap/Cargo.toml --frozen
Build completed unsuccessfully in 0:00:01

Running without sudo, I get

me@myplace rust $ ./x.py install
Updating submodules
    Finished <things> ...
    ...
Finished release [optimized] target(s) in 0.0 secs
Install docs stage2 (Some("x86_64-unknown-linux-gnu"))
touch: cannot touch '/usr/local/lib/rust-install-probe': Permission denied
install: error: can't write to destination. consider `sudo`.


command did not execute successfully: "sh" "/home/andrew/Projects/software-projects/systems/rust/build/tmp/dist/rust-docs-1.24.0-dev-x86_64-unknown-linux-gnu/install.sh" "--prefix=/usr/local" "--sysconfdir=/etc" "--docdir=/usr/local/share/doc/rust" "--bindir=/usr/local/bin" "--libdir=/usr/local/lib" "--mandir=/usr/local/share/man" "--disable-ldconfig"
expected success, got: exit code: 1

Rust builds fine: I have an up to date rustc 1.24.0 built in the builds/ directory. I'd just wanted to be able to get a fresh system install of rust seamlessly from source, but this in particular is making it much more painful than my last source install of 1.13 :( My current workaround will just be to add the build directory to my $PATH, but I'd like to get this working right.

Unless someone here has a specific fix, I might try digging into the build to see what could be going wrong. Despite it seeming semi-resolved/workarounded, it seems pretty clear to me there's still some sort of problem here.

[CodeSandwich]

I'm using Ubuntu 17.10 and none of the above solutions worked for me. I've found a simple workaround, which solves all the issues I've faced:

• sudo ./x.py install can't update registry
• ./x.py install can't write to /usr/local
• Conflict between Rustup rustc and the built one
• Go to Rust source directory
• Rename config.toml.example to config.toml
• In config.toml, in section install set entry prefix = "~/.rustup/toolchains/dev"
• Run ./x.py build && ./x.py install
• Run rustup default dev

Now all the Rust tools use custom built rustc, you can even use cargo. Rebuilding requires only running ./x.py build && ./x.py install.

It seems, that Rustup is smart enough to use tools from last used default toolchain if they are not present among built ones, like cargo or rust-fmt. Make sure, that last default toolchain is the newest nightly.

[dtolnay]

@CodeSandwich's https://github.com/rust-lang/rust/issues/40108#issuecomment-368269981 worked the best for me on Ubuntu -- except that ~ was not accepted for me in prefix. It would report "could not canonicalize" and "No such file or directory" even when those directories exist relative to my home directory.

$ ./x.py install
Updating only changed submodules
Submodules updated in 0.03 seconds
    Finished dev [unoptimized] target(s) in 0.26s
thread 'main' panicked at 'could not canonicalize ~/.rustup/toolchains/dev: Os { code: 2, kind: NotFound, message: "No such file or directory" }', libcore/result.rs:945:5
note: Run with `RUST_BACKTRACE=1` for a backtrace.
failed to run: /rust/build/bootstrap/debug/bootstrap install
Build completed unsuccessfully in 0:00:01

Providing an absolute path to prefix is working fine.

[DevQps]

I guess we can close this issue now since #59334 got merged right? (@ewk Thanks for your work!)

[steveklabnik]

Yep, thanks!