Cargo: Failed to mmap

[miedzinski]

I can't install anything with cargo.

$ cargo install --verbose rustfmt
    Updating registry `https://github.com/rust-lang/crates.io-index`
error: failed to fetch `https://github.com/rust-lang/crates.io-index`

Caused by:
  [2/-1] Failed to mmap. Could not write data: Invalid argument

or by putting dependency in Cargo.toml

$ cat Cargo.toml
[package]
name = "test"
version = "0.1.0"
authors = ["test"]

[dependencies]
rustfmt = "0.5"
$ cargo build --verbose
    Updating registry `https://github.com/rust-lang/crates.io-index`
error: Unable to update registry https://github.com/rust-lang/crates.io-index

Caused by:
  failed to fetch `https://github.com/rust-lang/crates.io-index`

Caused by:
  [2/-1] Failed to mmap. Could not write data: Invalid argument

$ rustc --version
rustc 1.9.0 (e4e8b6668 2016-05-18)
$ cargo --version
cargo 0.10.0-nightly (10ddd7d 2016-04-08)

I am running up-to-date Ubuntu 16.04 LTS.

[alexcrichton]

Whoa, haven't seen that one before! Are you perhaps running on a system that's running low on memory? It may be worth running strace as well to take a look at the parameters to the mmap call that's failing.

[miedzinski]

mmap(NULL, 11, PROT_READ|PROT_WRITE, MAP_SHARED, 5, 0) = -1 EINVAL (Invalid argument)

There is also one thing I forgot to mention and it may be the cause. That Ubuntu is inside VirtualBox 5.0.22 on Windows host. I'm using this setup for years and never had any problems, so I haven't thought about mentioning this in first place.

Definitely RAM usage isn't an issue (host or guest).

---

This may be related to #1578. My whole home dir is on vbox shared folder.

$ df -T
Filesystem     Type     1K-blocks      Used Available Use% Mounted on
udev           devtmpfs    490492         0    490492   0% /dev
tmpfs          tmpfs       101592      3272     98320   4% /run
/dev/sda1      ext4      32894832   8020564  23180268  26% /
tmpfs          tmpfs       507956         8    507948   1% /dev/shm
tmpfs          tmpfs         5120         0      5120   0% /run/lock
tmpfs          tmpfs       507956         0    507956   0% /sys/fs/cgroup
/home          vboxsf   524287996 220802040 303485956  43% /home/dominik
tmpfs          tmpfs       101592         0    101592   0% /run/user/1000

I'm not sure whether it's important, but i'm using git from guest without any issues.

---

https://github.com/libgit2/git2go/issues/194#issuecomment-99105962 looks familiar.

[alexcrichton]

Interesting! Could you gist the full syscall log as well? It looks like it's attempting to map whatever file descriptor 5 is, and it may be good to see how that's in play

[miedzinski]

https://paste.ubuntu.com/17997204/

I guess it's /home/dominik/.cargo/registry/index/github.com-88ac128001ac3a9a/.git/objects/pack/pack_git2_iAED2h/

[alexcrichton]

cc @carlosmn, does this sound familiar in libgit2?

It looks like the relevant syscalls leading up to the failure here are:

open("/home/dominik/.cargo/registry/index/github.com-88ac128001ac3a9a/.git/objects/pack/pack_git2_iAED2h", O_RDWR|O_CREAT|O_EXCL, 0600) = 5
[..]
lseek(5, 10, SEEK_SET)                  = 10
write(5, "P", 1)                        = 1
mmap(NULL, 11, PROT_READ|PROT_WRITE, MAP_SHARED, 5, 0) = -1 EINVAL (Invalid argument)

I think that cargo is using libgit2 0.2.23 right now FWIW

[carlosmn]

It looks to be the same issue as the one in the git2go tracker. The VirtualBox filesystem does not like something we're doing, but it's not clear what.

It's possible that it doesn't support files with holes even though it doesn't actually complain about them, or maybe it's just with mmap. Someone reported the tests run fine, which is really odd as it's the same code. It probably doesn't help that it seems to be translating unix calls into Windows filesystem calls, which breaks the assumptions we're making in the library.

@miedzinski can you clone fine if you're on VBox but without a shared home dir?

[miedzinski]

@carlosmn I can use all git commands (including clone) on vboxsf. If you are asking about libgit2 (which git binary doesn't use, right?), then you have to wait few hours until I come back home and try it with cargo.

[miedzinski]

@carlosmn, @alexcrichton, it works flawlessly.

$ cargo install rustfmt
    Updating registry `https://github.com/rust-lang/crates.io-index`
 Downloading rustfmt v0.5.0
# downloading and compiling lots of crates
  Installing /home/libgit/.cargo/bin/rustfmt
  Installing /home/libgit/.cargo/bin/cargo-fmt
$ ~/.cargo/bin/rustfmt -V
0.5.0

This is the same virtual machine, but this time cargo directory is on ext4.

[alexcrichton]

@miedzinski could you try compiling this and see if it runs into the same error when cloning onto a shared directory?

#include <git2.h>
#include <assert.h>

int main() {
  git_repository *repo;
  assert(git_libgit2_init() == 1);
  assert(git_clone(&repo,
                   "https://github.com/rust-lang/crates.io-index",
                   "crates.io-index",
                   NULL) == 0);
}

[miedzinski]

a.out: test.c:10: main: Assertiongit_clone(&repo, "https://github.com/rust-lang/crates.io-index", "crates.io-index", NULL) == 0' failed.`

Built against 0.24.1-2 from Ubuntu repository.

[alexcrichton]

@miedzinski could you try using strace as well to ensure that it's failing with the same mmap error as before?

[miedzinski]

https://paste.ubuntu.com/18054099/

[alexcrichton]

Hm it's not clear from that gist that it's the same mmap error you're hitting with Cargo. It's also doing a different operation with a clone rather than a fetch, so that may be factoring in as well. It may be worth poking around various incantations to see if it can be reproduced with just libgit2 so we can report the bug upstream.

[carlosmn]

That error looks like libgit2 was not built with HTTPS support. Trying with git:// might provide more reliable results.

I've opened libgit2/libgit2#3845 to track this upstream, but I'm not sure that there's a fix that doesn't involve a lot of yak shaving so we don't need to read the data back from disk.

[alexcrichton]

Sounds good to me, thanks @carlosmn!

[miedzinski]

@carlosmn, there is definitely something wrong here. I have changed this code to:

#include <git2.h>
#include <assert.h>

int main() {
  git_repository *repo;
  assert(git_libgit2_init() == 1);
  assert(git_clone(&repo,
                   "git://[email protected]:rust-lang/crates.io-index.git"
                   "crates.io-index",
                   NULL) == 0);
}

and assertion was failing as well. I have tried different URLs (like [email protected]:rust-lang/crates.io-index.git) and after few tries I saw crates.io-index directory (it could have been there since first execution though). And what's weird, I couldn't delete it -- rm was complaining that it can't remove pack_xxx file with error Operation not permitted. I have deleted it straight from Windows, but since then I couldn't make libgit2 to create this directory again.
Like I said few times, official git binary produces valid repositories.

@alexcrichton, is it possible to change cargo directory location? It seems like a good workaround. If not, I can always mount ext4 filesystem on ~/.cargo/.

edit: symlink is sufficient. :)

[carlosmn]

"git://[email protected]:rust-lang/crates.io-index.git"

That's not going to work, since you're aksing for rust-lang to be the port. You're mixing URLs with scp/rsync syntax.

"Operation not permitted" is another error that seems to come from weird interactions between vboxfs and Windows. I'm not sure where the error comes from in that case.

[miedzinski]

It seems you are right. I have found which URL produced these files.

#include <git2.h>
#include <assert.h>

int main() {
  git_repository *repo;
  assert(git_libgit2_init() == 1);
  assert(git_clone(&repo,
                   "git://github.com/rust-lang/crates.io-index",
                   "crates.io-index",
                   NULL) == 0);
}

Unlike https version, this creates crates.io-index directory:

$ tree -a crates.io-index/
crates.io-index/
└── .git
    ├── objects
    │   └── pack
    │       └── pack_git2_Hfr6Me
    └── refs
        ├── heads
        └── tags

6 directories, 1 file

But assertion still fails. This time strace shows that unlink call returns Operation not permitted. I can't remove it from inside virtual machine as well.

# rm -rf crates.io-index/
rm: cannot remove 'crates.io-index/.git/objects/pack/pack_git2_Hfr6Me': Operation not permitted

strace ouput: https://paste.ubuntu.com/18115479/

[alexcrichton]

Oh that looks like it's a reproduction, from the strace logs:

open("/home/dominik/Projects/crates.io-index/.git/objects/pack/pack_git2_7Ak1Xv", O_RDWR|O_CREAT|O_EXCL, 0600) = 5
[..]
lseek(5, 10, SEEK_SET)                  = 10
write(5, "P", 1)                        = 1
mmap(NULL, 11, PROT_READ|PROT_WRITE, MAP_SHARED, 5, 0) = -1 EINVAL (Invalid argument)

[miedzinski]

Well, there is problem with chmods on vboxsf, because Windows doesn't support them. Using chmod (root or normal user) has no effect. My entry in /etc/fstab looks like this:

home /home/dominik vboxsf defaults,uid=dominik,gid=dominik,fmask=077,dmask=077 0 0

Which means all files have chmod 700 inside my $HOME directory.

[lilith]

With Rust 1.11 & cargo 1.12, I get what appears to be a related error:

``cargo search Updating registryhttps://github.com/rust-lang/crates.io-index`
warning: spurious network error (2 tries remaining): [2/-1] Failed to mmap. Could not write data: Invalid argument
warning: spurious network error (1 tries remaining): [2/-1] Failed to mmap. Could not write data: Invalid argument
error: failed to update registry https://github.com/rust-lang/crates.io-index

[splix]

I have same issue with cargo 1.16 and rust 1.15. The only difference that I'm using KVM+QEMU virtual machine, but with shared home folder as well. My host machine is on Debian, as well as guest.
If I change home folder to local, instead of shared, all works fine.

[Nercury]

I think I have hit the same issue on OSX while wanting to be clever and reuse cargo registry between builds on a docker:

docker run \
    -v $(pwd)/build/.cargo:/root/.cargo \
    -v $(pwd):/source jimmycuadra/rust:1.15.0 \
    /bin/bash -c "cargo build --release"

    Updating registry `https://github.com/rust-lang/crates.io-index`
warning: spurious network error (2 tries remaining): [2/-1] Failed to mmap. Could not write data: Invalid argument
warning: spurious network error (1 tries remaining): [2/-1] Failed to mmap. Could not write data: Invalid argument
error: failed to fetch `https://github.com/rust-lang/crates.io-index`

[miedzinski]

This issue is mostly related to vboxsf filesystem. Consider running cargo with strace (look at @alexcrichton comments above) and report to libgit2/libgit2#3845 if it fails the same way on overlayfs (or whatever docker is using right now) or create new issue.

[jml]

@Nercury Did you come up with a clever way of re-using registry without triggering this bug?

[Nercury]

@jml Sorry, no. My project is very small so this is minor for me.

[shepmaster]

Following the threads further, @jml's solution was to use a partial cache of the relevant pieces:

# We want to save the whole of /cargo between runs, but if we do that by
# sharing a mountpoint with the local filesystem, we hit a bug in libgit2 vs
# vboxsf:
#
# - https://github.com/rust-lang/cargo/issues/2808
# - https://github.com/libgit2/libgit2/issues/3845
#
# Instead, share the registry/cache, which is where most of the data is.

docker run ... -v $(REGISTRY_CACHE):/cargo/registry/cache ...

[omarabid]

@shepmaster I'm not getting any speed improvements using it. The cache folder is also empty. Mind that I only have one dependency which is "Rand" so I wonder if the cache is used for other stuff?

[jml]

Sorry haven't had a chance to look into this in a while and don't expect to any time soon. I've lost all mental context.

[carols10cents]

It sounds like there are either workarounds or upstream issues for this, so I'm going to close. Please reopen if I'm incorrect and there's something cargo should be doing here!

[jjyr]

Just have met the same bug in WSL ubuntu-20.

A hint for who has the same issue:

Using local docker volume to map .cargo won't trigger this.

docker run ... -v cargo-cache:/root/.cargo

The .cargo will be cached between different containers as expected.