Rust: incremental compilation: could not create session directory lock file: No locks available (os error 37)

[szignal]

I was unable to incrementally compile using cargo build.

I had added the rand dependency to the toml file, per the advice of the Guessing Game exercise in Rust Programming Language (Second Edition).

Before this particular error, I had just finished setting up a synced folder with NFS on my vagrant box (ubuntu 16.04) where rust was installed. There is some issue using the default vagrant shared folder setup with rust.

I tried this code:

use std::io;

fn main() {
    println!("Guess the number");

    println!("Please input your guess.");

    let mut guess = String::new();

    io::stdin().read_line(&mut guess)
        .expect("Failed to read line");

    print!("You guessed: {}", guess);
}

I expected to see this happen:

compile

Instead, this happened:

    Updating registry `https://github.com/rust-lang/crates.io-index`
 Downloading rand v0.3.22                                                       
 Downloading libc v0.2.40                                                       
 Downloading rand v0.4.2                                                        
   Compiling libc v0.2.40                                                       
   Compiling rand v0.4.2
   Compiling rand v0.3.22
   Compiling guessing_game v0.1.0 (file:///Rust/projects/guessing_game)
error: incremental compilation: could not create session directory lock file: No locks available (os error 37)

thread 'rustc' panicked at 'librustc/session/mod.rs:727: Trying to get session directory from IncrCompSession `NotInitialized`', librustc/session/mod.rs:1180:26
note: Run with `RUST_BACKTRACE=1` for a backtrace.

error: internal compiler error: unexpected panic

note: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/blob/master/CONTRIBUTING.md#bug-reports

note: rustc 1.25.0 (84203cac6 2018-03-25) running on x86_64-unknown-linux-gnu

error: Could not compile `guessing_game`.

rustc --version --verbose:

rustc 1.25.0 (84203cac6 2018-03-25)
binary: rustc
commit-hash: 84203cac67e65ca8640b8392348411098c856985
commit-date: 2018-03-25
host: x86_64-unknown-linux-gnu
release: 1.25.0
LLVM version: 6.0

Backtrace:

thread 'rustc' panicked at 'librustc/session/mod.rs:727: Trying to get session directory from IncrCompSession `NotInitialized`', librustc/session/mod.rs:1180:26
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
stack backtrace:
   0: std::sys::unix::backtrace::tracing::imp::unwind_backtrace
             at libstd/sys/unix/backtrace/tracing/gcc_s.rs:49
   1: std::sys_common::backtrace::_print
             at libstd/sys_common/backtrace.rs:71
   2: std::panicking::default_hook::{{closure}}
             at libstd/sys_common/backtrace.rs:59
             at libstd/panicking.rs:380
   3: std::panicking::default_hook
             at libstd/panicking.rs:396
   4: std::panicking::rust_panic_with_hook
             at libstd/panicking.rs:576
   5: std::panicking::begin_panic
   6: rustc::session::opt_span_bug_fmt::{{closure}}
   7: rustc::ty::context::tls::with_opt
   8: rustc::session::opt_span_bug_fmt
   9: rustc::session::bug_fmt
  10: rustc::session::Session::incr_comp_session_dir
  11: rustc_incremental::persist::fs::garbage_collect_session_directories
  12: rustc_driver::driver::phase_2_configure_and_expand_inner::{{closure}}
  13: rustc_driver::driver::phase_2_configure_and_expand_inner
  14: rustc_driver::driver::compile_input
  15: rustc_driver::run_compiler

[glaubitz]

This is an issue we are currently seeing on linux-sparc64 as well [1]:

``` Compiling serde v1.0.37
error: incremental compilation: could not create session directory lock file: No such process (os error 3)

error: internal compiler error: unexpected panic

note: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/blob/master/CONTRIBUTING.md#bug-reports

note: rustc 1.24.1 running on sparc64-unknown-linux-gnu

thread 'rustc' panicked at 'src/librustc/session/mod.rs:665: Trying to get session directory from IncrCompSession NotInitialized', src/librustc/session/mod.rs:1141:26
note: Run with RUST_BACKTRACE=1 for a backtrace.

error: Could not compile build_helper.
warning: build failed, waiting for other jobs to finish...
```

I have not yet figured out where this comes from. The same rust version worked fine in the past.

[1] https://buildd.debian.org/status/fetch.php?pkg=rustc&arch=sparc64&ver=1.25.0%2Bdfsg1-1%7Eexp2&stamp=1523052949&raw=0

CC @jrtc27 @psumbera @mkarcher

[michaelwoerister]

NFS does not support locking files very well, I guess. It would be good if there was some way to make Cargo put the incremental folder somewhere else (into /tmp for example).

cc @rust-lang/cargo

[jrtc27]

Although the original report mentions NFS, we're not using NFS for the sparc64 buildds; the two issues have similar but different error messages ("No locks available" vs "No such process") so they could well be different problems.

[michaelwoerister]

@jrtc27, yes, sounds like a different root cause.

[alexcrichton]

In Cargo we disable file locking whenever NFS is detected, although that may not work well for rustc as it's semi-reasonable to expect Cargo to not be run in parallel much on one system but rustc probably is being executed in parallel by Cargo at least.

An alternative to moving the incremental directory is moving the locks for the incremental directory. I've considered this for Cargo where we actually want to enable file locking on NFS directories. That'd require more intrusive support in rustc, though.

[glaubitz]

If anyone wants to reproduce this problem, there are sparc64 porterboxes available in the gcc compile farm, both running Solaris and Linux. Any open source developer can apply for an account:

https://cfarm.tetaneutral.net/machines/list/

[michaelwoerister]

Is this on a file system that should support locks in general?

An alternative to moving the incremental directory is moving the locks for the incremental directory. I've considered this for Cargo where we actually want to enable file locking on NFS directories. That'd require more intrusive support in rustc, though.

As long as we don't want to support two different machines working on the same NFS share simultaneously (which we don't really do anyway), we could indeed just allocate that lock files in /tmp. The only real restriction for lock files is that they mustn't be contained in the session directory that they protect.

[glaubitz]

Is this on a file system that should support locks in general?

It's a standard off-the-shelf installation of Debian on ext4. Nothing special. There is no NFS or anything involved and it affects sparc64 machines with the same configuration as comparable x86_64 machines.

[michaelwoerister]

Interesting. That's unexpected.

[glaubitz]

You can test it yourself on the Linux/sparc64 porterbox available in the gcc compile farm. It's a rather fast machine.

[michaelwoerister]

I've requested an account. Thanks for the tip!

[glaubitz]

I have root privileges on the sparc64 machine (but not the others), so I can install additional packages if required.

[glaubitz]

@michaelwoerister Did you have any success getting an account in the gcc compile farm?

[michaelwoerister]

@glaubitz Yes, I have an account. I haven't gotten around to actually using it though.

[glaubitz]

@michaelwoerister @alexcrichton

@karcherm has had a look at the Rust sources and he found the problem rather quickly. Rust is hard-wiring the constants for the fnctls from glibc assuming they are constant across which they are not.

Compare:

https://github.molgen.mpg.de/git-mirror/glibc/blob/master/sysdeps/unix/sysv/linux/sparc/bits/fcntl.h#L53

and:

https://doc.rust-lang.org/nightly/nightly-rustc/src/rustc_data_structures/flock.rs.html#45

Rust is trying to call "Set Lock" and uses function control number 6. But that's "F_SETOWN" on sparc64 which expects a PID as a parameter which is why the error message is "No such process".

[michaelwoerister]

Awesome find, @karcherm & @glaubitz!

[tpn-station]

I get this when compiling files stored on WSL2, but running cargo on windows. So running on windows and compiling on this path:

\\wsl$\Ubuntu\home\...

[nfwatsoncom]

Ditto from @tpn-station ... I likewise have source on WSL2 (Ubuntu-20.04 w/ native Kernel) filesystem with Ubuntu's root FS mounted as drive "N:" in Windows. To Windows it looks like a network drive. My Rust project is configured for Windows build (as opposed to running CLion in Ubuntu).

I also tried avoiding the "N:" drive-letter mapping and specifying source files directly as "'\\wsl$\Ubuntu-20.04\home..." and hit the same error.

[DrDinosaur]

Same issue on WSL 2 as above.

[daswafford]

Hi, I think I am seeing a related error. I was trying to build the hello world example from https://rust-cli.github.io/book/tutorial/setup.html and when issuing cargo run, get:

[email protected]:/Volumes/dswafford/CODE/rust/hello/grrs$cargo run --verbose
   Compiling grrs v0.1.0 (/Volumes/dswafford/CODE/rust/hello/grrs)
     Running `rustc --crate-name grrs --edition=2018 src/main.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C debuginfo=2 -C metadata=a4d4421a81652fc4 -C extra-filename=-a4d4421a81652fc4 --out-dir /Volumes/dswafford/CODE/rust/hello/grrs/target/debug/deps -C incremental=/Volumes/dswafford/CODE/rust/hello/grrs/target/debug/incremental -L dependency=/Volumes/dswafford/CODE/rust/hello/grrs/target/debug/deps`
error: incremental compilation: could not create session directory lock file: Operation not supported (os error 45)

thread 'rustc' panicked at 'trying to get session directory from `IncrCompSession`: NotInitialized', src/librustc_session/session.rs:690:48
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

error: internal compiler error: unexpected panic

note: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/blob/master/CONTRIBUTING.md#bug-reports

note: rustc 1.44.0 (49cae5576 2020-06-01) running on x86_64-apple-darwin

note: compiler flags: -C debuginfo=2 -C incremental --crate-type bin

note: some of the compiler flags provided by cargo are hidden

error: aborting due to previous error

error: could not compile `grrs`.

Caused by:
  process didn't exit successfully: `rustc --crate-name grrs --edition=2018 src/main.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C debuginfo=2 -C metadata=a4d4421a81652fc4 -C extra-filename=-a4d4421a81652fc4 --out-dir /Volumes/dswafford/CODE/rust/hello/grrs/target/debug/deps -C incremental=/Volumes/dswafford/CODE/rust/hello/grrs/target/debug/incremental -L dependency=/Volumes/dswafford/CODE/rust/hello/grrs/target/debug/deps` (exit code: 101)

I am on a MacOS computer running 10.15.4 and the current working directory where I ran "cargo" run" is an smb mount to a network-based file server (Ubuntu):

[email protected]:/Volumes/dswafford/CODE/rust/hello/grrs$mount
...
//[email protected]/dswafford on /Volumes/dswafford (smbfs, nodev, nosuid, mounted by dswafford)

[Tarnadas]

Current workaround for Windows compiling a project in WSL2:

# powershell
$env:CARGO_INCREMENTAL=0; cargo build

see https://github.com/rust-lang/rust/issues/66513#issuecomment-663205916