Rust: 'rustc' panicked at 'failed to acquire jobserver token: ...'

[matthiasbeyer]

This seems to be related to #42867, though not entirely the same.

I am in the process of cargo publishing my crate (multiple crates in a workspace project) imag v0.5.0 (from this commit).

Instead of a normal compile process, I got this:

error: internal compiler error: unexpected panic

note: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/blob/master/CONTRIBUTING.md#bug-reports

note: rustc 1.20.0 (f3d6973f4 2017-08-27) running on x86_64-unknown-linux-gnu

note: run with `RUST_BACKTRACE=1` for a backtrace

thread 'rustc' panicked at 'failed to acquire jobserver token: Error { repr: Os { code: 11, message: "Resource temporarily unavailable" } }', /checkout/src/libcore/result.rs:860:4
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
stack backtrace:
   0: std::sys::imp::backtrace::tracing::imp::unwind_backtrace
   1: std::sys_common::backtrace::_print
   2: std::panicking::default_hook::{{closure}}
   3: std::panicking::default_hook
   4: std::panicking::rust_panic_with_hook
   5: std::panicking::begin_panic_new
   6: std::panicking::begin_panic_fmt
   7: rust_begin_unwind
   8: core::panicking::panic_fmt
   9: core::result::unwrap_failed
  10: rustc_trans::back::write::execute_work
  11: rustc_trans::back::write::run_passes
  12: rustc_driver::driver::phase_5_run_llvm_passes
  13: rustc_driver::driver::compile_input
  14: rustc_driver::run_compiler

error: Could not compile `serde_derive_internals`.
warning: build failed, waiting for other jobs to finish...
error: failed to verify package tarball

Caused by:
  build failed

rustc --version --verbose:

rustc 1.20.0 (f3d6973f4 2017-08-27)
binary: rustc
commit-hash: f3d6973f41a7d1fb83029c9c0ceaf0f5d4fd7208
commit-date: 2017-08-27
host: x86_64-unknown-linux-gnu
release: 1.20.0
LLVM version: 4.0

[matthiasbeyer]

I experienced this error now multiple times. Maybe it is a good idea to include the complete build output here:

    Updating registry `https://github.com/rust-lang/crates.io-index`
warning: manifest has no documentation, homepage or repository.
See http://doc.crates.io/manifest.html#package-metadata for more info.
   Packaging libimagentryannotation v0.5.0 (file:///home/m/archive/development/rust/imag/lib/entry/libimagentryannotation)
   Verifying libimagentryannotation v0.5.0 (file:///home/m/archive/development/rust/imag/lib/entry/libimagentryannotation)
    Updating registry `https://github.com/rust-lang/crates.io-index`
 Downloading libimagentrylink v0.5.0
   Compiling num-traits v0.1.41
   Compiling cfg-if v0.1.2
   Compiling unicode-xid v0.0.4
   Compiling rustc-demangle v0.1.5
   Compiling glob v0.2.11
   Compiling itoa v0.3.4
   Compiling boolinator v2.4.0
   Compiling dtoa v0.4.2
   Compiling semver-parser v0.7.0
   Compiling lazy_static v0.2.11
   Compiling cc v1.0.3
   Compiling is-match v0.1.0
   Compiling libc v0.2.34
   Compiling serde v1.0.25
   Compiling lazy_static v1.0.0
   Compiling same-file v0.1.3
   Compiling percent-encoding v1.0.1
   Compiling gcc v0.3.54
   Compiling quote v0.3.15
   Compiling rustc-serialize v0.3.24
   Compiling either v1.4.0
   Compiling version v2.0.1
   Compiling utf8-ranges v1.0.0
   Compiling log v0.3.8
   Compiling regex-syntax v0.4.1
   Compiling unicode-normalization v0.1.5
   Compiling ansi_term v0.10.2
   Compiling void v1.0.2
   Compiling matches v0.1.6
   Compiling synom v0.11.3
   Compiling walkdir v1.0.7
   Compiling semver v0.8.0
   Compiling memchr v2.0.1
   Compiling time v0.1.38
   Compiling rand v0.3.18
   Compiling itertools v0.7.4
   Compiling unreachable v1.0.0
   Compiling unicode-bidi v0.3.4
   Compiling libimagerror v0.5.0
   Compiling syn v0.11.11
   Compiling backtrace-sys v0.1.16
   Compiling aho-corasick v0.6.4
   Compiling rust-crypto v0.2.36
   Compiling thread_local v0.3.5
   Compiling tempfile v2.2.0
   Compiling idna v0.1.4
   Compiling regex v0.2.3
   Compiling url v1.6.0
   Compiling toml v0.4.5
   Compiling serde_json v1.0.8
   Compiling serde_derive_internals v0.18.1
   Compiling backtrace v0.3.4
   Compiling error-chain v0.11.0
error: internal compiler error: unexpected panic

note: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/blob/master/CONTRIBUTING.md#bug-reports

note: rustc 1.20.0 (f3d6973f4 2017-08-27) running on x86_64-unknown-linux-gnu

note: run with `RUST_BACKTRACE=1` for a backtrace

thread 'rustc' panicked at 'failed to acquire jobserver token: Error { repr: Os { code: 11, message: "Resource temporarily unavailable" } }', /checkout/src/libcore/result.rs:860:4
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
stack backtrace:
   0: std::sys::imp::backtrace::tracing::imp::unwind_backtrace
   1: std::sys_common::backtrace::_print
   2: std::panicking::default_hook::{{closure}}
   3: std::panicking::default_hook
   4: std::panicking::rust_panic_with_hook
   5: std::panicking::begin_panic_new
   6: std::panicking::begin_panic_fmt
   7: rust_begin_unwind
   8: core::panicking::panic_fmt
   9: core::result::unwrap_failed
  10: rustc_trans::back::write::execute_work
  11: rustc_trans::back::write::run_passes
  12: rustc_driver::driver::phase_5_run_llvm_passes
  13: rustc_driver::driver::compile_input
  14: rustc_driver::run_compiler

error: Could not compile `serde_derive_internals`.
warning: build failed, waiting for other jobs to finish...
error: internal compiler error: unexpected panic

note: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/blob/master/CONTRIBUTING.md#bug-reports

note: rustc 1.20.0 (f3d6973f4 2017-08-27) running on x86_64-unknown-linux-gnu

note: run with `RUST_BACKTRACE=1` for a backtrace

thread 'rustc' panicked at 'failed to acquire jobserver token: Error { repr: Os { code: 11, message: "Resource temporarily unavailable" } }', /checkout/src/libcore/result.rs:860:4
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
stack backtrace:
   0: std::sys::imp::backtrace::tracing::imp::unwind_backtrace
   1: std::sys_common::backtrace::_print
   2: std::panicking::default_hook::{{closure}}
   3: std::panicking::default_hook
   4: std::panicking::rust_panic_with_hook
   5: std::panicking::begin_panic_new
   6: std::panicking::begin_panic_fmt
   7: rust_begin_unwind
   8: core::panicking::panic_fmt
   9: core::result::unwrap_failed
  10: rustc_trans::back::write::execute_work
  11: rustc_trans::back::write::run_passes
  12: rustc_driver::driver::phase_5_run_llvm_passes
  13: rustc_driver::driver::compile_input
  14: rustc_driver::run_compiler

error: Could not compile `toml`.
warning: build failed, waiting for other jobs to finish...
error: failed to verify package tarball

Caused by:
  build failed

[matthiasbeyer]

With rustc 1.22.1 I get this error in a different form:

thread '<unnamed>' panicked at 'failed to acquire jobserver token: Resource temporarily unavailable (os error 11)', /checkout/src/librustc_trans/back/write.rs:1619:28
note: Run with `RUST_BACKTRACE=1` for a backtrace.
error: failed to acquire jobserver token: Resource temporarily unavailable (os error 11)

error: Could not compile `serde_json`.
warning: build failed, waiting for other jobs to finish...
thread '<unnamed>' panicked at 'failed to acquire jobserver token: Resource temporarily unavailable (os error 11)', /checkout/src/librustc_trans/back/write.rs:1619:28
note: Run with `RUST_BACKTRACE=1` for a backtrace.
error: failed to acquire jobserver token: Resource temporarily unavailable (os error 11)

error: Could not compile `toml`.
warning: build failed, waiting for other jobs to finish...
thread '<unnamed>' panicked at 'failed to acquire jobserver token: Resource temporarily unavailable (os error 11)', /checkout/src/librustc_trans/back/write.rs:1619:28
note: Run with `RUST_BACKTRACE=1` for a backtrace.
error: failed to acquire jobserver token: Resource temporarily unavailable (os error 11)

error: Could not compile `backtrace`.
warning: build failed, waiting for other jobs to finish...
thread '<unnamed>' panicked at 'failed to acquire jobserver token: Resource temporarily unavailable (os error 11)', /checkout/src/librustc_trans/back/write.rs:1619:28
note: Run with `RUST_BACKTRACE=1` for a backtrace.
error: failed to acquire jobserver token: Resource temporarily unavailable (os error 11)

error: Could not compile `serde_derive`.
warning: build failed, waiting for other jobs to finish...
error: failed to verify package tarball

Caused by:
  build failed

[matthiasbeyer]

For the record: cargo build --release works on all crates where this bug was triggered (though I did not test all of them, just some).

[Ralith]

For whatever reason, this is making it effectively impossible for me to package projects with lots of dependencies for Nix.

[bennofs]

I can reliably hit this bug when compiling my project (https://github.com/bennofs/nix-index) with Nix

[bennofs]

I believe this is a bug in our rustc package in nixpkgs, we appear to build it against jobserver 0.1.6 which is bugged: https://nix-cache.s3.amazonaws.com/log/j4n6lk9fl5ngby7cvhxm5jys8zz62xf1-rustc-1.22.1.drv. Maybe it would be a good idea to have a hard dependency on jobserver >= 0.1.8 for rustc?

[bennofs]

There is an issue here: rustc version 1.22.1 has jobserver-0.1.6 in its lockfile, and the nix build appears to reliably trigger this bug while compiling 1.23.0. So we have a problem in nixpkgs that we cannot bootstrap 1.23.0 using the binary distribution from rust-lang.org.

@alexcrichton you are the author of jobserver-rs, is there something we could do to workaround this bug for bootstrapping 1.23.0? Or is it necessary to use a patched 1.22.1 for compiling 1.23.0?

[alexcrichton]

@bennofs the workaround would be to remove any ambient jobservers and let cargo/rustc manage it, if any

[Ralith]

Can't we bootstrap using 1.23.0 binaries?

[arianvp]

I can reliably hit this bug as well in nix

[kevincox]

I can reliably hit this bug using entr -r.

% entr -r cargo test <<</tmp/file
   Compiling proc-macro2 v0.4.26
   Compiling cc v1.0.28
   Compiling libc v0.2.48
   Compiling unicode-xid v0.1.0
   Compiling autocfg v0.1.2
   Compiling serde v1.0.85
   Compiling semver-parser v0.7.0
   Compiling ryu v0.2.7
thread '<unnamed>' panicked at 'failed to acquire jobserver token: early EOF on jobserver pipe', src/librustc_codegen_ssa/back/write.rs:1348:29
note: Run with `RUST_BACKTRACE=1` environment variable to display a backtrace.
error: failed to acquire jobserver token: early EOF on jobserver pipe

error: aborting due to previous error

thread '<unnamed>' panicked at 'failed to acquire jobserver token: early EOF on jobserver pipe', src/librustc_codegen_ssa/back/write.rs:1348:29
note: Run with `RUST_BACKTRACE=1` environment variable to display a backtrace.
error: failed to acquire jobserver token: early EOF on jobserver pipe

error: Could not compile `unicode-xid`.
warning: build failed, waiting for other jobs to finish...
error: aborting due to previous error

thread '<unnamed>' panicked at 'failed to acquire jobserver token: early EOF on jobserver pipe', src/librustc_codegen_ssa/back/write.rs:1348:29
note: Run with `RUST_BACKTRACE=1` environment variable to display a backtrace.
error: Could not compile `libc`.
warning: build failed, waiting for other jobs to finish...
...

As far as I am aware I don't have any jobservers running.

[kevincox]

I realized that the following issue was only occurring on nightly. However after the latest update this is now occurring on stable as well.

% cargo version
cargo 1.33.0 (f099fe94b 2019-02-12)
% rustc --version
rustc 1.33.0 (2aa4c46cf 2019-02-28)

[davidnorthetal]

still persists using entr
seems to happen because of this: https://bitbucket.org/eradman/entr/src/61cf1590a414110e43ef365a5c06d0a8e36396b2/entr.c?at=default&fileviewer=file-view-default#entr.c-406

[kevincox]

@alexcrichton Can you please take another look at this. It appears that your diagnosis is wrong and many people are experiencing this issue.

There is no workaround as far as I can tell and this is seriously workflow disrupting.

[alexcrichton]

A misconfigured jobserver can still cause issues in rustc, and this error is indicative of a misconfigured jobserver in one way or another. This indicates an environmental problem rather than a problem in rustc AFAIK. If someone else manages to fix rustc we can of course merge, but I don't personally believe there's much we can do in rustc for this.

[kevincox]

There is no other jobserver in my environment. I have checked numerous ways and am highly confident.

[AlvaroBernalG]

@alexcrichton I am also getting the same error as @kevincox when running entr with the reload flag.

find . -name "*.rs" | entr -rs 'cargo run'

Throws the following error:

error: failed to acquire jobserver token: early EOF on jobserver pipe

According to the entr documentation whats happening is :

1) immediately start the server
2) block until any of the listed files change
3) terminate the background process
4) wait for the server to exit before restarting

[davidnorthetal]

I created an upstream issue for entr as well - it surrounds entr closing STDIN, which was added in later entr versions for compatibility with some programs reasons - seems it then breaks with rustc.

https://bitbucket.org/eradman/entr/issues/81/restart_ops-closing-of-stdin-causes

[CleanCut]

It sounds like the underlying issue is https://github.com/rust-lang/rust/issues/60447

[dbohdan]

A workaround for those using entr:

find Cargo.* src/ static/ ... -type f | entr -r sh -c 'cargo run < /dev/null'

It works for me with rustc and cargo 1.40.0 on Linux.

[kevincox]

This is no longer happening for me. I don't know what has changed but I don't see this anymore.

% cargo --version
cargo 1.42.0 (86334295e 2020-01-31

[CleanCut]

It still happens for me with rustc 1.42.0 (b8cedc004 2020-03-09) and cargo 1.42.0 (86334295e 2020-01-31). 😢 Just tested it with cargo run <&- and got a panic starting with the same failed to acquire jobserver token... bit.

[petrochenkov]

So, I've found an entirely legitimate way to trigger this error (the "early EOF on jobserver pipe" case) on new versions of make, without having stdin closed or anything other exotic setup.

If you call cargo from an outer make-based build system

# Assume we are using `cmake`
add_custom_target(
    ...
    COMMAND cargo build
    ...
)

then cargo will receive file descriptors for the jobserver pipe through an env var called MAKEFLAGS.

Except that new versions of make will mark those descriptors as FD_CLOEXEC so they will be either dead or, worse, replaced with other random files when cargo tries to access them :D

The solution is to prevent make from closing the pipe by using string $(MAKE) anywhere in the command.
(Yep, that's how make works - https://www.gnu.org/software/make/manual/make.html#How-the-MAKE-Variable-Works.)

# Assume we are using `cmake`
add_custom_target(
    ...
    COMMAND SNEAK_DOLLAR_MAKE_INTO_THE_COMMAND=$(MAKE) cargo build
    ...
)

This is silly, but it seems like make doesn't leave any other way to inherit jobserver (at least not readily available from higher level makefile generators).