Nixpkgs: package: update vdirsyncer to account for the new rust dependancy

Created on 25 Dec 2017 · 14Comments · Source: NixOS/nixpkgs

Issue description

I am having some issues with vdirsyncer https://github.com/pimutils/vdirsyncer/issues/691, the author asked me to test the latest version so I looked into upgrading the package but the setup.py seems to compile a rust shared library so I am not sure how to tackle that (and I don't know rust):
https://github.com/pimutils/vdirsyncer/blob/master/setup.py#L41

I believe it would be best if nix could build the rust dependancy separately but I am not sure enough to actually ask vdirsyncer author to change that just for nixos.

@matthiasbeyer @jgeerds

Source

teto

Most helpful comment

I am working on this, it appears to be that our rust build is using an outdated version of the jobserver crate which had this bug.

bennofs on 31 Jan 2018

🎉5

All 14 comments

I'm on it.

matthiasbeyer on 25 Dec 2017

❤2

Ok. This will get messy and ugly... I doubt that I will succeed in this, especially as I do not have the resources to build rustc from source right now. Maybe @jgeerds can come up with a good idea.

matthiasbeyer on 25 Dec 2017

It's super hacky but at least this manages to build - https://github.com/NixOS/nixpkgs/compare/master...pbogdan:pr_33052?expand=1 (based on the referenced PR).
I don't actually use the package myself so not quite sure how to test it doesn't fail at runtime. Also the current version of rustc from nixpkgs seems to panic a fair amount when building so the build may need to be restarted a few times before it succeeds :-(.
@teto perhaps at the very least that would let you test if the new version fixes your issue? (assuming it doesn't fail at runtime because of my hacks).

pbogdan on 25 Dec 2017

Thanks for the help boys !

I tried the patch which fails on PATH="$HOME/.cargo/bin/:$PATH". I guess this part could be removed or replaced with $out/bin

$ nix-shell -p vdirsyncer                                                                                                                                                                                                                               ~/nixpkgs
these derivations will be built:
  /nix/store/wg6laqc5zlxil69w1n8wlshbig1q22i3-vdirsyncer-0.17.0a1.drv
building path(s) ‘/nix/store/zay1w7lach4gznhnxca9ji7idsz9ih0q-vdirsyncer-0.17.0a1’
bash: warning: setlocale: LC_ALL: cannot change locale (en_US.utf8)
unpacking sources
unpacking source archive /nix/store/2krb64npcsaix238ki403sxd7vv2n032-vdirsyncer-0.17.0a1.tar.gz
source root is vdirsyncer-0.17.0a1
setting SOURCE_DATE_EPOCH to timestamp 1512415031 of file vdirsyncer-0.17.0a1/vdirsyncer/version.py
patching sources
configuring
building
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.6
creating build/lib.linux-x86_64-3.6/vdirsyncer
copying vdirsyncer/__main__.py -> build/lib.linux-x86_64-3.6/vdirsyncer
copying vdirsyncer/version.py -> build/lib.linux-x86_64-3.6/vdirsyncer
copying vdirsyncer/utils.py -> build/lib.linux-x86_64-3.6/vdirsyncer
copying vdirsyncer/native.py -> build/lib.linux-x86_64-3.6/vdirsyncer
copying vdirsyncer/vobject.py -> build/lib.linux-x86_64-3.6/vdirsyncer
copying vdirsyncer/exceptions.py -> build/lib.linux-x86_64-3.6/vdirsyncer
copying vdirsyncer/metasync.py -> build/lib.linux-x86_64-3.6/vdirsyncer
copying vdirsyncer/_native__ffi.py -> build/lib.linux-x86_64-3.6/vdirsyncer
copying vdirsyncer/repair.py -> build/lib.linux-x86_64-3.6/vdirsyncer
copying vdirsyncer/http.py -> build/lib.linux-x86_64-3.6/vdirsyncer
copying vdirsyncer/_native.py -> build/lib.linux-x86_64-3.6/vdirsyncer
copying vdirsyncer/__init__.py -> build/lib.linux-x86_64-3.6/vdirsyncer
creating build/lib.linux-x86_64-3.6/vdirsyncer/storage
copying vdirsyncer/storage/memory.py -> build/lib.linux-x86_64-3.6/vdirsyncer/storage
copying vdirsyncer/storage/dav.py -> build/lib.linux-x86_64-3.6/vdirsyncer/storage
copying vdirsyncer/storage/base.py -> build/lib.linux-x86_64-3.6/vdirsyncer/storage
copying vdirsyncer/storage/google.py -> build/lib.linux-x86_64-3.6/vdirsyncer/storage
copying vdirsyncer/storage/singlefile.py -> build/lib.linux-x86_64-3.6/vdirsyncer/storage
copying vdirsyncer/storage/filesystem.py -> build/lib.linux-x86_64-3.6/vdirsyncer/storage
copying vdirsyncer/storage/http.py -> build/lib.linux-x86_64-3.6/vdirsyncer/storage
copying vdirsyncer/storage/etesync.py -> build/lib.linux-x86_64-3.6/vdirsyncer/storage
copying vdirsyncer/storage/__init__.py -> build/lib.linux-x86_64-3.6/vdirsyncer/storage
creating build/lib.linux-x86_64-3.6/vdirsyncer/cli
copying vdirsyncer/cli/utils.py -> build/lib.linux-x86_64-3.6/vdirsyncer/cli
copying vdirsyncer/cli/tasks.py -> build/lib.linux-x86_64-3.6/vdirsyncer/cli
copying vdirsyncer/cli/discover.py -> build/lib.linux-x86_64-3.6/vdirsyncer/cli
copying vdirsyncer/cli/fetchparams.py -> build/lib.linux-x86_64-3.6/vdirsyncer/cli
copying vdirsyncer/cli/config.py -> build/lib.linux-x86_64-3.6/vdirsyncer/cli
copying vdirsyncer/cli/__init__.py -> build/lib.linux-x86_64-3.6/vdirsyncer/cli
creating build/lib.linux-x86_64-3.6/vdirsyncer/sync
copying vdirsyncer/sync/status.py -> build/lib.linux-x86_64-3.6/vdirsyncer/sync
copying vdirsyncer/sync/exceptions.py -> build/lib.linux-x86_64-3.6/vdirsyncer/sync
copying vdirsyncer/sync/__init__.py -> build/lib.linux-x86_64-3.6/vdirsyncer/sync
[ "$READTHEDOCS" != "True" ] || make install-rust
cd ./rust && PATH="$HOME/.cargo/bin/:$PATH" cargo build --release
error: failed to open: /homeless-shelter/.cargo/registry/index/github.com-1ecc6299db9ec823/.cargo-index-lock

Caused by:
  Permission denied (os error 13)
make: *** [Makefile:146: rust-ext] Error 101
builder for ‘/nix/store/wg6laqc5zlxil69w1n8wlshbig1q22i3-vdirsyncer-0.17.0a1.drv’ failed with exit code 2
error: build of ‘/nix/store/wg6laqc5zlxil69w1n8wlshbig1q22i3-vdirsyncer-0.17.0a1.drv’ failed

teto on 26 Dec 2017

Was that from my branch? I thought https://github.com/NixOS/nixpkgs/compare/master...pbogdan:pr_33052?expand=1#diff-fa94f0db0527b004ae3809fa214083a0R40 ought to prevent that and the rust part would be built separately.

pbogdan on 26 Dec 2017

Right my bad. I freaked out when I clicked on your link that proposed to create a new PR so I clicked away, and I ended up testing matthias PR :<

I tested yours which goes a bit farther. I ve tried twice with the same error. I am in sandboxed mode on ixos-unstable so that might be why ? I can try rerunning with RUST_BACKTRACE=1 if that helps. Thanks for your work.

installing ‘vdirsyncer-0.17.0a1’
these derivations will be built:
  /nix/store/alvzcahav063a8ivpy6w23pgp19qbssa-vdirsyncer-0.17.0a1-native.drv
  /nix/store/6xixijkqaqhpvn0nd5anmd56m2km7dsr-vdirsyncer-0.17.0a1.drv
building path(s) ‘/nix/store/c47mprb5gg0ln26k8ny08n94rggjjc22-vdirsyncer-0.17.0a1-native’
unpacking sources
unpacking source archive /nix/store/2krb64npcsaix238ki403sxd7vv2n032-vdirsyncer-0.17.0a1.tar.gz
source root is vdirsyncer-0.17.0a1/rust
unpacking source archive /nix/store/gv2hi9ig9ji7m4m7q8h86nh9z5ahlka9-vdirsyncer-0.17.0a1-native-vendor
setting SOURCE_DATE_EPOCH to timestamp 1511808735 of file vdirsyncer-0.17.0a1/rust/src/item.rs
patching sources
configuring
building
Running cargo build --release 
   Compiling cfg-if v0.1.2
   Compiling quote v0.3.15
   Compiling dtoa v0.4.2
   Compiling ansi_term v0.9.0
   Compiling gcc v0.3.54
   Compiling futures v0.1.17
   Compiling vec_map v0.8.0
   Compiling bitflags v0.9.1
   Compiling serde v0.9.15
   Compiling cc v1.0.3
   Compiling untrusted v0.5.1
   Compiling num-traits v0.1.40
   Compiling either v1.3.0
   Compiling scopeguard v0.3.3
   Compiling rayon-core v1.2.1
   Compiling itoa v0.3.4
   Compiling rustc-demangle v0.1.5
   Compiling log v0.3.8
   Compiling strsim v0.6.0
   Compiling unicode-width v0.1.4
   Compiling libc v0.2.33
   Compiling lazy_static v0.2.9
   Compiling unicode-xid v0.0.4
   Compiling coco v0.1.1
   Compiling textwrap v0.9.0
   Compiling backtrace-sys v0.1.16
   Compiling synom v0.11.3
   Compiling atty v0.2.3
   Compiling rand v0.3.17
   Compiling num_cpus v1.7.0
   Compiling syn v0.11.11
   Compiling toml v0.3.2
   Compiling serde_json v0.9.10
   Compiling clap v2.27.1
   Compiling tempdir v0.3.5
   Compiling rayon v0.8.2
thread '<unnamed>' panicked at 'failed to acquire jobserver token: Resource temporarily unavailable (os error 11)', src/librustc_trans/back/write.rs:1420:28
note: Run with `RUST_BACKTRACE=1` for a backtrace.
error: failed to acquire jobserver token: Resource temporarily unavailable (os error 11)

error: Could not compile `rayon`.
warning: build failed, waiting for other jobs to finish...
error: build failed
note: keeping build directory ‘/tmp/nix-build-vdirsyncer-0.17.0a1-native.drv-1’
builder for ‘/nix/store/alvzcahav063a8ivpy6w23pgp19qbssa-vdirsyncer-0.17.0a1-native.drv’ failed with exit code 101

teto on 26 Dec 2017

Yes, unfortunately rustc seems to panic quite a lot in nondeterministic fashion :-(. I'm not sure why that is but I eventually succeeded building the package (with sandboxing enabled) after retrying a few times.
The panics happen with other packages too so I don't think it's something specific to vdirsyncer but other than retrying I don't know of any other work around.

pbogdan on 26 Dec 2017

Indeed it crashes at different steps yet I could not reach the end of the build progress despite many attempts. I guess I need to pay attention for newer rust versions too.
I will try compiling several batches of fewer rust packages at at time to reduce the risk of failures, like first installing

 Compiling cfg-if v0.1.2
   Compiling quote v0.3.15
   Compiling dtoa v0.4.2
   Compiling ansi_term v0.9.0
   Compiling gcc v0.3.54

then

Compiling futures v0.1.17
   Compiling vec_map v0.8.0
   Compiling bitflags v0.9.1
   Compiling serde v0.9.15
   Compiling cc v1.0.3
   Compiling untrusted v0.5.1

etc so that once vdirsyncer installs dependancies they are already compiled.

teto on 28 Dec 2017

This is being widely reported, especially under Nix for some reason. See https://github.com/rust-lang/rust/issues/46981

Ralith on 19 Jan 2018

I am working on this, it appears to be that our rust build is using an outdated version of the jobserver crate which had this bug.

bennofs on 31 Jan 2018

🎉5

@pbogdan after successfully installing another rsut-based package, I tried your patch and it worked.
I hope it can make it into nixpkgs through either Mathias' or your PR :+1:

teto on 5 Mar 2018

@bennofs any news :) ?

teto on 6 Apr 2018

@teto I think this should be fixed on master, since it uses a newer rustc release (1.24)

bennofs on 6 Apr 2018

PR merged. Thanks for your help

teto on 25 Apr 2018

Was this page helpful?

0 / 5 - 0 ratings