Wasmtime: Building for darwin on nix fails

[nomeata]

Trying to build on nix for darwin, (see https://github.com/NixOS/nixpkgs/pull/71316), I hit the following problem:

unpacking sources
unpacking source archive /nix/store/3ii33ri19pv2c092gzdr0vpjjqw1yrmj-source
source root is source
unpacking source archive /nix/store/g5zrf1aaw39w6cg7qfadfvrjy9cmky8a-wasmtime-20191018-vendor
patching sources
applying patch /nix/store/fh5c4jz5k0ny4fpmsiqkxk9qhqv2chh0-cargo-lock.patch
patching file Cargo.lock
configuring
building
++ env CC_x86_64-apple-darwin=/nix/store/hi40vi42w2rnf9pflyhmsalcj5jf2bsv-clang-wrapper-7.1.0/bin/cc CXX_x86_64-apple-darwin=/nix/store/hi40vi42w2rnf9pflyhmsalcj5jf2bsv-clang-wrapper-7.1.0/bin/c++ CC_x86_64-apple-darwin=/nix/store/hi40vi42w2rnf9pflyhmsalcj5jf2bsv-clang-wrapper-7.1.0/bin/cc CXX_x86_64-apple-darwin=/nix/store/hi40vi42w2rnf9pflyhmsalcj5jf2bsv-clang-wrapper-7.1.0/bin/c++ cargo build --release --target x86_64-apple-darwin --frozen
   Compiling log v0.4.8
   Compiling libc v0.2.64
   Compiling proc-macro2 v1.0.5
   Compiling unicode-xid v0.2.0
   Compiling cfg-if v0.1.10
   Compiling syn v1.0.5
   Compiling ryu v1.0.2
   Compiling serde v1.0.101
   Compiling semver-parser v0.7.0
   Compiling failure_derive v0.1.6
   Compiling proc-macro2 v0.4.30
   Compiling itoa v0.4.4
   Compiling unicode-xid v0.1.0
   Compiling cranelift-codegen-shared v0.46.1
   Compiling arrayvec v0.4.12
   Compiling memchr v2.2.1
   Compiling rustc-demangle v0.1.16
   Compiling syn v0.15.44
   Compiling lazy_static v1.4.0
   Compiling typenum v1.11.2
   Compiling glob v0.3.0
   Compiling nodrop v0.1.14
   Compiling byteorder v1.3.2
   Compiling scopeguard v1.0.0
   Compiling regex-syntax v0.6.12
   Compiling autocfg v0.1.6
   Compiling rayon-core v1.6.0
   Compiling smallvec v0.6.10
   Compiling quick-error v1.2.2
   Compiling byte-tools v0.3.1
   Compiling getrandom v0.1.12
   Compiling termcolor v1.0.5
   Compiling bitflags v1.2.1
   Compiling wasmtime-environ v0.2.0 (/private/tmp/nix-build-wasmtime-20191018.drv-0/source/wasmtime-environ)
   Compiling fake-simd v0.1.2
   Compiling plain v0.2.3
   Compiling ppv-lite86 v0.2.5
   Compiling unicode-segmentation v1.3.0
   Compiling wasmparser v0.39.2
   Compiling opaque-debug v0.2.3
   Compiling either v1.5.3
   Compiling fallible-iterator v0.2.0
   Compiling indexmap v1.2.0
   Compiling spin v0.5.2
   Compiling nix v0.15.0
   Compiling stable_deref_trait v1.1.1
   Compiling wasmparser v0.37.2
   Compiling void v1.0.2
   Compiling id-arena v2.2.1
   Compiling leb128 v0.2.4
   Compiling wasmtime v0.2.0 (/private/tmp/nix-build-wasmtime-20191018.drv-0/source)
   Compiling strsim v0.9.2
   Compiling wasi-common v0.1.0 (https://github.com/CraneStation/wasi-common?rev=c3bf040#c3bf0404)
   Compiling crossbeam-utils v0.6.6
   Compiling thread_local v0.3.6
   Compiling humantime v1.3.0
   Compiling block-padding v0.1.4
   Compiling semver v0.9.0
   Compiling bincode v1.2.0
   Compiling num-traits v0.2.8
   Compiling num-integer v0.1.41
   Compiling heck v0.3.1
   Compiling c2-chacha v0.2.2
   Compiling crossbeam-queue v0.1.2
   Compiling rustc_version v0.2.3
   Compiling base64 v0.10.1
   Compiling gimli v0.19.0
   Compiling aho-corasick v0.7.6
   Compiling generic-array v0.12.3
   Compiling num_cpus v1.10.1
   Compiling atty v0.2.13
   Compiling dirs-sys v0.3.4
   Compiling mach v0.2.3
   Compiling errno v0.2.4
   Compiling time v0.1.42
   Compiling filetime v0.2.7
   Compiling jobserver v0.1.17
   Compiling memoffset v0.5.1
   Compiling scroll v0.9.2
   Compiling directories v2.0.2
   Compiling digest v0.8.1
   Compiling block-buffer v0.7.3
   Compiling rand_core v0.5.1
   Compiling quote v0.6.13
   Compiling region v2.1.2
   Compiling sha2 v0.8.0
   Compiling quote v1.0.2
   Compiling rand_chacha v0.2.1
   Compiling cc v1.0.46
   Compiling chrono v0.4.9
   Compiling crossbeam-epoch v0.7.2
   Compiling rand v0.7.2
   Compiling crossbeam-deque v0.7.1
   Compiling rayon v1.2.0
   Compiling regex v1.3.1
   Compiling backtrace-sys v0.1.32
   Compiling zstd-sys v1.4.13+zstd.1.4.3
   Compiling cmake v0.1.42
   Compiling raw-cpuid v6.1.0
   Compiling wasmtime-runtime v0.2.0 (/private/tmp/nix-build-wasmtime-20191018.drv-0/source/wasmtime-runtime)
   Compiling scroll_derive v0.9.5
   Compiling wabt-sys v0.7.0
   Compiling backtrace v0.3.40
   Compiling goblin v0.0.24
   Compiling env_logger v0.6.2
   Compiling file-per-thread-logger v0.1.2
   Compiling pretty_env_logger v0.3.1
   Compiling serde_derive v1.0.101
   Compiling synstructure v0.12.1
   Compiling walrus-macro v0.12.0
   Compiling wasi-common-cbindgen v0.1.0 (https://github.com/CraneStation/wasi-common?rev=c3bf040#c3bf0404)
   Compiling failure v0.1.6
   Compiling zstd-safe v1.4.13+zstd.1.4.3
   Compiling zstd v0.4.28+zstd.1.4.3
error: linking with `/nix/store/hi40vi42w2rnf9pflyhmsalcj5jf2bsv-clang-wrapper-7.1.0/bin/cc` failed: exit code: 1
  |
  = note: "/nix/store/hi40vi42w2rnf9pflyhmsalcj5jf2bsv-clang-wrapper-7.1.0/bin/cc" "-m64" "-L" "/nix/store/r8ps3zwaxwam9bnw9i0n46z4qr2j7yzk-rustc-1.37.0/lib/rustlib/x86_64-apple-darwin/lib" "/private/tmp/nix-build-wasmtime-20191018.drv-0/source/target/x86_64-apple-darwin/release/deps/wasi_common-cb01afa41255bfe8.wasi_common.2pp5p6tf-cgu.0.rcgu.o" "/private/tmp/nix-build-wasmtime-20191018.drv-0/source/target/x86_64-apple-darwin/release/deps/wasi_common-cb01afa41255bfe8.wasi_common.2pp5p6tf-cgu.1.rcgu.o" "/private/tmp/nix-build-wasmtime-20191018.drv-0/source/target/x86_64-apple-darwin/release/deps/wasi_common-cb01afa41255bfe8.wasi_common.2pp5p6tf-cgu.10.rcgu.o" "/private/tmp/nix-build-wasmtime-20191018.drv-0/source/target/x86_64-apple-darwin/release/deps/wasi_common-cb01afa41255bfe8.wasi_common.2pp5p6tf-cgu.11.rcgu.o" "/private/tmp/nix-build-wasmtime-20191018.drv-0/source/target/x86_64-apple-darwin/release/deps/wasi_common-cb01afa41255bfe8.wasi_common.2pp5p6tf-cgu.12.rcgu.o" "/private/tmp/nix-build-wasmtime-20191018.drv-0/source/target/x86_64-apple-darwin/release/deps/wasi_common-cb01afa41255bfe8.wasi_common.2pp5p6tf-cgu.13.rcgu.o" "/private/tmp/nix-build-wasmtime-20191018.drv-0/source/target/x86_64-apple-darwin/release/deps/wasi_common-cb01afa41255bfe8.wasi_common.2pp5p6tf-cgu.14.rcgu.o" "/private/tmp/nix-build-wasmtime-20191018.drv-0/source/target/x86_64-apple-darwin/release/deps/wasi_common-cb01afa41255bfe8.wasi_common.2pp5p6tf-cgu.15.rcgu.o" "/private/tmp/nix-build-wasmtime-20191018.drv-0/source/target/x86_64-apple-darwin/release/deps/wasi_common-cb01afa41255bfe8.wasi_common.2pp5p6tf-cgu.2.rcgu.o" "/private/tmp/nix-build-wasmtime-20191018.drv-0/source/target/x86_64-apple-darwin/release/deps/wasi_common-cb01afa41255bfe8.wasi_common.2pp5p6tf-cgu.3.rcgu.o" "/private/tmp/nix-build-wasmtime-20191018.drv-0/source/target/x86_64-apple-darwin/release/deps/wasi_common-cb01afa41255bfe8.wasi_common.2pp5p6tf-cgu.4.rcgu.o" "/private/tmp/nix-build-wasmtime-20191018.drv-0/source/target/x86_64-apple-darwin/release/deps/wasi_common-cb01afa41255bfe8.wasi_common.2pp5p6tf-cgu.5.rcgu.o" "/private/tmp/nix-build-wasmtime-20191018.drv-0/source/target/x86_64-apple-darwin/release/deps/wasi_common-cb01afa41255bfe8.wasi_common.2pp5p6tf-cgu.6.rcgu.o" "/private/tmp/nix-build-wasmtime-20191018.drv-0/source/target/x86_64-apple-darwin/release/deps/wasi_common-cb01afa41255bfe8.wasi_common.2pp5p6tf-cgu.7.rcgu.o" "/private/tmp/nix-build-wasmtime-20191018.drv-0/source/target/x86_64-apple-darwin/release/deps/wasi_common-cb01afa41255bfe8.wasi_common.2pp5p6tf-cgu.8.rcgu.o" "/private/tmp/nix-build-wasmtime-20191018.drv-0/source/target/x86_64-apple-darwin/release/deps/wasi_common-cb01afa41255bfe8.wasi_common.2pp5p6tf-cgu.9.rcgu.o" "-o" "/private/tmp/nix-build-wasmtime-20191018.drv-0/source/target/x86_64-apple-darwin/release/deps/libwasi_common-cb01afa41255bfe8.dylib" "-Wl,-exported_symbols_list,/private/tmp/nix-build-wasmtime-20191018.drv-0/rustcqP4NQ5/list" "/private/tmp/nix-build-wasmtime-20191018.drv-0/source/target/x86_64-apple-darwin/release/deps/wasi_common-cb01afa41255bfe8.4mk5g2t6mk33db4q.rcgu.o" "-Wl,-dead_strip" "-nodefaultlibs" "-L" "/private/tmp/nix-build-wasmtime-20191018.drv-0/source/target/x86_64-apple-darwin/release/deps" "-L" "/private/tmp/nix-build-wasmtime-20191018.drv-0/source/target/release/deps" "-L" "/private/tmp/nix-build-wasmtime-20191018.drv-0/source/target/x86_64-apple-darwin/release/build/backtrace-sys-2d147214789255a0/out" "-L" "/nix/store/r8ps3zwaxwam9bnw9i0n46z4qr2j7yzk-rustc-1.37.0/lib/rustlib/x86_64-apple-darwin/lib" "/private/tmp/nix-build-wasmtime-20191018.drv-0/source/target/x86_64-apple-darwin/release/deps/libnix-dfd91aca8c39020a.rlib" "/private/tmp/nix-build-wasmtime-20191018.drv-0/source/target/x86_64-apple-darwin/release/deps/libvoid-9cdf6e4006aa038f.rlib" "/private/tmp/nix-build-wasmtime-20191018.drv-0/source/target/x86_64-apple-darwin/release/deps/libbitflags-a7bbe403b9c1735b.rlib" "/private/tmp/nix-build-wasmtime-20191018.drv-0/source/target/x86_64-apple-darwin/release/deps/librand-daaf851cf7868d45.rlib" "/private/tmp/nix-build-wasmtime-20191018.drv-0/source/target/x86_64-apple-darwin/release/deps/librand_chacha-97fa38918d53e690.rlib" "/private/tmp/nix-build-wasmtime-20191018.drv-0/source/target/x86_64-apple-darwin/release/deps/libc2_chacha-a0e1f0d831f8f9dd.rlib" "/private/tmp/nix-build-wasmtime-20191018.drv-0/source/target/x86_64-apple-darwin/release/deps/libppv_lite86-2a1a38c81ee18cd3.rlib" "/private/tmp/nix-build-wasmtime-20191018.drv-0/source/target/x86_64-apple-darwin/release/deps/liblazy_static-4cae960660a74250.rlib" "/private/tmp/nix-build-wasmtime-20191018.drv-0/source/target/x86_64-apple-darwin/release/deps/librand_core-4e1a0c41489f9461.rlib" "/private/tmp/nix-build-wasmtime-20191018.drv-0/source/target/x86_64-apple-darwin/release/deps/libgetrandom-28165c5b9affdbed.rlib" "/private/tmp/nix-build-wasmtime-20191018.drv-0/source/target/x86_64-apple-darwin/release/deps/liblog-47c2bc685ba53e9c.rlib" "/private/tmp/nix-build-wasmtime-20191018.drv-0/source/target/x86_64-apple-darwin/release/deps/libfiletime-e91ead56cad31a83.rlib" "/private/tmp/nix-build-wasmtime-20191018.drv-0/source/target/x86_64-apple-darwin/release/deps/libfailure-7e4295e10a48e1af.rlib" "/private/tmp/nix-build-wasmtime-20191018.drv-0/source/target/x86_64-apple-darwin/release/deps/libbacktrace-d1b25e4eda6cc091.rlib" "/private/tmp/nix-build-wasmtime-20191018.drv-0/source/target/x86_64-apple-darwin/release/deps/libbacktrace_sys-38b2028395269248.rlib" "/private/tmp/nix-build-wasmtime-20191018.drv-0/source/target/x86_64-apple-darwin/release/deps/liblibc-31f4ec99033d3060.rlib" "/private/tmp/nix-build-wasmtime-20191018.drv-0/source/target/x86_64-apple-darwin/release/deps/libcfg_if-1ee9105a5976112b.rlib" "/private/tmp/nix-build-wasmtime-20191018.drv-0/source/target/x86_64-apple-darwin/release/deps/librustc_demangle-627c318ee7b04a5c.rlib" "/nix/store/r8ps3zwaxwam9bnw9i0n46z4qr2j7yzk-rustc-1.37.0/lib/rustlib/x86_64-apple-darwin/lib/libstd-c6ca9251a199aa27.rlib" "/nix/store/r8ps3zwaxwam9bnw9i0n46z4qr2j7yzk-rustc-1.37.0/lib/rustlib/x86_64-apple-darwin/lib/libpanic_unwind-39ec93fd4165c255.rlib" "/nix/store/r8ps3zwaxwam9bnw9i0n46z4qr2j7yzk-rustc-1.37.0/lib/rustlib/x86_64-apple-darwin/lib/libbacktrace-a948cb44f048597a.rlib" "/nix/store/r8ps3zwaxwam9bnw9i0n46z4qr2j7yzk-rustc-1.37.0/lib/rustlib/x86_64-apple-darwin/lib/libbacktrace_sys-9f041fb877b98c64.rlib" "/nix/store/r8ps3zwaxwam9bnw9i0n46z4qr2j7yzk-rustc-1.37.0/lib/rustlib/x86_64-apple-darwin/lib/librustc_demangle-3c4a5c551a922378.rlib" "/nix/store/r8ps3zwaxwam9bnw9i0n46z4qr2j7yzk-rustc-1.37.0/lib/rustlib/x86_64-apple-darwin/lib/libhashbrown-2c195470cc27ba70.rlib" "/nix/store/r8ps3zwaxwam9bnw9i0n46z4qr2j7yzk-rustc-1.37.0/lib/rustlib/x86_64-apple-darwin/lib/librustc_std_workspace_alloc-38fc89345f49f6a1.rlib" "/nix/store/r8ps3zwaxwam9bnw9i0n46z4qr2j7yzk-rustc-1.37.0/lib/rustlib/x86_64-apple-darwin/lib/libunwind-e60112f814e80d3d.rlib" "/nix/store/r8ps3zwaxwam9bnw9i0n46z4qr2j7yzk-rustc-1.37.0/lib/rustlib/x86_64-apple-darwin/lib/libcfg_if-ebddb7552a11bbd5.rlib" "/nix/store/r8ps3zwaxwam9bnw9i0n46z4qr2j7yzk-rustc-1.37.0/lib/rustlib/x86_64-apple-darwin/lib/liblibc-46d8e16ab2813df4.rlib" "/nix/store/r8ps3zwaxwam9bnw9i0n46z4qr2j7yzk-rustc-1.37.0/lib/rustlib/x86_64-apple-darwin/lib/liballoc-349e6b1effd8559c.rlib" "/nix/store/r8ps3zwaxwam9bnw9i0n46z4qr2j7yzk-rustc-1.37.0/lib/rustlib/x86_64-apple-darwin/lib/librustc_std_workspace_core-1a46406c8cbb31ae.rlib" "/nix/store/r8ps3zwaxwam9bnw9i0n46z4qr2j7yzk-rustc-1.37.0/lib/rustlib/x86_64-apple-darwin/lib/libcore-2922fa7cbacf5093.rlib" "/nix/store/r8ps3zwaxwam9bnw9i0n46z4qr2j7yzk-rustc-1.37.0/lib/rustlib/x86_64-apple-darwin/lib/libcompiler_builtins-96d54cc5512fa6bf.rlib" "-lSystem" "-lresolv" "-lc" "-lm" "-dynamiclib" "-Wl,-dylib"
  = note: Undefined symbols for architecture x86_64:
            "_utimensat", referenced from:
                _$LT$$u5b$u8$u5d$$u20$as$u20$nix..NixPath$GT$::with_nix_path::h2c1b78ca714c817b in wasi_common-cb01afa41255bfe8.wasi_common.2pp5p6tf-cgu.12.rcgu.o
          ld: symbol(s) not found for architecture x86_64
          clang-7: error: linker command failed with exit code 1 (use -v to see invocation)


error: aborting due to previous error

error: Could not compile `wasi-common`.
warning: build failed, waiting for other jobs to finish...
error: build failed
builder for '/nix/store/6h89ffzlsxmxf8wp0abap155i9g8nw46-wasmtime-20191018.drv' failed with exit code 101

I read somewhere that on nix, certain symbols are removed from libsystem that are not available on all versions of MacOS. (See first two messages on https://logs.nix.samueldr.com/nix-darwin/2019-07-26)

[kubkon]

Awesome, thanks for this! So, we do indeed use _utimensat in our codebase [here]. As far as I remember, _utimensat was added to darwin when macOS 13 was released, and before then, only _utimes was available. My proposition here is to try and use [filetime] crate which handles different darwin versions dynamically, which should hopefully fix the problem. I'd ask you to verify that it builds fine for you with this fix though. Would that be OK?

[nomeata]

I have no idea about darwin and little idea about nix, but I am happy to test fixes.

[kubkon]

OK, so I think there is a potential fix for this, but it requires some thought before committing to it. @alexcrichton @sunfishcode I'd very much appreciate your opinion on this. The potential fix, in principle, would require the same mechanism as the one used in @alexcrichton's [filetime] crate for [checking whether a symbol exists at runtime]:

static ADDR: AtomicUsize = AtomicUsize::new(0);
let func: Option<unsafe extern "C" fn(...)> = unsafe {
    fetch(&ADDR, b"utimensat\0").map(|sym| mem::transmute(sym))
};
if let Some(func) = func {
    // Success! "utimensat" symbol was found on our host
    // Call utimensat(...)
    func(fd, path, ...);
} else {
    // Sad days, "utimensat" is not available on this host
}

In case of the negative route, when utimensat symbol is not found on the host, [filetime] resorts back to utimes. However, our use case is slightly more constrained than [filetime]'s in the sense that we always operate on a combination of a (pre)opened file descriptor and a relative path. Now, AFAIK there is no "at" version of utimes, i.e., something like utimesat, which makes things a little bit more complicated than I originally envisioned. We could improvise by calling openat on the provided pair (fd, rel_path), and then adjust filetimes using futimes syscall like so:

let fd = unsafe { libc::openat(dirfd, path, libc::O_NOFOLLOW | libc::O_RDWR) };
unsafe { libc::futimes(fd, times.as_ptr()) };

But this becomes somewhat messy, plus it adds up to two additional syscalls (one to get the file descriptor using openat, and second to read current values of atime and mtime -> no UTIME_OMIT or UTIME_NOW available if utimensat is not available :-().

Originally, I thought of extending [filetime] crate with this use case in mind, but I'm now I'm hesitant as it could potentially break the symmetry in implementation between Linux and BSD hosts.

Anyhow, the question here is should we support hosts without utimensat capabilities, or not?

[alexcrichton]

I'm generally of the opinion that on recent (ish) versions of a platform is where a program should be the most optimized, and if a slower polyfill is required on older platforms that's better than not having the functionality at all. In that sense if the polyfill for older platforms requires some hoops and a few more syscalls I personally think that's ok.

Also FWIW if this is on Linux I'd recommend using libc::syscall since that is generally a bit more robust than dlopen in the face of static linking and musl. It's only available on Linux though!

[kubkon]

I'm generally of the opinion that on recent (ish) versions of a platform is where a program should be the most optimized, and if a slower polyfill is required on older platforms that's better than not having the functionality at all. In that sense if the polyfill for older platforms requires some hoops and a few more syscalls I personally think that's ok.

Makes sense! :-)

Also FWIW if this is on Linux I'd recommend using libc::syscall since that is generally a bit more robust than dlopen in the face of static linking and musl. It's only available on Linux though!

Yep, absolutely. I was going to basically steal much of the "meat" from your lib, filetime and that would include both Linux and BSD implementations. I guess I'll prepare a draft PR with a potential resolution of the issue and cc all of you guys in, and coordinate with @nomeata for some testing, if that's OK of course! How does that sound for everyone?

[sunfishcode]

That sounds like a good approach to me!

[marmistrz]

What's done in filetime for utimensat and futimens sounds like a quite repetitive piece of code, which may be useful in different crates.

How about writing some macro that would automatically generate the Option<unsafe extern "C" fn(...)> getters and publishing it as a crate? Alternatively, you could write a function, generic over the function type, which would accept the symbol name and return Option<unsafe extern "C" fn(...)>.

Idea:

maybe_function!("futimens", fn(c_int, *const timespec) -> c_int);

or

let futimens = maybe_function<fn(c_int, *const timespec) -> c_int>("futimens");

I'm not sure if we can create convert fn to unsafe extern "C" fn, though.

[nomeata]

@kubkon, were you able to make progress here?

[kubkon]

Hey @nomeata, apologies but I didn't have time to properly sit down to it yet. Having said that I've got some more time this week, so hopefully, I'll get most of it (if not all) done this week. The moment I get something working and reviewed, I shall let you know and kindly ask for your help in verifying that the changes are actually correct.

[kubkon]

Hi @nomeata, could you do me a favour and try building wasmtime using [kubkon/wasmtime/tree/nixos_utimensat] branch, and see if this fixes your issue?

[kubkon]

I believe #535 fixed this issue as confirmed by @nomeata in #535, so closing :-)