Nix: darwin: nix-daemon crashes due to OBJC_DISABLE_INITIALIZE_FORK_SAFETY

[periklis]

On macOS 10.14 i observe lately in my system.log that nix-daemon crashes due to the apple changes on fork since 10.13. The following message comes in regurarly:

nix-daemon[38220]: objc[38220]: +[__NSPlaceholderArray initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.

Related discussion in nixpkgs for other packages: https://github.com/NixOS/nixpkgs/issues/44160
Related info: http://sealiesoftware.com/blog/archive/2017/6/5/Objective-C_and_fork_in_macOS_1013.html

Do we need to update the SDK dependency for our buildsystem?

[copumpkin]

Why is Objective C even relevant to nix-daemon? This seems pretty weird!

[LnL7]

Yeah, I don't understand this either. I'm running 10.14 on my new machine and don't have any problems with builds or the daemon.

[copumpkin]

@periklis can you run your daemon with DYLD_PRINT_LIBRARIES=1 in the environment?

[periklis]

Here is the log: https://gist.github.com/periklis/a4357cfd21cd67dafa9122f9ca119592

Is it maybe because nix-daemon is run by launchctl that in turn can be an objc software?

[peel]

I've hit the same thing. However... as @copumpkin noticed the problem is deep blow the nix stack and in fact is related to some awkward security features introduced somewhere in High Sierra, setting export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES in deamon environment was enough to avoid crashing the daemon. However, that's not a proper solution (as per: https://blog.phusion.nl/2017/10/13/why-ruby-app-servers-break-on-macos-high-sierra-and-what-can-be-done-about-it/)

[LnL7]

We got a bit closer to figuring out what's going on. The released nix binaries work fine, however using a build from nixpkgs master/unstable (on 10.14.1?) causes this problem.

Either this is caused by the new CoreFoundation from swift-corelibs or something else is pulling in the Objective-C runtime. With changes like https://github.com/NixOS/nixpkgs/pull/29785 libraries like libcurl have become pretty bloated IMHO, pulling in eg. kerberos in by default (not only for curlFull). One of those could also be causing this.

[periklis]

yeah, my setup has a managed nix binary by nix-darwin

[LnL7]

First I thought it was a problem there somehow, but using the same binary from the installer there works fine.

{ config, lib, pkgs, ... }:
{
  nix.package = lib.toDerivation /nix/store/dkjlfkrknmxbjmpfk3dg4q3nmb7m3zvk-nix-2.1.3 // { version = "2.1.3"; };
}

[LnL7]

I updated my machine but still can't reproduce this for some reason.

[periklis]

@LnL7 10.14.1 or 10.14.0? i am still on later.

[LnL7]

I can reproduce this now. A bunch of frameworks are definitively getting loaded and I'm not sure why yet, however this doesn't seem to be anything if I compare it with the working version.

@@ -1,4 +1,4 @@
-dyld: loaded: good/bin/nix-daemon
+dyld: loaded: bad/bin/nix-daemon
 dyld: loaded: /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-libsodium-1.0.16/lib/libsodium.23.dylib
 dyld: loaded: /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-nix-2.2pre6526_9f99d624/lib/libnixexpr.dylib
 dyld: loaded: /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-boehm-gc-8.0.0/lib/libgc.1.dylib
@@ -50,7 +50,7 @@
 dyld: loaded: /usr/lib/libc++.1.dylib
 dyld: loaded: /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-xz-5.2.4/lib/liblzma.5.dylib
 dyld: loaded: /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-bzip2-1.0.6.0.1/lib/libbz2.1.dylib
-dyld: loaded: /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-openssl-1.0.2p/lib/libcrypto.1.0.0.dylib
+dyld: loaded: /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-openssl-1.0.2q/lib/libcrypto.1.0.0.dylib
 dyld: loaded: /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-brotli-1.0.7-lib/lib/libbrotlienc.1.dylib
 dyld: loaded: /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-brotli-1.0.7-lib/lib/libbrotlidec.1.dylib
 dyld: loaded: /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-boost-1.67_0/lib/libboost_context.dylib
@@ -62,15 +62,15 @@
 dyld: loaded: /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-libiconv-osx-10.11.6/lib/libiconv-nocharset.dylib
 dyld: loaded: /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-libiconv-osx-10.11.6/lib/libcharset.1.dylib
 dyld: loaded: /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-libc++abi-5.0.2/lib/libc++abi.1.dylib
-dyld: loaded: /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-sqlite-3.24.0/lib/libsqlite3.0.dylib
+dyld: loaded: /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-sqlite-3.26.0/lib/libsqlite3.0.dylib
 dyld: loaded: /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-curl-7.59.0/lib/libcurl.4.dylib
 dyld: loaded: /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-aws-sdk-cpp-1.6.52/lib/libaws-cpp-sdk-transfer.dylib
 dyld: loaded: /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-aws-sdk-cpp-1.6.52/lib/libaws-cpp-sdk-s3.dylib
 dyld: loaded: /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-aws-sdk-cpp-1.6.52/lib/libaws-cpp-sdk-core.dylib
 dyld: loaded: /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-zlib-1.2.11/lib/libz.dylib
-dyld: loaded: /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-nghttp2-1.34.0-lib/lib/libnghttp2.14.dylib
+dyld: loaded: /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-nghttp2-1.35.0-lib/lib/libnghttp2.14.dylib
 dyld: loaded: /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-libssh2-1.8.0/lib/libssh2.1.dylib
-dyld: loaded: /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-openssl-1.0.2p/lib/libssl.1.0.0.dylib
+dyld: loaded: /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-openssl-1.0.2q/lib/libssl.1.0.0.dylib
 dyld: loaded: /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-libkrb5-1.15.2/lib/libgssapi_krb5.2.2.dylib
 dyld: loaded: /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-Libsystem-osx-10.11.6/lib/libresolv.9.dylib
 dyld: loaded: /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-swift-corefoundation/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
@@ -81,10 +81,10 @@
 dyld: loaded: /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-ICU-osx-10.10.5/lib/libicucore.A.dylib
 dyld: loaded: /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-curl-7.62.0/lib/libcurl.4.dylib
 dyld: loaded: /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-libxml2-2.9.8/lib/libxml2.2.dylib
-dyld: loaded: /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-nghttp2-1.34.0-lib/lib/libnghttp2.14.dylib
+dyld: loaded: /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-nghttp2-1.35.0-lib/lib/libnghttp2.14.dylib
 dyld: loaded: /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-libssh2-1.8.0/lib/libssh2.1.dylib
-dyld: loaded: /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-openssl-1.0.2p/lib/libssl.1.0.0.dylib
-dyld: loaded: /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-openssl-1.0.2p/lib/libcrypto.1.0.0.dylib
+dyld: loaded: /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-openssl-1.0.2q/lib/libssl.1.0.0.dylib
+dyld: loaded: /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-openssl-1.0.2q/lib/libcrypto.1.0.0.dylib
 dyld: loaded: /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-libkrb5-1.15.2/lib/libgssapi_krb5.2.2.dylib
 dyld: loaded: /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-libkrb5-1.15.2/lib/libkrb5.3.3.dylib
 dyld: loaded: /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-libkrb5-1.15.2/lib/libk5crypto.3.1.dylib
@@ -238,5 +238,7 @@
 dyld: loaded: /usr/lib/libcmph.dylib
 dyld: loaded: /System/Library/Frameworks/CoreData.framework/Versions/A/CoreData
 dyld: loaded: /System/Library/Frameworks/ServiceManagement.framework/Versions/A/ServiceManagement
-dyld: loaded: /System/Library/PrivateFrameworks/BackgroundTaskManagement.framework/Versions/A/BackgroundTaskManagement
+dyld: loaded: /System/Library/PrivateFrameworks/BackgroundTaskManagement.framework/Versions/A/BackgroundTaskManagement
 dyld: loaded: /usr/lib/libxslt.1.dylib
+objc[38361]: +[__NSPlaceholderArray initialize] may have been in progress in another thread when fork() was called.
+objc[38361]: +[__NSPlaceholderArray initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.

[LnL7]

I think I found the culprit, something dlopen's libnetwork.dylib which then pulls in lots of system stuff and frameworks.

dlopen_internal(/usr/lib/libnetwork.dylib, 0x00000005)
dyld: Using shared cached for /usr/lib/libnetwork.dylib
dyld: loaded: /usr/lib/libnetwork.dylib

EDIT: possibly curl -> libresolv -> Libinfo

[LnL7]

Found some more information the dlopen comes from curl. However this is not something new so I'm not sure what changed or how to fix it.

$ dtrace -n 'pid$target:libdyld.dylib:dlopen:entry { ustack(); }' -W nix-daemon
CPU     ID                    FUNCTION:NAME
  4  22854                     dlopen:entry
              libdyld.dylib`dlopen
              libsystem_info.dylib`_gai_load_libnetwork_once+0xd4
              libsystem_pthread.dylib`__pthread_once_handler+0x42
              libsystem_platform.dylib`_os_once_callout+0x12
              libsystem_pthread.dylib`pthread_once+0x38
              libsystem_info.dylib`_gai_load_libnetwork+0x1b
              libsystem_info.dylib`_gai_nat64_v4_address_requires_synthesis+0x1f
              libsystem_info.dylib`_gai_nat64_second_pass+0x200
              libsystem_info.dylib`si_addrinfo+0x7ba
              libsystem_info.dylib`_getaddrinfo_internal+0xf2
              libsystem_info.dylib`getaddrinfo+0x3d
              libcurl.4.dylib`Curl_getaddrinfo_ex+0x32
              libcurl.4.dylib`getaddrinfo_thread+0x53
              libcurl.4.dylib`curl_thread_create_thunk+0x1d
              libsystem_pthread.dylib`_pthread_body+0x7e
              libsystem_pthread.dylib`_pthread_start+0x46
              libsystem_pthread.dylib`thread_start+0xd

[LnL7]

I made an example program to reproduce the behaviour (without forking). It looks like some kind of fallback that happens when no ipv6 address can be resolved for the host.

https://gist.github.com/LnL7/5c43975144c0a9a0f5de74cc155a01d5

[yurrriq]

I'm having the same (or similar?) experience on 10.14.2 fwiw.

Edit: I'm also having some weird errors with substituters, so I can't neither download cached binaries, nor build with --option build-use-substitutes false. I do, however, have an old generation that works (darwin-rebuild -G 227) How can I diff that generation and my current/desire one?

[LnL7]

I think I might have found the problem thanks to https://github.com/NixOS/nixpkgs/issues/41312#issuecomment-395896215

Particularly noteworthy is that the version complained about (7.55) is where threaded resolver was first enabled by default I think. Err well it's 7.55.1 not 7.55.0, hmm.

A build from master still seems to have the same problem, while this does not.

with import <nixpkgs> {};

let
  curlFix = curl.overrideAttrs (drv: {
    configureFlags = drv.configureFlags or [] ++ ["--disable-threaded-resolver"];
  });
in

nix.override {
  curl = curlFix;
  aws-sdk-cpp = aws-sdk-cpp.override { curl = curlFix; };
}

EDIT: While this definitively improves the situation there are still cases where problems occur. For some bizarre reason substituting from my local cache still fails while cache.nixos.org now works consistently.

[matthewbauer]

Do we think this is a curl bug or a Nix bug? It could be the way that Nix is downloading with curl is wrong.

I just started hitting this after updating to 10.14.3.

[LnL7]

That's possible. But since this only started to happen with a recent nixpkgs/curl version I suspect it's curl or libresolv itself.

[corps]

Had to write a script to add the OBJC_DISABLE_INITIALIZE_FORK_SAFETY to coworker's nix-daemons when this started happening. In case anyone's curious, you can use PlistBuddy and a plist file to do this in a bash script pretty easily:

add-obj-c-fork-env.plist:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>OBJC_DISABLE_INITIALIZE_FORK_SAFETY</key>
  <string>YES</string>
</dict>
</plist>

set -e
set -o pipefail
NIX_DAEMON_PLIST=/Library/LaunchDaemons/org.nixos.nix-daemon.plist
set -x

sudo launchctl unload $NIX_DAEMON_PLIST

while ps -ax | grep -v grep | grep "nix-daemon"
  echo "Waiting for nix-daemon to stop..."
  sleep 1 # not great but generally the daemon closes very fast if you're not using it.
end

  sudo /usr/libexec/PlistBuddy -c "merge $PWD/add-obj-c-fork-env.plist EnvironmentVariables" $NIX_DAEMON_PLIST || true
  sudo launchctl load -w $NIX_DAEMON_PLIST

[jhenahan]

@corps One other option, if you use nix-darwin and don't want to hack plists, is to set launchd.daemons.nix-daemon.environment.OBJC_DISABLE_INITIALIZE_FORK_SAFETY = "YES"; in your config.

[dpausp]

Ran into the same problem and tried the workaround script, but I had to fix it. It doesn't need the separate plist file anymore:

~~~sh
set -x
set -e
set -o pipefail
NIX_DAEMON_PLIST=/Library/LaunchDaemons/org.nixos.nix-daemon.plist

sudo launchctl unload $NIX_DAEMON_PLIST

while ps -ax | grep -v grep | grep "nix-daemon"; do
echo "Waiting for nix-daemon to stop..."
sleep 1 # not great but generally the daemon closes very fast if you're not using it.
done

sudo /usr/libexec/PlistBuddy
-c "Add :EnvironmentVariables:OBJC_DISABLE_INITIALIZE_FORK_SAFETY string YES"
$NIX_DAEMON_PLIST || true
sudo launchctl load -w $NIX_DAEMON_PLIST
~~~

[mexisme]

Ran into the same problem and tried the workaround script, but I had to fix it. It doesn't need the separate plist file anymore:

It looks like @LnL7's 34d897e0b603bf2688809e726b073ead1cfc0c6a hasn't hit unstable yet?
Your fix worked for me, however, @dpausp.
Thanks!

[ldub]

I still get this issue, and the OBJC_DISABLE_INITIALIZE_FORK_SAFETY plist fix does not fix it for me.

This only happens in multi-user mode, regular nix install works fine.

Heres my nix-info from single user mode:

~ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-darwin"`
 - host os: `Darwin 18.5.0, macOS 10.14.4`
 - multi-user?: `no`
 - sandbox: `no`
 - version: `nix-env (Nix) 2.2.2`
 - channels(lev): `"nixpkgs-19.09pre184207.8789ff3179a"`
 - nixpkgs: `/Users/lev/.nix-defexpr/channels/nixpkgs`

[LnL7]

Are you sure the daemon is running with that variable?

$ launchctl print system/org.nixos.nix-daemon | grep OBJC_DISABLE_INITIALIZE_FORK_SAFETY
                OBJC_DISABLE_INITIALIZE_FORK_SAFETY => YES

[ldub]

EDIT 2: My issue was unrelated. I had some kind of mess left over from a previous nix install that prevented the default profile from being set up properly.

Click to see original comment

Yes, I do see exactly that. I also see that variable on a fresh install, without even running the fix (I'm assuming this made it into the latest release).

I sometimes see different error messages: sometimes it only says unexpected end-of-file, and sometimes it complains about a cert as well.

➜  ~ nix-shell -p nix-info --run "nix-info -m"
error (ignored): unable to download 'https://cache.nixos.org/jl50iznh81znnf2jw9fy32hs9m0s577d.narinfo': Problem with the SSL CA cert (path? access rights?) (77)
error: unexpected end-of-file
➜  ~ nix-shell -p nix-info --run "nix-info -m"
error (ignored): unable to download 'https://cache.nixos.org/7q9s8jlnz3m2079960jpbcpg4nsvrk1m.narinfo': Problem with the SSL CA cert (path? access rights?) (77)
error: unexpected end-of-file
➜  ~ nix-shell -p nix-info --run "nix-info -m"
error: unexpected end-of-file

A full -v --debug log: https://hastebin.com/alolafeteh.cs

➜  ~ cat /Library/LaunchDaemons/org.nixos.nix-daemon.plist
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
  <dict>
    <key>EnvironmentVariables</key>
    <dict>
      <key>OBJC_DISABLE_INITIALIZE_FORK_SAFETY</key>
      <string>YES</string>
    </dict>
    <key>Label</key>
    <string>org.nixos.nix-daemon</string>
    <key>KeepAlive</key>
    <true/>
    <key>RunAtLoad</key>
    <true/>
    <key>Program</key>
    <string>/nix/store/87i4fp46jfw9yl8c7i9gx75m5yph7irl-nix-2.2.2/bin/nix-daemon</string>
    <key>StandardErrorPath</key>
    <string>/var/log/nix-daemon.log</string>
    <key>StandardOutPath</key>
    <string>/dev/null</string>
  </dict>
</plist>
➜  ~ launchctl print system/org.nixos.nix-daemon | grep OBJC_DISABLE_INITIALIZE_FORK_SAFETY
        OBJC_DISABLE_INITIALIZE_FORK_SAFETY => YES

Here's what I've seen in /var/log/nix-daemon.log, though I'm not sure when these lines were logged:

dyld: Library not loaded: /nix/store/bsnc02rq3s4304s87pk96s8cs45qa7ag-boehm-gc-7.6.6/lib/libgc.1.dylib
  Referenced from: /nix/store/87i4fp46jfw9yl8c7i9gx75m5yph7irl-nix-2.2.2/bin/nix-daemon
  Reason: image not found
accepted connection from pid <unknown>, user lev
libc++abi.dylib: terminating with uncaught exception of type nix::Interrupted: interrupted by the user
accepted connection from pid <unknown>, user lev

Also, @LnL7 thank you for the quick response. I much appreciate it. I think the cert problem means that I'm commenting in the wrong issue. Do you know what the right place to triage this would be? Please let me know if there is some other log I can provide to help diagnose the issue.

EDIT:

I think its related to the fact that in the multi-user install, my ~/.nix-profile is a dangling symlink:

single-user install:

➜  ~ ls -la /nix/var/nix/profiles/per-user/lev
total 0
drwxr-xr-x  7 lev  wheel  224 Jun 28 19:51 .
drwxr-xr-x  3 lev  wheel   96 Jun 28 19:51 ..
lrwxr-xr-x  1 lev  wheel   15 Jun 28 19:51 channels -> channels-1-link
lrwxr-xr-x  1 lev  wheel   60 Jun 28 19:51 channels-1-link -> /nix/store/q889jm9c9bf5xryjlqf1mb8h97586hfl-user-environment
lrwxr-xr-x  1 lev  wheel   14 Jun 28 19:51 profile -> profile-2-link
lrwxr-xr-x  1 lev  wheel   60 Jun 28 19:51 profile-1-link -> /nix/store/x5qa5dsjn37nhrymy6s0b03hj2k2ljch-user-environment
lrwxr-xr-x  1 lev  wheel   60 Jun 28 19:51 profile-2-link -> /nix/store/wc3lcg14yh4mpk8wkv20dp7ib1k7kbnn-user-environment

multi-user install just doesn't have the profile and profile*-link symlinks:

➜  ~ ls -la /nix/var/nix/profiles/per-user/lev
total 0
drwxr-xr-x  7 lev  wheel  224 Jun 28 19:51 .
drwxr-xr-x  3 lev  wheel   96 Jun 28 19:51 ..
lrwxr-xr-x  1 lev  wheel   15 Jun 28 19:51 channels -> channels-1-link
lrwxr-xr-x  1 lev  wheel   60 Jun 28 19:51 channels-1-link -> /nix/store/q889jm9c9bf5xryjlqf1mb8h97586hfl-user-environment

And so the SSL certs in ~/.nix-profile/etc/ssl/ are obviously not found.

[angerman]

In addition to what @ldub said, you'll see

error: unexpected end-of-file

as well if /etc/ssl/certs/ca-certificates.crt is a dead symlink.

[mpfaff]

I've just had a very similar issue to @ldub, the difference being that my nix-profile was completely empty, not even the channel files.

[webframp]

@ldub How did you fix your issue? I see you describe your problem but there's no details on the fix

[ldub]

@webframp my issue was that ~/.nix-profile was a dangling symlink. Unlikely to be related to you problem.