darling hangs and /dev/mach has the wrong permissions

[ahyattdev]

darling shell hangs indefinetly. There is no output.

/dev/mach has 700 permissions by default.

dyld64 complains that it can't open /dev/mach. This was fixed by changing the permissions of the device. After this fix, dyld64 can launch bash. This fix does not make darling shell work.

[bugaevc]

Oh, so I have broken stuff after all. Try running darling shutdown and then darling shell again. If it doesn't help, try fully uninstalling darling and reinstalling it again — some of my recent changes might require this (although they shouldn't). Definitely clean up your prefix & let darling shell recreate it.

[bugaevc]

If this doesn't help, let me see the output of darling shutdown && sudo strace -f -u $USER darling shell

[ahyattdev]

darling-shutdown reports that no container is running.

I did sudo rm -rf /usr/local/libexec/darling /usr/local/lib/darling/ ~/.darling and reinstalled. It did nothing.

I checked htop and 4 instances of darling shell were running in the background. I killed them. I then ran darling-shell again and it gave this error:

/Volumes/SystemRoot/usr/local/bin/dyld64: error while loading shared libraries: libdyld.so: cannot open shared object file: No such file or directory

Here is the log from strace.

[bugaevc]

So, obviously, it cannot find libdyld.so, which is located under /Volumes/SystemRoot/usr/local/lib/darling/ -- and it should know to look there because we explicitly set $LD_LIBRARY_PATH to this folder here -- but it doesn't even try to look there, unlike on my system, where it does and finds it successfully.

Maybe try to change dyld here for "/Volumes/SystemRoot/bin/busybox", (const char *[2]) {"sh", NULL} and poke around to see if the environment variable is set up correctly and whether the library is indeed accessible through that path.

[ahyattdev]

At runtime, this is what it looks like:

SYSTEM_ROOT:  /Volumes/SystemRoot
LIB_PATH:  /usr/local/lib//darling
LD_LIBRARY_PATH: /Volumes/SystemRoot/usr/local/lib//darling

I guess this means that /Volumes/SystemRoot isn't being set up properly.

I am on Xubuntu 16.04.

[bugaevc]

So can you run e.g. BusyBox inside the target namespace (or you can use nsenter) to see what's underneath /Volumes/SystemRoot? What seems odd to me is that ld-linux doesn't even try to look there, based on the log from strace.

[ahyattdev]

I used nsenter -t <PID> to check out the state of it. /Volumes/does not exist. No overlays are made. It is the same as my Linux system root.

[bugaevc]

You mean you used nsenter -t <PID> -m (to enter the mount namespace), right? Otherwise it won't do anything.

If you did and it shows the regular Linux layout, then it means prelaunchd is setting up the namespace incorrectly. Your strace log only shows attaching to an already running prelaunchd, can you kill it (darling shutdown should work, by the way, it's really strange if it doesn't) and get me a new log (with both prelaunchd and bash)?

[ahyattdev]

You are right, I wasn't mounting it. This is the result after running darling shtudown (it works) then darling shell, which crashes because it can't find libdyld, but the other process continues running.

andrewhyatt@andrewhyatt-VirtualBox:~$ sudo nsenter -t 21132 -m
Failed to execute self through Darling dyld.
Illegal instruction (core dumped)

[bugaevc]

Right, and that's because it's trying to execute bash and either can't find dyld or ld-linux (which is, again, very strange, because the previous bash launched from darling shell found them successfully). Can you cat /proc/21132/mounts to see what's mounted?

[ahyattdev]

/dev/mapper/xubuntu--vg-root /Volumes/SystemRoot ext4 rw,relatime,errors=remount-ro,data=ordered 0 0
udev /Volumes/SystemRoot/dev devtmpfs rw,nosuid,relatime,size=2002280k,nr_inodes=500570,mode=755 0 0
devpts /Volumes/SystemRoot/dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
tmpfs /Volumes/SystemRoot/dev/shm tmpfs rw,nosuid,nodev 0 0
hugetlbfs /Volumes/SystemRoot/dev/hugepages hugetlbfs rw,relatime 0 0
mqueue /Volumes/SystemRoot/dev/mqueue mqueue rw,relatime 0 0
tmpfs /Volumes/SystemRoot/run tmpfs rw,nosuid,noexec,relatime,size=404628k,mode=755 0 0
tmpfs /Volumes/SystemRoot/run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0
tmpfs /Volumes/SystemRoot/run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=404628k,mode=700,uid=1000,gid=1000 0 0
gvfsd-fuse /Volumes/SystemRoot/run/user/1000/gvfs fuse.gvfsd-fuse rw,nosuid,nodev,relatime,user_id=1000,group_id=1000 0 0
sysfs /Volumes/SystemRoot/sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
securityfs /Volumes/SystemRoot/sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0
tmpfs /Volumes/SystemRoot/sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0
cgroup /Volumes/SystemRoot/sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd 0 0
cgroup /Volumes/SystemRoot/sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0
cgroup /Volumes/SystemRoot/sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0
cgroup /Volumes/SystemRoot/sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0
cgroup /Volumes/SystemRoot/sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0
cgroup /Volumes/SystemRoot/sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0
cgroup /Volumes/SystemRoot/sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0
cgroup /Volumes/SystemRoot/sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0
cgroup /Volumes/SystemRoot/sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0
cgroup /Volumes/SystemRoot/sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0
cgroup /Volumes/SystemRoot/sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0
pstore /Volumes/SystemRoot/sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0
debugfs /Volumes/SystemRoot/sys/kernel/debug debugfs rw,relatime 0 0
fusectl /Volumes/SystemRoot/sys/fs/fuse/connections fusectl rw,relatime 0 0
proc /Volumes/SystemRoot/proc proc rw,nosuid,nodev,noexec,relatime 0 0
systemd-1 /Volumes/SystemRoot/proc/sys/fs/binfmt_misc autofs rw,relatime,fd=36,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0
binfmt_misc /Volumes/SystemRoot/proc/sys/fs/binfmt_misc binfmt_misc rw,relatime 0 0
/dev/sda1 /Volumes/SystemRoot/boot ext2 rw,relatime,block_validity,barrier,user_xattr,acl 0 0
andyhyatt /Volumes/SystemRoot/media/sf_andyhyatt vboxsf rw,nodev,relatime 0 0
gvfsd-fuse /Volumes/SystemRoot/home/andrewhyatt/.gvfs fuse.gvfsd-fuse rw,nosuid,nodev,relatime,user_id=0,group_id=0 0 0
overlay / overlay rw,relatime,lowerdir=/usr/local/libexec/darling,upperdir=/home/andrewhyatt/.darling,workdir=/home/andrewhyatt/.darling.workdir 0 0
proc /proc proc rw,relatime 0 0

[ahyattdev]

When I run tree on the darling prefix in my home dir:
The setup didn't finish.

.darling
├── Applications
├── usr
│   └── local
│       └── share
└── Volumes

[bugaevc]

Looks completely normal.

When I run tree on the darling prefix in my home dir: The setup didn't finish.

No, this is how it should be now.

[bugaevc]

I think you'd need to do what I said here:

Maybe try to change dyld here for "/Volumes/SystemRoot/bin/busybox", (const char *[2]) {"sh", NULL} and poke around to see if the environment variable is set up correctly and whether the library is indeed accessible through that path.

That is, change the source to start busybox shell instead of Darling bash — and see what's there from inside.

On a somewhat unrelated note, is this reproducible on a clean Xubuntu + Darling installation? I should probably try it myself when I have time.

[ahyattdev]

That got me into the prefix.

libdyld.so is at the path it should expect.

/Volumes/SystemRoot/home/andrewhyatt $ stat /Volumes/SystemRoot/usr/local/lib/darling/libdyld.so
  File: /Volumes/SystemRoot/usr/local/lib/darling/libdyld.so
  Size: 7306368     Blocks: 14272      IO Block: 4096   regular file
Device: fc00h/64512d    Inode: 1319615     Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2017-01-07 17:30:10.000000000
Modify: 2017-01-06 02:22:47.000000000
Change: 2017-01-07 22:28:50.000000000

[ahyattdev]

Strange!

/Volumes/SystemRoot/home/andrewhyatt $ echo $LD_LIBRARY_PATH
/Volumes/SystemRoot/usr/local/lib//darling

/Volumes/SystemRoot/home/andrewhyatt $ /Volumes/SystemRoot/usr/local/bin/dyld64
/Volumes/SystemRoot/usr/local/bin/dyld64: error while loading shared libraries: libdyld.so: cannot open shared object file: No such file or directory

Yes, it is an unmodified installation of Xubuntu in a VM I just use for this.

[bugaevc]

Strange!

Exactly 😕

No idea why this could happen, especially given that it works for me (and probably for @LubosD since he accepted my pull request).

[bugaevc]

Yes, it is an unmodified installation of Xubuntu in a VM I just use for this.

Well, I'll try it myself tomorrow afternoon then.

[LubosD]

I'm just writing in that it seems to work fine for me, currently on my laptop with Debian Testing.

[bugaevc]

Okay now this is interesting. I'm trying to build Darling under Xubuntu 16.04, and I get the following error:

[ 29%] Linking CXX shared library libicucore.so
/usr/bin/ld: BFD (GNU Binutils for Ubuntu) 2.26.1 assertion fail ../../bfd/elflink.c:2677
/usr/bin/ld: BFD (GNU Binutils for Ubuntu) 2.26.1 assertion fail ../../bfd/elflink.c:2677
clang: error: linker command failed with exit code 1 (use -v to see invocation)
src/external/icu/icuSources/CMakeFiles/icucore.dir/build.make:9683: recipe for target 'src/external/icu/icuSources/libicucore.so' failed
make[2]: *** [src/external/icu/icuSources/libicucore.so] Error 1
CMakeFiles/Makefile2:6527: recipe for target 'src/external/icu/icuSources/CMakeFiles/icucore.dir/all' failed
make[1]: *** [src/external/icu/icuSources/CMakeFiles/icucore.dir/all] Error 2
Makefile:149: recipe for target 'all' failed
make: *** [all] Error 2

This is obviously not a problem with darling, but with ld. /usr/bin/ld --version reports version 2.26.1 -- are you using a different one?

[ahyattdev]

I am running the same version of ld. Even more strange. 😕

[bugaevc]

So I decided to try some other version of Ubuntu and picked Kubuntu 16.04. Everything has built just fine this time (although it's running the exact same version of ld). I can reproduce both darling shell hanging and darling shutdown not seeing an already running container. Let's see whether I can figure this out.

[bugaevc]

Well that was a really stupid race condition. I'm surprised it didn't come up before. Expect a fix in a few minutes. I'm not sure it'll resolve the dylib problem though.

[bugaevc]

I can also reproduce the build error mentioned in #224:

awk: line 1: regular expression compile failed (bad class -- [], [^] or [)
^[^
awk: line 1: syntax error at or near ]
awk: line 1: regular expression compile failed (bad class -- [], [^] or [)
^[^
awk: line 1: syntax error at or near ]
awk: line 1: regular expression compile failed (bad class -- [], [^] or [)
^[^
awk: line 1: syntax error at or near ]

This seems to be causing the problem with finding the shared libs (it can't find libstdc++ in my case, and it _does_ find libdyld)

[bugaevc]

The latest commit (https://github.com/darlinghq/darling/commit/dce27e3d6017e4a5bf5865616142ab230e6082f0) fixes this for me and I'm able to normally start the Darling shell.

Please reinstall Darling and test whether this works for you.

[ahyattdev]

No more hangs, that fixed it!

/Volumes/SystemRoot/usr/local/bin/dyld64: error while loading shared libraries: libdyld.so: cannot open shared object file: No such file or directory is still an issue.

[bugaevc]

I'm trying to build Darling under Xubuntu 16.04, and I get the following error

I decided to reinstall Xubuntu to see if that error would go away, and it did! Darling built cleanly and worked out-of-the-box.

Maybe you too have a bug in ld-linux that's specific to your installation?

[bugaevc]

I know what's going on!

Your dyld64 probably has cap_sys_rawio=ep set, and this makes ld-linux ignore LD_LIBRARY_PATH for security reasons, as it would be in case of SETUID files.

The proper way to fix this is to stop using LD_LIBRARY_PATH. Copy-paste from #222:

Pass LD_LIBRARY_PATH for /usr/local/lib/darling. This is because I couldn't get it to work with /etc/ld.so.conf.d/darling.conf; this won't be hopefully needed after the Great Mach-O Bringup (see #221).

So we should either make it work with /etc/ld.so.conf.d/darling.conf (note: I'm aware that the file order there matters -- the problem is somewhere else) or wait for the Mach-O bringup. In the meantime, you can just unset the capability.

Also see darlinghq/darling-installer#3

[LubosD]

Or we could add an extra RPATH, then we wouldn't need LD_LIBRARY_PATH.

[bugaevc]

@LubosD sounds great -- just note that ld-linux(8) says DT_RPATH is deprecated in favor of DT_RUNPATH

[ahyattdev]

@bugaevc cap_sys_rawio was exectly the issue. I can get into the shell now. Thanks!

[chuckhacker]

@bugaevc Also getting the assertion error. Ideally, I don't reinstall my entire OS on my work laptop.

[bugaevc]

@TheChuckster what are you talking about, what assertion? Why would you have to reinstall your entire OS? Please make a proper bug report, ideally with details of your setup and steps to reproduce.

[chuckhacker]

@bugaevc CTRL+F assertion

[chuckhacker]

OS reinstall was also part of this discussion. As you can imagine, this one is going to be hard to repro, but maybe you could still take a look?

[bugaevc]

@TheChuckster Ah, if you're talking about the assertion in ld that I was getting, then this is obviously not an issue with Darling, but a symptom of ld being corrupted.

There's nothing we (as Darling developers) can do to help you, but I may suggest you instruct your package manager to reinstall the package that contains ld (e.g. sudo apt reinstall binutils in case of Ubuntu)