Darling: Commands in bash crash with SIGSEGV. Works if ptrace() on child

[yan12125]

First, I add the following Arch-specific patch for successful running: (see https://github.com/darlinghq/darling/issues/221#issuecomment-285445186 for previous discussions)

diff --git a/src/dyld/darling.c b/src/dyld/darling.c
index b989d0f1..0db1d204 100644
--- a/src/dyld/darling.c
+++ b/src/dyld/darling.c
@@ -354,6 +354,8 @@ void setupChild(const char *curPath)
        setenv("PWD", buffer2, 1);
        chdir(buffer2);
    }
+
+   setenv("LD_LIBRARY_PATH", "/lib", 1);
 }

 void showHelp(const char* argv0)

Then I built darling on using-machos-experiment branch with the following command:

mkdir -p build/x86-64
cd build/x86-64
cmake ../.. -DCMAKE_TOOLCHAIN_FILE=../../Toolchain.cmake -DCMAKE_BUILD_TYPE=Debug
make -j1
sudo make install

And external commands fail with SIGSEGV. For example:

$ darling shell
Darling [~/Projects/tmp/darling/src/lkm]$ ls
Segmentation fault: 11 (core dumped)
Darling [~/Projects/tmp/darling/src/lkm]$ 

There are two workarounds:

1. sudo strace -fp $(pgrep mldr) >& /dev/null (I redirect all outputs to /dev/null or darling will be slowed down)
2. sudo gdb, attach the mldr process and set follow-fork-mode child

Seems if I call ptrace on the child then everything works. As a result I can't debug it :/

Environment: Arch Linux x86_64, Linux kernel 4.10.1, clang 3.9.1

[bugaevc]

Seems if I call ptrace on the child then everything works. As a result I can't debug it :/

But you can look at the produced core file, right?

Does it work for you on other distros?

[bugaevc]

cc @X0rg

[X0rg]

I'll take a look, but I don't think I can help you.
The package I maintain use master branch, not using-machos-experiment.

[yan12125]

But you can look at the produced core file, right?

Thanks for reminding me that.

$ gdb /usr/local/libexec/darling/bin/mldr core.mldr.1000.139fbdf3dd18476c87fc1bae9338f305.18384.1489168793000000000000
Reading symbols from /usr/local/libexec/darling/bin/mldr...done.
[New LWP 51]
Core was generated by `/bin/mldr!/bin/ls ls'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f6cf37a4d80 in ?? ()
(gdb) info proc mappings 
Mapped address spaces:

          Start Addr           End Addr       Size     Offset objfile
            0x400000           0x607000   0x207000        0x0 /bin/mldr
      0x7f6cf37a4000     0x7f6cf37c7000    0x23000        0x0 /Volumes/SystemRoot/usr/lib/ld-2.25.so

Seems the crash is in ld.so. I'll rebuild glibc with debugging symbols and come back later.

[yan12125]

Turns out that debugging symbols don't help becase the crash occurs in hand-written assembly codes. Here are partial results from objdump --disassemble-all /usr/lib/ld-2.25.so:

0000000000000d5b <oom>:
     d5b:   48 8d 35 ce e6 01 00    lea    0x1e6ce(%rip),%rsi        # 1f430 <__PRETTY_FUNCTION__.8380+0x78>
     d62:   bf 02 00 00 00          mov    $0x2,%edi
     d67:   48 83 ec 08             sub    $0x8,%rsp
     d6b:   31 c0                   xor    %eax,%eax
     d6d:   e8 ce f5 00 00          callq  10340 <_dl_dprintf>
     d72:   bf 7f 00 00 00          mov    $0x7f,%edi
     d77:   e8 84 86 01 00          callq  19400 <_Exit>
     d7c:   0f 1f 40 00             nopl   0x0(%rax)

0000000000000d80 <_start>:
     d80:   48 89 e7                mov    %rsp,%rdi
     d83:   e8 e8 3d 00 00          callq  4b70 <_dl_start>

And disassemble results from gdb:

(gdb) disassemble/r 0x00007f243c082d5b,+50
Dump of assembler code from 0x7f243c082d5b to 0x7f243c082d8d:
   0x00007f243c082d5b:  48 8d 35 ce e6 01 00    lea    rsi,[rip+0x1e6ce]        # 0x7f243c0a1430
   0x00007f243c082d62:  bf 02 00 00 00  mov    edi,0x2
   0x00007f243c082d67:  48 83 ec 08     sub    rsp,0x8
   0x00007f243c082d6b:  31 c0   xor    eax,eax
   0x00007f243c082d6d:  e8 ce f5 00 00  call   0x7f243c092340
   0x00007f243c082d72:  bf 7f 00 00 00  mov    edi,0x7f
   0x00007f243c082d77:  e8 84 86 01 00  call   0x7f243c09b400
   0x00007f243c082d7c:  0f 1f 40 00     nop    DWORD PTR [rax+0x0]
=> 0x00007f243c082d80:  48 89 e7        mov    rdi,rsp
   0x00007f243c082d83:  e8 e8 3d 00 00  call   0x7f243c086b70
   0x00007f243c082d88:  49 89 c4        mov    r12,rax
   0x00007f243c082d8b:  8b 05 c7 1e 22 00       mov    eax,DWORD PTR [rip+0x221ec7]        # 0x7f243c2a4c58

Seems the mldr process crashes just before _start of ld.so... (broken execve()?)

[LubosD]

It smells like https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=749122

Try removing this line from src/dyld/CMakeLists.txt:

set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -Wl,-Ttext-segment,0x400000 -Wl,-Tbss,0x410000 -Wl,-Tdata,0x420000")

[yan12125]

Thanks for the tip, but removing that line doesn't work. Child processes still crash with SIGSEGV. (And ptrace() workaround still works)

[LubosD]

Remember to always use darling shutdown before retrying, because overlayfs may be caching the file that has changed (mldr in this case).

[yan12125]

Sorry, I didn't know that command. To prevent any possible inconsistent states in the kernel, I reboot my machine. I also removed ~/.darling and ~/.darling.workdir before trying again. The result is the same.

[LubosD]

For now, I've fixed the ld.so search path problem. "Good" news is I can reproduce the SIGSEGV problem. It even triggers a kernel BUG :-D

[LubosD]

OK, so I wasn't able to figure anything out. Even if mldr is built as a static executable, which makes its loading a trivial case of mapping a few segments into memory, it still segfaults on the very first instruction. This smells like a kernel bug. Hell, it even prints BUG into dmesg :-)

Arch uses Linux 4.10, the newest kernel I have running elsewhere is 4.9. @yan12125 Could you possibly try running an older kernel if it makes any difference? My Arch Linux skills are close to zero.

[X0rg]

The linux-lts package is 4.9, so if you don't already have it @yan12125, install it. 😉

[yan12125]

The linux-lts package is 4.9, so if you don't already have it @yan12125, install it

Thanks for the tip. I also installed linux-lts-headers to compile lkm for 4.9.

Could you possibly try running an older kernel if it makes any difference?

On 4.9 commands work fine without any SIGSEGV, and mldr don't trigger kernel bugs. Awesome!

Hell, it even prints BUG into dmesg :-)

Yep I noticed that too. The bug "scheduling while atomic" also occurs when the ptrace() workaround is applied. So I guess they are different issues.

[yan12125]

Good news - there are no longer segmentation faults with newlkm! Tested on 4.10.8-1-ARCH. (There were still segfaults with the current lkm on 4.10.8)

[LubosD]

@yan12125 Are you sure? Because the segfault was happening before /dev/mach was even opened. At least that was the case when I tried it in a VM.

[yan12125]

I rebuilt everything - darling shutdown, sudo rmmod darling_mach, remove files under /usr/local/libexec/darling and /usr/local/bin/darling, git clean -dfx, check nothing is left via git status, cmake, make, sudo make install, make in src/lkm, sudo insmod darling-mach.ko, ...

Yes, it works.

[ahyattdev]

I think we can close this now