Delve: On macOS a SIGSEGV (EXC_BAD_ACCESS) can not be propagated back to the target

[mnelson-bah]

1. What version of Delve are you using (dlv version)?
   Version: 1.0.0-rc.1
2. What version of Go are you using? (go version)?
   go version go1.8.2 darwin/amd64
   note: also occurs with 1.8.1
3. What operating system and processor architecture are you using?
   OSX El Capitan
4. What did you do?
   Attempt to debug go program that uses go-kit/kit/log
5. What did you expect to see?
   Program runs normally
6. What did you see instead?
   Program stops at go-stack/stack/stack.go/stack.go:228

Here is the source code:
~~~~
package main

import (
"os"
"github.com/go-kit/kit/log"
)

func main() {
log.NewLogfmtLogger(os.Stdout)
}
~~~~

Here's what the delve session looks like:
~~~~
~/dev/go/src/mat/foo $ dlv debug
Type 'help' for list of commands.
(dlv) continue

mat/vendor/github.com/go-stack/stack.findSigpanic.func1() /Users/mathewnelson/dev/go/src/mat/vendor/github.com/go-stack/stack/stack.go:228 (PC: 0x10b8ec3)
223: }
224: }
225: }
226: }()
227: // intentional nil pointer dereference to trigger sigpanic
=> 228: return *p
229: }()
230: return fn
231: }
232:
233: var sigpanic = findSigpanic()
(dlv)
~~~~

This was working with 0.12.2.

Thank you.

[aarzilli]

lldb and debugserver can not recover from a EXC_BAD_ACCESS, this is an upstream bug that's been open for years: https://bugs.llvm.org//show_bug.cgi?id=22868.

[aarzilli]

I'll leave this open for reference, the upstream in question is LLVM/OSX, not Go.

[dkushner]

@aarzilli, in addition to the outstanding bugs in the native backend you mentioned, this workaround does not work for many systems with endpoint security enabled (most company-issued laptops) due to #714.

If you could help me clarify, the issue here seems to be that go-kit/kit/log is forcing a panic to grab the stack and therefore the call location for the logging method. Is there any other way they could approach this? How can this be done in a way that would not interfere with this limitation of LLVM? Just trying to grab some info to perhaps open up an issue/PR on their repo.

I totally recognize its not your job to fix others' code, just curious what alternative approaches may work here since the provided workaround is untenable for me (and many others) and that LLVM bug may outlive us all.

[dkushner]

Ah, just found go-stack/stack#11. It looks like the author is aware of this issue and a workaround and has simply not gotten around to implementing it.

[aarzilli]

There's actually a much smaller fix, wherever go-stack does fn != sigpanic it could do fn.Name() != "runtime.sigpanic".

[ChrisHines]

This issue should be fixed in the go-stack project: https://github.com/go-stack/stack/releases/tag/v1.5.4

[dlsniper]

@aarzilli, do you think there's a better way to inform the users about this issue? See https://youtrack.jetbrains.com/issue/GO-5183#comment=27-2658938 for reference. Thank you!

[aarzilli]

We should probably print a better error message client side "process received signal: bad access" and then print the listing normally. Other clients should do the same, I don't think there's anything more that we can do.

[jedops]

Hello, I'm seeing this on some external libraries other than those in go-stack/go-kit (particularly in Kubernetes) which effectively makes dlv impossible to use for debugging certain in these cases.

I've attached a gist which I believe is a cut down example of how this issue can occur.

https://gist.github.com/jedops/83b26c2a439772a2de89403d1d5c5c77

go test
will pass against this, but
dlv test
will crash

In the event that a fix within delve is not possible. Is anyone aware of a workaround we could implement so that we can use the debugger?

[aarzilli]

See https://github.com/go-delve/delve/issues/1371#issuecomment-429555169

[daxmc99]

@aarzilli I have a Mac and I am willing to build lldb and the debug server. From my understanding distributing binaries may be difficult but perhaps we could document how to build your own lldb to work around this?

[aarzilli]

@daxmc99 See #1371

[daxmc99]

I'm not very familiar with how delve works internally but would I simply need to add the --backend lldb flag for delve to pick up the custom lldb and debugserver I have set in my PATH?

[aarzilli]

The path to debugserver is hardcoded in pkg/proc/gdbserial/gdbserver.go (there's a constant called debugserverExecutable), you would have to change that.

[CermakM]

Is there any workaround for this? This is really a pain.

[daxmc99]

I haven't experienced this since upgrading to Go 1.15.4 🤞
I think https://github.com/golang/go/commit/1b49a86ecece3978ceba60c372327b9cbcc68501 may provide the workaround

Building the lldb debug server manually wasn't straightforward or easy 😞

[aarzilli]

Note: even though this bug is closed you will continue to experience it until a version of debugserver with the corresponding patch is installed. At the time of this message no released version of xcode command line tools has it.