Delve: Cannot debug go plugins on mac

[nd]

1. What version of Delve are you using (dlv version)?

Delve Debugger
Version: 1.2.0
Build: cb658772972d74042ba75f006ac14023e552c7cd

1. What version of Go are you using? (go version)?

go version go1.12.6 darwin/amd64

1. What operating system and processor architecture are you using?

darwin/amd64

1. What did you do?

I'm trying to debug a simple go plugin from https://github.com/vladimirvivien/go-plugin-example.

I've build plugins using the command:

go build -gcflags "all=-N -l" -x -v -buildmode=plugin -o eng/eng.so eng/greeter.go

I'm debugging the code calling the plugin (https://github.com/vladimirvivien/go-plugin-example/blob/master/greeter.go).

Once debugger reaches the breakpoint on line 58, I'm adding a breakpoint at eng/greeter.go:8 and continue or trying to step into the call.

1. What did you expect to see?

Adding a breakpoint inside the plugin succeeds. If I continue, debugger hits the breakpoint inside the plugin. Step into the call in plugin also works.

1. What did you see instead?

Create breakpoint call fails with the "could not find file /Users/nd/go/src/github.com/vladimirvivien/go-plugin-example/eng/greeter.go" error. Step into plugin call also fails.

Debug log: https://gist.github.com/nd/feb71212f6ef9dcfbb33c4293e18f146

Debug works fine on linux.

[juhwany]

Debugging plugin on mac is really needed for Golang developers. Hope delve supports it :)

[aarzilli]

I didn't try implementing this because it would be affected by https://github.com/golang/go/issues/25841 at least. I also vaguely remembering some versions of go not producing any dwarf at all for macOS. What's the output of readelf -h -S path_to_a_go_plugin.so?

[nd]

It returns an error:

readelf: Error: Not an ELF file - it has the wrong magic bytes at the start

[aarzilli]

It returns an error:

Right, of course, because they aren't elfs. My bad. I guess the right one would be otool -l or something like that. Or gobjdump -h ... if it's installed.

[nd]

otool -l output:

eng/eng.so:
Mach header
      magic cputype cpusubtype  caps    filetype ncmds sizeofcmds      flags
 0xfeedfacf 16777223          3  0x00           6    13       1984 0x00100004
Load command 0
      cmd LC_SEGMENT_64
  cmdsize 392
  segname __TEXT
   vmaddr 0x0000000000000000
   vmsize 0x00000000000dd000
  fileoff 0
 filesize 905216
  maxprot 0x00000007
 initprot 0x00000005
   nsects 4
    flags 0x0
Section
  sectname __text
   segname __TEXT
      addr 0x0000000000002040
      size 0x00000000000dabd0
    offset 8256
     align 2^4 (16)
    reloff 0
    nreloc 0
     flags 0x80000400
 reserved1 0
 reserved2 0
Section
  sectname __stubs
   segname __TEXT
      addr 0x00000000000dcc10
      size 0x0000000000000156
    offset 904208
     align 2^1 (2)
    reloff 0
    nreloc 0
     flags 0x80000408
 reserved1 0 (index into indirect symbol table)
 reserved2 6 (size of stubs)
Section
  sectname __stub_helper
   segname __TEXT
      addr 0x00000000000dcd68
      size 0x000000000000024a
    offset 904552
     align 2^2 (4)
    reloff 0
    nreloc 0
     flags 0x80000400
 reserved1 0
 reserved2 0
Section
  sectname __unwind_info
   segname __TEXT
      addr 0x00000000000dcfb4
      size 0x0000000000000048
    offset 905140
     align 2^2 (4)
    reloff 0
    nreloc 0
     flags 0x00000000
 reserved1 0
 reserved2 0
Load command 1
      cmd LC_SEGMENT_64
  cmdsize 1112
  segname __DATA
   vmaddr 0x00000000000dd000
   vmsize 0x0000000000122000
  fileoff 905216
 filesize 1064960
  maxprot 0x00000007
 initprot 0x00000003
   nsects 13
    flags 0x0
Section
  sectname __nl_symbol_ptr
   segname __DATA
      addr 0x00000000000dd000
      size 0x0000000000000010
    offset 905216
     align 2^3 (8)
    reloff 0
    nreloc 0
     flags 0x00000006
 reserved1 57 (index into indirect symbol table)
 reserved2 0
Section
  sectname __got
   segname __DATA
      addr 0x00000000000dd010
      size 0x0000000000001500
    offset 905232
     align 2^3 (8)
    reloff 0
    nreloc 0
     flags 0x00000006
 reserved1 59 (index into indirect symbol table)
 reserved2 0
Section
  sectname __la_symbol_ptr
   segname __DATA
      addr 0x00000000000de510
      size 0x00000000000001c8
    offset 910608
     align 2^3 (8)
    reloff 0
    nreloc 0
     flags 0x00000007
 reserved1 731 (index into indirect symbol table)
 reserved2 0
Section
  sectname __mod_init_func
   segname __DATA
      addr 0x00000000000de6d8
      size 0x0000000000000008
    offset 911064
     align 2^3 (8)
    reloff 0
    nreloc 0
     flags 0x00000009
 reserved1 0
 reserved2 0
Section
  sectname __rodata
   segname __DATA
      addr 0x00000000000de6e0
      size 0x0000000000052d58
    offset 911072
     align 2^5 (32)
    reloff 0
    nreloc 0
     flags 0x00000000
 reserved1 0
 reserved2 0
Section
  sectname __typelink
   segname __DATA
      addr 0x0000000000131440
      size 0x0000000000001434
    offset 1250368
     align 2^5 (32)
    reloff 0
    nreloc 0
     flags 0x00000000
 reserved1 0
 reserved2 0
Section
  sectname __itablink
   segname __DATA
      addr 0x0000000000132878
      size 0x00000000000000a0
    offset 1255544
     align 2^3 (8)
    reloff 0
    nreloc 0
     flags 0x00000000
 reserved1 0
 reserved2 0
Section
  sectname __gosymtab
   segname __DATA
      addr 0x0000000000132918
      size 0x0000000000000000
    offset 1255704
     align 2^0 (1)
    reloff 0
    nreloc 0
     flags 0x00000000
 reserved1 0
 reserved2 0
Section
  sectname __gopclntab
   segname __DATA
      addr 0x0000000000132920
      size 0x000000000009a6de
    offset 1255712
     align 2^5 (32)
    reloff 0
    nreloc 0
     flags 0x00000000
 reserved1 0
 reserved2 0
Section
  sectname __noptrdata
   segname __DATA
      addr 0x00000000001cd000
      size 0x000000000000cbdc
    offset 1888256
     align 2^5 (32)
    reloff 0
    nreloc 0
     flags 0x00000000
 reserved1 0
 reserved2 0
Section
  sectname __data
   segname __DATA
      addr 0x00000000001d9be0
      size 0x0000000000006ad0
    offset 1940448
     align 2^5 (32)
    reloff 0
    nreloc 0
     flags 0x00000000
 reserved1 0
 reserved2 0
Section
  sectname __bss
   segname __DATA
      addr 0x00000000001e06c0
      size 0x000000000001b830
    offset 0
     align 2^5 (32)
    reloff 0
    nreloc 0
     flags 0x00000001
 reserved1 0
 reserved2 0
Section
  sectname __noptrbss
   segname __DATA
      addr 0x00000000001fbf00
      size 0x0000000000002518
    offset 0
     align 2^5 (32)
    reloff 0
    nreloc 0
     flags 0x00000001
 reserved1 0
 reserved2 0
Load command 2
      cmd LC_SEGMENT_64
  cmdsize 72
  segname __LINKEDIT
   vmaddr 0x00000000001ff000
   vmsize 0x0000000000072000
  fileoff 1970176
 filesize 465632
  maxprot 0x00000007
 initprot 0x00000001
   nsects 0
    flags 0x0
Load command 3
          cmd LC_ID_DYLIB
      cmdsize 112
         name /var/folders/qv/lg43663s45jbmrkjvy64zh5r0000gn/T/go-build695961820/b001/exe/a.out.so (offset 24)
   time stamp 1 Thu Jan  1 01:00:01 1970
      current version 0.0.0
compatibility version 0.0.0
Load command 4
            cmd LC_DYLD_INFO_ONLY
        cmdsize 48
     rebase_off 1970176
    rebase_size 21536
       bind_off 1991712
      bind_size 65752
  weak_bind_off 0
 weak_bind_size 0
  lazy_bind_off 2057464
 lazy_bind_size 1080
     export_off 2058544
    export_size 43032
Load command 5
     cmd LC_SYMTAB
 cmdsize 24
  symoff 2105928
   nsyms 9259
  stroff 2257224
 strsize 178584
Load command 6
            cmd LC_DYSYMTAB
        cmdsize 80
      ilocalsym 0
      nlocalsym 6646
     iextdefsym 6646
     nextdefsym 2555
      iundefsym 9201
      nundefsym 58
         tocoff 0
           ntoc 0
      modtaboff 0
        nmodtab 0
   extrefsymoff 0
    nextrefsyms 0
 indirectsymoff 2254072
  nindirectsyms 788
      extreloff 0
        nextrel 0
      locreloff 0
        nlocrel 0
Load command 7
     cmd LC_UUID
 cmdsize 24
    uuid 0CD073D4-4257-345E-8157-2ECBC871E244
Load command 8
      cmd LC_VERSION_MIN_MACOSX
  cmdsize 16
  version 10.12
      sdk 10.12
Load command 9
      cmd LC_SOURCE_VERSION
  cmdsize 16
  version 0.0
Load command 10
          cmd LC_LOAD_DYLIB
      cmdsize 56
         name /usr/lib/libSystem.B.dylib (offset 24)
   time stamp 2 Thu Jan  1 01:00:02 1970
      current version 1238.60.2
compatibility version 1.0.0
Load command 11
      cmd LC_FUNCTION_STARTS
  cmdsize 16
  dataoff 2101576
 datasize 4352
Load command 12
      cmd LC_DATA_IN_CODE
  cmdsize 16
  dataoff 2105928
 datasize 0

[aarzilli]

As you can see, no debug sections. Now that I think about it I knew this because I noticed when I changed the linker to make it able to emit debug symbols for plugins.

[juhwany]

@aarzilli

First of all, I don't know how debugger works well. But It seems that it's impossible to debug golang plugin at this moment. Because as you said, the built plugin doesn't contain debugging symbols at all.
Is this golang issue? Should I wait for new version of golang?

[akamensky]

This is a major obstacle for me as I am working on same code base that uses plugins and I often switch between Linux and Mac while working on this.

Meantime I believe lack of debugging symbols in plugins on Darwin is still an issue in Go https://github.com/golang/go/issues/27502

[erikdw]

@aarzilli : Thanks for your diligence in pushing this support forward. It's nice that https://github.com/golang/go/issues/27502 has now been fixed, but it seems like the ability to debug go plugins on Mac will still be blocked by https://github.com/golang/go/issues/25841, according to:

• your comment above, and
• your comment in https://github.com/golang/go/issues/27502.

Given this, perhaps you can revisit your comment in https://github.com/golang/go/issues/25841, where you responded that it was not important? Seems that @heschik was willing to dig into that issue before.

[erikdw]

I discovered that the fix for https://github.com/golang/go/issues/27502 is present in go1.14beta1, so I installed that version:

go get golang.org/dl/go1.14beta1
go1.14beta1 download

Then I rebuilt the binaries from the example project:

go1.14beta1 build -gcflags 'all=-N -l' -buildmode=plugin -o eng/eng.so eng/greeter.go
go1.14beta1 build -gcflags 'all=-N -l' -o greeter greeter.go

I then used otool -l to validate that the eng.so binary had the DWARF debug sections.

✗ otool -l eng/eng.so | grep debug
  sectname __zdebug_line
  sectname __zdebug_pubname
  sectname __zdebug_loc
  sectname __zdebug_ranges
  sectname __zdebug_pubtype
  sectname __zdebug_aranges
  sectname __zdebug_info
  sectname __zdebug_frame
  sectname __zdebug_abbrev
  sectname __zdebug_str

So I got my hopes up, but when I tried debugging the greeter binary I hit the same problem.

✗ dlv --check-go-version=false exec ./greeter -- english
Type 'help' for list of commands.
(dlv) b greeter.go:58
Breakpoint 1 set at 0x40f33ac for main.main() ./greeter.go:58
(dlv) c
> main.main() ./greeter.go:58 (hits goroutine(1):1 total:1) (PC: 0x40f33ac)
    53:         fmt.Println("unexpected type from module symbol")
    54:         os.Exit(1)
    55:     }
    56: 
    57:     // 4. use the module
=>  58:     greeter.Greet()
    59: 
    60: }
(dlv) b eng/greeter.go:8
Command failed: Location "eng/greeter.go:8" not found
(dlv) s
Stopped at: 0x7ce92d0
=>no source available
(dlv) s
Hello Universe
Process has exited with status 0
(dlv) 

[aarzilli]

I think it would behoove you to comment yourself on that thread, your first hand account of why you need it would be more persuasive than my second hand perception that some people might need it.

[erikdw]

@aarzilli : I've put in a comment on that issue. But frankly I'm too ignorant about this topic to understand what the issues are here, and how exactly golang/go#25841 is blocking this issue.

e.g., I don't understand how you obtained the output that you cite, they look like they came from running readelf, but the ticket is about Mac. So I'm guessing maybe you targeted GOOS=linux in the build. And if so then I'm not seeing how that relates to this issue.

Furthermore, I seem to be seeing at least 2 different problems in my example above, but I'm not sure if they are symptoms of one underlying cause.

cannot set the breakpoint

On Mac, where it fails in this scenario, it looks like this:

(dlv) b eng/greeter.go:8
Command failed: Location "eng/greeter.go:8" not found

On linux where it works it looks like this:

(dlv) b eng/greeter.go:8
Breakpoint 2 set at 0x7f7d8c5c838f for plugin/unnamed-380bc3019cae9f99f0f646d134e7c21e1405b74f.greeting.Greet() ./eng/greeter.go:8

cannot step into the plugin code

On Mac. where it fails in this scenario, it looks like this:

(dlv) c
> main.main() ./greeter.go:58 (hits goroutine(1):1 total:1) (PC: 0x40f33ac)
    53:         fmt.Println("unexpected type from module symbol")
    54:         os.Exit(1)
    55:     }
    56: 
    57:     // 4. use the module
=>  58:     greeter.Greet()
    59: 
    60: }
(dlv) s
Stopped at: 0xcde92d0
=>no source available

On linux where it works it looks like this:

(dlv) c
> main.main() ./greeter.go:58 (hits goroutine(1):1 total:1) (PC: 0x533921)
    53:         fmt.Println("unexpected type from module symbol")
    54:         os.Exit(1)
    55:     }
    56: 
    57:     // 4. use the module
=>  58:     greeter.Greet()
    59: 
    60: }
(dlv) s
> plugin/unnamed-380bc3019cae9f99f0f646d134e7c21e1405b74f.(*greeting).Greet() <autogenerated>:1 (PC: 0x7f972c3cf421)
(dlv) s
> plugin/unnamed-380bc3019cae9f99f0f646d134e7c21e1405b74f.greeting.Greet() ./eng/greeter.go:7 (PC: 0x7f972c3cf381)
     2: 
     3: import "fmt"
     4: 
     5: type greeting string
     6: 
=>   7: func (g greeting) Greet() {
     8:     fmt.Println("Hello Universe")
     9: }
    10: 
    11: // exported
    12: var Greeter greeting
(dlv) s
> plugin/unnamed-380bc3019cae9f99f0f646d134e7c21e1405b74f.greeting.Greet() ./eng/greeter.go:8 (PC: 0x7f972c3cf38f)
     3: import "fmt"
     4: 
     5: type greeting string
     6: 
     7: func (g greeting) Greet() {
=>   8:     fmt.Println("Hello Universe")
     9: }
    10: 
    11: // exported
    12: var Greeter greeting

[aarzilli]

e.g., I don't understand how you obtained the output that you cite, they look like they came from running readelf

they are from objdump --dwarf, the GNU version, not the apple version.

[erikdw]

Thanks for the response. Hmm, for some reason gobjdump (GNU objdump) doesn't show any details when run against a Mac-targeted binary. Maybe I'm missing some necessary flags for the gobjdump binary?

✗ go build -gcflags='-N -l' -buildmode pie -o greeter greeter.go
✗ /usr/local/Cellar/binutils/2.33.1/bin/gobjdump --dwarf greeter

greeter:     file format mach-o-x86-64

Whereas when I crosscompile the binary for being run on linux (again from Mac), gobjdump --dwarf shows all kinds of fun dwarf details.

✗ GOOS=linux GOARCH=amd64 go build -gcflags '-N -l' -o greeter greeter.go
✗ /usr/local/Cellar/binutils/2.33.1/bin/gobjdump --dwarf greeter | head     

greeter:     file format elf64-x86-64

Contents of the .zdebug_abbrev section:

  Number TAG (0x0)
   1      DW_TAG_compile_unit    [has children]
    DW_AT_name         DW_FORM_string
    DW_AT_language     DW_FORM_data1
    DW_AT_stmt_list    DW_FORM_sec_offset
...

NOTE: I wasn't able to run this linux-targeted build in -buildmode pie, per the errors below:

✗ GOOS=linux GOARCH=amd64 go build -gcflags '-N -l' -buildmode pie -o greeter greeter.go
# command-line-arguments
loadinternal: cannot find runtime/cgo
/usr/local/Cellar/go/1.13.5/libexec/pkg/tool/darwin_amd64/link: running clang failed: exit status 1
ld: unknown option: -z
clang: error: linker command failed with exit code 1 (use -v to see invocation)

[aarzilli]

It's probably because of dwarf compression, you probably have to disable it with -ldflags='-compressdwarf=false'

[erikdw]

@aarzilli : woot! That was it. Thanks for teaching me something.

✗ go build -gcflags='-N -l' -ldflags='-compressdwarf=false' -buildmode pie -o greeter greeter.go 
✗ /usr/local/Cellar/binutils/2.33.1/bin/gobjdump --dwarf greeter                      

greeter:     file format mach-o-x86-64

Raw dump of debug contents of section .debug_line:

  Offset:                      0x0
  Length:                      1288
  DWARF Version:               2
...