Delve: go1.8devel and delve seem to not work well when stepping (problem with line number table?)

[dr2chase]

This is a heads-up; I work on the Go compiler, I spotted this problem, the cause was not immediately evident to me but I am digging. Someone more skillful with Dwarf might spot the cause more quickly; there are differences, not sure which ones are important and which are just artifacts. This may be a bug in the Go compiler instead of in Delve, but people might look for this problem here, so I am filing this bug.

1. What version of Delve are you using (dlv version)?
   Delve Debugger
   Version: 0.11.0-alpha
   Build: 46803551b888c017e22bef94d6bf223a9eb704a3

2. What version of Go are you using? (go version)?
   For the debugged code go version devel +aa8c8e7 Thu Nov 3 13:32:05 2016 +0000 darwin/amd64
   I see this bug either when compiling Delve itself with the version above, or with 1.7.3.

3. What operating system and processor architecture are you using?
   OSX 10.11.6, amd64

4. What did you do?
   Compiled a simple program with both go-1.7.3 and go-devel/recent and attempted to debug it.

package main

//go:noinline
func asdfasdf() {
    for i := 0; i < 5; i++ {
        for i := 0; i < 5; i++ {
            //...
        }
        //...
    }
}

func main() {
    asdfasdf()
}

and

dlv debug
b main.main
c
s
s
s
s
s

1. What did you expect to see?
   Orderly single stepping (seen for 1.7.3 compilation, not for 1.8 compilation)

2. What did you see instead?
   On 1.8, "steps" that leaped forwards through the program, finally ending in a (debugged program) panic of some sort:

(dlv) s
> [unrecovered-panic] runtime.startpanic() /Users/drchase/GoogleDrive/work/go/src/runtime/panic.go:568 (hits goroutine(0):1 total:1) (PC: 0x1022150)
SIGILL: illegal instruction
PC=0x10237f4 m=0 sigcode=1

goroutine 1 [running]:
runtime.main()
    /Users/drchase/GoogleDrive/work/go/src/runtime/proc.go:198 +0x214 fp=0xc420035fe0 sp=0xc420035f88
runtime.goexit()
    /Users/drchase/GoogleDrive/work/go/src/runtime/asm_amd64.s:2184 +0x1 fp=0xc420035fe8 sp=0xc420035fe0

rax    0x0
rbx    0x1092d70
rcx    0xc4200001a0
rdx    0x1066080
rdi    0x0
rsi    0xc42001a001
rbp    0xc420035fd0
rsp    0xc420035f88
r8     0xc41ffff2fc
r9     0x0
r10    0xc42001a060
r11    0xffffffff
r12    0x0
r13    0xffffffee
r14    0x60
r15    0x55
rip    0x10237f4
rflags 0x10246
cs     0x2b
fs     0x0
gs     0x0

One difference is that "main" is now the last package linked into the binary instead of the first.
Another is in the Line Number Statements.
1.7:

  [0x000011b7]  Extended opcode 2: set Address to 0x2000
  [0x000011c2]  Advance PC by 64 to 0x2040
  [0x000011c5]  Advance Line by 2 to 3
  [0x000011c7]  Copy
  [0x000011c8]  Special opcode 58: advance Address by 14 to 0x204e and Line by 1 to 4
  [0x000011c9]  Special opcode 98: advance Address by 24 to 0x2066 and Line by 1 to 5
  [0x000011ca]  Special opcode 168: advance Address by 42 to 0x2090 and Line by -1 to 4
  [0x000011cb]  Advance PC by 22 to 0x20a6
  [0x000011cd]  Advance Line by 6 to 10
  [0x000011cf]  Copy
  [0x000011d0]  Special opcode 42: advance Address by 10 to 0x20b0 and Line by 1 to 11
  [0x000011d1]  Set File Name to entry 2 in the File Name Table
...
  [0x000103d5]  Copy
  [0x000103d6]  Special opcode 22: advance Address by 5 to 0x4cf45 and Line by 1 to 139
  [0x000103d7]  Special opcode 18: advance Address by 4 to 0x4cf49 and Line by 1 to 140
  [0x000103d8]  Special opcode 6: advance Address by 1 to 0x4cf4a and Line by 1 to 141
  [0x000103d9]  Special opcode 10: advance Address by 2 to 0x4cf4c and Line by 1 to 142
  [0x000103da]  Advance PC by 4 to 0x4cf50
  [0x000103dc]  Advance Line by 4 to 146
  [0x000103de]  Copy
  [0x000103df]  Special opcode 22: advance Address by 5 to 0x4cf55 and Line by 1 to 147
  [0x000103e0]  Special opcode 18: advance Address by 4 to 0x4cf59 and Line by 1 to 148
  [0x000103e1]  Special opcode 6: advance Address by 1 to 0x4cf5a and Line by 1 to 149
  [0x000103e2]  Special opcode 10: advance Address by 2 to 0x4cf5c and Line by 1 to 150
  [0x000103e3]  Extended opcode 1: End of Sequence

vs 1.8:

Line Number Statements:
  [0x00000fe3]  Extended opcode 2: set Address to 0x1001000
  [0x00000fee]  Advance PC by 41 to 0x1001029
  [0x00000ff0]  Advance Line by 42 to 43
  [0x00000ff2]  Special opcode 239: advance Address by 23 to 0x1001040 and Line by 5 to 48
  [0x00000ff3]  Special opcode 166: advance Address by 16 to 0x1001050 and Line by 2 to 50
  [0x00000ff4]  Advance PC by constant 24 to 0x1001068
  [0x00000ff5]  Special opcode 55: advance Address by 5 to 0x100106d and Line by 1 to 51
  [0x00000ff6]  Advance PC by 29 to 0x100108a
  [0x00000ff8]  Special opcode 243: advance Address by 24 to 0x10010a2 and Line by -1 to 50
  [0x00000ff9]  Special opcode 147: advance Address by 14 to 0x10010b0 and Line by 3 to 53
  [0x00000ffa]  Advance PC by constant 24 to 0x10010c8
...
  [0x00009d88]  Advance PC by 104 to 0x104aa38
  [0x00009d8a]  Special opcode 244: advance Address by 24 to 0x104aa50 and Line by 0 to 1
  [0x00009d8b]  Set File Name to entry 66 in the File Name Table
  [0x00009d8d]  Advance PC by 201 to 0x104ab19
  [0x00009d90]  Special opcode 236: advance Address by 23 to 0x104ab30 and Line by 2 to 3
  [0x00009d91]  Special opcode 145: advance Address by 14 to 0x104ab3e and Line by 1 to 4
  [0x00009d92]  Special opcode 245: advance Address by 24 to 0x104ab56 and Line by 1 to 5
  [0x00009d93]  Advance PC by constant 24 to 0x104ab6e
  [0x00009d94]  Special opcode 183: advance Address by 18 to 0x104ab80 and Line by -1 to 4
  [0x00009d95]  Advance Line by 1 to 5
  [0x00009d97]  Special opcode 229: advance Address by 22 to 0x104ab96 and Line by 5 to 10
  [0x00009d98]  Special opcode 105: advance Address by 10 to 0x104aba0 and Line by 1 to 11
  [0x00009d99]  Extended opcode 1: End of Sequence

The -N -l generated code and the compiler's belief about code-to-line-number correspondence for the simple example are both the same.

[aarzilli]

One thing I know doesn't work with go-devel is prologue detection, probably due to new ssa optimizations. The SIGILL thing sounds like we are having problems with PC to line number info.
I was going to write a compatibility patch for 1.8 this week but it took a backseat to macOS 10.12.1...

For the line numbers stuff I wonder if we are using the right endianness, we've had that problem before and DWARF isn't exaclty clear about that.

[dr2chase]

The PC-line sync is off, I think I might be closer to figuring this out in a few minutes, pretty much of a matter of logging and watching where they go out of sync (it goes wrong before the second file).

[dr2chase]

Something's wrong in constaddpc -- delve consistently adds one more than gobjdump thinks it should.

[ignatov]

Any news?

[aarzilli]

667 fixes this and other problems with go1.8, it will be merge soon.

[ignatov]

That's cool, do you have any estimates?

[vsivsi]

@aarzilli Any timetable for releasing these fixes? Would be great if the current brew install of delve (0.12.1) worked reliably with the current brew install of golang (1.8).

I'm new to all of this, and I just wasted the better part of an afternoon trying to understand why something as basic as single stepping through a trivial program causes an unhandled panic using the current latest released versions of everything...

I can confirm that installing using brew install --HEAD go-delve/delve/delve works as expected.

[aarzilli]

@derekparker manages the releases.

[ignatov]

@vsivsi Feel free to use signed Delve binaries backed by JetBrains – https://bintray.com/jetbrains/golang/delve/view

[vsivsi]

Okay thanks for the information. I'm going to open a separate issue for this because as of right now, following the MacOS install instructions leads to a completely broken experience.

At least having an easy to find open issue stating this will save the next person in my situation from wasting an hour or two trying to figure out what is wrong.

[justinclift]

@ignatov Ahhh, so that's why the debugger in recent point builds of Gogland (which will become EAP 7) works ok, unlike EAP 6 and earlier. :smile:

[ignatov]

@justinclift You're right! Feel free to use our builds.