Crystal: DWARF sections cannot be parsed properly

[lugia-kun]

Summary of the problem

Following code produces OverflowError under my developing Linux distribution.

begin
  pp caller
rescue e : Exception
  p e.class
end

Note about the test code

(Later I noticed that I should print the class and message of Exception, without calling #class, to be more informative, and it is safe. However, doing it makes built binary different and It does not reproduce the problem. Therefore, I'm going with this test code at first)

To summarize below details, in my developing Linux distribution:

• Crystal cannot parse DWARF sections of binaries built by our BFD ld.
  
  
  
  • Exposed problem is OverflowError, but it is not all.
  
  
  • In other major Linux distributions, this problem seem not to be raised.
  
  
• GDB and objdump can parse DWARF data without error.
• Linking binary with GOLD linker also seems to be OK.

I'll try to find affected patches applied to binutils later.

Test of original test code

[momonga@elma ~/crystal x86_64]$ cat test.cr
begin
  pp caller
rescue e : Exception
  p e.class
end
[momonga@elma ~/crystal x86_64]$ bin/crystal build test.cr
Using compiled compiler at `.build/crystal'
/usr/bin/ld: skipping incompatible /usr/lib/libm.so when searching for -lm
/usr/bin/ld: skipping incompatible /usr/lib/libpthread.so when searching for -lpthread
/usr/bin/ld: skipping incompatible /usr/lib/librt.so when searching for -lrt
/usr/bin/ld: skipping incompatible /usr/lib/libdl.so when searching for -ldl
/usr/bin/ld: skipping incompatible /usr/lib/libc.so when searching for -lc
[momonga@elma ~/crystal x86_64]$ ./test
OverflowError

Getting backtrace using GDB

[momonga@elma ~/crystal x86_64]$ gdb ./test
GNU gdb (GDB) 8.1
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-momonga-linux".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./test...done.
(gdb) b __crystal_raise
Breakpoint 1 at 0x441840: file /opt/momonga/crystal/src/raise.cr, line 191.
(gdb) r
Starting program: /opt/momonga/crystal/test
warning: Error querying the rpm file `/lib64/ld-linux-x86-64.so.2': glibc-2.30.9000-2m.mo8.nosrc.rpm-debuginfo.x86_64
Missing separate debuginfo for /lib64/ld-linux-x86-64.so.2
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/23/66b2fce8e5b6b8b7d1f6316458094144e7dd85.debug
warning: Error querying the rpm file `/lib64/libm.so.6': glibc-2.30.9000-2m.mo8.nosrc.rpm-debuginfo.x86_64
Missing separate debuginfo for /lib64/libm.so.6
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/97/93abd3f4464ecda09afd118b77b13222e7a3c5.debug
warning: Error querying the rpm file `/lib64/libpthread.so.0': glibc-2.30.9000-2m.mo8.nosrc.rpm-debuginfo.x86_64
Missing separate debuginfo for /lib64/libpthread.so.0
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/f5/c2f51492747eeca1aaaf05d6567460874832be.debug
warning: File "/usr/lib64/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load:/usr/bin/mono-gdb.py".
To enable execution of this file add
        add-auto-load-safe-path /usr/lib64/libthread_db-1.0.so
line to your configuration file "/opt/momonga/.gdbinit".
To completely disable this security protection add
        set auto-load safe-path /
line to your configuration file "/opt/momonga/.gdbinit".
For more information about this security protection see the
"Auto-loading safe path" section in the GDB manual.  E.g., run from the shell:
        info "(gdb)Auto-loading safe path"
warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
warning: Error querying the rpm file `/lib64/librt.so.1': glibc-2.30.9000-2m.mo8.nosrc.rpm-debuginfo.x86_64
Missing separate debuginfo for /lib64/librt.so.1
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/05/a49312e6a9447f7a2b60526bf47433fa0af05d.debug
warning: Error querying the rpm file `/lib64/libdl.so.2': glibc-2.30.9000-2m.mo8.nosrc.rpm-debuginfo.x86_64
Missing separate debuginfo for /lib64/libdl.so.2
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/9c/aa310d95423523528971266405144e6a0791ae.debug
warning: Error querying the rpm file `/lib64/libc.so.6': glibc-2.30.9000-2m.mo8.nosrc.rpm-debuginfo.x86_64
Missing separate debuginfo for /lib64/libc.so.6
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/6d/eb7ca426e5a06263e979e24f675c24ca1abd3f.debug
[New LWP 27236]
[New LWP 27237]
[New LWP 27238]

Thread 1 "test" hit Breakpoint 1, __crystal_raise ()
    at /opt/momonga/crystal/src/raise.cr:191
191     {% unless flag?(:win32) %}
(gdb) bt
#0  __crystal_raise () at /opt/momonga/crystal/src/raise.cr:191
#1  0x000000000044efee in raise () at /opt/momonga/crystal/src/raise.cr:191
#2  0x00000000004418ce in __crystal_raise_overflow ()
    at /opt/momonga/crystal/src/raise.cr:239
#3  0x000000000048d5aa in read_opcodes ()
    at /opt/momonga/crystal/src/debug/dwarf/line_numbers.cr:240
#4  0x000000000048cd76 in decode_sequences ()
    at /opt/momonga/crystal/src/debug/dwarf/line_numbers.cr:229
#5  0x000000000048c34c in initialize ()
    at /opt/momonga/crystal/src/debug/dwarf/line_numbers.cr:176
#6  0x000000000048c2a4 in new ()
    at /opt/momonga/crystal/src/debug/dwarf/line_numbers.cr:173
#7  0x00000000004566e8 in read_dwarf_sections ()
    at /opt/momonga/crystal/src/callstack.cr:218
#8  0x0000000000453e1d in decode_line_number ()
    at /opt/momonga/crystal/src/callstack.cr:218
#9  0x00000000004535a8 in decode_backtrace ()
    at /opt/momonga/crystal/src/callstack.cr:163
#10 0x0000000000453462 in printable_backtrace ()
    at /opt/momonga/crystal/src/callstack.cr:52
#11 0x000000000044c5dd in caller () at /opt/momonga/crystal/src/callstack.cr:23
#12 0x0000000000440c26 in __crystal_main ()
    at /opt/momonga/crystal/src/time/location/loader.cr:2
#13 0x00000000004bd879 in main_user_code ()
    at /opt/momonga/crystal/src/crystal/main.cr:97
#14 0x00000000004bd7d9 in main ()
    at /opt/momonga/crystal/src/crystal/main.cr:86
#15 0x000000000044c2a6 in main ()
    at /opt/momonga/crystal/src/crystal/main.cr:106
(gdb) quit
A debugging session is active.

        Inferior 1 [process 27232] will be killed.

Quit anyway? (y or n) y
[momonga@elma ~/crystal x86_64]$

Dumping DWARF data using objdump -g seems to be completed without error.
Attached the output here (about 6MB).

Linking with GOLD linker

Linking with GOLD linker works fine.

[momonga@elma ~/crystal x86_64]$ rm -rfv ~/.cache/crystal
...
removed directory '/opt/momonga/.cache/crystal'
[momonga@elma ~/crystal x86_64]$ CC="cc -fuse-ld=gold" bin/crystal build test.cr
Using compiled compiler at `.build/crystal'
/usr/bin/ld.gold: warning: skipping incompatible /usr/lib/libm.so while searching for m
/usr/bin/ld.gold: warning: skipping incompatible /usr/lib/libpthread.so while searching for pthread
/usr/bin/ld.gold: warning: skipping incompatible /usr/lib/librt.so while searching for rt
/usr/bin/ld.gold: warning: skipping incompatible /usr/lib/libdl.so while searching for dl
/usr/bin/ld.gold: warning: skipping incompatible /usr/lib/libc.so while searching for c
[momonga@elma ~/crystal x86_64]$ ./test
["test.cr:2:6 in '__crystal_main'",
 "src/crystal/main.cr:97:5 in 'main_user_code'",
 "src/crystal/main.cr:86:7 in 'main'",
 "src/crystal/main.cr:106:3 in 'main'",
 "__libc_start_main",
 "_start",
 "???"]
[momonga@elma ~/crystal x86_64]$

Using unpatched (vanilla) GNU binutils

I have built "vanilla" GNU binutils 2.33.1, and it works fine.

Result

[momonga@elma ~/crystal x86_64]$ $HOME/bintuils/2.33.1/bin/ld -v
GNU ld (GNU Binutils) 2.33.1
[momonga@elma ~/crystal x86_64]$ ld -v
GNU ld version 2.33.1-1m.mo8
[momonga@elma ~/crystal x86_64]$ env PATH=$HOME/bintuils/2.33.1/bin:$PATH bin/crystal build test.cr
Using compiled compiler at `.build/crystal'
/opt/momonga/bintuils/2.33.1/bin/ld: skipping incompatible /usr/lib/libm.so when searching for -lm
/opt/momonga/bintuils/2.33.1/bin/ld: skipping incompatible /usr/lib/libpthread.so when searching for -lpthread
/opt/momonga/bintuils/2.33.1/bin/ld: skipping incompatible /usr/lib/librt.so when searching for -lrt
/opt/momonga/bintuils/2.33.1/bin/ld: skipping incompatible /usr/lib/libdl.so when searching for -ldl
/opt/momonga/bintuils/2.33.1/bin/ld: skipping incompatible /usr/lib/libc.so when searching for -lc
[momonga@elma ~/crystal x86_64]$ ./test
["test.cr:2:6 in '__crystal_main'",
 "src/crystal/main.cr:97:5 in 'main_user_code'",
 "src/crystal/main.cr:86:7 in 'main'",
 "src/crystal/main.cr:106:3 in 'main'",
 "__libc_start_main",
 "_start",
 "???"]

"Monkey patching" test 1

@asterite provided me a snippet that fixes above error here (Thanks!).

The error line is:
https://github.com/crystal-lang/crystal/blob/10d92bbac60dbe39d89b8a32b9000392da07c821/src/debug/dwarf/line_numbers.cr#L240

After adding it to the test code, it still raises OverflowError but in another location.

I have confirmed that there is a case that sequence.opcode_base == 0. But I'm wondering whether these binaries are really in valid format.

module Debug
  module DWARF
    struct LineNumbers
      private def read_opcodes(sequence)
        return unless sequence.opcode_base > 0

        1.upto(sequence.opcode_base - 1) do
          sequence.standard_opcode_lengths << @io.read_byte.not_nil!
        end
      end
    end
  end
end

begin
  pp caller
rescue e : Exception
  p e.class
end

[momonga@elma ~/crystal x86_64]$ rm -rfv ~/.cache/crystal
...
removed directory '/opt/momonga/.cache/crystal'
[momonga@elma ~/crystal x86_64]$ bin/crystal build test.cr
Using compiled compiler at `.build/crystal'
/usr/bin/ld: skipping incompatible /usr/lib/libm.so when searching for -lm
/usr/bin/ld: skipping incompatible /usr/lib/libpthread.so when searching for -lpthread
/usr/bin/ld: skipping incompatible /usr/lib/librt.so when searching for -lrt
/usr/bin/ld: skipping incompatible /usr/lib/libdl.so when searching for -ldl
/usr/bin/ld: skipping incompatible /usr/lib/libc.so when searching for -lc
[momonga@elma ~/crystal x86_64]$ ./test
OverflowError



md5-d3395c33509c11540a72b942f8538cb0



[momonga@elma ~/crystal x86_64]$ gdb -q ./test
Reading symbols from ./test...done.
(gdb) b __crystal_raise
Breakpoint 1 at 0x441840: file /opt/momonga/crystal/src/raise.cr, line 191.
(gdb) r
Starting program: /opt/momonga/crystal/test
(omitted)
[New LWP 27492]
[New LWP 27493]
[New LWP 27494]

Thread 1 "test" hit Breakpoint 1, __crystal_raise ()
    at /opt/momonga/crystal/src/raise.cr:191
191     {% unless flag?(:win32) %}
(gdb) bt
#0  __crystal_raise () at /opt/momonga/crystal/src/raise.cr:191
#1  0x000000000044efee in raise () at /opt/momonga/crystal/src/raise.cr:191
#2  0x00000000004418ce in __crystal_raise_overflow ()
    at /opt/momonga/crystal/src/raise.cr:239
#3  0x000000000048de03 in read_filename_table ()
    at /opt/momonga/crystal/src/debug/dwarf/line_numbers.cr:260
#4  0x000000000048d091 in decode_sequences ()
    at /opt/momonga/crystal/src/debug/dwarf/line_numbers.cr:231
#5  0x000000000048c34c in initialize ()
    at /opt/momonga/crystal/src/debug/dwarf/line_numbers.cr:176
#6  0x000000000048c2a4 in new ()
    at /opt/momonga/crystal/src/debug/dwarf/line_numbers.cr:173
#7  0x00000000004566e8 in read_dwarf_sections ()
    at /opt/momonga/crystal/src/callstack.cr:218
#8  0x0000000000453e1d in decode_line_number ()
    at /opt/momonga/crystal/src/callstack.cr:218
#9  0x00000000004535a8 in decode_backtrace ()
    at /opt/momonga/crystal/src/callstack.cr:163
#10 0x0000000000453462 in printable_backtrace ()
    at /opt/momonga/crystal/src/callstack.cr:52
#11 0x000000000044c5dd in caller () at /opt/momonga/crystal/src/callstack.cr:23
#12 0x0000000000440c26 in __crystal_main () at /opt/momonga/crystal/test.cr:16
#13 0x00000000004bd8a9 in main_user_code ()
    at /opt/momonga/crystal/src/crystal/main.cr:97
#14 0x00000000004bd809 in main ()
    at /opt/momonga/crystal/src/crystal/main.cr:86
#15 0x000000000044c2a6 in main ()
    at /opt/momonga/crystal/src/crystal/main.cr:106
(gdb) quit
A debugging session is active.

        Inferior 1 [process 27488] will be killed.

Quit anyway? (y or n) y
[momonga@elma ~/crystal x86_64]$



md5-8daeeb9ecf9625b956e57c2635fc4612

[momonga@elma ~/crystal x86_64]$ rm -rf ~/.cache/crystal/
[momonga@elma ~/crystal x86_64]$ bin/crystal build test.cr
Using compiled compiler at `.build/crystal'
/usr/bin/ld: skipping incompatible /usr/lib/libm.so when searching for -lm
/usr/bin/ld: skipping incompatible /usr/lib/libpthread.so when searching for -lp
thread
/usr/bin/ld: skipping incompatible /usr/lib/librt.so when searching for -lrt
/usr/bin/ld: skipping incompatible /usr/lib/libdl.so when searching for -ldl
/usr/bin/ld: skipping incompatible /usr/lib/libc.so when searching for -lc
[momonga@elma ~/crystal x86_64]$ ./test

[momonga@elma ~/crystal x86_64]$ gdb -q ./test
Reading symbols from ./test...done.
(gdb) b __crystal_raise
Breakpoint 1 at 0x441800: file /opt/momonga/crystal/src/raise.cr, line 191.
(gdb) r
Starting program: /opt/momonga/crystal/test
[New LWP 28472]
[New LWP 28473]
[New LWP 28474]

Thread 1 "test" hit Breakpoint 1, __crystal_raise ()
at /opt/momonga/crystal/src/raise.cr:191
191 {% unless flag?(:win32) %}
(gdb) bt

0 __crystal_raise () at /opt/momonga/crystal/src/raise.cr:191

1 0x000000000044ef6e in raise () at /opt/momonga/crystal/src/raise.cr:191

2 0x000000000044188e in __crystal_raise_overflow ()

at /opt/momonga/crystal/src/raise.cr:239

3 0x000000000048d52a in read_opcodes ()

at /opt/momonga/crystal/src/debug/dwarf/line_numbers.cr:240

4 0x000000000048ccf6 in decode_sequences ()

at /opt/momonga/crystal/src/debug/dwarf/line_numbers.cr:229

5 0x000000000048c2cc in initialize ()

at /opt/momonga/crystal/src/debug/dwarf/line_numbers.cr:176

6 0x000000000048c224 in new ()

at /opt/momonga/crystal/src/debug/dwarf/line_numbers.cr:173

7 0x0000000000456668 in read_dwarf_sections ()

at /opt/momonga/crystal/src/callstack.cr:218

8 0x0000000000453d9d in decode_line_number ()

at /opt/momonga/crystal/src/callstack.cr:218

9 0x0000000000453528 in decode_backtrace ()

at /opt/momonga/crystal/src/callstack.cr:163

10 0x00000000004533e2 in printable_backtrace ()

at /opt/momonga/crystal/src/callstack.cr:52

11 0x000000000044c59d in caller () at /opt/momonga/crystal/src/callstack.cr:23

12 0x0000000000440c20 in __crystal_main ()

at /opt/momonga/crystal/src/time/location/loader.cr:2

13 0x00000000004bd866 in main_user_code ()

at /opt/momonga/crystal/src/crystal/main.cr:97

14 0x00000000004bd7c9 in main ()

at /opt/momonga/crystal/src/crystal/main.cr:86

15 0x000000000044c266 in main ()

at /opt/momonga/crystal/src/crystal/main.cr:106

(gdb) q
A debugging session is active.

    Inferior 1 [process 28468] will be killed.

Quit anyway? (y or n) y
[momonga@elma ~/crystal x86_64]$

</details>
<details><summary>Result (with the first monkey patch)</summary>

```crystal
module Debug
  module DWARF
    struct LineNumbers
      private def read_opcodes(sequence)
        return unless sequence.opcode_base > 0

        1.upto(sequence.opcode_base - 1) do
          sequence.standard_opcode_lengths << @io.read_byte.not_nil!
        end
      end
    end
  end
end

begin
  pp caller
rescue e : Exception
  p e
end

[momonga@elma ~/crystal x86_64]$ rm -rf ~/.cache/crystal/
[momonga@elma ~/crystal x86_64]$ bin/crystal build test.cr
Using compiled compiler at `.build/crystal'
/usr/bin/ld: skipping incompatible /usr/lib/libm.so when searching for -lm
/usr/bin/ld: skipping incompatible /usr/lib/libpthread.so when searching for -l$
thread
/usr/bin/ld: skipping incompatible /usr/lib/librt.so when searching for -lrt
/usr/bin/ld: skipping incompatible /usr/lib/libdl.so when searching for -ldl
/usr/bin/ld: skipping incompatible /usr/lib/libc.so when searching for -lc
[momonga@elma ~/crystal x86_64]$ ./test
#<IndexError:Index out of bounds>
[momonga@elma ~/crystal x86_64]$ gdb -q test
Reading symbols from test...done.
(gdb) b __crystal_raise
Breakpoint 1 at 0x441800: file /opt/momonga/crystal/src/raise.cr, line 191.
(gdb) r
Starting program: /opt/momonga/crystal/test
[New LWP 28589]
[New LWP 28590]
[New LWP 28591]

Thread 1 "test" hit Breakpoint 1, __crystal_raise ()
    at /opt/momonga/crystal/src/raise.cr:191
191     {% unless flag?(:win32) %}
(gdb) bt
#0  __crystal_raise () at /opt/momonga/crystal/src/raise.cr:191
#1  0x0000000000441e5e in raise () at /opt/momonga/crystal/src/raise.cr:191
#2  0x000000000046d1cd in [] () at /opt/momonga/crystal/src/indexable.cr:73
#3  0x000000000049054c in register_to_matrix ()
    at /opt/momonga/crystal/src/debug/dwarf/line_numbers.cr:363
#4  0x000000000048e920 in read_statement_program ()
    at /opt/momonga/crystal/src/debug/dwarf/line_numbers.cr:287
#5  0x000000000048d1f0 in decode_sequences ()
    at /opt/momonga/crystal/src/debug/dwarf/line_numbers.cr:234
#6  0x000000000048c2cc in initialize ()
    at /opt/momonga/crystal/src/debug/dwarf/line_numbers.cr:176
#7  0x000000000048c224 in new ()
    at /opt/momonga/crystal/src/debug/dwarf/line_numbers.cr:173
#8  0x0000000000456668 in read_dwarf_sections ()
    at /opt/momonga/crystal/src/callstack.cr:218
#9  0x0000000000453d9d in decode_line_number ()
    at /opt/momonga/crystal/src/callstack.cr:218
#10 0x0000000000453528 in decode_backtrace ()
    at /opt/momonga/crystal/src/callstack.cr:163
#11 0x00000000004533e2 in printable_backtrace ()
    at /opt/momonga/crystal/src/callstack.cr:52
#12 0x000000000044c59d in caller () at /opt/momonga/crystal/src/callstack.cr:23
#13 0x0000000000440c20 in __crystal_main () at /opt/momonga/crystal/test.cr:16
#14 0x00000000004bd896 in main_user_code ()
    at /opt/momonga/crystal/src/crystal/main.cr:97
#15 0x00000000004bd7f9 in main ()
    at /opt/momonga/crystal/src/crystal/main.cr:86
#16 0x000000000044c266 in main ()
    at /opt/momonga/crystal/src/crystal/main.cr:106
(gdb) q
A debugging session is active.

        Inferior 1 [process 28585] will be killed.

Quit anyway? (y or n) y
[momonga@elma ~/crystal x86_64]$



md5-1b2e6d952d38e08777be6594a4332186



[momonga@elma ~/crystal x86_64]$ bin/crystal version
Using compiled compiler at `.build/crystal'
Crystal 0.32.0-dev [10d92bbac] (2019-10-18)

LLVM: 8.0.1
Default target: x86_64-unknown-linux-gnu



md5-505fb252ea14513ef3e35ac518f2d213



[momonga@elma ~/crystal x86_64]$ gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/libexec/gcc/x86_64-momonga-linux/9.2.1/lto-wrapper
Target: x86_64-momonga-linux
Configured with: ../configure --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --with-pkgversion='Momonga Linux 9.2.1-0.6m.mo8' --program-suffix= --enable-shared --enable-threads=posix --enable-checking=release --with-system-zlib --enable-__cxa_atexit --enable-multilib --disable-libunwind-exceptions --enable-gnu-unique-object --enable-linker-build-id --with-linker-hash-style=gnu --enable-plugin --enable-initfini-array --host=x86_64-momonga-linux --build=x86_64-momonga-linux --target=x86_64-momonga-linux --with-default-libstdcxx-abi=gcc4-compatible --with-build-config=bootstrap-lto --with-isl --enable-libmpx --enable-gnu-indirect-function --with-tune=generic --without-system-libunwind --enable-languages=c,c++,objc,obj-c++,fortran,lto,go --with-ppl --enable-cloog-backend=isl --enable-gnu-indirect-function --with-tune=generic --with-arch_32=i686 --build=x86_64-momonga-linux
Thread model: posix
gcc version 9.2.1 20190928 (Momonga Linux 9.2.1-0.6m.mo8)



md5-505fb252ea14513ef3e35ac518f2d213



[momonga@elma ~/crystal x86_64]$ uname -a
Linux elma 5.2.11-1m.mo8.x86_64 #1 SMP Fri Aug 30 20:23:12 JST 2019 x86_64 x86_64 x86_64 GNU/Linux

• binutils 2.33.1 (source)
• gcc 9.2.1 (source)
• LLVM 8.0.1 (source)
• GDB 8.1 (source)

[lugia-kun]

Added result of using unpatched GNU binutils (and it worked fine).

[ysbaddaden]

Here is what the DWARF 4 specs has to say:

1. opcode_base (ubyte)
   The number assigned to the first special opcode. Opcode base is typically one greater than the highest-numbered standard opcode defined for the specified version of the line number information (12 in DWARF Version 3 and Version 4, 9 in Version 2). If opcode_base is less than the typical value, then standard opcode numbers greater than or equal to the opcode base are not used in the line number table of this unit (and the codes are treated as special opcodes). If opcode_base is greater than the typical value, then the numbers between that of the highest standard opcode and the first special opcode (not inclusive) are used for vendor specific extensions.

2. standard_opcode_lengths (array of ubyte)
   This array specifies the number of LEB128 operands for each of the standard opcodes. The first element of the array corresponds to the opcode whose value is 1, and the last element corresponds to the opcode whose value is opcode_base - 1. (...)

It seems the opcode_base is meant to be greater than zero, but this isn't prohibited, but in that case no all opcodes should be treated as special... which is fishy.

It probably doesn't hurt to be careful and skip parsing standard_opcode_lengths when opcode_base is zero, or check what readelf and/or objdump decodes —is it really zero? or is there a decoding bug in Debug::DWARF?

[lugia-kun]

I find that the .debug_line section (and other debug-related sections) seem to be compressed, after comparing the result printed the content of Debug::DWARF::LineNumbers::Sequence, the result decoded with readelf and the hex dump of section .debug_line.

Crystal seems not supported compressed sections, does it?

Details of what inspected about generated object by my dev Linux

Heading of hexdump of .debug_line section is following: (by readelf of elfutils)

Hex dump of section [35] '.debug_line', 45192 bytes at offset 0x146f42:
  0x00000000 01000000 00000000 db070200 00000000 ................
  0x00000010 01000000 00000000 789cec7d 079c54d5 ........x..}..T.
  0x00000020 f5ff7bf7 ee7dfb76 96656176 e9084b93 ..{..}.v.eav..K.
  0x00000030 3e606f80 25262683 1a139338 d8006117 >`o.%&&....8..a.
  0x00000040 50048459 da369d05 35d6017b d4017b5b P..Y.6..5..{..{[
  0x00000050 6c299ac5 164b5cc4 a8b12c96 18db62d7 l)...K\...,...b.
  0x00000060 e0a2c6a8 d1fff77b ee9bd9d9 020221f9 .......{......!.
  0x00000070 fdf2fffc fce0ce7d f7dd77eb b9a7dd73 .......}..w....s
  0x00000080 ceadece4 3839cec7 ca715cd7 fdaa7301 ....89...q\...s.
  0x00000090 7f5c074f f83766ee bcf89853 e79e3a77 .\.O.7f....S..:w
  0x000000a0 ce8ca963 a6cd5fb2 203e75b6 d351e698 ...c.._. >u..Q..
  0x000000b0 05f3a78d 995e7a52 f98c2dbe def277b3 .....^zR..-...w.
  0x000000c0 679d3479 da98c5fb ee3d79ef 3d47cf9e g.4y.....=y.=G..
  0x000000d0 35a77cf1 e81973ca c7ecc017 63162c59 5.|...s.....c.,Y
  0x000000e0 b0e5af66 6ca5c6ef 1cdcbcf9 73a7952e ...fl.......s...

Heading of the output of readelf decodes:

DWARF section [35] '.debug_line' at offset 0x146f42:

Table at offset 0:

 Length:                         3196
 DWARF version:                  4
 Prologue length:                750
 Address size:                   8
 Segment selector size:          0
 Min instruction length:         1
 Max operations per instruction: 1
 Initial value if 'is_stmt':     1
 Line base:                      -5
 Line range:                     14
 Opcode base:                    13

Opcodes:
  [ 1]  0 arguments
  [ 2]  1 argument
  [ 3]  1 argument
  [ 4]  1 argument
  [ 5]  1 argument
  [ 6]  0 arguments
  [ 7]  0 arguments
  [ 8]  0 arguments
  [ 9]  1 argument
  [10]  0 arguments
  [11]  0 arguments
  [12]  1 argument

(omit)

The output of Crystal DWARF parser reads:

[momonga@elma ~/crystal x86_64]$ ./test
Debug::DWARF::LineNumbers::Sequence(
 @default_is_stmt=false,
 @file_names=[{"", 0, 0, 0}],
 @header_length=131792896,
 @include_directories=[""],
 @line_base=0,
 @line_range=0,
 @maximum_operations_per_instruction=1,
 @minimum_instruction_length=2,
 @offset=0,
 @opcode_base=0,
 @standard_opcode_lengths=[0],
 @unit_length=1,
 @version=0)
Debug::DWARF::LineNumbers::Sequence(
 @default_is_stmt=false,
 @file_names=[{"", 0, 0, 0}],
 @header_length=2013265920,
 @include_directories=[""],
 @line_base=125,
 @line_range=7,
 @maximum_operations_per_instruction=1,
 @minimum_instruction_length=156,
 @offset=15,
 @opcode_base=156,
 @standard_opcode_lengths=[0],
 @unit_length=256,
 @version=0)

diff to output above

diff --git a/src/debug/dwarf/line_numbers.cr b/src/debug/dwarf/line_numbers.cr
index b4a11139b..0f285222f 100644
--- a/src/debug/dwarf/line_numbers.cr
+++ b/src/debug/dwarf/line_numbers.cr
@@ -225,13 +225,16 @@ module Debug
           sequence.line_base = @io.read_bytes(Int8)
           sequence.line_range = @io.read_byte.not_nil!
           sequence.opcode_base = @io.read_byte.not_nil!
+          pp sequence

-          read_opcodes(sequence)
-          read_directory_table(sequence)
-          read_filename_table(sequence)
+          if sequence.opcode_base > 0
+            read_opcodes(sequence)
+            read_directory_table(sequence)
+            read_filename_table(sequence)

-          if @io.tell - @offset < sequence.offset + sequence.total_length
-            read_statement_program(sequence)
+            if @io.tell - @offset < sequence.offset + sequence.total_length
+              read_statement_program(sequence)
+            end
           end
         end
       end

Inspecting binary generated under openSUSE, under my dev Linux

Heading of hexdump of .debug_line section:

Hex dump of section [35] '.debug_line', 112719 bytes at offset 0xeec1b:
  0x00000000 58000000 03003000 00000101 fb0e0d00 X.....0.........
  0x00000010 01010101 00000001 0000012e 2e2f7379 ............./sy
  0x00000020 73646570 732f7838 365f3634 00007374 sdeps/x86_64..st
  0x00000030 6172742e 53000100 00000009 02f0c443 art.S..........C
  0x00000040 00000000 00033e01 03102e42 233e4d24 ......>....B#>M$
  0x00000050 24757603 10746802 01000101 23000000 $uv..th.....#...
  0x00000060 03001d00 00000101 fb0e0d00 01010101 ................
  0x00000070 00000001 00000100 696e6974 2e630000 ........init.c..
  0x00000080 00000061 00000003 002f0000 000101fb ...a...../......
  0x00000090 0e0d0001 01010100 00000100 00012e2e ................
  0x000000a0 2f737973 64657073 2f783836 5f363400 /sysdeps/x86_64.
  0x000000b0 00637274 692e5300 01000000 00090200 .crti.S.........
  0x000000c0 c0430000 00000003 c100014c 753d2f02 .C.........Lu=/.
  0x000000d0 02000101 00090230 cb4b0000 00000003 .......0.K......
  0x000000e0 d3000102 04000101 bd0c0000 0400db02 ................
  0x000000f0 00000101 01fb0e0d 00010101 01000000 ................
  0x00000100 01000001 2f686f6d 652f6c75 67696100 ..../home/lugia.
  0x00000110 2f686f6d 652f6c75 6769612f 63727973 /home/lugia/crys

Heading of the output of readelf decodes:

DWARF section [35] '.debug_line' at offset 0xeec1b:

Table at offset 0:

 Length:                         88
 DWARF version:                  3
 Prologue length:                48
 Address size:                   8
 Segment selector size:          0
 Min instruction length:         1
 Max operations per instruction: 1
 Initial value if 'is_stmt':     1
 Line base:                      -5
 Line range:                     14
 Opcode base:                    13

Opcodes:
  [ 1]  0 arguments
  [ 2]  1 argument
  [ 3]  1 argument
  [ 4]  1 argument
  [ 5]  1 argument
  [ 6]  0 arguments
  [ 7]  0 arguments
  [ 8]  0 arguments
  [ 9]  1 argument
  [10]  0 arguments
  [11]  0 arguments
  [12]  1 argument

[ysbaddaden]

Crystal seems not supported compressed sections, does it?

First time I hear about compressed sections in ELF executables; it's probably not supported.

We parse garbage while readelf works, which means there is something broken in Debug::ELF or Debug::DWARF.

[lugia-kun]

I looked for information about compressing debug sections, and I found or confirmed following things:

• objcopy can control the compression of the debug sections.
• I confirmed that Crystal fails to parse it if the debug sections are compressed
• ld (of GNU binutils) compresses debug sections in default if --enable-compressed-debug-sections argument is passed with the value all or ld to the configure, while building GNU binutils.
  
  
  
  • While this argument is passed with value all, but I'm not sure why gold does not compress them. I may need to check more about the building processes.
  
  
• Otherwise, ld compresses debug sections if --compress-debug-sections is passed with value other then none.

My problem here is that the almost of all executables (including Crystal compiler itself) cannot parse DWARF sections, so it should be fixed by supporting compressed sections.

The specification and useful discussion about compressing debug sections are available on these documents:

• https://docs.oracle.com/cd/E53394_01/html/E54813/man-cds.html
• https://docs.oracle.com/cd/E53394_01/html/E54813/section_compression.html
• https://reviews.llvm.org/D31941
  
  
  
  • This is the code review when LLVM introduced the support of compressing debug sections to their linker, LLD.
  
  
  • ~The feature of compressed sections seems to be already included at that time.~
  
  

https://github.com/crystal-lang/crystal/blob/10d92bbac60dbe39d89b8a32b9000392da07c821/src/debug/elf.cr#L5
This document also has the description about compressing sections: http://www.sco.com/developers/gabi/latest/ch4.sheader.html (Around SHF_COMPRESSED flags)

According to these documents and source of Crystal, we may need to add uncompressing process to Debug::ELF#read_section?.
https://github.com/crystal-lang/crystal/blob/10d92bbac60dbe39d89b8a32b9000392da07c821/src/debug/elf.cr#L208-L217

They seem to have another specification that compressing debug sections, but it uses another section names, .zdebug*. So it won't prevent the application from running, and it will just return the result same as the one if .debug_* sections are not found.

There is a referenceable implementation of reading ELF and DWARF: elfufils, https://sourceware.org/elfutils/, which is mainly or originally developed by RedHat.

Example of testing about compressing debug sections

With debug sections are compressed:

[momonga@elma ~/crystal x86_64]$ objcopy --compress-debug-sections .build/crystal
[momonga@elma ~/crystal x86_64]$ bin/crystal play -p 80
Using compiled compiler at `.build/crystal'
Failed to raise an exception: END_OF_STACK
[0x7ecac6] *CallStack::print_backtrace:Int32 +118
[0x7b74f6] __crystal_raise +86
[0x7e3e09] *CallStack::read_dwarf_sections:(Array(Tuple(UInt64, UInt64, String)) | Nil) +697
[0x7e394d] *CallStack::decode_line_number<UInt64>:Tuple(String, Int32, Int32) +45
[0x7e30fe] *CallStack#decode_backtrace:Array(String) +302
[0x7e2fb2] *CallStack#printable_backtrace:Array(String) +50
[0x142617d] *Exception+ +77
[0x1425f98] *Exception+ +120
[0x8c7ac3] *AtExitHandlers::run<Int32>:Int32 +579
[0x12862eb] *Crystal::main<Int32, Pointer(Pointer(UInt8))>:Int32 +139
[0x7c1d76] main +6
[0x7f167c2f77cb] __libc_start_main +235
[0x7b5baa] _start +42
[0x0] ???

[momonga@elma ~/crystal x86_64]$ cat test.cr
begin
  pp caller
rescue e : Exception
  p e.class
end
[momonga@elma ~/crystal x86_64]$ bin/crystal build test.cr
Using compiled compiler at `.build/crystal'
/usr/bin/ld: skipping incompatible /usr/lib/libm.so when searching for -lm
/usr/bin/ld: skipping incompatible /usr/lib/libpthread.so when searching for -lpthread
/usr/bin/ld: skipping incompatible /usr/lib/librt.so when searching for -lrt
/usr/bin/ld: skipping incompatible /usr/lib/libdl.so when searching for -ldl
/usr/bin/ld: skipping incompatible /usr/lib/libc.so when searching for -lc
[momonga@elma ~/crystal x86_64]$ objcopy --compress-debug-sections test
[momonga@elma ~/crystal x86_64]$ ./test
OverflowError



md5-43d7ef88f4b7cca8b2242f2bf09e6dd3



[momonga@elma ~/crystal x86_64]$ objcopy --decompress-debug-sections .build/crystal
[momonga@elma ~/crystal x86_64]$ bin/crystal play -p 80
Using compiled compiler at `.build/crystal'
bind: Permission denied (Errno)
  from src/socket/tcp_server.cr:75:15 in 'initialize:reuse_port'
  from src/socket/tcp_server.cr:32:3 in 'new:reuse_port'
  from src/http/server.cr:207:5 in 'bind_tcp'
  from src/compiler/crystal/tools/playground/server.cr:531:17 in 'start'
  from src/compiler/crystal/command/playground.cr:37:5 in 'playground'
  from src/compiler/crystal/command.cr:74:7 in 'run'
  from src/compiler/crystal/command.cr:46:5 in 'run'
  from src/compiler/crystal/command.cr:45:3 in 'run'
  from src/compiler/crystal.cr:8:1 in '__crystal_main'
  from src/crystal/main.cr:97:5 in 'main_user_code'
  from src/crystal/main.cr:86:7 in 'main'
  from src/crystal/main.cr:106:3 in 'main'
  from __libc_start_main
  from _start
  from ???
Error: you've found a bug in the Crystal compiler. Please open an issue, including source code that will allow us to reproduce the bug: https://github.com/crystal-lang/crystal/issues



md5-505fb252ea14513ef3e35ac518f2d213



[momonga@elma ~/crystal x86_64]$ cat test.cr
begin
  pp caller
rescue e : Exception
  p e.class
end
[momonga@elma ~/crystal x86_64]$ bin/crystal build test.cr
Using compiled compiler at `.build/crystal'
/usr/bin/ld: skipping incompatible /usr/lib/libm.so when searching for -lm
/usr/bin/ld: skipping incompatible /usr/lib/libpthread.so when searching for -lpthread
/usr/bin/ld: skipping incompatible /usr/lib/librt.so when searching for -lrt
/usr/bin/ld: skipping incompatible /usr/lib/libdl.so when searching for -ldl
/usr/bin/ld: skipping incompatible /usr/lib/libc.so when searching for -lc
[momonga@elma ~/crystal x86_64]$ objcopy --decompress-debug-sections test
[momonga@elma ~/crystal x86_64]$ ./test
["test.cr:2:6 in '__crystal_main'",
 "src/crystal/main.cr:97:5 in 'main_user_code'",
 "src/crystal/main.cr:86:7 in 'main'",
 "src/crystal/main.cr:106:3 in 'main'",
 "__libc_start_main",
 "_start",
 "???"]



md5-7069f4ac86a2940ac38bb2674918d516



[momonga@elma ~ x86_64]$ objcopy --help
Usage: objcopy [option(s)] in-file [out-file]
 Copies a binary file, possibly transforming it in the process
 The options are:
  -I --input-target <bfdname>      Assume input file is in format <bfdname>
  -O --output-target <bfdname>     Create an output file in format <bfdname>
  -B --binary-architecture <arch>  Set output arch, when input is arch-less
...
     --compress-debug-sections[={none|zlib|zlib-gnu|zlib-gabi}]
                                   Compress DWARF debug sections using zlib
     --decompress-debug-sections   Decompress DWARF debug sections using zlib



md5-8badefeab7ff696f43273b2754eed2a7



[momonga@elma ~/binutils-2.33.1 x86_64]$ ./configure --help
`configure' configures this package to adapt to many kinds of systems.

Usage: ./configure [OPTION]... [VAR=VALUE]...

To assign environment variables (e.g., CC, CFLAGS...), specify them as
VAR=VALUE.  See below for descriptions of some of the useful variables.

Defaults for the options are specified in brackets.
...
Optional Features:
...
  --enable-compressed-debug-sections={all,gas,gold,ld,none}
                          Enable compressed debug sections for gas, gold or ld
                          by default



md5-b3a6eaf98168e983b059e67329d3131a



[momonga@elma ~ x86_64]$ ld --help
Usage: ld [options] file...
Options:
  -a KEYWORD                  Shared library control for HP/UX compatibility
  -A ARCH, --architecture ARCH
                              Set architecture
...
  --compress-debug-sections=[none|zlib|zlib-gnu|zlib-gabi]
                              Compress DWARF debug sections using zlib
                               Default: zlib-gabi

For GOLD linker, you may not see Default: clause.

[ysbaddaden]

@lugia-kun could you upload an executable with compressed DWARF sections? I'll try to uncompress them. Apparently it's a mere zlib, with a specific header.

[lugia-kun]

@ysbaddaden Uploaded here (compressed with gzip): test2.gz

Source code of this code is:

raise ""

[ysbaddaden]

Thanks. The sections are indeed compressed with ZLIB, and... it's the first time I'm seeing an ELF64 executable in the wild. I'm happy to see that Debug::ELF can read it just like it reads ELF32 :)

EDIT: I spent too many years on Ubuntu Trusty. Modern LD seems to always link ELF64 by default, now. That's weird, ELF64 is only meant for binaries larger than UInt32::MAX (i.e. should never happen).

[ysbaddaden]

I can decompress the COMPRESSED sections with Zlib::Reader, but that breaks Debug::ELF and Debug::DWARF afterwards, since they only expected an IO::FileDescriptor not a IO+... I'm changing #read_section? to fully read sections in memory with transparent decompression and returning an IO::Memory but still have some issues afterwards.

I'll keep digging, and try not to fully refactor Debug...

[rdp]

Example works OK ubuntu and OS X FWIW :)

[rdp]

It seems that some parsers view opcode_base of 0 as an "error" and some don't. Weird. LLDB doesn't seem to handle it (except possibly on accident LOL): https://llvm.org/doxygen/DWARFDebugLine_8cpp_source.html GDB also used to not handle it: http://stanshebs.github.io/gdb-doxy-test/gdb-xref/dwarf2read_8c_source.html line 16571 would blow up but seems it might handle it now "on accident" https://github.com/bminor/binutils-gdb/blob/508559397632e046a1f734f27d2b1971b1d172d1/bfd/dwarf2.c#L2222 but if it's been seen in the wild, may as well allow for it I'd say :)

[ysbaddaden]

Did you read the thread? The sections are COMPRESSED and thus have to be decompressed to read the DWARF data, otherwise it's garbage.

It has NOTHING to do with LLDB or GDB or even Debug::DWARF. It only has to do with Debug::ELF.

I have something working, but haven't finished it, yet.

[rdp]

Oh OK that makes more sense, my bad, thanks for the clarification!