Go: cmd/link: compress debug info

[josharian]

Compressing our debug info might offer a significant, cheap file size win.

Things to do:

• Figure out which targets to compress debug info for by default.
• Provide a linker flag to control compression on/off.
• Teach the linker to compress debug info.
• Teach the Go packages how to read compressed debug info.

@ianlancetaylor commented off-list:

Compressed debug info is supported by gdb and the GNU binutils.
Yes, we should do it in our toolchain for relevant targets.

Related: #11773

[mwhudson]

It's only tangentially related, but it's been at the back of my mind to try to figure out what would be required to make stripping Go binaries (and so being able to use detached debugging symbols) more of a supported thing.

[josharian]

cc @crawshaw @derekparker

I think that this could reduce the size of cmd/go by about 15%.

How I got that number (OS X commands):

$ dwarfdump -R `which go`
[snip]
Segments
Segment Name     vmaddr           vmsize           fileoff          filesize         maxprot  initprot nsects   flags
---------------- ---------------- ---------------- ---------------- ---------------- -------- -------- -------- --------
__PAGEZERO       0000000000000000 0000000000001000 0000000000000000 0000000000000000 00000000 00000000 00000000 00000000
__TEXT           0000000000001000 00000000007bf000 0000000000000000 00000000007bf000 00000007 00000005 00000006 00000000
__DATA           00000000007c0000 0000000000045060 00000000007bf000 0000000000020ac0 00000003 00000003 00000005 00000000
__LINKEDIT       0000000000806000 0000000000083c4c 0000000000ac6000 0000000000083c4c 00000007 00000003 00000000 00000000
__DWARF          0000000000805060 0000000000000000 00000000007e0000 00000000002e5968 00000000 00000000 00000008 00000000

Sections
Section Name     Segment Name     addr             size             offset   align    reloff   nreloc   flags    reserv1  reserv2  reserv3  size     size %
---------------- ---------------- ---------------- ---------------- -------- -------- -------- -------- -------- -------- -------- -------- ======== ======
__text           __TEXT           0000000000002000 00000000003a6f38 00001000 00000004 00000000 00000000 00000400 00000000 00000000 00000000    3.65M 33.49%
__rodata         __TEXT           00000000003a8f40 0000000000283f0c 003a7f40 00000005 00000000 00000000 00000000 00000000 00000000 00000000    2.52M 23.06%
__typelink       __TEXT           000000000062ce50 000000000000e7f0 0062be50 00000003 00000000 00000000 00000000 00000000 00000000 00000000   57.98K  0.52%
__gosymtab       __TEXT           000000000063b640 0000000000000000 0063a640 00000000 00000000 00000000 00000000 00000000 00000000 00000000    0      0.00%
__gopclntab      __TEXT           000000000063b640 000000000018415b 0063a640 00000005 00000000 00000000 00000000 00000000 00000000 00000000    1.52M 13.90%
__symbol_stub1   __TEXT           00000000007bf7a0 00000000000000d8 007be7a0 00000005 00000000 00000000 80000408 00000000 00000006 00000000  216      0.00%
__nl_symbol_ptr  __DATA           00000000007c0000 0000000000000138 007bf000 00000002 00000000 00000000 00000006 00000024 00000000 00000000  312      0.00%
__noptrdata      __DATA           00000000007c0140 0000000000014008 007bf140 00000005 00000000 00000000 00000000 00000000 00000000 00000000   80.01K  0.72%
__data           __DATA           00000000007d4160 000000000000c960 007d3160 00000005 00000000 00000000 00000000 00000000 00000000 00000000   50.34K  0.45%
__bss            __DATA           00000000007e0ac0 000000000001e138 00000000 00000005 00000000 00000000 00000001 00000000 00000000 00000000  120.30K  1.08%
__noptrbss       __DATA           00000000007fec00 0000000000006460 00000000 00000005 00000000 00000000 00000001 00000000 00000000 00000000   25.09K  0.22%
__debug_abbrev   __DWARF          0000000000805060 00000000000000ff 007e0000 00000000 00000000 00000000 02000000 00000000 00000000 00000000  255      0.00%
__debug_line     __DWARF          000000000080515f 00000000000ac904 007e00ff 00000000 00000000 00000000 02000000 00000000 00000000 00000000  690.25K  6.18%
__debug_frame    __DWARF          00000000008b1a63 000000000005c574 0088ca03 00000000 00000000 00000000 02000000 00000000 00000000 00000000  369.36K  3.31%
__debug_info     __DWARF          000000000090dfd7 0000000000154b49 008e8f77 00000000 00000000 00000000 02000000 00000000 00000000 00000000    1.33M 12.20%
__debug_pubnames __DWARF          0000000000a62b20 000000000005f4c1 00a3dac0 00000000 00000000 00000000 02000000 00000000 00000000 00000000  381.19K  3.41%
__debug_pubtypes __DWARF          0000000000ac1fe1 000000000002897f 00a9cf81 00000000 00000000 00000000 02000000 00000000 00000000 00000000  162.37K  1.45%
__debug_aranges  __DWARF          0000000000aea960 0000000000000030 00ac5900 00000000 00000000 00000000 02000000 00000000 00000000 00000000   48      0.00%
__debug_gdb_scri __DWARF          0000000000aea990 0000000000000035 00ac5930 00000000 00000000 00000000 02000000 00000000 00000000 00000000   53      0.00%
$ # manually convert fileoff and filesize from hex to dec, calculate rough compressed dwarf size
$ head -c 8257536 `which go` | tail -c 3037544 | gzip | wc -c
 1048123
$ # find current binary size
$ wc -c `which go`
 11836492
$ # calculate savings as a percent of current size
$ python -c "print(100*(3037544-1048123)/11836492.)"
16.807522026

I also double-checked that I'm interpreted fileoff and filesize correctly by using otool and xxd to dump just the __debug_info section and compressed it; the compression ratio for __debug_info is even better than for the entirety of the dwarf segment.

Note that this probably includes updating the dwarf stdlib to handle compression.

See also #5158.

[minux]

I think debug/elf already supports compressed sections.

Could we compress other sections as well (and do decompression at runtime)?

The only segment we can't decompress at runtime is the code segment.

[josharian]

I took yet another stab at this, but I am having a hard time making friends with the linker, so I'm going to leave this for @aarzilli or @heschik.

Here are some very useful implementation details from @ianlancetaylor, copied from #20463:

---

DWARF compression is done at the file container level. There are two ways to do it. The older way is to prefix the DWARF section name with a 'z', as in '.zdebug_info'. The newer way, which applies to sections of all types, not just DWARF sections, is to set the SHF_COMPRESSED flag in the ELF section flags.

Clearly Mach-O doesn't support the SHF_COMPRESSED flag, but I don't see any reason that it wouldn't support .zdebug_*. Looking at the gdb source code, it seems that it ought to work. I suppose the only way to find out is to try it and see what happens.

A .zdebug-* section starts with the four characters "ZLIB" followed by a uint64 of the uncompressed size in big-endian order followed by the compressed data.

SHF_COMPRESSED works differently. The section data starts with a compression header: a 32-bit word with the compression type (1 for ZLIB), on 64-bit systems a 32-bit padding word, a 32-bit or 64-bit word with the uncompressed section size, a 32-bit or 64-bit word with the uncompressed alignment.

[heschik]

I've spent a little time digging into this and I'll leave some notes for myself or whoever comes next.

The linker doesn't have the .debug_info section as a whole unit in memory at any point. dwarfp contains each top-level DIE as an individual symbol, and they're written to the output file individually by Dwarfblk in data.go. So that's the only place that we can insert compression of the whole section, perhaps by checking a new flag in the Section type.

This won't work for external linking, because the .debug_info section contains relocations to text symbols that will be placed by the external linker. New versions of GNU ld (I'm not sure when it was added exactly) support --compress-debug-sections to compress debug sections passed to. (gcc has another flag -gz, but presumably that's just for the individual CUs it's creating, not the whole section. That doesn't help with the Go CUs.)

[ianlancetaylor]

I don't think we should compress the debug info when doing external linking. The object file that we create is only going to live briefly, probably in /tmp, likely on a RAM disk. It will be deleted after the external linker runs. Compressing the debug info in that case will buy us nothing; we should leave it to the external linker.

[heschik]

I agree, it's both impossible and undesirable :)

I might work on this for 1.11 as part of getting DWARF location lists enabled by default, to offset the increase in binary size.

[ghost]

What about providing the option to completely strip debug info? The -w and -s flags don't remove this info.

[ianlancetaylor]

@r04r That is a completely different problem that should be discussed on a different issue, not here. Thanks.

[ghost]

@ianlancetaylor This issue seems to be about acquiring a "significant, cheap file size win.". Instead of compressing, just removing this information seems more significant, and cheaper. Is it viable? Should I make another issue?

[ianlancetaylor]

@r04r This issue is about a cheap file size win without losing any information. Removing the debug info produces a binary that can not be used with a debugger, so that is a different matter--not a win at all for people who want to use a debugger.

Yes, if the linker's -s and -w flags do not work as documented, please open a different issue.

[josharian]

Marking as release-blocker for 1.11, since https://go-review.googlesource.com/c/go/+/100738 just went in and executable sizes are up >15%.

[heschik]

Spent a little more time. More notes, adding to https://github.com/golang/go/issues/11799#issuecomment-315392705.

Revisiting it, I'm not totally sure where to put the compression any more. The linker calculates the output's layout in (*Link).address(), then uses it in (*Link).reloc() to compute relocation targets and rewrite symbols as necessary. The DWARF cannot have been compressed before that, since it contains relocations. But compression could alter the layout, invalidating the previous results. In practice, the only things that come after the .debug sections on my Linux/AMD64 machine are .note.go.buildid, .symtab, and .strtab. I don't think any of those ought to be relocation targets. As long as we don't mind making that a guarantee, we can add a compression step beween reloc and Asmb, using the .zdebug approach for both Mach-O and ELF based on Ian's commentary above. I don't know about PE.

[aarzilli]

Other options:

1. Move all .debugs all the way to the end of the file
2. Do the relocations for .debugs only, compress, do the relocations for everything else.

[aclements]

To expand on @heschik's comment, perhaps the root of the problem here is that (*Link).address() does two things: it assigns virtual addresses, which has to be done before reloc; and it assigns file offsets. These are logically independent. If we could disentangle them, we could assign virtual addresses, then apply relocations, then compress the DWARF section, then do file layout. I suspect the reason they're entangled is simply that there's no simple list of segments anywhere. The list is all unrolled into code and it would be awful to duplicate that code, so it's all in address.

[bradfitz]

I'd like to stress that this really should happen for Go 1.11, as our binaries have gotten 20% bigger since Go 1.10 and that's not something users will be super happy about.

[aclements]

I'll see if I can put together a fix for this, but it's definitely not an easy change. We'll see how it turns out.

[ianlancetaylor]

@aclements I'm also happy to look at this if you prefer.

[aclements]

I have a specific plan in mind that I'll spend an hour or two on and see where it gets us.

[aclements]

Is compress/zlib non-deterministic? I think I nearly have this working, but make.bash is failing to converge when building toolchain3.

[bradfitz]

It should be deterministic. I have no recollection of it ever not being so. There are no maps in compress/* and no time.Now.

[ianlancetaylor]

CC @dsnet for compress/zlib question, but as far as I know it is deterministic.

[dsnet]

It is deterministic, but I won't be surprised if the output is different is when Write is called at different boundaries (e.g., w.Write("he"); w.Write("llo") vs w.Write("hello")).

It is definitely not stable though and subject to change as improvements are made to the flate implementation.

[gopherbot]

Change https://golang.org/cl/111682 mentions this issue: cmd/link: separate virtual address layout from file layout

[gopherbot]

Change https://golang.org/cl/111683 mentions this issue: cmd/link: compress DWARF sections in ELF binaries

[aclements]

Russ figured out why it's not converging: the different bootstrap stages are using different compress/zlib versions, which are leading to different compressed DWARF section contents.

[aclements]

It looks like the trybots are still on GDB 7.7, which is the current minimum requirement for the runtime GDB test. Support for compressed sections was added in either 7.10 or 7.11 (it doesn't work with 7.9, it does work with 7.11, it's not in GDB's change log, and the only GDB commit that seems obviously related is from 2012, so I'm not sure exactly when it was added).

[bradfitz]

We can update the trybots from Debian jessie to Debian stretch.

[gopherbot]

Change https://golang.org/cl/111895 mentions this issue: cmd/link: compress debug sections in external linking mode

[ianlancetaylor]

According to the gdb NEWS file support for reading compressed debug sections was added in gdb 7.0. I'm guessing that was support for zlib-gnu, though, not for zlib-gabi (zlib is a synonym for zlib-gabi). The reason you don't see anything in the gdb ChangeLog is that the work is actually done in the BFD library. There I see support for zlib-gabi added in May, 2015, which should be in gdb 7.10.

[gopherbot]

Change https://golang.org/cl/118276 mentions this issue: cmd/link: compress DWARF sections in ELF binaries

[gopherbot]

Change https://golang.org/cl/118277 mentions this issue: cmd/link: compress debug sections in external linking mode

[jcajka]

bd83774 seems to be cause of issue #25863

[gopherbot]

Change https://golang.org/cl/118716 mentions this issue: cmd/link: separate virtual address layout from file layout

[robpike]

This should be left open until all platforms, not just ELF, do this.

[agnivade]

Then #25927 should be closed I believe.

[aclements]

I think it's fine to leave #25927 open for the Windows-specific aspects of this. There's already been a decent amount of discussion there.

For macOS, I'm concerned because my understanding is that lldb doesn't support compressed DWARF sections in Mach-O binaries (in fact, it didn't support them in ELF binaries until last month). And I believe dsymutil doesn't support them either (@heschik?)

[heschik]

Well, dsymutil doesn't matter so much since we only use it during external linking, and we would need the host linker to do the compression in that case. But there's no --compress-debug-sections on macOS. I suppose we could do the compression in machoCombineDwarf.

Then there's the question of whether we want to. dwarfdump and lldb won't support it, but Delve could. So I suppose it's technically feasible as long as we're willing to leave lldb behind. I would want a flag so that I could use dwarfdump when necessary though.

[gopherbot]

Change https://golang.org/cl/119816 mentions this issue: cmd/link: enable DWARF compression on Windows

[gopherbot]

Change https://golang.org/cl/119815 mentions this issue: debug/elf,macho,pe: support compressed DWARF

[gopherbot]

Change https://golang.org/cl/120155 mentions this issue: cmd/link: support DWARF compression on Darwin

[heschik]

Here's the state of things as of that last commit. We now enable DWARF compression by default on almost all platforms. I didn't do anything for plan9, wasm, and nacl, but I'm not actually sure if they get debug information anyway.

To debug a binary with DWARF compression, you'll need a modern version of GDB or a version of Delve after https://github.com/derekparker/delve/commit/440b4405626d0b27b960a7d070da9b5211b7ccef compiled with tip Go. Tip versions of lldb might work, but I haven't tested them. Compression can be disabled with -ldflags=-compressdwarf=false if there's any problems, or if you need to use inspection tools like macOS dwarfdump that don't understand the compressed DWARF.

I'm not aware of any significant remaining work to do here, so closing. Please comment if I missed something.

[gopherbot]

Change https://golang.org/cl/138182 mentions this issue: doc: mention -compressdwarf=false on gdb page

[gopherbot]

Change https://golang.org/cl/138184 mentions this issue: [release-branch.go1.11] doc: mention -compressdwarf=false on gdb page