Nixpkgs: Enable debug symbols for all packages

[stapelberg]

PR https://github.com/NixOS/nixpkgs/pull/15539 mentions this, but I think it’s worthwhile to have a separate GitHub issue.

Currently, a large number of libraries are lacking debug symbols even though environment.enableDebugInfo = true; is specified in my /etc/nixos/configuration.nix. The following list is taken from a gdb session on NetworkManager:

(gdb) info sharedlibrary 
From                To                  Syms Read   Shared Object Library
0x00007ffff7ddaab0  0x00007ffff7df5990  Yes         /nix/store/y5js9j7zgrmz4j2dqcy0nvaqyk3izbv7-glibc-2.24/lib/ld-linux-x86-64.so.2
0x00007ffff7bd6630  0x00007ffff7bd7d01  Yes (*)     /nix/store/33z8lb3yzh18z10aqszc07imwlfkyhgb-util-linux-2.28.1/lib/libuuid.so.1
0x00007ffff78c7c80  0x00007ffff7993a44  Yes (*)     /nix/store/3yvgy2mnsm2cfvbjkdcaxlzg1xwh80fj-gnutls-3.4.14/lib/libgnutls.so.30
0x00007ffff764b830  0x00007ffff767a1cf  Yes (*)     /nix/store/yzb1maz797c80606aqj78hafbc4lidcr-p11-kit-0.23.2/lib/libp11-kit.so.0
0x00007ffff7408fd0  0x00007ffff740d6e6  Yes (*)     /nix/store/4b33d5w92qxaw464kgyfs6ncskwmp056-libidn-1.33/lib/libidn.so.11
0x00007ffff71f5d50  0x00007ffff7200b6b  Yes (*)     /nix/store/13lpz809nmpzl12hmn09xnd6yk2jvy7s-libtasn1-4.8/lib/libtasn1.so.6
0x00007ffff6fc4ee0  0x00007ffff6fe15ef  Yes (*)     /nix/store/5rqh0fg8004z9gkwvywbpg75ydf3w9gy-nettle-3.1.1/lib/libnettle.so.6
0x00007ffff6d8f270  0x00007ffff6d9c1e8  Yes (*)     /nix/store/5rqh0fg8004z9gkwvywbpg75ydf3w9gy-nettle-3.1.1/lib/libhogweed.so.4
0x00007ffff6b04280  0x00007ffff6b6ee68  Yes (*)     /nix/store/gb81bp3xbwrhkqd8cw7cvwxl0qlzah4x-gmp-6.1.1/lib/libgmp.so.10
0x00007ffff68edb90  0x00007ffff68f0c62  Yes (*)     /nix/store/z9nhrhcgiqcag9frhdz8lwriypvl8kc5-libgudev-230/lib/libgudev-1.0.so.0
0x00007ffff7fd5c60  0x00007ffff7fe6886  Yes (*)     /nix/store/lamh95q185ymigm217s64x08qcmnfhvm-systemd-231/lib/libudev.so.1
0x00007ffff66d13c0  0x00007ffff66deb35  Yes (*)     /nix/store/vw0wcq2zz1w3l86ly5pzw2dphfpychym-libnl-3.2.28/lib/libnl-3.so.200
0x00007ffff7f59760  0x00007ffff7fae2aa  Yes (*)     /nix/store/lamh95q185ymigm217s64x08qcmnfhvm-systemd-231/lib/libsystemd.so.0
0x00007ffff64c4fe0  0x00007ffff64c6784  Yes (*)     /nix/store/z4mg6wiy5y26p0djr95b4b6cnpf3m7qb-libndp-1.6/lib/libndp.so.0
0x00007ffff622a2e0  0x00007ffff6270aef  Yes (*)     /nix/store/3dg2ld3q204xsi2ml12073nw9bng7xf7-libsoup-2.54.1/lib/libsoup-2.4.so.1
0x00007ffff5eb07a0  0x00007ffff5f93160  Yes (*)     /nix/store/9didfvj6wba723zcnls1c5iwp9jv3l3l-libxml2-2.9.4/lib/libxml2.so.2
0x00007ffff5b83660  0x00007ffff5beffba  Yes         /nix/store/y5js9j7zgrmz4j2dqcy0nvaqyk3izbv7-glibc-2.24/lib/libm.so.6
0x00007ffff58b2440  0x00007ffff5955100  Yes (*)     /nix/store/1bx29pqdzwc722an88nzxxvn24al2nyb-sqlite-3.14.1/lib/libsqlite3.so.0
0x00007ffff5555490  0x00007ffff562892b  Yes (*)     /nix/store/nvclglzzqj57gzclmq44aziw33y7d2c4-glib-2.48.2/lib/libgio-2.0.so.0
0x00007ffff531d2b0  0x00007ffff531e1b5  Yes (*)     /nix/store/nvclglzzqj57gzclmq44aziw33y7d2c4-glib-2.48.2/lib/libgmodule-2.0.so.0
0x00007ffff51081a0  0x00007ffff51143d9  Yes (*)     /nix/store/ahjpbwi957n7pqp7i7ys6s2vxmk4dr06-zlib-1.2.8/lib/libz.so.1
0x00007ffff4ef2810  0x00007ffff4efe93b  Yes         /nix/store/y5js9j7zgrmz4j2dqcy0nvaqyk3izbv7-glibc-2.24/lib/libresolv.so.2
0x00007ffff4ca79e0  0x00007ffff4cd83a6  Yes (*)     /nix/store/nvclglzzqj57gzclmq44aziw33y7d2c4-glib-2.48.2/lib/libgobject-2.0.so.0
0x00007ffff4a95aa0  0x00007ffff4a9a35c  Yes (*)     /nix/store/3zzw0gc6078nsp829x2va9690kca237j-libffi-3.2.1/lib/../lib64/libffi.so.6
0x00007ffff479ef80  0x00007ffff4815bb9  Yes (*)     /nix/store/nvclglzzqj57gzclmq44aziw33y7d2c4-glib-2.48.2/lib/libglib-2.0.so.0
0x00007ffff4516730  0x00007ffff4565996  Yes (*)     /nix/store/10jk5h18ahh2xaad74ng3mgx0lfjmfj6-pcre-8.38/lib/libpcre.so.1
0x00007ffff42fd810  0x00007ffff430a531  Yes         /nix/store/y5js9j7zgrmz4j2dqcy0nvaqyk3izbv7-glibc-2.24/lib/libpthread.so.0
0x00007ffff40f4d70  0x00007ffff40f590e  Yes         /nix/store/y5js9j7zgrmz4j2dqcy0nvaqyk3izbv7-glibc-2.24/lib/libdl.so.2
0x00007ffff3d75930  0x00007ffff3e9e1a3  Yes         /nix/store/y5js9j7zgrmz4j2dqcy0nvaqyk3izbv7-glibc-2.24/lib/libc.so.6
0x00007ffff3b526d0  0x00007ffff3b53fd6  Yes (*)     /nix/store/4am908pjq2j4bzqcy051mx62ab0rd1xa-libcap-2.25-lib/lib/libcap.so.2
0x00007ffff394af70  0x00007ffff394dd1f  Yes         /nix/store/y5js9j7zgrmz4j2dqcy0nvaqyk3izbv7-glibc-2.24/lib/librt.so.1
0x00007ffff3725e30  0x00007ffff373bff2  Yes (*)     /nix/store/3qq5qv97jdqg0ywq14spj7kmif39fhnw-xz-5.2.2/lib/liblzma.so.5
0x00007ffff35132a0  0x00007ffff351f5bb  Yes (*)     /nix/store/bppnjqhm8jjbwc8215i9qvg4ws39avz1-lz4-131/lib/liblz4.so.1
0x00007ffff320ce00  0x00007ffff32cce48  Yes (*)     /nix/store/i7z7i541bfa6gxw1g71wzf82gj5ms8s7-libgcrypt-1.7.3/lib/libgcrypt.so.20
0x00007ffff2ff0a30  0x00007ffff2ffa099  Yes (*)     /nix/store/4ixpz8hqaqv4h4hibm0m8ljpbv38wd3n-libgpg-error-1.24/lib/libgpg-error.so.0
0x00007fffea9c30d0  0x00007fffea9c8d91  Yes         /nix/store/y5js9j7zgrmz4j2dqcy0nvaqyk3izbv7-glibc-2.24/lib/libnss_files.so.2
0x00007fffea7a6800  0x00007fffea7b74c6  Yes         /tmp/nm/nix/store/g05c3qil03b5lg02nqlz5viv2ad3bsam-network-manager-1.2.2/lib/NetworkManager/libnm-device-plugin-wifi.so
(*): Shared library is missing debugging information.

In the above list, only glibc seems to come with debug symbols, no other library does.

https://nixos.org/nixos/manual/options.html#opt-environment.enableDebugInfo explains how to override individual packages, but that’s tedious and requires recompilation of all reverse-dependencies, which is really really inconvenient (my machine is compiling spidermonkey since half an hour… :-/).

Debian recently started building debug symbol paackages by default and Fedora seems to have debuginfo RPMs built by default as well. I think having debug symbols on by default is a feature that users of modern Linux distributions have come to expect :).

Could we enable debug symbols for all packages by default please?

cc @kevincox

[edolstra]

I don't know if we should enable it for all packages. We should probably have some guesstimate of the additional storage costs. However, it's definitely a good idea to enable it for some major libraries.

[stapelberg]

I’m assuming you’re referring to the build output stored on build servers, since the debug info on the disks of end users is configurable using enableDebugInfo :).

For comparison, here’s a list of the sizes (in bytes) of all 1726 Debian packages with a “-dbg” suffix currently in Debian testing (amd64): debian-debug-sizes.txt
The sum of all of these packages is about 10 GiB.

Here’s a list of the sizes (in percent, e.g. 100 is 100%) relative to the corresponding package: relative.txt
The average size of a debug package is 4557% of the base package.

Note that these should be treated as rough estimates: the base package matching was done textually (as opposed to understanding the source-package relationship).

Based on these values from Debian, can you do a guesstimate as to how much additional space would be required? AFAICT (without any knowledge of the NixOS build infrastructure), the additional demand seems to range in the tens of gigabytes region, which should be totally doable…?

[kevincox]

I would love to see this happen. As for my other PR life got in the way and there was debate that never got resolved. The basics work but you can see the checkboxes to see what features I would like added (although they can likely be added as a later patch). I should have time to pick it up around now, hopefully banging it into a shape that is generally agreed to be acceptable.

As for the package sizes this is obviously a code tradeoff and it will depend on what hydra/S3 can have available. It would definitely be awesome to have most libraries built with debug symbols.

[copumpkin]

Would we put them in a separate output?

[kevincox]

On Sep 12, 2016 23:08, "Daniel Peebles" [email protected] wrote:

Would we put them in a separate output?

I'm pretty sure this is a requirement.

[vcunat]

Would we put them in a separate output?

It should all be just a matter of adding separateDebugInfo = true; into the particular packages.

Note: the debian list you provided has 1726 lines. One Hydra evaluation of ours contains way more builds (~35k). There are some duplicates in there, and many would only have negligible debug info, but there still might be a significant difference. I can't see how to find easily, except to build one generation in that way.

[stapelberg]

I’m aware of that. Debian has >20k source packages, so we’re talking about a roughly 10% sample size here at best — that should be representative enough for a guesstimate.

I agree that the easiest way to get accurate numbers would be to just build a generation with debug symbols enabled, though :).

[vcunat]

What's their total size of non-debug stuff on this list? (EDIT: ratio of sums or averages seems much more reliable than average of ratios.)

[stapelberg]

It’s 8.19G of debug packages and 0.97G of corresponding non-debug packages.

[vcunat]

Well, 9-fold increase of required space would be a disaster. Hopefully they don't compress their debug files; we do it.

[stapelberg]

On the package level, compression is not accounted for in my numbers: the sizes are for the unpacked, uncompressed package contents.

On the debug file level, I can’t tell whether they are compressed or not…?

$ file /usr/lib/debug/.build-id/ee/13cb8a18d72bea00fd7f7ff22544d3024eb5c3.debug
/usr/lib/debug/.build-id/ee/13cb8a18d72bea00fd7f7ff22544d3024eb5c3.debug: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter *empty*, for GNU/Linux 2.6.32, BuildID[sha1]=ee13cb8a18d72bea00fd7f7ff22544d3024eb5c3, not stripped

[vcunat]

Hmm, I see the compression doesn't make that great difference (~30 %) and even after it the debug files are expected to be 2–4x size of the corresponding binaries (for complex C++ at least). https://gcc.gnu.org/wiki/DebugFission But we store more than just ELFs.

We've got glibc example where the increase is ~40 %:

22M     /nix/store/6fix3zqpnahyml8zp2sxi2rwan55rgb8-glibc-2.24
2.7M    /nix/store/g2f08f2gb4dpc2l0g57jy6yp26bfc4hj-glibc-2.24-bin
14M     /nix/store/2gfgppppk96xi91xv90xlz85z73m8prj-glibc-2.24-debug
3.2M    /nix/store/8sl4jfs3nq0pkq4gg655s3axrxdx7z29-glibc-2.24-dev
8.2M    /nix/store/m2mvna9pk9hv1qgxwi124avzdq1i5q96-glibc-2.24-static
50M     total

Perhaps if we restricted the system-wide enabling to x86_64-linux for now, the total increase might be as small as 10 %.

[vandenoever]

Having debug symbols enabled should be the default in my opinion, especially in a partially self-compiled distributions like NixOS. The threshold to submitting bugs should be low. Great to see it as a milestone for 17.03.

[volth]

If the storage space on cache.nixos.org is the only concern, the debug symbols can be deleted immediately after the build (or have very short time-to-live).

It is a cache, a cache miss should not be a problem.
A cache miss would fall back to local build, but with a big advantage over the currently recommended way to get debug symbols with libxxx = pkgs.libxxx.overrideAttrs (oldAttrs: { separateDebugInfo = true; });: only one library to be compiled, not all its reverse-dependencies

[volth]

And.. what about debug symbols for kernel?

[0xABAB]

Storage costs also depend on the device being used to store them on. On some systems, I would much rather store the debugging symbols on cheap hard disk space as opposed to an NVME or SSD disk, while keeping the rest of the Nix store on fast storage.

For me it would be the difference between enabling debug symbols for everything and not.

[kevincox]

@0xABAB I suspect that you wouldn't install debugging symbols on most machines, they would be kept on your binary cache until you need to symbolize a stack trace. This would likely be done on developer machines or dedicated servers.

[0xABAB]

@kevincox In that case the specific feature being discussed here is not clear.

As a developer oriented user, I would like to:

1. run a program with a segfault
2. get a dialog that it has crashed
3. have an option in configuration.nix to say to automatically download debugging symbols on demand
4. see a beautiful back trace which in KDE's user interface has three stars or contains no question marks/hidden symbols in simple English.
5. purge these downloaded debugging symbols automatically after two weeks if free disk space is below 30%.

I can imagine an operator of a binary cache to want low operational costs, but it seems that with a properly designed Nix the costs are only dependent on the costs of S3, not of the server's attached storage.

I don't see why any user would ever be required to build packages. @volth apparently has reasons to believe there are still cases like that, but I don't see them.

[domenkozar]

I think the next step is rebuilding nixos-small with debug on and compressing all debug outputs to xz to see how much each package size increases on average.

[edolstra]

.debug files are already compressed using --compress-debug-sections=zlib. (Whether this is useful is questionable, given that NARs in the binary cache are already compressed using xz, so probably we just end up with a worse compression ratio...)

Note that since a few months, cache.nixos.org provides a index of .debug files (indexed by their build ID). For example, to get the debug symbols for libc.so.6:

$ readelf -a /nix/store/kjwbqnh13dxh6w4pk2gb3ddmhpiaihqg-glibc-2.25/lib/libc.so.6 | grep 'Build ID'
    Build ID: 443f38a7db6510d81a7afe03b1a42de8ef6c6ebf

$ curl https://cache.nixos.org/debuginfo/443f38a7db6510d81a7afe03b1a42de8ef6c6ebf
{"archive":"../nar/0xx62iin8dpql0fn70ad7fy8d2w4vzc10d7nbl84al8k723np3p4.nar.xz","member":"lib/debug/.build-id/44/3f38a7db6510d81a7afe03b1a42de8ef6c6ebf.debug"}

So https://cache.nixos.org/nar/0xx62iin8dpql0fn70ad7fy8d2w4vzc10d7nbl84al8k723np3p4.nar.xz contains the desired debug symbols.

I also wrote a little FUSE file system that fetches debug symbols automatically from the binary cache. So you can mount the file system on (say) /run/debug, point the NIX_DEBUG_INFO_DIRS environment variable to it, and gdb and eu-stack will be able to obtain debug info for any pacakge in the binary cache that has debug info enabled.

[domenkozar]

@edolstra if I understand correctly one needs to recompile nixpkgs with separateDebugInfo = true to get this working right?

[edolstra]

@domenkozar Yes.

[lheckemann]

In general, what would the preferred solution for actually getting everything building with debug symbols be, assuming that the storage problem is somehow solved (even if it's just by not uploading debug outputs to the binary cache)?

[edolstra]

I think that would just be a one-line change to stdenv to set separateDebugInfo by default.

[lheckemann]

I tried

diff --git a/pkgs/stdenv/generic/make-derivation.nix b/pkgs/stdenv/generic/make-derivation.nix
index 7b5f9f7d6b0..ab506267c54 100644
--- a/pkgs/stdenv/generic/make-derivation.nix
+++ b/pkgs/stdenv/generic/make-derivation.nix
@@ -59,7 +59,7 @@ rec {
         (if attrs.meta.description or null != null
           then builtins.unsafeGetAttrPos "description" attrs.meta
           else builtins.unsafeGetAttrPos "name" attrs)
-    , separateDebugInfo ? false
+    , separateDebugInfo ? ! (attrs ? outputHashAlgo || attrs ? buildCommand)
     , outputs ? [ "out" ]
     , __impureHostDeps ? []
     , __propagatedImpureHostDeps ? []

Because fixed-output derivations can't have multiple outputs, and because buildCommand overrides fixupPhase, but there's still a bunch of stuff that fails to build (because of not producing the debug output) even inside stdenv.

[layus]

Is there some way to enable separateDebugInfo based on the presence of a c/cpp/... compiler in the build inputs ? That would force to use stdenvNoCC when there is no need to. And would also help catching rogue overrides of buildCommand.

[vcunat]

It's not difficult, but I think it would be over-optimistic. Standard stdenv is still used in many derivations that don't produce (c-style) binaries.

[timor]

Is the motivation for this issue being able to easily run a debugger for a specific package? When I played around with debug information, I noticed that in addition to the debug symbols, the relevant source code is usually also not there anymore.

I handled that problem with this quite intrusive piece of nix code: https://github.com/timor/timor-overlay/blob/master/default.nix#L16-L108, which might be of interest for this issue.

The basic idea is to be able to source-level-debug a certain package and its (manually specified) dependencies. The way it works is that the source code of the specified packages is saved after build, debug symbols generated, and then an environment dedicated for running gdb for that particular set of packages is generated.

That by itself does not provide zero-effort stack traces for arbitrary applications, though. It also could be extended to trickle down the separateDebugInfo attribute to its dependencies, if not already enabled.

[matthewbauer]

We could do mkdir -p $debug in these cases to avoid the failure.

[matthewbauer]

Is everyone on board to try this for 19.09? I think it's a good idea to do this by default. Having a few of these be empty directories doesn't seem too bad.

FWIW I have a branch to get this working with LLVM/Darwin builds here:

https://github.com/matthewbauer/nixpkgs/commits/macho

It's probably better to get this figured out on Linux first, but it seems to be fairly easy to get this working with macho binaries.

[edolstra]

@matthewbauer On board with what exactly? Enabling debug info for all packages? Exporting sources? Adding mkdir -p $debug?

[matthewbauer]

I mean to try to enable all debug symbols globally. I don't really care about specifics, but the mkdir -p $debug hack seemed like the easiest way to avoid tons of separateDebugInfo = false for everything in the tree.

[edolstra]

I'm not sure if that's a good idea in light of some of the statistics above (e.g. https://github.com/NixOS/nixpkgs/issues/18530#issuecomment-246820586). It seems that enabling debug symbols would significantly increase storage costs and increase build times (since debug symbols need to be uploaded to S3).

[matthewbauer]

That seems reasonable if it really is the costly. Perhaps we could try some smaller set of packages like haskellPackages and enable it there? That has been requested a few times here:
https://github.com/NixOS/nixpkgs/issues/51987

[nh2]

enabling debug symbols would significantly increase storage costs and increase build times

I'd like to add that having debug symbols for all packages easily available would be a big boon for troubleshooting issues in production quickly.

Debian/Ubuntu's -dbg packages are super useful here, and NixOS should have them too to be equally useful!

I don't have a good answer to build times, but when it comes to storage, I wouldn't be surprised if the cost of one of us debugging a production segfault without debug info once per year far outweighs the storage costs for debug info.

[vcunat]

Our main stumbling point seems to be the uncertainty about the size this adds, so now I tried to get an estimate of what tree-wide separateDebugInfo = true would do in _our_ NixPkgs.

ATM the sample is smaller than I'd like, but it seems likely that growth of one nixos-unstable evaluation (on x86_64-linux) wouldn't be too far from 20–25 GB. I expect that would be OK-ish, especially if restricted to this single platform (for now?). I'm not sure if a separate GC strategy for these is easy to implement for our S3 (suggested above, good idea).

Notes:

• I suspect this flag isn't enough for haskellPackages, as their sizes seem suspiciously tiny. This experiment could be extended... or done specifically on a subset like haskellPackages (greater accuracy with similarly small sample size).
• A large part of our packages are various languages (python, perl, ...), which could explain why our ratio of sizes seems very different from Debian.
• Sampling like this will always suffer from the risk of missing "black swans"... but we could find such cases retrospectively and just disable .debug for them if they don't seem worth it. They statistically can't be in a very large number – a couple percent (of count) at most, even with this small sample size.
• We'll need some tweaks besides just enabling the flag, otherwise quite a large fraction of packages breaks. I did some quick hacks to pull this off and suggested a few more: https://github.com/vcunat/nixpkgs/commits/v/try-debug

Methodology, more numbers

I downloaded a store-path list for a current channel release. The paths are ordered by crypto-hashes, so a simple prefix makes quite a high-quality random sample. I "assigned" the (missing) .debug to each .out I encountered in that sample, and found its du-size when rebuilt. The debug info is compressed, so .nar.xz would be just a few percent smaller (I tried xz -9 on a couple).
Estimate ATM:

88904 KiB * 33667/126  / 1024^2 = ~22.7 GiB

I consider the sample small, because the distribution is clearly very asymmetric. For nice one, mere 30 would be fine. Non-zero tail:

8       /nix/store/1ccjbi37bnslppim88kq9xbncf6ny45j-nano-erl-0.1.0.1-debug
8       /nix/store/31dydkpmipn89gz6a5kq9jdjww3iwsw8-prettyprinter-compat-ansi-wl-pprint-1.0.1-debug
8       /nix/store/4rq0vni3v43779sgpmhvkjygswafrmkz-restless-git-0.7-debug
8       /nix/store/6a1nlcj161q90xs146ip9af00bq1xvlr-ifstat-legacy-1.1-debug
8       /nix/store/732yhkf8hq0gm3jpwixi9xiyl5ghs144-text-icu-0.7.0.1-debug
8       /nix/store/85px3zyajm2nxiha5qf7d32b1g6a3ln2-bench-1.0.12-debug
8       /nix/store/8kj422wv2m1c8731762h1ywd2c7qmmlf-timer-wheel-0.1.0-debug
8       /nix/store/98mwjfw4nh3vnqvasq5ji1a7ndl0ixwb-leancheck-instances-0.0.3-debug
8       /nix/store/9cdg2nd5xfjbpx1g08rmhg122kzyxd3l-tmp-postgres-0.1.1.1-debug
8       /nix/store/9gx8yyq5cvi99fk6n5miqjf7457nrd64-monad-hash-0.1.0.2-debug
8       /nix/store/anid0qr48hjy0gk2x4kfbqkj65hr08b4-acl2-0.0.1-debug
8       /nix/store/cy7x1ldfvwpx4dpn667r8v6qypjkjqv7-alarmclock-0.6.0.2-debug
8       /nix/store/igrz4jm4ah8877myay2g8qkv0an4c41i-complex-integrate-1.0.0-debug
8       /nix/store/jrix237vl6navdh8s1y0gkrxrlz7yz39-detrospector-0.3-debug
8       /nix/store/kz06bh1afrja7h0q7anzxy2vph8d8cxn-aeson-generic-compat-0.0.1.3-debug
8       /nix/store/lcfayl3ylz0zq3ap33grhcr2yny6cniv-html2hamlet-0.3.0-debug
8       /nix/store/n7lvvr1cgqsxm7v81n4yl3d6w21qwc8a-language-cil-0.4.0-debug
8       /nix/store/p2x5ccdwyziibncb3ilq81l7y9h39663-osx-ar-0.11-debug
8       /nix/store/qxlrd12yk9q65kmxdhv3c945d4svwfg5-yesod-transloadit-0.7.1.0-debug
8       /nix/store/ry2v88mcwk7zji4kw2q736n3p7pcbya5-wrap-0.0.0-debug
8       /nix/store/s1a0xd0ci1bkvqaqz1vlvw3lwsvwvh7h-multiplicity-0.1.0-debug
8       /nix/store/snlj5p6mkqr64vfirw3cz7i9f0hzv7n0-amazonka-efs-1.6.1-debug
8       /nix/store/wq5fa2hg68rr0c31l6g551bc5z8xrkwz-machines-0.6.4-debug
16      /nix/store/andq2i1shnpn5p2c5xi9c7qbw46xpai6-fst-0.10.0.1-debug
16      /nix/store/n6vpf2ngj0bc3ccicgc42x3wk8cf0pzd-cab-0.2.18-debug
16      /nix/store/sg2p75lf4v54v2h9jacks70zwcg8zdw8-foobar-0.1.0.0-debug
24      /nix/store/dc1qs1hvfgcs84n9jyhbfm1zpv5rnz5p-perfect-hash-generator-0.2.0.6-debug
24      /nix/store/f7rgh0s8py4w04panm9anq7qph6zza5v-msilbc-2.1.2-debug
32      /nix/store/72c6bjyvbgvm1knrpm25cyaxdmynmpxm-elementary-print-shim-0.1.3-debug
32      /nix/store/bbqihknyq85nj163bk6w9cl2bl4ssf9h-lua5.2-lrexlib-posix-2.9.0-1-debug
44      /nix/store/vk4bgrfmxd6hzb2a7vy72rr4547yam3c-smemstat-0.02.00-debug
48      /nix/store/87lqfns07js7bcf1761qd6iy8h8whrq0-ispell-3.3.02-debug
52      /nix/store/wy6x5yhpkj8gi0598h929ajf1sjv8xdf-perl5.29.6-B-Utils-0.27-debug
64      /nix/store/9cs6cvvpxp75vfl24fhk7a3r1y58bsbp-python2.7-simplejson-3.16.0-debug
72      /nix/store/86gxba59x9xfjzh3797m01mdwcfih3mg-armadillo-9.200.7-debug
112     /nix/store/8xnnq1h33zc3xyz84hisgc28l25qxakc-gpsd-3.16-debug
116     /nix/store/xskbklx0isbfyk8qgx58wl4c3pdp0ppx-libmilter-8.15.2-debug
232     /nix/store/bmzabsyqg5zgw908aga2xlyl5s36dabm-python2.7-pyodbc-4.0.25-debug
324     /nix/store/1lss2a7h9swfc9z22jq01qvfimsl6m0m-epdfview-0.1.8-debug
380     /nix/store/cqgn8g526hp49pbywkv9sr1g619sagz4-kmediaplayer-5.54.0-debug
588     /nix/store/5s7862r6fljwh2fxs5683g938jxb9aqc-python3.7-python-imread-0.6-debug
596     /nix/store/s3xpkhqqyb4l64fag7gprvpxg3bf4agn-python2.7-pygobject-3.30.4-debug
608     /nix/store/9ys9shyjpbnjyldk660g10ln1k9ha17n-can-utils-20170830-debug
732     /nix/store/rjzmn88g157qy15zndng3xdj21wqnh00-libtcod-1.5.1-debug
1164    /nix/store/bgk1bya00p95in9w1qrpy9l5cclw1sgl-swh-lv2-1.0.16-debug
1456    /nix/store/8yl8bmqcb93ixnk2qzm00jbamlsn4c7j-klibc-2.0.4-debug
1536    /nix/store/csp7ijbmk7j01297nqnw8l0n5xcirnri-python2.7-mpi4py-3.0.0-debug
2632    /nix/store/p8mcfhkgc2176m7kfdsib9wfaq6fj0qi-cargo-update-1.5.2-debug
2904    /nix/store/gzii09fj42ik09czvnqxbph6cgx6iywx-vdr-fritzbox-1.5.3-debug
3008    /nix/store/m6na0qsj7dbs1xkdby1h8rfnq6q6bnjq-neovim-unwrapped-0.3.4-debug
7964    /nix/store/560zq6kzy5za1bslpzrl1q41dmw5csfm-eventviews-18.12.1-debug
15924   /nix/store/6lq9slb05ndbx6g5b9g2ads25hr8sink-xqilla-2.3.4-debug
48004   /nix/store/4m07q2v2l34a62njwxg3y0ap4r8czfqd-openbabel-2.4.1-debug

EDIT: well, perhaps it would've been better to sample from all the binary cache paths instead of just those from a single channel release, as different packages will have different rebuild "rates" that might well be significantly correlated to the ratio of their .debug size. The problem is that I do not have that data, so I couldn't do such sampling.

[nh2]

• I suspect this flag isn't enough for haskellPackages, as their sizes seem suspiciously tiny.

@vcunat Yes, you need a DWARF-enabled Haskell compiler. I see you already found that info here, but you may not have seen that I had already gathered some basic statistics: https://github.com/NixOS/nixpkgs/pull/52255#issuecomment-447646279. In that case, the DWARF overhead was 3.5x for this large Haskell executable, and 3x when xz-compressed.

[vcunat]

Apparently for just haskellPackages the total extra cost will be larger than for the rest of NixPkgs. Very quick estimate: 30–40 GB, based on the haskellPackages subset of the list above (26 pkgs) and code from the other PR (which does not _split_ an output).

[Moredread]

How do these numbers compare to builds without debug symbols?

[vcunat]

I'm not sure ATM. I only tried to estimate the .debug sizes so far, not the size of the rest.

[edolstra]

We should set up a jobset with debug symbols enabled globally so we can compare the total size with nixpkgs master.

[vandenoever]

Debug symbols for large packages like Qt, Firefox and LibreOffice would be very welcome. The amount of work a user needs to do to write useful bug reports for upstream is unrealistic.

I do not know how to enable debug symbols for libraries. I figured out that for leaf packages I can add an overlay like so:

self: super:
{
  kcalc = super.kcalc.overrideAttrs (oldAttrs: rec {
    separateDebugInfo = true;
  });
}

in ~/.config/nixpkgs/overlays/. But that just lets me debug the top layer of the stack.

To send useful crash reports to projects or do debugging, debug info for the entire stack is needed.

How can I enable debug info for all Qt and KDE libraries?

[grahamc]

I would like to revisit enabling debugging symbols for all packages. From what I've read and heard, doing so will have some negative effects:

• binary cache storage will go up significantly
• build and upload time will be lengthened, increasing load on the Hydra coordinating machine

I don't have a solution for the second point, but for the first: would it be valid to periodically trawl the debuginfo directory in the s3 bucket for files older than 1 month, and delete the NARs associated with it?

Would this eliminate the first concern, by keeping only a rolling set of debuginfo files?

[edolstra]

We could revive the binary cache garbage collection script. That would reduce storage costs a lot, not just wrt debug symbols. Otherwise deleting NARs referenced by the buildid links would be dangerous (you don't want to delete something reachable in some other way, and debug symbols don't have to be in a separate output).

[Ericson2314]

I think we're making a mistake conflating nixpkgs and cache.nixos.org policy here. Even if we never stored debug outputs, but still built them, it would be useful to CI that the building still works. I'm OKish doing a mass rebuild for myself to get those outputs, but only if I'm sure that mass rebuild won't be in vain because it isn't tesed and bitrots.