Nixpkgs: GCC has unwanted flags

[andrewrk]

andy@xps:~/tmp$ NIX_DEBUG=true gcc
(some good flags for dealing with /nix/store/* and then...)
  -O2
  -D_FORTIFY_SOURCE=2
  -fstack-protector-strong
  --param
  ssp-buffer-size=4
  -fPIC
  -fno-strict-overflow
  -Wformat
  -Wformat-security
  -Werror=format-security

These flags should be enabled when compiling nix packages, but they should not be always on for general compiler use.

This has caused a broken debugging experience for me on the project I was working on.

Most projects have a "debug build" that has optimizations off, and this prevents that from working correctly.

NixOS version 16.09pre90254.6b20d5b (Flounder)

[andrewrk]

I believe this was caused by #12895

[andrewrk]

Not only that, but these flags are in the extraAfter section, which means I can't even override the flags, for example with -Og in a Makefile or build script.

[andrewrk]

One workaround is putting

    hardeningDisable = [ "fortify" ];

in a nix-shell. Since I was already using a nix-shell for my environment, this is an acceptable workaround. Still, seems weird to assert -O2 by default and have to disable it to get previously expected behavior.

[abbradar]

I think this can be fixed by adding a special flag, NIX_ENFORCE_HARDENING, that is only set in stdenvs. This way the flags won't be enforced in regular gcc invocations and could be disabled in shells by setting this flag to 0.

BTW having -O2 always added is not nice because some applications might want -O3 or -Ofast (which would be overridden). We may want to make this conditional somehow...

[rasendubi]

/cc @globin @fpletz

[fpletz]

You can use

hardeningDisable = [ "all" ];

to disable all hardening flags. This should also work if set as an environment variable.

We will work on a solution after 16.09 using gcc spec files that can detect for example if libraries are built or debugging is enabled. This was just our first iteration and we agree that it is not perfect. Changes to the cc-wrapper always require a full rebuild, which is very painful. More feedback on #12895 would've been helpful.

We didn't anticipate and test that those flags would be propagated to regular gcc invocations outside of nix builds. Not entirely sure how to fix it yet but @abbradar's proposal sounds reasonable. I will look into it after 16.09.

-O2 is required for -D_FORTIFY_SOURCE=2. Not sure though if higher optimizations also work.

[abbradar]

FWIW I found that level 1 fortifying requires inlining:

As I said in that post, the special fortified functions (those that are available in the form of __$func_chk in the libc.so file and provide warnings at build time, and proper stack traces at runtime) only get enabled if inline functions are enabled, so are totally ignored at -O0 (simply disabling inlines, by using -fno-inline won’t stop them from being used, though).

I haven't been able to find anything except of anecdotal evidence that -O2 is required for level 2 (I'm not questioning your choice, rather I was trying to find a list of optimizations that are required so that we can enable them individually).

[fpletz]

(I'm not questioning your choice, rather I was trying to find a list of optimizations that are required so that we can enable them individually).

I'm always open for suggestions. Thanks! :smiley: We'll have to try that.

[abbradar]

Unfortunately it may need peeking into rat nest GCC/glibc's codebases to determine this. Meanwhile I think it's okay to leave -O2.

[copumpkin]

Just as a quick reminder when you talk about gcc: there's also clang to consider on Darwin (and in some cases on Linux too)

[jxy]

So this is why recently gdb suddenly tells me different things in my code and gcc no longer does a good job optimizing my code. I almost thought gcc 5.4 had optimization regressions.

How do I get back a reasonable working gcc? Which nixpkgs version should I revert to?

Does this mean I should put hardeningDisable = [ "all" ]; in all of my custom nix files for my HPC production code, eg, mpich3?

I'm using nixpkgs on a ubuntu without root privilege.

It is NOT okay to leave -Oanything in extraAfter!

[andrewrk]

We're all in agreement here @jxy and going to fix it soon. This is why it's called unstable!

The workaround you mentioned is working for me and is probably the reasonable thing to do until this is fixed.

[jxy]

Right, thanks for the workaround. It wasn't easy to find this thread. I guess I'll need some performance regression tests after changing my system.

[jxy]

Does NixOS 16.09 release version have this issue? If so, I'll put off upgrading until this is fixed.

[vcunat]

Yes, both 16.09 and unstable/master.

[fpletz]

I'm currently working on a fix.

[andrewrk]

even though I knew about this, I temporarily forgot and it caused me to file a bogus issue report on another project: https://github.com/thejoshwolfe/legend-of-swarkland/issues/36

[mboisson]

I'm running into this issue when compiling older versions of GCC.

gcc 4.8 compiles fine, but at run time, it tries to use stack-protector-strong, which is not supported.

I've wasted about 8 hours so far trying to fix this broken thing....
[mboisson@build-node easybuild-easyconfigs]$ gcc --version
gcc: erreur: unrecognized command line option ‘-fstack-protector-strong’
gcc (GCC) 4.8.5

Our fork of nixpkgs is based on 16.09 and is here:
github.com/computecanada/nixpkgs

is there any commit I could pull to fix this ?

[globin]

There is no optimal fix yet, workaround is still to export hardeningDisable=all in your shell while developing.

[mboisson]

Thanks.

[Ralith]

This cost me untold hours of frustration; eventually I just gave up getting anything useful out of a debugger on NixOS. Eagerly looking forward to a fix.

It's important to me that building nix packages with useful debug info (i.e. without -O2) even with nix-build (rather than nix-shell) is possible as well. For example, debugging software that uses LLVM is much easier with a debug build of LLVM which I'd rather not have to make by hand.

[viric]

What about not adding any hardening if IN_NIX_SHELL=1 ? What is bad in this?

[Ralith]

That would be a huge improvement, but would still make it more difficult than necessary to build nix packages with debug symbols. IMO no -O flags should be supplied (let alone in extraAfter!) if debug symbols are requested by the derivation.

[edolstra]

It's not desirable to have nix-shell behave very different from a regular build. The whole point of nix-shell after all is to get an interactive build environment that is otherwise identical to a regular Nix build.

Not supplying -O when debug symbols are requested doesn't sound right either. I do optimized builds with debug symbols all the time... After all, it's pretty useful to be able to get meaningful results out of a production core dump.

However: The real issue here is that add-hardening.sh adds -O2 after the user's CFLAGS. So you might think you're building with -O3, but you're not. That's a pretty critical bug...

A possible fix would be for add-hardening.sh to not add -O2 if any previous -O flag was seen.

[globin]

Just a note if this is not clear, -D_FORTIFY_SOURCE=2 needs at least -O2 to work, but yes generally -O3 should not be overwritten. But I don't think we should not stop adding the -O2 if we see a lower optimisation level or -g because that will remove the hardening on all derivations that have a broken build environment and that are exactly those that need hardening more than others.

[edolstra]

@globin It's probably not a big deal if it overwrites -O1, but it should not overwrite -O0. Maybe the best thing to do in that case is to exit with an error message informing the user that there is a conflict between optimization and hardening flags.

[globin]

@edolstra -O1 disables some checks -O0 all of them (only talking about the fortify stuff).. I think the problem here is that we can't check if the build just defaults to setting one of those which we do want to override or if a user has specified it explicitly e.g. in nix-shell.

[joachifm]

Has anybody looked at basing the hardening logic on an existing wrapper, e.g. https://github.com/thestinger/hardening-wrapper?

[rszibele]

This also caused me countless hours of frustration from being unable to determine the cause of not being able to debug any of my programs through any debugger on NixOS. I was also puzzled why a simple test program with ~10 LOC compiled with "g++ -g -O0 main.cpp" kept jumping around during debugging.

I'll be using export hardeningDisable=all until this is fixed as it seems to work just fine, though it might be a good idea to insert a warning when using g++ while this is still an issue so others wont have to waste time trying to figure out why their debugging just does not work as expected.

[musteresel]

Why are the hardening flags added after the user specified flags after all? Why not add them before, to make them override-able?
https://github.com/NixOS/nixpkgs/blob/master/pkgs/build-support/cc-wrapper/cc-wrapper.sh#L124

[mboisson]

Why add them period ? If the end-user wants to use hardening flags, he or she can add them.

[cstrahan]

Rather than have the cc-wrapper require opt-out, how about we do this: make hardening opt-in, but we'll make the stdenv do the opting-in by default. We could have add-hardening.sh look for something like $NIX_CC_HARDENING_ENABLE, which would have all the non-disabled flags handed off from the Nix expression side -- the idea being that this should require no changes to the API of stdenv.mkDerivation. If that sounds good, I could try to whip up a PR over the next week or two (a little too busy to get to it at the moment) -- or someone else can take a stab at it now, if they'd like.

[lambdafu]

In GnuPG we use the kernel's jitter entropy gatherer, which does not compile when __OPTIMIZE__ is defined. I spent quite a bit of time tracking down why -O0 does not work as expected, and finally found this issue. This is really unconventional.

[cstrahan]

So, 5 thumbs up on my last recommendation. I'll create a PR over the next day or two, and then hopefully we can finally call this resolved.

If anyone thinks this isn't a good solution, please tell me now.

[Ralith]

It's not immediately obvious to me how that gives the correct behavior (i.e. no involuntary -O2) for nix-shell. Also, as @edolstra said:

However: The real issue here is that add-hardening.sh adds -O2 after the user's CFLAGS. So you might think you're building with -O3, but you're not. That's a pretty critical bug...

Wouldn't the proposed change fail to address the root problem, while also being more complex than just supplying flags in the correct sequence in the first place? I don't think -O2 by default is a problem, so long as explicit user-provided -O flags aren't overridden.

[cstrahan]

It's not immediately obvious to me how that gives the correct behavior (i.e. no involuntary -O2) for nix-shell.

Here are my thoughts:

If I install gcc as a systemPackage in NixOS, I shouldn't then have to stuff export hardeningDisable=all; in my profile to ensure that I get the expected result when I'm building with e.g. my IDE. In reallity, it's more complicated than that, as there are many scenarios where env vars are dropped (e.g. sudo), and I'd have to work around each of those cases.

However, for nix-shell, this isn't a bug -- things are working as intended. As @edolstra says:

It's not desirable to have nix-shell behave very different from a regular build.

If you want to use nix-shell simply to bring a bunch of dev tools/libraries into your environment, and not necessarily because you want to replicate the build environment of a package, then the solution is straight-forward: add hardeningDisable = "all"; to your shell.nix. Here's (a rather minimal) one from a recent project:

with import ~/src/nixpkgs { };

let
  # from TARGET_RECIPES
  deps = [
    c-ares
    backward-cpp
    libevent
    pythonPackages.gcovr
    gtest
    gperftools
    http-parser
    /* lightstep */
    nghttp2
    /* libnghttp2 */
    protobuf3_2
    tclap
    rapidjson
    spdlog
    boringssl
  ];

in

runCommand "dummy" {
  hardeningDisable = "all";
  buildInputs = [
    bazel
  ] ++ deps;
  shellHook = ''
  '';
} ""

I suspect most people are reporting here because they're running into issues with the gcc wrapper installed globally, rather than within nix-shell. To prevent future headaches for nix-shell users, we should provide a warning in the docs where nix-shell is introduced.

Also, as @edolstra said:

However: The real issue here is that add-hardening.sh adds -O2 after the user's CFLAGS. So you might think you're building with -O3, but you're not. That's a pretty critical bug...

I'll also try to address that too, thought I haven't decided how exactly. Perhaps I can make the logic be: if passed -ON where N < 2, add the -O2; else leave the commands as is. Thoughts?

First, off to bed ;)

[Ralith]

It's not desirable to have nix-shell behave very different from a regular build.

The current behavior is just as wrong in a regular build as it is in a nix-shell. If any -O is passed explicitly, it should not be clobbered.

Perhaps I can make the logic be: if passed -ON where N < 2, add the -O2; else leave the commands as is. Thoughts?

The only thing that needs to be done is to move the injected options before, rather than after, the user-supplied options, so that the user-supplied options will correctly override them. As has been discussed at length above, overriding -O2 with lower optimizations is an immensely important capability so as to generate debuggable binaries.

[edolstra]

We got bitten at work by this again today, i.e. a debug build at -O0 silently got changed into -O2. This really needs to be fixed since this issue has been a blocker for the last two releases.

How about this: cc-wrapper adds -O2 only if it hasn't seen a preceding -O flag. If it has seen -O where n >= 2, it does nothing.If is has seen -O0 or -O1, it should fail with an error message like:

cc-wrapper: Hardening is incompatible with -O0. Set hardeningDisable = ["bla"] to disable hardening.

This way hardening can be enabled by default, but we don't get silent optimization with -O0.

(What about -Os?)

[vcunat]

Oh, I had been wondering why my debug builds don't feel like compiled with -O0.

[gebner]

If is has seen -O0 or -O1, it should fail with an error message like:

I agree that it's better to fail loudly instead of silently changing flags. But it won't solve this issue: either way using gcc outside of nixpkgs is broken. nix-shell -p gcc --run "gcc -O0 foo.c" should work and call gcc with the -O0 flag and no further optimization. I really expect this to just work when developing C/C++ code.

IMHO stdenv and the standalone gcc/clang attributes should use different wrappers. stdenv should have hardening enabled by default, and fail with -O0 as you suggested. The standalone attributes should use the minimal wrapper that is necessary to compile programs, but not enable hardening. In this setup, you can still use nix-shell -A to troubleshoot a regular build since it would use the stdenv wrapper.

[vcunat]

-O0 silently decreasing hardening might be "unexpected" as well, but outside nix build/shell it seems OK to be without hardening by default. The wrapper would better be the same; stdenv may e.g. set some specific variable like we have IN_NIX_SHELL already.

[edolstra]

@gebner Yes, I guess hardeningDisable should be hardeningEnable. I.e. if that variable (which would be set by default in stdenv) is not set, you don't get hardening so -O0 works as expected outside of a Nix build or nix-shell. Inside a nix-shell hardening would still be enabled by default, but you would get an error when using -O0.

[Ralith]

@vcunat

outside nix build/shell it seems OK to be without hardening by default

Silently breaking -O0 inside nix-shell is not okay. Using nix-shell to manage development environments is one of Nix's most compelling features.

Having an explicit gcc/clang build input change the behavior would be weird and unintuitive, especially when you want to build a nix package with debug symbols (which is very common when e.g. developing against LLVM). It's not really an improvement on the current state of affairs; in either case, nix silently defeats your efforts unless you know exactly the right magic incantation.

@edolstra

How about this: cc-wrapper adds -O2 only if it hasn't seen a preceding -O flag. If it has seen -O where n >= 2, it does nothing.If is has seen -O0 or -O1, it should fail with an error message like:

Why can't we just place all automatically-injected options before manually supplied ones? This would be massively more intuitive, require less complexity, and protect against similar issues in the future. Yes, there might(!) exist some small number of packages which won't be fully hardened as a result; those packages are broken, and can be fixed.

[vcunat]

@Ralith. ~I didn't mean to claim that at all. I'm sorry for the misunderstanding.~ I'm sorry; I'm too tired.

[ElvishJerricco]

I agree that if the only cost of simply putting the arguments before the supplied ones is that some packages break, that seems like by far the most useful and intuitive approach. The problem is tracking down which packages are broken and verifying that we actually can fix them

[vcunat]

@Ralith: the problem about nix-shell is that some people expect it to be an interactive nix-build and some use it just to bring packages into scope. EDIT: before I managed to finish the thought properly, Gebner did it for me below.

[gebner]

If hardening is only enabled by stdenv, then

• nix-shell -p gcc would just bring gcc into scope without hardening, and
• nix-shell -A hello still works as an interactive nix-build and enables hardening by default.

AFAICT this would address both usages of nix-shell.

[Ralith]

@vcunat Those aren't mutually exclusive, we just need to make the nix-build semantics sane, i.e. don't clobber explicitly passed options.

@gebner As discussed in my prior comment, that doesn't really improve things. I maintain build environments with a shell.nix containing a derivation and often want debug builds of nix packages, so stdenv very much applies. Adding an explicit build input for a compiler that stdenv would otherwise supply is just another magic incantation required to prevent nix from silently sabotaging me.

[vcunat]

Using mkDerivation from nixpkgs is indistinguishable from a real nixpkgs build. If you want something different, you should explicitly differ "in some way".

[Ralith]

@ElvishJerricco To be clear, respecting explicitly supplied arguments will almost certainly not break any packages, it only risks marginally weaken the hardening applied to any packages in the unlikely event that they have hardcoded -O0 in their makefiles or similar. I'm skeptical that there even exist any packages which do this and aren't for some reason specifically unable to use higher optimization levels, in which case this could actually fix some packages.

[Ralith]

@vcunat I do not want something different. I want real nixpkgs builds to respect the flags I pass.

[ElvishJerricco]

Frankly, I think cc-wrapper needs to be as simple as possible. The more rules someone has to know in order to use $CC properly, whether in a nix-shell or whatever else, the more likely it is to generate problems like this issue. Rather than adding rules about gcc and stdenv.cc being different and keeping complex rules about which arguments are automatically added where, it makes more sense to me to remove rules and simply say “there may be some arguments prepended to your argument list.”

[cstrahan]

@edolstra et al:

Is my PR (https://github.com/NixOS/nixpkgs/pull/28029/files) not suitable? If not, why? I think it gets us most of the way there - I’m willing to make necessary changes to fit the precise semantics we want with respect to failing fast (e.g. in the presence of multiple -O). Just tell me how to improve it and I’ll do so.

I should note that, as it stands, it makes nix-build and nix-shell behave the same way, while making system-wide installed gcc behave like what most people would expect (unhardened). So that part should please @Ralith, I’d think.

[andrewrk]

I'm going to throw out a crazy idea here:

It's up to each individual application how they want their software compiled. We can have hardening by default, but if they override it to disable hardening, the application wanted it that way. And that decision should be respected, and if we want to counter that decision we should file an upstream bug.

[musteresel]

@Ralith Yes, there are packages which must disable optimizations for (parts of) their build. An example is (IIRC, it's been a while since I worked on its sources) CLISP, which does some non-C-standard stuff and thus relies on the optimizer not messing that up.

[cstrahan]

/ping

@edolstra I hate to bug you personally/directly, but would you be able to specify your desired approach to fixing this bug - or if you can't spare the cycles for that, could you name an individual or two that you'd feel comfortable deferring to?

There's enough subtlety to the problem that I doubt everyone here will come to anything approaching a consensus. Nonetheless, the problem needs fixing, and I'd like to do the work to fix it - if someone with authority can state what an acceptable solution is.

Just as a reminder, I have this PR open: #28029

There's some feedback regarding cross compilation from @Ericson2314 that I can address, but I'm first waiting to know if we're going to accept that general approach, a variant of it, or something else entirely.

[edolstra]

@cstrahan The PR looks good to me, thanks. It might be nice to issue an error/warning if there is a conflict between a hardening flag and a user-supplied -O flag, but that's not super important.

[cstrahan]

@edolstra Awesome - thanks! I'll address @Ericson2314's points and try to get it ready for merging soon as I can - probably before the weekend is up.

[andrewrk]

Even though I originally filed the bug, and it bit me again after that, I still forgot about it again and got bit by this issue again, wasting the musl-lib developers' time.

[Ericson2314]

Hopefully @cstrahan's PR will be done soon. My bad recently for reviewing it slowly when he took it up again.

[glaebhoerl]

I ran into this as well, even though I'm on 18.09. After writing out most of a comment here to ask why, I realized I needed to re-instantiate my default.nix in order to actually pull in the new stuff. After doing that, I promptly got:

/nix/store/akak0rxhbi4n87z3nx78ipv76frvj841-glibc-2.27-dev/include/features.h:381:4: error: #warning _FORTIFY_SOURCE requires compiling with optimization (-O) [-Werror=cpp]
 #  warning _FORTIFY_SOURCE requires compiling with optimization (-O)

whenever I tried to build my project (which uses -Werror, yes), with no indication of what to do about it. Is this the expected behavior?

Since I've been reading this thread, I knew to try hardeningDisable = [ "fortify" ];, which thankfully fixed it, but Google was of no help at all. Now that I've written this comment, hopefully that might at least change. :)

(Yes I could've disabled -Werror, and if I only have to do that when also using -O0 I guess it's not such a huge deal, but it's surprising and perplexing nonetheless.)

[nh2]

We continue to be plagued by this; I've opened a new issue on #60919.

Copying the descrpition here, so that people subscribed here can see what it's about and subscribe there if interested (I think the discussion should probably continue there):

---

This is in some way a continuation of #18995 GCC has unwanted flags.

Even though that one is marked as solved, I have over the last 2 years encountered many situations where silently -O2 -D_FORTIFY_SOURCE=2 is added to builds where I have explicitly requested that a debug build be done (which should result in -O0 / no -O flag being passed).

For example, right now on meson-based builds (on Darwin, but I believe on Linux too), if I set

mesonBuildType = "debug";

it does not have the desired effect unless I also add

hardeningDisable = [ "fortify" ];

because the fortify hardening adds -O2 -D_FORTIFY_SOURCE=2 no matter what.

The same happend to me in various C++ builds, and it's always a big drag in development time when you just can't get your -O0 debug builds working.

---

Reading the PR description of #28029, it says

user supplied -O flags should take precedence over the hardening flags

so I suspect that this isn't good enough, because most development builds don't add -O0 explicitly -- instead they add no -O flag at all, so even with the PR -O2 is still added.

I think we must do _something_ about it because it's a total pain.