Nixpkgs: [Discussion] Compiler Testing

[dtzWill]

(splitting this from a comment I was about to make on #28223, see that for context-- 'stdenv' here refers to the clang stdenv, as in llvmPackages_4.stdenv)

Testing whether this stdenv works on even basic code requires going a bit "out of the way" to produce an ad-hoc expression like I gave earlier-- on top of remembering to explicitly check this.

It seems like it would be useful to add this to some testing, perhaps ensuring simple C (and C++) apps build on various compiler configurations? (gcc5, gcc6, clang, clang+libcxx)...?

Nix is well-suited to expressing these things (and I'd be happy to work on it),
but first I'd like to gauge interest and thoughts on where this should be implemented
to best integrate with existing development and testing workflows.

Thoughts on the idea on where this should live code-wise to best work with our testing/release workflows?

[vcunat]

Very simple tests tend to be good within builds of packages themselves. I don't see a difference if it's a stdenv. Large tests (resource-wise) tend to be better split into a separate build, and there's a recent thread already: #27604.

[7c6f434c]

Maybe it could be hello_by_stdenv attribute set with a derivation for
each stdenv case produced by explicitly passing stdenv? Getting all
these to build would already be a good sanity check.

[copumpkin]

We already have a simple test for the Linux & Darwin bootstrap tools like this:

https://github.com/NixOS/nixpkgs/blob/master/pkgs/stdenv/darwin/make-bootstrap-tools.nix#L275-L332

It only runs when building bootstrap tools though, which doesn't happen very often.

[Ericson2314]

I made a tests attribute set in all-packages. Maybe let's put that there instead? Can still import it in the make-bootstrap-tools expressions.

[vcunat]

Bootstrap tools are tested as part of the unstable job.

[Ericson2314]

@vcunat well I mean it would be nice to use the a similar test for any stdenv, not just a minimally rigged-up bootstrap tools.

[LnL7]

I made this https://github.com/NixOS/nixpkgs/pull/28943 while fixing the clangStdenv, there also was an issue recently that caused gcc to generate broken binaries on darwin.