Nix: Allow printing full log on nix build

[jtojnar]

When I am creating a package, I often iterate short fix–rebuild cycles. Now, for figuring out a problem, I usually need a full log but this does not seem to be available. The only way I found is copying the *.drv path in the output and pasting it as an argument to nix log and this is far from convenient.

It looks like the tailing is controlled by verboseBuild variable:

https://github.com/NixOS/nix/blob/d4e93532e2695fa7204937a96573b686a3b1a1ea/src/libstore/build.cc#L1548

but it apparently can only be disabled, not enabled:

https://github.com/NixOS/nix/blob/3a5a241b3209f14f8801b902ba20b5cb0666c9df/src/libmain/shared.cc#L155-L159

I am using Nix 2.0pre5968_a6c0b773.

[dtzWill]

Same! For this reason (and the "--dry-run"-doesn't-work-unless-you-attempt-a-non-dry-run-first problem) I find myself using "nix-build" for development/testing.

A flag to show build output instead of progress bar would be great, as suggested here I believe, but for this particular workflow I think it'd be super awesome if there was some sort of "show-log-on-failure" option that used the nice progress bar by default but if something went wrong performed equivalent of "nix log $failing-build.drv".

I've previously thought maybe a "nix lastlog" to show log of last failed build.... but I think that's a bad idea for a few reasons such as likely having unexpected behavior when multiple nix-build invocations are executed in parallel, etc.

[lheckemann]

You should be able to just replace build with log in your nix command to get the full log.

[knedlsepp]

@lheckemann: I guess we should at least unify the nix build and nix log UI: When I want to build my default.nix, I can simply type nix build. I however can't use nix log to get the build log: "more arguments are required".

[lheckemann]

Oh, I guess that was overseen. They are unified, except apparently for the case where no arguments are passed. Other than that, e.g. nix build -f . and nix log -f . should work.

[knedlsepp]

Yes, that case was probably overseen. Actually nix log -f . doesn't work either, but nix log -f . "" does what I would expect from nix log.

[djahandarie]

The biggest problem with just running nix log -f . "" afterwards is that you can't get log until the build is entirely finished.

For very long builds I'd to get warnings as soon as possible to shorten my development feedback loop, and this isn't possible with nix build at the moment.

@shlevy Have you taken a look at this lately?

[lheckemann]

https://github.com/NixOS/nix/pull/2483 proposed adding a mode similar to classic nix-build where log output was immediately printed, which @edolstra didn't like. In the meantime you can still use nix-build.

[dtzWill]

Bumping log-lines is a nice helper in the meantime, however! (yes?)

[lheckemann]

@dtzWill not entirely, because

you can't get log until the build is entirely finished.

still applies.

[bb010g]

If the main blocker for this is mixing when printing build logs live, apenwarr's "redo, buildroot, and serializing parallel logs" covers how they handled this for redo. I don't think everything directly applies from there, but what might work well here is allowing specification of specific derivations to show live logs for on the command line.

Since Nix derivations can't contain / or *, we could glob on names to show logs for, and control if builds they cause should be shown. For example, with a gitignore-style syntax, --verbose-build 'hello-*' would show build logs for any derivation built by that expression with a name matching hello, 'hello-*/*' would show the logs for that derivation and anything it directly invokes to build (i.e. its inputDrvs, nix-store -q --references style), /* would show all derivations directly built by the command, hello-*/** would show the logs for all derivations built that it depends on (-q --requisites style), and * would show all derivation build logs.

Past that, logs should be ordered first by their order on the command line (--verbose-build 'b-*' --verbose-build 'a-*' would first show the logs of all b-* derivations built, then all b-* derivations), then by their glob ordering (* does not impose ordering, but if we supported braces, {b,a}-* would order), then by derivation name (store derivation name 82vssrpg4s76f1izb5jcsb73311gr531-python3-3.5.7 would come before 48pw8p0gqfcbyd5gc8zlma03a54bzjxp-python3-3.6.8, despite Nix's normal lexicographic ordering placing python3-3.6.8's derivation before python3-3.5.8), then by derivation hash (qn9hma1hxrgmdka8g79aaj7l6hjqagkk-hello-2.10 would come before zm44ahh0lnb1rx5hz7624izibaqq7ds9-hello-2.10). If a derivation matches multiple times, the earliest match should be used.

I think this would help relieve any usability issues only streaming one live log at a time causes, as (provided we show a distinct header before every build) the user can sanely adjust their command line flags if working on a specific derivation to place that build's log(s) first after a couple of tries, resulting in guaranteed live logs for that build, even if the hash changes, and the glob syntax allows for situtations where versions can possibly be changed in the Nix source but nix's output remains easy to skim for what you're interested in due to the consistent ordering across suffix and hash changes.

As we'd be linearizing the logs still, indentation growth from the deep dependency chains Nix can have on fresh builds is not an issue.

[grahamc]

diff --git a/src/nix/main.cc b/src/nix/main.cc
index 3ec5f48d..41456a36 100644
--- a/src/nix/main.cc
+++ b/src/nix/main.cc
@@ -96,7 +96,7 @@ void mainWrapped(int argc, char * * argv)
     }

     verbosity = lvlError;
-    settings.verboseBuild = false;
+    //settings.verboseBuild = false;
     evalSettings.pureEval = true;

     NixArgs args;

=>

$ ./inst/bin/nix build
unpacking sources
unpacking source archive /nix/store/9pkrwlsf78bxi35c5h2c3d1rnpxw5qy2-source
source root is source
corepkgs/local.mk
corepkgs/derivation.nix
corepkgs/unpack-channel.nix
corepkgs/imported-drv-to-derivation.nix
corepkgs/fetchurl.nix
corepkgs/config.nix.in
corepkgs/buildenv.nix
.dist-files
Makefile
mk/libraries.mk
mk/lib.mk
mk/templates.mk
mk/jars.mk
mk/install.mk
mk/README.md
mk/clean.mk
mk/tracing.mk
mk/dist.mk
mk/patterns.mk
mk/functions.mk
mk/programs.mk
mk/tests.mk
[1/1/2 built, 0.0 MiB DL] building nix-2.3pre6852_0000000 (buildPhase): ...

[grahamc]

master has:

$ ./inst/bin/nix build  -L -f ../test.nix
hi> hello!
hi> bye!
[1/0/1 built, 0.0 MiB DL] running post-build-hook for hi

so I think this is fixed! :)

[doronbehar]

-L | --print-build-logs are not documented in nix build --help

[archseer]

I agree, this issue should be reopened. The flag is documented on nix --help, but it should be propagated to nix build --help.

[doronbehar]

-L | --print-build-logs are not documented in nix build --help

Perhaps it's because the flag has meaning for other commands? Though I can't imagine for which..