Nixpkgs: Zero Test Failures (ZTF) for NixOS 18.09

[xeji]

While failures of channel-blocking NixOS tests tend to be fixed quickly, we generally seem to give less attention to non-blocking test failures. A number of these have been around for quite some time.

But tests are there for a reason, so I propose we adopt a slightly stricter approach to NixOS tests for 18.09:

• All tests in nixos/tests should either be dropped or become blockers for the full release channel and automatically be included in the tested Hydra job.
• Exceptions include tests of unstable/experimental features or tests under development. These could be tagged with an attribute like meta.experimental = true; to exclude them from the tested job.

Some benefits of this approach:

• Leverages all existing tests for QA as we will have to act upon each failure (at the very least, disable the test temporarily)
• Gives us an incentive to fix or get rid of unstable, randomly failing test cases
• Helps us clarify what is expected to work on partially supported platforms like i686 (which has a number of failing tests but not much seems to be done about it).

Implementing this will involve some tedious cleanup work but I believe this is worth it.

/cc @vcunat @srhb @matthewbauer

[srhb]

I fear that the meta.experimental idea just shifts the responsibility for what is a blocker and what is not. That might be okay, but I'm not sure it will actually do anything to encourage real change.

[vcunat]

Of course it's better if all the tests just worked, but I'm not sure what exactly to do about it? It feels similar direction to the idea of updating a channel only if _all_ builds succeed (i.e. "hard ZHF" all the time).

I agree we should disable tests that always fail for a longer time, as that's clearly a waste of Hydra resources, so meta.broken or something similar can be used, as completely removing the tests from nixpkgs typically doesn't seem suitable (even in case the failure is for all platforms). And we may consider adding (currently) reliable tests to release blockers, especially those that test something "important".

[dtzWill]

It might be helpful to increase visibility of test results-- with a badge or something on the main page?

[vcunat]

Hmm, I wonder if we should have some kind of "how can I help" section on a visible place, e.g. (linked from) here. I expect most contributors will "help" by fixing/improving stuff they already use, but this might still help to steer some kind of people to work that's generally considered useful for the project but not particularly affecting any single person.

[xeji]

Thanks everyone for your feedback. Considering the valid concerns raised, let's try to improve our test set without adopting a strict policy first. I have started to investigate and fix currently failing tests.

And we may consider adding (currently) reliable tests to release blockers, especially those that test something "important".

Let's do that for 18.09 after some cleanup.

[jcrben]

Perhaps this discussion could be consolidated with https://github.com/NixOS/nixpkgs/issues/40257?

[xeji]

This is more about policy while #40257 is about fixing specific tests, so I'd prefer to keep them separate.

[c0bw3b]

Should we rename this to Zero Test Failures (ZTF) for NixOS 19.03 and update the milestone accordingly?
It's still a thing but we're past 18.09 anyway :)

[worldofpeace]

Stale, closing.