Vector: Officially support nix as a distribution channel

[binarylogic]

Following up on #1238, I'd like to officially support the nix package manager as a distribution channel. @thoughtpolice was nice enough to already add Vector. We should be able to create a release script for this, similar to what we did with homebrew. Before we begin work I would like to answer the following questions:

1. Dose an automated release process make sense for this channel? There are other things going on in this file that could change (feature flags, dependencies, etc).
2. Should we consider moving the nix files file into our /distribution folder so that we can maintain changes through our workflow, then sync the latest version when we tag a release? This obviously has the downside of getting out of sync with changes in the official repository.

Noting, it's fine if we decide to manually maintain this. I would just like to remove the manual/human aspect from the release process where it reasonably makes sense.

Caveats

• [ ] Add notice that releases are delayed a few weeks for stability reasons.

[a-rodin]

I think we can do the following:

• Put a nix package build script template to Vector's repository;
• Check that it works in CI as part of our testing workflow to ensure that the package build script is in sync with the latest changes in Vector;
• Check that our build script can be merged to NixOS/nixpkgs as part of our testing workflow;
• Automatically create PRs to NixOS/nixpkgs when a new release is tagged.

With this approach answers to both question are positive.

[a-rodin]

I'm having issues with building a nix package for the new release, as it results in errors from Cargo saying that it cannot pull leveldb from the repository. This needs some investigation.

[thoughtpolice]

Vector is now at 0.6.0 in nixpkgs; see nixos/nixpkgs@3e88e1b14439e9c8c3e1f3d7058116fffefad5a9

The reason you got a weird leveldb failure is because of the way Nix downloads Cargo dependencies, and the way the expression is written. In brief, Nix calls cargo twice: once to download the "vendored" set of dependencies into a local directory (in a highly restricted environment that is allowed network access), and once again to build everything using those dependencies in a second phase (with no network access.)

Second, Cargo uses feature flags when it wants to resolve dependencies, obviously. The Nix expression for Vector upstream uses an "inclusive" set of feature flags -- nothing unlisted is enabled. In Vector 0.6.0, the LevelDB code was reworked to use a fork, which added a new dependency and a new feature flag. The nix expression for 0.5.0 specified leveldb, but now it has to specify that and also leveldb/leveldb-sys-2. One is for the high level library, the other for the thin bindgen wrappers. If you don't enable the flag, it doesn't get downloaded into the vendored package set.

So this adds up to a situation like the following:

• You run cargo vendor to download the source.
  
  
  
  • It resolve dependencies
  
  
  • It gets leveldb, since that --feature was enabled
  
  
  • It does not get leveldb-sys-2, because that feature was not enabled. Cargo vendor does not check that dependencies like this are resolvable/valid at the download phase; it only calculates the exact versions of everything it "sees" and grabs it.
  
  
• Next, you check the hashes for the source code of all the code. They're good.
• So you start the build, and tell Cargo to use the vendored source, not the network.
  
  
  
  • Cargo tries to resolve dependencies
  
  
  • Oops: the leveldb crate needed by vector requires leveldb-sys-2, but it wasn't downloaded during the vendor step, because that feature wasn't enabled.
  
  
  • Cargo cannot download from the network at this point, so the build fails.
  
  

The long and short of this is that the upstream Nix expression should probably move to using feature list exclusion rather than inclusion, so cases like this are handled better. The only reason we need the flags right now is because I haven't worked out Kafka support, because rdkafka is a PITA...

[binarylogic]

Thanks @thoughtpolice. @a-rodin, before we close this out I want to make sure we:

• [x] Get https://github.com/NixOS/nixpkgs/pull/75767 merged.
• [ ] If possible, add @a-rodin to this list of maintainers here.
• [x] Add the nix package manager to our docs.

[a-rodin]

We also need to ensure that all necessary run-time dependencies are added to the Nix package. In particular, I think we want systemd to be in that list to enable journald source.

[a-rodin]

To do this, I think we can proceed with the approach outlined in https://github.com/timberio/vector/issues/1242#issuecomment-557817516, namely build the Nix package in CI and test it. I've created a separate issue https://github.com/timberio/vector/issues/1430 to track progress on it.

[Hoverbear]

Happy to help with this if there's anything left to do. :) I've been using this package.

[windware-ono]

Have you stopped updating nix packages? It's still at 0.8.1 released in March.

https://search.nixos.org/packages?query=vector

[thoughtpolice]

@windware-ono The automatic update robot for the vector nix package failed to work because the updates from 0.8 to 0.10 were more substantial than just a version bump. I simply hadn't gotten around to updating things. Luckily, a contributor stepped up a few weeks ago to fix this, and I got around to merging it today. nixos/nixpkgs@d892551a834f826875d892519375c480b7fa7de3 (in the master branch) has been applied and bumps vector to 0.10.