Nixpkgs: vmTools: support specifying multiple repositories

[FRidh]

Issue description

The distro's listed in vmTools currently take a single list of packages as arguments. It should support multiple so that we can include other packages as well. This is needed in order to build a Nix RPM (https://github.com/NixOS/nix/pull/1141#issuecomment-347374010) on CentOS.

[FRidh]

cc @therealpxc @bhipple

I was actually looking at this during the weekend, because I also want to build an RPM for on CentOS. If I understood the code correctly, it uses import from derivation to build the package list. I think we could write a script that merges the xml files and wrap that in a Nix function.

[FRidh]

Another option, but requiring more work, is having a maintainer script that parses the XML and creates JSON or a Nix expression listing per package its dependencies and source. We could create for each XML file such a file, and then use a fixed-point combinator to merge the "lists".

[therealpxc]

Thanks for creating an issue on the appropriate repo for this.

Neither option you've presented sounds bad to me, but given that it is more work, what do you see as the main advantage of the second option? That the same expressions could be applied for other Nix-y purposes not yet anticipated?

[FRidh]

The latter option prevents building during evaluation.

How RPM deals with dependencies isn't trivial though. Packages don't necessarily directly depend on other packages, but instead they specify which files they require and which files they provide. Based on that one can find multiple packages that fulfill requirements.

[FRidh]

Turns out that the current implementation already supports using multiple package lists.

diff --git a/pkgs/build-support/vm/default.nix b/pkgs/build-support/vm/default.nix
index e31f513c666..50615f025b0 100644
--- a/pkgs/build-support/vm/default.nix
+++ b/pkgs/build-support/vm/default.nix
@@ -1303,17 +1303,33 @@ rec {
       packages = commonCentOSPackages ++ [ "procps-ng" ];
     };

-    centos74x86_64 = rec {
+    centos74x86_64 = let
+      # N.B. Switch to vault.centos.org when the next release comes out
+      lists = [
+        rec {
+          urlPrefix = http://mirror.centos.org/centos-7/7.4.1708/os/x86_64;
+          packagesList = fetchurl {
+            url = "${urlPrefix}/repodata/b686d3a0f337323e656d9387b9a76ce6808b26255fc3a138b1a87d3b1cb95ed5-primary.xml.gz";
+            sha256 = "1mayp4f3nzd8n4wa3hsz4lk8p076djkvk1wkdmjkwcipyfhd71mn";
+          };
+        }
+        rec {
+          urlPrefix = http://dl.fedoraproject.org/pub/epel/7/x86_64;
+          packagesList = fetchurl {
+            url = "${urlPrefix}/repodata/e1e16427f1cfc8a2b8d483e3298743d5eff22064be1e31501725dee272a8117c-primary.xml.gz";
+            sha256 = "0z0im1rf5pi52x8327mychhg5vym8f3jkqw3sjwa5j6gy4kn9qg1";
+          };
+        }
+      ];
+      urlPrefixes = lib.catAttrs "urlPrefix" lists;
+      packagesLists = lib.catAttrs "packagesList" lists;
+    in rec {
       name = "centos-7.4-x86_64";
       fullName = "CentOS 7.4 (x86_64)";
-      # N.B. Switch to vault.centos.org when the next release comes out
-      urlPrefix = http://mirror.centos.org/centos-7/7.4.1708/os/x86_64;
-      packagesList = fetchurl {
-        url = "${urlPrefix}/repodata/b686d3a0f337323e656d9387b9a76ce6808b26255fc3a138b1a87d3b1cb95ed5-primary.xml.gz";
-        sha256 = "1mayp4f3nzd8n4wa3hsz4lk8p076djkvk1wkdmjkwcipyfhd71mn";
-      };
+      inherit urlPrefixes;
+      inherit packagesLists;
       archs = ["noarch" "x86_64"];
-      packages = commonCentOSPackages ++ [ "procps-ng" ];
+      packages = commonCentOSPackages ++ [ "procps-ng" "libsodium-devel" ];
     };
   };

Unfortunately the gcc in CentOS 7.4 is too old to build Nix master:

+ /usr/bin/make
  GEN    Makefile.config
  CC     src/linenoise/linenoise.o
  CXX    src/nix/add-to-store.o
g++: error: unrecognized command line option '-std=c++14'
make: *** [src/nix/add-to-store.o] Error 1
error: Bad exit status from /var/tmp/rpm-tmp.IDua0c (%build)


RPM build errors:
    Bad exit status from /var/tmp/rpm-tmp.IDua0c (%build)
[  233.129985] reboot: Power down
builder for '/nix/store/871gkdl8qcl9xfnx31k6pkvclrxcs3hq-nix-rpm-centos-7.4-x86_64-1.12pre1234.abcdef.drv' failed with exit code 1
error: build of '/nix/store/871gkdl8qcl9xfnx31k6pkvclrxcs3hq-nix-rpm-centos-7.4-x86_64-1.12pre1234.abcdef.drv' failed

[dtzWill]

gcc from the devtoolset-7 is 7.2.1 and can be used on CentOS 7.4:

https://www.softwarecollections.org/en/scls/rhscl/devtoolset-7/

Which I think could be grabbed using something like:

http://mirror.centos.org/centos/7/sclo/x86_64/rh/repodata/4fa6a0369ce6428286d2f663789abaf1c9bb44abbee8a12da7593e64335da666-primary.xml.gz

Two notes, though:

• Will need to wrap in 'scl enable devtoolset-7 bash` or adjust PATH to include devtools paths (/opt/something/...)
• Looking at listing in that directory it looks like these only live for a few months, maybe? I'm not sure of all the details but this might require periodic maintenance or impact long-term reproducibility.

EDIT: Whoops looks like all of this is mentioned over on https://github.com/NixOS/nixpkgs/issues/32900#issue-283604597 !