Nixpkgs: fetchurl for private resources

[bobvanderlinden]

Issue description

When using Nix in a company, some resources are needed to be retrieved from private webservers. These require (for instance) an Authentication header. Currently I don't see a trivial way to make such build processes seamless:

• There is fetchurl, but adding a username and password to each url altering the build process.
• There is requireFile, which does not actually download files at all. This makes the process very cumbersome as each new version of each application needs to be re-prefetched manually.

For Git, there is fetchgitPrivate, but it seems that the general consensus is that it is hard to setup.

What options are available (or what options need to be created) for retrieving files from private resources?

[orivej]

Do you have a continuous integration system that knows how to download sources from private webservers? How does it do so?

Adding a username and password to each url does not cause rebuilds as long as the output name is the same, which can be set in the name attribute of fetchurl and other fetchers.

If you can not store credentials in the nix store but also want sandboxed builds, the only option I know is impureEnvVars (see Nix manual and examples in Nixpkgs).

You may also set up a proxy that will add authentication to the unauthenticated requests and enable it with http_proxy env var.

[bobvanderlinden]

Thanks a lot for the extensive reply!

Yes, CI has set up tokens, but developers have personal tokens. CI has environment variables. Most developers use package-manager-specific configuration files (like ~/.m2/settings.xml).

Setting up a proxy seems like defeating the purpose of personal tokens, so I'm guessing I can't push such a thing into a company. However, the impureEnvVars is an interesting one. Would it be possible to use that to pass along an Authentication header or -u username:password?

In addition, as there are multiple different sources where files need to be retrieved from, the authentication information would need to be set based on the domain-name of the url being retrieved. From what I gather from impureEnvVars in fetchurl, this is not available yet.

https://github.com/NixOS/nix/blob/8efe937a350eca57cbc3cb9837f28d985e0d281f/corepkgs/fetchurl.nix#L25-L31

Is that correct? If so, would adding something alike to nixpkgs be worth the time?

[orivej]

fetchurl accepts netrcPhase which is responsible for creating a netrc file in the form:

machine github.com
  login username
  password password1
machine github.company.com
  login corporate
  password password2

and netrcImpureEnvVars which lists environment variables of the machine starting the build that will be accessible to the netrcPhase. You can build your authenticating fetcher from this. The only restriction is that if some users want to build in the sandbox, all authenticating information must be available in the environment variables as sandbox will not let you access arbitrary files.

[orivej]

However, I guess you can try to fallback to parsing ~/.m2/settings.xml if the environment variables are not set for convenience of the users of single user unsandboxed Nix…

[bobvanderlinden]

Thanks for suggesting netrc! It seems to be working nicely by placing a netrc file next to default.nix.

That's quite a hidden feature, so I dug around and found the original PR for anyone wanting to know more: https://github.com/NixOS/nix/pull/1215

[orivej]

The issue you found is about the fetcher built into Nix. It is used only for fetching from binary caches and for bootstrapping Nixpkgs. The netrcPhase and netrcImpureEnvVars used for fetching of the original sources were introduced in b71b7ee62234dbf1863e9595bfeed961e951e491.

[bobvanderlinden]

Hmm, yes I noticed it not working (worked before due to cache I'm guessing).

The code I came up with:

    fetchurl_private = opts: pkgs.fetchurl (opts // {
      netrcPhase = ''
        if [ -z "''$NIX_JFROG_USERNAME" -o -z "''$NIX_JFROG_PASSWORD" ]; then
          echo "Error: NIX_JFROG_USERNAME and NIX_JFROG_PASSWORD environment variables not set" >&2
          exit 1
        fi
        cat > netrc <<EOF
        machine ***
                login ''$NIX_JFROG_USERNAME
                password ''$NIX_JFROG_PASSWORD
        EOF
      '';
      netrcImpureEnvVars = [ "NIX_JFROG_USERNAME" "NIX_JFROG_PASSWORD" ];
    });

Using it in buldMaven:

    pkgs = import <nixpkgs> {
      config.packageOverrides = pkgs: {
        buildMaven = pkgs.buildMaven.override {
          fetchurl = fetchurl_private;
        };
      };
    };

Then stopping nix-daemon. I'm under OSX, so:

sudo launchctl stop org.nixos.nix-daemon

And starting the daemon manually:

sudo NIX_JFROG_USERNAME=*** NIX_JFROG_PASSWORD=*** /nix/store/4j9jacx8mjd4jlj53wvymyhxq7dqyj5d-nix-1.11.14/bin/nix-daemon

This seems to be working. It is quite a chore though. Instructing others to do the same would be less than ideal.

Is it viable to have a global netrc file, configured in nix.conf, that nix-daemon picks up and passes to every fetchurl call (maybe have the contents of the netrc file be encoded as env variables, so that it can be passed to sandboxes more easily)? If so, I could look into working on such a feature.

[orivej]

AFAIK the variables do not have to be set before launching nix-daemon: nix-build/nix-env/etc should pass them to the daemon.

Is it viable to have a global netrc file, configured in nix.conf, that nix-daemon picks up and passes to every fetchurl call

This is not exactly what you are asking for, but nix.conf has sandbox-paths which extends the solution with reading from file to users with sandbox. People may set NIX_PATH=netrc=/path/to/netrc, add /path/to/netrc to sandbox-paths if they are building in the sandbox, and your fetcher will be able to copy netrc from <netrc>.

[bjornfor]

A variant of that is to use curlOpts:

  myFetcher = url: sha256: fetchurl {
    inherit url sha256;
    curlOpts = "--netrc-file /etc/nix/netrc";
  };

while also having nix.sandboxPaths = [ "/etc/nix/netrc" ] in configuration.nix. (I use that at $DAYJOB.)

[bobvanderlinden]

Yay! That is working mighty well! :D Thanks for the suggestion.

I have overridden fetchurl with the fetchurl that uses netrc like this:

     packageOverrides = pkgs: rec {
        fetchurlPrivate = opts: pkgs.fetchurl (opts // {
          curlOpts = "${pkgs.lib.optionalString (opts ? curlOpts) "${opts.curlOpts}"} --netrc-file /etc/nix/netrc";
        });
        fetchurl = fetchurlPrivate;
      };

And have the following in nix.conf:

build-sandbox-paths = /etc/nix/netrc
netrc-file = /etc/nix/netrc

I feel like something like this should be documented on the wiki. A new wiki-page with in-company usage information might be in order. Agreed?

[bobvanderlinden]

I forgot about this issue, but ran into the same problem again today. I've created a new wiki page that includes the methods above as well as some other information: https://nixos.wiki/wiki/Enterprise