nix-shell and nix-store unable to access private S3 bucket

[Ralith]

WIth substituters = s3://bucket-name and trusted-public-keys = cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY= my-key-name:my-key=, nix copy --from s3://bucket-name /nix/store/foo works, but nix-store -r /nix/store/foo fails with:

warning: AWS error checking bucket 'bucket-name': Access Denied
don't know how to build these paths:
  /nix/store/foo

nix-shell produces an identical warning: AWS error checking bucket 'bucket-name': Access Denied whenever it is ran, and locally builds derivations that it should instead be fetching from the cache.

[edolstra]

If you're using the Nix daemon, does the daemon have access to the AWS credentials?

[Ralith]

The Nix daemon has not been specially configured; AWS credentials are set up in my user's ~/.aws/credentials file, and are readable only to the user. This represents an ideal user experience and seems to be enough for nix copy; I'm not sure why older commands work differently.

[edolstra]

Substitution (-r) is done by the daemon, so it needs access to the AWS credentials.

[Ralith]

So this is the currently intended behavior? It would be an awfully convenient feature for substitution to behave in a manner consistent with nix copy.

[edolstra]

It's probably better to say that it's the expected behaviour. Ideally, substitutions and fixed-output derivations would be done by the client rather than the daemon. But that would require some fairly significant re-architecting.

[copumpkin]

A slightly less invasive option would be to allow propagation of credentials through the daemon protocol, but that could result in unexpected behavior in a multi-user system: substitution or builtins.fetchurl uses one user's credentials to grab something private, it goes into store, and now is accessible by other users who shouldn't have access to it.

[Ralith]

that could result in unexpected behavior in a multi-user system

That's the current behavior if you nix copy something private into the store, so I'm not sure that would be a significant change.

[bobvanderlinden]

You might want to use netrc-file in the Nix configuration file (/etc/nix/nix.conf). That way the command-line tools as well as the daemon will use the same credentials for the builtin fetchurl. I'm not entirely sure whether the fetch that Nix is doing in your case is using netrc-file though, but it would be very helpful if all HTTPS traffic in Nix would make use of this file.

[terlar]

I created this script to be able to let the nix-daemon (builders) borrow my credentials. Since I am using a profile with temporary access key/tokens, it was not really convenient to share the environment variables themselves.

Hope it is useful to someone else:

{ pkgs, ... }:

let
  nixDaemonS3CredentialsBin = pkgs.writeShellScriptBin "nix-daemon-s3-credentials" ''
    set -euo pipefail
    ACTION=''${1:-enable}
    AWS_SOURCE=~/.aws
    AWS_TARGET=/var/secrets/aws

    if [ "$ACTION" = "enable" ]; then
      sudo ${pkgs.bindfs}/bin/bindfs -o ro \
        -g nixbld \
        -p g+rD \
        "$AWS_SOURCE" "$AWS_TARGET"

      sudo systemctl set-environment AWS_PROFILE="$AWS_PROFILE"
      sudo systemctl set-environment AWS_SHARED_CREDENTIALS_FILE="$AWS_TARGET/credentials"
      sudo systemctl restart nix-daemon
      echo "Enabled nix-daemon S3 credentials..."
    else
      sudo umount /var/secrets/aws
      sudo systemctl unset-environment AWS_PROFILE AWS_SHARED_CREDENTIALS_FILE
      sudo systemctl restart nix-daemon
      echo "Disabled nix-daemon S3 credentials..."
    fi
  '';
in {
  environment.systemPackages = with pkgs; [
    nixDaemonS3CredentialsBin
  ];
};

[nghamilton]

I was able to have some success by including the region parameter in the s3 URL: ?region= ... but that success might have been attributed entirely due to my other AWS environment settings.

[expipiplus1]

An easy way to give the daemon access to credentials is to have them in /root/.aws/credentials in the format described here https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html