Dvc: Environment variables in `.dvc/config`

[dsuess]

Bug Report

In DVC 1.9 (and many versions prior to that) it was possible to use environment variables in the config. One useful application was to setup the credentialpath in a user-independent way like this:

credentialpath = $HOME/.aws/credentials

However, with 1.10 this doesn't work anymore and dvc push cancels with the following error:

2020-12-02 18:52:12,508 ERROR: failed to push data to the cloud - Unable to find AWS credentials. <https://error.dvc.org/no-credentials>: Unable to locate credentials

Please provide information about your setup

Output of dvc version:

DVC version: 1.10.2 (pip)
---------------------------------
Platform: Python 3.7.9 on Linux-4.15.0-109-generic-x86_64-with-debian-buster-sid
Supports: http, https, s3
Cache types: hardlink, symlink
Caches: local
Remotes: s3
Repo: dvc (subdir), git

Additional Information (if any):

If applicable, please also provide a --verbose output of the command, eg: dvc add --verbose.

[skshetry]

Most likely it's #4802. @dsuess, I don't think it was a supported feature.
A workaround that might work is using a relative path from the config location. Would that work for you?

[dsuess]

Thank's for your quick reply! Unfortunately, relative paths won't work for us. To give a bit more detail: We share repos with DVC between multiple users. We have a convention that everyone should have a certain named profile in their credentials file, to access the shared DVC remote bucket. This way, we don't have to commit sensitive information to a repo, users can push/pull to the bucket without any further setup, and we don't rely on 3rd party tools to manage credentials on a per-project basis. Since we can't tell users how to organize their home directory, we can't rely on relative paths.

If full variable substitution is too powerful for these use cases, maybe we can use expanduser to allow adding things like

credentialpath = ~/.aws/credentials

to such a repo. Another alternative would be to give users a global config to set these options?

[johnnychen94]

In case you haven't try it out yet; you can set credential informations in ~/.config/dvc/config file:

['remote "lflab"']
    url = s3://johnnychen/dvc
    endpointurl = http://oss.example.com
    access_key_id = <myaccount>
    secret_access_key = <mypassword>

[shcheklein]

@dsuess

In most case users do not need to setup credentialpath . If it is in it's default location, boto (the lib we use to access S3 or S3-compatible storages) will read them automatically.

In case someone needs to set a custom credentialpath w/o affecting others, they could utilize --global, --system, --local options for the dvc remote modify command . You can find more information on this logic here - https://dvc.org/doc/command-reference/config

Please, take a look and let us know if that works for you. And sorry for breaking things for you, I hope we'll find a good way for you to achieve what you want.

[dsuess]

Thanks for all the suggestions. I think the global options could work, but would be a lot of duplication (since we have multiple DVC remotes).

Also, I am not sure why we had to set this option in the first place, since we use the standard location of the credentials file, so boto should just find it. I'll close this issue since it's clear that this behavior wasn't intended from the beginning and open a new issue in case the standard location discovery doesn't work. Thanks everyone for your help.