Pydantic: Read values from secrets files when using BaseSettings

[ricardogsilva]

Feature Request

Output of python -c "import pydantic.utils; print(pydantic.utils.version_info())":

pydantic version: 1.4
pydantic compiled: True
install path: /home/likeno/.cache/pypoetry/virtualenvs/geospaca-yth2PwXS-py3.7/lib/python3.7/site-packages/pydantic
python version: 3.7.6 (default, Dec 28 2019, 05:50:43)  [GCC 8.3.0]
platform: Linux-5.3.0-40-generic-x86_64-with-debian-10.2
optional deps. installed: ['typing-extensions']

I'm using pydantic to provide an easy way to read my app's settings from the environment. Using BaseSettings it works great.

Now I'm moving to a setup where some sensitive settings are being managed as docker secrets and I'd like to have support for those baked into pydantic.BaseSettings too.

Basically this means BaseSettings would need to be able to read values from individual files, getting the name of the variable from the filename and its value from the file's contents.

I've actually implemented this is a derived class already, but would like to know if there is interest on incorporating it on pydantic.

Code looks like this:

# this is my settings file (it's actually a django settings file so I'm 
# showing only relevant stuff)
import pydantic


class MyConfig(pydantic.BaseSettings):
    rabbitmq_url: pydantic.AnyUrl = "amqp://guest:[email protected]:5672"

    class Config:
        env_prefix = "django_"

    def __init__(
            __pydantic_self__,
            *args,
            secrets_directory: typing.Optional[typing.Union[str, Path]] = None,
            **kwargs
    ):
        if secrets_directory is not None:
            extra_values = __pydantic_self__._scan_secrets_dir(
                Path(secrets_directory).expanduser())
            modified_values = __pydantic_self__._modify_secrets(extra_values)
            super().__init__(*args, **modified_values, **kwargs)
        else:
            super().__init__(*args, **kwargs)

    def _scan_secrets_dir(
            self,
            secrets_directory: Path
    ) -> typing.Dict[str, str]:
        result = {}
        if secrets_directory.is_dir():
            for item in secrets_directory.iterdir():
                if item.is_file():
                    contents = item.read_text().strip()
                    result[item.name] = contents
        return result

    def _modify_secrets(self, secrets: typing.Dict[str, str]):
        result = {}
        for name, value in secrets.items():
            if self.__config__.case_sensitive:
                new_name = name
            else:
                new_name = name.lower()
            new_name = new_name.replace(self.__config__.env_prefix, "")
            result[new_name] = value
        return result


_conf = MyConfig(secrets_directory="/run/secrets")

# now here come all of the usual django settings
# ...

RABBITMQ_URL = _conf.rabbitmq_url

In the snippet above, MyConfig gets the value for rabbitmq_url from the /run/secrets/DJANGO_RABBITMQ_URL file.

The code above seems to work well:

• some variables are gotten from the /run/secrets directory
• some are gotten from the environment

If this feature is something you'd be interested in incorporating into pydantic, I could make a PR with this code and some appropriate tests.

[samuelcolvin]

Happy to consider it, but only if others are keen. Let's see how many :+1: this issue gets.

[ricardogsilva]

I'll kick it off with my own :+1: - I hope it is not cheating :smile:

[vincenzoromeo]

Thank you so much @ricardogsilva

You code has been an invaluable example of how to inject outside sources of secrets into pydantic's settings class and has allowed me to extrapolate to using AWS Parameter Store. I couldn't find anything like this on my interwebs search.

I'm all for supporting your PR. At the very least, your code should be in the documentation to enable and encourage others to leverage third-party secrets management systems.

Much kudos to you, and the pydantic team :+1:

[ricardogsilva]

@samuelcolvin

I guess I'll go ahead and make a PR for this then - might take me a while though, I'm a bit swamped with other stuff due to the whole covid-19 situation at the moment. Just to let you know I'm still up for it.

[samuelcolvin]

agreed, thanks so much.

I'm a bit swamped with other stuff due to the whole covid-19 situation at the moment

me too!, tell me about it. No hurry, I'll review the PR when you have time to submit it, I'm already way behind on reviewing PRs 😨 🙏 🙇 .

[HacKanCuBa]

+1 to incorporate reading Docker secrets into BaseSettings
+1 to document it like above anyway so have it as an example that allows further extensibility

:smiley:

[FichteFoll]

I took a slightly different approach and implemented it as a subclass that my settings classes can then subclass themselves.

from pathlib import Path
from typing import Dict, Optional, Union

from pydantic.env_settings import BaseSettings, SettingsError


class SecretsSettings(BaseSettings):
    """Settings model that additionally reads values from a secrets directory.

    Configurable through Config.secrets_path, defaults to "/run/secrets".
    """

    def __init__(__pydantic_self__, *args, _secrets_path: Union[str, Path, None] = None, **kwargs):
        secrets_path = _secrets_path or __pydantic_self__.__config__.secrets_path
        if secrets_path is not None:
            path = Path(secrets_path).expanduser()
            secrets_dict = __pydantic_self__._scan_secrets_dir(path)
            kwargs = {**secrets_dict, **kwargs}
        # Currently, secrets override values from env
        super().__init__(*args, **kwargs)

    def _scan_secrets_dir(self, secrets_path: Path) -> Dict[str, str]:
        d: Dict[str, str] = {}

        if not secrets_path.is_dir():
            return d

        secret_paths = {
            (item.name if self.__config__.case_sensitive else item.name.lower()): item
            for item in secrets_path.iterdir()
            if item.is_file()
        }

        for field in self.__fields__.values():
            value: Optional[str] = None
            for env_name in field.field_info.extra['env_names']:
                if path := secret_paths.get(env_name):
                    if (value := path.read_text().strip()) is not None:
                        break
            else:
                continue

            if field.is_complex():
                try:
                    value = self.__config__.json_loads(value)  # type: ignore
                except ValueError as e:
                    raise SettingsError(f'error parsing JSON for "{env_name}"') from e

            if value is not None:
                d[field.alias] = value

        return d

    class Config:
        secrets_path = "/run/secrets"


class TestSettings(SecretsSettings):
    password: str

    class Config:
        env_prefix = "test_"

# reads from "/run/secrets/test_password"
print(TestSettings().password)
assert TestSettings(password="abc").password == "abc"

I can submit this as a PR, if desired. I only need to know whether the default class should be extended with this and whether environment variables or secrets should take priority.
(And I should probably be using deep_update for the kwargs, but this fulfills my needs.)

[freis]

@FichteFoll I see your comment about "secrets override values from env", I was wondering on how you would achieve the opposite.

[mdgilene]

Seems to me that the precedence order should be the following to keep with the current expected behavior.

1. arguments to __init__
2. environment variables
3. values defined in .env files
4. values defined in secret_files
5. defaults in the class definition

Also would it make sense to allow for secret_files to be a list of paths to search? Could be cases where secrets are stored in multiple places i.e. /var/run/secrets, ./secrets, etc.

[freis]

Wouldn't make more sense letting people decide on how the expected behavior should be? Like I see the use for a default case, but it wouldn't also make sense to let me choose to change how the priorities should be? (not sure on how hard it would be... but it would be nice at least)

[mdgilene]

There is no ability (as far as I am aware) to configure the priority order for regular environment variables so I don't see why there would be one for this as well. If that is something people do want it would probably be it's own feature request.

[FichteFoll]

@FichteFoll I see your comment about "secrets override values from env", I was wondering on how you would achieve the opposite.

I don't have a concrete plan for this. However, it would require a different injection method than extending the kwargs of __init__, which was a pretty convenient place to do so, since it also accepts _secrets_path. Because it was easier to implement and I didn't need environemtn variables to take priority, I went with that route.

If that is something people do want it would probably be it's own feature request.

I agree with this. Go with the sanest default; reconsider if different needs arise and make it configurable somehow, if necessary.

[ricardogsilva]

I'm currently very tight on my work schedule in order to PR this feature - in the way I had envisioned, which is admittedly more complex than the simple example I gave at the top.

Please feel free to take over it, as I see there is some interest in getting something like this in pydantic.

[mdgilene]

I have gone ahead and tried my hand at implementing this feature based on the discussion here. It is largely based on the work of @FichteFoll with a few modifications.

[PrettyWood]

Closed by #1820. Feel free to reopen if anything has been missed