Pydantic: Using multiple BaseSettings with different prefixes

[lesnek]

Question

Hi, I have problem with joining multiple BaseSettings into one config.

First I tried:

from pydantic import BaseSettings


class KubectlSecrets(BaseSettings):
    credentials: str
    google_auth_key: str

    class Config:
        env_prefix = "K8S_SECRET_"

class Settings(KubectlSecrets):
    environment: str = "development"
    redis_db: str = "0"

    class Config:
        env_prefix = ""

But this going to overwrite prefix and I cannot find k8s secrets

Next I tried

import os
from pydantic import BaseSettings

os.environ["K8S_SECRET_CREDENTIALS"] = "dummy"
os.environ["K8S_SECRET_GOOGLE_AUTH_KEY"] = "dummy_key"
os.environ["ENVIRONMENT"] = "prod"
os.environ["REDIS_DB"] = "1"


class KubectlSecrets(BaseSettings):
    credentials: str
    google_auth_key: str

    class Config:
        env_prefix = "K8S_SECRET_"


class EnvSettings(BaseSettings):
    environment: str = "development"
    redis_db: str = "0"


class Settings(KubectlSecrets, EnvSettings):
    ...


print(KubectlSecrets())
print(EnvSettings())
print(Settings())

Output:

credentials='dummy' google_auth_key='dummy_key'
environment='prod' redis_db='1'
environment='development' redis_db='0' credentials='dummy' google_auth_key='dummy_key'

It is again overwritten by prefix and default values are used, when there is not default val it raises error:

Traceback (most recent call last):
  File "/Users/lesnek/Library/Application Support/JetBrains/PyCharm2020.1/scratches/scratch.py", line 29, in <module>
    print(Settings())
  File "pydantic/env_settings.py", line 28, in pydantic.env_settings.BaseSettings.__init__
  File "pydantic/main.py", line 338, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 1 validation error for Settings
redis_db
  field required (type=value_error.missing)

Is there some best practices to join BaseSettings with different prefixes? We want it to get secrets nicely in namespace with multiple services.
I know the solution to use Field, but then I have to use it everywhere (30+ envs) or write it as big dict of fields or rewrite codebase to uses multiple settings.

[PrettyWood]

Hello @lesnek

Currently the subclass config overwrites the parent config making Settings.Config the main config (it overwrites KubectlSecrets.Config). We could modify the code but it needs to be discussed. I don't really know what is the best default behaviour.
In the meantime you could patch the current Config to use the env settings defined directly in the model.

from pydantic.env_settings import BaseSettings


class MyConfig(BaseSettings.Config):
    @classmethod
    def prepare_field(cls, field) -> None:
        if 'env_names' in field.field_info.extra:
            return
        return super().prepare_field(field)


class KubectlSecrets(BaseSettings):
    credentials: str
    google_auth_key: str

    class Config(MyConfig):
        env_prefix = 'K8S_SECRET_'


class Settings(KubectlSecrets):
    environment: str
    redis_db: str

    class Config(MyConfig):
        env_prefix = ''

Hope it helps !

[StephenBrown2]

This is something I've run into as well. It's a tricky thing, and I don't believe there is a solution using inheritance, since that will by definition merge things before initialization. One possibility I came up with just now is to merge the dictionaries of the resulting objects and create a new one:

import os
from pydantic import BaseSettings

os.environ["K8S_SECRET_CREDENTIALS"] = "dummy"
os.environ["K8S_SECRET_GOOGLE_AUTH_KEY"] = "dummy_key"
os.environ["ENVIRONMENT"] = "prod"
os.environ["REDIS_DB"] = "1"


class KubectlSecrets(BaseSettings):
    credentials: str
    google_auth_key: str

    class Config:
        env_prefix = "K8S_SECRET_"


class EnvSettings(BaseSettings):
    environment: str = "development"
    redis_db: str = "0"


class Settings:
    __dict__ = {}
    def __init__(self, *settings):
        for s in settings:
            self.__dict__.update(s)
        for k, v in self.__dict__.items():
            setattr(self, k, v)

    def __iter__(self):
        """
        so `dict(model)` works
        """
        yield from self.__dict__.items()

print(KubectlSecrets())
print(EnvSettings())

settings = Settings(KubectlSecrets(), EnvSettings())
print(f"{settings=}")
print(f"{dict(settings)=}")
print(f"{settings.environment=}")
print(f"{settings.redis_db=}")                                                                                                                                

which prints:

credentials='dummy' google_auth_key='dummy_key'
environment='prod' redis_db='1'
settings=<__main__.Settings object at 0x10b782af0>
dict(settings)={'credentials': 'dummy', 'google_auth_key': 'dummy_key', 'environment': 'prod', 'redis_db': '1'}
settings.environment='prod'
settings.redis_db='1'

Meaning you lose some of the pydantic niceness, but can still access things as attributes on the Settings class, after validation by Pydantic happens.

EDIT: Looks like I was a little too slow, and @PrettyWood 's solution looks much better.

[lesnek]

Thank you both guys, super fast response 👌. I had similar solution as @StephenBrown2 but I like solution of @PrettyWood .
Quick solution could be "local_env_prefix" which is used only where is defined in config.

Again thanks for help 👌