Pydantic: Is it possible to have arbitrary key names in pydantic?

[securisec]

I am trying to emulate a similar behavior to typescripts interface with arbitrary key names for a pydantic model, but am running in to some issues.

Consider the following in TS:

export interface SNSMessageAttributes {
    [name: string]: SNSMessageAttribute;
}

Is it possible to achieve this in pydantic?

Here is my python example code:

from pydantic import BaseModel, parse_obj_as
from typing import Optional

class Values(BaseModel):
    Value: str
    Type: str

class MessageAttributes(BaseModel):
    ArbitraryKey: Optional[Values] # question is how can i make ArbitraryKey truly arbitrary so that anything passed here will still have the values typed to Values?


class Mymodel(BaseModel):
    MessageAttributes: Optional[MessageAttributes]


data = {"MessageAttributes": {"ArbitraryKey": {"Type": "String", "Value": "TestString"}}}

parsed = parse_obj_as(Mymodel, data)

print(parsed.MessageAttributes.ArbitraryKey.Value)
# TestString

In this example, the property ArbitraryKey can be anything. But I cant possibly hardcode all the possible key names there could be. For instance, instead of ArbitraryKey, what if the key name was SomeTestKey?

I know I can use extra = 'allow in Config, but that wouldnt give the dot syntax that I can get when using parse_obj_as

My question is, if possible, can I do something similar to [name: string] in pydantic for property names?

[PrettyWood]

Hello @securisec
I just looked at the TS interface and wrote it with pydantic. Is it enough to guide you ?

from typing import Dict

from pydantic import BaseModel


class SNSMessageAttribute(BaseModel):
    Value: str
    Type: str


class SNSMessageAttributes(BaseModel):
    __root__: Dict[str, SNSMessageAttribute]

good_data = {
  'a': { 'Value': 'va', 'Type': 'ta' },
  'b': { 'Value': 'vb', 'Type': 'tb' },
}

bad_data = {
  'a': { 'Value': 'va', 'Type': 'ta' },
  'b': { 'Value': 'vb' },
}

SNSMessageAttributes.parse_obj(good_data)
# ok !

SNSMessageAttributes.parse_obj(bad_data)
# pydantic.error_wrappers.ValidationError: 1 validation error for SNSMessageAttributes
# __root__ -> b -> Type
#   field required (type=value_error.missing)

[securisec]

Thanks @PrettyWood. This is great!

So following the example, the parsing is done correctly, but accessing it will throw it off.

For example, print(parsed.MessageAttributes['ArbitraryKey'].Value) will lead to TypeError: 'MessageAttributes' object is not subscriptable

and doing print(parsed.MessageAttributes.dict()['ArbitraryKey']) will throw KeyError: 'ArbitraryKey'.

How do I overcome this to maintain the dot syntax, and/or access the data without first converting the whole thing back to a dict?

Full code and output

from pydantic import BaseModel, parse_obj_as
from typing import Dict, Optional

from pydantic import parse

class Values(BaseModel):
    Value: str
    Type: str

class MessageAttributes(BaseModel):
    __root__: Optional[Dict[str, Values]]


class Mymodel(BaseModel):
    MessageAttributes: Optional[MessageAttributes]


data = {"MessageAttributes": {"ArbitraryKey": {"Type": "String", "Value": "TestString"}}}

parsed = Mymodel.parse_obj(data)

print(parsed)
print(parsed.MessageAttributes)

MessageAttributes=MessageAttributes(__root__=None)
__root__=None

based on the

[PrettyWood]

@securisec
Yes it's normal. Everything is in the __root__ so by default you need to do parsed.MessageAttributes.__root__['ArbitraryKey'].Value.

But you could add some helpers! With my previous example you can do.

class SNSMessageAttributes(BaseModel):
    __root__: Dict[str, SNSMessageAttribute]

    def __getattr__(self, item):  # if you want to use '.'
        return self.__root__[item]

parsed = SNSMessageAttributes.parse_obj(good_data)
print(parsed.a.Value)
# va

Careful though as it can conflict with other attributes. If you have for example a key called Config, it will return the pydantic Config and not the field you expect.

A safer solution would be

class SNSMessageAttributes(BaseModel):
    __root__: Dict[str, SNSMessageAttribute]

    def __getitem__(self, item):  # if you want to use '[]'
        return self.__root__[item]

parsed = SNSMessageAttributes.parse_obj(good_data)
print(parsed['a'].Value)
# va

[securisec]

Understood. This really is super helpful. Learning a lot about pydantic internals via this conversation. I guess my issue is, following the example, I keep getting a None value for __root__. Here is the code:

from pydantic import BaseModel, parse_obj_as
from typing import Dict, Optional

from pydantic import parse

class Values(BaseModel):
    Value: str
    Type: str

class MessageAttributes(BaseModel):
    __root__: Optional[Dict[str, Values]]
    def __getattr__(self, item):  # if you want to use '.'
        return self.__root__[item]


class Mymodel(BaseModel):
    MessageAttributes: Optional[MessageAttributes]


data = {"MessageAttributes": {"ArbitraryKey": {"Type": "String", "Value": "TestString"}}}

parsed = Mymodel.parse_obj(data)

print(parsed)
print(parsed.MessageAttributes) 

# MessageAttributes=MessageAttributes(__root__=None)
# __root__=None

What I am doing wrong?

[PrettyWood]

__root__ is mostly used for top level in the model. So when we use parse_obj, we actually check if there is __root__ to still work without it.
But in your case it's a submodel so to work you should either

• add __root__ in the data: data = {"MessageAttributes": {"__root__": "ArbitraryKey": {"Type": "String", "Value": "TestString"}}}}, which is ugly
• remove the submodel, which is useless

class Values(BaseModel):
    Value: str
    Type: str


class Mymodel(BaseModel):
    MessageAttributes: Optional[Dict[str, Values]]


data = {"MessageAttributes": {"ArbitraryKey": {"Type": "String", "Value": "TestString"}}}
parsed = Mymodel.parse_obj(data)

print(parsed)
# MessageAttributes={'ArbitraryKey': Values(Value='TestString', Type='String')}
print(parsed.MessageAttributes['ArbitraryKey'].Value)
# TestString