Pydantic: [Question] How do we parse_obj with nested lists and some keys of the json input start with underscore _?

[simkimsia]

Question

version: 1.5.1

I have this dictionary:

expected_json = {
            "_items": [
                {
                    "_id": "5eb2841ce359f3e4db95312f",
                    "Quotation": "ASP-1",
                    "Item_Status": "Submit"
                },
                {
                    "_id": "5eb2841ce359f3e4db95312g",
                    "Quotation": "ASP-2",
                    "Item_Status": "Submit"
                }
              ]
}

I have the following models:

from typing import List, Optional
from pydantic import BaseModel

class LineitemFromJson(BaseModel):
    _id: str
    Quotation: str
    Item_Status: str

class ListFromJson(BaseModel):
    _items: List[LineitemFromJson]

Then i tried this:

        p = ListFromJson.parse_obj(expected_json)
        print(p)
        line = LineitemFromJson.parse_obj(expected_json["_items"][0])
        print(line)

the line works but p doesn't work. I think it's because the parse_json expects the dict to hold the class directly rather than it being a dict that holds nested dict.

Is there a way to parse_obj that works on nested obj?

[simkimsia]

Ok I think I know why now.. u have this line https://github.com/samuelcolvin/pydantic/blob/52af9162068a06eed5b84176e987a534f6d9126a/pydantic/main.py#L175

where you reject fields from dictionaries that have fields that start with _

Is there a way to warn about this when I do parse_obj or work around this?

[simkimsia]

Ok this is my workaround

class ListFromJson(BaseModel):
    items: List[LineitemFromJson]

    # have to swap _items with items because fields that start with underscore will be ignored
    # see https://github.com/samuelcolvin/pydantic/issues/1496#issuecomment-626134257
    @classmethod
    def parse_obj(cls: Type["Model"], obj: Any) -> "Model":
        new_obj = {}
        for k, v in obj.items():
            if k.startswith("_"):
                k = k.lstrip("_")
            new_obj[k] = v
        return super().parse_obj(new_obj)

Is there a better way than this?

[koxudaxi]

@simkimsia,
You should use Field(alias="_id")
this is a better way.
https://pydantic-docs.helpmanual.io/usage/models/#model-signature

from typing import List, Optional
from pydantic import BaseModel, Field

expected_json = {
    "_items": [
        {
            "_id": "5eb2841ce359f3e4db95312f",
            "Quotation": "ASP-1",
            "Item_Status": "Submit",
        },
        {
            "_id": "5eb2841ce359f3e4db95312g",
            "Quotation": "ASP-2",
            "Item_Status": "Submit",
        },
    ]
}


class LineitemFromJson(BaseModel):
    id: str = Field(alias="_id")
    Quotation: str
    Item_Status: str


class ListFromJson(BaseModel):
    items: List[LineitemFromJson] = Field(alias="_items")


p = ListFromJson.parse_obj(expected_json)
print(p)
# items=[LineitemFromJson(id='5eb2841ce359f3e4db95312f', Quotation='ASP-1', Item_Status='Submit'), LineitemFromJson(id='5eb2841ce359f3e4db95312g', Quotation='ASP-2', Item_Status='Submit')]
line = LineitemFromJson.parse_obj(expected_json["_items"][0])
print(line)
# id='5eb2841ce359f3e4db95312f' Quotation='ASP-1' Item_Status='Submit'

[simkimsia]

Thanks @koxudaxi

It works.

Just a side note, in case future readers find this.

If you adopt the alias in the Field, this no longer works

# THIS IS WRONG!! ❌ 
p_list = PRPODataListFromPDB(items=[p1, p2]) # note the lack of _

You need to change this to

# THIS IS GOOD!! ✅ 
p_list = PRPODataListFromPDB(_items=[p1, p2]) # note the _

Because the signature of the init has changed

Then when you call the pydantic model, you need to use

# this uses the alias _items to init 
p_list = PRPODataListFromPDB(_items=[p1, p2]) # note the _
# after that you need to use items on the pydantic model
print(p_list.items)
# [LineitemFromJson(id='5eb2841ce359f3e4db95312f', Quotation='ASP-1', Item_Status='Submit'), LineitemFromJson(id='5eb2841ce359f3e4db95312g', Quotation='ASP-2', Item_Status='Submit')]