Pydantic: How to use include/exclude with lists?

[parchd-1]

Question

When using include/exclude with nested structures, how do I specify what to include/exclude when the child model is a list?

Where the child is a single element and I only want the id of that element, I know from the docs that I can do:

A.dict(include={"id": ..., "child": {"id": ...}})

Where the child is a list of a model, and I only want a single element of that list, and only the id of that element, I know from reading #640 that I can do:

A.dict(include={"id": ..., "children": {0: {"id": ...}}})

But what is the syntax (assuming there is a syntax) to specify that I want all children, but only the "id" field for all children?

Also, what would be the syntax (again, assuming it is possible) to specify that for the Parent, I want all top level attributes, but only the "id" of the nested attributes?

Thank you for your help and for the library.

Please complete:

• OS: Linux
• Python version import sys; print(sys.version): 3.7.4
• Pydantic version import pydantic; print(pydantic.VERSION): 1.0

Please read the docs and search through issues to
confirm your question hasn't already been answered.

Where possible please include a self contained code snippet describing your question:

from __future__ import annotations
from typing import List, Optional

from pydantic import BaseModel, VERSION as pydantic_version
from dataclasses import dataclass


class Parent(BaseModel):
    id: int
    child: Child
    children: List[Child] = []

    class Config:
        orm_mode = True

class Child(BaseModel):
    id: int
    age: int

    class Config:
        orm_mode = True

Parent.update_forward_refs()

@dataclass
class OrmChild:
    id: int
    age: int

@dataclass
class OrmParent:
    id: int
    children: List[OrmChild]
    child: OrmChild

c1 = OrmChild(1, 12)
c2 = OrmChild(2, 7)
c3 = OrmChild(3, 2)
a = OrmParent(id=1, children=[c1, c2, c3], child=c1)

A = Parent.from_orm(a)

print(A.dict(
# WHAT GOES HERE?!
)

[samuelcolvin]

I think currently not possible.

Might be possible, but is it a common requirement?

[parchd-1]

I can only assume what other people would commonly require but I think

But what is the syntax (assuming there is a syntax) to specify that I want all children, but only the "id" field for all children?

This is something I expect people to want quite commonly - more commonly than wanting to have just the _n_th element of a list, which was one of the test cases you discussed when developing the syntax.

Also, what would be the syntax (again, assuming it is possible) to specify that for the Parent, I want all top level attributes, but only the "id" of the nested attributes?

The reason I want this is for an API, with nested models, some of which are quite large, I only want to send to the front end the top level attributes and a subset of children elements to be fetched if required. I think this is also quite commonly needed in APIs, but it is not my area of expertise.

[samuelcolvin]

The syntax would be easy, as you said you'd just specify the element you want by index.

The problem is how much more complexity it would add to the logic. Since it should be entirely backwards compatible, I'd be happy to accept a PR if the implementation is relatively streamlined.

[Lawouach]

Just to chime in on a common request, I certainly would appreciate the possibility to says something like

stuff.json(exclude={"*": {"field"}})

where the * would mean all elements of the top-level list.

[jceresini]

I'm not sure if this should be put in a separate issue, but we were able to exclude lists, though its not pretty cause we're building the exclude value to match the length of the list. It appears to be working as expected, however we get errors from mypy. Example script:

from pydantic import BaseModel
from typing import List

class Widget(BaseModel):
    foo: int
    bar: str

class LottaWidgets(BaseModel):
    name: str
    widgets: List[Widget]

widget_list = []

for i in range(10):
    widget_list.append(Widget(foo=i, bar=f'widget-{i}'))

x = LottaWidgets(name='things', widgets=widget_list)

exclude_widget_foo = {'widgets':
    {i: {'foo'} for i in range(len(x.widgets))}
}
print(x.json(
    exclude=exclude_widget_foo
))

# output > {"name": "things", "widgets": [{"bar": "widget-0"}, {"bar": "widget-1"}, {"bar": "widget-2"}, {"bar": "widget-3"}, {"bar": "widget-4"}, {"bar": "widget-5"}, {"bar": "widget-6"}, {"bar": "widget-7"}, {"bar": "widget-8"}, {"bar": "widget-9"}]}

However we're seeing the following from mypy:

foo.py:24: error: Argument "exclude" to "json" of "BaseModel" has incompatible type "Dict[str, Dict[int, Set[str]]]"; expected "Union[AbstractSet[Union[int, str]], Dict[Union[int, str], Any], None]"

This appears to be a mypy error, but I wanted to report it here as well.

Environment details:

$ cat mypy.ini
[mypy]
plugins = pydantic.mypy

$ pip freeze
mypy==0.770
mypy-extensions==0.4.3
mypy-r==0.0.0
pydantic==1.4
typed-ast==1.4.1
typing-extensions==3.7.4.1

$ python --version
Python 3.7.6

[jceresini]

https://github.com/python/mypy/issues/8554#issuecomment-600922299

This is because Dict is invariant. See https://mypy.readthedocs.io/en/stable/common_issues.html#invariance-vs-covariance. You probably want to use Mapping for argument types.

I'd be lying if I said I understood what they mean, but given that mypy says its intended behavior, does the type hinting in pydantic need to change?

[AlexECX]

1286 addresses this issue (use the "__all__" string instead of individual indexes), but excludes for sequences are modified by ValueItems so they cannot be reused.

For example, dictionaries are changed from:
{"__all__": some_excludes}
to:
{0 : some_excludes, 1 : some_excludes, ...}.

This means the same exclude dictionary or set cannot be used multiple times with different model instances unless they have the same number of items in the list. I think it would be more practical if ValueItems used a shallow copy of the items it receives, instead of directly modifying the exclude dictionary.