Pydantic: Custom root types

[jhonabreul]

Question

First, thank you for pydantic. I have saved lots of coding hours since I found it. So yeah, thanks!

As for my question, can I force the root to be of type "array" when generating a schema?

• OS: Fedora 29
• Python version: 3.7.1
• Pydantic version: 0.24

I need to provide a json schema file for the users to know how to create a json file for processing. The thing is, the json file MUST contain an array of users, that is, it cannot be a json object {} but an array []. Below there is an example file:

[
  {
    "name": "some name"
  },
  {
    "name": "another name"
  },
  ...
]

I have this litte sample model:

from typing import List
from pydantic import BaseModel, validator

class User(BaseModel):
  name = 'John Doe'

class Users(BaseModel):
  users: List[User]

  @validator('users', whole=True)
  def validateEmptyList(cls, value):
    if not value:
      raise ValueError('users list cannot be empty')

    return value

json_data = [
  {'name': 'some name'},
  {'name': 'another name'}
]

users = Users(users = json_data)
print(users.schema_json(indent=2))

This would print the following schema:

{
  "title": "Users",
  "type": "object",
  "properties": {
    "users": {
      "title": "Users",
      "type": "array",
      "items": {
        "$ref": "#/definitions/User"
      }
    }
  },
  "required": [
    "users"
  ],
  "definitions": {
    "User": {
      "title": "User",
      "type": "object",
      "properties": {
        "name": {
          "title": "Name",
          "default": "John Doe",
          "type": "string"
        }
      }
    }
  }
}

This schema tells the user that the file should contain something like this:

{
  "users" : [
    {
      "name": "some name"
    },
    {
      "name": "another name"
    },
    ...
  ]
}

Is there a way to create a model that tells the user that root type should be array which items should be "user" json objects?

Thank you very much in advance!

[samuelcolvin]

Not that I know of. Is this valid for jsonschema?

[samuelcolvin]

@tiangolo is this valid in jsonschema?

If not I don't think there's anything we can do to help.

[tiangolo]

Thanks for tagging me @samuelcolvin.

It is valid in JSON schema to declare something of any valid JSON type (object, array, number, etc).

Currently there's no way to declare a Pydantic model as a list (array).

But you can declare a Pydantic model's attribute/field as a list (array) of something. Like in users: List[User].

So, you could use Pydantic's internals/utils to declare what you need.

I was going to give you some hints, but I ended up implementing it:

from typing import List
import json
from pydantic import BaseModel, BaseConfig, Schema
from pydantic.fields import Field
from pydantic.schema import field_schema, get_model_name_map, get_flat_models_from_field


class User(BaseModel):
    name: str = "John Doe"


json_data = [{"name": "some name"}, {"name": "another name"}]

users_field = Field(
    name="name",
    type_=List[User],
    class_validators={},
    model_config=BaseConfig,
    default=[],
    required=False,
    schema=Schema(None),
)

users, errors = users_field.validate(json_data, values={}, loc="")

models = get_flat_models_from_field(users_field)
model_name_map = get_model_name_map(models)
users_schema, additional_definitions = field_schema(
    users_field, model_name_map=model_name_map
)
global_schema = {"definitions": additional_definitions, **users_schema}
print(json.dumps(global_schema, indent=2))
print(users)
print(errors)

That prints:

{
  "definitions": {
    "User": {
      "title": "User",
      "type": "object",
      "properties": {
        "name": {
          "title": "Name",
          "default": "John Doe",
          "type": "string"
        }
      }
    }
  },
  "title": "Name",
  "default": [],
  "type": "array",
  "items": {
    "$ref": "#/definitions/User"
  }
}
[<User name='some name'>, <User name='another name'>]
None

BTW, that output is valid JSON Schema.

---

Now, if by any chance you are using this with FastAPI, it takes, as body parameter declaration, a Python type that is converted to a Pydantic Field.

And the same for responses with response_model.

So, you could write an API like:

from typing import List
from fastapi import FastAPI
from pydantic import BaseModel


class User(BaseModel):
    name: str = "John Doe"


app = FastAPI()


@app.post("/users/", response_model=List[User])
async def create_users(users: List[User]):
    return users

And that would give you a full OpenAPI with JSON Schemas and included dynamic docs showing the correct type for users (a list/array of User).

[samuelcolvin]

@tiangolo that's very cool, thank you.

@jhonabreul, note that Field is not considered an external interface for pydantic so might change at any time.

@tiangolo I wonder if we could do anything to make this easier to use, both in fastapi and more widely?

My immediate thought is that it would be better to define something like:

class Users(BaseModel):
    __root__: List[User]

Then schema could return the scheme for Users.__root__ rather than for User and __init__ extra could be modified to only take one argument which is __root__.

Does that sound sensible?

I guess __root__ and can be anything so this would work not just for lists.

[tiangolo]

Nice! I think that would be a great addition.

Also, it could use the same trick of using *args in __init__ and taking the first item from it as in the PR for arbitrary instances. So, the first item of *args would be a list itself.

That way, although using __root__ internally, it wouldn't need to expose a parameter __root__ in __init__.

And I think it wouldn't even clash with the arbitrary classes PR, as that would be for objects with attributes, not sequences.

As both use cases are mutually exclusive (I think), the same *args could be used by both.

Should we wait for Cython and/or that PR first?

[samuelcolvin]

yes, let's wait for #548.

I agree about __init__, should have explained that, but you got what I meant.

[koxudaxi]

@samuelcolvin Thank you for your excellent work.
I have checked #548 is merged.
I'm trying to implement the "Custom root type" .

However, I have a question for this feature.

I think if a model has custom root, then the model should behave as "List"
It's mean the model should have interfaces for collections.abc.MutableSequence https://docs.python.org/3/library/collections.abc.html#collections.abc.MutableSequence

What do you think about this concept?
Could you please check the example for the feature?

from typing import List
from pydantic import BaseModel

class User(BaseModel):
    name: str = "John Doe"


class Users(BaseModel):
    __root__: List[User]


users = Users(*[User(), User(name='some_name')])  # or Users(__root__=[User(), User(name='some_name')])

print(f'users -> {users}')

for user in users:
    print(f'user in for loop -> {user}')

print(f'users[1] -> {users[1]}')

users[0] = User(name='another name')

del users[1]

result

users -> Users [<User name='John Doe'>, <User name='some_name'>]
user in for loop -> User name='John Doe'
user in for loop -> User name='some_name'
users[1] -> User name='some_name'

@tiangolo
Thank you for FastAPI.
If you have another idea, then tell it to me. I can not understand your thinking about the feature.

[samuelcolvin]

Hi custom root types aren't implemented yet, #548 was just a blocker, not the implementation.

The model will still look like a model, that can't be changed since the root type could be a list or a number or whatever.

[koxudaxi]

@samuelcolvin

Hi custom root types aren't implemented yet, #548 was just a blocker, not the implementation.

Sorry, I know it. I wanted to tell you that I want to implement the custom root types. And I will send you the pull request for the custom root types. I know you are very busy.

[samuelcolvin]

I agree with you concept, except to avoid confusion we shouldn't change __iter__ or __itemgetter__.

[koxudaxi]

Thank you. I understand it.
I have known you except any type include "List" for custom root types.
I will try it.

[tiangolo]

Custom __root__ types are now part of Pydantic: https://pydantic-docs.helpmanual.io/#custom-root-types

I think this issue can be closed :tada:

[koxudaxi]

I agree!
I just remembered this issue.

@samuelcolvin
There are a few issues which should be closed like this issue.
I think better to close the issues before release version 1.

[spolloni]

@samuelcolvin @tiangolo

and __init__ extra could be modified to only take one argument which is __root__.

That way, although using __root__ internally, it wouldn't need to expose a parameter __root__ in __init__.

Was this idea abandoned?