Fastapi: [BUG] Use third party class as property in pydantic schema

[Certary]

Describe the bug
I have a pydantic schema that needs a third party class (bson.objectid.ObjectID) as a property. For this reason I created a custom validator and encoder as per pydantic documentation.

Code

from bson.objectid import ObjectId
from pydantic import BaseModel
from pydantic import validators
from pydantic.errors import PydanticTypeError
from pydantic.utils import change_exception

class ObjectIdError(PydanticTypeError):
    msg_template = 'value is not a valid bson.objectid.ObjectId'

def object_id_validator(v) -> ObjectId:
    with change_exception(ObjectIdError, ValueError):
        v = ObjectId(v)
    return v

def get_validators() -> None:
    yield validators.not_none_validator
    yield object_id_validator

ObjectId.__get_validators__ = get_validators

def encode_object_id(object_id: ObjectId):
    return str(object_id)

class UserId(BaseModel):
    object_id: ObjectId = None

    class Config:
        json_encoders = {
            ObjectId: encode_object_id
        }

class User(UserId):
    email: str
    salt: str
    hashed_password: str

# Just for testing
user = User(object_id = ObjectId(), email="[email protected]", salt="12345678", hashed_password="letmein")
print(user.json())
# Outputs:
# {"object_id": "5c7e424225e2971c8c548a86", "email": "[email protected]", "salt": "12345678", "hashed_password": "letmein"}

As you can see at the bottom of the code, the serialization seems to work just fine. But when I use this schema as an argument (and/or response type) in API operations and then open the automatic documentation, I get presented with an error.

Code

from bson import ObjectId
from fastapi import FastAPI
from user import User, UserId

app = FastAPI()


@app.post("/user", tags=["user"], response_model=UserId)
def create_user(user: User):
    # Create user and return id
    print(user)
    return UserId(objectId=ObjectId())

Log

INFO: ('127.0.0.1', 2706) - "GET /openapi.json HTTP/1.1" 500
ERROR: Exception in ASGI application
Traceback (most recent call last):
  File "<project-path>\venv\lib\site-packages\uvicorn\protocols\http\h11_impl.py", line 373, in run_asgi
    result = await asgi(self.receive, self.send)
  File "<project-path>\venv\lib\site-packages\uvicorn\middleware\debug.py", line 83, in __call__
    raise exc from None
  File "<project-path>\venv\lib\site-packages\uvicorn\middleware\debug.py", line 80, in __call__
    await asgi(receive, self.send)
  File "<project-path>\venv\lib\site-packages\starlette\middleware\errors.py", line 125, in asgi
    raise exc from None
  File "<project-path>\venv\lib\site-packages\starlette\middleware\errors.py", line 103, in asgi
    await asgi(receive, _send)
  File "<project-path>\venv\lib\site-packages\starlette\exceptions.py", line 74, in app
    raise exc from None
  File "<project-path>\venv\lib\site-packages\starlette\exceptions.py", line 63, in app
    await instance(receive, sender)
  File "<project-path>\venv\lib\site-packages\starlette\routing.py", line 43, in awaitable
    response = await run_in_threadpool(func, request)
  File "<project-path>\venv\lib\site-packages\starlette\concurrency.py", line 24, in run_in_threadpool
    return await loop.run_in_executor(None, func, *args)
  File "C:\Program Files (x86)\Python37-32\lib\concurrent\futures\thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "<project-path>\venv\lib\site-packages\fastapi\applications.py", line 83, in <lambda>
    lambda req: JSONResponse(self.openapi()),
  File "<project-path>\venv\lib\site-packages\fastapi\applications.py", line 75, in openapi
    openapi_prefix=self.openapi_prefix,
  File "<project-path>\venv\lib\site-packages\fastapi\openapi\utils.py", line 230, in get_openapi
    flat_models=flat_models, model_name_map=model_name_map
  File "<project-path>\venv\lib\site-packages\fastapi\utils.py", line 45, in get_model_definitions
    model, model_name_map=model_name_map, ref_prefix=REF_PREFIX
  File "<project-path>\venv\lib\site-packages\pydantic\schema.py", line 461, in model_process_schema
    model, by_alias=by_alias, model_name_map=model_name_map, ref_prefix=ref_prefix
  File "<project-path>\venv\lib\site-packages\pydantic\schema.py", line 482, in model_type_schema
    f, by_alias=by_alias, model_name_map=model_name_map, ref_prefix=ref_prefix
  File "<project-path>\venv\lib\site-packages\pydantic\schema.py", line 238, in field_schema
    ref_prefix=ref_prefix,
  File "<project-path>\venv\lib\site-packages\pydantic\schema.py", line 440, in field_type_schema
    ref_prefix=ref_prefix,
  File "<project-path>\venv\lib\site-packages\pydantic\schema.py", line 643, in field_singleton_schema
    raise ValueError(f'Value not declarable with JSON Schema, field: {field}')
ValueError: Value not declarable with JSON Schema, field: object_id type=ObjectId default=None

To Reproduce
Copy my code and follow the instrcutions given in the "Describe the bug" section.

Expected behavior
No error should occur and the documentation should be able to show the schema correctly.

Environment:

• OS: Windows 10
• FastAPI: 0.6.3
• Python: 3.7.2

[tiangolo]

Thanks for the report, sorry for the delay.

I see/assume you are using MongoDB, right?

I hope to check and debug it soon, but it might take a bit as I have to set up a stack with mongo (and I don't have a project generator with Mongo just yet).

[Certary]

Thanks for your response.

You are right, I'm using MongoDB.
But the problem isn't related to MongoDB, so I don't think you need it to debug the error.

You just need to install the following pip packages:

• FastAPI
• bson

And then put the first code snippet in a module called user.py and the second one into some arbitrary module (e.g. server.py). When you then start the server and open the automatic documentation in your browser, you should be greeted by the error in the Log accordion above.

_Note: I had to slightly modify my code snippets to make it possible to just copy-paste them._

[tiangolo]

Excellent, I'll use that to debug/develop it.

[Charlie-iProov]

Was there a conclusion to this? Trying to parse mongos _id field is proving to be quite tricky unless i just delete it before returning the response

[tiangolo]

@Charlie-iProov not yet, but it's on the backlog.

[stefanondisponibile]

Not sure this being a _bug_ or a _feature_.
I'm still digging into fastAPI, but when you're saying:

@app.post("/user", tags=["user"], response_model=UserId)

you're basically declaring that your response will be a UserId, that is:

class UserId(BaseModel):
    object_id: ObjectId = None

if I was on the other side, receiving this response, I would then have to expect this kind of json:

{
  "object_id": ObjectId("5cdc01a6d8893f59a36d9957")
}

which would be pretty strange, since I couldn't have that ObjectId there.

Moreover, having to POST to that endpoint I would have a similar problem, since User inherits from UserId:

@app.post("/user", tags=["user"], response_model=UserId)
def create_user(user: User)

class User(UserId):
    email: str
    salt: str
    hashed_password: str

again, my problem would be what to send as an object_id:

{
  "object_id": ObjectId("5cdc01a6d8893f59a36d9957"),
  "email" : "[email protected]",
  "password": "letmein",
  "salt":"12345678"
}

That's why defining custom json_encoders wouldn't help here.

[tiangolo]

@stefanondisponibile I'm currently working on this PR in Pydantic: https://github.com/samuelcolvin/pydantic/pull/520

It will allow you to declare object_id: str and then FastAPI will take your ObjectId("5cdc01a6d8893f59a36d9957") and convert it to a string automatically.

[stefanondisponibile]

That's great!

[tiangolo]

About https://github.com/samuelcolvin/pydantic/pull/520, it was superseded by https://github.com/samuelcolvin/pydantic/pull/562.

While reviewing it I tested with bson, and I realized that it doesn't necessarily fix the problem, but that you can fix it like this:

from bson import ObjectId
from pydantic import BaseModel


class ObjectIdStr(str):
    @classmethod
    def __get_validators__(cls):
        yield cls.validate

    @classmethod
    def validate(cls, v):
        if not isinstance(v, ObjectId):
            raise ValueError("Not a valid ObjectId")
        return str(v)


class UserId(BaseModel):
    object_id: ObjectIdStr = None


class User(UserId):
    email: str
    salt: str
    hashed_password: str

# Just for testing
user = User(object_id = ObjectId(), email="[email protected]", salt="12345678", hashed_password="letmein")
print(user.json())
# Outputs:
# {"object_id": "5c7e424225e2971c8c548a86", "email": "[email protected]", "salt": "12345678", "hashed_password": "letmein"}

The trick is, there's no way to declare a JSON Schema for a BSON ObjectId, but you can create a custom type that inherits from a str, so it will be declarable in JSON Schema, and it can take an ObjectId as input.

Then, if you need the ObjectId itself (instead of the str version), you can create another model that has the ObjectId as you declared it before, and copy the values from the input/to the output.

[Certary]

That will do for now, thanks for you effort!

I will also test my example with the changes to Pydantic you referenced when I get around to it.

[stefanondisponibile]

I'm using the solution proposed by @tiangolo up above, just I preferred doing this:

class ObjectIdStr(str):
    @classmethod
    def __get_validators__(cls):
        yield cls.validate

    @classmethod
    def validate(cls, v):
        try:
            ObjectId(str(v))
        except InvalidId:
            raise ValuerError("Not a valid ObjectId")
        return str(v)

this way I can either pass a valid ObjectId string or an ObjectId instance.

This works pretty nicely also with mongoengine, as you'll be able to pass that ObjectIdStr directly to the db_model, and it will convert the ObjectIdStrings to actual ObjectIds in Mongo.

What I'm striving to understand now, though, is why can't I get an ObjectId back from the jsonable_encoder by setting this in Config's json_encoders property:

from bson import ObjectId
from pydantic import BaseModel
from upabove import ObjectIdStr

class SomeItem(BaseModel):
    some_id: ObjectIdStr

    class Config:
        json_encoders = {ObjectIdStr: lambda x: ObjectId(x)}

Why wouldn't some_id be converted to an ObjectId when calling jsonable_encoder on SomeItem instance? Is it maybe because being some_id a str it won't be passed further down to the custom json_encoders? This even if ObjectIds, are not json serializable.

[senjenathaniel]

Okay, so pardon me if I don't make much sense. I face this 'ObjectId' object is not iterable whenever I run the collections.find() functions. Going through the answers here, I'm not sure where to start. I'm new to programming, please bear with me.

Every time I hit the route which is supposed to fetch me data from Mongodb, I getValueError: [TypeError("'ObjectId' object is not iterable"), TypeError('vars() argument must have __dict__ attribute')].

Help

[stefanondisponibile]

Hi @senjenathaniel ! Are you sure your problem fits this issue? If you could give some more details, and an example of the code you're using, I think someone could give the proper advice :)