Fastapi: [BUG] Null value if the field has an alias

[francbartoli]

Describe the bug
A model defined by a field with an alias gives a null value in the response.

To Reproduce
Steps to reproduce the behavior:
1. Create the following files using the template full-stack-fastapi-postgresql
- models/test.py

from typing import List
from pydantic import BaseModel, Schema


class metadata(BaseModel):
    title: str = None


class baseType(BaseModel):
    metadata_: List[metadata] = Schema(None, alias="metadata")


class testSummary(baseType):
    tid: str = ...


class testCollection(BaseModel):
    tests: List[testSummary] = None


class TestBase(testSummary):
    pass


class TestCreate(TestBase):
    pass


# Properties shared by models stored in DB
class TestInDBBase(TestBase):
    id: int = None
    owner_id: int

    class Config:
        orm_mode = True


# Properties to return to client
class Test(TestInDBBase):
    pass

• db_models/test.py

from sqlalchemy import Column, ForeignKey, Integer, String
from sqlalchemy.dialects import postgresql
from sqlalchemy.orm import relationship

from app.db.base_class import Base


class Test(Base):

    __tablename__ = 'test'

    id                 = Column(Integer, primary_key=True, index=True)
    tid                = Column(String, index=True)
    title              = Column(String, index=True)
    metadata_          = Column(postgresql.JSONB, default=[])
    owner_id           = Column(Integer, ForeignKey("user.id"))
    owner              = relationship("User", back_populates="tests")

1. Add a path operation function with

from fastapi import APIRouter, Depends, HTTPException
from sqlalchemy.orm import Session

from app import crud
from app.models.test import testCollection, TestCreate, Test
from app.api.utils.db import get_db
from app.api.utils.security import get_current_active_user
from app.db_models.user import User as DBUser


router = APIRouter()


import logging


logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)


@router.get(
    "/tests/",
    operation_id="getTests",
    response_model=testCollection,
    status_code=200
)
def read_wps_tests(
    db: Session = Depends(get_db),
    skip: int = 0,
    limit: int = 100,
):
    """
    Retrieve available tests.
    """
    tests = crud.test.get_multi(db, skip=skip, limit=limit)
    if not tests:
        logger.info(f"======> The test collection is empty")
        tests = []
    return {"tests": tests}


@router.post(
    "/tests/",
    operation_id="createTest",
    response_model=Test,
    status_code=201,
    include_in_schema=True
)
def create_wps_test(
    *,
    db: Session = Depends(get_db),
    test_in: TestCreate,
    current_user: DBUser = Depends(get_current_active_user),
):
    """
    Create new wps test.
    """
    if not crud.user.is_superuser(current_user):
        raise HTTPException(
            status_code=403,
            detail="You are not allowed to register tests"
        )
    elif crud.test.get_by_tid(db_session=db, tid=test_in.tid):
        raise HTTPException(
            status_code=422,
            detail="The test has been already created"
        )
    test = crud.test.create(
        db_session=db,
        test_in=test_in,
        owner_id=current_user.id
    )
    return test

1. Make a POST with a payload of

{
  "metadata": [
    {
      "title": "my title"
    }
  ],
  "tid": "pippo”
}

1. Make GET request and see error

Expected behavior
Before 0.30.0 I've seen it working but the response had "metadata": null

Environment:

• OS: template with containers
• FastAPI 0.30.0

[tiangolo]

@francbartoli I think the problem is that you are specifying an alias metadata for an attribute metadata_, so the model will now expect to receive metadata as one of the things to validate/serialize, but your SQLAlchemy model has metadata_ too instead of metadata.

I guess the first thing to try is to change the metadata_ in your SQLAlchemy model to metadata.

[francbartoli]

@tiangolo I cannot use metadata cause it's a reserved name for SQLAlchemy with declarative_base

[tiangolo]

I see, then you can try allow_population_by_alias: https://pydantic-docs.helpmanual.io/#model-config (which actually means allow population by name and alias)

[francbartoli]

@tiangolo I've tried but now the field is populated with an empty list:

Body request:

{
  "metadata": [
    {
      "title": "thistle"
    }
  ],
  "tid": "thistle"
}

Response is 201 with

{
  "metadata": [
    {
      "title": "thistle"
    }
  ],
  "tid": "thistle",
  "owner_id": 1,
  "id": 3
}

But effectively the record has metadata_ = [] in postgresql

[stefanondisponibile]

@francbartoli I'm reading this from mobile and still haven't tested it but I think (as suggested by @tiangolo) you could try and define you model like

class BaseType(BaseModel):
    metadata : List[metadata] = Schema (None, alias="_metadata")

    class Config:
        allow_population_by_alias = True

This way your model's metadata property should be populated both by metadata and _metadata, but should be json encoded to the alias _metadata.

[francbartoli]

@stefanondisponibile Applying your changes would end up with the following error:

backend_1        | Traceback (most recent call last):
backend_1        |   File "/usr/local/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
backend_1        |     self.run()
backend_1        |   File "/usr/local/lib/python3.7/multiprocessing/process.py", line 99, in run
backend_1        |     self._target(*self._args, **self._kwargs)
backend_1        |   File "/usr/local/lib/python3.7/site-packages/uvicorn/main.py", line 305, in run
backend_1        |     loop.run_until_complete(self.serve(sockets=sockets))
backend_1        |   File "uvloop/loop.pyx", line 1451, in uvloop.loop.Loop.run_until_complete
backend_1        |   File "/usr/local/lib/python3.7/site-packages/uvicorn/main.py", line 312, in serve
backend_1        |     config.load()
backend_1        |   File "/usr/local/lib/python3.7/site-packages/uvicorn/config.py", line 182, in load
backend_1        |     self.loaded_app = import_from_string(self.app)
backend_1        |   File "/usr/local/lib/python3.7/site-packages/uvicorn/importer.py", line 20, in import_from_string
backend_1        |     module = importlib.import_module(module_str)
backend_1        |   File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module
backend_1        |     return _bootstrap._gcd_import(name[level:], package, level)
backend_1        |   File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
backend_1        |   File "<frozen importlib._bootstrap>", line 983, in _find_and_load
backend_1        |   File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
backend_1        |   File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
backend_1        |   File "<frozen importlib._bootstrap_external>", line 728, in exec_module
backend_1        |   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
backend_1        |   File "./app/main.py", line 5, in <module>
backend_1        |     from app.api.api_v1.api import api_router
backend_1        |   File "./app/api/api_v1/api.py", line 4, in <module>
backend_1        |     from app.api.api_v1.endpoints import (
backend_1        |   File "./app/api/api_v1/endpoints/login.py", line 7, in <module>
backend_1        |     from app import crud
backend_1        |   File "./app/crud/__init__.py", line 2, in <module>
backend_1        |     from . import user, process, job, test
backend_1        |   File "./app/crud/test.py", line 7, in <module>
backend_1        |     from app.models.test import TestCreate
backend_1        |   File "./app/models/test.py", line 9, in <module>
backend_1        |     class baseType(BaseModel):
backend_1        |   File "./app/models/test.py", line 10, in baseType
backend_1        |     metadata: List[metadata] = Schema(None, alias="metadata_")
backend_1        |   File "/usr/local/lib/python3.7/typing.py", line 251, in inner
backend_1        |     return func(*args, **kwds)
backend_1        |   File "/usr/local/lib/python3.7/typing.py", line 626, in __getitem__
backend_1        |     params = tuple(_type_check(p, msg) for p in params)
backend_1        |   File "/usr/local/lib/python3.7/typing.py", line 626, in <genexpr>
backend_1        |     params = tuple(_type_check(p, msg) for p in params)
backend_1        |   File "/usr/local/lib/python3.7/typing.py", line 139, in _type_check
backend_1        |     raise TypeError(f"{msg} Got {arg!r:.100}.")
backend_1        | TypeError: Parameters to generic types must be types. Got Schema(alias: 'metadata_', extra: {}).
backend_1        | INFO: Stopping reloader process [1]

I guess the problem is with the name metadata which cannot be applied in pydantic to a field and a model at the same time

[stefanondisponibile]

🤔 I'm not sure

from sqlalchemy import create_engine
from sqlalchemy import Column, Integer, String
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import sessionmaker
from pydantic import BaseModel
from pprint import pprint
from typing import Any
from fastapi.encoders import jsonable_encoder

Base = declarative_base()

class SomeClass(Base):
    __tablename__ = 'some_table'
    id = Column(Integer, primary_key=True)
    metadata_ =  Column(String(50))

class Test(BaseModel):
    id: int = 42
    metadata: Any = ...

    class Config:
        orm_mode = True
        fields = {"metadata": {"alias": "metadata_"}}

engine = create_engine('sqlite://')
Base.metadata.create_all(engine)

Session = sessionmaker(bind=engine)
session = Session()
c_in = Test(**{"metadata_": "foo"})
c_in = jsonable_encoder(c_in, by_alias=True)
c = SomeClass(**c_in)
session.add(c)
session.commit()
for t in session.query(SomeClass).all():
    pprint(vars(t))
    print()
    print("ORM MODE:")
    print(Test.from_orm(t).json())
    print(jsonable_encoder(Test.from_orm(t), by_alias=False))
    print("\nSTARRED:")
    print(Test(**vars(t)).json())
    print(jsonable_encoder(Test(**vars(t)), by_alias=False))

{'_sa_instance_state': <sqlalchemy.orm.state.InstanceState object at 0x1103fa4e0>,
 'id': 42,
 'metadata_': 'foo'}

ORM MODE:
{"id": 42, "metadata": "foo"}
{'id': 42, 'metadata': 'foo'}

STARRED:
{"id": 42, "metadata": "foo"}
{'id': 42, 'metadata': 'foo'}

[francbartoli]

Hi @stefanondisponibile, sorry perhaps I didn't explain my use case properly. I was meaning this situation which would be required to be consistently conforming with the models' name of my OpenAPI definition.

from sqlalchemy import create_engine
from sqlalchemy import Column, Integer, String
from sqlalchemy.dialects import postgresql
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import sessionmaker
from pydantic import BaseModel
from pprint import pprint
from typing import Any, List
from fastapi.encoders import jsonable_encoder

Base = declarative_base()

class SomeClass(Base):
    __tablename__ = 'some_table'
    id = Column(Integer, primary_key=True)
    metadata_ =  Column(postgresql.JSONB, default=[])

class metadata(BaseModel):
    title: str = None

class Test(BaseModel):
    id: int = 42
    metadata: List[metadata] = ...

    class Config:
        orm_mode = True
        fields = {"metadata": {"alias": "metadata_"}}

engine = create_engine('postgresql://')
Base.metadata.create_all(engine)

Session = sessionmaker(bind=engine)
session = Session()
c_in = Test(**{"metadata_": [{"title": "foo"}]})
c_in = jsonable_encoder(c_in, by_alias=True)
c = SomeClass(**c_in)
session.add(c)
session.commit()
for t in session.query(SomeClass).all():
    pprint(vars(t))
    print()
    print("ORM MODE:")
    print(Test.from_orm(t).json())
    print(jsonable_encoder(Test.from_orm(t), by_alias=False))
    print("\nSTARRED:")
    print(Test(**vars(t)).json())
    print(jsonable_encoder(Test(**vars(t)), by_alias=False))

[stefanondisponibile]

Oh yes, sorry, I hadn't read properly.
However, I tried plugging your example into a fastAPI app and it looks like working to me:

docker run --name tmp-postgres -p 5432:5432 -e POSTGRES_PASSWORD=foo -e POSTGRES_USER=foo postgres

from sqlalchemy import create_engine
from sqlalchemy import Column, Integer, String
from sqlalchemy.dialects import postgresql
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import sessionmaker
from pydantic import BaseModel
from pprint import pprint
from typing import Any, List, Dict
from fastapi.encoders import jsonable_encoder
from fastapi import FastAPI

Base = declarative_base()

class SomeClass(Base):
    __tablename__ = 'some_table'
    id = Column(Integer, primary_key=True)
    metadata_ =  Column(postgresql.JSONB, default=[])

class Metadata(BaseModel):
    title: str = None

class Test(BaseModel):
    metadata_: List[Metadata] = ...

    class Config:
        orm_mode = True
        fields = {"metadata_": {"alias": "metadata"}}
        allow_population_by_alias = True

engine = create_engine('postgresql://foo:foo@localhost:5432')
Base.metadata.create_all(engine)

Session = sessionmaker(bind=engine)
session = Session()

app = FastAPI()

@app.get("/test", response_model=List[Test])
def read_tests():
    tests = session.query(SomeClass).all()
    return tests

@app.post("/test", response_model=Test)
def create_test(test_in: Test):
    test_in = jsonable_encoder(test_in, by_alias=False)
    test = SomeClass(**test_in)
    session.add(test)
    session.commit()
    session.refresh(test)
    return test

curl -X POST -H "Content-Type: application/json" -d '{"metadata": [{"title": "foo"}, {"title": "bar"}]}' http://localhost:8000/test

```json
{
"metadata": [
{
"title": "foo"
},
{
"title": "bar"
}
]
}

```bash
curl -X GET http://localhost:8000/test

[
    {
        "metadata": [
            {
                "title": "foo"
            },
            {
                "title": "bar"
            }
        ]
    }
]

Is this the output you would expect @francbartoli ?

One thing I noticed, though, is that adding session.refresh(test) to the app POST method would mess things up in this example. I'm sorry I can't really understand if this could be an issue and would rely more on @tiangolo or others for this.

[stefanondisponibile]

Since I noticed some errors in my code (don't know why the heck hadn't put the response model into the decorator, sorry about that), I decided to cut to the chase and set up a little app for it.
Looks ok to me, try that out: git clone https://gitlab.com/stefanondisponibile/fastapi_327

[tiangolo]

Thanks for all your help here @stefanondisponibile !

@francbartoli It might be the case that the problem is related to https://github.com/tiangolo/fastapi/issues/332 if that's the case, you could try with 0.29.1 and see if it works.

Otherwise, the best would be to write a simple/minimal example program, possibly copying from the SQL tutorial, and replicating your error. That way it would be easier to debug what is happening and what you would expect to happen.

[francbartoli]

@tiangolo thanks, it is related to that issue. I'm going to test with Pydantic 0.30 if that solves the issue

[francbartoli]

closed by 0.33.0

[tiangolo]

Thanks for reporting back and closing the issue @francbartoli ! :tada: