Tortoise-orm: more further pydantic/serialization options

[AEnterprise]

Is your feature request related to a problem? Please describe.
The current pydantic integration is great but only one way, it's only useful for serializing and sending out data

Describe the solution you'd like
integration for deserializing models as well, this would make integrating with web frameworks that already leverage pydantic (like fastapi) much much easier

more specifically:

• a way to to make objects linked to the database based on a dict or even directly so classes can be used directly as type hints and fastapi/pydantic figure out the validation and returning of 422 (unprocessable entity) errors so we don't have to do this

• a way to then save this object to the database (saving the altered fields, primary key or unique key would ofc have to be present and valid for this to work)

Describe alternatives you've considered
doing it manually, it works but isn't great

Additional context
building an api using fastapi for handling the web requests and tortoise is my orm of choice to communicate with the database

[AEnterprise]

another suggestion i forgot to mention: a mode on the serialization where you can just tell it to ignore any missing data from relations instead of throwing an exception and nuking the entire thing

it's useful in some cases where you don't want to some subdata in general but only wana return it in some specific api routes, would be easier then to have to specifically ignore it in all places you don't want it

[grigi]

That would possibly require overriding the from_orm() that is provided by the base pydantic class.
It would make sense for 1:M or M:M relations, but for 1:1 where we expect a dict, doing that would generate an invalid object that would fail schema validation. So only for M2M or reverse FK can we get away with that.
Unless we make the object 'optional', but right now we are not managing optional at all.

New feature request:

• Manage 'optional' fields explicitly in schema
• Simple validation? e.g. char max/min length
• enums!

[AEnterprise]

1:1 makes sense it's always there cause one rarely means as much without the other

but in my case i have several layers to my data. there are inspections, those are couple to a location (and each location can have multiple inspections over time), that then is coupled to a company, list of authorized people, inspections also have actions taken and more, ....

it quickly gets a bit of a maze of what route should send or omit what data depending on what's useful as each seralization can take up to like 3 separate querries to get the full dataset

[jet10000]

Now it can directly convert to json, do you consider transferring json to orm? For example, I use the couchbase database to store data, but I expect to use a relational database mindset to design the model and execute related queries.

https://docs.couchbase.com/server/6.5/getting-started/try-a-query.html

[jet10000]

django can create a form based on the model, is it possible to map the tortoise-orm schema to pydantic for validating the form

[grigi]

Now it can directly convert to json, do you consider transferring json to orm?

Yes, that is the plan. Not yet, but soon.

django can create a form based on the model, is it possible to map the tortoise-orm schema to pydantic for validating the form

We currently create a pydantic model, but can only do basic type validation. We would want to extend it to validate more.
We have enough info for these: (looked at https://datatracker.ietf.org/doc/draft-handrews-json-schema-validation/?include_text=1 for spec)

• max length of strings
• min/max of ints
• enums
• required fields
• readonly (computed fields)
• date/time/date-time/duration
• uuid

That should at least prevent a db insert fail to happen? (It's also more than I thought)

:thinking: This is where hypothesis would be good. it already has support for generating data off a jsonschema and then we just have to ensure no errors leak through.
Fuzz testing at it's best!

How we should add the extra validation info is up for debate, as we could have validations inferred from type annotations, or explicitly added in the tortoise model, or annotate them with the pydantic schema creator.
We already do _some_ from the first one. e.g. JSONField defaults to Union[dict, list] but one could annotate it as foo: dict = JSONField() and it would only use the dict then.
But getting more than that out would be quite troublesome, and error-prone.

:thinking: Even using that kind of types should actually be validated against the supertype? e.g. this is wrong foo: Decimal = JSONfield() as it can't ever work, and we should fail hard when we see that?

[grigi]

Ok, in #318 I have added the following basic features:

• Ability to exclude all read-only & relational fields (generated/computed/relational)
  
  
  
  • This allows simple create/update to work with the reduced model
  
  
• Define required/nullable well (There is a catch in that we need to use **user.dict(exclude_unset=True) to exclude unset fields, else Pydantic fills them in, and then defaults can't get processed correctly by us.
• Simple validation for:
  
  
  
  • max length of strings
  
  
  • min/max of ints
  
  
• More pydantic-compatible way of handling List[...] schemas
• Convenience functions to get a single-instance or list from the queryset on the plain pydantic_model_creator essentially removing the need of ever using the pydantic_queryset_creator (Should we mark it as deprecated straight away?)

Further items of work identified:

• Aliasing
• Enforcing globally unique model names
• Strategy for Write-only fields (e.g. setting a password which is not available after compute?)
• Ability to reference partial child objects:
  
  
  
  • PK's
  
  
  • Some other unique scalar
  
  
  • An incomplete subset of fields
  
  
• Better recursion strategy where we allow a model to be self referential? (Or is this folly?)

[grigi]

Enforcing globally unique model names is part of 0.16.9

Items to still work on:

• Aliasing
• Ability to specify partial child objects
• Use .only() to fetch less data from DB
• Opportunistically lower prefetches into select-related for minimising DB round trips.

[long2ice]

Fix pydantic_model_creator return same model with twice call before init_models and after in https://github.com/tortoise/tortoise-orm/commit/8ceedc096f9f3bb65f88ea8f7f8cedce9f25dffb