Pydantic: Support for arbitrary generic types in fields

[ines]

First, thanks for the great work on pydantic! 🙌 I'm not 100% sure this is considered a bug, so feel free to relabel.

Bug

Output of python -c "import pydantic.utils; print(pydantic.utils.version_info())":

pydantic version: 1.3
pydantic compiled: False
install path: ...
python version: 3.7.2 (default, Oct  6 2019, 23:51:55)  [Clang 10.0.1 (clang-1001.0.46.4)]
platform: Darwin-19.0.0-x86_64-i386-64bit
optional deps. installed: ['typing-extensions']

I have a custom type that's used in a pydantic model with arbitrary types allowed. Sometimes it's just used as an arbitrary type, sometimes as a generic alias. In the second case, the type analysis fails and raises the error TypeError: Fields of type "<class '__main__.MyCustomClass'>" are not supported, because it triggers an origin check and its origin isn't supported.

from typing import TypeVar, Generic
from pydantic import BaseModel


InT = TypeVar("InT")
OutT = TypeVar("OutT")


class MyCustomClass(Generic[InT, OutT]):
    pass


class Model(BaseModel):
    # This works fine as it's an arbitrary type with no __origin__
    a: MyCustomClass
    # This doesn't, because it has an __origin__ that's not supported 
    # (MyCustomClass) so ModelField._type_analysis raises an error
    b: MyCustomClass[str, int]

    class Config:
        arbitrary_types_allowed = True

Is there a workaround I could use to make it treat MyCustomClass[str, int] like MyCustomClass and perform a simple instance check? Or maybe there's something else I'm missing?

It'd also be cool to have a way to customise ModelField._type_analysis, or have pydantic expose this logic as a helper like is_valid_type (or similar). In my specific use case, I'm creating the model dynamically, so I can perform additional checks – e.g. fall back to the type's origin or even Any if I know in advance that pydantic won't accept it.

Edit: I've already tried implementing a __get_validators__ on the class and that didn't make a difference.

[dmontagu]

First, thanks for the great work on pydantic!

Huge fan of spaCy here -- thanks for the great work on that!

---

I personally definitely think we should support this (both with and without __get_validators__).

My intuition is that it would be relatively simple to implement; @samuelcolvin are there any gotchas you can think of, or other reasons this shouldn't be supported?

I would be happy to review a PR to this end, I would also be happy to implement it myself assuming consensus on accepting the "feature".

---

I think one possible concern would be that we wouldn't know, in general, how to automatically validate the provided model complies with its specified __origin__ parameters. But perhaps that should be the responsibility of the __get_validators__ method anyway?

[ines]

Aw thanks! Very excited to officially adopt pydantic across our projects soon. (We're currently working on the new version of spaCy and our machine learning library Thinc which will both finally drop Python 2.7 and 3.5 🎉 So we're free to use all the cool new typing stuff.)

I think one possible concern would be that we wouldn't know, in general, how to automatically validate the provided model complies with its specified __origin__ parameters.

As @tiangolo noted in the comments of #1159, there isn't really a sensible way to use the generic type for more specific validation – at least, not really within the scope of pydantic. So if arbitrary_types_allowed is set, I think it'd be fine to just do a regular instance check, regardless of whether the type is MyCustomClass or MyCustomClass[str, int]. At least, from a user's perspective, that's what I'd expect. __get_validators__ can then provide additional validation if needed.

This would also make it easier to handle generic types from third-party code that we can't control. (In our use case, we're creating a pydantic model from function signatures and are using that to parse and validate config objects. And type hints on these functions can come from anywhere, including user code.)

[samuelcolvin]

@ines so pleased pydantic is helping.

Would you like to add a section about spaCy to the "using pydantic" section of our docs?

[ines]

@samuelcolvin Sure! It probably makes sense to wait until the new versions of spaCy (v3.0) and Thinc (v8.0) are out – the latest stable versions doesn't yet use pydantic.

We also use pydantic in Prodigy btw to parse and validate user config and incoming data. It's already included in the latest release 😃

[samuelcolvin]

awesome, I guess once spacy v3 is out and you add an item, you can include prodigy.