Pydantic: arbitrary_types_allowed not respected when validating stdlib dataclasses

[ines]

Checks

• [x] I added a descriptive title to this issue
• [x] I have searched (google, github) for similar issues and couldn't find anything
• [x] I have read and followed the docs and still think this is a bug

Bug

Output of python -c "import pydantic.utils; print(pydantic.utils.version_info())":

             pydantic version: 1.7
            pydantic compiled: True
                 install path: /Users/ines/Repos/explosion/spacy/.env37/lib/python3.7/site-packages/pydantic
               python version: 3.7.2 (default, Oct  6 2019, 23:51:55)  [Clang 10.0.1 (clang-1001.0.46.4)]
                     platform: Darwin-19.5.0-x86_64-i386-64bit
     optional deps. installed: ['typing-extensions']

As of v1.7, Pydantic validates dataclasses and their contents, but it fails if the dataclass specifies arbitrary types (which is pretty common in our code bases). Here's an example:

from typing import List
from dataclasses import dataclass
from pydantic import BaseModel

class ArbitraryType:
    def __init__(self):
        ...

@dataclass
class Test:
    foo: ArbitraryType
    bar: List[ArbitraryType]


class TestModel(BaseModel):
    a: ArbitraryType  # this is fine
    b: Test  # this raises RuntimeError

    class Config:
        arbitrary_types_allowed = True

Even though the BaseModel subclass sets arbitrary_types_allowed to True, this configuration doesn't seem to be taken into account when validating the dataclass fields.

Adding custom validation to our dataclasses isn't really an option, since we wouldn't want those Pydantic specifics to leak into the rest of the code base. I'm also wondering whether there should be an option to _not_ validate stdlib dataclasses as Pydantic dataclasses and just use a basic instance check instead, like it previously did (?) before v1.7?

[samuelcolvin]

I had this same problem yesterday.

it's not a bug since it matches the behaviour of models or dataclasses in the past. In this situation a model didn't know where it was going to be used when it was created (which is when this check takes place), so couldn't respect arbitrary_types_allowed.

I think it might be possible to find a work around for this, but it might not be trivial. @PrettyWood do you have any idea how hard it would be to respect arbitrary_types_allowed of the parent model? I guess in theory we can get access to config in the validator, so it shouldn't be hard hard.

Also, are there any other config attributes we should respect too? I guess not.

[tiangolo]

I think the main issue in this specific example is that it is using the standard lib dataclasses.dataclass instead of Pydantic's pydantic.dataclasses.dataclass.

So, here Test would be indeed an "arbitrary type", not a Pydantic model (I think).

---

Extending the example from @ines :

from typing import List
from dataclasses import dataclass
from pydantic import BaseModel


class ArbitraryType:
    def __init__(self, name: str):
        self.name = name


@dataclass
class Test:
    foo: ArbitraryType
    bar: List[ArbitraryType]


class TestModel(BaseModel):
    a: ArbitraryType  # this is fine
    b: Test  # this raises RuntimeError

    class Config:
        arbitrary_types_allowed = True


foo = ArbitraryType(name='Foo')
bar = [ArbitraryType(name='Bar'), ArbitraryType(name='Baz')]
test = Test(foo=foo, bar=bar)
test_model = TestModel(a=ArbitraryType(name='A'), b=test)


assert test_model.a.name == 'A'
assert test_model.b.bar[1].name == 'Baz'

Running this script with Pydantic < 1.7 passes. But raises in 1.7.

But interestingly, in Pydantic < 1.7, changing the line:

from dataclasses import dataclass

to

from pydantic.dataclasses import dataclass

then makes it raise the same way as it is raising now.

---

I feel like this could be related to @PrettyWood 's fix at https://github.com/samuelcolvin/pydantic/pull/2051, but I tried locally with the code from his PR and sadly it didn't fix this specific case.

[samuelcolvin]

It's because prior to 1.7 you couldn't use dataclasses as field types (well you could, but you had to add arbitrary_types_allowed and there were interpreted like any other arbitrary type).

With 1.7, pydantic will inspect the dataclass and do full validation on the dataclass fields, but problem is when the dataclass as unknown field types, the step of converting the stardard library dataclass to a pydantic dataclass doesn't respect config on the outer model.

[PrettyWood]

The fix is quite easy in fact. I added it in #2051 https://github.com/samuelcolvin/pydantic/pull/2051/commits/86aaff215de6f864ab218c58d09645a086f648e8

[samuelcolvin]

that's awesome. Thank you.

[tiangolo]

With 1.7, pydantic will inspect the dataclass and do full validation on the dataclass fields, but problem is when the dataclass as unknown field types, the step of converting the stardard library dataclass to a pydantic dataclass doesn't respect config on the outer model.

Thanks for the clarification @samuelcolvin ! I didn't know that was possible now, very cool! :rocket: :tada:

And thanks @PrettyWood , you rock! :guitar: