Pydantic: Optional fields loose some validation

[vdwees]

Bug

• OS: Ubuntu 18.04
• Python version: 3.7.3
• Pydantic versions: 0.30(pypi), 1.0a1 (master)

Example code:

from pydantic import BaseModel, Schema
from typing import Optional

class MyModel(BaseModel):
    my_int: Optional[int] = Schema(..., ge=3)

MyModel(my_int=2)

I would expect:

pydantic.error_wrappers.ValidationError: 1 validation error
my_int
  ensure this value is greater than or equal to 3 (type=value_error.number.not_ge; limit_value=3)

but MyModel gets instantiated just fine.

If I make my_int not optional, the behaviour is as expected.

[dmontagu]

@vdwees I've traced the source of this issue to this line.

The problem here is that Optional[X] is not considered by python to be a type. I definitely don't think the way it is currently handled is ideal, but it might be somewhat involved to fix this properly.

For now, here's a workaround:

from pydantic import BaseModel, Schema, ConstrainedInt
from typing import Optional

class ConInt3(ConstrainedInt):
    ge = 3

class MyModel(BaseModel):
    my_int: Optional[ConInt3] = Schema(...)

print(MyModel.schema())
# {'title': 'MyModel', 'type': 'object', 'properties': {'my_int': {'title': 'My_Int', 'minimum': 3, 'type': 'integer'}}}

print(MyModel(my_int=2))
"""
pydantic.error_wrappers.ValidationError: 2 validation errors for MyModel
my_int
  ensure this value is greater than or equal to 3 (type=value_error.number.not_ge; limit_value=3)
my_int
  value is not none (type=type_error.none.allowed)
"""

---

@samuelcolvin I'm not sure whether it would be better to modify the get_annotation_from_schema function to deal with the whole typing introspection rigamarole (like in, e.g., Field._populate_sub_fields), or to just require more careful handling (like in the "workaround" I provided above). Thoughts?

[samuelcolvin]

Thanks for reporting. Definitely looks like a bug, I'll look into in and see what I can do.

[dannymilsom]

Firstly thank you both for your contributions to this library, and +1 for resolving this issue if possible.

@dmontagu I have tried your workaround but I believe it has uncovered a second bug - if you look at your code snippet the schema now raises two issues when I would expect one (i.e. I assume it should not complain that the optional value is not none).

For a simpler example (using Pydantic 0.32.2)on MacOSX:

>>> from pydantic import BaseModel, PositiveInt
>>> from typing import Optional
>>> class ExampleModel(BaseModel):
...     foo: Optional[PositiveInt]
...
>>> ExampleModel(foo=0)
pydantic.error_wrappers.ValidationError: 2 validation errors for ExampleModel
foo
  ensure this value is greater than 0 (type=value_error.number.not_gt; limit_value=0)
foo
  value is not none (type=type_error.none.allowed)

[samuelcolvin]

hi @dannymilsom I'm not clear what second bug you've uncovered? Please can you elaborate.

[dmontagu]

@dannymilsom what's happening is that it's showing you, for each option in the union, why it can't be that option. The first option was PositiveInt, which fails. The second is None, which also fails (obviously).

This is because Optional[PositiveInt] is basically the same as Union[PositiveInt, None].

@samuelcolvin I feel like maybe some of this stuff has changed in v1 anyway; not sure whether the error messages still look the same? Either way, I don't think this is a buggy error message, just a slightly confusing one.

[samuelcolvin]

@dmontagu yes, you're right on all: it's not an error and it's clearer in v1.

It's also not related in anyway to this issue.

[dannymilsom]

@samuelcolvin Thank you for the speedy fix! And apologies for my misunderstanding around that error message - agree that isn't a bug 🤦‍♂ I had set genot le on my subclass of ConstrainedInt(not reflected in my simplified example code) and thought the none validation was the culprit...