Pydantic: Validate field based on other optional fields

[carmen-picwell]

Question

• OS: macOS Mojave
• Python version: 3.7.0
• Pydantic version: 0.21

I'm currently working with pydantic in a scenario where I'd like to validate an instantiation of MyClass to ensure that certain optional fields are set or not set depending on the value of an enum. However, I'm noticing in the @validator('my_field'), only required fields are present in values regardless if they're actually populated with values.

import enum
from pydantic import BaseModel, validator

class MyEnum(enum.Enum):
    ENUM_VALUE = 'foo'
    OTHER_VALUE = 'bar'


class MyClass(BaseModel):
    required_float: float
    my_enum: MyEnum
    optional_float: float = 0.0

    @validator('my_enum')
    def ensure_stuff_exists(cls, v, values, **kwargs):
        if v == MyEnum.ENUM_VALUE and values['optional_float'] is not None:
            raise KeyError('foo')


MyClass(
    my_enum=MyEnum.ENUM_VALUE,
    required_float=4.0,
    optional_float=22.0,
)
...

In the above example, only required_float and my_enum would be populated in values, I am unable to inspect the value of optional_float, even though it is populated with a value upon instantiation. Is there any way to perform this type of validation? I hope this example makes sense.

[samuelcolvin]

Yes, fields are populated in the order they're defined in the class, so just move my_enum down one line to below optional_float.

[SethMMorton]

I am not observing that fields are populated in the order they are defined - I am observing that first required fields are populated in the order they are defined, and then the optional fields are populated in the order they are defined. This is as of pydantic version 0.31.0 (Linux, Python 3.7.3). Here is a minimal example and it's output:

import traceback
from typing import List

import pydantic


class ExampleNoOptional(pydantic.BaseModel):
    foo: List[int]
    bar: int

    @pydantic.validator("bar")
    def add_length_of_foo_to_bar(cls, value, values, **kwargs):
        return value + len(values["foo"])

a = ExampleNoOptional(bar=5, foo=[1, 2, 3])
print(a.foo)
print(a.bar)
# outputs
# [1, 2, 3]
# 8

class ExampleWithOptional(pydantic.BaseModel):
    foo: List[int] = [1, 2]
    bar: int

    @pydantic.validator("bar")
    def add_length_of_foo_to_bar(cls, value, values, **kwargs):
        return value + len(values["foo"])

try:
    print(ExampleWithOptional(bar=5).bar)
except KeyError:
    traceback.print_exc()
# outputs
# Traceback (most recent call last):
#    ...
#    return value + len(values["foo"])
# KeyError: 'foo'

try:
    print(ExampleWithOptional(bar=5, foo=[1, 2, 3]).bar)
except KeyError:
    traceback.print_exc()
# outputs
# Traceback (most recent call last):
#    ...
#    return value + len(values["foo"])
# KeyError: 'foo'

class ExampleWithOptionalWorking(pydantic.BaseModel):
    foo: List[int] = [1, 2]
    bar: int

    @pydantic.validator("bar")
    def add_length_of_foo_to_bar(cls, value, values, **kwargs):
        return value + len(values.get("foo", []))

a = ExampleWithOptionalWorking(bar=5)
print(a.foo)
print(a.bar)
# outputs
# [1, 2]
# 5

a = ExampleWithOptionalWorking(bar=5, foo=[1, 2, 3])
print(a.foo)
print(a.bar)
# outputs
# [1, 2, 3]
# 5

This makes it challenging to validate required fields based on the values of optional fields.

[SethMMorton]

Additionally, as a sanity check, I copied the code in the original question and followed the advice of moving my_enum down one line below optional_float, and I get the same KeyError indicating that the optional_float is not yet in the values dictionary. I even tried it on version 0.21 to make sure and it failed even then.

Am I missing something?

[dmontagu]

@SethMMorton I think you are right. Following a debugger, I got an error in pydantic.main.validate_model, and noticed this:

def validate_model(  # noqa: C901 (ignore complexity)
    ...
    for name, field in model.__fields__.items():

So, it seemed like the order of the __fields__ would be what mattered. Sure enough:

from typing import List
import pydantic

class ExampleNoOptional(pydantic.BaseModel):
    foo: List[int]
    bar: int

class ExampleWithOptional(pydantic.BaseModel):
    foo: List[int] = [1, 2]
    bar: int

print(ExampleNoOptional.__fields__)
# {'foo': <Field(foo type=int required)>, 'bar': <Field(bar type=int required)>}

print(ExampleWithOptional.__fields__)
# {'bar': <Field(bar type=int required)>, 'foo': <Field(foo type=int default=[1, 2])>}

I don't have a work around yet (or even an understanding of why optional fields come later), but you aren't crazy.

[dmontagu]

@SethMMorton
This is the source of the issue:
https://github.com/samuelcolvin/pydantic/blob/master/pydantic/main.py#L207

# annotation only fields need to come first in fields

I just tested, and you can make it work by adding the "required" indicator:

from typing import List
from pydantic import BaseModel, Schema

class ExampleNoOptional(BaseModel):
    foo: List[int] = Schema(...)
    bar: int

class ExampleWithOptional(BaseModel):
    foo: List[int] = [1, 2]
    bar: int

This has actually been the source of several confusing validation bugs for me that I never really got to the bottom of (and worked around in other ways), so thanks for investigating this issue!

---

This is actually mentioned in the docs 😅. @samuelcolvin I think it might be good to put a red warning box in the validators section about this as well -- I very nearly missed it just now while looking closely for a reference about this in that section, and I suspect that section is a much more likely target than the mypy section when debugging a validation issue (speaking from experience...).

If you are open to it let me know and I'll make a pull request for the docs.

[SethMMorton]

Could it be due to the code starting at https://github.com/samuelcolvin/pydantic/blob/c28d469f5ba95b13114a148cf3b47f58857dfeb0/pydantic/main.py#L206?

I cannot say I fully understand what is going on or why, but the comment # annotation only fields need to come first in fields seems to imply that required (e.g. annotation only) will come first.

        if (namespace.get('__module__'), namespace.get('__qualname__')) != ('pydantic.main', 'BaseModel'):
            # annotation only fields need to come first in fields
            for ann_name, ann_type in annotations.items():
                if is_classvar(ann_type):
                    class_vars.add(ann_name)
                elif is_valid_field(ann_name) and ann_name not in namespace:
                    validate_field_name(bases, ann_name)
                    fields[ann_name] = Field.infer(
                        name=ann_name,
                        value=...,
                        annotation=ann_type,
                        class_validators=vg.get_validators(ann_name),
                        config=config,
                    )

            untouched_types = UNTOUCHED_TYPES + config.keep_untouched
            for var_name, value in namespace.items():
                if (
                    is_valid_field(var_name)
                    and (annotations.get(var_name) == PyObject or not isinstance(value, untouched_types))
                    and var_name not in class_vars
                ):
                    validate_field_name(bases, var_name)
                    fields[var_name] = Field.infer(
                        name=var_name,
                        value=value,
                        annotation=annotations.get(var_name),
                        class_validators=vg.get_validators(var_name),
                        config=config,
)

[dmontagu]

Also, @samuelcolvin if this field-ordering validation issue could be fixed (so that fields were always validated in the order declared, ignoring whether they were annotation only), would that be something you'd be willing to incorporate in 1.0?

(I'm assuming that's way too much of a potentially-breaking change for pre-1.0.)

[SethMMorton]

Digging deeper, this comment first appeared in https://github.com/samuelcolvin/pydantic/commit/5efa54d80d8a584e4c369e2ccdf30d723e54db3b, with the note "switch annotation only fields to come first in fields list not last". It seems like they have always been separated, but before they were after optional, not before.

Based on some of the diffs I saw in that commit, I got an idea to make all my required fields non-annotation-only by adding ....

class ExampleWithOptional(pydantic.BaseModel):
    foo: List[int] = [1, 2]
    bar: int = ...

    @pydantic.validator("bar")
    def add_length_of_foo_to_bar(cls, value, values, **kwargs):
        return value + len(values["foo"])

This got it to work as well.

---

So, there is a workaround! If it is not possible to get what I was originally trying to do to work, it might be nice to add to the docs that you can make it work with = ....

[SethMMorton]

@dmontagu Looks like we were digging and posting at the same time!

I agree with you that I did not see that warning in the docs because it was in the mypy section, so I assumed it would be mypy only.

[samuelcolvin]

Happy to make the warning more explicit and move it.

To my knowledge there's no way of fixing this since annotations are a completely different object from namespace. Short of manually opening the file and reading the ast this would be impossible.

[dmontagu]

@samuelcolvin
Maybe I'm missing something, but it looks like annotations has everything, and in the proper order:

from typing import List
from pydantic import BaseModel, Schema

class ExampleNoOptional(BaseModel):
    foo: List[int] = Schema(...)
    bar: int = 1

print(ExampleNoOptional.__annotations__)
print(ExampleNoOptional.__fields__)
# {'foo': typing.List[int], 'bar': <class 'int'>}
# {'foo': <Field(foo type=int required)>, 'bar': <Field(bar type=int default=1)>}

class ExampleWithOptional(BaseModel):
    foo: List[int] = [1, 2]
    bar: int

print(ExampleWithOptional.__annotations__)
print(ExampleWithOptional.__fields__)
# {'foo': typing.List[int], 'bar': <class 'int'>}
# {'bar': <Field(bar type=int required)>, 'foo': <Field(foo type=int default=[1, 2])>}

~Maybe it's a python 3.6 difference?~ I get the same result in python 3.6 and 3.7. Maybe it has something to do with what's going on in the metaclass? Or inheritance?

---

In case it wasn't clear, the approach I had in mind would be similar to:

new_fields = {k: fields[k] for k in annotations if k in fields}
new_fields.update({k: fields[k] for k in fields if k not in new_fields})
fields = new_fields

But I'd guess it would require more nuance to handle inheritance etc.