Pydantic: Incorrect type for dynamic default

[pikeas]

Bug

From https://pydantic-docs.helpmanual.io/usage/validators/#validate-always

from datetime import datetime
from pydantic import BaseModel, validator


class DemoModel(BaseModel):
    ts: datetime = None

    @validator("ts", pre=True, always=True)
    def set_ts_now(cls, v):
        return v or datetime.now()

The default value of None for ts is incorrect (flagged by type checking), as None isn't a datetime. A type signature of Optional[datetime] would also be incorrect, as the validator ensures that ts is always set to a datetime (or fails regular validation). However, a model with the default removed will fail to validate:

from datetime import datetime
from pydantic import BaseModel, validator


class DemoModel(BaseModel):
    ts: datetime

    @validator("ts", pre=True, always=True)
    def set_ts_now(cls, v):
        return v or datetime.now()


model = DemoModel()

###
$ python foo.py
Traceback (most recent call last):
  File "foo.py", line 14, in <module>
    model = DemoModel()
  File "pydantic/main.py", line 346, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 1 validation error for DemoModel
ts
  field required (type=value_error.missing)
 dm = DemoModel()

pydantic.utils.version_info()

          pydantic version: 1.6.1
            pydantic compiled: True
                 install path: <...>/.venv/lib/python3.8/site-packages/pydantic
               python version: 3.8.5 (default, Jul 23 2020, 17:26:51)  [Clang 11.0.3 (clang-1103.0.32.62)]
                     platform: macOS-10.15.5-x86_64-i386-64bit
     optional deps. installed: ['typing-extensions']

[PrettyWood]

Hi @pikeas
You're right it's not really compatible with mypy you should just add a # type: ignore or something. But it's not a bug.
I think the best would be to use the recently added default_factory, which is shorter and type safe

from datetime import datetime

from pydantic import BaseModel, Field


class DemoModel(BaseModel):
    ts: datetime = Field(default_factory=datetime.now)

Hope it helps!

[pikeas]

@PrettyWood Thanks for the quick response! Unfortunately, that won't work for my use case, which is more like this:

class DemoModel(BaseModel):
    p1: Path
    p2: Path # Needs = None to work

    @validator("p2", pre=True, always=True)
    def default_p2(cls, v, values):
        if (not v) and (p1 := values.get('p1'):
            return p1 / "subfolder"
        return v

Generically, I have a field F2 which always exists on the model but is an optional parameter. When F2 isn't given, its value depends on another field F1.

[PrettyWood]

@pikeas I don't really understand why it should be Path and not Optional[Path]. It just means that setting nothing as p2 is valid, which is true because you actually set a default value afterwards.
You can either set p2: Optional[Path] or just p2: Path = None (which actually does the same as it changes Path to Optional[Path]) but you will have to add # type: ignore afterwards for mypy.
There is currently no other solution

[pikeas]

@PrettyWood What's the purpose of providing types on a model? If it's to specify what Pydantic receives as input, then I agree that Optional[Path] is correct. However, it's my understanding that the types represent a contract on the output, in which case Path is correct.

[pikeas]

From https://pydantic-docs.helpmanual.io/usage/models/

pydantic is primarily a parsing library, not a validation library. Validation is a means to an end: building a model which conforms to the types and constraints provided.
In other words, pydantic guarantees the types and constraints of the output model, not the input data.

I remain a bit stumped by this. How can I express "this field is guaranteed to exist and always has this type, but may be absent as input (in which case here is a derived value)"?

[PrettyWood]

Well I understand your point and maybe we could change that in v2 but it would be a significant change.

As for now if you want to keep the type as it is you can just use a default value of the same type (instead of None) like '' (so p2: Path = '') or use a sentinel but you'll have to tweak it a bit to work with some classes.

[pikeas]

Well I understand your point and maybe we could change that in v2 but it would be a significant change.

Sure, breaking backwards compatibility is a big deal and deserves a lot of thought. But since v2 will already include breaking changes, now seems like a great time to consider this.

To summarize, this seems like the behavior most users would expect:

class Foo(BaseModel):
    a: str

    @validator("a", pre=True, always=True)
    def default_a(cls, v):
        return v or 'a'

foo = Foo()

Currently, this throws a missing value error, and alternatives such as Optional or default values have different semantics (such as my earlier example with two paths).

[PrettyWood]

Hi again.
Coming back here while having an overlook on all open issues :)
Currently we indeed set required when inferring the field based on default value versus no default value.
We could possibly:

• add something like if any(v.pre and v.always for v in class_validators.values()): required = False but it seems messy and not very bullet-proof since we have no guarantee the validators actually return something valid in case of missing value.
• check fields after the pre validators
• have a way to force required = False like we do for required = True with ... (but I feel like it's something that we want to avoid with #990)
• probably a better solution than the three above

@samuelcolvin I feel like this could/should be taken into consideration for v2