Pydantic: Re-use validators?

[chopraaa]

Is it possible to create a validator and import it everywhere else it's used?

I have validators for certain fields that are present in multiple classes that I'd like to call simply without defining them.

[samuelcolvin]

I'm on my mobile, but should be as simple as doing something like:

class MyModel(BaseModel):
    check_foo = validator('foo')(reused_validator)

Have a try and let me know if that works or doesn't work or I have the syntax slightly wrong.

[dmontagu]

@samuelcolvin I don't think that works due to this check:

        if not in_ipython():  # pragma: no branch
            ref = f.__module__ + '.' + f.__qualname__
            if ref in _FUNCS:
                raise ConfigError(f'duplicate validator function "{ref}"')
            _FUNCS.add(ref)

I think we can probably fix this though.

[dmontagu]

Here's a self contained demonstration that it doesn't currently work.

from pydantic import BaseModel, validator


def double_validator(cls, value):
    return value * 2


class A(BaseModel):
    x: int

    double = validator('x')(double_validator)


class B(BaseModel):
    x: int

    double = validator('x')(double_validator)

# pydantic.errors.ConfigError: duplicate validator function "__main__.double_validator"

[samuelcolvin]

Humm, possibly. We could either drop that check or make it optional.

Or you could setup a validator which just calls the reused validator.

Maybe allow_name_reuse as another argument on a validator?

[dmontagu]

Here's a hack that works for now:

from copy import deepcopy
from typing import Callable

from pydantic import BaseModel, validator
from pydantic.typing import AnyCallable


def reused_validator(
    *fields: str,
    pre: bool = False,
    each_item: bool = False,
    always: bool = False,
    check_fields: bool = True,
    whole: bool = None,
) -> Callable[[AnyCallable], classmethod]:
    def dec(f: AnyCallable) -> classmethod:
        f = deepcopy(f)
        f.__qualname__ = f"{f.__qualname__}{reused_validator.count}"
        reused_validator.count += 1
        return validator(*fields, pre=pre, whole=whole, always=always, check_fields=check_fields)(f)

    return dec


reused_validator.count = 0


def double_validator(cls, value):
    return value * 2


class A(BaseModel):
    x: int
    double = reused_validator('x')(double_validator)


class B(BaseModel):
    x: int
    double = reused_validator('x')(double_validator)


print(A(x=1))
print(B(x=1))
# x=2
# x=2

This is using v1 imports, but I also checked that if you fix the imports it works in v0.32.2.

@samuelcolvin related -- are models supposed to print like that now? On master I'm not seeing the model class name when I print the model.

[dmontagu]

@samuelcolvin I think it makes sense to keep that error in many cases because the most common situation is that you've accidentally duplicated the name of the validator inside the class, which would cause problems if the logic changed (and is presumably why you added the error in the first place).

I was thinking it could make sense to just look for a . in the __qualname__ before raising the error. If the function was defined globally in a module, you almost certainly meant to reuse it.

Separately, I also think it would be good to add allow_name_reuse as a keyword argument so that you could do the same thing with a function defined at the class level.

Also, we could add a check for whether something is already a classmethod when building the validator so that you could define it as such inside the class and not have a misleading signature.

I can start putting together a PR for this.

[samuelcolvin]

are models supposed to print like that now? On master I'm not seeing the model class name when I print the model

Yes, that's standard for __str__ methods, I think you reviewed the PR. Bit late to change now.

[dmontagu]

Yeah, I remember it now 😅; I'm fine with it since we are calling the repr on nested values. I just haven't seen it in the wild much yet (still using 0.32.2 with fastapi :/), so it caught me by surprise.

[chopraaa]

Thanks for the quick response on this. I wasn't even aware we could call validators using check_foo = validator('foo')(reused_validator).

[dmontagu]

@chopraaa that's basically all a python decorator does behind the scenes

[skewty]

I came across this issue this morning. Thanks for the temporary work around idea.

@dmontagu unless I am failing to see / understand something, I don't see where/how count ever gets incremented. That said it seems to compile and run. This fact almost interested me more :)

def reused_validator(
    *fields: str,
    pre: bool = False,
    each_item: bool = False,
    always: bool = False,
    check_fields: bool = True,
    whole: bool = None,
) -> Callable[[AnyCallable], classmethod]:
    def dec(f: AnyCallable) -> classmethod:
        f = deepcopy(f)
        f.__qualname__ = f"{f.__qualname__}{reused_validator.count + 1}"
        return validator(*fields, pre=pre, whole=whole, always=always, check_fields=check_fields)(f)

    return dec


reused_validator.count = 0

so I added

print(f"Function name will be: {f.__qualname__}")

and this is what came out:

Function name will be: partition_range_check1
Function name will be: partition_range_check11

which explains why it works. That said count is perhaps a poor name given the reason it works :) (unless we are counting notches on a stick)

I wasn't sure if this was by design or by mistake. I Just wanted to share for others who might also be struggling to understand.

[dmontagu]

That was just a mistake -- I meant to actually increment the value. It's now fixed above!

(Clearly there are some edge cases this wouldn't handle well, e.g., if you end the validator name with a number. This is only intended as an admittedly hacky interim solution.)

[pmav99]

The issue was closed by #941 but allow_reuse seems not to have been documented yet. If anyone is interested, it can be used like this:

import typing
import pydantic

def normalize(value: str) -> str:
    return value.lower()

class Goo(pydantic.BaseModel):
    a: str
    b: str

    # validators
    _ensure_a_is_normalized: classmethod = pydantic.validator("a", allow_reuse=True)(normalize)
    _ensure_b_is_normalized: classmethod = pydantic.validator("b", allow_reuse=True)(normalize)

g = Goo(a="A", b="BB")
assert g.a == "a"
assert g.b == "bb"

We can further reduce repetition in the models by defining a help function:

import typing
import pydantic

def normalize(value: str) -> str:
    return value.lower()

def normalizing_validator(field: str) -> classmethod:
    decorator = pydantic.validator(field, allow_reuse=True)
    validator = decorator(normalize)
    return validator

class Hoo(pydantic.BaseModel):
    a: str
    b: str

    # validators
    _ensure_a_is_normalized: classmethod = normalizing_validator("a")
    _ensure_b_is_normalized: classmethod = normalizing_validator("b")

h = Hoo(a="A", b="BB")
assert h.a == "a"
assert h.b == "bb"

[samuelcolvin]

@pmav99: Would be great if you could submit a PR with documentation for this?

[samuelcolvin]

(thanks for reminding us it's not documented)

[pmav99]

Sure.

1. Do you want to have both version in the docs? The first one is simpler and somewhat easier to understand, while the second one is a bit more realistic.
2. Which section? Maybe between "validate always" and "root-validators"? https://pydantic-docs.helpmanual.io/usage/validators/#root-validators
3. I should remove types for the docs, right?

[samuelcolvin]

Really up to you on all three, do what you think is clearest and we can discuss on the PR.

Thanks so much.

[SylannBin]

Sorry to dig up this old topic, but I am curious to know why this check for duplicate validators even is a thing.

Considering the discussion above, it doesn't seem to be a critical problem?

I am asking because I find having to add allow_reuse=True everywhere a bit annoying.
And the alternative is even worse: declaring custom Models with the validator class methods and all is like heavy artillery.