Pydantic: Callables are attached as bound methods when used as default args

[dgjustice]

Bug

Methods used as default args to Callable are attached as bound methods of the BaseModel class. Things work fine if you pass the method as an argument to the constructor. One workaround (if you must have a default arg) is to pass the method as a functools.partial object.

Output of python -c "import pydantic.utils; print(pydantic.utils.version_info())":

             pydantic version: 1.5.1
            pydantic compiled: True
                 install path: /usr/local/lib/python3.7/site-packages/pydantic
               python version: 3.7.7 (default, May 20 2020, 21:10:21)  [GCC 8.3.0]
                     platform: Linux-4.19.76-linuxkit-x86_64-with-debian-10.4
     optional deps. installed: ['typing-extensions']

Test

from pydantic import BaseModel
import typing as t
from functools import partial

def bar(*args):
    print(f"bar args: {args}")

class Foo(BaseModel):
    callback: t.Callable[[int], int]
    method: t.Callable[..., t.Any] = bar

m = Foo(callback=lambda x: x)
print(m.callback(42))
print(m.method)
m.method(42)
print(f"\n\n{'#' * 42}\n\n")

class Foo2(BaseModel):
    callback: t.Callable[[int], int]
    method: t.Callable[..., t.Any]  # no default

n = Foo2(callback=lambda x: x, method=bar)  # This works fine
print(n.callback(42))
print(n.method)
n.method(42)
print(f"\n\n{'#' * 42}\n\n")

class Foo3(BaseModel):
    callback: t.Callable[[int], int]
    method: t.Callable[..., t.Any] = partial(bar)  # workaround

o = Foo3(callback=lambda x: x)
print(o.callback(42))
print(o.method)
o.method(42)

Output

# python foo.py 
42
<bound method bar of Foo(callback=<function <lambda> at 0x7f596462b9e0>)>
bar args: (Foo(callback=<function <lambda> at 0x7f596462b9e0>), 42)


##########################################


42
<function bar at 0x7f5964621dd0>
bar args: (42,)


##########################################


42
functools.partial(<function bar at 0x7f5964621dd0>)
bar args: (42,)

[samuelcolvin]

Thanks for reporting, not sure how hard this would be to fix.

In the meantime you can probably use default_factory as a workaround.

[dgjustice]

It's not a show stopper by any stretch. I really appreciate the library, and have enjoyed using it!

[antonl]

@samuelcolvin I ran into the same bug and I did try what you've suggested, except with the dataclasses. The default_factory approach doesn't work, and worse, for the dataclasses version it results in pydantic not knowing about that field. I think there's a fundamental difference about how pydantic handles binding of functions when compared to the stdlib dataclasses. The good news is that the workaround with partial (or even staticmethod) works. Here's a test bench:

from typing import Callable
from dataclasses import field as dc_field, dataclass as dc_dataclass
from pydantic import dataclasses, BaseModel, Field
import pytest


def foo(arg1, arg2):
    return arg1, arg2


class WorkaroundCallable(Callable):
    @classmethod
    def validate_callable(cls, v):
        if not callable(v):
            raise ValueError("invalid callable passed")

        return v

    @classmethod
    def __get_validators__(cls):
        yield cls.validate_callable


@dataclasses.dataclass()
class HasCallables:
    non_default_callable: Callable
    default_callable: Callable = lambda x: foo(x, "default")
    default_callable_factory: Callable = dc_field(default=lambda x: foo(x, "factory"))


@dataclasses.dataclass()
class HasCallablesStatic:
    non_default_callable: Callable
    default_callable: Callable = staticmethod(lambda x: foo(x, "default"))
    default_callable_factory: Callable = dc_field(default=staticmethod(lambda x: foo(x, "factory")))


@dataclasses.dataclass()
class HasCallablesWorkaround:
    non_default_callable: WorkaroundCallable
    default_callable: WorkaroundCallable = lambda x: foo(x, "default")
    default_callable_factory: WorkaroundCallable = dc_field(default_factory=lambda: lambda x: foo(x, "factory"))


class HasCallablesModel(BaseModel):
    non_default_callable: Callable
    default_callable: Callable = lambda x: foo(x, "default")
    default_callable_factory: Callable = Field(default_factory=lambda: lambda x: foo(x, "factory"))


@dc_dataclass()
class HasCallablesDC:
    non_default_callable: Callable
    default_callable: Callable = lambda x: foo(x, "default")
    default_callable_factory: Callable = dc_field(default_factory=lambda: lambda x: foo(x, "factory"))


@pytest.mark.parametrize("cls", [HasCallables, HasCallablesModel, HasCallablesStatic])
def test_pydantic_callable_static(cls):
    """tests pydantic callable behavior"""
    non_default_callable = lambda x: foo(x, "nondefault")
    a1 = cls(non_default_callable=non_default_callable)
    a2 = HasCallablesDC(non_default_callable=non_default_callable)

    # call non_default
    assert a1.non_default_callable("hello") == a2.non_default_callable("hello")

    # call default_factory
    assert a1.default_callable_factory("hello") == a2.default_callable_factory("hello")

    # call default
    assert a1.default_callable("hello") == a2.default_callable("hello")

[juanarrivillaga]

FWIW: This isn't a problem in the built-in dataclasses.dataclass because in their approach it ends up assigning default arguments as instance attributes, which shadow the class attribute function objects, thus, the descriptor protocol is never initiated to do the binding of the first argument.

[PrettyWood]

@antonl @juanarrivillaga Thanks for reporting! I missed this issue...
I just made a quick fix and took the liberty of copying your test @antonl.
Please tell me if it looks good to you

[antonl]

@PrettyWood thanks! This looks great!