Mypy: functools.partial support

[berdario]

I haven't been able to find another issue for this, weird that this hasn't been reported already.

from typing import *
from functools import partial

def inc(a:int) -> int:
    return a+1

def foo(f: Callable[[], int]) -> int:
    return f()

foo(partial(inc, 1))

This will fail with

error: Argument 1 to "foo" has incompatible type partial[int]; expected Callable[[], int]

a similar code by using a dyadic add instead of inc will yield:

error: Argument 1 to "foo" has incompatible type partial[int]; expected Callable[[int], int]

(so, partial apparently doesn't currently retain any type information on the non-applied inputs... I guess this'll make the fix non trivial)

[JukkaL]

The error message is generated because a class with a __call__ method (in this case, partial) isn't considered a subtype of Callable, even though it should be. See #797.

Also, partial only retains information about the return type, since the type system can't represent a more precise type.

[berdario]

the type system can't represent a more precise type.

You mean that the partial class doesn't represent it at runtime, right?
I think mypy could special case partial, and coerce it to the appropriate Callable type transparently

[JukkaL]

I meant that mypy would have to special case partial, since we can't write a good enough stub using PEP 484 features only. Mypy already does some special casing, but for this I'd rather have a more general mypy plugin/extension system instead of even more ad-hoc special case logic in the type checker.

[gvanrossum]

Maybe this could be supported by #3299

[JukkaL]

Yes, this looks like a good candidate for function plugins. Increasing priority since this is may be pretty easy to implement now. Just supporting positional arguments for partial should cover the majority of uses and would probably be good enough.

[ilevkivskyi]

This particular case and probably other basic use cases are already supported by protocols (see tests in PR #3132), simply because partial has __call__ and therefore is a structural subtype of Callable.

[JukkaL]

Protocols won't help with inferring the argument types of the result, though?

[ilevkivskyi]

Protocols won't help with inferring the argument types of the result, though?

Yes, I didn't do any special-casing, partial in typeshed is only generic in one type _T which is the original return type. Maybe with the new support for flexible Callable this might be improved, but probably a plugin will still be required if we want precise types.

[ilevkivskyi]

The original example is now fixed by #3132. This issue can be kept open to track better support of partial using a plugin.

[astrojuanlu]

Not sure if this has been fixed entirely or perhaps it's a different issue:

(py34) juanlu@centauri /tmp $ cat test.py 
from functools import reduce, partial

def sum_two(a, b, c=0):
  return a + b + c

f_0 = partial(sum_two, c=0)
print(reduce(f_0, [1, 2, 3], 10))
(py34) juanlu@centauri /tmp $ mypy --version
mypy 0.550
(py34) juanlu@centauri /tmp $ mypy --ignore-missing-imports --check-untyped-defs test.py 
test.py:7: error: No overload variant of "reduce" matches argument types [functools.partial[Any], builtins.list[builtins.int*], builtins.int]

[nicktimko]

I think this crops up also when trying to use a partial for a defaultdict factory:

from collections import defaultdict
from functools import partial
from typing import cast, Callable, Mapping

# error: No overload variant of "defaultdict" matches argument types [functools.partial[builtins.int*]]
x: Mapping[str, int] = defaultdict(partial(int, 10))

# (ok, but feels like it shouldn't be needed)
y: Mapping[str, int] = defaultdict(cast(Callable, partial(int, 10)))

[JukkaL]

python/typeshed#2878 changed how partial works. Now mypy can often preserve the argument types, but it loses keyword argument names. It may still be worth it to special case partial using a plugin.

[wkschwartz]

+1 for supporting partial's instance attributes: func, args, and keywords.

[socketpair]

def test(x: str, y: int):
    pass


z1 = partial(test, 42)
z2 = partial(test, 42, 6.7)

It also does not generate any warnings.

[berdario]

To clarify, after 2019 June 20th, the change that JukkaL talked about has been rolled back: https://github.com/python/typeshed/issues/3077

This means that

partial's instance attributes: func, args, and keywords.

Should now be supported again, but simple examples like the one that socketpair added are not caught anymore. This still needs a typechecker plugin.

[sobolevn]

Fully typed partial is supported in dry-python/returns project: https://returns.readthedocs.io/en/latest/pages/curry.html#partial

You can install it with pip, add our mypy plugin, and use it in your code!
Here are the sources for this feature:

• Definition: https://github.com/dry-python/returns/blob/accee054a546749fd73fd6ee7d9c100a4d45a7ad/returns/curry.py#L7
• Plugin: https://github.com/dry-python/returns/blob/accee054a546749fd73fd6ee7d9c100a4d45a7ad/returns/contrib/mypy/_curry.py#L195

Example:

from returns.curry import partial

def some_function(first: int, second: int) -> float:
    return first / second

reveal_type(partial(some_function, 1))
# => def (second: builtins.int) -> builtins.float*
# => Which is fair!

Here are the tests to give you a better impression of its features: https://github.com/dry-python/returns/tree/master/typesafety/test_curry

[oakkitten]

not sure if the other comments mention this, but partial doesn't typecheck TypeVars for me:

from typing import *
from functools import partial

T = TypeVar("T")

def foo(t: T) -> T: ...

i: int = partial(foo, 1)()

main.py:8: error: Incompatible types in assignment (expression has type "T", variable has type "int")

[Behoston]

4 years and still the same problem with partial function. Is really that hard to implement partial introspection?

[sobolevn]

@Behoston yes, it is 🙂 I have spent several weeks working on it part-time.

You can check out the code here: https://github.com/dry-python/returns/blob/master/returns/contrib/mypy/_features/partial.py Or try to implement it yourself if that solutions does not feet you for some reason (it has some flaws).