Mypy: Making a decorator which preserves function signature

[sharmaeklavya2]

Currently this is how I annotate decorators which don't modify the function signature:

from typing import Any, Callable, TypeVar

FuncT = TypeVar('FuncT', bound=Callable[..., Any])

def print_on_call(func: FuncT) -> FuncT:
    def wrapped(*args, **kwargs):
        print("Running", func.__name__)
        return func(*args, **kwargs)
    return wrapped # type: ignore

I have to use the # type: ignore otherwise I get this error:

decorator.py: note: In function "print_on_call":
decorator.py:9: error: Incompatible return value type (got Callable[[Any, Any], Any], expected "FuncT")

I'm wondering if there is a better way to do this which doesn't need the type: ignore. What do you all suggest?

Maybe mypy can figure out from (*args, **kwargs) that the parameter types are the same. Maybe it can also see that the return value of func is the return value of wrapped so the return type is the same, and hence the function signatures are the same.

[sharmaeklavya2]

Also the type of wrapped in the error message should not be Callable[[Any, Any], Any]. It should be Callable[..., Any], isn't it?

[gvanrossum]

I use a cast(FuncT, wrapper) but # type: ignore works too -- I'm not aware of a better way (though you should probably add annotations to wrapper otherwise mypy won't look inside it at all).

I do agree there are two separate bugs here:

• def wrapper(*args, **kwds) is not considered a match for Callable[..., Any]
• mypy seems to display the type of wrapper here incorrectly

[sixolet]

This looks like it's an instance of https://github.com/python/typing/issues/239#issuecomment-236765463

[gvanrossum]

This looks like it's an instance of python/typing#239 (comment)
https://github.com/python/typing/issues/239#issuecomment-236765463

Yeah, although in this case you can do it already (since there's no change
to the signature). The problem is that the universal-varargs function isn't
seen as a match for the specific callable.

[rwbarton]

I think that def wrapper(*args, **kwds) would be considered a match for Callable[..., Any]. But here we need it to be a match for FuncT, which is some specific Callable type chosen by the caller of print_on_call. I guess it should still be considered a match because def wrapper(*args, **kwds) purports to be able to accept any sort of arguments, but mypy doesn't use that logic I guess.

There's something a bit unsettling here because if mypy did accept wrapper as having type FuncT, there's nowhere that we check that wrapper actually calls func with the type of arguments func accepts. The call to func is accepted because func has type FuncT which is a subtype of Callable[..., Any], which can accept arbitrary arguments; but in reality func probably can't accept arbitrary arguments. This is a loophole that arises from the subtyping relationships with Any. So it's probably not really worth fixing the first bullet point of @gvanrossum's earlier comment. Instead, we should have a way of typing this more accurately, along the lines of the typing proposal mentioned above.

[gvanrossum]

I follow your reasoning that we probably shouldn't fix the need for a cast or type-ignore on the return, but even if we were able to type this example using variadic type variables, the wrapper implementation would probably still use *args, **kwds so it would _still_ require a cast or type-ignore, right?

[rwbarton]

That's right unless you also had the ability to do something like this:

def print_on_call(func: Callable[[*Args, **Kwargs], Res]) -> Callable[[*Args, *Kwargs], Res]:
    def wrapped(*args: *Args, **kwargs: **Kwargs) -> Res:
        print("Running", func.__name__)
        return func(*args, **kwargs)
    return wrapped

Note the new syntax in the signature of wrapped.

[rwbarton]

(To maybe clarify, Args and Kwargs are supposed to be new kinds of type variables such that [*Args, **Kwargs] as the first argument of Callable quantifies over all possible argument signatures of a function. I haven't yet looked at @sixolet's proposal to see how it compares syntactically or semantically.)

[sixolet]

New kinds of type variables are in fashion.

I wish the unary * and ** were valid syntax where you put them 😞 , but I don't think they are. That's a minor issue, though, syntax is fungible.

This example highlights something interesting about the whole domain. "All the same arguments", in python, is spelled with two separate special ways of argument passing, that happen to work in tandem to provide very specifically correct transferral of all positional arguments including variadic ones (*args), and all optional arguments, including both named optional arguments and implicit-shoved-into-a-dict ones (**kwargs). The transferral feels particularly happenstance, in that when you splat a list of *args into your func, they will actually quite likely correspond to specific positional or optional arguments, not just star arguments, and when you splat in the **kwargs, they will actually quite likely correspond to specific positional or optional arguments too, not just star2 arguments. While func(*args, **kwargs) does in fact correctly call func with exactly all the same arguments, and maps the arguments correctly, there's no clean typecheck-time mapping you can make about which of the arguments were part of Args and which were part of Kwargs -- they only work that way in tandem.

My unmodified proposal over there has a way of saying "all kinds of arguments, no matter what", but then the typechecking of the body of the function ends up with some erasure of types (and argument names) that I thought was unavoidable, and ends up equivalent to replacing the internal argument lists with [...] and then casting on the way out. All that would be done without the user having to say # type: ignore or anything, but it's still not ideal.

I've been trying to concoct some kind of way of resolving this problem so that I think we _can_ typecheck the body cleanly, but I am not there yet. The closest I've come is wishing for some new even-more-magical python argument-passing syntax:

Ts = TypeVar('Ts', variadic=True)
Arg = ArgVar('Arg')

def print_on_call(func: Callable[[Expand[Arg[Ts]]], Res]) -> Callable[[Expand[Arg[Ts]]], Res]:
    def wrapped(***super_args: Ts) -> Res:
        print("Running", func.__name__)
        return func(***super_args)
    return wrapped

Or something. Wishful thinking.

I also may be misunderstanding something about @rwbarton's proposal.

[refi64]

I wish the unary * and ** were valid syntax where you put them 😞 , but I don't think they are. That's a minor issue, though, syntax is fungible.

Does that not work with PEP 448?

[sixolet]

@kirbyfan64 I don't think the ones to the right of the colons in the type annotation of the function work out.

[rwbarton]

I wish the unary * and ** were valid syntax where you put them :disappointed: , but I don't think they are.

Right, as far as I know they actually aren't legal.

I also may be misunderstanding something about @rwbarton's proposal.

I don't think you are and I haven't really been thinking about this very carefully. The issue you mention with *args and **kwargs not necessarily matching up with the actual keyword arguments occurred to me too after I made my comment.

[gvanrossum]

@sixolet, Is this fixed by your #3113? (Not counting the better syntax you're proposing in #3157.)

[sixolet]

Yes, I think so

On Apr 21, 2017, at 9:54 AM, Guido van Rossum [e-mail] wrote:

@sixolet, Is this fixed by your #3113? (Not counting the better syntax you're proposing in #3157.)

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[gvanrossum]

Hm, the OP's example still gives

error: Incompatible return value type (got Callable[[StarArg(Any), KwArg(Any)], Any], expected "FuncT"

if you remove the # type: ignore from the last line.

[gvanrossum]

Hm, I think this would be fixed by variadic type variables. Your #3113 fixes other things, like decorator factories. So I think this is not fixed and we need to keep it open until we have variadic type variables. (For which we don't seem to have a PR yet, though there's a typing issue: https://github.com/python/typing/issues/193). Sorry for the confusion.

[gvanrossum]

Raising priority based on internal needs.

[JukkaL]

Lowering priority because function plugins can often be used to work around this issue.