Jedi: Autocomplete broken when typing a function decorator

[lion3ls]

Hi there,

I discovered lately that autocompletion got broken when typing a function decorator, here is the minimal example:

from typing import TypeVar

R = TypeVar("R")

def dec(fn : R) -> R:
    return fn

@dec
def my_func(a, b, c):
    ...

def __main__():
    my_func()

Actual:

As you can see the signature of the function fallback to FunctionType instead of the actual type.

Expected:

I removed the return type of the decorator to get this work, but it's not suitable for my use case.

Thanks for your help !

[davidhalter]

Yeah, this is a hard one. It's just not that easy to get this stuff working. TypeVars typically play with types and not values. So here you actually just want the function to be passed.

A possible solution is probably instead of type(dec) -> FunctionType to use a bit more logic for annotations where it would result in FunctionType<dec> and if you execute that you would not get a random function type that isn't really valuable, but you would get the dec function back.

Does this make sense?

[fsenart]

The root problem is to be able to preserve the exact type of the decorated function. So that dec(fn) ~= fn (in term of signature). I'm not sure if I understood your solution but it seems that with FunctionType<dec> we will have a more detailed FunctionType but it won't reflect the type of the decorated function, isn't it? Or with FunctionType<dec> vscode will be able to understand the signature of the decorated function?
PS: when there is no type involved in the definition of the decorator, everything rules and vscode understands that the decorator preserves the signature of the decorated function. However, we lose MyPy ability to enforce this fact.
PS2: regarding the above PS, it may be a bug, not a feature?!

[davidhalter]

I label stuff that was never implemented as features, because it was never supposed to actually work - so it's a new feature. With your argumentation you could argue that not understanding meta classes is a bug ;-).

You don't really have to understand what I meant with FunctionType<dec>. It was more a note to myself, how we could solve this. :)

[davidhalter]

I have looked into this for a bit. I think you're typing wrong. This is probably also not something mypy would understand. You should probably type it like this:

import typing

R = TypeVar("R")

def dec(fn : typing.Callable[[...], R]) ->  typing.Callable[[...], R]:
    return fn

This is also how lru_cache is typed in typeshed. Would you agree? This only works on Jedi/Parso master.

If you don't typing like this, we should probably wait for mypy: https://github.com/python/mypy/issues/3157 and see if they change anything.

So I guess I'm closing this, because there seems to be a clear way how you can type decorators.

[PeterJCLaw]

I've hit this recently and it would be great to see this supported. I appreciate that this is a complex area (even before we allow the signature changes which https://github.com/python/mypy/issues/3157 starts to allow), though I think it would be really valuable to get function signatures working for annotated decorators decorating annotated functions.

I've just tried something like this on master (in VSCode, with it pointed to my pip-install of the master branch in my python 3.5 virtualenv) and for both annotated and non-annotated functions I'm not seeing the "real" signature of the target function when the decorator is annotated.

I've put all my examples in a gist here: https://gist.github.com/PeterJCLaw/8b01cc26bdd0c23b3631c625b892f39c

@davidhalter does this still work for you on master?

It would be useful to know which of those you're expecting to work and which you believe are waiting on https://github.com/python/mypy/issues/3157.

Currently I would favour re-opening this, as I don't seem to be able to find any spellings of annotated decorators which offer the correct signatures. I'd be happy to either be shown something I'm missing or open a fresh issue if that would be preferred. I'd also be happy to contribute to this if I can.

[davidhalter]

For me most things work. There are two exceptions:

1. I use def annotated_decorator(fn: Callable[[...], R]) -> Callable[[...], R]:.

I added the brackets that you omitted. I thought these were necessary, when they are actually not. Feel free to open an issue for this. I'm also happy to let you contribute a fix. The relevant code is in inference/gradual/typing.py and inference/gradual/annotation.py. It's probably all about trying to find why an ellipsis does not work there.

1. Using this type var: TCallable = TypeVar('TCallable', bound=Callable[..., Any]) results in a return value of Any, because that's what it says. So Jedi does everything correct there. Unless there's a change in specification, I'm not willing to change that.

[PeterJCLaw]

Thanks for the pointers, I'll have a look into the ellipses part, hopefully this weekend.

Regarding TCallable, I don't believe the return type of the generic variable should be interpreted as being always Any. This is because the type of TCallable is generic first and secondly bounded to being callable. Importantly, it's the _bound_ which is Any rather than the type of the generic argument annotated by the TypeVar. (My understanding is that bounded generics are different to generics with constraints; the latter would expect to match (one of) the given types exactly)

For any usages of a type which has a generic component, I would expect that jedi to preserve the type of any value passed through such a generic regardless of whether the generic was bound or not. (There are probably cases where jedi should observe the bound of the generic, though I think they're all from within classes or functions which have a bounded TypeVar as their generic component)

I've put some examples in https://gist.github.com/PeterJCLaw/1b046afad70ac0aac67b8c453ad14aaa/5ea035ba28391ab66895b2d53fb3e4e0ea317cfa, which show various operations using values bounded and unbound generics and how the type-preserving aspect of them is unaffected by the bound.

Hopefully these examples also show why typing a decorator's argument with a single TypeVar (e.g: R or TCallable) is preferable to annotating with a Callable[..., R] -- by making the annotation fully generic, the full signature of the decorated function is preserved rather than just the return type.

[davidhalter]

The issue in general is just: What is the type of function f? At the moment it's just types.FunctionType. But it is not something like types.FunctionType[f]. So basically in this sort of type system we lose the information. Now we could kind of hack around that, but that would be really really ugly (and AFAIK also not part of all the RFCs).

To avoid these kind of hacks we would probably need something like def decorator(f: FunctionVar[T]) -> FunctionVar[T] or something similar. This type of construct doesn't exist at the moment and is what some people are still working out over at pyre/mypy.

[davidhalter]

BTW: This was fixed by #1613 and most people will probably now be happy with how it works with function decorators.

I'm still not sure this is the correct interpretation of the PEP, but I guess people will be happy :)

[ievgennaida]

I have looked into this for a bit. I think you're typing wrong. This is probably also not something mypy would understand. You should probably type it like this:

import typing

R = TypeVar("R")

def dec(fn : typing.Callable[[...], R]) ->  typing.Callable[[...], R]:
    return fn

This is also how lru_cache is typed in typeshed. Would you agree? This only works on Jedi/Parso master.

If you don't typing like this, we should probably wait for mypy: python/mypy#3157 and see if they change anything.

So I guess I'm closing this, because there seems to be a clear way how you can type decorators.

Pylance is saying

"..." not allowed in this contextPylance

RTYPE = TypeVar("RTYPE")
def log_method(func: Callable[[...], RTYPE]) ->  Callable[[...], RTYPE]:

Could you please advise how declaration should look like to solve this IntelliSense problem?
Thanks in advance.

[PeterJCLaw]

Pylance is correct -- Callable[[...], RTYPE] is not a valid type annotation construct.

Depending on what your method signature is, one of the following should get things working for you:

R = TypeVar('R')

def foo(func: Callable[..., R]) -> Callable[..., R]:
   ...

This says that your function foo takes some callable with a return type R and returns a callable with return type R (where R is generic). Note that it doesn't make any guarantee that the argument signature is preserved.

TCallable = TypeVar('TCallable', bound=Callable[..., Any])

def foo(func: TCallable) -> TCallable:
    ...

This says that your function foo takes some callable with signature expressed by the generic TCallable and returns a function which is completely compatible with that signature. It doesn't guarantee that it's the same function (that would make foo rather limited in what it can do), but you normally don't need this guarantee.

In general for decorators which don't modify the signature I would recommend the latter (i.e: using TCallable) as it offers much stronger assurances to the behaviour of the decorator.

Aside: if you have a decorator factory function, then this is the signature you want:

def factory(...) -> Callable[[TCallable], TCallable]:
    def decorator(func: TCallable) -> TCallable:
        ...
    return decorator

[ievgennaida]

@PeterJCLaw Thanks, works like a charm with both sync and async func arguments.