Pyright: Type "TResult" cannot be assigned to type "TResult"

[dbrattli]

I'm still getting some Type "TResult" cannot be assigned to type "TResult" "so not sure if this is related to #1235. I've used quite a lot of time trying to fix this issue in my code so I think this is a bug.

Describe the bug

Getting errors such as:

• "Awaitable[Try[Context[TResult]]]" cannot be assigned to return type "HttpFuncResultAsync[TResult]" even if one is an alias of the other.

To Reproduce

from typing import Any, Awaitable, Callable, Generic, TypeVar

TSource = TypeVar("TSource")
TResult = TypeVar("TResult")
TNext = TypeVar("TNext")


class Try(Generic[TSource]):
    pass


class Context(Generic[TSource]):
    pass


# HttpFuncResult[TResult]
HttpFuncResult = Try[Context[TResult]]
# HttpFuncResultAsync[TResult]
HttpFuncResultAsync = Awaitable[Try[Context[TResult]]]

# HttpFunc[TNext, TResult]
HttpFunc = Callable[
    [Context[TNext]],
    HttpFuncResultAsync[TResult],
]

# HttpHandler[TNext, TResult, TSource]
HttpHandler = Callable[
    [HttpFunc[TNext, TResult], Context[TSource]],
    HttpFuncResultAsync[TResult],
]


def with_url_builder(builder: Callable[[Any], str]) -> HttpHandler[TSource, TResult, TSource]:
    def _with_url_builder(
        next: HttpFunc[TSource, TResult],
        context: Context[TSource],
    ) -> HttpFuncResultAsync[TResult]:
        return next(context)

    return _with_url_builder


# This gives error
def with_url(url: str) -> HttpHandler[TSource, TResult, TSource]:
    def _with_url(
        next: HttpFunc[TSource, TResult],
        context: Context[TSource],
    ) -> HttpFuncResultAsync[TResult]:
        fn = with_url_builder(lambda _: url)
        return fn(next, context)

    return _with_url

The last function gives 3 errors with pyright:

Expected behavior

I think that the code should pass type checking. While trying to figure out what was wrong I discovered that the function could be eta reduced, and removing the abstraction also removes the error:

# This works
def with_url_eta_reduced(url: str) -> HttpHandler[TSource, TResult, TSource]:
    return with_url_builder(lambda _: url)

I guss it might be because this code is much simpler, but I still feel the other version should type check as well, since they should be identical.

VS Code extension or command-line

vscode extension: 1.1.94

[erictraut]

I think Pyright is doing the right thing here. The TSource TypeVar used in the with_url_builder function is completely different from the TSource TypeVar used in the with_url function. Those are two completely different scopes. You could have allocated two different TypeVars with different names (e.g. TSource1 and TSource2) and used one in the first function and the other in the second function.

For what it's worth, mypy also reports errors in this code.

One tip for you... If you write a generic function that includes only one instance of a TypeVar in its signature, that's probably a bug. Both with_url_builder and with_url use TResults only once in their signatures.

[dbrattli]

The last example I gave type checks even if it does the exact same thing, so how can it type check when the other doesn't? That cannot be by design.

[erictraut]

The last example is not the same as your previous example.

Your second example should arguably generate an error of some sort as well. It's a false negative due to your misuse of TypeVars in the signature of with_url_eta_reduced. As I mentioned previously, any time you use a TypeVar only once in a signature, you're probably doing something wrong. Furthermore, a TypeVar that appears only within the return type is almost always a bug. A TypeVar that appears within a return type should also appear within one or more of the input parameter types. The job of the TypeVar solver is to use the caller's argument types to solve for the TypeVars that appear within the parameter types and then apply those solved TypeVar values to determine the return type. If the TypeVars appear only within the return type annotation without appearing in any of the input parameter types, there's no way for the TypeVar solver to determine the return type based on caller's arguments.

There was recently discussion on the typing-sig (the forum that's used to discuss issues related to Python typing standards) about flagging these misuses of TypeVars in generic function signatures. Pyright doesn't currently flag these as errors.

The issues that you've filed are increasing my conviction that these should be reported as errors. Maybe I should take another run at the typing-sig to try to convince the members that these should be considered errors. Another option is to add a diagnostic check in Pyright that is off by default in "basic" type checking mode but on in "strict" mode.

[dbrattli]

But I think a function returning an inner function needs to take the signature of the inner function into account. E.g:

def take(count: int) -> Callable[[Iterable[TSource]], Iterable[TSource]]:
    def inner(source: Iterable[TSource]) -> Iterable[TSource]:
        return source

    return inner

The signature of the inner function should be valid since it uses the generic argument twice in its signature. Returning a validly typed function should be valid, not? But how can I type hint a function returning such a function?

Declared as Callable[..., Any] will loose the type information.

def take(count: int) -> Callable[..., Any]:
    def _take(source: Iterable[TSource]) -> Iterable[TSource]:
        return source

    return _take

The only workaround I can think of is to use classes instead to avoid having to declare the generic callable return type. Is this valid or is it also a false negative?

class take:
    def __init__(self, count: int):
        pass

    def __call__(self, source: Iterable[TSource]) -> Iterable[TSource]:
        return source

... and can take generic parameters as well:

class map(Generic[TSource, TResult]):
    def __init__(self, mapper: Callable[[TSource], TResult]):
        self.mapper = mapper

    def __call__(self, source: Iterable[TSource]) -> Iterable[TResult]:
        return (self.mapper(x) for x in source)

This seems to work but feels kind of silly (objects are poor man's closures)

[erictraut]

Now that you better understand the way it works, I'll leave it to you to find a solution that meets your particular needs.

[erictraut]

I'm going to leave this issue open as a reminder to look more deeply at whether your second example above should generate an error.

[erictraut]

I found the reason why your second example wasn't generating an error. That fix will be in the next release.

[dbrattli]

Nice! I will be great to get rid of all the false negatives. I've think I might have found a way around it using "callback protocols". The idea is to instead use a generic callback method (delegate) where the generic type appears in both the input and output of the method. It seems to work, so hopefully it's not another false negative?

class FilterFn(Protocol):
    def __call__(self, source: Iterable[TSource]) -> Iterable[TSource]:
        raise NotImplementedError

def take(count: int) -> FilterFn:
    def _take(source: Iterable[TSource]) -> Iterable[TSource]:
        n = count

        def gen():
            nonlocal n
            for x in source:
                if n > 0:
                    yield x
                    n -= 1
                else:
                    break

        return gen()
    return _take

[erictraut]

@gvanrossum, you were interested in cases where a single instance of a TypeVar was used in a generic function. Here's an example. Refer to the code sample at the top of this issue.

I've proposed that we implement a new diagnostic rule in pyright that detects this case and reports a warning (by default) or error (in strict mode). Here's the pending PR that implements this check: https://github.com/microsoft/pyright/pull/1272. It's worth noting that a bunch of pyright's unit tests needed to be updated as part of this PR so as to avoid this problem.

[dbrattli]

FYI: I've btw updated my code to use generic protocols instead. Hope this is the right way to do things like this!? https://github.com/cognitedata/Expression/blob/main/oryx/handler.py

The handlers themselves might look scary but the resulting app looks really nice, type checks with pyright, but not with mypy so could be false negatives: https://github.com/cognitedata/Expression/blob/main/oryx/examples/app.py

[erictraut]

Pyright 1.1.95, which I just published, now properly flags the second example with an error.

The new diagnostic rule is implemented in a PR pending review. It is not included in Pyright 1.1.95 but will likely be in a future release.