Pyright: fmap, identity, and Optional[T] do not play together

[Congee]

Describe the bug
Ummm, sorry I do not know how to briefly describe this bug. 😕 Please see code

To Reproduce
See code below

Expected behavior
See code below

Screenshots or Code

from typing import TypeVar, Callable, Optional

T = TypeVar('T')
def identity(x: T) -> T:
    return x


# double underscore prefix to avoid ambiguity with `T` due to lack of namespace for type variables
__fmap_T = TypeVar('__fmap_T')
__fmap_U = TypeVar('__fmap_U')
def fmap(f: Callable[[__fmap_T], __fmap_U], maybe: Optional[__fmap_T]) -> Optional[__fmap_U]:
    """
    Example:
        >>> assert fmap(lambda x: x + 1, None) is None
        >>> assert fmap(lambda x: x + 1, 0) == 1
    """
    return None if maybe is None else f(maybe)


x: Optional[int] = 0
q = fmap(lambda x: x + 1, x)  # q inferred as `int | None`, correct

q = fmap(lambda x: x, x)  # q inferred as `int | None`, correct
y = fmap(identity, x)  # expect y to be an `int | None`, but got `T | None`
z: int = y
if y is not None:
    w: int = y  # type checks because y is of `T`
    z: str = y  # type checks because y is of `T`. It should fail instead

VS Code extension or command-line
coc-pyright with pyright 1.1.66

Additional context
N/A

[Congee]

Am I doing anything wrong to the identity function

[erictraut]

Yeah, this won't work the way you've written it. When a Python type checker encounters a function call, and that function's parameters are generic, it "solves for" the types of those generic parameters. In your case, a call to fmap is solving for the types of __fmap_T and __fmap_U. It does not try to solve for other type variables, such as those in function identity.

There is arguably a bug in Pyright here. It should probably provide an error associated with the statement fmap(identity, x). I just tried your code in mypy, and it flags an error here.

[Congee]

Yeah, it makes sense. Is there any relevant formal specification (PEP) about the type inference strategy that pyright is using here?
Also, should I define identity as identity = lambda x: x?

[Congee]

In Haskell (GHC 8.8.4) there isn't much difference when inferring the result of id and lambda \x -> x, to say the least. The type parameter b in Maybe b is constrained by Num.

> :t id
id :: a -> a
> :t (\x -> x)
(\x -> x) :: p -> p
> :t fmap id (Just 1)
fmap id (Just 1) :: Num b => Maybe b  -- not `=> Maybe a` where `a` is the type parameter of `id`
> :t fmap (\x -> x) (Just 1)
fmap (\x -> x) (Just 1) :: Num b => Maybe b

Naturally, I expect the same behavior from pyright.

[Congee]

Also, it maybe interesting and helpful to be aware of the difference between predefined id and an "immediate" id expression.

myid = lambda x: x

x: Optional[int] = 0
y = fmap(myid, x)  # y: Unknown | None
z = fmap(lambda x: x, 0) # z: Literal[0] | None

[Congee]

Anyway, I cannot be demanding for pyright because Python's typing system looks toyish and pyright already is trying to be considerate. Maybe an error for these cases suffices 😉

[layday]

https://github.com/python/typing/issues/548

[erictraut]

Good news. I was able to get "higher-order type variables" to work. This functionality will be included in the next release of Pyright and Pylance.

The type checker now correctly determines that the type of T should be int in your example above. I've used your sample as part of the unit test for this functionality.
https://github.com/microsoft/pyright/commit/ebe82b8b44923fe2b0e9fdf2a5583917d7d41342

Also note that you won't need to use a different TypeVars (e.g. __fmap_T versus T). You can use the same name (e.g. T) in both cases because Pyright now differentiates between different uses of the same TypeVar.

[Congee]

Das AWESOME!

[erictraut]

This is now implemented in Pyright 1.1.69, which I just published. It will appear in the next Pylance release as well.