Pyright: Typevar can't be inferred when overloading __init__ with annotated self param

[sbdchd]

Describe the bug

When overloading the __init__ method with the self param annotated, the TypeVar doesn't get inferred.

However, the TypeVar is resolved when overloading __new__.

_T isn't resolved when overloading via __init__:

# mypy: disallow_any_explicit=False
from __future__ import annotations

from typing import Any, Generic, Optional, TypeVar, overload

from typing_extensions import Literal

_T = TypeVar("_T", bound=Optional[str])


class TextField(Generic[_T]):
    @overload
    def __init__(self: TextField[str], *, null: Literal[False] = ...) -> None:
        ...

    @overload
    def __init__(
        self: TextField[Optional[str]],
        *,
        null: Literal[True] = ...,
    ) -> None:
        ...

    def __init__(self, *, null: bool = ...) -> None:
        ...

    def __get__(self: TextField[_T], instance: Any, owner: Any) -> _T:
        ...


class Table:
    name = TextField()
    name_nullable = TextField(null=True)


table = Table()

reveal_type(table.name)
# Revealed type is 'builtins.str*' (mypy)
# Type of "table.name" is "_T" (Pylance)
reveal_type(table.name_nullable)
# Revealed type is 'Union[builtins.str, None]' (mypy)
# Type of "table.name_nullable" is "_T" (Pylance)

Works when overloading __new__:

# mypy: disallow_any_explicit=False
from __future__ import annotations

from typing import Any, Generic, Optional, TypeVar, overload

from typing_extensions import Literal

_T = TypeVar("_T", bound=Optional[str])


class TextField(Generic[_T]):
    @overload
    def __new__(cls, *, null: Literal[False] = ...) -> TextField[str]:
        ...

    @overload
    def __new__(
        cls,
        *,
        null: Literal[True] = ...,
    ) -> TextField[Optional[str]]:
        ...

    def __new__(cls, *, null: bool = ...) -> TextField[Any]:
        ...

    def __get__(self: TextField[_T], instance: Any, owner: Any) -> _T:
        ...


class Table:
    name = TextField()
    name_nullable = TextField(null=True)


table = Table()

reveal_type(table.name)
# Revealed type is 'builtins.str*' (mypy)
# Type of "table.name" is "str" (Pylance)
reveal_type(table.name_nullable)
# Revealed type is 'Union[builtins.str, None]' (mypy)
# Type of "table.name_nullable" is "str | None" (Pylance)

To Reproduce

Use the above code

Expected behavior

TypeVars should be inferred properly when overloading via __init__ like they are when overloading via __new__

Screenshots or Code

see above

VS Code extension or command-line

Pylance: 2020.11.2

also tested via:
pyright 1.1.89

[erictraut]

I wouldn't expect this to work. This is not a legitimate way to use specialization, at least by my reading of PEP 484. I wonder if it just happens to work in mypy by chance.

Here's how PEP 484 says this should work. When a method is called, TypeVars that appear within the method signature (comprised of the parameters and return type) are "solved" based on the arguments that are passed to the method. In your example, the __init__ method signature contains no TypeVars, so there's nothing to "solve". By contrast, the __new__ method does contain a TypeVar, which is why it works in that case.

It sounds like you are expecting this to work in reverse in this particular case — i.e. you want the _argument_ type to be solved based on the type annotation of the self parameter.

If you can point me to any documentation or specs that indicate this should work as you expect, I'd appreciate it. Otherwise, I will mark this as "as designed" and close it.

[sbdchd]

Yeah can't find anything in PEP 484, the mypy docs briefly describe annotating the self param in __init__ for more complicated cases pointing to Popen

In particular, an __init__() method overloaded on self-type may be useful to annotate generic class constructors where type arguments depend on constructor parameters in a non-trivial way, see e.g. Popen.

However, in the typeshed Popen is typed using __new__ overloads.

The functionality of annotating self seems to discussed in https://github.com/python/typing/issues/680#issuecomment-541759627 and later implemented via https://github.com/python/mypy/pull/7860.

I think these overloads in the typeshed are using this functionality:

https://github.com/python/typeshed/blob/6d697e7f2c7b512865c7cc2ef6999f121d4594d1/stdlib/3/email/feedparser.pyi#L8-L11

class FeedParser(Generic[_M]):
    @overload
    def __init__(self: FeedParser[Message], _factory: None = ..., *, policy: Policy = ...) -> None: ...
    @overload
    def __init__(self, _factory: Callable[[], _M], *, policy: Policy = ...) -> None: ...

https://github.com/python/typeshed/blob/6d697e7f2c7b512865c7cc2ef6999f121d4594d1/stdlib/2and3/array.pyi#L18-L25

class array(MutableSequence[_T], Generic[_T]):
    typecode: _TypeCode
    itemsize: int
    @overload
    def __init__(self: array[int], typecode: _IntTypeCode, __initializer: Union[bytes, Iterable[_T]] = ...) -> None: ...
    @overload
    def __init__(self: array[float], typecode: _FloatTypeCode, __initializer: Union[bytes, Iterable[_T]] = ...) -> None: ...
    @overload
    def __init__(self: array[Text], typecode: _UnicodeTypeCode, __initializer: Union[bytes, Iterable[_T]] = ...) -> None: ...
    @overload
    def __init__(self, typecode: str, __initializer: Union[bytes, Iterable[_T]] = ...) -> None: ...

https://github.com/python/typeshed/blob/6d697e7f2c7b512865c7cc2ef6999f121d4594d1/stdlib/3/tempfile.pyi#L172-L198

class SpooledTemporaryFile(IO[AnyStr]):
    # bytes needs to go first, as default mode is to open as bytes
    if sys.version_info >= (3, 8):
        @overload
        def __init__(
            self: SpooledTemporaryFile[bytes],
            max_size: int = ...,
            mode: Literal["rb", "wb", "ab", "xb", "r+b", "w+b", "a+b", "x+b"] = ...,
            buffering: int = ...,
            encoding: Optional[str] = ...,
            newline: Optional[str] = ...,
            suffix: Optional[str] = ...,
            prefix: Optional[str] = ...,
            dir: Optional[str] = ...,
            *,
            errors: Optional[str] = ...,
        ) -> None: ...
        @overload
        def __init__(
            self: SpooledTemporaryFile[str],
            max_size: int = ...,
            mode: Literal["r", "w", "a", "x", "r+", "w+", "a+", "x+", "rt", "wt", "at", "xt", "r+t", "w+t", "a+t", "x+t"] = ...,
            buffering: int = ...,
            encoding: Optional[str] = ...,
            newline: Optional[str] = ...,
            suffix: Optional[str] = ...,
            prefix: Optional[str] = ...,
            dir: Optional[str] = ...,
            *,
            errors: Optional[str] = ...,
        ) -> None: ...
        @overload
        def __init__(
            self,
            max_size: int = ...,
            mode: str = ...,
            buffering: int = ...,
            encoding: Optional[str] = ...,
            newline: Optional[str] = ...,
            suffix: Optional[str] = ...,
            prefix: Optional[str] = ...,
            dir: Optional[str] = ...,
            *,
            errors: Optional[str] = ...,
        ) -> None: ...

https://github.com/python/typeshed/blob/6d697e7f2c7b512865c7cc2ef6999f121d4594d1/third_party/2and3/redis/client.pyi#L192-L253

[erictraut]

Thanks for all of that information. Very helpful.

The mypy docs and all of the cases in typeshed appear to be legitimate uses of a specialized self annotations for purposes of selecting an overload. For example, if you allocate a FeedParser by explicitly specializing it with a type argument of Message (i.e. FeedParser[Message](), then the first overload is considered. Pyright supports the use of self annotations in an __init__ method for purposes of selecting overloads.

I wouldn't expect any of these to work for specializing the constructed class instance, but looking at the bug report you cited, it appears that mypy did add special-case logic to handle this case specifically for __init__. This behavior isn't documented in any PEPs or in any mypy documentation though.

I'm torn on whether to support this. It seems like a hack — an abuse of the type system and an inconsistency with the way TypeVar solving works in all other cases. I'll think about it more.

[erictraut]

I've decided to implement this since it appears to be an implicit standard among other type checkers. This will be included in the next release.

[erictraut]

This is now addressed in pyright 1.1.90, which I just published. It will also be included in the next release of pylance.