Pylance-release: Pylance is not detecting multiple constraints when using TypeVar

[ErikSniper]

Environment data

• Language Server version: 2020.9.8
• OS and version: Windows 10
• Python version (& distribution if applicable, e.g. Anaconda): Python 3.8.3

Expected behaviour

Pylance properly detecting multiple constraints when using TypeVar.

Actual behaviour

Pylance doesn't detect multiple constraints in TypeVar but it detects only first one. Because of that, any function which argument is of that type will not be detected properly by Pylance because it didn't detect multiple constraints of that type which causes Pylance lint any argument passed to that function.

Code Snippet / Additional information

Example for reproducing problem:

Line 3: Defining type T which can be either int or float.
Line 10: Calling function with previously defined type as argument and passing float as argument to the same function will cause Pylance to raise reportGeneralTypeIssues because Pylance didn't detect that type T can also be float not only int.

from typing import TypeVar

T = TypeVar("T", int, float)


def add(a: T, b: T) -> T:
    return a + b


print(add(3, 5.5))

[jakebailey]

That doesn't really feel like legal code; if a function is parameterized over some typevar T, then that T needs to be the same for all uses of T in that scope, no? In your example, what should the return type be? int or float? It might be obvious to the reader for this definition of T, but there are other constrained types that wouldn't have such behavior.

For reference, see this section of PEP 484: https://www.python.org/dev/peps/pep-0484/#function-method-overloading

Namely, this example:

A constrained TypeVar type can often be used instead of using the @overload decorator. For example, the definitions of concat1 and concat2 in this stub file are equivalent:

from typing import TypeVar, Text

AnyStr = TypeVar('AnyStr', Text, bytes)

def concat1(x: AnyStr, y: AnyStr) -> AnyStr: ...

@overload
def concat2(x: str, y: str) -> str: ...
@overload
def concat2(x: bytes, y: bytes) -> bytes: ...

So by that logic, your example is equivalent to:

@overload
def add(a: int, b: int) -> int: ...
@overload
def add(a: float, b: float) -> float: ...
def add(a, b):
    return a + b

Strangely, mypy says the return type is builtins.float* (asterisk?) for your code, but if I just add an int to a float, it says builtins.float (no asterisk). I'm not sure how it's deducing that unless it's special casing the conversion rules for these number types.

https://mypy-play.net/?mypy=latest&python=3.8&gist=4eb9656871755d8c7eebf3b8fe9dd2e9

[erictraut]

Thanks for the bug report. This is a legitimate bug. Pyright wasn't properly handling the case where one of the constrained types was a narrower version of another constrained type. In this particular example, "int" is narrower than "float". Pyright was previously picking the first match but not widening to accommodate another use of the type variable. You can see this by doing either of the following:

1. Swap the int and float in the TypeVar definition for T
2. Swap the int and float arguments to the add function call

I've fixed the problem so it now picks the narrowest constraint but widens on further uses if necessary. This provides correct and consistent behavior regardless of ordering.

The fix will be included in the next release.

[jakebailey]

Thanks for the clarification. I didn't know this'd work, clearly. 🙂

[ErikSniper]

@jakebailey

Return type of add function in my example depends on types of values passed to the function. In this case, it works like this:

• int + int = int
• float + float = float
• int + float = float

Apperently, Python will decide which type it will return depending on which types of values are passed to that function and depending on which constraints are defined with TypeVar for that type.

You can see that in this simple example:

from typing import TypeVar

T = TypeVar("T", int, float)


def add(a: T, b: T) -> T:
    return a + b


print(type(add(3, 5)))
print(type(add(3.5, 5.5)))
print(type(add(3, 5.5)))

@erictraut

Thank you for fixing the issue so quick. You can close the issue.

[erictraut]

We will leave it open until we publish the release that fixes it.

I should point out that your example is interesting because the behavior is ambiguous based on a strict reading of PEP 484. That's because int and float are part of the same inheritance hierarchy; int is effectively a subclass of float. In the case of add(3, 5), the constraints for T are met with either int or float. PEP 484 doesn't provide any guidance about which solution is preferred. I've updated the logic to always prefer the narrowest solution possible. That's consistent with the behavior of mypy. So in this particular example, Pyright will now prefer the int solution over the float solution.

[ErikSniper]

Yeah I understand what you want to say but from what I know, this is how it works:

When calling function add with both arguments as int, firstly, int arguments need to be initialized and Python will initialize them as integers since they don't have floating point.

Then, they will be passed to the function and in this case they are both int so there is no need for Python to convert them to float, that's why function add in that case will return int and not float.

When one argument is int and other is float, Python will automatically make float return type because it's more precise than int.

This is simple example:

a = int(3)
b = float(5.5)

print(type(a + b))

This example shows that when operations between int and float are being done, Python will automatically make float return type because it's more precise than int.

That is basic functionality of Python programming language and has nothing to do with PEP 484.

[erictraut]

What you're describing is the implementation of the add function, and I agree with your description.

What I'm referring to is the process by which a Python type checker interpreters the type information provided. The type checker has no knowledge of the internal implementation. It simply follows the rules outlined in PEP 484. My point is that there are ambiguities in PEP 484. By a strict reading of the spec, both int and float are correct answers when solving for T in the expression add(3, 5) when given the type definitions in your example.

By augmenting PEP 484's rules with the additional (unstated) rule that the type checker should always return the narrowest solution possible, the ambiguity is removed. And in this particular case at least, that also allows the type results to match the implementation.

[jakebailey]

This issue has been fixed in version 2020.10.0, which we've just released. You can find the changelog here: https://github.com/microsoft/pylance-release/blob/master/CHANGELOG.md#2020100-7-october-2020