Mypy: error: Overloaded function signatures 1 and 2 overlap with incompatible return types

[chantisnake]

I am trying to implement currying in python, it passes Pycharm type checks (which is not as comprehensive as mypy), but my py gives me tons of error like this:

pythonz\basic.py:54: error: Overloaded function signatures 1 and 2 overlap with incompatible return types

Here is a sample of my code

@overload
def curry(f: Callable[[A1], R]) -> PZFunc[A1, R]:
    """curry a function with 1 parameter"""
    ...


@overload
def curry(f: Callable[[A1, A2], R]) -> PZFunc[A1, PZFunc[A2, R]]:
    """curry a function with 2 parameters"""
    ...

def curry(f: Callable) -> PZFunc:
    """To curry a function, only support less than 7 parameters

    :param f: the normal un-curried python function
    :return: A curried PythonZ function
    """
    return PZFunc(lambda x: curry(lambda *args: f(x, *args)))

[chantisnake]

Full code is here: https://github.com/chantisnake/PythonZ/blob/master/pythonz/basic.py

[ilevkivskyi]

My guess is that mypy considers Callable[[A1], R] and Callable[[A1, A2], R] overlapping types. For example something like def fun(*args): ... matches both.

I am not sure what to do here, maybe we can allow the definition, and give an error at the call site if an argument matches two overloads with incompatible return types. Or maybe we can infer a union of return types if the argument matches multiple overloads.

There is still no 100% agreement on semantics of overloads, see https://github.com/python/typing/issues/253

[ilevkivskyi]

This appeared again, so I raise the priority. Also https://github.com/python/typing/issues/253 is one of the few issues remaining in the "PEP 484 finalization" typing milestone that approaches soon.

[Daenyth]

In addition to curry/uncurry style functions, I ran into this when I was handling a decorator that was agnostic to the arity of what it worked on by using *args, **kwargs for re-invocation.

[chadrik]

I ran into this same problem with NamedTuple as well as Callable:

Here are my simple reproductions (using mypy 0.540):

from typing import overload, NamedTuple, Callable

A = NamedTuple('A', [('foo', int)])
B = NamedTuple('B', [('bar', str)])

@overload
def this(arg):
    # type: (A) -> str
    pass

@overload
def this(arg):
    # type: (B) -> int
    pass

def this(arg):
    pass

# -- callable

@overload
def that(arg):
    # type: (str) -> str
    pass

@overload
def that(arg):
    # type: (Callable) -> int
    pass

def that(arg):
    pass

[ilevkivskyi]

@chadrik I think your particular examples might be related to the fact that if mypy can't determine whether types are overlapping, then it always assumes that they are.

[JelleZijlstra]

The last example might actually overlap for real with protocols; you could have a str subclass that also defines __call__ and is therefore Callable.

[chadrik]

The last example might actually overlap for real with protocols; you could have a str subclass that also defines __call__ and is therefore Callable.

The same would be true for any protocol, which effectively means that protocols can't be used with @overload, right? And if mypy takes potential subclasses into consideration when determining overlap, then wouldn't multiple inheritance mean that any two types could overlap? i.e. for any pair of classes A and B there can be a class C that inherits from both, so by the same logic A and B overlap.

Since this is such an easy trap to fall into, it might be worth considering some adjustments to make it easier to avoid. Here are some ideas:

• only error if overlap is actually present in the declared types. find matches by applying overloads in order of appearance
• provide a way to declare a type X, but not subclasses of X

I think your particular examples might be related to the fact that if mypy can't determine whether types are overlapping, then it always assumes that they are.

Why can't mypy determine that NamedTuples A and B don't overlap?

[Daenyth]

provide a way to declare a type X, but not subclasses of X - I don't think this is productive. A constraint like that basically means that you expect inheritors of X to break LSP. I think it's reasonable for people who break LSP to see errors.

only error if overlap is actually present in the declared types. - This seems much more sensible to me

[JukkaL]

The way mypy handles overloads is inconsistent. Originally mypy did overlapping checks pretty strictly -- for example, two user-defined classes A and B were considered overlapping, because of potential multiple inheritance. Built-in types were tagged so you could still overload between int and str, for example. This turned out to be too restrictive. Also, as PEP 484 didn't support tagging built-in classes, the rules were relaxed so unrelated classes A and B are no longer considered as overlapping, even though there could still be a common subclass defined somewhere. Some other overlapping rules have been relaxed as well, but we've never generalized the rules to all kinds of types in a consistent way.

provide a way to declare a type X, but not subclasses of X

I don't think that this would be useful, since "type X or any subclass" is so prevalent that this would be very restrictive in practice. For example, all functions in stubs return "type X or any subclass" so about the only the way you can get a guaranteed instance of type X is to call X (or use a type(o) is X check, but that's considered a code smell).

only error if overlap is actually present in the declared types.

I assume that this means that mypy would complain if the actual argument types match multiple overload items with incompatible return types (we'd have to be silent in case of Any argument types to avoid false positives). This might be reasonable thing to do. Alternatively, the type of the call could be inferred to be a union of the return types of all matching overload items.

[ilevkivskyi]

Originally mypy did overlapping checks pretty strictly

IIUC this is related to the fact that long time ago mypy did some kind of runtime dispatch for overloads.
I will try write a short summary of how I view overloads now, here or in https://github.com/python/mypy/issues/4159 or in https://github.com/python/typing/issues/253 so that we can move with a PoC implementation as Jukka proposed in https://github.com/python/mypy/issues/4159#issuecomment-339814841

[ilevkivskyi]

All examples in this issue now pass on master.