Mypy: Cannot specify return type with Union[..., NotImplemented] (and others, e.g. Ellipsis)

[kamahen]

[UPDATE: The reported behavior happens only with --warn-return-any or --strict.]

Mypy gives warning: Returning Any from function declared to return "bool" for

class C:
    def m(self) -> bool:
        return NotImplemented

So, try this:

from typing import Union
class C:
    def m(self) -> Union[bool, type(NotImplemented)]:
        return NotImplemented

which gives error

q.py:3: error: invalid type comment or annotation
q.py:3: note: Suggestion: use type[...] instead of type(...)

Following mypy's suggestion results in:

q.py:3: error: Return type becomes "Union[bool, Type[Any]]" due to an unfollowed import
q.py:4: warning: Returning Any from function declared to return "Union[bool, Type[Any]]"

which also gives a runtime error: TypeError: Parameters to generic types must be types. Got NotImplemented. or (when attempting type[...]) TypeError: 'type' object is not subscriptable.

(See also pull #4770, which attempts to fix some magic methods)

There seem to be two issues:

1. Only magic methods can return NotImplemented without a complaint from mypy
2. If the programmer tries to get rid of the mypy complaint by specifying -> Union[bool, type(NotImplemented)], mypy doesn't like that and suggests changing it to -> Union[bool, type[NotImplemented]], which gets a runtime error (Type[NotImplemented] doesn't work either).

[Strilanc]

A similar issue occurs with Ellipsis. For example, if you try to specify the type of a method which operates on slices possibly involving ellipses, there's no way to say "I return a tuple containing items that are an integer, a slice instance, or Ellipsis".

I suggest that these, and other, builtin singleton values be handle the same way that None is handled. If you say Union[int, None], that's the same as saying Optional[int] even though technically you should say type(None) or NoneType instead of None because None is a value not a type.

[Strilanc]

Using the string 'NotImplemented' appears to work when invoking mypy and at runtime.

Union[int, 'NotImplemented']

[ilevkivskyi]

Yes, I think we can allow NotImplemented as a singleton (you can open an issue on typing tracker for the runtime part). But Ellipsis will be harder, since it already has special meaning, for example Tuple[int, ...] is a variable length tuple. How would we distinguish it from a tuple of two elements? (A similar problem appears for Callable).

[whatisaphone]

I suggest mypy continues to treat ... as it does today, but treats the long spelled-out form Ellipsis as the type of the singleton. In other words:

foo: Tuple[int, ...] = (2, 3, 4)
foo: Tuple[int, Ellipsis] = (2, ...)

Switching them around would cause errors in both cases:

foo: Tuple[int, Ellipsis] = (2, 3, 4)  # error: 3 elements found, but only 2 expected
foo: Tuple[int, ...] = (2, ...)  # error: type(Ellipsis) is not assignable to int

[jack1142]

Could also just support usage of type(Ellipsis)/type(...) as a type alias (maybe explicit type alias, i.e. PEP 613), so that one could do:

# the commented version could be used instead,
# if this were to use explicit type aliases feature:
# EllipsisType: TypeAlias = type(Ellipsis)
EllipsisType = type(Ellipsis)

foo: Tuple[int, EllipsisType] = (2, ...)

How does that look?

I suggest mypy continues to treat ... as it does today, but treats the long spelled-out form Ellipsis as the type of the singleton.

As far as I know, this would not be really possible as Tuple[int, ...] and Tuple[int, Ellipsis] mean the same at runtime.

[hauntsaninja]

Some more discussion at https://mail.python.org/archives/list/[email protected]/thread/277RFXNQYLT6D3R4EYZZOPPGX56SNIK6/

[julian-klode]

So, NotImplemented worked fine as a return type in e.g. Union[int, NotImplemented] in mypy 0.782, but now in mypy 0.790 it says

apt/package.py:473: error: Variable "builtins.NotImplemented" is not valid as a type
apt/package.py:473: note: See https://mypy.readthedocs.io/en/latest/common_issues.html#variables-vs-type-aliases

Making it a Literal[NotImplemented] does not help:

apt/package.py:473: error: Parameter 1 of Literal[...] is invalid

I'm a bit unhappy.

[hauntsaninja]

mypy_primer bisects this to https://github.com/python/typeshed/pull/4222 to fix https://github.com/python/typeshed/issues/3315
Previously NotImplemented was just Any, which is why you got the complaint with --warn-return-any.

Seems like we should sprinkle some more special casing on this. Not the greatest workaround, but you could use NotImplementedT = Any. You could also dangerously rely on typeshed implementation details here and do some if TYPE_CHECKING stuff.
In most cases, I think the best thing to do is to not put NotImplemented in the return type and just type ignore if using with --warn-return-any. (You should only want to be able to put NotImplemented in your return type if you want all callers to be aware of that and have to check that the returned value is NotImplemented)

[julian-klode]

Yeah, um, in my case I'm not sure what the code is doing, as it wraps a cmp-style function, and the callers then do > 0, for __gt__ for example, which produces a TypeError with TypeError: '>' not supported between instances of 'NotImplementedType' and 'int'. I think that code is buggy, as the __gt__ functions and friends should forward NotImplemented return, but then mypy never gave me a bug about that :/

[EPronovost]

@julian-klode Not sure about your situation but I just encountered issues related to NotImplemented after upgrading to mypy 0.790. Here is the commit that fixed things for me. https://github.com/numba/numba/pull/6222/commits/55b6edaa0e5395476809b14af94c67ad1c5cefa0