Mypy: type alias to union is invalid in runtime context

[glyph]

As of MyPy 0.620:

# aliases.py
from typing import Union
x = Union[int, str]
def y() -> None:
    print(x)

This produces the error:

aliases.py:5: error: The type alias to Union is invalid in runtime context

However, type variables are quite useful in "runtime context" in order to implement things like serialization; for example, https://github.com/Tinche/cattrs makes heavy use of introspecting Union objects to determine which types are valid.

Those of us hacking around code like this have known that the current API for enumerating the members of a Union was private & internal, and not for public consumption, so we've been keeping pace with implementation details while waiting for a public API to get this information to emerge. These errors are super annoying because they imply the only future-looking direction for this type of code will involve manually re-typing the same list of types for every union alias defined anywhere.

Can this change be backed out? If not, is there any guidance for how to define a Union such that it is introspectable?

[glyph]

I guess this was a result of #5221 ?

[ilevkivskyi]

From mypy's point of view it is hard to distinguish whether this is intentional or a mistake. The point is that this change discovered actual errors in real code, so I am not so keen to revert this change even though it was unintentional.

On the other hand, such aliases will likely fail at runtime in most scenarios where it is an error quite soon. So the value of discovering them statically is not so big.

One possible way is to turn the type aliases into variables by giving them explicit types:

X: Type[Any] = Union[A, B, C]
print(X)  # OK

but then X can't be used in annotations. I would like to see more input here before making a decision here.

[glyph]

I can see your point as far as this being a useful thing to check most of the time. And given that we're relying on private implementation details of typing I know it's my own responsibility to keep up with such changes :). Mainly what I'm looking for here is a non-horrible way forward that translates to something that will be close to a one-day public API. For example, I'm very curious what you mean by

such aliases will likely fail at runtime in most scenarios where it is an error quite soon

what will make this the case?

(previous complaint Type[Any] approach deleted, that was an error that was fixed in 0.620)

[glyph]

OK, now that I'm actually testing on the correct version:

x = Union[int, str]
z: Type[Any] = x

still says "The type alias to Union is invalid in runtime context", which means any type I want to use in this way needs to be fully typed out twice.

[glyph]

Is this a workable way to express this for the time being?

x = Union[int, str]
x_runtime: Type[Any]
if not TYPE_CHECKING:
    x_runtime = x

[JelleZijlstra]

What if you do x: Type[Any] = Union[int, str]?

[ilevkivskyi]

After a discussion with @JukkaL I think it is OK to allow the original example. This is however relatively low priority since there are workarounds like using Type[Any] and/or # type: ignore and/or if TYPE_CHECKING: ....

[glyph]

FWIW I agree that the priority on this shouldn't be super high; the main thing it makes annoying is testing. In most real code using this system, the Union is buried in a result of get_type_hints and not something that is being directly referenced.

[glyph]

Just hit this problem again with Callable; we are using one as a sentinel value for dependency injection and mypy complains. It's possible to work around with the X and X_runtime thing above, but it would be nice for library users to not need to know about that distinction.

Would it be possible to reuse the runtime protocol decorator from typing_extensions like so?

from typing_extensions import runtime
x = runtime(Union[str, int])

which doesn't affect the semantics of x except to allow it to be referenced in this manner?

[ilevkivskyi]

As I mentioned above, I am fine with allowing type aliases where object is expected. Python has this conceptual invariant that everything is an object, so it is fine to keep that, even in the world of strong separation between type and runtime contexts. The problem here is not conceptual, we just don't have time to implement a fix.

[pando85]

I'm using python 3.7 and I cannot get it working:

x = Union[int, str]
x_runtime: Type[Any]
if not TYPE_CHECKING:
    x_runtime = x

a = 'a'
if isinstance(a, x_runtime):
    pass

or

x_runtime: Type[Any] = Union[int, str]

a = 'a'
if isinstance(a, x_runtime):
    pass

Error:

TypeError: Subscripted generics cannot be used with class and instance checks

[JukkaL]

@pando85 You can't use a union type in isinstance checks. This has nothing to do with mypy though, since the behavior happens at runtime.

[valtron]

Another example:

NoneType = type(None)
print(NoneType)

[bwo]

From mypy's point of view it is hard to distinguish whether this is intentional or a mistake. The point is that this change discovered actual errors in real code, so I am not so keen to revert this change even though it was unintentional.

but it also "discovers" and classifies as errors things that are not errors in real code, which seems very much in the category of "bug". After a much-delayed upgrade, we are now seeing this at my company and it's probably going to force us to downgrade.

[JukkaL]

@bwo Can you give an example of where you consider mypy behavior to be a bug? It's possible that error codes (a feature in development) will make it easy to selectively ignore only these errors. However, we may need to more examples where the current behavior is a problem for it to be general enough.

[bwo]

I'm having trouble getting a small, reliable reproduction. We have code that is morally like this:

from typing import Union, TypeVar, Generic

T = TypeVar('T')

convenient_union = Union[int, str]

class Spam(Generic[T]):
    def __init__(self, egg: T) -> None:
        self.egg = egg


spam = Spam[convenient_union](1) 

And we get the error on the equivalent of Spam[convenient_union](1). But … I can't actually tickle mypy with such a tiny example. In the actual code, the union is imported from a different module, but that doesn't seem to make a difference. The result in the actual codebase is a lot of repeated typing-out of Union[A, B, …] because using the alias gives the error.

[bwo]

More basically, if either of these works, shouldn't both?

def spam(u: Type[Union]) -> bytes:
    return b''


spam(convenient_union)
spam(Union[int, str])

We have codegen that creates functions that return Union[…] (for generative testing purposes). I suspect this is similar to the use @glyph has.

[Tishka17]

Hello,

I'm working on library which helps to create dataclass instances from dict (a bit like pydantic or marshmallow). The main idea is to have funciton like parse(data: Any, type_: Type[T]) -> T. It analyses what actual type as provied and does parsing based on detected infomation.

I am working on new feature polymrophic parsing and the use case is just to call parse(data, Union[ClassA, ClassB]). It works ok, but mypy detects an error The type alias to Union is invalid in runtime context. It is very strange to me that i cannot provide such a type explicitely, as at a same time i can detect it from annotations (and i do it for dataclass fields)

Minimal exmaple is:

from typing import Union, TypeVar, Type, Any

mytype = Union[str, int]

T = TypeVar("T")


def parse(value: T, type_: Type[T]) -> None:
    print(type_, value)


parse(1, mytype)  # error: The type alias to Union is invalid in runtime context

Some code from my library (https://github.com/Tishka17/dataclass_factory/commit/adc6f53013f0a277ddd0df635d0cedb3fea4dfd0)

What can i do with this?

[ilevkivskyi]

@Tishka17 The simplest way is to # type: ignore this for now and wait for the issue to be resolved.

[Tishka17]

@Tishka17 The simplest way is to # type: ignore this for now and wait for the issue to be resolved.

Any ideas when it can be fixed?

[ilevkivskyi]

Any ideas when it can be fixed?

As soon as someone submits a PR (i.e. no estimate).

[wsanchez]

Any chance we can give this message its own error code instead of [misc]?

[JukkaL]

Any chance we can give this message its own error code instead of [misc]?

Can you explain your use where this would be helpful?

[wsanchez]

Well [misc] lumps a lot of things into one bucket, which sort of defeats the point of having error codes.

If you look at this diff: https://github.com/twisted/klein/pull/301/files, you'll see I added a ton of #typedef ignore[misc] in the code, and I'd much rather have simply disable this warning globally in the config file, but I don't want to disable everything associated with [misc].

This is due to a hack @glyph added here to help mypy associate Zope interfaces with the classes that implement them.

[chadrik]

@JukkaL IIRC, there were plans to be able to configure which error codes to ignore -- globally and per module -- so further classification of errors in the "misc" bucket will become even more significant after that feature arrives, since ignoring the entire "misc" error code would be ill advised.

There are a handful of errors like this one that fall into the "you used a type object incorrectly at runtime" grouping, which could perhaps share the same error code.

[wsanchez]

@chadrik Heh I assumed that feature already existed. :-)