Mypy: Allow using TypedDict for more precise typing of **kwds

[ilevkivskyi]

There are some situations where a user wants to have more precisely typed **kwds. Current syntax only allows homogeneous **kwds:

def fun(x: int, *, **options: str) -> None:
    ...

However, in situations with many heterogeneous options listing all options in the signature could be verbose and will require rewriting some existing code. There is a vague idea to allow TypedDict for such situations. For example:

class Options(TypedDict):
    timeout: int
    alternative: str
    on_error: Callable[[int], None]
    on_timeout: Callable[[], None]
    ...

def fun(x: int, *, **options: Options) -> None:
    ...

Maybe for such cases the TypedDict used should be automatically understood as defined with total=False. Also it is worth mentioning that this feature will allow reusing the TypedDicts in modules where several functions have same (or similar) option sets.

[JukkaL]

The proposed syntax is problematic, as it could also mean a homogeneous **options where each value is an Options. We could work around this with some extra syntax, but it could look messy. Here's an idea:

from mypy_extensions import Expand
...
def fun(x: int, *, **options: Expand[Options]) -> None:
    ...

Expand[...] could also be used for variadic type variables in case we add support for them (for example, Callable[[int, Expand[X]], None] if X is a variadic type variable).

I don't like the idea of implicitly assuming total=False.

[ilevkivskyi]

Yes, the syntax would be ambiguous. Introducing Expand only for this purpose may be not worth it, but using it also for variadic type variables is a good idea (and it feels natural).

I don't like the idea of implicitly assuming total=False.

OK, let's then drop this. In any case it is easy to add it explicitly (I think it will be used often, since keyword arguments are often optional).

[ilevkivskyi]

Raising priority to normal, since this is a very common request (and we have agreed on using Expand[...]).

[rmorshea]

@ilevkivskyi @JukkaL I'm in pretty desperate need of this so I might be able to justify working on this to my employer. Do you have any ideas about how to implement this? If not, are there areas of the code-base that I should look at in order to understand how to approach the problem?

[ilevkivskyi]

Although this is not super tricky feature, it may be not the best choice for the first contribution, unless you like to live dangerously :-)

You will need to add a new special form Expand[...] to mypy_extensions (where TypedDict lives). This should be allowed only as annotation to **kwars, and must have only one type parameter which is a TypedDict. Then look at all things that contain words formal_to_actual and/or actual_to_formal, they might need to be updated.

Also it may be not the best time to do this, since we are in the middle of a big refactoring of semantic analyzer (one of the first steps in mypy logic, that binds names to symbols). It will take two-three more weeks to finish. Otherwise you would need to repeat some parts of logic twice: in mypy/semanal.py and in mypy/newsemanal/semanal.py.

[rmorshea]

@ilevkivskyi, hmm maybe I'll hold off then. Let me know if there's any work a new contributor could provide in order to help out.

[ilevkivskyi]

@rmorshea You can try one of these: https://github.com/python/mypy/labels/good-first-issue (unless they are too easy/not interesting).

[untitaker]

I got around to hacking something together in https://github.com/python/mypy/pull/7051. It's obviously very rough but hopefully I can build something off of that.

BTW, I think Expand completely breaks the typesystem. What does this do?

x: Expand[dict[str, int]] = 42

You could argue that this should work because having a type that just works for **kwargs is ugly. You could also argue that this is nonsense and should not be allowed.

I think Expand cannot be called a type. IMO the proper solution would be to introduce a breaking change in the typesystem and make **kwargs: X expand X automatically. Code that looks like this right now:

def foo(**kwargs: int): ...

would have to be changed to:

def foo(**kwargs: Dict[str, int]): ...

[untitaker]

@ilevkivskyi I would like to hear your thoughts on changing the behavior on kwargs to expand automatically. The more I think about it the more I resent the Expand type.

[ilevkivskyi]

If you mean this:

IMO the proper solution would be to introduce a breaking change in the typesystem

then no. Just the scale of possible breakages is already an enough reason not to do this. In addition this will introduce an inconsistency with *args (or you want to change them too?). Finally, Expand[...] will need to be introduced later anyway as part of support for variadic generics.

[untitaker]

Yes, I want to change *args the same way. I don't really understand why Expand is the only solution for variadic generics. In any case I feel that introducing a type that does not behave like a type at all is extremely ugly semantics to preserve backwards compat and work around syntactic limitations.

I think the mypy project will have to think about a system for introducing breaking changes (such as versioning hints) because right now it digs itself into a rabbit hole by building on mistakes for the sake of preserving compat.

[ilevkivskyi]

Just to save time, there is very little chance you will convince all of us. By us I mean the whole static typing community, there is also Pyre, pytype, PyRight, PyCharm, etc. and type system and syntax is shared by all of us and is standardized by several PEPs. If you really want to try this, start a discussion at [email protected].

[Nearoo]

I don't really feel authorative enough to post here, but I figured there's an off chance that you'll like my suggestion: What about **kwargs: **Options?

[griels]

I don't really feel authorative enough to post here, but I figured there's an off chance that you'll like my suggestion: What about **kwargs: **Options?

I'm happy with either Expand or **, but my current project would really benefit from this ASAP, so whatever gets the most agreement soonest is good for me. I see that the PEP 589 already mentions Expand as a proposition.

[Mattwmaster58]

This would be _really_ useful to have in a project I'm working on ATM. Is there any chance someone authoritative could take a look at #4441? Or is there any other way I could help out? My vote is for **kwargs: **Options, though I suspect it may be a little bit harder to implement.

[JukkaL]

**kwargs: **Options is a syntax error, so it can't be supported unless Python syntax is changed. Expand[...] seems to be the preferred option that is technically feasible.

I don't think that any of the core team members has time to work on this now, but if somebody would like to create a PR, we could give you hints.

[Mattwmaster58]

@JukkaL What needs to be added to mypy_extensions #18? I assume the missing constraints mentioned in the PR need to be added?

Then once that is finished, mypy itself will need to support these. What are the basic steps for doing so?

[wyfo]

**kwargs: **Options is a syntax error, so it can't be supported unless Python syntax is changed.

@JukkaL Indeed **kwargs: **Options is a syntax error, but **kwargs: '**Options' isn't. As a workaround to a possible future Python syntax enhancement in a few versions, it could work. Actually, we all did the same way for recursive references before 3.7 and from __future__ import annotations; why not a __future__ for this syntax too ?
By the way, I agree with @untitaker that using a new type (or something that really looks like) for something that is not a really a type could be quite confusing; I'd rather pay the price of adding two quotes. Less confusion too on the usage of this syntax, because it's obviously associated to variadic parameters (and it will maybe be simpler to track a misplaced use of the syntax than a misplaced use of an object). And last but not least, this solution doesn't need a typing/typing_extension library modification.

EDIT
I was wrong when I wrote that it would not require to modify typing/typing_extension because I hadn't think about typing.get_type_hints … So yes, that will need a modification, and without breaking change, that will need a new type that is not a type.
However, it there is a new type (Expand de facto), I will always prefer **kwargs: '**Options' to **kwargs: Expand[Options], and keeping this type (but considering it not like a type but a kind of wrapper) for internal use.
But I know … Python type annotations are currently not designed to be light but to work, so I imagine that we will have to pass by an heavy syntax like Tuple before a complete typing annotations refactoring …

[wyfo]

A possible workaround to type **kwargs would be to use typing.overload this way :

from typing import overload

@overload
def func(a: int = …, b: str = …):
    …

def func(**kwargs):
    print(kwargs)

BUT, the type of kwargs cannot be retrieved by typing.get_type_hints (like any other parameters typed by overloading). Furthermore, Mypy is complaining about this trick (but that could be arranged easily I think, at least by type: ignore), but it still successfully checks the call to the function.
However, as I use a lot typing annotation magic for code generation, I cannot be satisfied with this workaround because, as previously written, typing.get_type_hints cannot work. It can only be used by static checkers (what is still a beginning).

[antonagestam]

@wyfo I think it's probably better for mypy to not implement support for a new syntax without knowing it will be accepted into Python itself. And for it to be accepted into Python it would have to go through the PEP process building on top of PEP 484.

For that syntax change to be accepted by the Python core developers, they will probably first like to know that this is a sought after feature, or they risk binding themselves to supporting something that isn't widely used.

A feasible way to prove that it's a popular feature is to first implement it using already supported syntax. This is how almost every new feature is added to mypy. See for instance PEP 604 and PEP 585 that proposes specialized syntax for now stable typing features.

So, a realistic approach for the feature to gain specialized syntax support is for it to follow that same process through implementation using existing syntax, prove its popularity and then gain support in specialized syntax.

[JukkaL]

@antonagestam Well said. New syntax is generally only an option for an already supported feature that is known to be popular enough.