Mypy: TypedDict cannot be used where a normal dict is expected

[TimSimpsonR]

This is probably intended behavior, and if so it be good to have the motivation documented somewhere alongside TypedDict's current docs. Apologies if I'm missing something in the docs.

I would expect to be able to pass a TypedDict to any function that expects a plain old dict:

    import mypy_extensions as mt

    FancyDict = mt.TypedDict('FancyDict', {'a': str})

    fancy = FancyDict(a='1')

    def use_any_dict(arg: dict) -> None:
        print(arg)

    use_any_dict(fancy)

However the above snippet fails:

    error: Argument 1 to "use_any_dict" has incompatible type "FancyDict"; expected "Dict[Any, Any]"

TypedDict is a plain dictionary at runtime, which is what the signature of use_any_dict says it accepts. So why doesn't MyPy allow it?

Going the other direction- passing a Dict[Any,Any] to something expecting a certain TypedDict- also fails, which makes sense to me. But it leaves me wondering how code using normal dicts is supposed to interop with newer code using TypedDict aside from using casts or # type: ignore.

I've tested this with mypy 0.590 from PyPi as well as commit d6a22cf13a5a44d7db181ae5e1a3faf2c55c02b4 from trunk. I'm invoking MyPy as follows:

    mypy     \
        --ignore-missing-imports    \
        --strict-optional     \
        --check-untyped-defs     \
        --disallow-untyped-calls     \
        --disallow-incomplete-defs     \
        --disallow-untyped-defs \
        my_module

[JelleZijlstra]

This is probably related to invariance. See http://mypy.readthedocs.io/en/latest/common_issues.html#invariance-vs-covariance.

[JukkaL]

Yeah, this is a bit unfortunate, and it's related to invariance. You can replace dict with Mapping in the type annotation and things should work.

[TimSimpsonR]

Thank you very much. Using Mapping works for me.

I'm ready to make a pull request that adds a mention of this to the docs for TypedDict if that helps. Otherwise I'm ok with closing this.

[jasonkuhrt]

Can someone show a working code sample?

[rgant]

I think it's ridiculous to have to go through this, but this is a working method of handling a library function that only accepts Dict but you want to use TypedDict:

import csv
import typing

from mypy_extensions import TypedDict


User = TypedDict('User', {  # pylint: disable=invalid-name
    'uid': str,
    'email': str,
})

def write_to_csv(users: typing.List[User], outfile: str) -> None:
    """ Writes users list to CSV file. """
    fieldnames = list(users[0].keys())

    with open(outfile, 'w', newline='') as csvfile:
        writer = csv.DictWriter(outfile, fieldnames)
        writer.writeheader()

        for usr in users:
            # Cast to Dict until TypedDict is Dict compatable
            # https://github.com/python/mypy/issues/4976 
            usr_dict = typing.cast(typing.Dict[str, typing.Any], usr)
            writer.writerow(usr_dict)

Without the usr_dict typing.cast() you get the error:

Argument 1 to "writerow" of "DictWriter" has incompatible type "User"; expected "Dict[str, Any]"

[ilevkivskyi]

@JukkaL can we do something here, or should we just close this issue?

[JukkaL]

Making TypedDicts compatible with Dict[str, Any] would be nice, but it's unclear what's the best way to do this. We could special case this in subtype checks, but then the behavior under joins would be inconsistent. Joins would use the fallback type, so would generally produce Mapping types. Mapping is not compatible with Dict[str, Any], which results in the main inconsistency that I can see. Changing the TypedDict fallback type doesn't sound like a good idea to me.

Another idea would be to change typeshed signatures that only accept Dict[str, Any] to Mapping[str, Any] whenever it makes sense. If some functions don't accept arbitrary mappings, we wouldn't be able to do this, however.

Any thoughts?

[ilevkivskyi]

I think it is probably OK to special-case Dict[str, Any] (if one puts Any, not object there, then one already accepts the risk). However, I think we can just wait a bit more to see how popular is this request.

[rgant]

I guess I just don't understand mypy's plans for Dict and TypedDict. Because I mostly deal with JSON APIs, Dict is not something I ever intend to use (excepting casting). I guess I just assumed that eventually Dict and TypedDict would be merged.

I see in the documentation explaining why TypedDict doesn't extend Dict:

A TypedDict object is not a subtype of the regular Dict[...] type (and vice versa), since Dict allows arbitrary keys to be added and removed, unlike TypedDict.

But I don't know why a subtype of Dict cannot disallow arbitrary keys. I also don't see why TypedDict cannot accept arbitrary keys. TypeScript interfaces support all the complicated objects I've ever thrown at it.

After all TypedDict isn't a Python-ism, it's a mypy solution to a lack of Dict's ability to accept complicated patterns. I was really hoping that eventually I would be able to do something like typing.Dict[{'uid': str, 'email': str,}]

If we cannot do that, then can we just have TypedDict figure out the equivalent Dict syntax?

User = TypedDict('User', {  # pylint: disable=invalid-name
    'id': int,
    'email': str,
    'birthdate': datetime.date,
    'meta': UserMeta # A TypedDict
}) # == typing.Dict[str, typing.Union[int, str, datetime.date, UserMeta]]

Or add a third parameter that allows me to manually specify the Dict type of the TypedDict:

User = TypedDict('User', {  # pylint: disable=invalid-name
    'id': int,
    'email': str,
    'birthdate': datetime.date,
    'meta': UserMeta # A TypedDict
}, typing.Dict[str, typing.Any])

I just think it's very confusing to have to create a new variable to cast a variable for the type checker to be happy.

[JukkaL]

But I don't know why a subtype of Dict cannot disallow arbitrary keys.

In your example, if User would be compatible with the Dict type, it looks like you could do things like x['id'] = 'abc' through the Dict type, which would break type safety, since the value of the 'id' key is supposed to be an int. Also, a TypedDict subtype of User should define additional keys with values that are not covered by the union, so accessing .values() or .items() through the dict type would be unsafe as well.

I'm not quite sure what you mean by equivalent Dict syntax, as the two types don't seem to be equivalent. Can you provide a more complete code example that shows what isn't working right now as you'd expect, and explain how your proposal would resolve the issue? And maybe you can also explain how you'd deal with the issue in TypeScript.

[rgant]

In typescript I can do a mixture of known and unknown properties on my object like this:

interface MyInterface {
  fixedKey1: string,
  fixedKey2: number,
  [x: string]: string | number, 
}

And in my head I would translate that to something like:

MyInterface = TypedDict('MyInterface', 
    {'fixedKey1': str, 'fixedKey2': int},
    arbitrary=[str, Union[str, int]]
)

Then I could use the type like:

payload: MyInterface  = {'fixedKey1': 'foo', 'fixedKey2': 123, 'meta': 'bar'}

The important part being that I cannot make fixedKey2 a string as my API doesn't allow for that. But there could be other data that I haven't specified yet, or that my partner wants to shove in that my type checker could then handle.

[gricey432]

In my case, if I replace dict with Mapping all the errors go away on the function calls, however in my function I use .pop and now get a type error for that.

from typing import Mapping
from typing_extensions import TypedDict

def myfunc(d: Mapping):
    d.pop("key")

TD = TypedDict("TD", {"b": int})
b: TD = {"b": 2}
myfunc(b)

error: "Mapping[Any, Any]" has no attribute "pop"

If I replace Mapping with MutableMapping the .pop is now happy but the original problem is back.

error: Argument 1 to "myfunc" has incompatible type "TD"; expected "MutableMapping[Any, Any]"

[JukkaL]

@gricey43 One of the reasons why TypedDict types are compatible with Mapping is that it doesn't provide any mutation operations such as pop. What you want to do is not type safe, so you'll need to work around the type system by using a cast, for example.

[JukkaL]

After more practical experience with TypedDicts, not having TypedDict compatible with Dict[str, Any] can actually be quite useful when trying to ensure that all dictionaries in some code are precisely checked. In any case, the current situation seems acceptable, and I'm not convinced that it's worth adding special cases to type checking for this specific use case.