Typing: NamedTuple subclassing NamedTuple

[gvanrossum]

I naively thought that since I can write

class A(NamedTuple):
    x: int
    y: int

I would also be able to subclass this:

class B(A):
    z: int

That is accepted syntactically and at runtime but no new __new__ method is generated so calling B(1, 2, 3) is flagged as an error (too many arguments) by both mypy and runtime.

[ilevkivskyi]

I think this feels natural to you because you can do all such things (extending and merging) with TypedDict. It was simple with TypedDict since both functional and class syntax were developed at the same time, while with NamedTuple we have problem of backwards compatibility. The functional syntax exists for a long time (also in the form of collections.namedtuple). I have seen many times when people were subclassing named tuples, so that changing semantics of subclassing can potentially break things. It seems to me it was already briefly discussed some time ago, but I can't fins the discussion. There are three possible solutions now:

• Status quo.
• We can have different semantics for functional syntax (returning the result of plain collections.namedtuple call), and class syntax (returning some "smart class" that will support all the TypedDict tricks while mimicking collections.namedtuple).
• Make both functional and class syntax return the "smart class", people who want the old (i.e. current) semantics can just use collections.namedtuple.

All three options have some downsides. If we go with the second or third option, then I think it is better to do this fast (to minimise the effect of backwards incompatibility).

[gvanrossum]

Hm, the functional syntax cannot be used to create a subclass. I think it should not make a difference whether you use functional or class syntax for the base class. But the class syntax is new, so this problem is new too. I suppose the backward compatibility problem you're talking about is that in my example, one could also assume that B.z is a mutable instance variable? I think that would be a much less likely use case -- I can understand why you want to add methods, but mutable state seems quite a stretch for a NamedTuple subclass.

Or did I miss something? If not, now's the time to choose the more useful semantics.

[ilevkivskyi]

I suppose the backward compatibility problem you're talking about is that in my example, one could also assume that B.z is a mutable instance variable?

Also the fact that currently the signature of constructor of a subclass is not changed. One can have some variables added in a subclass and be surprised when the signature of constructor is changed in a new version of typing. This is still not a major backwards incompatibility, but probably something we should consider. For example:

Base = NamedTuple('Base', [('x', int), ('y', int)])

class Derived(Base):
    some_bookkeeping: List[int] = []  # mypy requires annotation here

Derived(1, 2)  # will fail in a new version

[gvanrossum]

I think we should allow ourselves complete freedom in changing this -- there's only the slightest mention of it in PEP 484 and none in PEP 526. That makes it a provisional feature (like everything in typing.py and in either of those PEPs).

Even for a legitimate class deriving from NamedTuple (e.g. A in my example) there seems to be an ambiguity with initial values -- e.g.

class A(NamedTuple):
  x: int
  y: int = 0
a = A(12)

passes in mypy but fails at runtime:

Traceback (most recent call last):
  File "__tmp__.py", line 11, in <module>
    a = A(12)
TypeError: __new__() missing 1 required positional argument: 'y'

[ilevkivskyi]

passes in mypy but fails at runtime:

It fails at runtime with _older_ versions of typing, but passes at runtime with the latest one (I just tried this on CPython master). In general I see your point, this is not explicitly documented, and typing is provisional, so that we can change this. I will play with this tomorrow (as I understand, we don't need to change anything in Python 2 version of typing).

[gvanrossum]

Thanks, no hurry. I actually lost track of the thought that led me to try that. Probably another random issue or PR in the tracker. :-(

[ilevkivskyi]

I would propose the following syntax:

class C(NamedTuple, get_fields_from=B):
    x: int

with the following rules:

• B can be any type with _field_types defined, the latter should be a dictionary mapping names to types;
• if B has _field_defaults attribute, then it will be also used to populate default values;
• C is not a subtype (static) and not a subclass (runtime) of B;
• fields from B cannot be overwritten by fields in C (this can be enforced only by static checkers as for TypedDicts currently).

Similar syntax can be also supported for functional form.

Possible rationale for this (at least what I have heard) is producing relatively many similar named tuples. In addition, in future we may switch to structural subtyping _between_ named tuples (it will be a minor extension over current protocols implementation, we just need to take care of fields order), so that:

class Point(NamedTuple):
    x: int
    y: int
class LabeledPoint(NamedTuple):
    x: int
    y: int
    label: str

def fun(p: Point): ...
fun(LabeledPoint(1, 2, 'test'))  # OK

consequently, this will also work:

class Point(NamedTuple):
    x: int
    y: int
class LabeledPoint(NamedTuple, get_fields_from=Point):
    label: str

def fun(p: Point): ...
fun(LabeledPoint(1, 2, 'test'))  # OK

@gvanrossum @JukkaL what do you think?

[gvanrossum]

There is basically no experience in the field with the current class-based NamedTuple (it wasn't even mentioned in PEP 526). For the functional API this could be solved much simpler (without new syntax) by teaching mypy about constant expressions so you can write

base_fields = [('x', int), ('y', int)]
Point = NamedTuple('Point', base_fields)
LabeledPoint = NamedTuple('LabeledPoint', base_fields + [('label', str)])

(And similar for collections.namedtuple.)

For anything fancier in 3.7 we can refer people to dataclasses.

[ilevkivskyi]

There is basically no experience in the field with the current class-based NamedTuple (it wasn't even mentioned in PEP 526).

It is true the class syntax is mostly unknown to people. Which is a bit strange, people seem to be always excited when they encounter it.

For the functional API this could be solved much simpler (without new syntax) by teaching mypy about constant expressions so you can write

It is also true that functional API already supports this, but the class API is more flexible, it supports default values, docstrings, custom __repr__.

For anything fancier in 3.7 we can refer people to dataclasses.

Yes, we don't need anything fancier for named tuples, but It would be nice if class syntax will be as "expressive" as functional API. Although the class keyword will not allow something more complex like:

base_fields = [('x', int), ('y', int)]
label = [('label', str)]
LabeledPoint3D = NamedTuple('LabeledPoint3D', base_fields + [('z', int)] + label)

it will probably be enough for most use cases. Finally, implementing class keyword will be very simple.

---

On the other hand, maybe it is OK if both class and functional syntax have some strengths. Functional API can be used for more "dynamic" namedtuple creation, while class syntax is more suitable for "static" cases, so just keeping the status quo in typing is also an acceptable option.

[gvanrossum]

It seems a lot of extra implementation complexity to occasionally save people a few lines of copied code...

[ilevkivskyi]

It seems a lot of extra implementation complexity to occasionally save people a few lines of copied code...

OK, so I understand that you vote for status quo :-)

[ilevkivskyi]

Following the discussion on mypy tracker, I think this issue can be closed as well.