Typing: NamedTuple doesn’t support overriding __new__

[flying-sheep]

I expected the following to work like with collections.namedtuple, but it doesn’t:

_FooBase = NamedTuple('_FooBase', [('a', FrozenSet[str])])

class Foo(_FooBase):
    def __init__(self, a: Iterable[str]):
        super().__init__(frozenset(a))

but I get:

In [3]: Foo(['x'])
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-9-a93a09ef10e2> in <module>()
----> 1 Foo(['x'])

<ipython-input-8-cbd998f4721b> in __init__(self, a)
      3 class Foo(_FooBase):
      4     def __init__(self, a: Iterable[str]):
----> 5         super().__init__(frozenset(a))
      6 

TypeError: object.__init__() takes no parameters

With the 3.6 syntax, there’s an earlier error:

In [2]: class foo(NamedTuple):
   ...:     a: FrozenSet[str]
   ...:     def __init__(self, a: Iterable[str]):
   ...:         super().__init__(frozenset(a))
   ...:         
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-3-c99525ef7a4f> in <module>()
----> 1 class foo(NamedTuple):
      2     a: FrozenSet[str]
      3     def __init__(self, a: Iterable[str]):
      4         super().__init__(frozenset(a))
      5 

/usr/lib/python3.6/typing.py in __new__(cls, typename, bases, ns)
   2152         for key in ns:
   2153             if key in _prohibited:
-> 2154                 raise AttributeError("Cannot overwrite NamedTuple attribute " + key)
   2155             elif key not in _special and key not in nm_tpl._fields:
   2156                 setattr(nm_tpl, key, ns[key])

AttributeError: Cannot overwrite NamedTuple attribute __init__

[ilevkivskyi]

This is by design. Note, tuple is immutable, and therefore one cannot call __init__ on it, everything is done by __new__. Also there are some other tricks to make named tuples compact and fast, so that it is prohibited to override _few_ special methods and attributes.

Named tuples are intended to be minimalistic, for everything more complex you can use dataclasses.

[flying-sheep]

oh, makes sense. then let’s make sure overriding __new__ works:

class Foo(NamedTuple):
    x: int
    def __new__(cls, x):
        return super().__new__(cls, int(x))
---------------------------------------------------------------------------
AttributeError: Cannot overwrite NamedTuple attribute __new__

[ethanhs]

I believe this is intentional so that people don't break classes that inherit from NamedTuple.

[ilevkivskyi]

oh, makes sense. then let’s make sure overriding __new__ works

I don't think it is possible either (at least it is not easy for sure). And from the example you provided it looks like you should really use something like dataclasses or attrs classes (the latter even have special support for converters).

[flying-sheep]

the example is of course just an example. luckily you don’t have to guess what i want, as i can tell you exactly 😉 i want a data container that:

• is typed
• is immutable
• has its items accessible by attribute access syntax
• has a custom constructor so that i can use defaults and coercion

NamedTuple has all that but the last one, which is prevented by its current implementation. so let’s fix that implementation!

I believe this is intentional so that people don't break classes that inherit from NamedTuple.

ah, so there’s magic in play that makes __new__ behave differently from normal. OK, then let’s just fix this (maybe in the metaclass?). the fact that it doesn’t work now is an understandable hack to protect people from confusion the current implementation would otherwise cause. but it is also surprising (hence this issue being filed), and python should be predictable.

i can override __repr__ and everything, so why not wrap the constructor?

[petered]

Agreed, this would be very useful - it's really the only think standing in the way of being able to nicely write immutable classes (well, that and the strange behaviour of inheritance)

[altaurog]

I am not sure what it means to subclass NamedTuple. How about the following?

In [1]: import collections
In [2]: class X(collections.namedtuple("XBase", "a,b")):
   ...:     def __new__(cls, a, b):
   ...:         return super().__new__(cls, a * 2, b + 10)
   ...:     
In [3]: X(1, 2)
Out[3]: X(a=2, b=12)

[ethanhs]

@altaurog, the point is to subclass and override typing.NamedTuple, to get the typing benefits. That is currently not supported. Of course you can use collections.namedtuple if you don't care about typing.

[ilevkivskyi]

and the strange behaviour of inheritance

I am not sure what it means to subclass NamedTuple.

Some people ask about

class Point(NamedTuple):
    x: int
    y: int
class LabeledPoint(Point):
    label: str

LabeledPoint(0, 0, "origin")

But this will not work. I know people like named tuples, but TBH I think this is at a boundary of abusing them, people should use dataclasses for complex cases (they can be also made immutable), while named tuples should be reserved for simpler scenarios. The problem is that named tuples are highly optimized, and it is impossible to implement the above functionality without breaking at least some of the optimizations.

Btw there also will be (or maybe already is) a backport of dataclasses on PyPI for older Python 3.6.

[petered]

There appears to be: https://github.com/ericvsmith/dataclasses

[jstriebel]

My use-case is to add some assertions in __new__ to check parameter validity. Therefore it would be great to be able to overwrite __new__. It seems to be possible in a subsubclass of NamedTuple, but having the new syntax it would be more natural in the subclass.

[kalessin]

You may implement a factory method instead:

class foo(NamedTuple):
    a: FrozenSet[str]

    @classmethod
    def build_foo(cls, a):
        return cls(frozenset(a))

[tueda]

Although a factory method would work as an alternative to overriding __new__ to some extent, it is bugprone that the normal construct (not forbidden and one may mistakenly use it) may lead to an unexpected result. For example:

from __future__ import annotations

from typing import NamedTuple, Sequence


class Foo(NamedTuple):
    a: Sequence[str]

    @classmethod
    def build(cls, a: Sequence[str]) -> Foo:
        return cls(tuple(a))


a = Foo.build(["1", "2"])
b = Foo.build(("1", "2"))
print(a == b)

a = Foo(["1", "2"])
b = Foo(("1", "2"))
print(a == b)

False

[heyakyra]

oh, makes sense. then let’s make sure overriding __new__ works:

class Foo(NamedTuple):
    x: int
    def __new__(cls, x):
        return super().__new__(cls, int(x))
---------------------------------------------------------------------------
AttributeError: Cannot overwrite NamedTuple attribute __new__

Does this work?

>>> class Foo(NamedTuple("Foo", [("x", int)])):
...     def __new__(cls, x):
...         return super().__new__(cls, int(x))
... 
>>> Foo("1")
Foo(x=1)