Mypy: Generic NewType?

[Daenyth]

I'd like to be able to write code like;

from typing import *
SortedList = NewType('SortedList', List)
A = TypeVar('A')
def my_sorted(items: List[A]) -> SortedList[A]:
    ...

Currently I get error: "SortedList" expects no type arguments, but 1 given

[gvanrossum]

Hm, that may actually be reasonable. After all NewType is an optimized version of subclassing that erases the distinction at runtime -- and generics are also erased at runtime, so that may be a reasonable match.

[ilevkivskyi]

NewType is an optimized version of subclassing

Because of this, at runtime SortedList is a function that returns its argument. Practically any way of making NewType subscriptable will make it much slower.

[Daenyth]

Ahh, I see. Could it not be done like:

class NewType(?):
    def __init__(self, typename, type):
        ?
    def __getitem__(self, key):
        (type things)
    def __call__(self, arg):
        return arg

Would that be such a large overhead?

It seems like potentially it could be made somewhat opt-in. By that I mean that if you do NewType('a', t) for a t that is not Generic, return the identity function as it already does, but in the case that t is generic, return the subscriptable object?

[Daenyth]

That would parallel fairly reasonably with expectations people may have from similar behavior in other languages. Value types in scala are usually erased, for example, except in some cases. Similarly for newtypes in haskell, as I understand it. And with traits in rust.

I don't think it's that bad of a drawback, considering that the drawback only occurs when supporting something that currently can't be done at all. As long as it's documented it seems reasonable, at least.

[ilevkivskyi]

@Daenyth

Would that be such a large overhead?

Approximately 30x slower:

>>> timeit('class C: ...', number=10000)
0.11837321519851685
>>> timeit('def f(x): return x', number=10000)
0.00439511239528656
>>> 

[ilevkivskyi]

@Daenyth

something that currently can't be done at all.

What about normal subclassing? Your example can be just this

class SortedList(List[T]):
    def __init__(self, lst: List[T]) -> None:
        ...

[Daenyth]

Well, "not at all" is definitely an exaggeration. The drawback to the wrapper class is that I either have to implement proxy methods for every single list interface or expose a .value accessor.

[carljm]

We have a slightly different use case for this. We want to be able to do:

T = TypeVar('T')

IGID = NewType('IGID', (int, Generic[T]))

user_id: IGID[User] = IGID(3)

Subclassing is not an option here due to the runtime overhead; these need to stay real ints at runtime.

Without the generic, we either lose the ability to distinguish different types of IGIDs in the type system, or we have to create a separate type (e.g. UserID = NewType(...)) for every object type with an IGID (and we have many).

This use case does require two new features: a) passing a tuple of types to NewType for multiple "inheritance", and b) supporting indexing the runtime NewType.

@Daenyth it's not a wrapper class, it's a subclass, and the list constructor accepts a list already, so add a super().__init__(lst) and the subclassing solution works without any extra proxy methods or accessors.

@ilevkivskyi

Approximately 30x slower:

The 30x overhead you timed would be paid only once per process, at NewType creation. The more likely critical cost is the one that you pay every time the type is used, which is more like 2-3x difference:

>>> timeit.timeit('c(1)', 'class C:\n    def __call__(self, arg): return arg\n\nc = C()') 
0.18850951734930277 
>>> timeit.timeit('f(1)', 'def f(arg): return arg')                                                                                         
0.08798552677035332 

That's still probably enough of a cost that we will just go with the "lots of separate types" solution for our case instead.

[SaschaSchlemmer]

I'd also like to use generic NewType in a manner inspired by phantom types to make APIs/ libraries I write type safe without introducing a lot of actual subclasses (for that I would have to implement/ pass through lots of methods). Take for example:

from typing import NewType, List, NoReturn, TypeVar

ListOfInts = List[int]

NonEmptyListOfInts = NewType('NonEmptyListOfInts', ListOfInts)


def prove_sequence_of_ints_is_nonempty(seq: ListOfInts) -> NonEmptyListOfInts: 
    if len(seq) > 0:
        return NonEmptyListOfInts(seq)
    else:
        raise ValueError('Sequence is empty')


def foo(seq: NonEmptyListOfInts) -> NoReturn: pass


a = [1, 2, 3]

b = prove_sequence_of_ints_is_nonempty(a)

foo(a) # Argument 1 to "foo" has incompatible type "List[int]"; expected "NonEmptyListOfInts"
foo(b)

This mostly works as expected (except for methods mutating b like b.pop() not causing mypy downgrade b to List[int] like c = a + b # c has type List[int] would do) but becomes tedious quite quickly because one has to introduce lots of type aliases like ListOfInts due to NewType not supporting generics.