Typing: Allow variadic generics

[NYKevin]

C++11 recently introduced the notion of variadic templates, which I believe Python could benefit from in a simplified form.

The idea is that you can have a generic class with a variable number of type variables, like typing.Tuple[] has. Here is a real-world variadic-generic class which is not Tuple; it is variadic in TagVar. As you can see, TagVar only appears in tuple contexts. Those tuples are sometimes heterogenous (Caution: annotations are a homegrown 3.4-compatible mishmash of nonsense), so repeating TagVar as shown is actually incorrect (but the closest approximation I could find).

Here's one possible syntax:

class MultiField(AbstractField[GetSetVar], Generic[(*TagVar,)]):
    def __init__(self, nbt_names: ty.Sequence[str], *, default:
                 GetSetVar=None) -> None:
        ...

    @abc.abstractmethod
    def to_python(self, *tags: (*TagVar,)) -> GetSetVar:
        ...

    @abc.abstractmethod
    def from_python(self, value: GetSetVar) -> ty.Tuple[(*TagVar,)]:
        ...

This is syntactically valid in Python 3.5 (if a bit ugly with the parentheses and trailing comma, which cannot be omitted without language changes), but doesn't currently work because type variables are not sequences and cannot be unpacked. It could be implemented by adding something like this to the TypeVar class:

def __iter__(self):
    yield StarredTypeVar(self)

StarredTypeVar would be a wrapper class that prefixes the repr with a star and delegates all other functionality to the wrapped TypeVar.

Of course, syntax isn't everything; I'd be fine with any syntax that lets me do this. The other immediately obvious syntax is to follow the TypeVar with an ellipsis, which conveniently does not require changes to typing.py. However, that might require disambiguation in some contexts (particularly since Tuple is likely to be involved with these classes).

[gvanrossum]

Hmm... why not make this a special property of type variables.Tthen you wouldn't need the funny syntax, you could just use a type variable that has a keyword saying it is variadic.

But maybe the bigger question is how exactly we should type check this.

[NYKevin]

Hmm... why not make this a special property of type variables.Tthen you wouldn't need the funny syntax, you could just use a type variable that has a keyword saying it is variadic.

This isn't _obviously_ objectionable, but for the following reasons I'm not sure it's actually a good idea. First, I would like to point out that I failed to notice a third possible syntax, which doesn't require the trailing comma:

class MultiField(AbstractField[GetSetVar], Generic[[*TagVar]]):
    def __init__(self, nbt_names: ty.Sequence[str], *, default:
                 GetSetVar=None) -> None:
        ...

    @abc.abstractmethod
    def to_python(self, *tags: [*TagVar]) -> GetSetVar:
        ...

    @abc.abstractmethod
    def from_python(self, value: GetSetVar) -> ty.Tuple[[*TagVar]]:
        ...

I'll return to this syntax in a moment.

But maybe the bigger question is how exactly we should type check this.

We should enforce that the variable only appears as the sole argument to Tuple (with a trailing ellipsis?), as a variable of a generic class, possibly (?) in the instantiation of a generic class other than Tuple, or as the type of a variadic parameter (*args, **kwargs). The generic class case is complicated, however, because there might be multiple variadic type variables in play under multiple inheritance. Even if there aren't, it could be difficult to parse.

We can overcome both problems by requiring specialized syntax when the generic class is subclassed or instantiated:

class UUIDField(MultiField[uuid.UUID, [tags.LongTag, tags.LongTag]]):
    ...

Note the extra pair of brackets. They indicate where the variadic typevar begins and ends, which removes any ambiguity when there are multiple variadic typevars, and simplifies parsing when there are non-variadic typevars in the same class (as in this case).

For reasons of uniformity, I would recommend we use the bracket syntax I showed above rather than making variadic-ness a property of the type variable. That makes instantiation and declaration look more like one another, and seems more intuitive to me.

[JukkaL]

Variadic generics would likely be useful at least occasionally. It would make sense to generalize them also to argument types in Callable. I remember that there are a few places in the std lib stubs where these would have let us have a more precise type for a library function. The canonical example might be something like this (in my counter-proposal being variadic is a property of a type variable, as it seems more readable to me):

Ts = TypeVar('Ts', variadic=True)
RT = TypeVar('RT')

def call_async(function: Callable[Ts, RT], args: Tuple[Ts]) -> Future[RT]:
    ...

In my proposal a variadic type variable Ts would be valid in at least these contexts:

• As Tuple[Ts] (the only argument to Tuple)
• As Callable[Ts, X] (the first argument to Callable)
• As type of *args, such as *args: Ts

It's less obvious how to generalize this to user-defined generic types. Maybe like this:

class C(Generic[Ts]):
    def __init__(self, *x: Ts) -> None: ...

c = C(1, '')  # type: C[int, str]   # ok!

The limitation would be that in order to have both variadic and non-variadic arguments, you'd have to create a dummy wrapper type for the variadic args:

class ArgTypes(Generic[Ts]): pass   # dummy wrapper type

class C(Generic[T, ArgTypes[Ts]]):
    def __init__(self, t: T, *args: Ts) -> None: ...

The apparent awkwardness of the above example probably wouldn't matter much as I expect this use case to be very rare.

There are additional implementation details that a type checker should probably get right to make this useful, but I'm not even trying to enumerate them here. The implementation would likely be tricky, but probably not excessively so.

I'm still unconvinced that this is a very important feature -- look at how long it took C++ to pick up this feature. I'd propose looking for more use cases to justify variadic generics and once we have a sufficient number of use cases collected here, we'd send this to python-ideas@ for discussion.

[JukkaL]

Examples where these could be useful:

• zip
• contextlib.contextmanager

[sixolet]

@JukkaL There's one extension to your proposal that would solve a problem I've been trying to figure out: I have some Query objects, which vary in the types of results they produce. I also have an execute function, which takes in some Querys and would like to return a tuple, with one result per Query it was passed.

R = TypeVar('R')
class Query(Generic[R]):
   ...

Rs = TypeVar('Rs', variadic=True)
Qs = Rs.map(Query) 

def execute(*args: Qs) -> Tuple[Rs]:
   ...

I am not at all wed to this syntax for mapping over variadic types variables, but I want to be able to do it, and I think it should be a thing we should consider when considering variadic type variables.

[gvanrossum]

It's not very clear from that example that execute() returns a tuple. I think Jukka's proposal would have you write Tuple[Rs] for the return type. I also wish the map() functionality was expressible as part of the signature of execute() rather than in the definition of Qs.

Anyway, this would be shorthand for an infinite sequence of definitions for execute(), like this, right?

R = TypeVar('R')
R1 = TypeVar('R1')
R2 = TypeVar('R2')
R3 = TypeVar('R3')
# etc. until R999
@overload
def execute(q1: Query[R]) -> R: ...
@overload
def execute(q1: Query[R1], q2: Query[R2]) -> Tuple[R1, R2]: ...
@overload
def execute(q1: Query[R1], q2: Query[R2], q3: Query[R3]) -> Tuple[R1, R2, R3]: ...
# etc. until a total of 999 variants

NOTE: The special case for the first overload is not part of the special semantics for variadic type variables; instead there could be two overload variants, one for a single query, one for 2 or more.

[sixolet]

Yes, execute returns a Tuple[Rs] and I simply was posting too quickly to check my work, apologies.

[sixolet]

And yes, this would be exactly that infinite series of overloads.

[sixolet]

Making the type mapping in the argument list would be nice, but requires some care as to exactly where you are parameterizing a type over each element of your variadic type variable, and where you are parameterizing a type using all elements of your variadic type variable.

For example, I'd love to be able to write Query[Rs] and mean "a variadic type variable that is a query type for each result type in Rs" but then I want to also be able to write something like Tuple[Rs] meaning "a tuple of every return type in Rs".

One possibility is, borrowing some stars and parens from @NYKevin to have, I think, somewhat of a different meaning:

def execute(*args: Query[Rs]) -> Tuple[(*Rs,)]:
    ...

[gvanrossum]

So maybe the notation ought to reflect that, and we should be able to write

Rn = TypeVar('Rn', variadic=True)
def execute(*q1: Query[Rn.one]) -> Tuple[Rn.all]: ...

Where .one and .all try to give hints on how to expand these.

[sixolet]

Ooh, I like not having it be some kind of obtuse operator but rather english words. Consider each as a possibility to mean what you're using one for above?

[sixolet]

def execute_tuple(queries: Tuple[Query[Rs.each].all]) -> Tuple[Rs.all]: ...

Unless, say, .all is the default, and you don't have to say it, and you have to say .each at every layer you mean to map the types along.

[sixolet]

Here's "all is the default, you have to specify each"

def execute_tuple(queries: Tuple[Query[Rs.each]]) -> Tuple[Rs]: ...

Here's "each is the default, you have to specify all"

def execute_tuple(queries: Tuple[Query[Rs].all]) -> Tuple[Rs.all]: ...

[gvanrossum]

I need to think about more examples. But note that we're talking about
*queries...

--Guido (mobile)

[sixolet]

For another example, here's map, using "each is the default" form, but where _specifically_ for a *args construction it knows you mean all when you say that *args is of a variadic type variable:

As = TypeVar('As', variadic=True)
R = TypeVar('R')
def map(f: Callable[[As.all], R], *args: Iterable[As]) -> Iterable[R]
  ...

And here's using "all is the default" form:

As = TypeVar('As', variadic=True)
R = TypeVar('R')
def map(f: Callable[[As], R], *args: Iterable[As.each]) -> Iterable[R]
  ...

[JukkaL]

I think that in each location, only one of Ts.one or Ts.all can be valid, so we could automatically infer the right variant. We could then write the example like this:

Ts = TypeVar('Ts', variadic=True)

def execute(*q: Query[Ts]) -> Tuple[Ts]: ...

Here's the map example:

As = TypeVar('As', variadic=True)
R = TypeVar('R')

def map(f: Callable[[As], R], *args: Iterable[As]) -> Iterable[R]: ...

Note that Iterable[As.all] doesn't make sense, since Iterable takes only a single argument.

[sixolet]

@JukkaL It might be a contrived example, but what about this (written with super-explicit each/all notation, to be clear about where you might find ambiguity):

Ts = TypeVar('Ts', variadic=True)

def make_lots_of_unary_tuples(*scalars: Ts) -> Tuple[Tuple[Ts.each].all]:
    return tuple(tuple([s]) for s in scalars)

[sixolet]

Vs this:

Ts = TypeVar('Ts', variadic=True)

def make_a_tuple_wrapped_in_a_tuple(*scalars: Ts) -> Tuple[Tuple[Ts.all]]:
    return tuple([tuple(scalars)])

[JukkaL]

The types aren't quite right in your examples, since tuple(s) isn't valid for an arbitrary object s and tuple(tuple(scalars)) is the same as tuple(scalars), so the tuples won't be nested in the second example. Can you tweak the examples to work?

[sixolet]

So the rule is that parameterizing something by a variadic type either results in another variadic type which represents one instance of the parametrized type per captured type variable in the variadic (each), or results in a non-variadic type, where the variadic type is interpreted as an expanded series of type parameters. You're right that _most_ places only one of these makes sense, but wherever you could have a variadic number of type parameters you could technically use either (even if most reasonable use cases mean to expand the variadic type).

Yeah, I'll tweak the examples.

[JukkaL]

Yeah, those look reasonable, though pretty contrived :-). I'm still not convinced that .all / .each are necessary, as we could also reject ambiguous code where both could be valid, at least if all the use cases we can come up with are silly enough. Single-item tuples seem like a pretty rare use case, but single-argument callables might be a more reasonable case which has the same issue. Even then you'd probably need nested callables, which are not very common.

Another thing to consider is type checking code that uses variadic type variables. Supporting them in stubs only could be much easier than supporting them in function bodies.

[sixolet]

Another exploration of the design space of syntax: TypeVar('Ts', variadic=True) vs. VariadicTypeVar('Ts')

[sixolet]

Ok, here I am a couple days later with a partially-working implementation under my belt. Here's what I think will work (I'm reëxplaining a decent amount here, and it's a wall of text, but bear with me):

Syntax

You create a variadic type variable with @JukkaL 's syntax: Ts = TypeVar('Ts', variadic=True) means "some series of types T_0, T_1 .. T_n".

Variadic type variables are contagious; when you use one as a type argument to some other type, it too becomes variadic: List[Ts] now means "some series of types List[T_0], List[T_1] .. List[T_n]". Each variadic type contains exactly one (for now, we can think about more later) variadic type variable it its tree of types-with-arguments.

To expand a variadic type in function arguments, give the variadic type as the type of a *args parameter. For example, def do_stuff(*args: Ts) -> None: ... says that the arguments to do_stuff, because Ts is variadic, are not uniformly the same (joined) type, but rather they each get their own type variable, however many are needed.

Without a return type, that particular type signature is not particularly useful. Aside from *args, there are two other places you can expand variadic types:

• Tuple[Ts, ...], because Ts is variadic, is on expansion a particular tuple type with however many type variables Ts needs to expand to, Tuple[T_0, T_1, .. T_n]. Similarly, Tuple[List[Ts]] becomes Tuple[List[T_0], List[T_1] .. List[T_n]]. Note that this preserves the notion that for a non-variadic type, the meaning of the syntax is a uniform but unknown-length series of that type, but for a variadic type, the meaning is a series of particular types, each with a bindable type variable at its core.
• Callable[[Ts, ...], R] is on expansion a callable taking as arguments however many variables Ts needs to expand to, Callable[[T_0, T_1 .. T_n], R]

Examples

With this syntax, here are the discussed use-case signatures (I think):

Ts = TypeVar('Ts', variadic=True)
R = TypeVar('R')

def zip(*args: Iterable[Ts]) -> List[Tuple[Ts, ...]]: ...

def map(fn: Callable[[Ts, ...], R], *args: Iterable[Ts]) -> List[R]: ...

def execute_all(*args: Query[Ts]) -> Tuple[Ts, ...]: ...

Some functions take some positional fixed arguments and some variadic arguments. For example, here's a version of map that expects a function that also takes an index:

def map_with_idx(fn: Callable[[int, Ts, ...], R], *args: Iterable[Ts]) -> List[R]:
    results = []  # type: List[R]
    for i, rotated_args in enumerate(zip(*args)):
        results.append(fn(i, *rotated_args))
    return results

(By extension, Tuple[int, Ts, ...] could also mean Tuple[int, T_0, T_1 .. T_n]. I don't consider it quite as important, but it's nice symmetry.)

Details: Function body typechecking

To typecheck the function body, there is very little we can conclude about the expansion of a variadic type -- the implementation of nearly any non-trivial function will almost certainly iterate over its *args at some point, and making next() return a different type for each iteration of the loop sounds too complicated. We can, however, replace all instances of our variadic type variable with Any while typechecking the body, and get a reasonably satisfactory result:

# This is how the typechecker sees it when checking the body
def map_with_idx(fn: Callable[[int, Any, ...], R], *args: Iterable[Any]) -> List[R]:
    results = []  # type: List[R]
    for i, rotated_args in enumerate(zip(*args)):
        results.append(fn(i, *rotated_args))
    return results

Details: Splatting arguments in

In the implementation of map_with_idx above, we're calling zip(*args), and for all the typechecker knows, args is of type Iterable[Iterable[Any]]. In this case, we don't actually have any known number of arguments at the call site to fill in an appropriate number of type variables -- so we replace the variadic type variable with Any again. To typecheck this particular call to zip where we can't calculate the number of substitutions, the variadic type expansion phase tells later phases to use the signature def zip(*args: Iterable[Any]) -> Tuple[Any, ...]

The true meaning of ... in a comma-separated series of types is something like:

• If the preceding type is variadic, and we have a valid substitution of a series of type variables handy, make the substitution.
• If the preceding type is variadic, and we have no such valid substitution, treat the replace the variadic type variable in the preceding type with its bound (if no bound, Any), making it non-variadic, and continue to the next bullet point.
• If the preceding type is non-variadic, treat it as an arbitrary-length sequence of that type.

Details: Implementation!!!

I implemented part of what I described. I might actually get to a little more tomorrow. What I've got so far:

• Knows how to look at *args to count how many variables to substitute
• Knows how to substitute that many type variables inTuple
• Knows how to do Any-replacement to typecheck function bodies and splats, but not yet the more accurate bound-replacement.
• Can successfully typecheck my execute example, and I think zip, but not yet map because I haven't gotten to Callable yet.

The buried lede:

https://github.com/python/mypy/compare/master...sixolet:variadic-types

[sixolet]

The link to the implementation above should now support full typechecking at the callsite of the following functions, as long as you don't splat args into them (and a fallback to some kind of reasonable use of Any if you _do_ splat args into them):

Ts = TypeVar('Ts', variadic=True)
R = TypeVar('R')

def my_zip(*args: Iterable[Ts]) -> Iterator[Tuple[Ts, ...]]:
    iterators = [iter(arg) for arg in args]
    while True:
        yield tuple(next(it) for it in iterators)

def make_check(*args: Ts) -> Callable[[Ts, ...], bool]:
    """Return a function to check whether its arguments are the same as this function's args"""
    def ret(*args2: Ts) -> bool:
        if len(args) != len(args2):
            return False
        for a, b in zip(args, args2):
            if a != b:
                return False
        return True
    return ret

def my_map(f: Callable[[Ts, ...], R], *args: Iterable[Ts]) -> Iterator[R]:
    for parameters in zip(*args):
        yield f(*parameters)

[sixolet]

(It's not pull-request-ready by any means, but all the type system features are there, just not all the error messages and not-crashing-if-you-do-something-I-didn't-think-of)

[gvanrossum]

FWIW, just to play devil's advocate, here's a more pedestrian approach: https://gist.github.com/gvanrossum/86beaced733b7dbf2d034e56edb8d37e

[ilevkivskyi]

I think the proposal by @sixolet is very interesting. I just have one question to clarify: assume that I have a decorator

def deco_int(func):
    def decorated(*args, **kwargs):
        return int(func(*args, **kwargs))
    return decorated

What should be the type of such decorator? Is it like this one:

R = TypeVar('R', bound=SupportsInt)
Ts = TypeVar('Ts', variadic=True)

Arg = Callable[[Ts, ...], R]
Return = Callable[[Ts, ...], int]

my_deco = deco_int # type: Callable[[Arg], Return]

is it OK to have **kwargs in the decorated function?

[gvanrossum]

Regardless of the code generation approach, let me try to describe what's going on in my own words.

• A _type variable_ is made variadic using T = TypeVar('T', variadic=True).
• A _type_ is variadic if it contains a variadic type variable somewhere.
• A _function definition_ is variadic if it contains a variadic type variable anywhere in its signature. For now we only consider a single variadic type variable per function definition.

A variadic function definition is a shorthand for an infinity (plus one :-) of overloads:

• One where the variadic type variable is replaced with its upper bound. (This is a fallback and should only be used when the argument count is unknown at compile time, i.e. for what Naomi calls "splatting arguments in".)
• One for each non-negative integer _n_ where the variadic type variable Ts is expanded into _n_ separate regular type variables Ti, where _i_ ranges from 1 through _n_ (inclusive).

For example if _n_ is 3, we have 3 regular type variables T1, T2 and T3. Note that if _n_ is 0 we have zero regular type variables.

The expansion is itself a loop for _n_ in 0, 1, 2, ...; at each iteration an overload is generated using the following rules:

• A _variadic varargs argument_, i.e. an argument of the form *<name>: X, where X is a variadic type, expands into _n_ separate arguments numbered _i_, whose types are expanded from X by replacing Ts with the corresponding Ti.
• A _variadic tuple_ of the form Tuple[X, ...], where the ... is a literal ellipsis and X is a variadic type, expands into a tuple with _n_ separate item types numbered _i_, whose types are expanded from X by replacing Ts with the corresponding Ti.
• A _variadic callable_ of the form Callable[[<args>, X, ...], R], where the ... is a literal ellipsis and X is a variadic type, expands into a callable with _n_ separate argument types numbered _i_, whose types are expanded from X by replacing Ts with the corresponding Ti. The expanded callable has no ....

There are some ambiguities and missing details in these rules; let me address them as follows:

• When expanding a variadic type X according to one of the above expansion rules, X must not itself contain a variadic tuple or callable. For example, Tuple[Tuple[Ts, ...], ...] is invalid because the X in the outer tuple is Tuple[Ts, ...] which is itself a variadic tuple.
• When expanding a variadic callable, the return type R should not be variadic. When the argument list is not variadic the return type is allowed to be variadic.
• After expansion according to the above rules, a function signature should not contain any left-over variadic type variables. For example, this is invalid (assuming Ts is variadic): def foo(arg: List[Ts]): pass because Ts does not occur inside a variadic tuple, variadic callable or varargs argument, so it would not be expanded. Similar for def foo(arg: Ts, *args: Ts): pass -- the second Ts is subject to expansion but the first is not.

Finally, note that Callable[..., R] (where ... is a literal ellipsis) is not a variadic callable.

[gvanrossum]

A separate comment: I'm not sure I like the "cute" syntax for indicating variadic callables and tuples. An alternative syntax would be to use Tuple[Expand[X]] instead of Tuple[X, ...] and Callable[[<args>, Expand[X]], R] instead of Callable[[<args>, X, ...], R. The various disambiguations then become:

• In Expand[X], X cannot contain a further Expand[...] form.
• Ts may only occur inside Expand[...] and in the type of a _varargs argument_.

In fact, there's probably further economy to be had by clarifying that a _variadic varargs argument_ has an implicit Expand[...].

Further notes:

• Maybe it should be Expand(X) instead of Expand[X] since expansion is only allowed in specific contexts.
• (Unrelated to Expand): In some cases we may want to start counting at 1 instead of 0; in particular, while Python allows zip() without arguments, it does not allow map(func) without further arguments.

[NYKevin]

If we are contemplating an explicit Expand[X] syntax, I would like to reiterate that [*X] is also a possible spelling (though not necessarily the _best possible_ spelling; I just want to ensure we don't forget that it exists).

[gvanrossum]

But [*X] is not valid Python 3.5 syntax.

[gvanrossum]

Oh wait. It is! PEP 448.

[sixolet]

If *Xs is valid python syntax (specifically, without the braces), I like it. It brings a nice symmetry, where the type of *args can be *Xs, the type of a Callable where the last argument is variadic can have a type that looks variadic, etc. The above syntax with Xs, ... had me wishing that *args was spelled args, ... (and I think @gvanrossum mentioned that in person, too)

[sixolet]

@ilevkivskyi , yeah, I don't know exactly how **args and optional arguments should be treated for a more complete description of all possible types of functions. The syntax we've been playing with (and the code I've been playing with) handles *args reasonably well, but not further yet. I read your comment here https://github.com/python/typing/issues/239#issuecomment-234716559 last night, and it got me pondering -- decorators are a great use case for being able to do better, and as of yet my branch can't even handle your "change the return type to int" decorator example well. That's because it relies on a variadic type as the type of a *args to count how many substitutions to produce for a call site.

With some care we could solve that problem and write a version that handles positional arguments just fine by looking at exactly how many arguments the input function has (and has some clever substitution if the input function has a *args (or even, and this hurts my head, if the input function has a variadically-typed *args, but the full generality we need to make it good as a general decorator needs at least good handling of **kwargs, like you point out.

I'll play with that on a whiteboard soon, and also do you have ideas as to how it could work?

[gvanrossum]

Well, *Xs is valid Python syntax only inside a list, tuple or set literal, and only in Python 3.5. And that does not include subscripts, so while you could indeed write Callable[[*Xs], R], you couldn't write Tuple[*Xs]. So it's far from ideal.

[NYKevin]

It could, at least, have a plausible non-type-hinting meaning:

foo[a, b, *rest] == foo[(a, b, *rest)]

Then you just have to leave the door open for foo[*rest] == foo[(*rest,)] == foo[tuple(rest)]. But this would need to be a whole new PEP and I've no idea if it would make it into 3.6 or if it's useful enough to bother with.

OTOH, if you're doing Expand[Xs] anyway, [*Xs] might be more concise (compare Tuple[Expand[Xs]] to Tuple[[*Xs]] or (with the above Optional[PEP]) just Tuple[*Xs]).

[gvanrossum]

If we allowed everything with a plausible meaning the language would quickly degenerate into perl. And more concise is not automatically better. The use case here is rare enough that I care more about being able to look up uncommon syntax than about being able to type it fast.

[sixolet]

Ok, @ilevkivskyi, I wrote a shot at an answer about how to deal with the types of higher-order functions in full generality over at the thread on it. https://github.com/python/typing/issues/239#issuecomment-236765463

It involves some complication. I think I wrote up enough that you can get the idea, but I'm happy to clarify any bits left unclear, and I still have to provide some examples to show how *args and **kwargs interactions would work.

[sixolet]

Resurrecting this thread a little! To add to the design space for how to show the expansion of a variadic type variable in an argument list or tuple:

Ts = TypeVar('Ts', variadic=True)
R = TypeVar('R')

def my_zip(*args: Iterable[T] for T in Ts) -> Iterator[Tuple[T for T in Ts]]:
    ...

def my_map(f: Callable[[T for T in Ts], R], *args: Iterable[T] for T in Ts) -> Iterator[R]:
    ...

[gvanrossum]

Didya check those aren't syntax errors? They may need uglyfying parens.

[sixolet]

Oh damn. Yes, they need uglifying parens. That kind of kills half the beauty. The other half is that it's clear exactly which variadic type var you'd like to expand, which was bothering me (in the presence of more than one -- I was going to disallow it)

[gvanrossum]

Another issue is that it won't be possible to find out the type at runtime, I think (or at least a tour de force).

[sixolet]

Because the runtime object is going to be a generator?

I do like the idea of being explicit about which type var to expand. That's possible in the Expand(Foo[Ts], Ts) syntax, but not so much with the ... syntax.

[gvanrossum]

In the meantime, a hacky workaround could be made to work easily: https://gist.github.com/gvanrossum/4232a4cdad00bd92a7a64cf3e2795820

[ilevkivskyi]

In the meantime, a hacky workaround could be made to work easily:

And one can use this in an automated way that you proposed earlier.

[shoyer]

To note another use-case, variadic generics would be quite useful for typing shapes for multi-dimensional arrays (#513, #516), e.g., Shape[N] and Shape[N, M].

[seaders]

The use case I'm looking for is very similar to @sixolet's, with sqlalchemy. With that, you're able to do a query like Base.query.with_entities(User, Role).all(), which returns List[Tuple[User, Role]], but isn't properly annotated, so is only identified as a list. This is an issue for a ton of sqlalchemy queries, but I've been able to get around them with shortcuts so far, but not this one.

For my use case, I have a function in Base,

@classmethod
def qwith_entities(cls, models: T) -> List[T]:
    return cls.query.with_entities(*models)

As above, with Base.qwith_entities((User, Role)), this correctly returns List[Tuple[User, Role]], but it's typehinted as List[Tuple[Type[User], Tupe[Role]]]. In my situation, I'd like to annotate models as something similar that's been described already, a tuple of multiple types, and "unpack" those types for the return value. Something like,

@classmethod
def qwith_entities(cls, models: Tuple[..., Type[T]]) -> List[Tuple[..., T]]:
    return cls.query.with_entities(*models)

I'm not at all wed to this syntax either, and do have a solution working via overloads, but would like to be able to do this without the need for them.

[gvanrossum]

CC @ilevkivskyi, who is working on basic SQLAlchemy stubs plus a mypy
plugin. (And also on type system changes, though I don't think we have
plans for variadic type variables yet.)

[ilevkivskyi]

@seaders I was going to use overloads for such scenarios in my SQLAlchemy stubs. It is unlikely that mypy will support variadic generics this year. Btw, we could join our efforts in writing SQLAlchemy stubs, I am going to publish them on GitHub when I am back from vacation.

[seaders]

@ilevkivskyi yeah, absolutely. I've done a load of work on generic-y helpers for my own app (the entire design app is very paired with the design of the dB), and I'd be more than happy to contribute them back to the community.

I'll start separating them out from my main codebase, and you can just shout me whenever's good for you.

[ilevkivskyi]

@seaders If you are still interested we are working on SQLAlchemy stubs (and soon some mypy plugins) at https://github.com/dropbox/sqlalchemy-stubs. It is still far from complete, so your help will be appreciated. You can look at issues and PRs to see what are the plans and what you can add.

[vsiles]

Hi ! I'm currently trying to see if the variadic generics could help to type some Pytorch Tensor operations. It definitely seems doable to type static functions, for example:

Ts = TypeVar('Ts', variadic=True)
class Tensor(Generic[Ts]):
   ...

A = TypeVar('A')
B = TypeVar('B')
C = TypeVar('C')
def mm(x : Tensor[A, B], y : Tensor[B, C]) -> Tensor[A, C]:
   ...

However a lot of these functions are usually used as instance methods (e.g. x.mm(y) instead of mm(x, y)). Do you think of allowing some kind of specialized methods ? Something like

class Tensor(Generic[Ts]):
   def __add__(self, x: Num) -> Tensor[Ts]:
    # adding a scalar to all cells of the tensor, no shape changes happen. Ok with the current proposition
       ...

  def mm(self: Tensor[A, B], y: Tensor[B, C]) -> Tensor[A, C]:
  # here it only is available for Tensor with exactly two dimensions. I don't see it working with the current propostion
     ...

[ilevkivskyi]

Yeah, methods are known to be tricky. They may be not supported in the initial implementation in mypy.

In general, there is a similar problem for normal generics where some methods are only available for given values of type variables.

However a lot of these functions are usually used as instance methods

If this will show to be a real problem, then we (mypy team) might prioritize support for restrictions on self (as in your example). This was already proposed in the context of normal generics.

[gvanrossum]

@jakebailey (MS Language Server representative)

[davidatbu]

Hi all. Thank you for mypy!

So, what is the plan going forward regarding variadic generics?

[mrahtz]

A few of us have actually been working on a draft of a PEP for variadics generics since last year, PEP 646. Eric Traut has very kindly contributed an initial implementation in Pyright, Pradeep Kumar Srinivasan has been working on an implementation in Pyre, and we're working on the additions to typing.py in this pull request. Sorry for the late notice in this thread; I do remember seeing this thread a long time ago but apparently it didn't stick in my memory.

@NYKevin As of the current draft, I think your use case with MultiField would look like this using the current PEP draft:

GetSetVar = TypeVar('GetSetVar')
TagVar = TypeVarTuple('TagVar')

class MultiField(AbstractField[GetSetVar], Generic[*TagVar]):
    def __init__(self, nbt_names: Sequence[str], *, default: GetSetVar = None) -> None:
        ...

    @abc.abstractmethod
    def to_python(self, *tags: *TagVar) -> GetSetVar:
        ...

    @abc.abstractmethod
    def from_python(self, value: GetSetVar) -> Tuple[*TagVar]:
        ...

@sixolet I think you've seen the thread in typing-sig about 646, but for the sake of other people reading this thread: the Query use case isn't supported by the current PEP; it used to be, but the PEP was getting too big, so we postponed this for a future PEP. The section we cut out is in this doc. Here's what the Query case would look like if we did use the proposal in that doc: (though of course there's still plenty of room for more discussion on this; there are some interesting proposals in this thread!)

R = TypeVar('R')
Rs = TypeVarTuple('Rs')

class Query(Generic[R]):
    ...

def execute(*args: *Map[Query, Rs]) -> Tuple[*Rs]:
    ...

@seaders I think your use case should be possible with:

Ts = TypeVarTuple('Ts')

@classmethod
def qwith_entities(cls, models: Tuple[*Ts]) -> List[Tuple[*Ts]]:
    return cls.query.with_entities(*models)

In any case, if you have any feedback on the current draft of the PEP, please do drop by the thread in typing-sig and leave us a message. Cheers!