Typing: Proposal: signature copying for kwargs.

[dmoisset]

There's a quite common pattern in python code which is:

def function(foo, *args, **kwargs):
    # do something with foo
    other_function(*args, **kwargs)
    # possibly do something else

def other_function(color: str=..., temperature: float=..., style: Stylesheet=..., timeout: Optional[int]=..., database_adaptor: Adaptor=..., strict: bool=..., output: IO[str], allow_frogs: bool=..., mode: SomeEnum=...):
    # do something with a lot of options

(a usual subcase of this one is when other_function is actually super().function ). This presents two problems for a static analyzer:

• the call from function to other_function can not be type-checked properly because of the *args, **kwargs in the call arguments.
• there is no sensible way to annotate function, so calls to it are unchecked.

The problem for me also affects readability of the code (which is for me one of the main problems that annotations tries to address). James Powell from numfocus even gave a pydata talk about the difficulties it brings at https://www.youtube.com/watch?v=MQMbnhSthZQ

Even if theoretically the args/kwargs packing feature of python can be used with more or less arbitrary data, IMO this use-case is common enough to warrant some special treatment. I was thinking on a way to flag this usage, for example:

@delegate_args(other_function)
def function(foo, *args, **kwargs):
    other_function(*args, **kwargs)

This could hint an analyzer so:

• On calls to function, the "extra" arguments are checked to match the signature of other_function
• The call to other_function is considered valid given that it uses the same arguments (I know that the code above could have modified the content of kwargs, but it's still more checking than what we have now).

For me, even without static analyzer, the readability benefits of seeing

@delegate_args(matplotlib.pyplot.plot)
def plot_valuation(ticker_symbol: str, start: date, end: date, *args, **kwargs): ...

and knowing that plot_valuation accepts any valid arguments from matplotlib's plot function, is worth it.

[gvanrossum]

Gotta make this quick: Yes, we experience this in our own (Dropbox) code bases a lot too. And I agree this is useful for readability even without static checking. I think there are some details to be worked out -- what I often see is that the wrapping function adds some fixed args to the function in delegates too, so those should not be accepted:

def raw(name='me', age=42): ...

def cooked(first_name, last_name, **extra):
    raw(name=first_name + ' ' + last_name, **extra)

cooked(age=100)  # OK
cooked(name='Guido')  # Error

Also of course the wrapper may also have some of its own arguments that are unrelated to the wrapped function.

[JukkaL]

I've seen this pattern quite frequently as well, and it would be nice to support it somehow. Refactoring existing code to use explicitly spelled out individual arguments instead of *args and **kwargs can be lot of work and make the code harder to maintain, especially if there are many args to propagate.

Expanding on the decorator idea, maybe it should be possible to declare that a function accepts all positional and keyword args another function accepts, _except_ for a set of args. Example:

def raw(name='me', age=42): ...

@delegate_args(raw, except=['name'])
def cooked(first_name, last_name, **extra):
    raw(name=first_name + ' ' + last_name, **extra)

cooked(age=100)  # OK
cooked(name='Guido')  # Error

Implementing this in a type checker looks a little tricky, especially when using both *args and **kwargs. Special-casing the whole thing looks possible, but sometimes programmers mutate the **kwargs dictionary and type checking _that_ would likely require some sort of support for "dict-as-struct".

[dmoisset]

OK, to support a bit of this with data I did a small experiment: take a random sample of stdlib files until I got about 100 function definitions with open keyword args; then another random sample in django with 100 more defs, and check manually what they do with the arguments. My finding say that:

1. 17% of the defs with **k are for functions that never use k
2. 18% of the defs with **k actually use k as a dictionary (calling get(), pop(), iterating on it, etc., but also storing it into a variable or attribute, or passing k as a parameter but without **)
3. 40% of the defs with **k pass it as is to another function, without adding more positional or keyword arguments on the call (This percentage also includes the functions that pass *varargs as is too).
4. 24% of the defs with **k pass it (with ** again) to another function, but the call includes some extra arguments (positional or keyword).
5. 1% of the defs with **k use it for self.__dict__.update(k).

Case 1 is trivial and works well already with Any or object. Case 2 is the hard one (when the dictionary is heterogeneous, which appears to be the most frequent scenario, although I didn't collect that data). Case 5 could be a verification target for a type checker but seems quite unusual. My proposal would cover case 3, and with @JukkaL addition, a part of case 4.

So "hard" cases would be reduced to about 20% of the def statements with **kwargs according to this semi-scientific experiment :) .

From what I saw of case 4, actually I think that a lot of cases could be covered automatically. The example that @JukkaL posted is not very common. The common case for those (at least in my sample) are the following two:

def raw(name=..., age=...): ...

@delegate_args(raw)
def cooked(name, **extra):
    raw(name=name.upper(), **extra)
    # here the typechecker could deduce that `**extra` never contains a `name`, because it's a named argument of "cooked"

def raw2(a, b, c, name=..., age=...): ...

@delegate_args(raw)
def cooked2(a, b, c, **extra):
    raw(a+1, b, c, **extra)
    # here the typechecker could deduce that `**extra` never contains `a`, `b`, or `c`, because they are named arguments of `cooked2`

So the except=... proposal is helpful, but in just a handful of cases (which I prefer, and also avoids the "except" keyword). There's also this case (which I included in my group 2) that could benefit from a similar proposal which is:

@delegate_args(raw, include=['foo'])
def cooked(**extra):
    foo = extra.pop('foo', SOME_DEFAULT)
    do_something_with(foo)
    raw( **extra)

I would have declared foo as an explicit argument and perhaps that's what we should recommend, but I'm a bit surprised on how relatively common this pattern is.

[JukkaL]

@dmoisset Thanks for the careful analysis! Having data like this makes decisions much easier. So the conclusion seems to be that this would be useful even without support for dict-as-struct. It might be useful for us to run a similar analysis against Dropbox code to get another data point.

[dfee]

I came across this issue, and wanted to present my solution to the problem:

import forge

def raw(name='he', age=42):
    return f'{name} is {age}'

@forge.compose(
    forge.copy(raw, exclude=('name')),
    forge.insert((forge.arg('first_name'), forge.arg('last_name')), index=0),
)
def cooked(first_name, last_name, **extras):
    extras['name'] = first_name + ' ' + last_name
    return forge.callwith(raw, extras)

(and some tests that validate that it works as expected)

import inspect
assert repr(inspect.signature(cooked)) == '<Signature (first_name, last_name, age=42)>'
assert raw(age=100)  == 'he is 100'
assert raw('Guido', age=42) == 'Guido is 42'
assert cooked('Guido', 'VR', 42) == 'Guido VR is 42'

try:
    cooked(name='Guido')  # Error
except TypeError as exc:
    assert exc.args[0] == "cooked() missing a required argument: 'first_name'"

forge is a package I recently released that allows for signature revision (add, remove, modify parameters as well as convert or validate argument values, etc.).

I'm curious as to whether a mypy plugin could be written that would allow for static analysis.

[alexmojaki]

This is quite the mouthful:

@forge.compose(
    forge.copy(raw, exclude=('name')),
    forge.insert((forge.arg('first_name'), forge.arg('last_name')), index=0),
)
def cooked(first_name, last_name, **extras):

How about instead forge.copy just doesn't interfere with arguments in the function it's decorating that don't have * or **? In other words the forge.insert could be automatic. If somehow that turns out to be a problem in some cases it could be turned off with a keyword argument.

I don't understand the need for this:

    extras['name'] = first_name + ' ' + last_name
    return forge.callwith(raw, extras)

Why not raw(name=first_name + ' ' + last_name, **extra)?

[dfee]

So forge can do more than solve the problem discussed here. You bring up an interesting point though, is that a common-enough problem to which a canned decorator should be provided? I'll share a snippet in a little bit.

The reason for callwith is that reconstituting arguments (especially the ordering of positional or keyword parameters is a non-trivial problem:

import forge

def func(a, b, c, d=4, e=5, f=6, *args):
    return (a, b, c, d, e, f, args)

@forge.sign(
    forge.arg('a', default=1),
    forge.arg('b', default=2),
    forge.arg('c', default=3),
    *forge.args,
)
def func2(*args, **kwargs):
    return forge.callwith(func, kwargs, args)

assert forge.repr_callable(func2) == 'func2(a=1, b=2, c=3, *args)'
assert func2(10, 20, 30, 'a', 'b', 'c') == (10, 20, 30, 4, 5, 6, ('a', 'b', 'c'))

The alternative to that is manual interpolation of argument values which is just as big of a problem:

import forge

def func(a, b, c, d=4, e=5, f=6, *args):
    return (a, b, c, d, e, f, args)

@forge.sign(
    forge.arg('a', default=1),
    forge.arg('b', default=2),
    forge.arg('c', default=3),
    *forge.args,
)
def func2(*args, **kwargs):
    return func(
        kwargs['a'],
        kwargs['b'],
        kwargs['c'],
        4,
        5,
        6,
        *args,
    )

assert forge.repr_callable(func2) == 'func2(a=1, b=2, c=3, *args)'
assert func2(10, 20, 30, 'a', 'b', 'c') == (10, 20, 30, 4, 5, 6, ('a', 'b', 'c'))

I explain that further, in the docs.

[alexmojaki]

OK, that's cool, but:

1. Anyone who writes

def func(a, b, c, d=4, e=5, f=6, *args):

instead of

def func(a, b, c, *args, d=4, e=5, f=6):

deserves to be slapped, and

1. If I understand correctly, the cooked/raw example can work fine without callwith?

[dfee]

Both are correct :)

[dfee]

Alright, I got a minute to draft out an extend revision:

import functools
import inspect

import forge


class extend(forge.Revision):
    """
    Extends a function's signature...
    """
    def __init__(self, callable, *, include=None, exclude=None):
        # pylint: disable=W0622, redefined-builtin
        self.callable = callable
        self.include = include
        self.exclude = exclude

    def revise(self, previous):
        extensions = forge.fsignature(self.callable)
        if self.include:
            extensions = list(forge.findparam(extensions, self.include))
        elif self.exclude:
            extensions = [
                param for param in extensions
                if param not in forge.findparam(extensions, self.exclude)
            ]

        params = [
            param for param in previous
            if param.kind is not forge.FParameter.VAR_KEYWORD
        ] + list(extensions)

        return forge.FSignature(params)

Usage is straightforward:

def raw(name='he', age=42):
    return f'{name} is {age}'

@extend(raw, exclude=('name'))
def cooked(first_name, last_name, **extras):
    return raw(name=f'{first_name} {last_name}', **extras)

And the tests still pass:

assert repr(inspect.signature(cooked)) == '<Signature (first_name, last_name, age=42)>'
assert raw(age=100)  == 'he is 100'
assert raw('Guido', age=42) == 'Guido is 42'
assert cooked('Guido', 'VR', 42) == 'Guido VR is 42'

try:
    cooked(name='Guido')  # Error
except TypeError as exc:
    assert exc.args[0] == "cooked() missing a required argument: 'first_name'"

The caveats to this approach are that a user must remain wary of the ordering of "parameter kind" and parameters with default values (or use within a @forge.compose(forge.extend(...), forge.sort()) construct.

[JukkaL]

python/mypy#5559 has an implementation of the basic proposal with exclude= by @alexmojaki as a mypy extension. Before moving on, I'd like to discuss the design a bit more.

In particular, there's the question of what syntax to use. All examples above use a decorator, but @ilevkivskyi also suggested (in python/mypy#5559) using a TypedDict (another mypy extension) or a protocol-based callable as an annotation. Here's an example using a callable annotation from the above PR:

class DoStuff(Protocol):
    def __call__(a: int, b: int = ..., c: int = ...) -> None: ...

f: DoStuff
def f(*args, **kwargs):
    ...

g: DoStuff
def g(*args, **kwargs):
    f(*args, **kwargs)

This is not optimal if the original function (such as f above) doesn't use *args and/or **kwargs for all arguments, since then we'd need to duplicate the argument names and types, even though that's what the proposal is trying to avoid. So there would arguably only be a benefit if there are two or more functions that delegate to f.

Here's another idea (which will likely be harder to implement):

def f(x: int, y: str = '') -> None:
    ...

g: TypeOf[f]
def g(*args, **kwargs):
    ...

We'd introduce a TypeOf[x] type operator that is equivalent to the type of the expression within square brackets. It could potentially also be used for other things, such as callbacks, though these use cases may be pretty marginal:

def default(x: int, y: str = '') -> None:
    ...

def do_stuff(cb: TypeOf[default] = default) -> int:
    ...

This would only cover uses cases where the signatures of the two functions are identical.

Even if we continue with the decorator proposal, we don't have an agreement on what we'd call it. The ideas above aren't quite self-explanatory enough, in my opinion. The feature is not very widely useful, so I feel like we should try to make name very clear and explicit. Here are a bunch of random ideas (using a real example from subprocess):

@copy_signature(Popen)
def call(*popenargs, timeout: Optional[float] = None, **kwargs)): ...

@inherit_signature(Popen)
def call(*popenargs, timeout: Optional[float] = None, **kwargs)): ...

@with_signature(Popen)
def call(*popenargs, timeout: Optional[float] = None, **kwargs)): ...

@use_signature(Popen)
def call(*popenargs, timeout: Optional[float] = None, **kwargs)): ...

@use_signature_from(Popen)
def call(*popenargs, timeout: Optional[float] = None, **kwargs)): ...

@apply_signature(Popen)
def call(*popenargs, timeout: Optional[float] = None, **kwargs)): ...

Here my rationale is that the decorator actually does no delegation -- the assumption is that the function delegates to another function, but that's actually not enforced by the feature, I assume. So the effect is to take the signature of another function and apply it to the decorated function.

[ilevkivskyi]

Here are few more random ideas/comments:

• Instead of TypeOf[f], that may give an impression that this applies to any other object, I would propose to allow single argument Callable like this:
  python def f(x: int, y: str = '') -> None: ... g: Callable[f] def g(*args, **kwargs): ... def do_stuff(cb: Callable[f] = f) -> int: ...

• ...protocol-based callable as an annotation...
  So there would arguably only be a benefit if there are two or more functions that delegate to f.

  Yes, the same also applies to using TypedDict to annotate **kwargs. Allowing TypedDict in annotations has its own use case, https://github.com/python/mypy/issues/4441, so probably it is a bit orthogonal here.

• If we go with the decorator approach, maybe again we can use (now stalled) @declared_type? (I am not sure I like this however.)

My preference would be probably to go with g: Callable[f].

[dfee]

That's great for copying a signature, but it does nothing for the just-as, if-not-more common case where you want to mutate the parameters: https://github.com/python/typing/issues/270#issuecomment-242800710

[alexmojaki]

A decorator allows performing actions at runtime such as setting the __signature__ attribute. This cannot be done with an annotation. To me this is a complete dealbreaker for using an annotation. I could elaborate some reasons for this, but do I need to? Does anyone feel that runtime signatures aren't such an important feature for this proposal?

[gvanrossum]

Yes, please explain your use cases and spare us further rhetoric (words like "complete dealbreaker" and "but do I need to?").

[alexmojaki]

Sorry, it was a genuine question, I didn't want to waste my time or anyone else's preaching to the choir.

Setting __signature__ on the decorated function means that inspect.signature and many of the nice features that follow (.parameters, .bind, .bind_partial, etc.) become automatically available. I don't know exactly what arguments led to the proposal and acceptance of these features in PEP 362 (Function Signature Object), but it seems to me that all or at least most of those same arguments should apply here. If a user has any interest in inspecting a 'normal' function such as raw, they would be equally interested in inspecting a delegating function such as cooked, and would be disappointed if they didn't get the same quality of information back.

This goes well beyond specific obscure use cases where a programmer wants to do some clever introspection for their own application. Here are some ways in which __signature__ can affect the experience of an average Python programmer trying to understand or use a function, depending on which tools they use.

In PyCharm (apparently the most popular editor for Python), a keyboard shortcut shows the parameters of a function for which the user is currently writing a call. Personally, I use this feature often, and it always annoys me when I get back a meaningless *args, **kwargs:

In the console, setting __signature__ fixes this problem:

In Jupyter notebooks, the most common editor for scientific developers after PyCharm (same survey above), the signature is used for parameter autocompletion:

And of course, there's the builtin help:

Help on function cooked:

cooked(name='me', age=42)

[chadrik]

In PyCharm (apparently the most popular editor for Python), a keyboard shortcut shows the parameters of a function for which the user is currently writing a call. Personally, I use this feature often, and it always annoys me when I get back a meaningless args, *kwargs:

Note that PyCharm's editor uses static analysis (like mypy), so setting __signature__ at runtime will have no affect on its completion, though it very well might in its interpreter/console (I can't say as I don't use this feature). For the editor at least, PyCharm will need to add explicit support for whatever convention is established as a result of this discussion.

[alexmojaki]

Yes, I did specifically mention the console. And seeing how much static analysis PyCharm has already implemented, I think it's most likely that they will implement this feature too. Personally I want this feature in the standard library precisely so that I can make use of it in PyCharm, so if they don't implement it, I might even do it myself. In any case none of this affects the decorator vs annotation question.

[alexmojaki]

There's already a clear precedent to a @delegate decorator: @wraps from functools. wraps is typically used as follows:

@wraps(f)
def wrapper(*args, **kwargs):
    # some other stuff
    return f(*args, **kwargs)

where f has a specific, 'concrete' signature. This is exactly how the typical use case of a delegate decorator would look, the only difference being the word wraps. Of course, the context is different, namely that the code above for wraps would be inside another function which is a decorator, but the similarity is still there. Making that connection may make it easier for people familiar with wraps to understand delegate and to remember how to use it.

Based on this similarity, I think it makes sense for the two decorators to behave similarly in other ways. @wraps sets __wrapped__, @delegate can set __delegate__ or some similar attribute pointing to the base function. inspect.signature follows chains of __wrapped__ by default, but this can be switched off. The same can be done for @delegate, and it wouldn't actually set __signature__ as I've previously suggested.

typing.get_type_hints can also use information provided by delegate to produce meaningful type hints based on the base function, just as it does with wraps. Whether delegate actually sets __annotations__ or get_type_hints just uses __delegate__ can be discussed.

__delegate__ may have other uses too. One idea that comes to mind now is that automatically generated HTML documentation could include a hyperlink to the base function.

Beyond that, attaching information at runtime may have uses that none of us think of, maybe even uses that none of us could think of because the relevant Python features don't exist yet. Using an annotation is a decision that would be messy to reverse and effectively rules out these possibilities.

In terms of readability, another problem I have with using an annotation is that annotations are usually (if not always) an absolute declaration of the type of the annotated object, without taking the annotated object into account. If I see:

cooked: TypeOf[raw]
def cooked(foo, *args, **kwargs):
    raw(*args, **kwargs)

the impression given by cooked: TypeOf[raw] is that cooked has the exact type of raw, i.e. the same signature, when in fact cooked has a slightly different signature because of the foo parameter. Certainly the name TypeOf is partly responsible for this, but it's also just the nature of annotations.

[gvanrossum]

I just want to note that you seem to be focused primarily on runtime behavior, while others on this thread are focused primarily on static checkers (which operate without running or importing the code they are checking).

[alexmojaki]

I'm fine with static analysis being the priority. I even wrote the PR for that. We can defer actually making decisions about or implementing the runtime stuff until much later. But I think it should be possible to implement those features eventually, and it essentially won't be if we use an annotation. All of this is just an argument in favour of using a decorator instead of an annotation, in contrast to @ilevkivskyi's stated preference for an annotation.

[gvanrossum]

I think there's still a misunderstanding though. Annotations are also introspectable at runtime (typically through some __annotations__ attribute somewhere).

[alexmojaki]

The suggestion is to use a variable annotation, which is the only possibility I see. Given a function, finding a corresponding variable annotation is messy at best, and AFAICT impossible if the function is locally defined. And since Python versions before 3.6 have to mimic variable annotations using comments, it's definitely impossible in all cases to attach runtime information for those versions. Of course this feature won't be directly available in those versions, but it could easily be available in a backport.

[JukkaL]

I prefer using a decorator over a variable annotation, mainly because it allows some differences in the signatures. Based on analysis by @dmoisset above, it's pretty common that some arguments are different, or *args are not passed, etc. It's much easier to cover these cases with the decorator-based syntax. Maybe further analysis would be helpful here, but I'm pretty much convinced already that this is important. The annotation-based syntax only makes sense to me if the signatures are identical.

Here are some potential signature differences that the decorator-based approach can support:

• The caller takes an extra argument that the callee shouldn't accept.
• The callee does not accept some argument accepted by the callee (it may be provided explicitly in the call).
• The caller only accepts **kwargs -- no *args -- but the callee also accepts some positional arguments.

I think that both approaches can support runtime introspection (in 3.6 and later), so it's probably not an important factor.

The similarity to @wraps noticed by @alexmojaki suggests some additional possible names for the decorator:

• @inherits_signature(f)
• @delegates_signature(f)
• @reuses_signature(f)
• @extends_signature(f)

More random ideas:

• @fallback_signature(f)
• @signature_fallback(f)

[alexmojaki]

Continuing on my previous comment:

I think the most important target audience for this feature is big popular libraries that are used by the largest number of people and have the most interest in being usable and well documented. If the feature is a variable annotation, those libraries will only be able to make use of it once they drop support for Python 3.5 and older. That delay will affect all users of the libraries even if they're using the latest Python version in their own application.

[rmorshea]

Any progress on this? The verbose alternative (enumerating all delegated arguments) is prohibitive for people who are new to MyPy. Where does this sit in the list of priorities?

[untitaker]

We have workarounds for this in the Python SDK for Sentry, but they are quite ugly.

[alexmojaki]

Hi @rmorshea and @untitaker, thanks for joining the discussion.

I wrote a PR a while back but it was decided it was too soon for an implementation. Right now the discussion is stalled on choosing a name and syntax. Gathering more opinions might be helpful - what API would you think is best?

[untitaker]

I wrote a related PR https://github.com/python/mypy/pull/7051 but dropped the ball on this one. I might have to pick it up again if we see that my workarounds don't work well enough.

I have interest in having a good solution to this problem but we are not committed to having good typing support in our libraries at all.

[rmorshea]

@alexmojaki @JukkaL in terms of naming suggestions, could we just reuse wraps? It's familiar and communicates the idea that the function being decorated is expected to call the thing being wrapped in some manner. As a close second, I'm 👍 for inherit_signature.

Another thought though, when I write a class, I'd prefer to avoid having to explicitly pass the function whose signature I'd like to extend (the name inherit_signature seems rather nice in this use case):

class Parent:
    def method(self, z): ....

class Child(Parent):
    @typing.inherit_signature
    def method(self, x, y, **kwargs): ...
    # VS
    @typing.inherit_signature(Parent.method)
    def method(self, x, y, **kwargs): ...

In other words if you use this on a class and don't pass the function being wrapped it would be implied that the thing being wrapped is the method from Parent. Of course you should also be able to be explicit in cases where you need to do this:

class Parent:
    def method(self, z): ....

class Child(Parent):
    @typing.inherit_signature(Parent.method)
    def wrapper_method(self, x, y, **kwargs): ...
    # Or maybe...
    @typing.inherit_signature("method")
    def wrapper_method(self, x, y, **kwargs): ...

[rmorshea]

On second thought, a majority of my use cases for this involve inheritance, so inherit_signature seems quite apt.

[alexmojaki]

I think having two similar but subtly different functions named wraps will cause confusion when reading discussions, articles, tutorials, or code that simply has @wraps. A unique name is also helpful when using find/search/grep.

I agree that inheritance is a very important use case - I think there's a good chance it's the majority overall, not just in your code. And not having to pass (Parent.method) would be great. I think the wording also sounds quite natural and intuitive even if there is no inheritance (or even classes) involved, but I can sort of imagine that it might throw some other people off. Does anyone think that inherits_signature would be confusing to others when there's no inheritance, especially to novice coders?

Otherwise, @inherits_signature has my vote. Note that that's for inherits, not inherit. I think inherits_signature reads more naturally, and looks similar to wraps.

In any case, we have plenty of time to decide on a name and changing it before release is very easy. I think it's best that we move the discussion on every other point forward and not let naming hold things back.

Currently the most contentious point is decorator vs. annotation. @rmorshea, @JukkaL, and I are in favour of a decorator, while @ilevkivskyi has voted for an annotation. @ilevkivskyi, do you still feel this way? Does anyone else prefer an annotation? If so, can you explain this preference, and can you propose an API for cases where the two functions have slightly different signatures?

[ilevkivskyi]

Note that the particular case of inheriting signature from the parent class doesn't require any special syntax, it can be just a type-checker flag, see for example proposal https://github.com/python/mypy/issues/3903 for mypy (btw there is a WIP PR that aims to add such flag https://github.com/python/mypy/pull/7548)

[rmorshea]

Having this be a flag is MUCH preferred over having to decorate anything I overwrite. It almost seems like the flag should be on be default at some point in the future since, as I've mentioned above, enumerating all delegated arguments is prohibitive for new users and, in my experience at least, is an extremely common pattern in OOP in Python.

Being able to inherit a signature for regular functions is still something I'd use, but IMO would be less critical than enabling it when overwriting methods.

[untitaker]

Having a flag instead of a decorator prohibits this from becoming a standardized part of the type system. Even if the default behavior changes you still have a flag around to change fundamental behavior that should be part of a pep. This seems wrong.

[rmorshea]

@untitaker I don't think the flag would be used _instead of_ but rather _in addition to_ the decorator. The decorator would still have usages for regular functions, or in cases where you want to adopt the signature of a method which has a different name.

[alexmojaki]

1. I don't fully understand the reasoning behind inheriting signatures being enabled by a flag instead of the default behaviour, but I take it to mean that enabling that behaviour globally could be annoying. A decorator would allow selectively inheriting signatures when you don't want to do so everywhere.
2. The best use cases for this feature would probably be for libraries, as I mentioned above. Libraries cannot control whether their users turn on a flag when type checking. So a library may be better off using a decorator to ensure that when users run a type checker it inherits the signature for decorated overridden methods regardless of flags.
3. Is the flag meant to handle base methods with complex explicit signatures being overridden by (*args, **kwargs)? I can't see anything like that in the issue discussion or the PR tests. That's the kind of thing the decorator is meant for. It seems to me like the flag should handle that, but maybe I'm missing something.

[rmorshea]

@alexmojaki you're right, that PR doesn't address the needs expressed here, I should have read closer. Though much of the core logic (i.e. signature copying) seems related to whatever feature would come out of this issue.

And as mentioned above, I agree that this makes sense as the default behavior when overwriting methods in classes - most people won't dig through the docs to find this decorator. With that said, if we are able to write a clever error that could direct people to this decorator when we detect that they may be trying to delegate arguments then we could keep it off by default.

[rmorshea]

@JukkaL and @ilevkivskyi thoughts on:

• The name signature_inheritance
• Whether signature inheritance for methods should be on by default

[ilevkivskyi]

Whether signature inheritance for methods should be on by default

Definitely no. About a decorator vs a flag, let's first implement the flag in mypy, and then if people will ask about even finer granularity (per-method rather than per-module) making it a decorator would be trivial (although I doubt it will happen, mypy for example has dozens of flags and I don't remember anyone ever asking making it per-function).

[rmorshea]

@ilevkivskyi would it be possible to turn this on by default at some point in the future? Again, as noted by many people above, this is a fairly common pattern, and the alternative (enumerating all delegated arguments) is prohibitive for new users and library maintainers hoping to gradually type their code.

[ilevkivskyi]

would it be possible to turn this on by default at some point in the future?

Maybe, but it is to soon to talk about this now.

[rmorshea]

Also, I think that in most situations where I would use functools.wraps I would also want to use inherit_signature, so there are definitely use cases for per-function specificity.

That said, I don't have a problem with implementing this gradually (i.e. do the flag first, and then the decorator).

[alexmojaki]

Please remember that this decorator is not simply a more fine grained version of the flag. There are plenty of cases it can be used where there is no inheritance involved (and also where functools.wraps would not be appropriate either).

@ilevkivskyi, do you still prefer an annotation over a decorator? If so, can you explain this preference, and can you propose an API for cases where the two functions have slightly different signatures?

[JukkaL]

would it be possible to turn this on by default at some point in the future?

I don't think that this will make sense to implicitly inherit signatures in overridden methods (though making this an opt-in thing may be reasonable).

Here are some reasons why I think that this is not a good idea:

• I believe that the main benefit of type annotations is improving the readability of code, and hiding type annotations would go against that. If I'd want to know the signature of a method, I would have to find the library stub (and these are often not trivial to find if using, say, PyCharm).
• Inheriting signatures implicitly goes against gradual typing -- unannotated code shouldn't cause false positives.
• Having some methods annotated and others unannotated (while everything is still type checked) would look inconsistent, in my opinion.
• If a base class signature is changed, all overridden methods would inherit that change, which could result in hard-to-understand errors inside function bodies. If all methods have signatures, a type checker can easily diagnose that the overridden method has an incompatible signature.
• This would be a backward compatibility break.

[ilevkivskyi]

@rmorshea @alexmojaki Please stop throwing everything into a single pile. There are several independent use cases here:

• Inheritance of methods: should be solved with an opt-in flag
• Precise typing of decorators: should be solved by ArgSpecs and variadic generics (hopefully soon)
• Precise typing of **kwargs: should be solved by allowing TypedDicts in annotations (hopefully soon)
• The only use case that requires some new special syntax is precise type-checking of bodies of non-inherited (complex) functions that relay arguments to other functions.

Couple notes about the last one:

• I don't have strong preference for the syntax. Annotation syntax just looks more pleasing visually to me.
• This is only about function _bodies_, copying signature for external callers already works.

F = TypeVar('F', bound=Callable[..., Any])

class copy_signature(Generic[F]):
    def __init__(self, target: F) -> None: ...
    def __call__(self, wrapped: Callable[..., Any]) -> F: ...

def f(x: bool, *extra: int) -> str: ...

@copy_signature(f)
def test(*args, **kwargs):
    return f(*args, **kwargs)

reveal_type(test)  # Revealed type is 'def (x: bool, *extra: int) -> str'

I am going to stop replying here for now, the discussion doesn't go anywhere and honestly becomes annoying.

[rmorshea]

My apologies, I was just trying to help push this forward since it had stalled for a long time. I'll also try to be more specific in the future as I think my understanding was aligned with yours (i.e. that there are several issues here that deserve their own independent solutions).

To your last point, I think there's more than just signature copying. As mentioned in a previous comment it would also be useful if one could add fixed arguments to test that are not a part of f:

def f(x: bool, *extra: int) -> str: ...

@copy_signature(f)
def test(fixed_argument: str, *args, **kwargs): ....

# desired
reveal_type(test)  # Revealed type is 'def (fixed_argument: str, x: bool, *extra: int) -> str'
# actual
reveal_type(test)  # Revealed type is 'def (x: bool, *extra: int) -> str'

In terms of aesthetics I can imagine a type annotation like the following that could work:

def f(x: bool, *extra: int) -> str: ...

def test(fixed_argument: str, *args: SameAs[f], **kwargs: SameAs[f]) -> str: ...

# for classes

class Parent:
    def method(self, y: int) -> int: ...

class Child(Parent):
    # implicit
    def method(self, x: int, **kwargs: SameAs[Parent]) -> int: ...
    # explicit
    def method(self, x: int, **kwargs: SameAs[Parent.method]) -> int: ...

I might actually prefer the above, but I don't feel too strongly about it.

[alexmojaki]

@ilevkivskyi I think one of us may be confused about the subject of this issue. To me this has always been about the typing of kwargs (you mentioned TypedDict before and both @JukkaL and I were not in favour of this as a solution as it is verbose and repetitive) and the copying of signatures. See e.g. the title. I've never been interested in checking the bodies of functions. Do other people here want to check function bodies? What is it that would be checked? How feasible is it?

I have never seen anything like your copy_signature construction. When did that become possible? Part of me thinks that maybe we should just put that in typing and call it a day - it solves the most common case and at this rate this issue might never be resolved otherwise. But as @rmorshea says I do think it's important to handle slight changes like additional arguments.

[rmorshea]

@alexmojaki with respect to checking function bodies it would, in principle, be possible to determine what *args and **kwargs are with more specificity:

def parent(x: int, y: int) -> int: ...

def child(z: int, **kwargs: SameAs[parent]) -> int:
    reveal_type(kwargs)  # Revealed type is TypedDict(x=int, y=int)

But again, my use cases for this are primarily concerned with having the correct signature for the wrapping function so I'd have to agree that this isn't a priority. Having more detailed type checking information in the wrapping function's body would be more of a nicety and could be added later since most of the time, kwargs is simply passed from child to parent without modification.

[rmorshea]

@JukkaL and @alexmojaki even though ilevkivskyi seems to have excluded himself from the conversation I think the idea of indicating signature copying via a type hint could be aesthetically acceptable. Such a type hint might be named something like:

• SameAs
• Same
• CopyFrom
• Copy
• Inherit
• InheritFrom

And would probably look a bit like this in practice:

class Parent:
    def method(self, y: int, y: int) -> int: ...

class Child(Parent):
    def method(self, z: int, *args: SameAs[Parent]) -> SameAs: ...
    # or
    def method(self, z: int, **kwargs: SameAs[Parent]) -> int: ...
    # or
    def method(self, z: int, *args: SameAs[Parent], **kwargs: SameAs[Parent]) -> int: ...

Implementation Concern

The ability to copy *args, **kwargs, and return separately might be more complicated to implement:

def parent(x: int, *, y: int = 0) -> int: ...

def child(z: int, *args: SameAs[parent]) -> int: ...  # SameAs should only copy `x` arg

Edge Case Benefits?

This may have some advantages when wrapping functions in slightly more complex ways. For example, this could allow you to delegate arguments to two different functions, or delegate arguments from one or more sources:

def f(*args: int) -> int: ...
def g(**kwargs: int) -> int: ...

def wrapper(*args: SameAs[f], **kwargs: SameAs[g]) -> int:
    return f(*args) + g(**kwargs)

T =  # not sure what to put here

def f(*args: int, **kwargs: int) -> int: ...

def chain_outer(*args: SameAs[f]) -> T:
    def chain_inner(**kwargs: SameAs[f]) -> SameAs[f]:
        return f(*args, **kwargs)
    return chain_inner

[rmorshea]

@msullivan PEP-612 appears related to this issue.

[lucmos]

Sorry if I intrude on the conversation.
I found this thread while I was researching how to solve one minor issue. I think it is common, thus I'd like to share my use case.

Use case

I want to type the function returned by functools.partial.

More precisely, I'd like be able to have some form of parameter hinting from PyCharm. (from what I understood, it is very probable to be implemented if this proposal is accepted)

Sample code

I have a register decorator that I use to register function or class names, in order to be able to retrieve them by name.
I'd like to organize such register in categories.

One possible implementation of a decorator to register objects of the "model category":

register_model = functools.partial( _register_decorator_factory, category="model_class")

However in this way it is not easy to use such decorator, because the signature is lost.

i.e. This is the signature of _register_decorator_factory:

def _register_decorator_factory(
    wrapped=None,
    *,
    names: Optional[Union[List[str], str]] = None,
    category: str,
) -> Any:

I would expect to automatically infer the register_model signature to:

def register_model(
    wrapped=None,
    *,
    names: Optional[Union[List[str], str]] = None,
) -> Any:

tl;dr

Consider the use case of automatically infer the signature of functools.partial's output

[rmorshea]

@lucmos I don't think your use case has been brought up in this thread before, so thanks for sharing! While it seems similar, to my eyes though it looks like you need a general solution for currying so PEP-612 (which will probably close this issue) won't help you out as it stands right now.

Doing a bit of digging though, it doesn't seem like anyone's brought up currying before so perhaps this warrants a new issue. TypeScript appears to have some solution to currying using recursive types, so that seems like a promising angle from which to approach the problem.

[gvanrossum]

Now that PEP 612 is accepted, can we close this issue?

[untitaker]

👍 even if it doesn't cover all usecases (no idea if that's the case, just started reading the spec) this thread has become sufficiently long so remaining usecases should probably get new threads.

Is there a ticket that tracks the implementation in mypy I can subscribe to?

[gvanrossum]

https://github.com/python/mypy/pull/9250

[matthewgdv]

Does PEP 612 offer a solution to the common usecase of copying the signature of the superclass method when overriding?

I've read through it and while it seems like the use-case of signature-preserving decorators is now well-supported, I can't see any way to indicate 'same as parent' or even 'same as x' where x is another callable. Could anyone tell me if I've just missed something glaringly obvious? I found some parts of the PEP a little hard to digest without first seeing more examples of its use out in the wild.

Honestly, I really like the use of attribute-access syntax to separate a ParamSpec out into P.args and P.kwargs.

If there is not yet a way to meet this need with PEP 612 I would like to start a new thread suggesting the following (in the spirit of PEP 612):

def raw(some: str, /, signature: int, goes: bool, *, here: dict) -> str:
    ...


def cooked(*args: raw.args, **kwargs: raw.kwargs) -> raw.ret:
    ret = raw(*args, **kwargs)
    ...  # do something with ret before returning it

Allowing func.args, func.kwargs, and (possibly, and subject to a name-change) func.ret as special type hints that are valid for any function.

And for the specific use-case of method overriding:

from typing import Super


class Parent:
    def some_method(self, some: str, /, signature: int, goes: bool, *, here: dict) -> str:
        ...


class Child(Parent):
    def some_method(self, *args: Super.args, **kwargs: Super.kwargs) -> Super.ret:
        ret = super().some_method(*args, **kwargs)
        ...  # do something with ret before returning it

If typing.Super were a special object that signified to type-checkers and IDEs that the parent's signature for that method should be used.

So yeah. Are these use-cases covered? If not, has there been a successor thread to this where I could post this as a suggestion? And if not, would it be okay if I started one?

Cheers

[rmorshea]

@matthewgdv if all you want to do is copy the signature, this comment describes a way to do that right now: https://github.com/python/typing/issues/270#issuecomment-555966301

That doesn't really help you though if you want to actually add arguments (e.g.(new_arg, *args, **kwargs)). In that case, I don't really see an obvious way to use ParamSpec to accomplish that.

Perhaps in a future revision the following could be made possible?

P1 = ParamSpec("P1")
P2 = ParamSpec("P2")

def concatenate_functions(f1: P1, f2: P2) -> Concatenate[P1, P2]:
    ...

def f(x: int) -> int: ...
def g(y: int) -> int: ...

h: Callable[[int, int], int] = concatenate_functions(f, g)

There might be a separate issue to track this though - a link here would be great if anyone knows of such an issue.