Typing: Partial Type Specifications For Callable

[rmorshea]

The Feature Request

It would be awesome if it were possible to create a partial type spec for Callable. For example, I might want to be able to specify that the first argument of a function must be an integer, but that any other parameters are allowed. This could be accomplished with the following extension to PEP-484 which would allow ... to be included in the argument list to Callable if it followed any extended callable types. As a result we could write a type spec FirstArgIsInt which would match all the following functions and any number of other arbitrary functions so long as their first argument is an integer:

FirstArgIsInt = Callable[[Arg(int), ...], int]

f1: FirstArgIsInt
def f1(x: int) -> int: ...

f2: FirstArgIsInt
def f2(x: int, y: int) -> int: ...

f3: FirstArgIsInt
def f3(x: int, *args, **kwargs) -> int: ...

A Real World Use Case

Partial type specifications for Callable are useful if parameters are being passed on to a function from a decorator as in request handlers for many web frameworks like Django's view funtions or Sanic's routes:

def route_1(request: Request) -> Response: ...
def route_2(request: Request, username: str) -> Response: ...
def route_3(request: Request, comment_id: int, content: str) -> Response: ...
# adinfinitum...

Imagine that you wanted to write an authenticated decorator which ensures a user who accesses any particular route is logged in. What would be the type of RouteHandler if not Callable[..., Response]:

RouteHandler = ?

def authenticated(r: RouteHandler) -> RouteHandler: ...

It would be Callable[[Arg(Request), ...], Any], or some other expression of the same idea.

Current Work Arounds

1. Callable[..., Response], however this provides no precision for the function parameters.
2. Use Callable[[Arg(Request), VarArg(), KwArg()], Response] however this is problematic since, these route handlers don't actually need to be able to accept arbitrary arguments.
3. Use Protocol with overloads, however there are cases (e.g. decorators for route handlers as above) where it's not feasible to enumerate all possible route handler implementations.

[srittau]

You should be able to solve this with callback protocols, which also support overloads for more complicated cases.

[rmorshea]

@srittau I've edited the description to address this. The problem with overloads is that I must know all possible implementations of, for example, a route handler for a web server, or use request: Request, *args: Any, **kwargs, which has the same problem as VarArg and KwArg in requiring the function to accept arbitrary arguments. The former isn't feasible, and the latter isn't correct.

Imagine that I wanted to write an authenticated decorator which ought to work for any route function. As mentioned before, I can use Callable[..., Any] but that gives up on any sort of specificity. In reality, I'd want to be able to write:

RouteHandler = Callable[[Arg(Request), ...], Any]

def authenticated(r: RouteHandler) -> RouteHandler: ...

[bryevdv]

@srittau It's entirely possible that the error is on my end, but here is a complicated example I was not able to get working with a callback protocol and had to fallback to Callable[...., T]

from typing import Any, Callable
from typing_extensions import Protocol

class Config: pass
class System: pass
class ActionReturn: pass

StepType = Callable[[Config, System], ActionReturn]

class VerifyFunctionType(Protocol):

    # not sure why this is necessary by mypy complains missing __name__ without
    __name__: str = "VerifyFunctionType"

    def __call__(self, config: Config, system: System, **kw: Any) -> None: ...

def collect_credential(**kw: str) -> Callable[[VerifyFunctionType], StepType]:
    def decorator(func: VerifyFunctionType) -> StepType:
        def wrapper(config: Config, system: System) -> ActionReturn:
            secrets = dict(foo="bar")
            func(config, system, **secrets)
            return ActionReturn()
        return wrapper
    return decorator


@collect_credential(token="SOME_TOKEN")
def verify_token_credentials(config: Config, system: System, *, token: str) -> None:
    pass

Which reports an error like this:

mypy ex_callback_deco.py
ex_callback_deco.py:27: error: Argument 1 has incompatible type "Callable[[Config, System, NamedArg(str, 'token')], None]"; expected "VerifyFunctionType"
Found 1 error in 1 file (checked 1 source file)

[gvanrossum]

I'm guessing this is because the VerifyFunctionType protocol has a __call__ that can be called with arbitrary keyword arguments (**kw: Any) but verify_token_credentials only has a specific keyword argument. Does adding a dummy **kw: Any to the latter's signature help? (You could assert at runtime that kw is empty.)

[bryevdv]

@gvanrossum No that just changes the error message to include KwArg(Any):

ex_callback_deco.py:27: error: Argument 1 has incompatible type "Callable[[Config, System, NamedArg(str, 'token'), KwArg(Any)], None]"; expected "VerifyFunctionType"

[gvanrossum]

Hm, then I'm guessing it would work if you also removed token: str, but that defeats the purpose.

I wonder if PEP 612 (not yet accepted or implemented) might help.

However this seems a different case from the original post.

[bryevdv]

However this seems a different case from the original post.

Quite possibly and apologies if so! I have been searching for a solution and latched on to

callback protocols, which also support overloads for more complicated cases.

and the OP mentions of decorators as possibly relevant (but I was not 100% sure)

[rmorshea]

It's been a while since I read through PEP-612 so I could be wrong, but I think that PEP might actually resolve this issue.

Taking the examples provided above, and my recollection, it seems like the following would be allowable:

p = ParamSpec("P")
RouteHandler = Callable[Concatenate[Request, P], Any]
def authenitcated(f: RouteHandler) -> RouteHandler: ...

[gvanrossum]

Now that PEP 612 is accepted, please re-review if you still need this issue open.

[rmorshea]

Closing - PEP-612 provides an example of a with_request context manager that leverages the ParamSpec and the Concatenate operator to achieve the effect of a "partial type specification for callables".