Invoke: Using python3 keyword-only arguments or annotated arguments causes a ValueError from inspect

[KuangEleven]

Defining a task that uses the new Python3 keyword-only or annotation argument syntax and invoking causes a ValueError: Function has keyword-only arguments or annotations, use getfullargspec() API which can support them and halts.

Per the docstring for Python3 inspect, getfullargspec() could be used, but that function is itself deprecated in favor of inspect.signature() Unfortunately, neither of those functions exists in Python 2.6, so some level of cascading will be necessary.

For example, neither of these can be invoked:

from invoke import task

@task
def func(ctx, *args, keyonlyarg=42):
    print(keyonlyarg)

@task
def func2(ctx, x:int=3):
    print(x)

[FranklinChen]

Any word on this? I use Python 3 with mypy for types and just realized that I'm blocked on using invoke right now.

[bitprophet]

Thanks for the bump, I'll take a look at #373 soon which seems to be the main patch around this.

@FranklinChen If you can check out that PR's branch and confirm it works against latest master & solves your particular flavor of the issue, that'd be a big help!

[FranklinChen]

@bitprophet Yes, I am actually using the fork at #373 and it works for me.

[nathanielford]

Hi! I'm still seeing this with version 1.1.1.

nford@Nathaniels-MacBook-Pro:~/repos/deploy-apps $ invoke --list -v
Traceback (most recent call last):
  File "/usr/local/bin/invoke", line 11, in <module>
    sys.exit(program.run())
  File "/usr/local/lib/python3.6/site-packages/invoke/program.py", line 324, in run
    self.parse_collection()
  File "/usr/local/lib/python3.6/site-packages/invoke/program.py", line 408, in parse_collection
    self.load_collection()
  File "/usr/local/lib/python3.6/site-packages/invoke/program.py", line 599, in load_collection
    module, parent = loader.load(coll_name)
  File "/usr/local/lib/python3.6/site-packages/invoke/loader.py", line 76, in load
    module = imp.load_module(name, fd, path, desc)
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/imp.py", line 245, in load_module
    return load_package(name, filename)
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/imp.py", line 217, in load_package
    return _load(spec)
  File "<frozen importlib._bootstrap>", line 684, in _load
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/Users/nmford/repos/deploy-apps/tasks/__init__.py", line 2, in <module>
    from tasks import install
  File "/Users/nmford/repos/deploy-apps/tasks/install.py", line 26, in <module>
    def domino(context, app: str):
  File "/usr/local/lib/python3.6/site-packages/invoke/tasks.py", line 313, in task
    return klass(args[0], **kwargs)
  File "/usr/local/lib/python3.6/site-packages/invoke/tasks.py", line 77, in __init__
    self.positional = self.fill_implicit_positionals(positional)
  File "/usr/local/lib/python3.6/site-packages/invoke/tasks.py", line 168, in fill_implicit_positionals
    args, spec_dict = self.argspec(self.body)
  File "/usr/local/lib/python3.6/site-packages/invoke/tasks.py", line 154, in argspec
    spec = inspect.getargspec(func)
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/inspect.py", line 1075, in getargspec
    raise ValueError("Function has keyword-only parameters or annotations"
ValueError: Function has keyword-only parameters or annotations, use getfullargspec() API which can support them

I am not sure what might need to be done to diagnose or resolve it from my end. Is anyone else still running into this?

[SantjagoCorkez]

@nathanielford Just run into this too. As it says in the exception itself, replaced inspect.getargspec at invoke/tasks.py with inspect.getfullargspec manually. Works for me.

[nathanielford]

@SantjagoCorkez I may not be understanding you correctly, but I don't think it'll work for us to manually fork the invoke library code. If I get a chance, though, I'll see if I can put a PR together to submit here, unless the maintainers already have one ready? I'm a little unclear on where they believe the state of the bug to be, given the merged PR.

[SantjagoCorkez]

@nathanielford Well, the bug is more than 2 years old at the moment. Developers still didn't apply a patch that is proposed by the exception message itself. So, they don't care. You're on your own.

[JesseTG]

@bitprophet Hello? How's it going?

[toejough]

@bitprophet - submitted a patch ^

Seems to be failing on 2.7 and pypy - I can work on adding some logic to fix that tomorrow.

[toejough]

passing now.

[eruvanos]

Hi,
still facing this issue, is there any update?

[shatil]

@italomaia and I encountered this using Fabric 2 with Python 3 (3.7.3 and 3.7.4 on Linux and macOS). Type hints in a Fabric task triggers ValueError complaining about getargspec: https://github.com/fabric/fabric/issues/1997

Several seemingly-easy solutions presented themselves over the years (since 2016?!) in PRs mentioning this issue. To recap:

1. Simplest is to use inspect.getfullargspec for Python 3: https://github.com/pyinvoke/invoke/pull/373

   diff --git a/invoke/tasks.py b/invoke/tasks.py
   index ed838c31..2ed82a2e 100644
   --- a/invoke/tasks.py
   +++ b/invoke/tasks.py
   @@ -11,8 +11,10 @@ from .util import six
   
   if six.PY3:
       from itertools import zip_longest
   +    getargspec = inspect.getfullargspec
   else:
       from itertools import izip_longest as zip_longest
   +    getargspec = inspect.getargspec
   
   from .context import Context
   from .parser import Argument, translate_underscores
   @@ -151,7 +153,7 @@ class Task(object):
           # TODO: __call__ exhibits the 'self' arg; do we manually nix 1st result
           # in argspec, or is there a way to get the "really callable" spec?
           func = body if isinstance(body, types.FunctionType) else body.__call__
   -       spec = inspect.getargspec(func)
   +       spec = getargspec(func)
           arg_names = spec.args[:]
           matched_args = [reversed(x) for x in [spec.args, spec.defaults or []]]
           spec_dict = dict(zip_longest(*matched_args, fillvalue=NO_DEFAULT))
   

2. Use invoke's vendored module called decorator, which already did the heavy lifting: https://github.com/pyinvoke/invoke/pull/373#issuecomment-437580030

   diff --git a/invoke/tasks.py b/invoke/tasks.py
   index ed838c31..eaf0b622 100644
   --- a/invoke/tasks.py
   +++ b/invoke/tasks.py
   @@ -4,7 +4,7 @@ generate new tasks.
   """
   
   from copy import deepcopy
   -import inspect
   +from .vendor import decorator
   import types
   
   from .util import six
   @@ -151,7 +151,7 @@ class Task(object):
           # TODO: __call__ exhibits the 'self' arg; do we manually nix 1st result
           # in argspec, or is there a way to get the "really callable" spec?
           func = body if isinstance(body, types.FunctionType) else body.__call__
   -       spec = inspect.getargspec(func)
   +       spec = decorator.getargspec(func)
           arg_names = spec.args[:]
           matched_args = [reversed(x) for x in [spec.args, spec.defaults or []]]
           spec_dict = dict(zip_longest(*matched_args, fillvalue=NO_DEFAULT))
   

3. Or, since clearly decorator has given this more thought, which was the original holdup for https://github.com/pyinvoke/invoke/pull/373, just copy that block into invoke/task.py.

[hypercodex]

Any action on this? Seems like using the inspect is now the way to go. Plz advise if this is something that can get resolved.

[exhuma]

I ran into this as well, and am currently using the following workaround:

from invoke import Context, task

def _mytask(ctx: Context, arg: str = 'foo') -> None:
    ctx.run('ls %s' % arg)

@task
def mytask(ctx, arg):
    _mytask(ctx, arg)

This gives me both invoke tasks and type-safety. But it is really cumbersome as the args need to be touched in 3 areas if they change. Can't wait to get this fix released 😉 🤞

Once this is released I can drop the delegator and just move the @task decorator to the real function.

[lsorber]

We're also using @exhuma's workaround, and would be grateful if this could get fixed!

[delucca]

Any updates on this?

[dperetti]

Come on! https://github.com/strongholdprojects/invoke/pull/1/files

[zelo]

Since this issue is ignored i rolled out little monkeypatch. What it does is

• use getfullargspec which won't raise exception
• translate it to same format that getargspec returns
• replace inspect.getargspec with patched argspec for duration of Task.getargspec call
• call original Task.getargspec

It needs only stdlib + invoke to work

# monkeypatch.py
from unittest.mock import patch
from inspect import getfullargspec, ArgSpec

import invoke


def fix_annotations():
    """
        Pyinvoke doesnt accept annotations by default, this fix that
        Based on: https://github.com/pyinvoke/invoke/pull/606
    """
    def patched_inspect_getargspec(func):
        spec = getfullargspec(func)
        return ArgSpec(*spec[0:4])

    org_task_argspec = invoke.tasks.Task.argspec

    def patched_task_argspec(*args, **kwargs):
        with patch(target="inspect.getargspec", new=patched_inspect_getargspec):
            return org_task_argspec(*args, **kwargs)

    invoke.tasks.Task.argspec = patched_task_argspec

at the top of your tasks.py do

from . import monkeypatch

monkeypatch.fix_annotations()

[ABitMoreDepth]

Can confirm @zelo's workaround works great. Look forward to a similar fix being landed... Sometime?

[0az]

This might also work:

F = TypeVar('F', bound=Callable)

def task(f: F) -> F:
    f.__annotations__ = {}
    return invoke.task(f)

Haven't tested it against anything but my current environment, so I don't know if this will break MyPy or other type checkers, etc.

[cahna]

This might also work:

F = TypeVar('F', bound=Callable)

def task(f: F) -> F:
    f.__annotations__ = {}
    return invoke.task(f)

Haven't tested it against anything but my current environment, so I don't know if this will break MyPy or other type checkers, etc.

Thanks, @0az! Worked for me. Only 1 change to make MyPy happy:

F = TypeVar("F", bound=Callable[..., Any])

[eruvanos]

We used invoke a lot, but now moving to https://typer.tiangolo.com/.
I can really recommend it, it is kind of powerful and simple to use, comes with a lot of example documentation and elevates type hints!

Looks like this project is not maintained anymore, or does not care about Python 3 features :/

[lsorber]

@eruvanos We experimented with Typer as well, but missed the ability to run simple commands like invoke docs. With Typer, the equivalent would presumably look something like python -m tasks docs, which is much more verbose. Did you find a solution for that?

[D3f0]

Typer is awesome, it leverages the type system and reduces the decorators you'd use with click. Although I agree with @lsorber about the simplicity Inovke provides, specially if your tasks can only depend on Python 3. In my experience, having something close to Inovke in Click/Typer would require to use https://github.com/amoffat/sh, and for some tasks can be tricky, for example, calling pytest with the flags you're using in CI.
For context, I found myself writing too either bash scripts or Makefile, or a combination of both for tasks like testing, building docs, building wheels, etc. Makefile is everywhere, but I'm OK with the trade off of having to install invoke in other developer machine.

[hangingman]

If I declare the variable 'c' as Context, I wish set the type. But, this problem prevent me to type-hinting...
I can't understand why this is not fixed.

[dralshehri]

I had to add Type comments as a workaround for code completion to work in PyCharm:

from invoke import context, task


@task
def hello(ctx, name):
    c = ctx  # type: context.Context
    c.run(f"echo 'Hello {name}'")

[lynncyrin]

Heya, it looks like this still isn't fixed? Any idea what the blockers are? It's been years