pyinfra.api.exceptions.PyinfraError : Nested operation called with global arguments

[rajgouravjain]

Describe the bug

While using below code and running pyinfra @local deploy.py command, following exception is raised by pyinfra:

pyinfra.api.exceptions.PyinfraError: Nested operation called with global arguments: ['sudo', 'sudo_user', 'use_sudo_login', 'use_sudo_password', 'preserve_sudo_env', 'su_user', 'use_su_login', 'preserve_su_env', 'su_shell', 'name', 'shell_executable', 'chdir', 'env', 'ignore_errors', 'success_exit_codes', 'timeout', 'get_pty', 'stdin', 'precondition', 'postcondition', 'on_success', 'on_error', 'parallel', 'run_once', 'serial'] (line 821 in ../usr/local/lib/python3.6/dist-packages/pyinfra/operations/server.py)

To Reproduce
Steps to reproduce the behavior (include code & usage example):

deploy.py file:

from install_prometheus import install_prometheus

install_prometheus()

install_prometheus.py file:

from pyinfra.api import deploy, DeployError
from pyinfra.operations import files, init, server

 @deploy('Install prometheus', data_defaults=None)
def install_prometheus(state, host):

    server.user(
        name='Create the prometheus user',
        user='rajgouravj',
        shell='/sbin/bash',
        state=state,
        host=host,
    )

Expected behavior
A clear and concise description of what you expected to happen.

The example is derived from [pyinfra-prometheus] (https://github.com/grantstephens/pyinfra-prometheus). It seems global arguments are not supported in Nested operations which has broken server.user and @deploy functions/decorators.

Meta

• Include output of pyinfra --support.

``` ot@f17dfbed55b2:/pyinfra-root# pyinfra --support
--> Support information:

If you are having issues with pyinfra or wish to make feature requests, please
check out the GitHub issues at https://github.com/Fizzadar/pyinfra/issues .
When adding an issue, be sure to include the following:

System: Linux
  Platform: Linux-5.4.39-linuxkit-x86_64-with-Ubuntu-18.04-bionic
  Release: 5.4.39-linuxkit
  Machine: x86_64
pyinfra: v1.3.1
Executable: /usr/local/bin/pyinfra
Python: 3.6.9 (CPython, GCC 8.4.0)

+ How was pyinfra installed (source/pip)? 
using pip

+ Include pyinfra-debug.log (if one was created)
NA

+ Consider including output with `-vv` and `--debug`.

Output:
--------
```ot@f17dfbed55b2:/pyinfra-root# pyinfra @local  deploy.py 
--> Loading config...
--> Loading inventory...

--> Connecting to hosts...
    [@local] Connected

--> Preparing operations...
    Loading: deploy.py
--> An unexpected exception occurred in: deploy.py:

  File "/usr/local/lib/python3.6/dist-packages/pyinfra_cli/util.py", line 52, in exec_file
    exec(PYTHON_CODES[filename], data)
  File "deploy.py", line 3, in <module>
    install_prometheus()
  File "/usr/local/lib/python3.6/dist-packages/pyinfra/api/deploy.py", line 118, in decorated_func
    func(*args, **kwargs)
  File "/pyinfra-root/install_prometheus.py", line 12, in install_prometheus
    host=host,
  File "/usr/local/lib/python3.6/dist-packages/pyinfra/api/operation.py", line 347, in decorated_func
    commands = unroll_generators(func(*actual_args, **actual_kwargs))
  File "/usr/local/lib/python3.6/dist-packages/pyinfra/api/util.py", line 180, in unroll_generators
    for item in generator:
  File "/usr/local/lib/python3.6/dist-packages/pyinfra/operations/server.py", line 821, in user
    state=state, host=host,
  File "/usr/local/lib/python3.6/dist-packages/pyinfra/api/operation.py", line 189, in decorated_func
    ).format(global_kwarg_keys, get_call_location()))
pyinfra.api.exceptions.PyinfraError: Nested operation called with global arguments: ['sudo', 'sudo_user', 'use_sudo_login', 'use_sudo_password', 'preserve_sudo_env', 'su_user', 'use_su_login', 'preserve_su_env', 'su_shell', 'name', 'shell_executable', 'chdir', 'env', 'ignore_errors', 'success_exit_codes', 'timeout', 'get_pty', 'stdin', 'precondition', 'postcondition', 'on_success', 'on_error', 'parallel', 'run_once', 'serial'] (line 821 in ../usr/local/lib/python3.6/dist-packages/pyinfra/operations/server.py)

[relaxdiego]

Interesting. I've used global args like sudo in nested operations without problems. Could it be that that "shell" arg (which appears to have been superseded by "shell_executable" based in the docs) is causing the problem?

[rajgouravjain]

performed a few tests and followings are results::

1. tried running above code after removing "shell='/sbin/bash'" option. But it failed with same error.

2. following code didn't work, as its again calling files.put operation in file server.py:338 (pyinfra.operations)

from pyinfra.api import deploy, DeployError
from pyinfra.operations import pip, files, init, server

@deploy('Install prometheus', data_defaults=None)
def install_prometheus(state, host):

     server.hostname(
     name= "Add hostname",
     hostname='server1.example.com',
     state=state,
     host=host,
     )

However,

1. Below code works very well :

from pyinfra.api import deploy, DeployError
from pyinfra.operations import pip, files, init, server

@deploy('Install prometheus', data_defaults=None)
def install_prometheus(state, host):

    pip.packages(
     packages=[ 'boto3' ],
     state=state,
     host=host,
    )
    files.directory(
    name='Ensure the /tmp/dir_that_we_want_removed is removed',
    path='/tmp/dir_that_we_want_removed',
    present=False,
    state=state,
    host=host,
    )

On going through code of pyinfra/api/operation.py (line 180 - 187), its clear that in_op state if global_kwarg_keys are present it will lead to nested global operations exception.

       # Configure operation
        #

        # Get the meta kwargs (globals that apply to all hosts)
        global_kwargs, global_kwarg_keys = pop_global_op_kwargs(state, kwargs)

        # If this op is being called inside another, just return here
        # (any unwanted/op-related kwargs removed above).
        if state.in_op:
            if global_kwarg_keys:
                raise PyinfraError((
                    'Nested operation called with global arguments: {0} ({1})'
                ).format(global_kwarg_keys, get_call_location()))
            return func(*args, **kwargs) or []

[Fizzadar]

This is indeed a bug in the handling here - operations that call other operations will produce this error when called within @deploy functions. I've fixed this in https://github.com/Fizzadar/pyinfra/commit/54e098ef9a6db9f2097862075fea8cbbd6711ead pending release of 1.3.2 tomorrow!

[Fizzadar]

Now released in v1.3.2!

[rajgouravjain]

awesome! Thank you for quick fix!