Pyinfra: nested routines causes re-ordering of operations

[morrison12]

Describe the bug
I'm not sure if this should be a bug or a feature request.

When there are multiple levels of python functions before an operation is invoked, the ordering line-number based mechanism appears re-orders the operations compared to one what might expect from reading the code. This is similar to the phenomenon with loops noted in the documentation.

One possible solution might be to number the operations as they are executed in the first pass and then use the occurrence number to order the operations prior to execution. One complication of such an approach would be that the occurrence sequence could be host-specific.

To Reproduce

"""
    a.py top-level
"""

from b import please_do_something

please_do_something("uno")

"""
    b.py - mid-level
"""
from c import do_something

def please_do_something(one: str):
    do_something(one, "alpha")
    do_something(one, "bravo")

"""
    c.py - where the action happens
"""
from pyinfra.operations import server

def do_something(one, two: str):

    server.shell(f"echo A {one} {two}", name=f"do A thing with {one} and {two}")
    server.shell(f"echo B {one} {two}", name=f"do B thing with {one} and {two}")

and execute with pyinfra @local a.py to find the following:

pyinfra @local a.py
--> Loading config...
--> Loading inventory...

--> Connecting to hosts...
    [@local] Connected

--> Preparing operations...
    Loading: a.py
    [@local] Ready: a.py

--> Proposed changes:
    Groups: @local
    [@local]   Operations: 4   Commands: 4   

--> Beginning operation run...
--> Starting operation: do A thing with uno and alpha 
    [@local] Success

--> Starting operation: do A thing with uno and bravo 
    [@local] Success

--> Starting operation: do B thing with uno and alpha 
    [@local] Success

--> Starting operation: do B thing with uno and bravo 
    [@local] Success

--> Results:
    Groups: @local
    [@local]   Successful: 4   Errors: 0   Commands: 4/4

Expected behavior
The expected behaviour is:

--> Beginning operation run...
--> Starting operation: do A thing with uno and alpha 
    [@local] Success

--> Starting operation: do B thing with uno and alpha 
    [@local] Success

--> Starting operation: do A thing with uno and bravo 
    [@local] Success

--> Starting operation: do B thing with uno and bravo 
    [@local] Success

--> Results:
    Groups: @local
    [@local]   Successful: 4   Errors: 0   Commands: 4/4

Meta
System: Darwin
Platform: macOS-10.14.6-x86_64-i386-64bit
Release: 18.7.0
Machine: x86_64
pyinfra: v1.0.4
Python: 3.8.5 (CPython, Clang 11.0.0 (clang-1100.0.33.17))

[Fizzadar]

Hi @morrison12! This is a limitation of the CLI mode deploy - using functions/imports such as this simply isn't compatible. There's a few methods that can be used to break up deploy code:

1. local.include - define all operations at the top level (does not support arguments)

# my_deploy.py
from pyinfra import local
local.include('tasks/some_other_operations.py')

# tasks/some_other_operations.py
from pyinfra.operations import server
server.shell(...)

1. Using the @deploy function wrapper, this should satisfy your exact use-case with arguments:

# my_deploy.py
from my_module import do_something
do_something('a', 'b')

# my_module.py
from pyinfra.api import deploy

@deploy
def do_something(state, host, a, b):
    server.shell(
        f'echo {a} {b}',
        state=state,
        host=host,
    )

1. Using the pyinfra API directly (not guaranteed stable yet!) - see the example.

---

I'm going to leave this open to improve the documentation around this on the writing deploys doc page.

[morrison12]

Thanks @Fizzadar. I had found and am using local.include (just left that layer off in my example) but I had somehow assumed that deploys could not have parameters (a bit odd in retrospect).

[Fizzadar]

Now updated (https://github.com/Fizzadar/pyinfra/commit/c155696de2632aee02168578c555ed9ec09b15c5)!