Pyinfra: Load group_data/all.py for @local

[gnat]

Is your feature request related to a problem? Please describe

I use @local to quickly develop tasks before running them on servers. Many of these tasks reference host.data which is normally unavailable on @local. It would make task writing more consistent and convenient to at least load group_data/all.py.

Describe the solution you'd like

Load group_data/all.py when invoking pyinfra on @local

Thanks!

[Fizzadar]

:+1: definitely an expected behaviour so counting this as a bug.

[gnat]

Thanks for the quick response @Fizzadar You rock!

[gnat]

Something to consider: a group_data/local.py loaded after group_data/all.py when using @local .

However to keep @local conventions extra simple, I'm also happy with group_data/all.py doing double duty as the local config by convention. I admire the simplicity of this.

[Fizzadar]

I just checked on this and the group_data/all.py file does appear to be loaded, so long as it's relative to the current directory...

Regarding the local.py, this is already possible but with @local.py!

Eg:

--> Loading config...
--> Loading inventory...
    [pyinfra_cli.inventory] Creating fake inventory...
    [pyinfra_cli.inventory] Looking for group data in: CWD/group_data/all.py
    [pyinfra_cli.inventory] Looking for group data in: CWD/src/group_data/@local.py

[@local]
--> Groups: @local, @local
--> Data:
{
    "is_all": "yes",
    "is_local": true
}

With files:

# group_data/@local.py
is_local = True

# group_data/all.py
is_all = 'yes'
is_local = False

[gnat]

Investigated this further and found the bug-- This only works on @local when the task directory is 1 level deep, otherwise it breaks. This bug only appears when running @local (ssh connector works as intended). Examples:

pyinfra @local tasks/audit.py Works!
pyinfra @local blahblah/test.py Works!
pyinfra @local tasks/all/audit.py Fails to load host.data.
pyinfra @local tasks/staging/test.py Fails to load host.data.

Otherwise /group_data/all.py and /group_data/@local.py both work as intended!

[Fizzadar]

Aha, this is a rubbish limitation of how pyinfra figures out the "deploy directory". I think a simple fix here is to work from the task directory up all the way to the current directory. This would also be the expected handling, I believe.

Things get messy when executing multiple files (currently pyinfra will only consider the first for the deployment directory), or multiple folders containing group_data (an anti pattern I suppose, can't think of any use-cases for that off the top of my head).

[gnat]

or multiple folders containing group_data (an anti pattern I suppose, can't think of any use-cases for that off the top of my head).

Agreed, I cannot think of one either.

[Fizzadar]

I've now reworked the deploy directory detection in https://github.com/Fizzadar/pyinfra/commit/966a6c2cb434a7af4e318a068e2a28e527a231f7, which handles any levels of nesting as described, as well as being clearer about how this works. Released just now in v1.4.1!

[gnat]

Working! Great work @Fizzadar !!