Pyinfra: files.directory() not working as expected with multiple directories

Created on 14 Jan 2020 · 2Comments · Source: Fizzadar/pyinfra

I think the files.directory() expects a single value, not a collection.

I tried to use a collection and got unexpected results. Perhaps multiple values should be expected, or it should complain loudly or ???

Using this:

from pyinfra.modules import files, server

SUDO = True

files.directory(
    {'Ensure the directories exist'},
    ['/netboot/tftp', '/netboot/nfs'],
)

server.shell(
    {'List dirs'},
    'ls -al /netboot',
)

It creates the /netboot dir and it creates a file called 'nfs]'.

$ pyinfra -vv --debug @docker/ubuntu mike.py
    [pyinfra_cli.main] Checking potential directory:
    [pyinfra_cli.main] Setting directory to:
--> Loading config...
--> Loading inventory...
    [pyinfra_cli.inventory] Creating fake inventory...
    [pyinfra.api.inventory] Expanding inventory connector: docker
    The @docker connector is in Alpha!
    [pyinfra_cli.inventory] Looking for group data in: group_data/all.py
    [pyinfra_cli.inventory] Looking for group data in: group_data/bsd.py
    [pyinfra.api.inventory] Expanding inventory connector: docker

--> Connecting to hosts...
localhost: >>> docker run -d ubuntu sleep 10000
localhost: 8edb7f0ddc842bdca0d4f95e3f71bb979e47dbd3698705cc7099ec455bd8faf8
    [pyinfra.api.state] Activating host: ubuntu

--> Preparing operations...
    Loading: mike.py
    [pyinfra.api.operation] Adding operation, {'Ensure the directories exist'}, called @ mike.py:7, opLines=(0, 7), ofrom pyinfra.modules import files, server
pHash=97d8032458c50f8c46e458a4ca15c67a69be92b3
    [pyinfra.api.facts] Getting fact: directory (ensure_hosts: (ubuntu,))
    [pyinfra.api.util] Building command (shell_executable: sh): docker exec 8edb7f0ddc842bdca0d4f95e3f71bb979e47dbd3698705cc7099ec455bd8faf8 sh -c 'ls -ld --time-style=long-iso ['"'"'/netboot/tftp'"'"', '"'"'/netboot/nfs'"'"'] 2> /dev/null || ls -ldT ['"'"'/netboot/tftp'"'"', '"'"'/netboot/nfs'"'"']'
[pyinfra.api.connectors.local] --> Running command on localhost: sh -c 'docker exec 8edb7f0ddc842bdca0d4f95e3f71bb979e47dbd3698705cc7099ec455bd8faf8 sh -c '"'"'ls -ld --time-style=long-iso ['"'"'"'"'"'"'"'"'/netboot/tftp'"'"'"'"'"'"'"'"', '"'"'"'"'"'"'"'"'/netboot/nfs'"'"'"'"'"'"'"'"'] 2> /dev/null || ls -ldT ['"'"'"'"'"'"'"'"'/netboot/tftp'"'"'"'"'"'"'"'"', '"'"'"'"'"'"'"'"'/netboot/nfs'"'"'"'"'"'"'"'"']'"'"''
[ubuntu] >>> sh -c 'docker exec 8edb7f0ddc842bdca0d4f95e3f71bb979e47dbd3698705cc7099ec455bd8faf8 sh -c '"'"'ls -ld --time-style=long-iso ['"'"'"'"'"'"'"'"'/netboot/tftp'"'"'"'"'"'"'"'"', '"'"'"'"'"'"'"'"'/netboot/nfs'"'"'"'"'"'"'"'"'] 2> /dev/null || ls -ldT ['"'"'"'"'"'"'"'"'/netboot/tftp'"'"'"'"'"'"'"'"', '"'"'"'"'"'"'"'"'/netboot/nfs'"'"'"'"'"'"'"'"']'"'"''
[ubuntu] ls: invalid tab size: '[/netboot/tftp,'
[pyinfra.api.connectors.local] --> Waiting for exit status...
[pyinfra.api.connectors.local] --> Command exit status: 2
    Loaded fact directory: (['/netboot/tftp', '/netboot/nfs'],)
    [ubuntu]  Ready: mike.py

--> Proposed changes:
    Groups: @docker
    [ubuntu]   Operations: 1   Commands: 1

--> Beginning operation run...
--> Starting operation: Ensure the directories exist
    [pyinfra.api.operations] Starting operation Ensure the directories exist on ubuntu
    [pyinfra.api.util] Building command (shell_executable: sh): docker exec 8edb7f0ddc842bdca0d4f95e3f71bb979e47dbd3698705cc7099ec455bd8faf8 sh -c 'mkdir -p ['"'"'/netboot/tftp'"'"', '"'"'/netboot/nfs'"'"']'
[pyinfra.api.connectors.local] --> Running command on localhost: sh -c 'docker exec 8edb7f0ddc842bdca0d4f95e3f71bb979e47dbd3698705cc7099ec455bd8faf8 sh -c '"'"'mkdir -p ['"'"'"'"'"'"'"'"'/netboot/tftp'"'"'"'"'"'"'"'"', '"'"'"'"'"'"'"'"'/netboot/nfs'"'"'"'"'"'"'"'"']'"'"''
[ubuntu] >>> sh -c 'docker exec 8edb7f0ddc842bdca0d4f95e3f71bb979e47dbd3698705cc7099ec455bd8faf8 sh -c '"'"'mkdir -p ['"'"'"'"'"'"'"'"'/netboot/tftp'"'"'"'"'"'"'"'"', '"'"'"'"'"'"'"'"'/netboot/nfs'"'"'"'"'"'"'"'"']'"'"''
[pyinfra.api.connectors.local] --> Waiting for exit status...
[pyinfra.api.connectors.local] --> Command exit status: 0
    [ubuntu] Success

--> Results:
    Groups: @docker
    [ubuntu]   Successful: 1   Errors: 0   Commands: 1/1
localhost: >>> docker commit 8edb7f0ddc84
localhost: sha256:481f7a2ff4365aff285faef5295ca08cd1da47d35d5273595bc0814e19459bc5
localhost: >>> docker rm -f 8edb7f0ddc84
localhost: 8edb7f0ddc84
    [ubuntu] docker build complete, image ID: 481f7a2ff436
(venv) mikekinneymbp:examples mikekinney$ vi mike.py
(venv) mikekinneymbp:examples mikekinney$ pyinfra -vv --debug @docker/ubuntu mike.py
    [pyinfra_cli.main] Checking potential directory:
    [pyinfra_cli.main] Setting directory to:
--> Loading config...
--> Loading inventory...
    [pyinfra_cli.inventory] Creating fake inventory...
    [pyinfra.api.inventory] Expanding inventory connector: docker
    The @docker connector is in Alpha!
    [pyinfra_cli.inventory] Looking for group data in: group_data/all.py
    [pyinfra_cli.inventory] Looking for group data in: group_data/bsd.py
    [pyinfra.api.inventory] Expanding inventory connector: docker

--> Connecting to hosts...
localhost: >>> docker run -d ubuntu sleep 10000
localhost: 6c15aa2abf3e36a41df96b4ce79cc3bdc80a3f7705fbd41e08e1601e61ad09c2
    [pyinfra.api.state] Activating host: ubuntu

--> Preparing operations...
    Loading: mike.py
    [pyinfra.api.operation] Adding operation, {'Ensure the directories exist'}, called @ mike.py:7, opLines=(0, 7), opHash=97d8032458c50f8c46e458a4ca15c67a69be92b3
    [pyinfra.api.facts] Getting fact: directory (ensure_hosts: (ubuntu,))
    [pyinfra.api.util] Building command (shell_executable: sh): docker exec 6c15aa2abf3e36a41df96b4ce79cc3bdc80a3f7705fbd41e08e1601e61ad09c2 sh -c 'ls -ld --time-style=long-iso ['"'"'/netboot/tftp'"'"', '"'"'/netboot/nfs'"'"'] 2> /dev/null || ls -ldT ['"'"'/netboot/tftp'"'"', '"'"'/netboot/nfs'"'"']'
[pyinfra.api.connectors.local] --> Running command on localhost: sh -c 'docker exec 6c15aa2abf3e36a41df96b4ce79cc3bdc80a3f7705fbd41e08e1601e61ad09c2 sh -c '"'"'ls -ld --time-style=long-iso ['"'"'"'"'"'"'"'"'/netboot/tftp'"'"'"'"'"'"'"'"', '"'"'"'"'"'"'"'"'/netboot/nfs'"'"'"'"'"'"'"'"'] 2> /dev/null || ls -ldT ['"'"'"'"'"'"'"'"'/netboot/tftp'"'"'"'"'"'"'"'"', '"'"'"'"'"'"'"'"'/netboot/nfs'"'"'"'"'"'"'"'"']'"'"''
[ubuntu] >>> sh -c 'docker exec 6c15aa2abf3e36a41df96b4ce79cc3bdc80a3f7705fbd41e08e1601e61ad09c2 sh -c '"'"'ls -ld --time-style=long-iso ['"'"'"'"'"'"'"'"'/netboot/tftp'"'"'"'"'"'"'"'"', '"'"'"'"'"'"'"'"'/netboot/nfs'"'"'"'"'"'"'"'"'] 2> /dev/null || ls -ldT ['"'"'"'"'"'"'"'"'/netboot/tftp'"'"'"'"'"'"'"'"', '"'"'"'"'"'"'"'"'/netboot/nfs'"'"'"'"'"'"'"'"']'"'"''
[ubuntu] ls: invalid tab size: '[/netboot/tftp,'
[pyinfra.api.connectors.local] --> Waiting for exit status...
[pyinfra.api.connectors.local] --> Command exit status: 2
    Loaded fact directory: (['/netboot/tftp', '/netboot/nfs'],)
    [pyinfra.api.operation] Adding operation, {'List dirs'}, called @ mike.py:12, opLines=(0, 12), opHash=4f11bec2ecafa9eb1c81b828658c2a03a093ff70
    [ubuntu]  Ready: mike.py

--> Proposed changes:
    Groups: @docker
    [ubuntu]   Operations: 2   Commands: 2

--> Beginning operation run...
--> Starting operation: Ensure the directories exist
    [pyinfra.api.operations] Starting operation Ensure the directories exist on ubuntu
    [pyinfra.api.util] Building command (shell_executable: sh): docker exec 6c15aa2abf3e36a41df96b4ce79cc3bdc80a3f7705fbd41e08e1601e61ad09c2 sh -c 'mkdir -p ['"'"'/netboot/tftp'"'"', '"'"'/netboot/nfs'"'"']'
[pyinfra.api.connectors.local] --> Running command on localhost: sh -c 'docker exec 6c15aa2abf3e36a41df96b4ce79cc3bdc80a3f7705fbd41e08e1601e61ad09c2 sh -c '"'"'mkdir -p ['"'"'"'"'"'"'"'"'/netboot/tftp'"'"'"'"'"'"'"'"', '"'"'"'"'"'"'"'"'/netboot/nfs'"'"'"'"'"'"'"'"']'"'"''
[ubuntu] >>> sh -c 'docker exec 6c15aa2abf3e36a41df96b4ce79cc3bdc80a3f7705fbd41e08e1601e61ad09c2 sh -c '"'"'mkdir -p ['"'"'"'"'"'"'"'"'/netboot/tftp'"'"'"'"'"'"'"'"', '"'"'"'"'"'"'"'"'/netboot/nfs'"'"'"'"'"'"'"'"']'"'"''
[pyinfra.api.connectors.local] --> Waiting for exit status...
[pyinfra.api.connectors.local] --> Command exit status: 0
    [ubuntu] Success

--> Starting operation: List dirs
    [pyinfra.api.operations] Starting operation List dirs on ubuntu
    [pyinfra.api.util] Building command (shell_executable: sh): docker exec 6c15aa2abf3e36a41df96b4ce79cc3bdc80a3f7705fbd41e08e1601e61ad09c2 sh -c 'ls -al /netboot'
[pyinfra.api.connectors.local] --> Running command on localhost: sh -c 'docker exec 6c15aa2abf3e36a41df96b4ce79cc3bdc80a3f7705fbd41e08e1601e61ad09c2 sh -c '"'"'ls -al /netboot'"'"''
[ubuntu] >>> sh -c 'docker exec 6c15aa2abf3e36a41df96b4ce79cc3bdc80a3f7705fbd41e08e1601e61ad09c2 sh -c '"'"'ls -al /netboot'"'"''
[ubuntu] total 12
[ubuntu] drwxr-xr-x 3 root root 4096 Jan 14 18:04 .
[ubuntu] drwxr-xr-x 1 root root 4096 Jan 14 18:04 ..
[ubuntu] drwxr-xr-x 2 root root 4096 Jan 14 18:04 nfs]
[pyinfra.api.connectors.local] --> Waiting for exit status...
[pyinfra.api.connectors.local] --> Command exit status: 0
    [ubuntu] Success

--> Results:
    Groups: @docker
    [ubuntu]   Successful: 2   Errors: 0   Commands: 2/2
localhost: >>> docker commit 6c15aa2abf3e
localhost: sha256:82b132a4bdf3c80eead6f07806a0c05a414a708759ad85e4ad8b51c2937ac89a
localhost: >>> docker rm -f 6c15aa2abf3e
localhost: 6c15aa2abf3e
    [ubuntu] docker build complete, image ID: 82b132a4bdf3

Bug

Source

mkinney

All 2 comments

The operation here is expecting a single directory, so you'd want to:

for dirname in ['a', 'b']:
    files.directory(
        {f'Ensure the {dirname} directory exists'},
        dirname,
    )

I'll leave this issue open to raise an exception when an incorrect value is passed into this operation (and files.file/files.template) as this behaviour is bad!