Pyinfra: Sync of directory with files with spaces in filename fails

Created on 15 Jun 2020 · 4Comments · Source: Fizzadar/pyinfra

Describe the bug

Sync of directory with files with spaces in filename fails.

To Reproduce
Put file called some file.txt into the Šablony directory. Call

files.sync("configs/Šablony", "~/Šablony")

Expected behavior
File with spaces in filename is copied to the target directory.

Meta

Include output of pyinfra --support.

--> Support information:

    If you are having issues with pyinfra or wish to make feature requests, please
    check out the GitHub issues at https://github.com/Fizzadar/pyinfra/issues .
    When adding an issue, be sure to include the following:

    System: Linux
      Platform: Linux-4.15.0-20-generic-x86_64-with-LinuxMint-19.1-tessa
      Release: 4.15.0-20-generic
      Machine: x86_64
    pyinfra: v0.14.5
    Executable: /home/bystrousak/.local/bin/pyinfra
    Python: 2.7.17 (CPython, GCC 7.5.0)

How was pyinfra installed (source/pip)?

pip install --user -U . from cloned repository on latest master.

Include pyinfra-debug.log (if one was created)

It wasn't.

Consider including output with -vv and --debug.

/home/bystrousak/.local/bin/pyinfra -vv --debug inventory.py configure_local_pc.py
    [pyinfra_cli.main] Checking potential directory: 
    [pyinfra_cli.main] Checking potential directory: 
    [pyinfra_cli.main] Checking potential directory: 
    [pyinfra_cli.main] Checking potential directory: 
--> Loading config...
--> Loading inventory...
    [pyinfra_cli.inventory] Creating fake inventory...

--> Connecting to hosts...
    [@local] Connected
    [pyinfra.api.state] Activating host: @local

--> Preparing operations...
    Loading: configure_local_pc.py
    Use of `pyinfra.modules` is deprecated, please use `pyinfra.operations`.
    [pyinfra.api.operation] Adding operation, set([u'Files/Sync']), called @ configure_local_pc.py:24, opLines=(0, 24), opHash=deaf136562af1c35af570a63cea148d4aa3eb15a
    [pyinfra.api.facts] Getting fact: directory (ensure_hosts: (@local,))
    [pyinfra.api.connectors.util] Building command (shell_executable: sh): ls -ld --time-style=long-iso ~/Šablony 2> /dev/null || ls -ldT ~/Šablony
[pyinfra.api.connectors.local] --> Running command on localhost: sh -c 'ls -ld --time-style=long-iso ~/Šablony 2> /dev/null || ls -ldT ~/Šablony'
[@local] >>> sh -c 'ls -ld --time-style=long-iso ~/Šablony 2> /dev/null || ls -ldT ~/Šablony'
[@local] drwxrwxr-x 2 bystrousak bystrousak 4096 2020-06-15 00:38 /home/bystrousak/Šablony
[pyinfra.api.connectors.util] --> Waiting for exit status...
[pyinfra.api.connectors.util] --> Command exit status: 0
    Loaded fact directory: (u'~/\u0160ablony',)
    [pyinfra.api.facts] Getting fact: file (ensure_hosts: (@local,))
    [pyinfra.api.connectors.util] Building command (shell_executable: sh): ls -ld --time-style=long-iso ~/Šablony/other file.txt 2> /dev/null || ls -ldT ~/Šablony/other file.txt
[pyinfra.api.connectors.local] --> Running command on localhost: sh -c 'ls -ld --time-style=long-iso ~/Šablony/other file.txt 2> /dev/null || ls -ldT ~/Šablony/other file.txt'
[@local] >>> sh -c 'ls -ld --time-style=long-iso ~/Šablony/other file.txt 2> /dev/null || ls -ldT ~/Šablony/other file.txt'
[@local] ls: chybná šířka tabelátoru: „/home/bystrousak/Šablony/other“
[pyinfra.api.connectors.util] --> Waiting for exit status...
[pyinfra.api.connectors.util] --> Command exit status: 2
    Loaded fact file: (u'~/\u0160ablony/other file.txt',)
    [@local] Ready: configure_local_pc.py

--> Proposed changes:
    Groups: my_hosts / inventory / @local
    [@local]   Operations: 1   Commands: 1   

--> Beginning operation run...
--> Starting operation: Files/Sync (u'configs/\u0160ablony', u'~/\u0160ablony')
    [pyinfra.api.operations] Starting operation Files/Sync on @local
    [pyinfra.api.connectors.util] Building command (shell_executable: sh): cp /tmp/tmpQJeuWX ~/Šablony/other file.txt
[pyinfra.api.connectors.local] --> Running command on localhost: sh -c 'cp /tmp/tmpQJeuWX ~/Šablony/other file.txt'
[@local] >>> sh -c 'cp /tmp/tmpQJeuWX ~/Šablony/other file.txt'
[@local] cp: cíl 'file.txt' není adresářem
[pyinfra.api.connectors.util] --> Waiting for exit status...
[pyinfra.api.connectors.util] --> Command exit status: 1
    [@local] Command IO error: IOError(u"cp: c\xedl 'file.txt' nen\xed adres\xe1\u0159em",)
    [@local] Error
    [pyinfra.api.state] Failing hosts: @local
--> pyinfra error: No hosts remaining!
Makefile:5: recipe for target 'build' failed
make: *** [build] Error 1

Bug

Source

Bystroushaak

All 4 comments

Issue can be probably solved by putting quotes around arguments called in connector and files api.

But, it can't just be quoted, as the paths like ~/something stops working, because ~ is not expanded in quotes. It is possible to do replacement of the filename with .replace('~', '$HOME'), which I can confirm works.

Suggestion: Maybe it would be better to use sh library I linked in the other issue, as it solves this kind of problems (and much more) automagically.

Bystroushaak on 15 Jun 2020

👍1

Nice catch @Bystroushaak, I'm amazed this hasn't come up before over the last few years! I'm looking at supporting this by proper escaping (ie path\ with\ space).

Avoiding quotes in this situation as this could cause unexpected side effects (possibly people already using environment variables/similar in path). Overall this has brought into question the string-based commands - in hindsight a list of arguments which can later be quoted and merged is better (although shlex.join appears in Python3.8 only).

Currently the fact or operation is responsible for correctly quoting or escaping such paths - very keen to investigate the possibility of handling this elsewhere in future; doing so will require major changes to the operation output.