Pyinfra: SHA1 file fact on macOS host (catalina)

[harold-b]

It seems that files are always uploaded even if they exist because the fact relies on sha1sum which is not present on macOS (at least not in Catalina). There is shasum, however.

Would it be possible to support this one macOS targets by using shasum on those targets?

[Fizzadar]

Interesting - it currently looks for either a sha1sum or sha1 command which definitely used to work! I can expand this to include shasum for Catalina.

[Fizzadar]

Implemented @ https://github.com/Fizzadar/pyinfra/commit/2189bb7f5c611d2a37d4ecf2ca452650c518654b, will release in v1 and v0.16.3 later today.

[harold-b]

Thanks very much taking care of this! I will test today with this version.

[harold-b]

I just tested this on a Catalina target. The facts report the file exist, but the files are still being uploaded even if they exist and are not changed:

Operation

files.put(
    name  = 'Upload Large File'
    ,src  = 'files/file.dmg'
    ,dest = f'{workdir}/file.dmg'
    ,mode = '644'
    ,create_remote_dir = False
    ,force = False
)

Target machine before first deploy:

First deploy:

[pyinfra_cli\main] Checking potential directory: 
    [pyinfra_cli\main] Checking potential directory: 
    [pyinfra_cli\main] Checking potential directory: inventories
    [pyinfra_cli\main] Checking potential directory: 
--> Loading config...
--> Loading inventory...
    [pyinfra_cli\inventory] Creating fake inventory...

--> Connecting to hosts...
    [pyinfra\api\connectors\ssh] Connecting to: catalinavm ({'allow_agent': True, 'look_for_keys': True, 'timeout': 10})
    [catalinavm] Connected
    [pyinfra\api\state] Activating host: catalinavm

--> Preparing operations...
    Loading: sandbox.py    
    [pyinfra\api\operation] Adding operation, {'Upload Large File'}, called @ sandbox.py:59, opLines=(0, 59), opHash=3de946074b3ce4c600425a99258959345bbce0b8
    [pyinfra\api\facts] Getting fact: file (ensure_hosts: (catalinavm,))
    [pyinfra\api\connectors\ssh] Running command on catalinavm: (pty=False) sh -c 'ls -ld --time-style=long-iso /Users/harito/.machine-setup/file.dmg 2> /dev/null || ls -ld --full-time /Users/harito/.machine-setup/file.dmg 2> /dev/null || ls -lde /Users/harito/.machine-setup/file.dmg 2> /dev/null || ls -ldT /Users/harito/.machine-setup/file.dmg'
[catalinavm] >>> sh -c 'ls -ld --time-style=long-iso /Users/harito/.machine-setup/file.dmg 2> /dev/null || ls -ld --full-time /Users/harito/.machine-setup/file.dmg 2> /dev/null || ls -lde /Users/harito/.machine-setup/file.dmg 2> /dev/null || ls -ldT /Users/harito/.machine-setup/file.dmg'
[catalinavm] ls: /Users/harito/.machine-setup/file.dmg: No such file or directory
    [pyinfra\api\connectors\ssh] Waiting for exit status...
    [pyinfra\api\connectors\ssh] Command exit status: 1
    Loaded fact file: ('/Users/harito/.machine-setup/file.dmg',)
    [catalinavm] Ready: sandbox.py

--> Proposed changes:
    Groups: mac-minis / mac_minis
    [catalinavm]   Operations: 1   Commands: 2

--> Beginning operation run...
--> Starting operation: Upload Large File ('src=files/file.dmg', 'dest=/Users/harito/.machine-setup/file.dmg', 'mode=644', 'create_remote_dir=False', 'force=False')
    [pyinfra\api\operations] Starting operation Upload Large File on catalinavm
[catalinavm] file uploaded: /Users/harito/.machine-setup/file.dmg
    [pyinfra\api\connectors\ssh] Running command on catalinavm: (pty=None) sh -c 'chmod 644 /Users/harito/.machine-setup/file.dmg'
[catalinavm] >>> sh -c 'chmod 644 /Users/harito/.machine-setup/file.dmg'
    [pyinfra\api\connectors\ssh] Waiting for exit status...
    [pyinfra\api\connectors\ssh] Command exit status: 0
    [catalinavm] Success

--> Results:
    Groups: mac-minis / mac_minis
    [catalinavm]   Successful: 1   Errors: 0   Commands: 2/2

Target machine after first deploy:

Local file sha1:

Second deploy:

   [pyinfra_cli\main] Checking potential directory: 
    [pyinfra_cli\main] Checking potential directory: 
    [pyinfra_cli\main] Checking potential directory: inventories
    [pyinfra_cli\main] Checking potential directory: 
--> Loading config...
--> Loading inventory...
    [pyinfra_cli\inventory] Creating fake inventory...

--> Connecting to hosts...
    [pyinfra\api\connectors\ssh] Connecting to: catalinavm ({'allow_agent': True, 'look_for_keys': True, 'timeout': 10})
    [catalinavm] Connected
    [pyinfra\api\state] Activating host: catalinavm

--> Preparing operations...
    Loading: sandbox.py
    [pyinfra\api\operation] Adding operation, {'Upload Large File'}, called @ sandbox.py:59, opLines=(0, 59), opHash=3de946074b3ce4c600425a99258959345bbce0b8
    [pyinfra\api\facts] Getting fact: file (ensure_hosts: (catalinavm,))
    [pyinfra\api\connectors\ssh] Running command on catalinavm: (pty=False) sh -c 'ls -ld --time-style=long-iso /Users/harito/.machine-setup/file.dmg 2> /dev/null || ls -ld --full-time /Users/harito/.machine-setup/file.dmg 2> /dev/null || ls -lde /Users/harito/.machine-setup/file.dmg 2> /dev/null || ls -ldT /Users/harito/.machine-setup/file.dmg'
[catalinavm] >>> sh -c 'ls -ld --time-style=long-iso /Users/harito/.machine-setup/file.dmg 2> /dev/null || ls -ld --full-time /Users/harito/.machine-setup/file.dmg 2> /dev/null || ls -lde /Users/harito/.machine-setup/file.dmg 2> /dev/null || ls -ldT /Users/harito/.machine-setup/file.dmg'
[catalinavm] -rw-r--r--  1 harito  staff  210653788 Jul 13 12:57 /Users/harito/.machine-setup/file.dmg
    [pyinfra\api\connectors\ssh] Waiting for exit status...
    [pyinfra\api\connectors\ssh] Command exit status: 0
    Loaded fact file: ('/Users/harito/.machine-setup/file.dmg',)
    [catalinavm] Ready: sandbox.py

--> Proposed changes:
    Groups: mac-minis / mac_minis
    [catalinavm]   Operations: 1   Commands: 2

--> Beginning operation run...
--> Starting operation: Upload Large File ('src=files/file.dmg', 'dest=/Users/harito/.machine-setup/file.dmg', 'mode=644', 'create_remote_dir=False', 'force=False')
    [pyinfra\api\operations] Starting operation Upload Large File on catalinavm
[catalinavm] file uploaded: /Users/harito/.machine-setup/file.dmg
    [pyinfra\api\connectors\ssh] Running command on catalinavm: (pty=None) sh -c 'chmod 644 /Users/harito/.machine-setup/file.dmg'
[catalinavm] >>> sh -c 'chmod 644 /Users/harito/.machine-setup/file.dmg'
    [pyinfra\api\connectors\ssh] Waiting for exit status...
    [pyinfra\api\connectors\ssh] Command exit status: 0
    [catalinavm] Success

--> Results:
    Groups: mac-minis / mac_minis
    [catalinavm]   Successful: 1   Errors: 0   Commands: 2/2

File being re-uploaded on subsequent deploy:

I am unaware if the hash reported by opHash is supposed to be the same sha1 from the file, or if any extra data is hashed with it. But if it is supposed to be the sha1 hash from the file, then it is not matching, although the fact is consistent with its opHash ( it is always 3de946074b3ce4c600425a99258959345bbce0b8), where has the result I get from shasum or sha1sum is 9cea0e7068aa32909f18d2a0edfb7335ac51220c. That last bit of info may be irrelevant to you... 🤷

[Fizzadar]

@harold-b it looks like the issue here is with the file fact - it's not returning anything because the sha1 checksum isn't even attempted. I've pushed a fix for parsing that date format (https://github.com/Fizzadar/pyinfra/commit/acae97d863e66874fb7d57f08643f9194f98e4e2) and am about to release 1.0.1!

[harold-b]

Thanks again! I will give it another try with 1.0.1 :)

[harold-b]

@Fizzadar Just tried it. Everything is working perfectly now.
Thank you for the quick fix!