Pyinfra: files.line() shows changed when there should not be any changes

[mkinney]

It looks like if you have a line with a " (double quote), it may always show as a change. I've tried escaping the double quote and using the re.escape() function.

The example below was run on docker, but I get similar results when running on a vm.

For instance, this deploy:

from pyinfra.modules import files
import re

SUDO = True

# pyinfra @docker/ubuntu mike.py

line1 = 'QUOTAUSER=\\"\\"'
test1 = files.line(
    {'test1'},
    '/etc/adduser.conf',
    r'^{}$'.format(line1),
    replace=line1,
)
print(test1.changed)

line2 = 'QUOTAUSER=""'
test2 = files.line(
    {'test2'},
    '/etc/adduser.conf',
    re.escape('^{}$'.format(line2)),
    replace=line2,
)
print(test2.changed)

Shows both operations as changed, when there really is no change.

$ pyinfra -vv --debug @docker/ubuntu mike.py
    [pyinfra_cli.main] Checking potential directory:
    [pyinfra_cli.main] Setting directory to:
--> Loading config...
--> Loading inventory...
    [pyinfra_cli.inventory] Creating fake inventory...
    [pyinfra.api.inventory] Expanding inventory connector: docker
    The @docker connector is in Alpha!
    [pyinfra_cli.inventory] Looking for group data in: group_data/all.py
    [pyinfra_cli.inventory] Looking for group data in: group_data/bsd.py
    [pyinfra.api.inventory] Expanding inventory connector: docker

--> Connecting to hosts...
localhost: >>> docker run -d ubuntu sleep 10000
localhost: 414437174a1dc3360b28d41ac244234be2a0eed1653cba70a5638cd1b76d6b76
    [pyinfra.api.state] Activating host: ubuntu

--> Preparing operations...
    Loading: mike.py
    [pyinfra.api.operation] Adding operation, {'test1'}, called @ mike.py:13, opLines=(0, 13), opHash=7d6cc86e00acf8e2dc7e09db45da85a7fc2d8cbf
    [pyinfra.api.facts] Getting fact: find_in_file (ensure_hosts: (ubuntu,))
    [pyinfra.api.util] Building command (shell_executable: sh): docker exec 414437174a1dc3360b28d41ac244234be2a0eed1653cba70a5638cd1b76d6b76 sh -c 'grep "^QUOTAUSER=\"\"$" /etc/adduser.conf 2> /dev/null || (find /etc/adduser.conf -type f > /dev/null && echo "__pyinfra_exists_/etc/adduser.conf")'
[pyinfra.api.connectors.local] --> Running command on localhost: sh -c 'docker exec 414437174a1dc3360b28d41ac244234be2a0eed1653cba70a5638cd1b76d6b76 sh -c '"'"'grep "^QUOTAUSER=\"\"$" /etc/adduser.conf 2> /dev/null || (find /etc/adduser.conf -type f > /dev/null && echo "__pyinfra_exists_/etc/adduser.conf")'"'"''
[ubuntu] >>> sh -c 'docker exec 414437174a1dc3360b28d41ac244234be2a0eed1653cba70a5638cd1b76d6b76 sh -c '"'"'grep "^QUOTAUSER=\"\"$" /etc/adduser.conf 2> /dev/null || (find /etc/adduser.conf -type f > /dev/null && echo "__pyinfra_exists_/etc/adduser.conf")'"'"''
[ubuntu] QUOTAUSER=""
[pyinfra.api.connectors.local] --> Waiting for exit status...
[pyinfra.api.connectors.local] --> Command exit status: 0
    Loaded fact find_in_file: ('/etc/adduser.conf', '^QUOTAUSER=\\"\\"$')
True
    [pyinfra.api.operation] Adding operation, {'test2'}, called @ mike.py:22, opLines=(0, 22), opHash=b4e88706f398aed3f3887a084f64b712e0ba8e59
    [pyinfra.api.facts] Getting fact: find_in_file (ensure_hosts: (ubuntu,))
    [pyinfra.api.util] Building command (shell_executable: sh): docker exec 414437174a1dc3360b28d41ac244234be2a0eed1653cba70a5638cd1b76d6b76 sh -c 'grep "^.*\^QUOTAUSER=""\$" /etc/adduser.conf 2> /dev/null || (find /etc/adduser.conf -type f > /dev/null && echo "__pyinfra_exists_/etc/adduser.conf")'
[pyinfra.api.connectors.local] --> Running command on localhost: sh -c 'docker exec 414437174a1dc3360b28d41ac244234be2a0eed1653cba70a5638cd1b76d6b76 sh -c '"'"'grep "^.*\^QUOTAUSER=""\$" /etc/adduser.conf 2> /dev/null || (find /etc/adduser.conf -type f > /dev/null && echo "__pyinfra_exists_/etc/adduser.conf")'"'"''
[ubuntu] >>> sh -c 'docker exec 414437174a1dc3360b28d41ac244234be2a0eed1653cba70a5638cd1b76d6b76 sh -c '"'"'grep "^.*\^QUOTAUSER=""\$" /etc/adduser.conf 2> /dev/null || (find /etc/adduser.conf -type f > /dev/null && echo "__pyinfra_exists_/etc/adduser.conf")'"'"''
[ubuntu] __pyinfra_exists_/etc/adduser.conf
[pyinfra.api.connectors.local] --> Waiting for exit status...
[pyinfra.api.connectors.local] --> Command exit status: 0
    Loaded fact find_in_file: ('/etc/adduser.conf', '^.*\\^QUOTAUSER=""\\$')
True
    [ubuntu]  Ready: mike.py

--> Proposed changes:
    Groups: @docker
    [ubuntu]   Operations: 2   Commands: 2

--> Beginning operation run...
--> Starting operation: test1
    [pyinfra.api.operations] Starting operation test1 on ubuntu
    [pyinfra.api.util] Building command (shell_executable: sh): docker exec 414437174a1dc3360b28d41ac244234be2a0eed1653cba70a5638cd1b76d6b76 sh -c 'sed -i="" "s/^QUOTAUSER=\"\"$/QUOTAUSER=\"\"/" /etc/adduser.conf'
[pyinfra.api.connectors.local] --> Running command on localhost: sh -c 'docker exec 414437174a1dc3360b28d41ac244234be2a0eed1653cba70a5638cd1b76d6b76 sh -c '"'"'sed -i="" "s/^QUOTAUSER=\"\"$/QUOTAUSER=\"\"/" /etc/adduser.conf'"'"''
[ubuntu] >>> sh -c 'docker exec 414437174a1dc3360b28d41ac244234be2a0eed1653cba70a5638cd1b76d6b76 sh -c '"'"'sed -i="" "s/^QUOTAUSER=\"\"$/QUOTAUSER=\"\"/" /etc/adduser.conf'"'"''
[pyinfra.api.connectors.local] --> Waiting for exit status...
[pyinfra.api.connectors.local] --> Command exit status: 0
    [ubuntu] Success

--> Starting operation: test2
    [pyinfra.api.operations] Starting operation test2 on ubuntu
    [pyinfra.api.util] Building command (shell_executable: sh): docker exec 414437174a1dc3360b28d41ac244234be2a0eed1653cba70a5638cd1b76d6b76 sh -c 'echo "QUOTAUSER=""" >> /etc/adduser.conf'
[pyinfra.api.connectors.local] --> Running command on localhost: sh -c 'docker exec 414437174a1dc3360b28d41ac244234be2a0eed1653cba70a5638cd1b76d6b76 sh -c '"'"'echo "QUOTAUSER=""" >> /etc/adduser.conf'"'"''
[ubuntu] >>> sh -c 'docker exec 414437174a1dc3360b28d41ac244234be2a0eed1653cba70a5638cd1b76d6b76 sh -c '"'"'echo "QUOTAUSER=""" >> /etc/adduser.conf'"'"''
[pyinfra.api.connectors.local] --> Waiting for exit status...
[pyinfra.api.connectors.local] --> Command exit status: 0
    [ubuntu] Success

--> Results:
    Groups: @docker
    [ubuntu]   Successful: 2   Errors: 0   Commands: 2/2
localhost: >>> docker commit 414437174a1d
localhost: sha256:239412717cf464b125d4e5cd9adb91d615b777143c06081ed8c606c2cd1e0e90
localhost: >>> docker rm -f 414437174a1d
localhost: 414437174a1d
    [ubuntu] docker build complete, image ID: 239412717cf4

and on a vm:

$ pyinfra  --user vagrant --password vagrant 192.168.2.232 mike.py
--> Loading config...
--> Loading inventory...

--> Connecting to hosts...
    [192.168.2.232] Connected

--> Preparing operations...
    Loading: mike.py
True
True
    [192.168.2.232]  Ready: mike.py

--> Proposed changes:
    Ungrouped:
    [192.168.2.232]   Operations: 2   Commands: 2

--> Beginning operation run...
--> Starting operation: test1
    [192.168.2.232] Success

--> Starting operation: test2
    [192.168.2.232] Success

--> Results:
    Ungrouped:
    [192.168.2.232]   Successful: 2   Errors: 0   Commands: 2/2

[Fizzadar]

Definitely looks like a bug, hope to get a chance to look at this ASAP, files.line is such an important operation.

[Fizzadar]

So the second version is how I designed files.line to behave - you pass the string exactly (unescaped) as the replace value.

Found two bugs in the find_in_file fact, fixed w/https://github.com/Fizzadar/pyinfra/commit/697873ac7729cdd7b3366853b2df9112395494d7 + https://github.com/Fizzadar/pyinfra/commit/b7e3a54c9912c003d474005bb4ef6e4ed60edc2f.

This should now work:

line2 = 'QUOTAUSER=""'
test2 = files.line(
    {'test2'},
    '/etc/adduser.conf',
    '^{}$'.format(line2),
    replace=line2,
)
print(test2.changed)

The escaping isn't required!

[Fizzadar]

Released in v0.13.1!