Salt: module.file.replace string seems to be mutated into arrays

[grinapo]

Description of Issue/Question

$ salt host1 file.replace /tmp/test.yaml pattern='- test' repl='- ct'
resulted lots of really funny (but pretty much broken) lines like:
clu['ct']['ct']['ct']r: myclu['ct']['ct']['ct']r

Seems it's not always a regex...
May have been fixed in the new versions, but maybe not and you'd like to see that.

Versions Report

$ salt --versions-report
Salt Version:
Salt: 2016.11.1
Dependency Versions:
cffi: Not Installed
cherrypy: Not Installed
dateutil: 2.5.3
gitdb: 2.0.0
gitpython: 2.1.1
ioflo: Not Installed
Jinja2: 2.8
libgit2: Not Installed
libnacl: Not Installed
M2Crypto: Not Installed
Mako: 1.0.6
msgpack-pure: Not Installed
msgpack-python: 0.4.8
mysql-python: Not Installed
pycparser: Not Installed
pycrypto: 2.6.1
pygit2: Not Installed
Python: 2.7.13 (default, Jan 19 2017, 14:48:08)
python-gnupg: Not Installed
PyYAML: 3.12
PyZMQ: 16.0.2
RAET: Not Installed
smmap: 2.0.1
timelib: Not Installed
Tornado: 4.4.3
ZMQ: 4.2.1
System Versions:
dist: debian 9.0
machine: x86_64
release: 4.4.49-1-pve
system: Linux
version: debian 9.0

[gtmanfred]

Can you give me an example file that this breaks with so that I can replicate it and we can figure out what is going on?

Thanks,
Daniel

[grinapo]

Sure.

Original:

this: is sad
roles:
  - test
main:
  - cluster: mycluster

Command

salt host1 file.replace /tmp/test.yaml pattern='- test' repl='- ct'

Result

['ct']hi['ct']: i['ct'] ['ct']ad
rol['ct']['ct']:
  - ['ct']['ct']['ct']['ct']
main:
  - clu['ct']['ct']['ct']r: myclu['ct']['ct']['ct']r

[lorengordon]

I think when passing inputs on the cli, the values are parsed with yaml. That's actually a feature. Try double-quoting the values, or otherwise escaping the inputs a bit so yaml sees them as strings.

[grinapo]

I thought something like that, but the question is then whether it's well documented? I kind of read all the docs and don't remember such one, but I may be mistaken.
(Obviously I can avoid that if I'm aware that I have to; yet to see what's this good for.)

[lorengordon]

I think it's a convenience so that dictionaries and lists can be passed _relatively_ easily on the cli. These types are used quite commonly for many execution module arguments. I think there are quite a few such examples in the docs. Although you're probably right, I'm not sure where in the docs it would be spelled out explicitly as a feature though, if it is at all.

[gtmanfred]

that is definitely what is happening, it is converting the input into a list as if it was yaml.

@saltstack/team-core do we have this documented anywhere that cli variables are loaded through the yaml loader?

Thanks,
Daniel

[grinapo]

Still, I wonder how. I mean, why "a member of a sequences of scalars" became an array? If I guess correctly
- test
have became
['t','e','s','t']
and I fail to see why, based on yaml specs.

[lorengordon]

If I had to guess, has something to do with pattern being a regex. If I'm right, I would expect - test is yaml-fied to ['test'] (by the cli/loader/whatever), which is then treated as a regex pattern (by file.replace) that matches those individual characters.

[gtmanfred]

That is exactly what is happening, it actually gets mutated to be [test] so it will match any character in there, and since it is the word you are trying to match, it will replace each letter with one of your second sequence which is rendered to [ct], and since that is not pattern matching, it just puts that object in place of every letter.

[terminalmage]

YAML lists are prepended with dashes, so the leading dash is making the loader assume a list:

>>> yaml.safe_load('''\
... - foo
... - bar
... - baz''')
['foo', 'bar', 'baz']
>>> yaml.safe_load('- foo')
['foo']

https://github.com/saltstack/salt/pull/46322 fixes this. We have similar logic to make sure foo: bar doesn't get loaded as a dictionary, we just needed logic to force the same when the result of YAML loading is a list and the string doesn't begin with [.

One temporary workaround would be to use an extra layer of quoting:

[root@7bfc22e7baee /]# salt-call --local test.arg '"- foo"' --out=pprint
{'local': {'args': ('- foo',),
           'kwargs': {'__pub_fun': 'test.arg',
                      '__pub_jid': '20180303210525196830',
                      '__pub_pid': 367,
                      '__pub_tgt': 'salt-call'}}}