Salt: Multiline strings tricky when used with ``file.managed`` ``contents`` param

[whiteinge]

This one is best articulated with an example. In short, if the contents param of file.managed is a function call that returns a multi-line string the template indentation is wrong.

SLS file:

/tmp/zomg:
  file:
    - managed
    - contents: |
        {{ salt['pillar.get']('zomg:text') }}

Pillar file:

zomg:
  text: |
    This is a
    multiline test.

Results in this traceback from the PyYAML parser:

    Rendering SLS main failed, render error: while scanning a simple key
...
could not found expected ':'
  in "<unicode string>", line 7, column 1:

Because it is trying to render invalid yaml:

/tmp/zomg:
  file:
    - managed
    - contents: |
        This is a
multiline test.

This can be worked around with a little Jinja magic:

/tmp/zomg:
  file:
    - managed
    - contents: |
        {{ salt['pillar.get']('zomg:text') | indent(8) }}

This works because the indent filter does not indent the first line by default. However it is less than ideal for obvious reasons, the least of which is you have to manually count how many spaces to indent by.

[basepi]

Good to know, though I'm not sure how to fix this. =\

[whiteinge]

Me neither. The problem is YAML / template dependent so it doesn't make
sense to alter the contents handling in the file module. But the templating
layer isn't likely to know how to deal with this.

Two possible options:

Have contents insert a raw string with newlines inline so the template
isn't messed up but the newlines are preserved. This may not be possible
with yaml.

Or

Document it. :-)

[terminalmage]

Yeah, I've run into this before. The solution for me was to use a template file and put the jinja doing the reference to pillar inside the template file, rather than in the SLS.

[basepi]

Right, but the idea with the contents variable is that you don't have any source file or template. You just specify the file contents right there in your SLS.

[terminalmage]

Yeah, I'm aware. It's just how I solved this in the past. I think it's probably best just to document this, since jinja isn't really doing anything wrong. It's replacing the curly braces with the multiline string as requested, it's the YAML formatting requirements that are causing the problem.

Trying to fix this by being clever with the renderers is a recipe for tears, IMO. Better just to document it.

[jollyroger]

Just as a note: there are possible ways to mess even this hack, for example, if you distribute some SSL certificate in such way. OpenSSL always generates text output similar to manual running openssl x509 -in certfile.crt -text -noout right before actual certificate inside the PEM file and as a result renderer will see lots of tree-like indented information (data is forged and stripped):

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 16338703134708870008 (0xe2bebbef52e22b78)
    Signature Algorithm: sha1WithRSAEncryption
        Issuer: C=US, ST=New York, O=My Awesome Company, CN=myawesomecompany.com
        Validity
            Not Before: Jan 1 00:00:00 1970 GMT
            Not After : Jan 19 03:14:08 2038 GMT
        Subject: C=US, ST=New York, O=My Awesome Company, CN=myawesomeserver.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Basic Constraints: 
                CA:FALSE
            Netscape Comment: 
                OpenSSL Generated Certificate
            X509v3 Subject Key Identifier: 
                7C:F1:4A:63:84:58:83:1E:7F:DC:61:0B:C3:91:C0:DD:B7:3D:23:03
            X509v3 Authority Key Identifier: 
                keyid:23:E8:EB:A0:D3:69:72:8E:A4:52:10:01:69:5A:DA:EB:D8:7A:CD:F8

Salt will brake rendering and exit with an error on one of these lines containing ":" symbol. In my case it was the Issuer field.

[whiteinge]

@jollyroger I think it should work so long as YAML can unambiguously interpret it as a string by using the | multiline syntax. I don't know the exact command you're running but I tried the following two and they both worked correctly:

/tmp/blah:
  file:
    - managed
    - contents: |
        {{ salt['cmd.run']('openssl x509 -in myfile.crt -text -noout') | indent(8) }}

/tmp/blah:
  file:
    - managed
    - contents: |
        Certificate:
            Data:
                Version: 3 (0x2)
                Serial Number: 16338703134708870008 (0xe2bebbef52e22b78)
            Signature Algorithm: sha1WithRSAEncryption
                Issuer: C=US, ST=New York, O=My Awesome Company, CN=myawesomecompany.com
                Validity
                    Not Before: Jan 1 00:00:00 1970 GMT
                    Not After : Jan 19 03:14:08 2038 GMT
                Subject: C=US, ST=New York, O=My Awesome Company, CN=myawesomeserver.com
                Subject Public Key Info:
                    Public Key Algorithm: rsaEncryption
                        Public-Key: (2048 bit)
                        Exponent: 65537 (0x10001)
                X509v3 extensions:
                    X509v3 Basic Constraints: 
                        CA:FALSE
                    Netscape Comment: 
                        OpenSSL Generated Certificate
                    X509v3 Subject Key Identifier: 
                        7C:F1:4A:63:84:58:83:1E:7F:DC:61:0B:C3:91:C0:DD:B7:3D:23:03
                    X509v3 Authority Key Identifier: 
                        keyid:23:E8:EB:A0:D3:69:72:8E:A4:52:10:01:69:5A:DA:EB:D8:7A:CD:F8

[v6]

Has anyone other than @whiteinge verified that multi-line YAML syntax still works with python compilation?

Are the conditions under which it will work / not work written somewhere?

Does > work?

Also, I found a discussion of this here:

http://stackoverflow.com/questions/3790454/in-yaml-how-do-i-break-a-string-over-multiple-lines

Sorry to bug you about something so petty.

[whiteinge]

Paste your code and what error you're seeing so we can try to reproduce.

On Sat, Jan 24, 2015, 3:18 AM Nathan Dataguake Basanese <
[email protected]> wrote:

Has anyone other than @whiteinge https://github.com/whiteinge verified
that multi-line YAML syntax still works with python compilation?

Methinks not.

Let me know if anyone else has tested this, or if there is a solution out
there already.

—
Reply to this email directly or view it on GitHub
https://github.com/saltstack/salt/issues/5480#issuecomment-71309257.

[pprince]

Hmm... Could the solution to this [entire class of] problem[s] be as simple as…

file.managed:
- contents: {{ salt['pillar.get']('zomg:text') | yaml_encode }}

…?

yaml_encode…Serializes a single object into a YAML scalar with any necessary handling for escaping special characters. This will work for any scalar YAML data type: ints, floats, timestamps, booleans, strings, unicode. It will _not_ work for multi-objects such as sequences or maps.
— the docs for yaml_encode

IMO, it pays to be mindful, defensive, and paranoid when interpolating data into your YAML… there are a bunch of different ways you can get burned (most more subtle than this one.) Most Salt users seem happy to make cavalier assumptions about the contents/structure of the data they're in the Jinja→YAML dance (maybe because they're usually the ones who set it originally), but I cannot be so trusting ;)

<off-topic>

That doc describes another Salt-custom Jinja filters that I am currently loving: sequence. If your code currently handles ['lists', 'of', 'strings'], you can sprinkle in sequence and it will suddenly also handle "single string values" just as well.

At this point, I was going to suggest re-visiting the list of filters that come baked into Jinja, but as I look through them now I'm not seeing any super-useful or super-relevant ones… Nonetheless, it's worthwhile to occassionally re-visit the full Jinja template writer's doc; I'll frequently pick up a new test/filter/function/feature that I hadn't noticed before because I hadn't needed it before. And similarly, YAML is complex (and sometimes unintuitive) enough that doing a serious read-through of the Salt-relevant YAML specification is ultimately a time-saving proposition.

</off-topic>

Half-way through this comment, I seem to have switched over to full-on off-topic blog-post mode... since I don't have a blog, I'll leave the last couple of paragraphs here rather than delete them. I'll surely feel very silly if it turns out that yaml_encode doesn't even solve the problem.

[whiteinge]

@pprince Great suggestion! That filter is much cleaner than the indent route.

And you are very right to be wary about nested templating.