Cfn-python-lint: [Question] Issues Running on TravisCI

[ferricoxide]

cfn-lint version: (cfn-lint --version)

Presumably latest from Pypi (invocation-method follows to explain the "presumably"):

• Initial problem:

Using:

~~~
install:

• pip install --user cfn-lint
  ~~~

Resulted in numerous warnings related to SSL/TLS handlers and installed version of pip.

• Next break-point:

Updating to:

~~~
install:

• pip install --user --upgrade pip
• pip install --user cfn-lint
  ~~~

However, this resulted in a different job-failure:

~
$ pip install --user cfn-lint
Traceback (most recent call last):
File "/usr/local/bin/pip", line 7, in
from pip import main
ImportError: cannot import name main
The command "pip install --user cfn-lint" failed and exited with 1 during .
~

• Functional work-around:

Googled and found a thread that indicated that pip 10+ changed how they handled import main and a recommendation to pin the pip upgrade to a version < 10. Ultimately, I modified my .travis.yml to do:

~~~
install:

• pip install --user --upgrade pip==9.0.3
• pip install --user cfn-lint
  ~~~

The above allowed cfn-lint to run.

Description of issue

Need advice for sustainability of current work-around for use with Travis, given the above. While the above works, staying on 9.0.3 is likely sub-optimal. Wanted to know if there was a more-recommended path forward.

Also: might some of this become a non-issue after Travis finishes migrating all projects to Xenial

Additional Information

• Worker information
  hostname: [e-mail]-1-worker-com-gce-8ng8
  version: v6.1.0 https://github.com/travis-ci/worker/tree/a6071dd64658ecf13ec3190bfb9e9ca3c8f3e10d
  instance: travis-job-89a61bc2-af5a-4002-a51c-70443a743e7c travis-ci-connie-trusty-1512502258-986baf0 (via amqp)

Forcing use of Xenial as test platform also results in breakage (presumably because the base pip version is too high):

~
$ pip install --user cfn-lint
Traceback (most recent call last):
File "/usr/local/bin/pip", line 7, in
from pip._internal import main
ImportError: No module named _internal
The command "pip install --user cfn-lint" failed and exited with 1 during .
~

[kddejong]

@ferricoxide can you provide your .travis.yml file? or a part of it. Or what version of Python are you using?

I used this for a file and had no issues.

language: python
sudo: true
dist: xenial
stages:
- Lint
jobs:
  include:
  - stage: Lint
    python: 3.6
    script: cfn-lint **/*.yaml
install:
- pip install cfn-lint

Forced failure with linting error: https://travis-ci.com/kddejong/cloudformation/builds/95992462
Successful: https://travis-ci.com/kddejong/cloudformation/builds/95992355

[ferricoxide]

Hi, @kddejong, thanks for responding.

I should probably request this be closed: I was able to request they migrate my projects to their updated infrastructure which enabled me to use the updated job syntax. Things are working, now — just had to add language: python and python: 3.6 to the linter stage.

Only thing killing me, now, is that E1029 doesn't yet handle file:content: that includes BASH scripts that perform variable-reads. So, added a -i E1029 to my job, for now.

Great tool, by the way. Discovered I have a lot of templates with sloppiness in them (other frameworks I use are a lot less forgiving than CFn is — so, slopiness is more self-evident without use of a linter). Thanks to your team for making it public!

[kddejong]

@ferricoxide if you could generalize and provide an example I would love to take a look at this. I would hope you don't have to ignore any rules.

[ferricoxide]

Currently having to ignore E1029 and E2015.

• E1029 doesn't seem to grok how to deal with stuff like:

~~~
"GlusterSetup": {
  "files": {
    "/etc/cfn/scripts/volumeSetup.sh": {
      "content": {
        "Fn::Join": [
          "",
          [
            "#!/bin/bash\n",
            "# Script to configure an OS for use with Gluster\n",
            "#\n",
            "#################################################################\n",
            "PROGNAME=\"$(basename \"${0}\")\"\n",
            "VOLNAME=\"${1:-vol01}\"\n",
            "VOLMOUNT=\"${3:-/var/glusterfs/${VOLNAME}}\"\n",
            "REPLCNT=\"replica ${2:-2}\"\n",
            "if [[ ${REPLCNT} =~ .*2 ]]\n",
            "then\n",
[...elided...]
~~~

• Sadly, I haven't figured out what it is that E2015 if choking on

I mean, yeah, might be better off shovin the above type of stuff into a script that CFn downloads and executes ...but then I need to tell CFn where to find it and sort out access to the file.

[kddejong]

I'll take a look.

[kddejong]

We have exceptions for E1029. I have added content to those exceptions. Too much complicated data can exist in those files to do this correctly.

554

[kddejong]

Not sure how to help with E2015

[ferricoxide]

Yeah. No worries. Not even sure what's triggering it. CFn deploys the stacks with the error, so, it's not a fatal error, but the error message, itself:
~
E0002 Unknown exception while processing rule E2015: multiple repeat
Templates/make_EC2-node.tmplt.json:1:1
~

Doesn't even tell me _where_ the problem exists.

If you need to eyeball something, this template — among several in that and a few other of my projects — will trigger it.

[kddejong]

@ferricoxide we seem to be failing on the regex [a-z0-9]{6,12}+ There are a lot of parts of regex I sometimes don't fully get but I believe the + is causing the issue. Python is saying multiple repeat because it is counting the {6,12} as redundant with the +. Interesting as this isn't causing an issue with CloudFormation as a few of the online tools that I have tried also don't like the +. If this regex is actually working what may be in order is silently continuing when we get this type of error as I'm not sure we can mimic the code that CloudFormation is using to parse that regex. Another thing may be an new rule that checks the validity of the regex (however we may not be able to fully test validity compared to CloudFormation based on this issue).

[kddejong]

we will wrap a try/except around the regex test. Since CloudFormation using a different engine and we can't conform from a Python perspective. So best is we test it if we can.

[kddejong]

@ferricoxide should be released with v0.11.1

[ferricoxide]

At this point, I forget why I had that. The Java regex manual is a bit dense. I _think_ I was trying to establish a userid-length between 6 and 12 chars and the + might have been to place a soft upper bound? I'll need to go back to the dox to figure out why I'm using the +