Cfn-python-lint: Does not lint deeply nested template paths

[phene]

cfn-lint version: 0.9.2

Using ** path globs does not behave as expected. Assume the following directory structure

project:
  templates:
    - a:
      - b:
        - c:
          - foo.yaml
          - bar.yaml
        - d:
          - baz.yaml
          - quux.yaml
      - e:
        - ipsum.yaml
        - lorem.yaml
    - f:
      - zyx.yaml
      - quuz.yaml
    - main.yaml

If I use the glob format in .cfnlintrc:

templates:
- project/templates/**/*.yaml

The only files that are checked are project/templates/f/zyx.yaml and project/templates/f/quux.yaml. All others are ignored. I would expect project/templates/**/*yaml to match all deeply nested templates as well as project/templates/main.yaml.

[kddejong]

The easy fix for Python 3.5 or higher is to add , recursive=True to https://github.com/awslabs/cfn-python-lint/blob/master/src/cfnlint/config.py#L439

I'm debating punting on the backwards compatibility of this and just saying we support what glob can do based on the version of Python available.

[fatbasstard]

@phene What OS and what Python version are you running?

[cmmeyer]

I like @kddejong's solution. Based on my research, getting recursive globbing working on 3.4 would require moving to something like pathlib. Adding recursive=True gets us compatibility back to 3.5 (which is pretty good) without the need for any significant code changes. I'm going to go ahead and approve it.

[phene]

@cmmeyer This is still not behaving appropriately:

a: 
  - b:
    - c.yaml
  - d.yaml

I expect a/**/*yaml to match both a/b/c.yaml and a/d.yaml. It is only matching a/b/c.yaml

$ python --version
Python 3.7.2
$ pipenv run cfn-lint --version
cfn-lint 0.11.1

[PatMyron]

@phene could you try shopt -s globstar beforehand if using bash? Solved a similar issue

may need to upgrade bash, especially if using Mac. This is how I upgraded bash on Mac:

brew install bash
sudo bash -c 'echo /usr/local/bin/bash >> /etc/shells'
chsh -s /usr/local/bin/bash

[phene]

This shouldn't require bash to function. It _should_ just work on systems that just have /bin/sh or use zsh.