Super-linter: FILTER_REGEX_INCLUDE seems not to work

[TeraNovaLP]

Describe the bug
I added the FILTER_REGEX_INCLUDE environment var with this regex .*\.(cs|css|scss|ts|js|yml|ps1|vue). But file endings that are not in this regex are checked anyway. I tested my regex with the following script, there it works: https://github.com/github/super-linter/pull/684#issuecomment-687603635

To Reproduce
Steps to reproduce the behavior:

1. Create a folder with a json file.
2. Run the linter with this command: docker run -e RUN_LOCAL=true -e FILTER_REGEX_INCLUDE='.*\.(cs|css|scss|ts|js|yml|ps1|vue)' -v PATH_TO_FOLDER:/tmp/lint github/super-linter

Expected behavior
I expect that only that files are checked where the regex applies.

Screenshots

Additional context

[theodore-s-beers]

Yeah, I can't get FILTER_REGEX_INCLUDE or FILTER_REGEX_EXCLUDE to work at all. It even confuses me that the README section about this has Bash syntax (e.g. FILTER_REGEX_EXCLUDE=.*test/.*). I thought this was supposed to be an env variable set in YAML. (edit: I guess it depends whether you're running the linter locally…) It would help, especially for less experienced users, if someone could post a known good example of a config file with these variables included.

I'm aware that the PR in question was merged only days ago. Maybe there are still a few kinks to work out. Thank you to everyone involved in this effort.

[GaboFDC]

@nvuillam can you help us look at this?

[nvuillam]

I'll have a look :)

[theodore-s-beers]

Thanks. Another question I have is whether VALIDATE_ALL_CODEBASE would override FILTER_REGEX_EXCLUDE (preferably not).

[nvuillam]

FILTER_REGEX_EXCLUDE is (supposed to be) applied on all files while filtering, even in case of override of other configuration variables

[nvuillam]

I think my check to see if FILTER_REGEX_INCLUDE and FILTER_REGEX_EXCLUDE are set is wrongly done

https://stackoverflow.com/a/3601734/7113625 solution would deserved to be tried

  if [[ -n "$FILTER_REGEX_INCLUDE" ]]; then
    for index in "${!LIST_FILES[@]}"; do
      [[ ! (${LIST_FILES[$index]} =~ $FILTER_REGEX_INCLUDE) ]] && unset -v 'LIST_FILES[$index]'
    done
  fi

  #################################################
  # Filter files if FILTER_REGEX_EXCLUDE is set #
  #################################################
  if [[ -n "$FILTER_REGEX_EXCLUDE" ]]; then
    for index in "${!LIST_FILES[@]}"; do
      [[ ${LIST_FILES[$index]} =~ $FILTER_REGEX_EXCLUDE ]] && unset -v 'LIST_FILES[$index]'
    done
  fi

Unfortunately I'm on windows and I don't have anything to run locally yet... Will test that once I can, except if someone else tried before :)

[nvuillam]

It could also be escaping characters matters because of regex in command line argument

Some debug about variables FILTER_REGEX_INCLUDE & FILTER_REGEX_EXCLUDE values may help

[nvuillam]

Soooo I tried an attempt to solve the issue, when seems to be a good old argument wrong order ( shame on me 😡 )

@admiralAwkbar maybe it is possible to generate a docker image of a beta version (with #723 ) , so we can test with docker run before publishing to an official release ?

[theodore-s-beers]

I don't think there was a problem with the original syntax for checking whether these variables are set. -n is the default test, so it doesn't need to be indicated explicitly, and a single bracket should be fine (provided the variable is quoted). Double brackets as a Bash-specific feature have certain advantages, which aren't really meaningful in this case.

I guess the revision offers a lot of redundancy: double brackets, explicit -n, quoted variable. But, if I understand correctly, it was fine (and arguably better) to begin with.

[nvuillam]

@admiralAwkbar > CI failed again, no way to test if my corrections are effective... maybe you can manually run it again ? :)

https://github.com/github/super-linter/actions/runs/255599210 (same good old phar.io error)

Step 38/75 : RUN wget --tries=5 -O phive.phar https://phar.io/releases/phive.phar     && wget --tries=5 -O phive.phar.asc https://phar.io/releases/phive.phar.asc     && gpg --keyserver pool.sks-keyservers.net --recv-keys 0x9D8A98B29B2D5D79     && gpg --verify phive.phar.asc phive.phar     && chmod +x phive.phar     && mv phive.phar /usr/local/bin/phive     && rm phive.phar.asc     && phive install --trust-gpg-keys 31C7E470E2138192,CF1A108D0E7AE720,8A03EA3B385DBAA1
 ---> Running in 3af0ed00573f
Connecting to phar.io (188.94.27.6:443)
Connecting to github.com (140.82.114.3:443)
Connecting to github-production-release-asset-2e65be.s3.amazonaws.com (52.216.112.3:443)
saving to 'phive.phar'
phive.phar           100% |********************************|  187k  0:00:00 ETA
'phive.phar' saved
Connecting to phar.io (188.94.27.6:443)
Connecting to github.com (140.82.113.4:443)
Connecting to github-production-release-asset-2e65be.s3.amazonaws.com (52.216.133.211:443)
saving to 'phive.phar.asc'
phive.phar.asc       100% |********************************|   854  0:00:00 ETA
'phive.phar.asc' saved
gpg: directory '/root/.gnupg' created
gpg: keybox '/root/.gnupg/pubring.kbx' created
gpg: keyserver receive failed: Address not available
The command '/bin/sh -c wget --tries=5 -O phive.phar https://phar.io/releases/phive.phar     && wget --tries=5 -O phive.phar.asc https://phar.io/releases/phive.phar.asc     && gpg --keyserver pool.sks-keyservers.net --recv-keys 0x9D8A98B29B2D5D79     && gpg --verify phive.phar.asc phive.phar     && chmod +x phive.phar     && mv phive.phar /usr/local/bin/phive     && rm phive.phar.asc     && phive install --trust-gpg-keys 31C7E470E2138192,CF1A108D0E7AE720,8A03EA3B385DBAA1' returned a non-zero code: 2
2020-09-15 12:05:21 [FATAL ]   failed to [build] Dockerfile!

[nvuillam]

@TeraNovaLP > The docker build seems correct now, please can you tell if it works better with your test case , using latest version ?

docker run -e RUN_LOCAL=true -e FILTER_REGEX_INCLUDE='.*\.(cs|css|scss|ts|js|yml|ps1|vue)' -v PATH_TO_FOLDER:/tmp/lint github/super-linter:latest

[nvuillam]

Works on my side with latest docker image version

docker pull github/super-linter:latest
git clone https://github.com/nvuillam/npm-groovy-lint.git
docker run -e RUN_LOCAL=true -e FILTER_REGEX_INCLUDE='(.*lib/.*)' -e FILTER_REGEX_EXCLUDE='(.*lib/example/.*)' -v ~/npm-groovy-lint:/tmp/lint github/super-linter

As result, it lints only files inside /lib folder , but not files inside /lib/example folder, so the filtring seems to be performed correctly 🎈

[theodore-s-beers]

Thanks. I'll try again once these changes are reflected in the GitHub Action.

[admiralAwkbar]

@nvuillam Seems the restart worked on the job... i hate that phar...

[TeraNovaLP]

@TeraNovaLP > The docker build seems correct now, please can you tell if it works better with your test case , using latest version ?

docker run -e RUN_LOCAL=true -e FILTER_REGEX_INCLUDE='.*\.(cs|css|scss|ts|js|yml|ps1|vue)' -v PATH_TO_FOLDER:/tmp/lint github/super-linter:latest

@nvuillam It works now as expected. Thanks

[nvuillam]

I confirm that with latest version it works fine for me 👍

      - name: Lint Code Base
        uses: github/super-linter@v3
        env:
          DEFAULT_BRANCH: master
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          FILTER_REGEX_EXCLUDE: .*(lib/example|\.github|\.vscode|pull_request_template).*
          VALIDATE_EDITORCONFIG: false
          VALIDATE_JAVASCRIPT_STANDARD: false
          LINTER_RULES_PATH: "."
          JAVASCRIPT_ES_CONFIG_FILE: .eslintrc.js

[theodore-s-beers]

Yeah, it works. Thanks so much for doing this! I wanted a setup where everything is linted according to the defaults, while carving out a few narrow exceptions. And now that's possible.

[nvuillam]

You're welcome 🤓