Semgrep: add `pattern-regex`

[DrewDennison]

to enable semgerp to completely replace any PCRE regex tool

[aryx]

I like this.

[ievans]

It is nice that our boolean composition work will work seamlessly with this. I wonder what this means for thefuture of the python parts of semgrep.

[aryx]

Honestly, it can be nice to have semgrep written half in OCaml half in Python so if people want to contribute, it's less scary if they think they can probably do stuff on the Python side. I feel the split with the set-based operators outside semgrep-core is still pretty good. I don't think we suffer much from it.

[DrewDennison]

basic feature: if pattern-regex is defined, read the file in python and return the range that the regex matches. Acceptance criteria is that pattern-regex should work with the boolean logic so I can match something like

boto3.client(host="123.123.123.123")

with

patterns:
  - pattern: boto3.client(host="...")
  - pattern-regex: "\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}"

[DrewDennison]

cc @nbrahms

[ievans]

@DrewDennison in that example, I think you would want:

patterns:
  - pattern-inside: boto3.client(host="...")
  - pattern-regex: "\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}"

So that the regex range must be inside the semgrep range

[nbrahms]

Just a heads up for @dlukeomalley that pattern-regex will be PCRE-compliant, whereas string regexes in pattern will not be.

[nbrahms]

Also, FWIW:

(?<!\d)[12]?\d{1,2}(?:\.[12]?\d{1,2}){3}

~should be the~ is a closer PCRE for a host 🤷 😛

[mschwager]

@DrewDennison in that example, I think you would want:

patterns:
  - pattern-inside: boto3.client(host="...")
  - pattern-regex: "\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}"

So that the regex range must be inside the semgrep range

I don't think pattern vs. pattern-inside matters in this situation since they're using the same value and thus will return the same results. semgrep doesn't distinguish between two patterns returning the same information, it just filters the information it gets back. I.e. pattern-regex will simply be filtering the results it receives back from semgrep-core, it doesn't care if the results came from pattern or pattern-inside. We could change this, but that would take a larger refactor and may change the existing behavior.

Note another limitation of the current pattern-regex approach is that it can only filter existing results, it can't be used by itself (e.g. as a top-level field). This is because pattern-regex relies on results generated from other patterns (it can only filter the semgrep-core results sent to it). Why can't we just make it do its own thing prior to invoking semgrep-core? The semgrep Python code doesn't know about which file paths are being considered, it simply sends the CLI arguments down to semgrep-core. This means it doesn't know the file paths and thus can't regex filter their contents. Since include/exclude functionality is implemented in semgrep-core we can't know for sure which file paths are being considered until semgrep-core sends us results. So, long story short, the Python code cannot generate results with top-level fields, it can only filter them. Again, we could change this, but that's a much larger refactor/piece of work.

[DrewDennison]

@mschwager I think a hard requirement is the also support new languages that doesn't exist in semgrep yet. i.e. I would also like for someone to write a regex to scan terraform, ruby, etc while we work on tree-sitter support for new languages. I think this means it should be a top-level key

[dlukeomalley]

Note another limitation of the current pattern-regex approach is that it can only filter existing results, it can't be used by itself (e.g. as a top-level field).

@mschwager Thanks for calling this out. Based on the feedback we've gotten, the expectation is that pattern-regex works standalone. The goal is to be able to deprecate any grep-based tooling people have and use semgrep's infrastructure fully.