Jsonnet: stdlib: add support regexp match and replace for strings

[mikedanese]

Would be easy to implement as a builtin.

https://github.com/google/re2

[sparkprime]

Hard in Jsonnet, easy as a builtin, but the trick is to make sure every language that someone might want to implement Jsonnet in (and therefore have to provide an implementation for each of the builtins) has a native regular expression library with exactly the same regex syntax and semantics.

[mikedanese]

PCRE seems to be implemented in many languages. Perhaps it would be better to implement this as a native extension (#108) to the language and not part of the core.

[sparkprime]

Does it support unicode typically?

[nand0p]

+1

[benley]

It appears that PCRE does typically support unicode: http://man7.org/linux/man-pages/man3/pcreunicode.3.html

[sparkprime]

Would be great if the 3 of you could offer some real use cases for this functionality so I can figure out how to prioritize it.

[nand0p]

For our use case, we need to strip all non-alphanumeric chars from a string variable. i am thinking this can be done currently by splitting string to array of chars, checking each char, and then rejoining.... but that seems very ugly.

This may be able to be done more easily with a new function like std.toAlpha(x), but i would think full-on regex capabilities would be a more complete solution.

[sparkprime]

It's not too bad but I can see why you'd rather write it with 0-9A-Za-z type ranges and on one line.

local is_alpha(x) =
    std.setMember(x,"0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz");
local to_alpha(str) = std.join("", std.filter(is_alpha, std.stringChars(str)));
to_alpha("He.llo World123")

[emmanuel]

I have a case where I'd like to be able to replace all instances of - with _. That seems even more cumbersome than the filter-to-alphanumeric use-case discussed above. Regex is overkill for my use-case, but either regex-based or tr-style replace functionality would be helpful.

On a nearer-term note, can the stdlib/built-ins be composed to produce a 'replace each instance of character x with an instance of character y' behavior? I'm not coming up with it, though I'm quite new to Jsonnet.

[sbarzowski]

Here's an example implementation of proposed _'replace each instance of character x with an instance of character y'_

local replaceChars(str, mapping) =
    std.join("", std.map(function(c) if c in mapping then mapping[c] else c, std.stringChars(str)));
replaceChars("abcd", {"b": "!", "d": "?"})

produces:
"a!c?"

The implementation above also support deleting characters or replacing them with multiple characters:
replaceChars("abcd", {"b": "xx", "d": ""})
produces:
"axxc"

It may be useful generally enough to add it to stdlib. @sparkprime what do you think?

[emmanuel]

@sbarzowski thank you, that is fantastic. Not only a clean interface to accomplish what I'm looking to get done, but also a good bit of insight about how to approach programming jsonnet. I'd be in favor of adding this to stdlib, but I'm not on the hook for maintenance, so perhaps merely adding to the documentation would be sufficient to help future seekers like myself.

Whatever the decision about adding to stdlib or docs, thank you for the help @sbarzowski.

[sparkprime]

tr-like functionality is definitely a good candidate for stdlib.

[sparkprime]

Since this has come up again - do we have compatible implementations of PCRE in Go and C++ that will work with unicode?

[sbarzowski]

Well, there is this thing: https://github.com/glenn-brown/golang-pkg-pcre. This is an interface to libpcre. It seems to hardcode assumptions about where libpcre is installed, though... I couldn't find anything else. Probably using libpcre directly with cgo would be a better option.

[kamalmarhubi]

My guess is that that package defeats part of the purpose of go-jsonnet, which is to allow go programs to use jsonnet without cgo. Could be wrong though. :-)

[sparkprime]

Yeah I think unless we can find a library that has native Go and C++ support (for exactly the same regex syntax) we'll have to leave regexes as something that people add with native extensions.

[dcoles]

Coming back full circle, would RE2 along with Go's built-in regexp package not be a good fit? There're Unicode aware and syntax compatible.

From Go's regexp package documentation:

"The syntax of the regular expressions accepted is the same general syntax used by Perl, Python, and other languages. More precisely, it is the syntax accepted by RE2 and described at https://golang.org/s/re2syntax, except for \C."

(The re2syntax link goes to the actual RE2 documentation)

[sparkprime]

In that case I guess RE2 is the way forward after all :)

[dcoles]

I'm currently prototyping RE2 regexp support in my https://github.com/google/jsonnet/compare/master...dcoles:re2 branch.

Boolean matches can be implemented pretty trivially, but positional and named captures are going to require a bit more thought. The current plan is to have a match return an object upon successful match or null otherwise. For example:

$ jsonnet -e 'std.regexFullMatch("hello", "h(?P<mid>.*)o")'
{
   "captures": [
      "ell"
   ],
   "namedCaptures": {
      "mid": "ell"
   },
   "string": "hello"
}

This way you can still do things like assert std.regexFullMatch(self.foo, "pattern") != null for validation or use the object fields for accessing captured values.