Elasticsearch: Enhance painless documentation

[gmoskovicz]

We state that painless is similar than Groovy, but to be fair most of the code that you paste from groovy will need to be modified in order to work with this new language.

One simple example is the following that works in Groovy but not in Painless:

variable = list.sort { a, b -> a.value == b.value ? 0 : a.value < b.value ? -1 : 1 }

Now, this could be translated into the following:

variable = list.sort( (a, b) -> a.value == b.value ? 0 : a.value < b.value ? -1 : 1 );

Also, looks like joda.time.DateTime cannot be used, and there is no clear explanation how to deal with dates. See https://github.com/elastic/elasticsearch/issues/22162.

As you can see, it's similar, but still you usually need to modify a big part of the code. Unfortunately this type of similarities are not documented, nor you can understand in an easy way how to translate groovy or how to develop with painless. My proposal is to:

1. Clearly explain in the documentation that while it's similar to groovy, there are a lot of differences which means that you will need to translate most of your code into painless.
2. Enhance the documentation with sections like "how to translate groovy to painless".
3. Add more sample docs around painless.
4. Add documentation on how the language works, what things we support, and what things we don't support.

We shouldn't expect to have a large documentation, but at least to have the general concepts. Just to see how other languages are explained:

• C#: https://msdn.microsoft.com/en-us/library/67ef8sbd.aspx
• Java: https://docs.oracle.com/javase/specs/jls/se8/html/index.html
• Groovy http://groovy-lang.org/documentation.html (note the "differences with Java section")

[jasontedor]

We state that painless is similar than Groovy, but to be fair most of the code that you paste from groovy will need to be modified in order to work with this new language.

To be fair, when developers hear that two languages are similar, they do not have an expectation that they can paste code from one language into another. For example, developers freely admit that C# and Java are similar (in fact, very similar), but no developer comfortable with one of those languages expects they can paste code back and forth between the two (sometimes you can paste a snippet, the vast majority of the time you can not).

This does not negate the point that we should always work to improve our documentation, especially for a feature as nascent as Painless is, but I do think it's worth correcting the expectation that developers have when they say and hear that two languages are similar.

[jdconrad]

Agree 100% the documentation needs improvements. I'm currently working on making some, but there's a lot to cover and it's going to take some time to get there. I also see how comparing this language to Groovy could be confusing given the lack of documentation here.

[nik9000]

I'm still working on prerequisites for an API reference.... I think I have one review left before I can revive my work on the reference. That'll help. Some.

[nik9000]

The API reference I was talking about: https://github.com/elastic/elasticsearch/pull/22775

[PhaedrusTheGreek]

My findings from working with painless a bit:

• You can't declare global scope variables in painless, like you can in groovy. If you need to share variables, pass them in functions. (not documented)
• All functions must be declared at the top of the script (documented)
• In compile errors, Character number is thrown, not Line number. In vim you can do :goto <character> (not documented, existing issue)
• in .vimrc , add this for syntax highlighting: au BufRead,BufNewFile *.painless set filetype=groovy (helpful hints?)
• You can't use groovy shorthand for loop iteration (list.each {}), you must use the Java style syntax: for (Object it : list) { } (documented)

Attached is an example that will traverse each JSON document in order to fix invalid fields names. Can be attached to an ingest pipeline:

• trim_dots.painless

[jdconrad]

@PhaedrusTheGreek Thank you for the feedback here! Appreciate it.

[dakrone]

I wouldn't want Emacs to be left out, so:

(add-to-list 'auto-mode-alist '("\\.painless$" . groovy-mode))

(requires groovy-mode to already be installed of course)

[rjernst]

in .vimrc , add this for syntax highlighting: au BufRead,BufNewFile *.painless set filetype=groovy
I wouldn't want Emacs to be left out

I would not do this. If we want syntax highlighting, then we should create our own filetype plugins for vim/emacs. Otherwise you will think eg a closure is valid, because of syntax highlighting, but painless does not have closures. Likewise, it would not properly format lambdas, since groovy does not support them.

[nik9000]

You can't use groovy shorthand for loop iteration (list.each {}), you must use the Java style syntax: for (Object it : list) { } (documented)

I believe you can do it java style like list.stream.forEach(e -> {code}). Painless doesn't support groovy's funky rules around parameters.

[dakrone]

we should create our own filetype plugins for vim/emacs

Sure, I'll add this to my TODO list

[jdconrad]

@dakrone If/when you find the time to do this, maybe you can just pull from the existing grammar files to know what valid syntax is.

[nik9000]

maybe you can just pull from the existing grammar files to know what valid syntax is.

Don't forget the semicolon hacks!

[Cinerar]

@jdconrad Please find some time to answer StackOverFlow questions, right now it is very hard learning curve.

[jdconrad]

@Cinerar I'd be happy to. Do you have any particular questions or links to questions that you would like to me to prioritize?

[gmoskovicz]

@Cinerar it is more appropriate to open a new issue in http://discuss.elastic.co/ if you have a specific question. I am sure that @jdconrad or any other Engineer will be more than happy to help 😃 .

[Cinerar]

@jdconrad There are not so much of them right now.
http://stackoverflow.com/search?q=%5Belasticsearch%5D+painless

I'm interested in http://stackoverflow.com/questions/42009778/elasticsearch-painless-script-how-to-iterate-in-an-array-of-nested-objects

Thank you.

[PhaedrusTheGreek]

A few things

• Semicolons are required, unlike groovy
• In script file, as noted here, params are accessible via params.<your_var> instead of in the base scope.

[jdconrad]

Closing this in favor of (https://github.com/elastic/elasticsearch/issues/23777)