Fluent-bit: Stream processing: WHERE field IS (NOT) NULL

[dperetti]

Is your feature request related to a problem? Please describe.
It does not seem possible do use a query checking for the existence of a field using sql WHERE IS NOT NULL.
This would be super useful to tag and route to a different ouput depending on the existence of a field created by a parser.

[edsiper]

cc: @koleini

[edsiper]

hmm to validate if a key exists we would need a different syntax, for normal case:

• IS NULL => would mean "key" have a value equal to msgpack null
• IS NOT NULL => opposite (value != msgpack null)

should we use comething like:

• WHERE key EXISTS
• WHERE key NOT EXISTS

ideas ??

[koleini]

As @edsiper mentioned, we need a different syntax to represent the non-existence of a key, since semantically, NULL means the key exists, but doesn't have a value (however, currently, we don't support NULL checking).

I'm now working on checking the existence of the key in WHERE condition, since it is a useful feature to have.

[edsiper]

Fixed by #1356, thanks to @koleini you now can use:

SELECT * FROM STREAM:test WHERE key IS NULL;

or

SELECT * FROM STREAM:test WHERE key IS NOT NULL;

further checks if a key exists or not, we are working in a different approach since for our binary records "NULL" is a valid value for a key.

note: code changes in git master, to be available in the next major version.

[rolandjitsu]

@edsiper is the EXISTS functionality implemented in master? E.g.

SELECT * FROM STREAM:test WHERE NOT EXISTS key;

[koleini]

@rolandjitsu This feature is in master, but with different syntax (I will update docs today). You can use @record.contains(key) to check the existence of a key. For instance, your query will be:

SELECT * FROM STREAM:test WHERE NOT @record.contains(key);

[rolandjitsu]

@koleini I see, thanks. I will try it out.

[koleini]

@rolandjitsu docs are updated to cover this:
https://github.com/fluent/fluent-bit/tree/11c3e48a4391af26a22978bdefa0773f9c5abc27/src/stream_processor
https://docs.fluentbit.io/stream-processing/getting_started/fluent_bit_sql