Logstash: `if .. in []` doesn't match for single-element arrays

[praseodym]

A Logstash filter such as if "a" in ["a", "b"] matches but if "a" in ["a"] does not, which does not make sense. I'd consider this to be a bug.

Example pipeline:

input { generator { count => 1 } }

filter {
  if "a" in ["a"] {
    mutate { add_tag => "1" }
  }

  if "a" in ["a", "b"] {
    mutate { add_tag => "2" }
  }
}

output { stdout {} }

Output:

{
      "@version" => "1",
          "tags" => [
        [0] "2"
    ],
          "host" => "hostname",
    "@timestamp" => 2018-08-22T20:07:25.211Z,
      "sequence" => 0,
       "message" => "Hello world!"
}

I would expect to see both "1" and "2" as tags.

[yaauie]

I believe that the only way to fix this is to introduce a new, unambiguous syntax for arrays.

There is currently an ambiguity in the pipeline syntax that is resolved by favouring any sequence in a conditional clause that starts and ends with square brackets as a Field Reference; the sequence if "a" in ["a"] is currently interpreted as:

"if the string a is in the values found on the event at field reference ["a"]".

---

I've run into this before, and every attempt I have made to find a fix that works within the existing pipeline syntax ends up breaking configs used in the real world:

• (a) prohibit quotes in field references
  
  
  
  • breaks any config that legitimately uses quotes in field references
  
  
  • does not fix single-element integer-array use-case
  
  
• (b) favour arrays to field references when parsing
  
  
  
  • breaks: field reference [foo] interpreted as single-element array containing the string foo
  
  
• (c) favor non-bareword arrays to field references when parsing
  
  
  
  • breaks: integer array has same problem as (b); field reference [1] interpreted as a single-element array containing integer 1
  
  

---

As a workaround, I have seen people prepend their "haystack" array with a value known not to match the "needle" search key:

~
if "a" in ["___", "a"] {
~

[praseodym]

I see, at least I now understand what's going on. With the stricter field reference parser in Logstash 6.4 there could be an opportunity to prohibit quoted field references when they are not needed. Then again, as long as the field reference and array syntax are so similar, there will probably be more cases like this one.

[TheVastyDeep]

Appears to be the same as 5591 to me.

[Zeal0us]

This also causes :exception=>#[]>, I assume from FileReference.java if you use a 1 item array as a "path" for the file input plugin.

Edit: Nevermind, I think this was something else, it seems to be happening again. Probably unrelated?
Edit 2: Looks like it is related, I just had to clear out the queued data.

[JohnLyman]

At very least, this should be documented here. It's not unreasonable to expect in to match single element arrays, and it is very confusing and difficult to debug when it does not.

[NotAProfessionalDeveloper]

Just wanted to let y'all know that this has wasted at least 45 minutes of my life :neutral_face:

[seadub]

Chiming in, this issue has wasted a solid week of time for my team.

[lhzw]

Oh dear, waste 2 hours for me, what a shame.