Logstash: Ruby event API does not work with non-ASCII keys

[maxirmx]

I reproduced it for 6.2 and 6.3
It looks like this problem exists for 5.x and 6.x versions

Test config

input { stdin { } }

filter {
mutate {
add_field => { "Русское название" => "Content of the field" }
}
date {
match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
}
}
output {
stdout { codec => rubydebug }
}

Output

[2018-06-22T18:15:11,760][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"C:/maxirmx/logstash/modules/fb_apache/configuration"}
[2018-06-22T18:15:11,791][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"C:/maxirmx/logstash/modules/netflow/configuration"}
[2018-06-22T18:15:12,094][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2018-06-22T18:15:13,047][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.2.4"}
[2018-06-22T18:15:13,797][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2018-06-22T18:15:19,885][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2018-06-22T18:15:20,119][INFO ][logstash.pipeline ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#"}
The stdin plugin is now waiting for input:
[2018-06-22T18:15:20,260][INFO ][logstash.agent ] Pipelines running {:count=>1, :pipelines=>["main"]}
Some data
{
"message" => "Some data\r",
"@version" => "1",
"@timestamp" => 2018-06-22T15:15:28.309Z,
"host" => "NSC181278",
"? ?\u0083?\u0081?\u0081?????? ???°?·???°??????" => "Content of the field"
}

* Unit test that will fail *
The tests below go to _logstash-core/spec/logstash/event_spec.rb_
Ruby API fails but Java API works OK

describe "#event-key-non-ASCII" do
it "should set and get values for non-ASCII keys" do
e = LogStash::Event.new()
expect(e.set("фуу", "bar")).to eq("bar")
expect(e.get("фуу")).to eq("bar")

 e = LogStash::Event.new({"фуу" => "test"})
 expect(e.set("фуу", "bar")).to eq("bar")
 expect(e.get("фуу")).to eq("bar")

end

it "should set and get values for non-ASCII keys through java APIs" do
e = LogStash::Event.new()
e.to_java.setField("фуу", "bar")
expect(e.to_java.getField("фуу")).to eq("bar")
end

[maxirmx]

The problem is in _JrubyEventExtLibrary.java_

   @JRubyMethod(name = "get", required = 1)
   public IRubyObject ruby_get_field(ThreadContext context, RubyString reference)
   {
       return Rubyfier.deep(
           context.runtime,
          this.event.getUnconvertedField(FieldReference.from(reference.getByteList()))
      );
 }

_reference.getByteList()_ is not correct above. It shall rather be _reference.getValue()_
It shall be fixed for _set_, _get_ and _remove_

[jsvd]

this is strange, I cannot reproduce on 6.3.0:

/tmp/logstash-6.3.0 % bin/logstash -i irb
2.3.0 :001 > e = LogStash::Event.new()
 => #<LogStash::Event:0x27ecdcf8> 
2.3.0 :002 > e.set("言葉", "word") == e.get("言葉")
 => true 
2.3.0 :003 > e.set("фуу", "bar") == e.get("фуу")
 => true 

[jsvd]

nevermind, confirmed:

2.3.0 :007 > e = LogStash::Event.new()
 => #<LogStash::Event:0x506794a9> 
2.3.0 :008 > e.set("фуу", "bar")
 => "bar" 
2.3.0 :009 > e.to_json
 => "{\"@timestamp\":\"2018-06-27T09:50:48.202Z\",\"Ñ\u0084Ñ\u0083Ñ\u0083\":\"bar\",\"@version\":\"1\"}" 

[maxirmx]

yep
In the first example it was nil on both sides of '=='

[maxirmx]

btw fix for 6.2 is here
https://github.com/maxirmx/logstash/tree/6.2

[jsvd]

@maxirmx do you mind submitting a PR with the fix and the specs you wrote?

[maxirmx]

Sure

[danhermann]

This is fixed in #9825 and #9821.