Logstash: How to test filter?

[tofdragon]

how to unit test filter?
version:6.4.2

I have seen it on the Internet, rspec can be unit tested, but the bin in the window's decompression package can't find the rspec file, install the ruby environment, but can't find logstash/devutils/rspec/spec_helper,

demo:https://gquintana.github.io/2016/09/07/Testing-Logstash-configuration.html

View the official documentation, did not see where there is relevant guidance

[guyboertje]

@tofdragon
I presume you are not talking about code testing along the lines of RSpec or JUnit.
I usually create a simple config using the generator input and the stdout output, e.g.:

input {
  generator {
    message => '{"environment":"urban","envType":"outdoor","positionNumber":1,"frameIndex":15,"fcntUp":22324,"numberOfGateways":3,"numberOfTimestamps":0,"serverTimestamp":"23:44:36","lnsAppEui":"4883C7DF30040000","lnsDevEui":"4883C7DF3004223A","lnsNetId":"000007","lnsServerTimestamp":"27/10/2017 23:44:36","gtwFrequency":868.1,"gtwDataRate":"SF10_BW_125","estimatedLongitudeWGS84":null,"estimatedLatitudeWGS84":null,"deviceLongitudeWGS84":2.289511063734628,"deviceLatitudeWGS84":48.88373587201055,"spreadingFactor":10,"accuracy":null,"estimatedAccuracy":null,"averageISD":null,"hdopDevice":null,"hdopEstimated":null,"gateways":[{"antennaLongitude":2.2957080835783388,"rssi":-107.7376019773414,"rssiStandardDeviation":-95.0,"elapsedTimeSince1PPS":0.0,"antennaID":"1","antennaLatitude":48.89112433031253,"gatewayTimestamp":"27/10/2017 23:44:31","snr":-12.5,"frequencyOffset":0,"lnsServerTimestamp":"27/10/2017 23:44:36","rssiSignal":-95.0,"gatewayID":"M15279"},{"antennaLongitude":2.2932473657963164,"rssi":-107.1244260279434,"rssiStandardDeviation":-103.0,"elapsedTimeSince1PPS":0.0,"antennaID":"1","antennaLatitude":48.88193968215203,"gatewayTimestamp":"27/10/2017 23:44:32","snr":-2.0,"frequencyOffset":0,"lnsServerTimestamp":"27/10/2017 23:44:36","rssiSignal":-103.0,"gatewayID":"M15073"},{"antennaLongitude":2.3014574627818787,"rssi":-104.59612087980607,"rssiStandardDeviation":-94.0,"elapsedTimeSince1PPS":0.0,"antennaID":"1","antennaLatitude":48.87684647097623,"gatewayTimestamp":"27/10/2017 23:44:32","snr":-10.2,"frequencyOffset":0,"lnsServerTimestamp":"27/10/2017 23:44:36","rssiSignal":-94.0,"gatewayID":"M15913"}],"loraDevAddr":"0F127D87"}'
    count => 1
  }
}

filter {
  json {
    source => "message"
  }
  ruby {
    code => '
      gateways_size = event.get("[gateways]").size
      gateways_size.times do |index|
        event.set("[gateways][#{index}][position][lat]", event.get("[gateways][#{index}][antennaLatitude]"))
        event.set("[gateways][#{index}][position][lon]", event.get("[gateways][#{index}][antennaLongitude]"))
      end
    '
  }
}

output {
  stdout { codec => rubydebug }
}

Above, I am testing/developing the ruby filter code. It is sometimes convenient to use the json filter to create a structured event before the filter of interest.

input {
  generator {
    lines => [
      '10.2.3.40;from-P2;22.95;101',
      '10.2.3.20;from-P2;22.95;100',
      '10.2.3.30;from-P2;22.95;101'
    ]
    count => 1
  }
}

filter {
  dissect {
    mapping => {
      "message" => "%{from_ip};%{app};%{amount};%{loggedin_userid}"
    }
    convert_datatype => {
      "amount" => "float"
      "loggedin_userid" => "int"
    }
  }
  jdbc_streaming {
    statement => "select descr as description from ref.local_ips where ip = :substitute"
    parameters => {substitute => "[from_ip]"}
    target => "server"
    jdbc_user => "logstash"
    jdbc_password => "???"
    jdbc_driver_class => "org.postgresql.Driver"
    jdbc_driver_library => "/elastic/tmp/postgresql-42.1.4.jar"
    jdbc_connection_string => "jdbc:postgresql://localhost:5432/ls_test_2"
  }
}

output {
  stdout {
    codec => rubydebug {metadata => true}
  }
}

Above, I am testing jdbc_streaming but also using dissect to create the structured event.

input {
  generator {
    lines => [
      '<01>- localhost {"foo": 11, "bar": 12}',
      '<13>Oct 11 17:46:22 10.55.1.11 AgentDevice=WindowsLog   PluginVersion=7.2.8.91'
    ]
    count => 1
  }
}

filter {
  grok {
    match => {
      "message" => [
        '^<%{INT:priority:int}>- %{DATA:ip_or_host} %{DATA:kv_or_json}$',
        '^<%{INT:priority:int}>%{SYSLOGTIMESTAMP:timestamp} %{IPORHOST:ip_or_host} %{DATA:kv_or_json}$'
      ]
    }
    break_on_match => true
  }
  if [kv_or_json] =~ /^\{/ {
    json {
      source => '[kv_or_json]'
    }
  } else {
    kv {
     source => "[kv_or_json]"
    }
  }
}

output {
  stdout {
    codec => rubydebug
  }
}

Above, I am working through grok patterns to parse two very different line formats in a combined logging scenario.

[LanceGG]

If you want to test grok match patterns, you can find a website to test your formats on website.

If you got the right pattern, the result will appear

[tofdragon]

@LanceGG grok match Just a test of the grok, for a complete configuration, unit tests are definitely better

[tofdragon]

@guyboertje Your code is a test, but you have to go to the console manually, because I am not familiar with ruby, good unit tests should be self-verifying.

[tofdragon]

https://www.elastic.co/blog/logstash-functionality-through-testing
but window run error

no such file to load -- logstash/devutils/rspec/spec_helper

demo:https://gquintana.github.io/2016/09/07/Testing-Logstash-configuration.html

The problem has been solved. The version 6.4.2 has no rspec and rspec.bat files in the bin directory of the installation package downloaded from https://www.elastic.co/downloads/logstash. In the github source repository, download logstash containing rspec and Rspec.bat, decompress it and copy it to bin, you can run unit test normally.