Logstash: tcp output plugin does not send newlines

[whizz]

The tcp output plugin is supposed to send events separated by newline, but the newline is not there. I tested using nc.

Logstash config:

input {
  file {
    path => "/tmp/in"
  }
}
output {
  tcp {
    host => "127.0.0.1"
    port => 4322
  }
}

I tried posting the events like this:

# echo test1 >> /tmp/in
# echo test2 >> /tmp/in

In other console I ran nc, the events arrived like this:

$ nc -l 4322
{"message":"test1","@version":"1","@timestamp":"2014-08-20T13:13:18.281Z","host":"0.0.0.0","path":"/tmp/in"}{"message":"test2","@version":"1","@timestamp":"2014-08-20T13:13:31.301Z","host":"0.0.0.0","path":"/tmp/in"}

No newline between them.

[jordansissel]

This is a bug in the default configuration. For now, you can work around it by setting codec => json_lines

output {
  tcp {
    codec => json_lines
    host => "127.0.0.1"
    port => 4322
  }
}

[jordansissel]

This is a bug in the default configuration. For now, you can work around it by setting codec => json_lines

output {
  tcp {
    codec => json_lines
    host => "127.0.0.1"
    port => 4322
  }
}

[jordansissel]

For fixing the code, we should make the tcp output use fix_streaming_codecs like the tcp and stdin inputs do.

[whizz]

Thanks very much, the workaround works.

[mrmanc]

We encountered this issue using the tcp output with a plain codec to send to Graphite. Using the graphite codec got us past the issue although it didn’t allow us to carry along a timestamp from the input.

I wasn’t aware of the line codec until I searched the code for ‘fix_streaming_codecs’, and having switched to that everything still works :+1:

Here is our new configuration:

output {
  stdout {}
  tcp {
    codec => line { format => "%{metric_name} %{metric_value} %{timestamp}" }
  }
}

[jordansissel]

I think this is fixed in 1.5 (smarter selection of line-oriented codec for tcp plugins) and a workaround of forcing the codec to json_lines is working.

[phil-lavin]

FYI, not fixed in 1.5.0. Workaround still works, though.

[Signus]

I will make note that I have experienced this very same problem in both Logstash 2.3.4 and 5.0.0-alpha4 with logstash-output-tcp 3.2.0. It is still required to use codec => "json_lines"

I have multiple messages going through:

{
       "message" => "some_field=1234 testing_field=testing14",
      "@version" => "1",
    "@timestamp" => 2016-08-08T08:01:11.532Z,
          "host" => "host",
          "port" => 51901
}
{
       "message" => "some_field=1234 testing_field=testing15",
      "@version" => "1",
    "@timestamp" => 2016-08-08T08:01:20.532Z,
          "host" => "host",
          "port" => 51901
}
{
       "message" => "some_field=1234 testing_field=testing16",
      "@version" => "1",
    "@timestamp" => 2016-08-08T08:01:41.532Z,
          "host" => "host",
          "port" => 51901
}

Configuration:

output {
    tcp {
        #codec => "json_lines"
        host => "host"
        port => portnum
    }
}

Result on secondary Logstash host:

{
    "message" => "{\"message\":\"some_field=1234 testing_field=testing14\",\"@version\":\"1\",\"@timestamp\":\"2016-08-08T08:01:11.532Z\",\"host\":\"host\",\"port\":51901}{\"message\":\"some_field=1234 testing_field=testing15\",\"@version\":\"1\",\"@timestamp\":\"2016-08-08T08:01:20.532Z\",\"host\":\"host\",\"port\":51901}{\"message\":\"some_field=1234 testing_field=testing16\",\"@version\":\"1\",\"@timestamp\":\"2016-08-08T08:01:31.532Z\",\"host\":\"host\",\"port\":51901}
    "@version" => "1",
    "@timestamp" => "2016-08-08T17:20:21.699Z",
    "host" => "host",
    "port" => 43182,
    "type" => "sometype"
}

After codec => "json_lines" is added, this is no longer a problem.

Is this supposedly fixed or is the workaround considered to be sufficient?

[brettinman]

I had this issue today, shipping from one logstash agent to another via tcp output/input.

This output:
output { tcp { host => "hostname.fqdn" port => 7002 } }

This input:
input { tcp { port => 7002 tags => [ "sometags" ] } }

Resulted in nothing. Adding codec => "json_lines" to the output and I was getting messages. Hope this helps someone else. For what it's worth, the input to the first one was from the GELF plugin, default codec "plain".

Logstash version 2.4.1.

[ayalaio]

I found for me using json_lines works, but not quite well.
Instead I use line. But on both, the tcp input and tcp output.
Using json_lines will escape double quotes. It does not happen with lines

    tcp {
        codec => "line"
        host => "host"
        port => portnum
    }

[okazymyrov]

The same issue in logstash 5.4.3.

[wadejensen]

I've been able to work around this issue with the json_lines codec. https://www.elastic.co/guide/en/logstash/current/plugins-codecs-json_lines.html

My use case is a bit strange: I'm sending events from one logstash to another over tcp.

Sample config:
Logstash 1

input {
  beats {
    port => 5044
  }
filter {
  # filtery stuff
}

output {
  tcp {
      host => "myhostname@someserver"
      port => 5044 
      codec => json_lines
    }
}

Logstash 2

input {
  tcp {
    port => 5044
    codec => json_lines  # codec => json also works, but no codec at all will result in your message field being its own json string which is nasty
  }
}

filter {
  # moar filtery stuff
}

output {
  stdout{
     codec => json_lines # for debugging
  }
  file {
    path => "/my/output/path"
    codec => json_lines   # can be whatever you want ( I'm reading the json in Spark and therefore newline delimiting makes life easy)
 }
}

Curiously enough, input configurations using tcp with standard json codec are automatically switched to the json_lines codec in at least logstash 5.4.x (I suspect someone at Elastic noticed and decided to kind of help out without adding docs)
Logstash 2 stdout

[2017-07-05T12:14:54,986][INFO ][logstash.pipeline        ] Starting pipeline {"id"=>"main", "pipeline.workers"=>5, "pipeline.batch.size"=>250, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>1250}
[2017-07-05T12:14:54,990][INFO ][logstash.inputs.tcp      ] Automatically switching from json to json_lines codec {:plugin=>"tcp"}
[2017-07-05T12:14:54,993][INFO ][logstash.inputs.tcp      ] Starting tcp input listener {:address=>"0.0.0.0:5044"}
[2017-07-05T12:14:54,994][INFO ][logstash.pipeline        ] Pipeline main started

Unfortunately I also needed json_lines on the tcp output, and that doesn't seem to be switched automagically, so I had to guess that part.