Fluentd: incomming chung is broken, warning

[epcim]

Infrastructure, environment information, e.g. OS.
td-agent-bit --> td-agent(in k8s env)

Fluentd or td-agent version.
0.14.22 - fluentd
0.12.7 -fluentbit

Your configuration
(i may post, but in_tail on fluentbit is reading openstack logs and forwards them to fluentd)

Your problem explanation
Some recorrds are receaved by fluentd and processed but there is huge number of [input_forward_24224] incoming chunk is broken errors in fluend log.

Full log line

2017-11-09 15:01:27 +0000 [warn]: #0 [input_forward_24224] incoming chunk is broken: host="name resolusion failed" msg=92

Related:

1. https://github.com/fluent/fluentd/issues/660 (already spotted back in 2015, quite deep analyse)
2. https://github.com/fluent/fluent-bit/issues/415 (as I thought it may escape issue, send error on fluentbit)
3. https://github.com/fluent/fluentd/blob/3f8630d082c2fe9d02277fd63f5da59dbbe4779e/lib/fluent/plugin/in_forward.rb#L275

Would it be possible to better log where/what is affected. Also, what happens's to this log record. Is it totally skipped or it's partially received and processed? If that is an parsing issue (as #660 proved) (json on input) how that could be treated (in order to capture correct start of the message and skip just one failed, log it properly so we know whether the record is messed when sent or it's parsing issue only at fluentd in_forward?

[repeatedly]

Does this happen with deny_keepalive false in in_forward?

Is it totally skipped or it's partially received and processed?

The problem is msgpack's unpacker receives broken msgpack stream from socket.
Do you hit this problem with low traffic?

[japzio]

hi. experience this from a network load balancer health check on aws running td-agent on a container. any way to suppress this? cant find it on docs :-(

-- update --
@log_level error on in_forward and should be deny_keepalive false for the health check to succeed

[ganmacs]

it's a stale issue. I'm closing. if you have any problem, updating fluentd might help it.