Telegraf: POST request to http_listener fails with EOF

[salarali]

Bug report

Intermittently the post request to the http_listener fails and I get the following error message:
Post http://localhost:8186/write: EOF

Relevant telegraf.conf:

Vanilla conf. with http_listener enabled.

[sparrc]

what version are you using? do you have any steps to reproduce? can you provide more log context and the full log line?

[salarali]

Sorry for the lack of information.

Telegraf version: 1.1.2

I am making the http post request from a golang program.

resp, err := http.Post(API_URL, "", bytes.NewBuffer([]byte(metricsString)))

if err != nil || resp.StatusCode != 204 {
    logger.Error("Telegraf push failed: ", err)
} else {
    defer resp.Body.Close()
}

metricsString looks like this:

test num_error_no_category=122 1484680136892545372
test num_error_http_setup=0 1484680136892545372
test num_active_go_routines=2045 1484680136892545372
test num_error=1803 1484680136892545372
test num_duplicate=760 1484680136892545372

And this is all the log that go provides to me. Is there a way I can get more information?

I have turned debug logging on for telegraf but I do not see any errors there.

I am sending metrics to telegraf every 10 seconds. Currently, I have only experienced this error on production. I will try to reproduce the error locally.

[danielnelson]

@salarali Any chance you have found a way to reproduce this?

[ncode]

I am receiving the same error, but it only happens with 21+ Goroutines pushing metrics to telegraf. Also It never happens if you use the discard as output.

[danielnelson]

Possibly related to #2919?

[jpereira]

I am experiencing the same issue in my setup.

2017-09-29T17:53:01Z E! InfluxDB Output Error: Post https://metrics.internal.lan:443/write?db=telegraf&rp=default: EOF

[danielnelson]

@jpereira Just to clarify, is the error when you are sending to the http_listener in Telegraf, or when you are sending to InfluxDB?

[jpereira]

@danielegozzi Sending to the InfluxDB Cluster.

[danielnelson]

Okay, this is a different issue. It indicates that InfluxDB closed the connection, I would check the InfluxDB logs to see if anything is reported.

[jpereira]

@danielegozzi I am using the latest version 1.4.1

[ILoGM]

We're experiencing the same issue as @salarali did.
We have erlang app which sends metrics to telegraf via http locally. Sometimes it looks like http_listener just closes the connection. Just like that:

2018-04-17 16:08:17.147 Undefined [error] influx_client: HTTP sending error: {error,closed}
2018-04-17 16:08:17.147 Undefined [warning] influx_client: Reconnecting success: {http,<<"127.0.0.1">>,8186,undefined,undefined}

Telegraf debug logs say nothing useful about what happens. I'd rather see some info about max_body_size/max_line_size exceeding or something like that.

Version used:

telegraf --version
Telegraf v1.5.3 (git: release-1.5 1e519698)

We're still investigating.

[ILoGM]

We figured out that we exceeded max_line_size.
I set it as twice as default value and disconnects disappeared.
But still, I believe that such kind of situations must be mentioned in logs, at least at debug level.

[danielnelson]

Thanks for debugging this @ILoGM, I think we should return 413 Request Entity Too Large.

[russorat]

let's look at a refactor of the entire http_listener code for the next release.