Nomad: Nomad UI gives errors when watching logs

[kaspergrubbe]

Nomad version

$ nomad version
Nomad v0.8.7 (21a2d93eecf018ad2209a5eab6aae6c359267933+CHANGES)

Operating system and Environment details

Amazon Linux 2.

Issue

When I go to an allocation and click logs nothing shows up, but after a while this shows up:

My browser console shows me this:

LOG FETCH: Couldn't connect to /v1/client/fs/logs/03758951-bfe4-5457-f804-d82554109b99?follow=true&offset=50000&origin=end&task=whoami&type=stdout

However, if I in the interface clicks on Head or Tail the logs will show up:

Also, nomad alloc logs :alloc_id work just fine:

$ nomad alloc logs 03758951
Listening on :8000
I'm 8b845094ad9d
I'm 8b845094ad9d
I'm 8b845094ad9d
I'm 8b845094ad9d

Here is an small video of the problem:

Nomad Server logs (if appropriate)

I haven't seen anything related to these errors.

Nomad Client logs (if appropriate)

I haven't seen anything related to these errors.

[robloxrob]

We have seen the same issue since upgrading to 0.8.7 (CentOS 7.5)

[robloxrob]

In our case we are behind Nginx. To fix the issue we disabled proxy_buffering off; in our Nginx config and the problem went away.

[kaspergrubbe]

@robloxrob Thats a nice fix.

Sadly, we run Fabio in front, so no Nginx, but it might be possible that there are some tweaks that can be applied.

[daktari]

@robloxrob Thats a nice fix.

Sadly, we run Fabio in front, so no Nginx, but it might be possible that there are some tweaks that can be applied.

We have the same problem with Fabio 1.5.9 and Nomad 0.8.4

Maybe is related with https://github.com/fabiolb/fabio/issues/324.

Request:

Request URL: http://localhost:9999/v1/client/fs/logs/6da04ba5-b143-49b8-9460-c8fc8616823c?follow=true&offset=50000&origin=end&task=redis&type=stdout
Accept: */*
Accept-Encoding: gzip, deflate, br
Cache-Control: no-cache
Connection: keep-alive
Host: localhost:9999
Referer: http://localhost:9999/ui/allocations/6da04ba5-b143-49b8-9460-c8fc8616823c/redis/logs

When there are many logs (>BufferSize) everything is Ok.

[kaspergrubbe]

So we upgraded to Nomad 0.9.1 and the problem persists, which we expected since it is probably a Fabio thing, we haven't yet decided if we want to continue with Fabio (their newest release broke something for us that we haven't had the time to debug) or if we are switching to pure Haproxy and live without the cool Consul bindings that Fabio gives us.

[tommyvicananza]

So we upgraded to Nomad 0.9.1 and the problem persists, which we expected since it is probably a Fabio thing, we haven't yet decided if we want to continue with Fabio (their newest release broke something for us that we haven't had the time to debug) or if we are switching to pure Haproxy and live without the cool Consul bindings that Fabio gives us.

@kaspergrubbe maybe you can use this it solves our problem. What we did it's to add a Target option which adds flush to desired route.

E.g:
route add nomad nomad.tommyvicananza.io http://192.168.50.4:4646 opts "flush=-1s"

Hope that can help you

[kaspergrubbe]

@tommyvicananza Thank you for the workaround, that did actually solve it for us.

The fix solved it for us while we migrated away from Fabio, and we have now replaced it with Haproxy and aren't seeing any issues.

[mylesw42]

I'm experiencing the same thing, except I'm configured for TLS and on v0.9.3. I'm not fronting the Nomad UI with any proxy, I connect to directly.

I see the same "Cannot fetch logs" error, except the logs will never show in the window.

I'm also seeing this in the nomad server logs:

Jun 27 17:43:17 q9icajnomad01 nomad[92505]: http: request failed: method=GET path=/v1/client/fs/logs/2afbba45-4fb0-f5b5-85a3-d3bfc2037f06?follow=true&offset=50000&origin=end&task=nginx&type=stdout error="only encoded map or array can be decoded into a struct" code=500
Jun 27 17:43:17 q9icajnomad01 nomad[92505]: 2019-06-27T17:43:17.135-0400 [ERROR] http: request failed: method=GET path=/v1/client/fs/logs/2afbba45-4fb0-f5b5-85a3-d3bfc2037f06?follow=true&offset=50000&origin=end&task=nginx&type=stdout error="only encoded map or array can be decoded into a struct" code=500

And this in the nomad UI console:

LOG FETCH: Couldn't connect to //10.103.28.55:4646/v1/client/fs/logs/2afbba45-4fb0-f5b5-85a3-d3bfc2037f06?follow=true&offset=50000&origin=end&task=nginx&type=stdout
LOG FETCH: Couldn't connect to /v1/client/fs/logs/2afbba45-4fb0-f5b5-85a3-d3bfc2037f06?follow=true&offset=50000&origin=end&task=nginx&type=stdout

This particular task has been running for a few hours, and just now after clicking back on the Logs tab, the logs loaded properly. I'm at a loss to find any consistent behaviour :/

The "nomad alloc logs " command works all the time.

When I disable TLS on the cluster, everything works in the UI.

[jacksod1]

I also get the same error with Nomad version 0.9.3. We have put Traefik in front on Nomad, however I get this error when accessing Nomad via the load balanced address and when accessing the Web UI directly over port 4646. We also have TLS enabled on the cluster.

[mylesw42]

I also get the same error with Nomad version 0.9.3. We have put Traefik in front on Nomad, however I get this error when accessing Nomad via the load balanced address and when accessing the Web UI directly over port 4646. We also have TLS enabled on the cluster.

I believe this issue is being resolved through [GH-5954] for the upcoming 0.9.4 release.