Consul: Consul UI send 404 with "The backend responded with an error" with no errors in log-file

[kaspergrubbe]

I run my Consul cluster like this:

{
  "server": true,
  "datacenter": "eu-aws",
  "encrypt": "abcdef==",
  "enable_script_checks": true,
  "leave_on_terminate": true,
  "rejoin_after_leave": true,
  "ui": true,
  "bootstrap_expect": 3,
  "verify_outgoing": true,
  "verify_incoming": true,
  "data_dir": "/opt/consul",
  "ca_file": "/etc/consul.d/ssl/ca.pem",
  "cert_file": "/etc/consul.d/ssl/server.pem",
  "key_file": "/etc/consul.d/ssl/server.key",
  "connect": {
    "enabled": true
  },
  "retry_join": ["provider=aws tag_key=role tag_value=consulserver"]
}

And I run Nginx in front on port 8585:

user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;

events {
    worker_connections 1024;
}

http {
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile            on;
    tcp_nopush          on;
    tcp_nodelay         on;
    keepalive_timeout   65;
    types_hash_max_size 2048;

    include             /etc/nginx/mime.types;
    default_type        application/octet-stream;

    # Load modular configuration files from the /etc/nginx/conf.d directory.
    # See http://nginx.org/en/docs/ngx_core_module.html#include
    # for more information.
    include /etc/nginx/conf.d/*.conf;

    server {
    listen       8585 default_server;
        listen       [::]:8585 default_server;
        server_name  _;
        root         /usr/share/nginx/html;

        location /ui {
          proxy_pass http://127.0.0.1:8500/ui;
          proxy_buffering off;
        }
    }
}

But when I hit http://servername:8585 I get the following:

I have tried turning debugging on by adding "log_level": "debug", to the JSON configuration and restarting it, however there are no log-lines telling me what is failing in the ui when I use consul monitor.

Are there any way to figure out what is making the UI fail so I can debug further?

[pierresouchay]

The UI needs other paths than /ui (/v1 for instance), so you have to forward more paths than just /ui

[johncowen]

Hi @kaspergrubbe ,

Just to confirm what Pierre says here. I think that even after this, proxying the UI on anything but /ui won't work. I believe it's a mix between how our frontend framework works plus some extra things to take into consideration on how we serve the UI on the backend (I'll try and find the relevant issue shortly and post here).

Apart from this, right now I believe we wouldn't support running the UI on anything other than /ui because this is what we test everything against. Although @noelledaley on our team is currently looking into how we can standardise testing things in our UI products across other environments/setups (through reverse proxies, embedded UI, and of course further cross browser testing).

This work could possibly lead us to being able to support running our UI on things like /consul/, which would be great as I've seen similar 'wants' for this from the community.

I'll try and keep you posted.

[johncowen]

Here's the relevant issue:

https://github.com/hashicorp/consul/issues/1930

[kaspergrubbe]

The UI needs other paths than /ui (/v1 for instance), so you have to forward more paths than just /ui

Aha! That totally makes sense, and changing my Nginx config to this worked:

        location / {
          proxy_pass http://127.0.0.1:8500;
          proxy_buffering off;
        }

Thanks a bunch!

Apart from this, right now I believe we wouldn't support running the UI on anything other than /ui because this is what we test everything against. Although @noelledaley on our team is currently looking into how we can standardise testing things in our UI products across other environments/setups (through reverse proxies, embedded UI, and of course further cross browser testing).

That sounds great, would that include making it easier to expose the http ui? It was very easy with Nomad, but I couldn't get Consul to play ball to listen on 0.0.0.0, so instead I just threw Nginx in front.

Thanks for your answers!

[johncowen]

Hey @kaspergrubbe !

Glad we were helpful.

If your are still having trouble exposing the UI maybe trying the mailing list? https://www.consul.io/community.html (also make sure you are specifying using -client not -bind)

Generally putting a proxy in front of consul is not a bad plan, as you should always be extra careful about exposing things publicly, but the mailing list is a good place to ask and check about these things.

Please shout if you hit anything else

Thanks,

John