Consul: Consul UI makes invalid requests when Connect is disabled

[vassilvk]

Overview of the Issue

When deploying a Consul server with disabled Connect, navigating to a service's page in Consul UI leads to HTTP 500 on the server side with an error indicating that Connect must be enabled in order to access said endpoint.

Reproduction Steps

1. Start up a Consul agent as a server
2. Make sure Connect is disabled for the Consul server
3. Make sure Consul UI is enabled
4. Open up a browser and navigate to the Consul UI endpoint at /ui/dc1/services/consul
5. Observe the following error in the Consul agent logs (Actual IP replaced with xxx below):

agent.http: Request error: method=GET url=/v1/discovery-chain/consul?dc=dc1 from=xxx.xxx.xxx.xxx:50354 error="rpc error making call: Connect must be enabled in order to use this endpoint"

Consul info for both Client and Server

Client info

agent:
        check_monitors = 0
        check_ttls = 1
        checks = 5
        services = 5
build:
        prerelease =
        revision = 8b4a3d95
        version = 1.7.3
consul:
        acl = disabled
        known_servers = 3
        server = false
runtime:
        arch = amd64
        cpu_count = 2
        goroutines = 163
        max_procs = 2
        os = linux
        version = go1.13.7
serf_lan:
        coordinate_resets = 0
        encrypted = false
        event_queue = 0
        event_time = 3
        failed = 0
        health_score = 0
        intent_queue = 0
        left = 0
        member_time = 96
        members = 4
        query_queue = 0
        query_time = 1

Server info

agent:
        check_monitors = 0
        check_ttls = 0
        checks = 0
        services = 0
build:
        prerelease =
        revision = 8b4a3d95
        version = 1.7.3
consul:
        acl = disabled
        bootstrap = false
        known_datacenters = 1
        leader = false
        leader_addr = 10.1.1.29:8300
        server = true
raft:
        applied_index = 29000
        commit_index = 29000
        fsm_pending = 0
        last_contact = 37.640092ms
        last_log_index = 29000
        last_log_term = 97
        last_snapshot_index = 16390
        last_snapshot_term = 54
        latest_configuration = [{Suffrage:Voter ID:729b7718-3a9a-95b8-f1fd-3a59f54c407d Address:10.1.1.29:8300} {Suffrage:Voter ID:2cd025ca-771a-4311-e873-ae699816d4f9 Address:10.1.1.30:8300} {Suffrage:Voter ID:30cfa0ae-1f1f-67a8-5470-137ce45ae193 Address:10.1.1.26:8300}]
        latest_configuration_index = 0
        num_peers = 2
        protocol_version = 3
        protocol_version_max = 3
        protocol_version_min = 0
        snapshot_version_max = 1
        snapshot_version_min = 0
        state = Follower
        term = 97
runtime:
        arch = amd64
        cpu_count = 2
        goroutines = 148
        max_procs = 2
        os = linux
        version = go1.13.7
serf_lan:
        coordinate_resets = 0
        encrypted = false
        event_queue = 0
        event_time = 3
        failed = 0
        health_score = 0
        intent_queue = 0
        left = 0
        member_time = 96
        members = 4
        query_queue = 0
        query_time = 1
serf_wan:
        coordinate_resets = 0
        encrypted = false
        event_queue = 0
        event_time = 1
        failed = 0
        health_score = 0
        intent_queue = 0
        left = 0
        member_time = 72
        members = 3
        query_queue = 0
        query_time = 1

Operating system and Environment details

Docker 19.03.3 running on Ubuntu 18.04.

Log Fragments

    2020-06-05T20:00:57.183Z [ERROR] agent.client: RPC failed to server: method=DiscoveryChain.Get server=xxx.xxx.xxx.xxx:8300 error="rpc error making call: Connect must be enabled in order to use this endpoint"
    2020-06-05T20:00:57.183Z [ERROR] agent.http: Request error: method=GET url=/v1/discovery-chain/consul?dc=dc1 from=xxx.xxx.xxx.xxx:50354 error="rpc error making call: Connect must be enabled in order to use this endpoint"

[mkeeler]

@johncowen Did you recently add something to the UI to prevent using Connect functionality when Connect is disabled?

[johncowen]

Hey @vassilvk !

A fix for this was added in Consul 1.7.3 could you possibly double check the version of the UI you are seeing this in? Incase you missed it, the UI version is in the footer of the UI.

Thanks,

[johncowen]

Edit: I've just sanity checked this as much as I can and the issue that was originally reported mentions the fix being added in 1.7.1, I've added here for further info/issue linking:

https://github.com/hashicorp/consul/issues/7288
https://github.com/hashicorp/consul/pull/7291

All in all, it would be great if you could confirm the UI version where you are seeing this, I'm intrigued now myself!

[mnuic]

Hi everyone,

We are on v1.7.3 and seeing the same error. When on consul UI, clicking on the service name immediately shows error:

Jun  9 12:12:16 NODE consul[32461]:     2020-06-09T12:12:16.473+0200 [ERROR] agent.http: Request error: method=GET url=/v1/discovery-chain/SERVICE1?dc=NODE from=A.B.C.D:54437 error="Connect must be enabled in order to use this endpoint"
Jun  9 12:12:26 NODE consul[32461]:     2020-06-09T12:12:26.341+0200 [ERROR] agent.http: Request error: method=GET url=/v1/discovery-chain/SERVICE2?dc=NODE from=A.B.C.D:54434 error="Connect must be enabled in order to use this endpoint"
Jun  9 12:12:30 NODE consul[32461]:     2020-06-09T12:12:30.889+0200 [ERROR] agent.http: Request error: method=GET url=/v1/discovery-chain/SERVICE3?dc=NODE from=A.B.C.D:54524 error="Connect must be enabled in order to use this endpoint"

Connect is disabled.

[johncowen]

Hi @mnuic

Thanks for the extra info here.

Currently the UI will make the discovery chain request and fail gracefully if it receives a 500 (ie. a 500 = connect is disabled), the UI should still function correctly. Does the UI still function correctly in your case?

Unfortunately you will see these errors in your Consul logs until we decide on a better way in the UI to detect when connect is disabled and therefore not make that extra request at all.

Let me know.

Thanks,

[mnuic]

Hi @johncowen

The UI functions correctly, I can confirm that, only that logs are showing errors, but that's expected as I read from #7288.

Thanks

[vassilvk]

Thank you @johncowen

Unfortunately you will see these errors in your Consul logs until we decide on a better way in the UI to detect when connect is disabled and therefore not make that extra request at all.

Will you track the above in another issue?

[johncowen]

Hey @vassilvk

When you mentioned I was about to make another issue, but then I figured we could keep this one here open? It gives a good explanation of the issue incase others encounter it. Does that sound good?

Thanks,

[vassilvk]

@johncowen - sounds good, thanks 👍

[johncowen]

Thanks @vassilvk I've PRed a potential improvement above ^. This will still produce one error log per reload of the entire UI, but thats an improvement on one error log every time you navigate to a service.

I'm aiming to get that in our next release.

Longer term I've spoken to folks here about some sort of solution which means we can check these things per DC without relying on errors etc, so this is likely to be further improved upon at a later date.