Sensu-go: cluster health throws error with misconfigured cluster

[treydock]

Expected Behavior

I'd expect sensuctl cluster health to handle a misconfigured cluster of non-existence nodes in initial-cluster.

Current Behavior

[root@sensu-backend-peer1 ~]# sensuctl cluster health
Error: json: cannot unmarshal object into Go struct field ClusterHealth.Err of type error

Possible Solution

Steps to Reproduce (for bugs)

Configure 2 node cluster with a 3rd non-existent node in initial-cluster and then run sensuctl cluster health. I think that's how I produced the error.

YAML used where backend3 doesn't exist.

---
listen-client-urls: http://0.0.0.0:2379
listen-peer-urls: http://0.0.0.0:2380
initial-cluster: backend1=http://192.168.52.21:2380,backend2=http://192.168.52.22:2380,backend3=http://192.168.52.26:2380
initial-advertise-peer-urls: http://192.168.52.21:2380
initial-cluster-state: new
name: backend1

Context

Was working on sensu-puppet and misconfigured my test environment which resulted in errors getting cluster health. CC @ghoneycutt.

Your Environment

• Sensu version used (sensuctl, sensu-backend, and/or sensu-agent): sensuctl version 2.0.0-nightly#204a7cb, build 204a7cb3a5ff7aa7d26fa943fa5d3bea926422f1, built 2018-09-04T09:21:21+0000
• Installation method (packages, binaries, docker etc.): RPM
• Operating System and version (e.g. Ubuntu 14.04): EL7

[nikkictl]

Hey @treydock, thanks so much for filing this issue and bringing it to our attention! I'm able to reproduce the issue and will begin looking into potential solutions shortly.

[nikkictl]

When an unhealthy cluster member is recognized and logged on the backend, cliErr shows as dial tcp 10.0.1.107:2379: connect: connection refused and is stored in healthResponse.ClusterHealth.Err.

However, when reproducing this bug using sensuctl, the error in the resulting body of GET /health resembles:
"Err":{"Op":"dial","Net":"tcp","Source":null,"Addr":{"IP":"10.0.1.107","Port":2379,"Zone":""},"Err":{"Syscall":"connect","Err":111}} and therefore cannot be unmarshaled into healthResponse.ClusterHealth.Err's type error.

I'm still trying to dig into the reason for this discrepancy since both are of type error.

[DJM0]

@nikkiki I'm seeing the same thing, on what I believe to be a healthy etcd cluster.

> sensuctl version
sensuctl version 2.0.0-beta.4-4#a05ad41, build a05ad411199077286f49c06629eed4823331176c, built 2018-08-16T20:59:49+0000

> sensuctl cluster member-list
         ID            Name                Peer URLs                   Client URLs
 ────────────────── ────────── ────────────────────────────────── ─────────────────────
  ab888ec02ba6ac7c   backend2   http://backend2.domain.com:2380   http://0.0.0.0:2379
  e7fd91709d2b305e   backend1   http://backend1.domain.com:2380   http://0.0.0.0:2379
  fe97ffc5044844ee   backend3   http://backend3.domain.com:2380   http://0.0.0.0:2379

> sensuctl cluster health
Error: json: cannot unmarshal object into Go value of type []*types.ClusterHealth

Then from one of the backends (they all show the same):

> sensu-backend version
sensu-backend version 2.0.0-nightly#0de35c2, build 0de35c245019cf87db54eae57c084ea2dfcb78c3, built 2018-09-12T18:31:17+0000

> etcdctl member list
ab888ec02ba6ac7c: name=backend2 peerURLs=http://backend2.domain.com:2380 clientURLs=http://0.0.0.0:2379 isLeader=true
e7fd91709d2b305e: name=backend1 peerURLs=http://backend1.domain.com:2380 clientURLs=http://0.0.0.0:2379 isLeader=false
fe97ffc5044844ee: name=backend3 peerURLs=http://backend3.domain.com:2380 clientURLs=http://0.0.0.0:2379 isLeader=false

> etcdctl cluster-health
member ab888ec02ba6ac7c is healthy: got healthy result from http://0.0.0.0:2379
member e7fd91709d2b305e is healthy: got healthy result from http://0.0.0.0:2379
member fe97ffc5044844ee is healthy: got healthy result from http://0.0.0.0:2379
cluster is healthy

[nikkictl]

@davidmaitland Thanks for the info! I'm not able to reproduce the unmarshal error on a _healthy_ cluster. I do notice that you have differing versions of sensuctl and sensu-backend, so I suspect that could be a contributing factor to the unmarshal error you are seeing (we recently added alarm info to the cluster health command https://github.com/sensu/sensu-go/commit/613facf2e9894c2fbded012b4aedc78949d6c8d3 so the returned type is no longer a []*types.ClusterHealth and rather a *types.HealthResponse). If you are still able to reproduce the error you are describing after updating both images, please feel free to file a new issue!

[DJM0]

@nikkiki ah that's it! I was trying to use sensuctl on my Mac with the latest binary available (https://storage.googleapis.com/sensu-binaries/2.0.0-beta.4-1/darwin/amd64/sensuctl)

Trying from docker:

> sensuctl version
sensuctl version 2.0.0-nightly#0de35c2, build 0de35c245019cf87db54eae57c084ea2dfcb78c3, built 2018-09-12T18:31:43+0000

> sensuctl cluster health
         ID            Name     Error   Healthy
 ────────────────── ────────── ─────── ─────────
  ab888ec02ba6ac7c   backend2   <nil>   true
  e7fd91709d2b305e   backend1   <nil>   true
  fe97ffc5044844ee   backend3   <nil>   true

👍

[nikkictl]

UREKA! Turns out, we were storing the error as type error (which is an interface) rather than as type string. I'll get a PR for this ASAP!

         ID            Name                              Error                           Healthy  
 ────────────────── ─────────── ─────────────────────────────────────────────────────── ───────── 
  1e8b542f60611ee    backend-2   dial tcp x:2379: connect: connection refused            false    
  7918dd0d1856841b   backend-0                                                           true     
  f7cdce3577ddf3fc   backend-1                                                           true