Sensu-go: etcd-cipher-suites in backend.yml break sensu-backend

Created on 4 Jun 2019  路  4Comments  路  Source: sensu/sensu-go


When etcd-cipher-suites in backend.yml are defined, the sensu-backend service won't start anymore. It does not matter what ciphers you define, the result is the same.

Expected Behavior




The sensu-backend should start normally when etcd-cipher-suites is defined in backend.yml.

Current Behavior



The sensu-backend service fails and is continuously restarted by systemd.

Possible Solution



Removing etcd-cipher-suites definition from backend.yml restores the functionality, but does not fix the real issue here. In technical perspective, no idea what the solution could be.

Steps to Reproduce (for bugs)


  1. Form a 3-member sensu-backend 5.9.0 cluster which leverage backend.yml for configuration (with SSL enabled) but do not define the etcd-cipher-suites.
  2. Make sure that the cluster works as intended.
  3. Insert the following definition into each member's backend.yml:
etcd-cipher-suites:
  - TLS_RSA_WITH_RC4_128_SHA
  - TLS_RSA_WITH_3DES_EDE_CBC_SHA
  - TLS_RSA_WITH_AES_128_CBC_SHA
  - TLS_RSA_WITH_AES_256_CBC_SHA
  - TLS_RSA_WITH_AES_128_CBC_SHA256
  - TLS_RSA_WITH_AES_128_GCM_SHA256
  - TLS_RSA_WITH_AES_256_GCM_SHA384
  - TLS_ECDHE_ECDSA_WITH_RC4_128_SHA
  - TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA
  - TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA
  - TLS_ECDHE_RSA_WITH_RC4_128_SHA
  - TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA
  - TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
  - TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
  - TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256
  - TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
  - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
  - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
  - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
  - TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
  - TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305
  - TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305
  1. Restart each sensu-backend member. The cluster should be inoperable now.

Context


I want to remove all the RC4 and 3DES cipher suites from being used as they are considered insecure.

Following logs show (part of) the problematic behavior of a first sensu-backend in a 3 member cluster by executing _journalctl -u sensu-backend -f_:

Jun 04 12:19:08 sensu-backend1 systemd[1]: sensu-backend.service: main process exited, code=exited, status=1/FAILURE
Jun 04 12:19:08 sensu-backend1 systemd[1]: Unit sensu-backend.service entered failed state.
Jun 04 12:19:08 sensu-backend1 systemd[1]: sensu-backend.service failed.
Jun 04 12:19:09 sensu-backend1 systemd[1]: sensu-backend.service holdoff time over, scheduling restart.
Jun 04 12:19:09 sensu-backend1 systemd[1]: Stopped The Sensu Backend service..
Jun 04 12:19:09 sensu-backend1 systemd[1]: Started The Sensu Backend service..
Jun 04 12:19:09 sensu-backend1 sensu-backend[7061]: {"component":"etcd","level":"warning","msg":"simple token is not cryptographically signed","pkg":"auth","time":"2019-06-04T12:19:09+03:00"}
Jun 04 12:19:14 sensu-backend1 systemd[1]: sensu-backend.service: main process exited, code=exited, status=1/FAILURE
Jun 04 12:19:14 sensu-backend1 systemd[1]: Unit sensu-backend.service entered failed state.
Jun 04 12:19:14 sensu-backend1 systemd[1]: sensu-backend.service failed.
Jun 04 12:19:14 sensu-backend1 systemd[1]: sensu-backend.service holdoff time over, scheduling restart.
Jun 04 12:19:14 sensu-backend1 systemd[1]: Stopped The Sensu Backend service..
Jun 04 12:19:14 sensu-backend1 systemd[1]: Started The Sensu Backend service..
Jun 04 12:19:14 sensu-backend1 sensu-backend[7080]: {"component":"etcd","level":"warning","msg":"simple token is not cryptographically signed","pkg":"auth","time":"2019-06-04T12:19:14+03:00"}
Jun 04 12:19:20 sensu-backend1 sensu-backend[7080]: {"component":"etcd","level":"warning","msg":"lost the TCP streaming connection with peer 8356720591a69734 (stream MsgApp v2 reader)","pkg":"rafthttp","time":"2019-06-04T12:19:20+03:00"}
Jun 04 12:19:20 sensu-backend1 sensu-backend[7080]: {"component":"etcd","level":"error","msg":"failed to read 8356720591a69734 on stream MsgApp v2 (unexpected EOF)","pkg":"rafthttp","time":"2019-06-04T12:19:20+03:00"}
Jun 04 12:19:20 sensu-backend1 sensu-backend[7080]: {"component":"etcd","level":"warning","msg":"lost the TCP streaming connection with peer 8356720591a69734 (stream Message reader)","pkg":"rafthttp","time":"2019-06-04T12:19:20+03:00"}
Jun 04 12:19:20 sensu-backend1 sensu-backend[7080]: {"component":"etcd","level":"warning","msg":"lost the TCP streaming connection with peer 15cbcd8b3ddcaa83 (stream MsgApp v2 reader)","pkg":"rafthttp","time":"2019-06-04T12:19:20+03:00"}
Jun 04 12:19:20 sensu-backend1 sensu-backend[7080]: {"component":"etcd","level":"error","msg":"failed to read 15cbcd8b3ddcaa83 on stream MsgApp v2 (unexpected EOF)","pkg":"rafthttp","time":"2019-06-04T12:19:20+03:00"}
Jun 04 12:19:20 sensu-backend1 sensu-backend[7080]: {"component":"etcd","level":"warning","msg":"lost the TCP streaming connection with peer 15cbcd8b3ddcaa83 (stream Message reader)","pkg":"rafthttp","time":"2019-06-04T12:19:20+03:00"}
Jun 04 12:19:20 sensu-backend1 systemd[1]: sensu-backend.service: main process exited, code=exited, status=1/FAILURE
Jun 04 12:19:20 sensu-backend1 systemd[1]: Unit sensu-backend.service entered failed state.
Jun 04 12:19:20 sensu-backend1 systemd[1]: sensu-backend.service failed.
Jun 04 12:19:21 sensu-backend1 systemd[1]: sensu-backend.service holdoff time over, scheduling restart.
Jun 04 12:19:21 sensu-backend1 systemd[1]: Stopped The Sensu Backend service..
Jun 04 12:19:21 sensu-backend1 systemd[1]: Started The Sensu Backend service..
Jun 04 12:19:21 sensu-backend1 sensu-backend[7097]: {"component":"etcd","level":"warning","msg":"simple token is not cryptographically signed","pkg":"auth","time":"2019-06-04T12:19:21+03:00"}

Your Environment

  • Sensu version used (sensuctl, sensu-backend, and/or sensu-agent): 5.9.0
  • Installation method (packages, binaries, docker etc.): official yum packages
  • Operating System and version (e.g. Ubuntu 14.04): CentOS 7.6.1810
bug
Source
picture hezor

Most helpful comment

Hi @hezor,

Just to let you know that I was able to reproduce this issue. It will require further investigation because I wasn't able to find anything obvious right away.

Thanks

picture palourde on 11 Jun 2019
👍2

All 4 comments

Hi @hezor,

Could you provide us your full backend.yml configuration file so we can try to reproduce it?

Thanks

palourde picture palourde on 7 Jun 2019

Hello @palourde!

Full backend.yml for the first member of the cluster (sensu-backend1):

---
##
# general configuration
##
state-dir: "/var/lib/sensu/sensu-backend"

##
# agent configuration
##
#agent-host: "[::]" # listen on all IPv4 and IPv6 addresses
#agent-port: 8081

##
# api configuration
##
api-listen-address: "[::]:8080" # listen on all IPv4 and IPv6 addresses
api-url: "https://localhost:8080"
cert-file: "/etc/sensu/certs/api-certificate.pem"
key-file: "/etc/sensu/certs/api-privatekey.pem"
trusted-ca-file: "/etc/pki/tls/certs/ca-bundle.crt"

##
# ssl configuration
##
etcd-peer-cert-file: "/etc/sensu/certs/cluster-certificate.pem"
etcd-peer-key-file: "/etc/sensu/certs/cluster-privatekey.pem"
etcd-peer-trusted-ca-file: "/etc/sensu/certs/cluster-ca-certificate.pem"
etcd-peer-client-cert-auth: true
etcd-cert-file: "/etc/sensu/certs/cluster-certificate.pem"
etcd-key-file: "/etc/sensu/certs/cluster-privatekey.pem"
etcd-trusted-ca-file: "/etc/sensu/certs/cluster-ca-certificate.pem"

##
# store configuration
##
etcd-advertise-client-urls: "https://172.18.22.223:2379"
etcd-listen-client-urls: "https://172.18.22.223:2379"
etcd-listen-peer-urls: "https://0.0.0.0:2380"
etcd-initial-cluster: "sensu-backend1=https://172.18.22.223:2380,sensu-backend2=https://172.18.22.224:2380,sensu-backend3=https://172.18.22.225:2380"
etcd-initial-advertise-peer-urls: "https://172.18.22.223:2380"
etcd-initial-cluster-state: "new"
etcd-initial-cluster-token: "sensu"
etcd-name: "sensu-backend1"
etcd-cipher-suites:
  - TLS_RSA_WITH_RC4_128_SHA
  - TLS_RSA_WITH_3DES_EDE_CBC_SHA
  - TLS_RSA_WITH_AES_128_CBC_SHA
  - TLS_RSA_WITH_AES_256_CBC_SHA
  - TLS_RSA_WITH_AES_128_CBC_SHA256
  - TLS_RSA_WITH_AES_128_GCM_SHA256
  - TLS_RSA_WITH_AES_256_GCM_SHA384
  - TLS_ECDHE_ECDSA_WITH_RC4_128_SHA
  - TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA
  - TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA
  - TLS_ECDHE_RSA_WITH_RC4_128_SHA
  - TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA
  - TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
  - TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
  - TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256
  - TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
  - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
  - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
  - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
  - TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
  - TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305
  - TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305

##
# dashboard configuration
##
#dashboard-host: "[::]" # listen on all IPv4 and IPv6 addresses
#dashboard-port: 3000

##
# other
##
#cache-dir: "/var/cache/sensu/sensu-backend"
#config-file: ""
#debug: false
#deregistration-handler: ""
log-level: "warn"

Full backend.yml for the second member of the cluster (sensu-backend2):

---
##
# general configuration
##
state-dir: "/var/lib/sensu/sensu-backend"

##
# agent configuration
##
#agent-host: "[::]" # listen on all IPv4 and IPv6 addresses
#agent-port: 8081

##
# api configuration
##
api-listen-address: "[::]:8080" # listen on all IPv4 and IPv6 addresses
api-url: "https://localhost:8080"
cert-file: "/etc/sensu/certs/api-certificate.pem"
key-file: "/etc/sensu/certs/api-privatekey.pem"
trusted-ca-file: "/etc/pki/tls/certs/ca-bundle.crt"

##
# ssl configuration
##
etcd-peer-cert-file: "/etc/sensu/certs/cluster-certificate.pem"
etcd-peer-key-file: "/etc/sensu/certs/cluster-privatekey.pem"
etcd-peer-trusted-ca-file: "/etc/sensu/certs/cluster-ca-certificate.pem"
etcd-peer-client-cert-auth: true
etcd-cert-file: "/etc/sensu/certs/cluster-certificate.pem"
etcd-key-file: "/etc/sensu/certs/cluster-privatekey.pem"
etcd-trusted-ca-file: "/etc/sensu/certs/cluster-ca-certificate.pem"

##
# store configuration
##
etcd-advertise-client-urls: "https://172.18.22.224:2379"
etcd-listen-client-urls: "https://172.18.22.224:2379"
etcd-listen-peer-urls: "https://0.0.0.0:2380"
etcd-initial-cluster: "sensu-backend1=https://172.18.22.223:2380,sensu-backend2=https://172.18.22.224:2380,sensu-backend3=https://172.18.22.225:2380"
etcd-initial-advertise-peer-urls: "https://172.18.22.224:2380"
etcd-initial-cluster-state: "new"
etcd-initial-cluster-token: "sensu"
etcd-name: "sensu-backend2"
etcd-cipher-suites:
  - TLS_RSA_WITH_RC4_128_SHA
  - TLS_RSA_WITH_3DES_EDE_CBC_SHA
  - TLS_RSA_WITH_AES_128_CBC_SHA
  - TLS_RSA_WITH_AES_256_CBC_SHA
  - TLS_RSA_WITH_AES_128_CBC_SHA256
  - TLS_RSA_WITH_AES_128_GCM_SHA256
  - TLS_RSA_WITH_AES_256_GCM_SHA384
  - TLS_ECDHE_ECDSA_WITH_RC4_128_SHA
  - TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA
  - TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA
  - TLS_ECDHE_RSA_WITH_RC4_128_SHA
  - TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA
  - TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
  - TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
  - TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256
  - TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
  - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
  - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
  - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
  - TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
  - TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305
  - TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305

##
# dashboard configuration
##
#dashboard-host: "[::]" # listen on all IPv4 and IPv6 addresses
#dashboard-port: 3000

##
# other
##
#cache-dir: "/var/cache/sensu/sensu-backend"
#config-file: ""
#debug: false
#deregistration-handler: ""
log-level: "warn"

Full backend.yml for the third member of the cluster (sensu-backend3):

---
##
# general configuration
##
state-dir: "/var/lib/sensu/sensu-backend"

##
# agent configuration
##
#agent-host: "[::]" # listen on all IPv4 and IPv6 addresses
#agent-port: 8081

##
# api configuration
##
api-listen-address: "[::]:8080" # listen on all IPv4 and IPv6 addresses
api-url: "https://localhost:8080"
cert-file: "/etc/sensu/certs/api-certificate.pem"
key-file: "/etc/sensu/certs/api-privatekey.pem"
trusted-ca-file: "/etc/pki/tls/certs/ca-bundle.crt"

##
# ssl configuration
##
etcd-peer-cert-file: "/etc/sensu/certs/cluster-certificate.pem"
etcd-peer-key-file: "/etc/sensu/certs/cluster-privatekey.pem"
etcd-peer-trusted-ca-file: "/etc/sensu/certs/cluster-ca-certificate.pem"
etcd-peer-client-cert-auth: true
etcd-cert-file: "/etc/sensu/certs/cluster-certificate.pem"
etcd-key-file: "/etc/sensu/certs/cluster-privatekey.pem"
etcd-trusted-ca-file: "/etc/sensu/certs/cluster-ca-certificate.pem"

##
# store configuration
##
etcd-advertise-client-urls: "https://172.18.22.225:2379"
etcd-listen-client-urls: "https://172.18.22.225:2379"
etcd-listen-peer-urls: "https://0.0.0.0:2380"
etcd-initial-cluster: "sensu-backend1=https://172.18.22.223:2380,sensu-backend2=https://172.18.22.224:2380,sensu-backend3=https://172.18.22.225:2380"
etcd-initial-advertise-peer-urls: "https://172.18.22.225:2380"
etcd-initial-cluster-state: "new"
etcd-initial-cluster-token: "sensu"
etcd-name: "sensu-backend3"
etcd-cipher-suites:
  - TLS_RSA_WITH_RC4_128_SHA
  - TLS_RSA_WITH_3DES_EDE_CBC_SHA
  - TLS_RSA_WITH_AES_128_CBC_SHA
  - TLS_RSA_WITH_AES_256_CBC_SHA
  - TLS_RSA_WITH_AES_128_CBC_SHA256
  - TLS_RSA_WITH_AES_128_GCM_SHA256
  - TLS_RSA_WITH_AES_256_GCM_SHA384
  - TLS_ECDHE_ECDSA_WITH_RC4_128_SHA
  - TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA
  - TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA
  - TLS_ECDHE_RSA_WITH_RC4_128_SHA
  - TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA
  - TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
  - TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
  - TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256
  - TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
  - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
  - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
  - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
  - TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
  - TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305
  - TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305

##
# dashboard configuration
##
#dashboard-host: "[::]" # listen on all IPv4 and IPv6 addresses
#dashboard-port: 3000

##
# other
##
#cache-dir: "/var/cache/sensu/sensu-backend"
#config-file: ""
#debug: false
#deregistration-handler: ""
log-level: "warn"

The cluster works perfectly fine when the etcd-cipher-suites definition is omitted.

hezor picture hezor on 11 Jun 2019

Hi @hezor,

Just to let you know that I was able to reproduce this issue. It will require further investigation because I wasn't able to find anything obvious right away.

Thanks

palourde picture palourde on 11 Jun 2019
👍2

Hi @hezor,

Turns out the problem was simply our documentation. It appears etcd is sensible to the order the cipher suites are declared (see https://github.com/etcd-io/etcd/issues/10652#issuecomment-499696567), and the suites we presented as default were not the right ones.

We have just merged a change to our docs, which is now live and available under https://docs.sensu.io/sensu-go/5.10/reference/backend/#general-configuration-flags, that proposes a recommended list of cipher suites instead.

Please let us know if you still face any issues with these cipher suites!

palourde picture palourde on 24 Jun 2019
Was this page helpful?
0 / 5 - 0 ratings

Related issues

echlebek picture echlebek  路  4Comments
blufor picture blufor  路  4Comments
spykermj picture spykermj  路  4Comments
cyphus picture cyphus  路  5Comments
amdprophet picture amdprophet  路  4Comments