Charts: [stable/prometheus] How to monitor etcd with ssl?

[minhdanh]

Is this a request for help?: Yes

---

Is this a BUG REPORT or FEATURE REQUEST? (choose one): FEATURE REQUEST

Version of Helm and Kubernetes:

helm version
Client: &version.Version{SemVer:"v2.8.2", GitCommit:"a80231648a1473929271764b920a8e346f6de844", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.8.2", GitCommit:"a80231648a1473929271764b920a8e346f6de844", GitTreeState:"clean"}

kubectl version
Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.2", GitCommit:"81753b10df112992bf51bbc2c2f85208aad78335", GitTreeState:"clean", BuildDate:"2018-04-27T09:10:24Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.2", GitCommit:"81753b10df112992bf51bbc2c2f85208aad78335", GitTreeState:"clean", BuildDate:"2018-04-27T09:10:24Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}



md5-a8c6e39406f10c2f518c30c45603eaa0



      - job_name: 'kubernetes-etcd'
        scheme: https
        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
          insecure_skip_verify: true
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        static_configs:
          - targets: ['10.6.16.99:2379','10.6.2.41:2379','10.6.18.105:2379']



md5-4333f9dd5779db7ebcd581b94763c541



Get https://10.6.16.99:2379/metrics: remote error: tls: bad certificate



md5-7b01b5af10ee719a846b22fe3cf740ea



additionalFiles:
   etcd_ca_file: |
     [content of ca.crt]
   etcd_cert_file: |
     [...]
   ...

Then those file will be mounted to Prometheus in /prometheus.

Then the job configs for etcd can refer to these files.

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know:

[Nodraak]

I ended up manually loading etcd's certificates into a Kubernetes Secret and adding this to prometheus chart's config :

  extraSecretMounts:
    - name: prometheus-etcd-certificates
      mountPath: /etcd-ssl
      secretName: prometheus-etcd-certificates
      readOnly: true

[...]
      - job_name: 'etcd'
        [...]
        tls_config:
          insecure_skip_verify: true
          cert_file: /etcd-ssl/tls.crt
          key_file: /etcd-ssl/tls.key

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[stale[bot]]

This issue is being automatically closed due to inactivity.

[aleks-mariusz]

in case this helps anyone, i built an on-prem v1.13.5 kubernetes cluster using kubeadm..

i also deploy my prometheus-operator into its own "metrics" namespace.

so during cluster initialization, while bootstrapping the cluster, i also capture the generated etcd certs on the first master node using:

D="$(mktemp -d)"
cp /etc/kubernetes/pki/etcd/{ca.crt,healthcheck-client.{crt,key}} $D
kubectl create ns metrics
kubectl -n metrics create secret generic etcd-client --from-file="$D"
rm -fr "$D"

and then adding this to my values.yaml:

kubeEtcd:
  serviceMonitor:
    scheme: https
    insecureSkipVerify: true
    caFile: /etc/prometheus/secrets/etcd-client/ca.crt
    certFile: /etc/prometheus/secrets/etcd-client/healthcheck-client.crt
    keyFile: /etc/prometheus/secrets/etcd-client/healthcheck-client.key

prometheus:
  prometheusSpec:
    secrets:
      - etcd-client

then the target gets scraped properly

[caihy05]

挂载证书解决问题

[caihy05]

第一步：
kubectl create secret generic prome-etcd-cert --from-file=/etc/etcd/ssl/etcd.pem --from-file=/etc/etcd/ssl/etcd-key.pem -n monitoring
第二步：
volumeMounts:
- name: ssl
mountPath: /ssl
volumes:
- name: ssl
secret:
secretName: prome-etcd-cert

第三步：
- job_name: 'etcd'
scheme: https
tls_config:
insecure_skip_verify: true
cert_file: /ssl/etcd.pem
key_file: /ssl/etcd-key.pem
static_configs:
- targets: ['172.16.66.89:2379','172.16.66.90:2379','172.16.66.91:2379']