Prometheus-operator: Invalid hostnames for kubelet endpoint

Created on 26 Feb 2017 · 2Comments · Source: prometheus-operator/prometheus-operator

What did you do?
Update the prometheus-operator deployment with the argument for the kubelet service.
The hostnames of the servers registered as nodes are:

cluster1-node1.test.example.net
cluster1-node2.test.example.net
cluster1-node3.test.example.net
cluster1-node4.test.example.net

It seems this is just related to the hostnames of the nodes.

What did you expect to see?
An endpoint with the addresses of the kubelet nodes.

What did you see instead? Under which circumstances?
Errors about the hostnames in the prometheus-operator log that the endpoint can't be created.
I see why the regex for the hostname validation in the service is not matching, but

Environment

Kubernetes version information:

Client Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.3", GitCommit:"029c3a408176b55c30846f0faedf56aae5992e9b", GitTreeState:"clean", BuildDate:"2017-02-15T06:40:50Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.2", GitCommit:"08e099554f3c31f6e6f07b448ab3ed78d0520507", GitTreeState:"clean", BuildDate:"2017-01-18T10:52:03Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"}

Kubernetes cluster kind:
Kubernetes bare metal cluster installed using kubernetes/contrib Ansible deployment
Manifests:

apiVersion: "monitoring.coreos.com/v1alpha1"
kind: "ServiceMonitor"
metadata:
  name: "kubelet"
  namespace: monitoring
  labels:
    app: kubelet
spec:
  namespaceSelector:
    matchNames:
      - kube-system
  selector:
    matchLabels:
      k8s-app: kubelet
  endpoints:
  - port: "https-metrics"

Prometheus Operator Logs:

ts=2017-02-26T13:05:56Z caller=operator.go:333 component=prometheusoperator msg="updating kubelet service object failed" err="Service \"kubelet\" is invalid: metadata.resourceVersion: Invalid value: \"\": must be specified for an update"
ts=2017-02-26T13:05:56Z caller=operator.go:344 component=prometheusoperator msg="updating kubelet enpoints object failed" err="Endpoints \"kubelet\" is invalid: [subsets[0].addresses[0].hostname: Invalid value: \"cluster1-node1.test.example.net\": must match the regex [a-z0-9]([-a-z0-9]*[a-z0-9])? (e.g. 'my-name' or '123-abc'), subsets[0].addresses[1].hostname: Invalid value: \"cluster1-node2.test.example.net\": must match the regex [a-z0-9]([-a-z0-9]*[a-z0-9])? (e.g. 'my-name' or '123-abc'), subsets[0].addresses[2].hostname: Invalid value: \"cluster1-node3.test.example.net\": must match the regex [a-z0-9]([-a-z0-9]*[a-z0-9])? (e.g. 'my-name' or '123-abc'), subsets[0].addresses[3].hostname: Invalid value: \"cluster1-node4.test.example.net\": must match the regex [a-z0-9]([-a-z0-9]*[a-z0-9])? (e.g. 'my-name' or '123-abc')]"

Source