Datadog-agent: Kubernetes Daemonset manifest does not work on clusters < 1.7

[jonmoter]

Kubernetes manifest files don't work on Kubernetes clusters < 1.7

In your example manifests, there's this line:

https://github.com/DataDog/datadog-agent/blob/ea7982569bdf0c62c8adf00e2c5b855cc266f71c/Dockerfiles/manifests/agent.yaml#L34-L37

Support for using status.hostIP was added in kubernetes/kubernetes#42717, which was part of the 1.7 release.

If I try deploying with your manifest, I get an error like:

DaemonSet.extensions "datadog-daemonset" is invalid: spec.template.spec.containers[0].env[3].valueFrom.fieldRef.fieldPath: Invalid value: "status.hostIP": error converting fieldPath: field label not supported: status.hostIP

I'm running Kubernetes 1.5.7. (Yes, it's an old release, and we're working on upgrading.)

Is the DD_KUBERNETES_KUBELET_HOST environment variable necessary for the kubernetes integration to work?

If you only support Kubernetes 1.7 or higher, please document this. If not, please offer an alternative to how to make this work on earlier versions.

[rltvty]

I used spec.nodeName here instead of status.hostIP to get the agent to install. But I'm having lots of other errors still, and can't get the pod to stay up.

I'm on kubernetes 1.6.8

[jonmoter]

I tried using spec.nodeName and now I'm getting errors like:

[ AGENT ] 2018-03-20 19:12:34 UTC | ERROR | (runner.go:276 in work) | Error running check kubelet: [{"message": "404 Client Error: Not Found for url: http://ip-W-X-Y-Z.us-west-1.compute.internal:10255/metrics/cadvisor", "traceback": "Traceback (most recent call last):\n  File \"/opt/datadog-agent/bin/agent/dist/checks/__init__.py\", line 332, in run\n    self.check(copy.deepcopy(self.instances[0]))\n  File \"/opt/datadog-agent/embedded/lib/python2.7/site-packages/datadog_checks/kubelet/kubelet.py\", line 128, in check\n    self.process(self.metrics_url, send_histograms_buckets=send_buckets, instance=instance)\n  File \"/opt/datadog-agent/embedded/lib/python2.7/site-packages/datadog_checks/checks/prometheus/prometheus_base.py\", line 310, in process\n    response = self.poll(endpoint)\n  File \"/opt/datadog-agent/embedded/lib/python2.7/site-packages/datadog_checks/checks/prometheus/prometheus_base.py\", line 432, in poll\n    response.raise_for_status()\n  File \"/opt/datadog-agent/embedded/lib/python2.7/site-packages/requests/models.py\", line 862, in raise_for_status\n    raise HTTPError(http_error_msg, response=self)\nHTTPError: 404 Client Error: Not Found for url: http://ip-W-X-Y-Z.us-west-1.compute.internal:10255/metrics/cadvisor\n"}]

Where ip-W-X-Y-Z.us-west-1.compute.internal is the name of the EC2 instance it's running on.

[jonmoter]

Actually, I think the issue is that the /metrics/cadvisor endpoint doesn't exist in Kubernetes 1.5.x. I am able to curl ip-W-X-Y-Z.us-west-1.compute.internal:10255/metrics from the container, but not /metrics/cadvisor

[irabinovitch]

@jonmoter Thats correct. It becomes available in 1.7.x. We are currently digging into making 6.x work with k8s 1.3, 1.4, 1.5, 1.6 via older APIs

[jonmoter]

Okay, thanks for the verification.

Any idea what the approximate timeframe is for support of the earlier versions? I don't need a commitment, but an idea of O(weeks) or O(months) would be helpful.

[irabinovitch]

@jonmoter We're actively working on this at the moment. You should start to see some open PRs in the next week or two. Our goal is to have something available to test later in April.

[jonmoter]

Hey, any word on this? Any PRs that I could follow along on, to support earlier versions of k8s?

[xvello]

Hi @jonmoter

The two main parts of 1.3+ support are already merged in master, and will ship with 6.2 early next month:

• https://github.com/DataDog/datadog-agent/pull/1570 :kube_* creator tags collection
• https://github.com/DataDog/integrations-core/pull/1339 : collect metrics from the cadvisor port

The testing image datadog/agent-dev:xvello-test1-3 listed in the second PR includes both changes. While we run our internal QA process, early feedback on it would be much appreciated.

[xvello]

Legacy Kubernetes versions are documented in https://github.com/DataDog/datadog-agent/blob/master/Dockerfiles/agent/README.md#legacy-kubernetes-versions
Closing the issue