Apm: [agents] Read k8s data from environment variables

[roncohen]

It would be great if agents could read kubernetes data from environment variables and add it automatically to _metadata_ sent to APM Server. This would work both for kubernetes setups where APM Server runs self-hosted and in the future where APM Server could be hosted.

I'll file a separate issue for the APM Server work.

Users set the environment variables through the DownwardAPI.

Here's a table mapping the environment variable to the metadata event field:

| Environment variable | Field name |
| ----------------------------- | ------------- |
| KUBERNETES_NODE_NAME | system.kubernetes.node.name |
| KUBERNETES_POD_NAME | system.kubernetes.pod.name |
| KUBERNETES_NAMESPACE | system.kubernetes.namespace |
| KUBERNETES_POD_UID | system.kubernetes.pod.uid |

Elasticsearch documents will use kubernetes.node.name (and not system.kubernetes.node.name) etc., so it aligns with the fields generated by filebeat, except for a few:

• container.* fields which we don't seem to have access to unfortunately.
• annotations: off by default in filebeat because people use them to store all sorts of large objects
• labels: could be very useful down the line, but lets start without as an MVP

Here's how a user would add the environment variables to their kubernetes pod spec:

         - name: KUBERNETES_NODE_NAME
            valueFrom:
              fieldRef:
                fieldPath: spec.nodeName
          - name: KUBERNETES_POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: KUBERNETES_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
          - name: KUBERNETES_POD_UID
            valueFrom:
              fieldRef:
                fieldPath: metadata.uid

@elastic/apm-agent-devs Please link your issue or PR and then mark the checkbox when the implementation is merged:

• [x] - Java - https://github.com/elastic/apm-agent-java/issues/344
• [x] - Python https://github.com/elastic/apm-agent-python/issues/347
• [x] - Ruby - https://github.com/elastic/apm-agent-ruby/issues/252
• [x] - Go - https://github.com/elastic/apm-agent-go/issues/329
• [x] - dotNet - https://github.com/elastic/apm-agent-dotnet/issues/181
• [x] - Node.js - https://github.com/elastic/apm-nodejs-http-client/pull/34
• [x] - Documentation - https://github.com/elastic/apm-server/pull/1650
• [x] - APM Server https://github.com/elastic/apm-server/issues/1621

Potential additional fields that we should not do now

| Environment variable | Field name |
| ----------------------------- | ------------- |
| KUBERNETES_POD_IP | system.kubernetes.pod.ip |
| KUBERNETES_NODE_IP | system.kubernetes.node.ip |
| KUBERNETES_LABELS_* | system.kubernetes.labels.* |

          - name: KUBERNETES_POD_IP
            valueFrom:
              fieldRef:
                fieldPath: status.podIP
          - name: KUBERNETES_NODE_IP
            valueFrom:
              fieldRef:
                fieldPath: status.hostIP
          - name: KUBERNETES_LABELS_APP
            valueFrom:
              fieldRef:
                fieldPath: metadata.labels['app']

[roncohen]

Updated to reflect that data should be added to metadata under system.

[watson]

I assume we can just publish this in the agents without coordinating with a specific release of the APM Server?

[ruflin]

@roncohen I think the k8s fields are without a system.* prefix.
@exekias For awareness.

[roncohen]

@ruflin this is for agent devs. The intake api uses system. I'll make it more clear

[beniwohli]

Where should we document how the user has to set the environment variables? In a central location or in each of the agent docs?

[roncohen]

@bmorelli25 probably makes sense to describe the process for setting up the k8s environment variables somewhere centrally instead of in each agent docs. WDYT?

[watson]

I was thinking we probably should still link to this central page from each agent. That got me thinking if asciidoc might even allow us to "embed" this central docs into the actual agent docs. But maybe this is too complicated or doesn't make sense. Anyway, just a thought 😃

[bmorelli25]

@watson, your idea would work. We could create an asciidoc file in the APM Overview directory, then include:: it in each agent's documentation. Wouldn't be difficult to implement or maintain, I just worry that it might overly complicate things for users. Having the same information six times scares me a bit. If the documentation is identical, I prefer the centralized docs idea (like in APM Overview) with all agents linking to it.

[axw]

@roncohen in the description you've got KUBERNETES_POD_NAMESPACE -> system.kubernetes.namespace

I think that should just be KUBERNETES_NAMESPACE?

EDIT: I guess not, given that https://kubernetes.io/docs/tasks/inject-data-application/environment-variable-expose-pod-information/#use-pod-fields-as-values-for-environment-variables also uses "MY_POD_NAMESPACE". I suppose that makes sense since it's only relevant to the pod, and not the node. I was kinda thrown by the fact that the JSON path doesn't have "pod" in it.

[roncohen]

True. Will correct it. Thanks!

On Thu, 29 Nov 2018 at 10.48, Andrew Wilkins [e-mail]
wrote:

@roncohen https://github.com/roncohen in the description you've got KUBERNETES_POD_NAMESPACE
-> system.kubernetes.namespace

I think that should just be KUBERNETES_NAMESPACE?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/elastic/apm/issues/21#issuecomment-442771243, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAAC6DI2y0Casqj9Popt8unmusNqf6j_ks5uz61egaJpZM4Yw5aU
.

[watson]

@roncohen wrote:

True

@axw wrote:

I guess not

.... sooo I'm not really sure if that means we should use KUBERNETES_NAMESPACE or KUBERNETES_POD_NAMESPACE 😅

[roncohen]

Let's go with KUBERNETES_NAMESPACE. I'll update.

[axw]

There are a couple of properties which can be extracted from /proc/self/cgroup when running under Kubernetes: container ID and pod UID. Additionally, the hostname is set to the pod name; when we detect we're running within Kubernetes, we can assume the pod name is the hostname.

If everyone agrees, I think we should we should do the following:

• check /proc/self/cgroup for the kubernetes format (see below), extracting the pod UID and container ID, and setting the pod name == hostname
• override/extend anything found above with values specified via KUBERNETES_{NAMESPACE,NODE_NAME,POD_NAME,POD_UID}

The cgroup path will look like one of

• (cgroupfs) /kubepods/<QoS-class>/pod<pod-UID>/<container-ID>
• (systemd) /kubepods.slice/kubepods-<QoS-class>.slice/kubepods-<QoS-class>-pod<pod-UID>.slice/docker-<container-ID>.scope

For each line in /proc/self/cgroup, I check for one which satisfies the below:

1. Ensure the line has 3 fields split by :
2. Ignore the first 2 fields; split the third field on the final /, so you've got the directory (call it dir) and basename (call it id): e.g. "/docker/123456" is split into dir="/docker/" and id="123456"
3. If id ends with ".scope", trim everything up to and including the first -
4. Check first if dir matches one of the kubernetes regular expressions:
   
   
   
   • :^/kubepods/[^/]+/pod([^/]+)/$ +
   
   
   • :^/kubepods\.slice/kubepods-[^/]+\.slice/kubepods-[^/]+-pod([^/]+)\.slice/$
   
   
5. If dir doesn't match, then check if id is a hex-encoded sha256 (Docker or cri-o container ID):
   
   
   
   • ^[[:xdigit:]]{64}$
   
   

When dir matches one of the /kubepods paths, then we blindly use id as the container ID; we're relying on the format of the path to imply that the final path segment is the container ID, and we don't require the Docker container ID format. We also extract the regexp capturing group value as the pod UID, and get the hostname to use as the pod name.

When dir does not match /kubepods, but id is a hex-encoded sha256 (64 hex digits), then we assume it is a container ID. I was previously checking for "docker" in the path, but that would preclude other container runtime. After discussing with Eyal, we decided that it was unlikely enough that something other than a container runtime would use such a cgroup path, so I removed that check.

You can find my implementation for the Go agent in https://github.com/elastic/apm-agent-go/pull/342. Here's some test cases:

12:devices:/docker/051e2ee0bce99116029a13df4a9e943137f19f957f38ac02d6bad96f9b700f76

• container.id = "051e2ee0bce99116029a13df4a9e943137f19f957f38ac02d6bad96f9b700f76"

1:name=systemd:/system.slice/docker-cde7c2bab394630a42d73dc610b9c57415dced996106665d427f6d0566594411.scope

• container.id = "cde7c2bab394630a42d73dc610b9c57415dced996106665d427f6d0566594411"

1:name=systemd:/kubepods/besteffort/pode9b90526-f47d-11e8-b2a5-080027b9f4fb/15aa6e53-b09a-40c7-8558-c6c31e36c88a

• container.id = "15aa6e53-b09a-40c7-8558-c6c31e36c88a" (NOTE: UUID format like rkt; I made this up though)
• pod.uid = "e9b90526-f47d-11e8-b2a5-080027b9f4fb"
• pod.name = <hostname>

1:name=systemd:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90d81341_92de_11e7_8cf2_507b9d4141fa.slice/crio-2227daf62df6694645fee5df53c1f91271546a9560e8600a525690ae252b7f63.scope

• container.id = "2227daf62df6694645fee5df53c1f91271546a9560e8600a525690ae252b7f63" (NOTE: "crio-" is trimmed off)
• pod.uid = "90d81341_92de_11e7_8cf2_507b9d4141fa"
• pod.name = <hostname>

[roncohen]

there's also the question about how we should handle invalid format vs. missing /proc/self/cgroup. Generally, I think if the /proc/self/cgroup is missing, we should just silently ignore it and we should log a warning if it's there but we're not able to extract the information we expected from it. This goes for #22 as well btw.

[axw]

we should log a warning if it's there but we're not able to extract the information we expected from it

/proc/self/cgroup is probably always going to be there on Linux, but naturally it won't have the container format paths in a non-container context. I'm treating non-matching contents as "not containerised", which doesn't warrant a warning.

I guess we could look for "docker" or "kube" in the path and warn if we don't get a match? Did you have something specific in mind?

[beniwohli]

@axw awesome, thanks for putting together a few test cases. I put them into a python dict, might be useful for others (at least for nodejs it should be a valid JS object)

cases = {
    "12:devices:/docker/051e2ee0bce99116029a13df4a9e943137f19f957f38ac02d6bad96f9b700f76": {
        "container": {
            "id": "051e2ee0bce99116029a13df4a9e943137f19f957f38ac02d6bad96f9b700f76",
        }
    },
    "1:name=systemd:/system.slice/docker-cde7c2bab394630a42d73dc610b9c57415dced996106665d427f6d0566594411.scope": {
        "container": {
            "id": "cde7c2bab394630a42d73dc610b9c57415dced996106665d427f6d0566594411",
        }
    },
    "1:name=systemd:/kubepods/besteffort/pode9b90526-f47d-11e8-b2a5-080027b9f4fb/15aa6e53-b09a-40c7-8558-c6c31e36c88a": {
        "container": {
            "id": "15aa6e53-b09a-40c7-8558-c6c31e36c88a",
        },
        "pod": {
            "uid": "e9b90526-f47d-11e8-b2a5-080027b9f4fb",
        }
    },
    "1:name=systemd:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90d81341_92de_11e7_8cf2_507b9d4141fa.slice/crio-2227daf62df6694645fee5df53c1f91271546a9560e8600a525690ae252b7f63.scope": {
        "container": {
            "id": "2227daf62df6694645fee5df53c1f91271546a9560e8600a525690ae252b7f63",
        },
        "pod": {
            "uid": "90d81341_92de_11e7_8cf2_507b9d4141fa",
        }
    }
}

[roncohen]

/proc/self/cgroup is probably always going to be there on Linux, but naturally it won't have the container format paths in a non-container context. I'm treating non-matching contents as "not containerised", which doesn't warrant a warning.

wasn't aware of that. Makes sense to treat it in that way.

[beniwohli]

As @axw pointed out in his review of my PR (https://github.com/elastic/apm-agent-python/pull/352#pullrequestreview-182046011), my implementation of the cgroup file parsing differs quite a bit from his. While he uses regexes in the Go agent, I opted to use substring search and string splitting. I also don't enforce the IDs to hex digits of a certain length.

My rationale for this implementation as opposed to regexes is that I feel like regex matching is too strict in the face of a file format (if it even merits that name) that is underdefined and unstable. Using simple string splitting allows for some leeway in the format.

The drawback of my approach is that the code turns out quite ugly and branchy, a fertile ground for bugs.

I do agree with @axw that we should try and align implementations to ensure consistent results. What do y'all think?

(btw, I don't think performance considerations should play a role in deciding which way to go. Ideally, this code is only run once, during startup. A few allocations and microseconds difference shouldn't be a factor)

[axw]

As @axw pointed out in his review of my PR (elastic/apm-agent-python#352 (review)), my implementation of the cgroup file parsing differs quite a bit from his. While he uses regexes in the Go agent, I opted to use substring search and string splitting. I also don't enforce the IDs to hex digits of a certain length.

Just to be clear, my mention of regex was purely about keeping the code size compact and neat. What I most care about is keeping aligned is the behaviour. For behaviour, the two approaches are:

1. check that the last path segment is exactly 64 hex digits, and take it as the container ID
2. check that the path contains "docker", and take the last path segment as the container ID

(1) will pick up non-Docker container IDs as long as they use the same hex-encoded sha256 format. This doesn't cover all container runtimes/schedulers, but it covers a few: Docker, cri-o, AWS ECS. If any of those start using a different format, it won't work any more.

(2) will work if the Docker container ID format changes, but won't pick up cri-o, AWS ECS. OTOH, if the cgroup format changes to something like "/docker//suffix", then we'll start reporting "suffix" as the container ID.

[axw]

ECS [0] says this about host.hostname in a Kubernetes context:

Hostname of the host.
It normally contains what the hostname command returns on the host machine.

I think where it says host, it means node; I'm seeking clarification on this. This likely the cause of https://github.com/elastic/kibana/issues/23317#issuecomment-476151029.

I propose that we change the following change to APM Server:

• If system.kubernetes.node.name is set in the metadata, set host.hostname in the event to its value
• If system.kubernetes.* is set, but system.kubernetes.node.name is not, then don't set host.hostname at all. We could go a step further, and modify APM Server to map the pod's namespace and name if it has the access to do so
• Otherwise set host.hostname to system.hostname

Alternatively, we could change each of the agents to set system.hostname accordingly. Then system.kubernetes.node.name is redundant. Also, I'm not sure if it's reasonable/viable for the pod to know its own node name.

[0] https://github.com/elastic/ecs/blob/master/use-cases/kubernetes.md

[alvarolobato]

@elastic/apm-server can you have a look at Andrew's comment and give your opinion, please?

[graphaelli]

This proposal makes sense from the ECS definition, but does it work for the referenced kibana issue? Filtering logs by kubernetes node doesn't seem like something a user would want to do very often. If it's an option with lower priority than filtering by pod then perhaps it doesn't hurt anything. Probably better discussed in the other issue.

We could go a step further, and modify APM Server to map the pod's namespace and name if it has the access to do so

Like a beats kubernetes metadata processor would, this is great.

I'm not sure if it's reasonable/viable for the pod to know its own node name.

Agreed.

[axw]

This proposal makes sense from the ECS definition, but does it work for the referenced kibana issue?

I think so. Depending on the presence of the fields, the UI will show some or all of the following links:

• "Host" logs/metrics
• Container logs/metrics
• Pod logs/metrics

(As mentioned previously, in the context of k8s "host" means "node" -- Mathieu confirmed this.)

In https://github.com/elastic/kibana/issues/23317#issuecomment-476151029, Felix mentions that the host links aren't working. I'm not 100% sure since I haven't looked at the data, but I'm surmising that the issue is related to us using the pod's hostname rather than the node's. Presumably Filebeat is setting the fields correctly, so searching filebeat indices for "host.hostname: " isn't going to come up with anything.

Filtering logs by kubernetes node doesn't seem like something a user would want to do very often. If it's an option with lower priority than filtering by pod then perhaps it doesn't hurt anything. Probably better discussed in the other issue.

I concur that it's probably an unusual thing to do; yes, let's discuss the utility elsewhere. Regardless of the utility, I think we should ensure the data adheres to ECS.

Like a beats kubernetes metadata processor would, this is great.

Yes, we can probably reuse some code too.

[graphaelli]

Implemented in apm-server - tests only cover unit and non-k8s integration. Planning to manually test on kubnernetes before backporting.

https://github.com/elastic/kibana/issues/34259 opened to track implementation in migration script.

[graphaelli]

This made it into apm-server 7.0.0 and should be included in kibana 7.0.1.

https://github.com/elastic/apm-agent-dotnet/issues/181 opened to track implementation for that agent. Closing as all GA agents have implemented this.

[watson]

When we talk about environment variables, we normally leave out the ELASTIC_APM_ prefix as it takes up too much space. So when the Node.js agent implemented this issue, we assumed the environment variables should include the prefix like all our other environment variables.

But it looks like all the other agents have implemented them without that prefix. Is that because these environment variables names are already well established in the Kubernetes world, or was it an oversight in this spec?

[axw]

This was discussed in the agents weekly meeting briefly, and the outcome was that the Node.js agent should be updated to use KUBERNETES_FOO.

I lack a link to the meeting and I was kinda half asleep already, but what I took away from the meeting was: it's different because it's not so much agent configuration as it is conveying information about the environment to the agent; it's Kubernetes-centric rather than agent-centric.

IMO it's still a little bit debatable, as we have ELASTIC_APM_HOSTNAME... but that one never really sit that well with me anyway. Anyway, I think this is the most pragmatic way forward. Brandon is going to update the docs to make it all clear to users how they can configure Kubernetes.

[nerddelphi]

@axw It'll going to be very useful if APM Agents send kubernetes.labels.* as well.

[axw]

@nerddelphi could you please create a feature request issue for that? The more you can include about motivation, the better. Thank you.