Datadog-agent: Kubernetes Logs with AutoDiscovery doesn't work without ContainerCollectAll config

[gottfrois]

Describe what happened:

When trying to configure logs aggregation on a K8s cluster using pod annotations, the datadog agent won't consume the annotated pods unless the global logsConfigContainerCollectAll config option is configured.

Describe what you expected:

I would expect to avoid collecting logs from all containers and use annotated pods to consume logs from pods I'm interested by only.
Reading the agent codebase I found the place where the config is required: https://github.com/DataDog/datadog-agent/blob/59d94018c188dc238914a788b7b9663ed45c137d/pkg/logs/input/container/scanner.go#L20-L23

Steps to reproduce the issue:

Using datadog helm chart with values:

datadog.apiKey=xxx
datadog.apmEnabled=true
datadog.logsConfigContainerCollectAll=true
datadog.logsEnabled=true
datadog.nonLocalTraffic=true
datadog.logLevel=WARN

and pod annotation:

ad.datadoghq.com/foo.logs=[{"source": "ruby", "service": "foo"}]

Additional environment details (Operating System, Cloud provider, etc):

$ kubectl version --short
Client Version: v1.11.1
Server Version: v1.10.5-gke.4
$ helm version --short
Client: v2.10.0+g9ad53aa
Server: v2.10.0+g9ad53aa

Helm version is 1.0.4 https://github.com/helm/charts/tree/master/stable/datadog

[ajacquemot]

Hi @gottfrois,
Thanks for opening this issue, I am already working on a fix, for more details see: https://github.com/DataDog/datadog-agent/pull/1965#issuecomment-418098906

[gottfrois]

@ajacquemot a quick question while your head is wired up with this context, is it possible to benefit from auto-discovery from both pods annotations and docker labels? On k8s, the agent seems to be auto-configured to pull pods annotations but wouldn't be a good idea to have both?

[NBParis]

Hello @gottfrois ,

First of all, thanks a lot for helping us on this one and raising the issue you find. This helps us a lot improving the product and contribution are always warmly welcomed.

Regarding your latest question, would you have an example where both would be needed?

As pod annotation have been introduced because Labels were not easily accessible on containers in Kubernetes, it seemed that we would never face a case were both would be used at the same time.

Moreover, let's imagine there are both container labels and pod annotation for a container. Which one should be used?
Some might argue that container labels are more granular so it should be used, other would state that pod annotation are the general rules for all the container described in the annotation.

Therefore if you have some use cases in mind where both would be required, feel free to share them with us.

On a side note, let's keep in mind that this issue was open on Pod annotation not working therefore it will probably get closed when the new version of the Agent will be released so you might want to go through [email protected] and mention this issue so we can continue the discussion.

Thanks

[gottfrois]

Thanks @NBParis for your answer, I think you are right in a sense where both of them are not really useful or desire. I thought about this just because we use k8s for our pre-prod cluster and docker swarm (at the minute) for our production cluster. So we have to configure both pods annotation and docker labels in order to benefit from it, and i was thinking it could be nice if k8s could fallback into docker labels if no pods annotations were found.

But it's not a big deal really, the important part is that we should be able to stream logs of pods with annotations even if the logsConfigContainerCollectAll is set to false.

[NBParis]

Thanks for the clarification @gottfrois.

So in your case it should work as expected. We either look for Labels or pod annotations but we don't look for both for each container.
Therefore if you have either a pod annotation or configuration in container labels, it should be picked up.

That said, this use case has not been tested yet as we focused our efforts on labels container or pod annotations but never both configuration together.

[ajacquemot]

Hi @gottfrois,

We've just released a new release candidate with the fix 6.5.0-rc.3, can you confirm that everything works fine on your side please ?
Thanks

[gottfrois]

@ajacquemot works like a charm, nice work!