Splunk-connect-for-kubernetes: Question: Capture specific logs for a given container?

[ajsaclayan]

Hi all,

Following this example:
https://github.com/splunk/splunk-connect-for-kubernetes/blob/develop/helm-chart/splunk-kubernetes-logging/examples/use_logs.yaml

I have some logs (log1 and log2) that live in acontainer at path /opt/containername/. I've SSH'd into the container and verified the path to these logs.

I want to tail these logs. In my values.yaml I've defined this snippet:

logs:
  container-name:
    from:
      file:
        path: /opt/containerName/log1.log

Retrieving the logs of the splunk-connect pod that lives on the same node as the container:
2019-12-17 21:35:16 +0000 [warn]: #0 /opt/containername/log1.log not found. Continuing without tailing it.

Any guidance would be appreciated.

Thanks!
AJ

[rockb1017]

Hello,
the file should be the splunk-connect-logging pod. Not just any container.
do you still see the file in that path of splunk-connect pod?

[ajsaclayan]

Thanks for the quick response. I'm a little new to this. Is this something I have to configure on fluentd to forward these logs?

[rockb1017]

Normally, have you application print logs to standard out, then the rest is automatically handled by kubelet and docker.
when application is run inside container by docker, docker will write all printed logs to a file. ie) /var/lib/docker/container//.-json.log
Kubelet also creates symlinks of these files under /var/log/pod/... and /var/log/container/.... when this docker container is created through kubernetes.
Then this connector will read all these container log files and send it to splunk.

[matthewmodestino]

In this scenario, where the logs are not sent to stdout/stderr, you would use a “sidecar” approach.

https://kubernetes.io/docs/concepts/cluster-administration/logging/#sidecar-container-with-a-logging-agent

Splunk Connect for Kubernetes is designed to be a Node Agent collecting all logs on a node.

https://kubernetes.io/docs/concepts/cluster-administration/logging/#using-a-node-logging-agent

The node agent usually covers 90% of all the things you need to collect, then you can use the sidecar or mount the volumes you are looking to monitor into the node agent pod.

To get started I recommend you try something like this example...its a docker example but applies to k8s as well if you translate it to k8s yaml....

https://github.com/matthewmodestino/container_workshop/blob/master/universalforwarder/sidecar/sidecar.yaml

This example demonstrates the concept you are after, where the splunk uf monitors a directory from another container by using volume mounts. You could use fluentd in place of the if here as well. See if this helps...

[ajsaclayan]

Thanks all for your suggestions! I believe we had the application misconfigured, so we weren't seeing logs in stdout. We have since been able to capture logs in splunk.