Splunk-connect-for-kubernetes: Windows pods logs

[doryanmzr]

What would you like to be added:

We have hybrid cluster on EKS with windows pods and we'd like to send those logs to Splunk
AFAIK, there is support only for Linux daemonset and not for windows.

Why is this needed:

Send logs of windows pods to Splunk

[matthewmodestino]

can you please provide more info?

can you use docker uf instead? where are the logs from the windows pods?

[doryanmzr]

Hi,
we installed the Splunk connector using helm on the cluster, once Splunk-connect has deployed , it actually deploys daemonset( nodeselector linux) of fluentd which collect logs only from the Linux pods and then send them to splunk , so far so good, but we have windows pods as well that we want collect theirs logs from k8s and send them to Splunk, I assume with windows fluentd as well.
are you going to develop daemonset for windows nodes in order to support windows pods?

[matthewmodestino]

The logging strategy on windows is completely different.

Probably best to use the docker-uf as a sidecar for windows workloads or review how you are going to collect the logs from the windows nodes.

Please refer to Windows support. I don't think logs get aggregated into a central spot on windows...please confirm...

[doryanmzr]

The logging strategy on windows is completely different.

Probably best to use the docker-uf as a sidecar for windows workloads or review how you are going to collect the logs from the windows nodes.

Please refer to Windows support. I don't think logs get aggregated into a central spot on windows...please confirm...

AFAIK, there is daemonset (windows container based) of fluentd for cloudwatch so I assume it's possible to customize fluentd for Splunk to collect the windows logs from central spot.

as a workaround, we installed universal forwarder inside the docker image, it's working just fine but it's not optimal

[matthewmodestino]

https://github.com/bgsilvait/k8s-fluentd-windows

I see no reason why this wouldn't work...the only part that I am not sure about is how /var/log symlink is maintained by kubelet...

In this same configmap, it looks like Windows can resolve the linux path for the pod log symlink ???

     containers:
      - name: fluentd-windows
        image: bgsilvait/winfluentd:1809
        env:
          - name: AWS_REGION
            valueFrom:
              configMapKeyRef:
                name: fluentd-windows
                key: AWS_REGION
          - name: CLUSTER_NAME
            valueFrom:
              configMapKeyRef:
                name: fluentd-windows
                key: CLUSTER_NAME
        resources:
          limits:
            memory: 2Gi
          requests:
            cpu: 100m
            memory: 1Gi
        volumeMounts:
        - name: fluentdconftemp
          mountPath: /etc/temp/
        - name: fluentdconf
          mountPath: /etc/fluent
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: C:\ProgramData\Docker\containers
          readOnly: true

what voodoo is this? :)

[matthewmodestino]

Ah it looks like windows indeed does keep the same kubelet symlink method...

https://blog.smokinserver.com/on-premise-Windows-kubernetes-logging-with-IIS-fluentd-elasticsearch

[felixep]

We few months ago I ran into the same problem and I was able to build a solution. It's working for me and I wanted to share it with you guys ... check it out https://github.com/felixep/splunk-k8s-win-logging