Splunk-connect-for-kubernetes: metrics are stopped and not restarting when kubelet is not reachable

[schose]

What happened:

We are closely monitoring if metrics from clusters are ingested into splunk enterprise. using following search:

`k8s_objects` sourcetype="kube:objects:nodes" earliest=-15m@m | stats values(metadata.name) as nodename by cluster_name | mvexpand nodename
| join type=left nodename
[
  | mstats count as metriccount WHERE metric_name="kube.node.cpu.usage" `k8s_metrics` earliest=-15m@m latest=-0m@m by node
  | rename node as nodename
]
| fillnull value=0
| search metriccount=0

We see that sometimes metrics are not send anymore. Root-cause seems to be that kubelet is not responding for any reason. The metric container shows errors like:

"2020-10-24 08:08:26 +0000 [error]: #0 Unexpected error raised. Stopping the timer. title=:cadvisor_metric_scraper error_class=RestClient::Unauthorized error="401 Unauthorized"
  2020-10-24 08:08:26 +0000 [error]: #0 /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/abstract_response.rb:249:in `exception_with_response'
  2020-10-24 08:08:26 +0000 [error]: #0 /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/abstract_response.rb:129:in `return!'
  2020-10-24 08:08:26 +0000 [error]: #0 /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/request.rb:836:in `process_result'
  2020-10-24 08:08:26 +0000 [error]: #0 /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/request.rb:743:in `block in transmit'
  2020-10-24 08:08:26 +0000 [error]: #0 /usr/share/ruby/net/http.rb:910:in `start'
  2020-10-24 08:08:26 +0000 [error]: #0 /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/request.rb:727:in `transmit'
  2020-10-24 08:08:26 +0000 [error]: #0 /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/request.rb:163:in `execute'
  2020-10-24 08:08:26 +0000 [error]: #0 /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/request.rb:63:in `execute'
  2020-10-24 08:08:26 +0000 [error]: #0 /opt/app-root/src/gem/fluent-plugin-kubernetes-metrics-1.1.4/lib/fluent/plugin/in_kubernetes_metrics.rb:660:in `scrape_cadvisor_metrics'
  2020-10-24 08:08:26 +0000 [error]: #0 /usr/share/gems/gems/fluentd-1.9.1/lib/fluent/plugin_helper/timer.rb:80:in `on_timer'
  2020-10-24 08:08:26 +0000 [error]: #0 /usr/share/gems/gems/cool.io-1.6.0/lib/cool.io/loop.rb:88:in `run_once'
  2020-10-24 08:08:26 +0000 [error]: #0 /usr/share/gems/gems/cool.io-1.6.0/lib/cool.io/loop.rb:88:in `run'
  2020-10-24 08:08:26 +0000 [error]: #0 /usr/share/gems/gems/fluentd-1.9.1/lib/fluent/plugin_helper/event_loop.rb:93:in `block in start'
  2020-10-24 08:08:26 +0000 [error]: #0 /usr/share/gems/gems/fluentd-1.9.1/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2020-10-24 08:08:26 +0000 [error]: #0 Timer detached. title=:cadvisor_metric_scraper%

same happens if kubelet is stopped for a while:

2020-11-05 16:15:14 +0000 [error]: #0 Unexpected error raised. Stopping the timer. title=:cadvisor_metric_scraper error_class=Errno::ECONNREFUSED error="Failed to open TCP connection to 192.168.53.112:10250 (Connection refused - connect(2) for \"192.168.53.112\" port 10250)"
...
2020-11-05 16:15:14 +0000 [error]: #0 Timer detached. title=:cadvisor_metric_scraper

container is not restarted, as there is no liveness check defined and processes are still running in container while error occurs.

k exec my-splunk-connect-splunk-kubernetes-metrics-h9q96 ps ax
  PID TTY      STAT   TIME COMMAND
    1 ?        Ssl    0:02 /usr/bin/fluentd -c /fluentd/etc/fluent.conf
    9 ?        Sl     0:08 /usr/bin/ruby -Eascii-8bit:ascii-8bit -r/usr/local/share/gems/gems/bundler-2.1.4/lib/bundler/setup /usr/bin/fluentd -c /fluentd/etc/fluent.conf --under-supervisor
  162 pts/0    Ss+    0:00 /bin/bash
  168 ?        Rs     0:00 ps ax

What you expected to happen:

metrics scraping is restarted when kublet is available again.

How to reproduce it (as minimally and precisely as possible):

1. install sck
2. stop kubelet for some some seconds
3. start kubelet
4. see errors in logs metrics are not delivered anymore

Anything else we need to know?:

reproduceable in openshift and k8s.

Environment:

• Kubernetes version (use kubectl version): kubernetes v1.18.9 and openshift v4.5.13
• Ruby version (use ruby --version):
• OS (e.g: cat /etc/os-release): ubuntu 18.04 and coreos 4.5
• Splunk version: 8.0.5.1
• Splunk Connect for Kubernetes helm chart version: v1.4.3
• Others:

[schose]

I added a liveness probe in my fork which i adjusted from here fluentd-gcp

you also need to adjust your values.yml .

splunk-kubernetes-metrics:
...
  buffer:
  "@type": file
  "path": "/tmp/fluentd-buffers"

seems to reboot the container reliable when being in the stuck state.