Splunk-connect-for-kubernetes: Question: Upgrading chart does not redeploy logging containers

[paulalex]

Hi,

I am trying to upgrade the splunk connector release using the following helm command:

helm upgrade --tiller-namespace splunk-connect-k8s splunk-connect-k8s -f values.yaml custom-splunk-connect-for-kubernetes.tgz

Everytime I redeploy it creates new containers for the metrics and meta aspects of the release but it never redeploys the kubernetes logging and I need this to redeploy as well.

splunk-eks (develop) $ k get pods
NAME                                                              READY   STATUS    RESTARTS   AGE
splunk-connect-k8s-splunk-kubernetes-logging-ltrt7                1/1     Running   0          98m
splunk-connect-k8s-splunk-kubernetes-logging-p7snv                1/1     Running   0          102m
splunk-connect-k8s-splunk-kubernetes-logging-zddff                1/1     Running   0          102m
splunk-connect-k8s-splunk-kubernetes-metrics-49nlm                1/1     Running   0          42s
splunk-connect-k8s-splunk-kubernetes-metrics-agg-cc79cfcdbt62rg   1/1     Running   0          63s
splunk-connect-k8s-splunk-kubernetes-metrics-g8wqp                1/1     Running   0          50s
splunk-connect-k8s-splunk-kubernetes-metrics-kwdlr                1/1     Running   0          28s
splunk-connect-k8s-splunk-kubernetes-objects-84b4c9596c-drrjn     1/1     Running   0          63s
tiller-deploy-66fb8fc48c-x2wcn                                    1/1     Running   0          102m

Is there a reason why the logging pods do not redeploy and what do I need to do to get them to redeploy?

[rockb1017]

Hello, I am aware of this behavior. we will soon fix it.
in the mean time, you can delete logging pods, it will be restarted automatically and new configMap will be applied.

[paulalex]

Ok this is part of a switch from a self hosted splunk install to cloud where I have amended the logging chart in k8s to send to individual application namespaces.

I am guessing I will lose some logs as I kill the logging pods?

Would an upgrade (if it worked) still result in a loss of logs as the daemon set is redeployed or will the logs 'catch up'?

Sorry for the questions but I dont know much about splunk.

[rockb1017]

the logging pod keeps the record of positions where it has send to splunk, so once it gets restarted, it will pick up from where it left off.

[paulalex]

Great thanks, one last thing if I deleted the helm install because I wanted to rename the release would this still be ok or would I lose logs then?

[rockb1017]

it will still be okay. you won't lose logs.

[paulalex]

Thanks!!

[matthewmodestino]

you can also use the helm command:

helm upgrade myChart -f myValues.yaml ./myChart —recreate-pods

https://helm.sh/docs/helm/#helm-upgrade

[paulalex]

@matthewmodestino I did look and it doesnt appear possible to upgrade and also rename the release at the same time so it looks like I will have to delete it first to recreate with a --name parameter as the currently deployed release name adds randomly generated names.

I have created a pipeline function in jenkins to take care of upgrades and installs for future use but this requires the release name to be set to splunk-connect-k8s.

When I run helm install --name splunk-connect-k8s .... after running a helm delete how does splunk connect know at which point it stopped sending logs? Given when I redeploy I wont be sending logs to the same splunk server anymore.

[matthewmodestino]

Why would you change the release name? Helm tracks in revisions of the release.

Please take a look at how fluentd does checkpointing:

https://docs.fluentd.org/input/tail

[paulalex]

I want to change the release name because it was deployed manually originally and each environment has a random name attached to the release.

I want to deploy and upgrade from a jenkins pipeline and so having a consistent name for the release in each environment makes things a lot easier for me going forwards.

Is this strange?

[matthewmodestino]

Not at all and thanks for sharing!

I recommend checking out the easy install in Splunk App for Infrastructure, as it acts as a template generator, which you could likely plug into your pipelines (by taking a look at how we did it - HINT look in this directory within the app: appserver/static/kubernetes_connect. it has a variable for "release name" that should make it easier to templates it for different enviros.

https://docs.splunk.com/Documentation/InfraApp/1.4.1/Admin/AddDataKubernetes

[paulalex]

Thanks I will take a look!

[chaitanyaphalak]

Resolved in https://github.com/splunk/splunk-connect-for-kubernetes/releases/tag/1.3.0