splunk-connect-for-kubernetes stuck in CrashLoopBackOff

[modevops]

I am trying to install splunk-connect-for-kuberenetes on minikube for testing and learning.
I have sucessfully installed docker-splunk in mini-kube but when I try to deply splunk-connect-for-kubernetes i get error below.
splunk install i have replaced hec token with x's
export MONITORING_MACHINE=https://192.168.99.103:8443 && export HEC_TOKEN=xxxxxxxxxxxxxxxx && export HEC_PORT=8088 && export GLOBAL_HEC_INSECURE_SSL=true && export OBJECTS_INSECURE_SSL=true && export METRICS_INSECURE_SSL=true && export JOURNALD_PATH=/run/log/journal && export KUBELET_PROTOCOL=https && export METRICS_INDEX=k8_metrics && export LOG_INDEX=k8_events && export META_INDEX=k8_objects && export CLUSTER_NAME=mini-splunk && export HELM_RELEASE_NAME=splunk-k8s && export KUBERNETES_NAMESPACE=splunk && export CORE_OBJ=pods,nodes && export APPS_OBJ= && files=("kubernetes_connect_template.yaml" "deploy_sck_k8s.sh") && for each in "${files[@]}"; do wget -O- --no-check-certificate http://192.168.99.103:31800/en-US/static/app/splunk_app_infrastructure/kubernetes_connect/"$each" > $each; done && wget https://github.com/splunk/splunk-connect-for-kubernetes/releases/download/1.2.0/splunk-connect-for-kubernetes-1.2.0.tgz -O splunk-connect-for-kubernetes.tgz && bash deploy_sck_k8s.sh

Errors and logs
sed: 1: "s/${monitoring_machine} ...": bad flag in substitute command: '/'

chown: /fluentd/etc/splunk/..data: Read-only file system
chown: /fluentd/etc/splunk/splunk_hec_token: Read-only file system
chown: /fluentd/etc/splunk/..2019_10_22_01_09_22.178706774/splunk_hec_token: Read-only file system
chown: /fluentd/etc/splunk/..2019_10_22_01_09_22.178706774: Read-only file system
chown: /fluentd/etc/splunk/..2019_10_22_01_09_22.178706774: Read-only file system
chown: /fluentd/etc/splunk: Read-only file system
chown: /fluentd/etc/splunk: Read-only file system
chown: /fluentd/etc/..2019_10_22_01_09_22.899916051/fluent.conf: Read-only file system
chown: /fluentd/etc/..2019_10_22_01_09_22.899916051: Read-only file system
chown: /fluentd/etc/..2019_10_22_01_09_22.899916051: Read-only file system
chown: /fluentd/etc/..data: Read-only file system
chown: /fluentd/etc/fluent.conf: Read-only file system
chown: /fluentd/etc: Read-only file system
chown: /fluentd/etc: Read-only file system
2019-10-22 01:26:01 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
2019-10-22 01:26:02 +0000 [error]: config error file="/fluentd/etc/fluent.conf" error_class=Fluent::ConfigError error="hec_host (${monitoring_machine}) and/or hec_port (8088) are invalid."

Successfully assigned splunk/splunk-k8s-splunk-kubernetes-logging-w7jb2 to minikube
default-scheduler
-
1
Pulling image "splunk/fluentd-hec:1.1.1"
kubelet minikube
spec.containers{splunk-fluentd-k8s-logs}
5
Successfully pulled image "splunk/fluentd-hec:1.1.1"
kubelet minikube
spec.containers{splunk-fluentd-k8s-logs}
4
Created container splunk-fluentd-k8s-logs
kubelet minikube
spec.containers{splunk-fluentd-k8s-logs}
4
Started container splunk-fluentd-k8s-logs
kubelet minikube
spec.containers{splunk-fluentd-k8s-logs}
4
warning
Back-off restarting failed container
kubelet minikube
spec.containers{splunk-fluentd-k8s-logs}
72

[matthewmodestino]

Hey @modevops!

Looks like the fluentd.conf is rendering something it doesn't like..We dump the rendered config to the pod logs, so lets check kubectl -n <yourNamespace> logs <yourPod>. if you can share the config in the logs i can likely pick out the issue.

[modevops]

Here is what I see in the logs
splunk-k8s-splunk-kubernetes-metrics-agg-5b68fb879f-27577
splunk-k8s-splunk-kubernetes-metrics-cb4c9
[splunk-k8s-splunk-kubernetes-metrics-cb4c9] chown: /fluentd/etc/splunk/..data: Read-only file system
[splunk-k8s-splunk-kubernetes-metrics-cb4c9] chown: /fluentd/etc/splunk/splunk_hec_token: Read-only file system
[splunk-k8s-splunk-kubernetes-metrics-cb4c9] chown: /fluentd/etc/splunk/..2019_10_25_00_02_47.059959492/splunk_hec_token: Read-only file system
[splunk-k8s-splunk-kubernetes-metrics-cb4c9] chown: /fluentd/etc/splunk/..2019_10_25_00_02_47.059959492: Read-only file system
[splunk-k8s-splunk-kubernetes-metrics-cb4c9] chown: /fluentd/etc/splunk/..2019_10_25_00_02_47.059959492: Read-only file system
[splunk-k8s-splunk-kubernetes-metrics-agg-5b68fb879f-27577] chown: /fluentd/etc/splunk/..data: Read-only file system
[splunk-k8s-splunk-kubernetes-metrics-cb4c9] chown: /fluentd/etc/splunk: Read-only file system
[splunk-k8s-splunk-kubernetes-metrics-agg-5b68fb879f-27577] chown: /fluentd/etc/splunk/splunk_hec_token: Read-only file system
[splunk-k8s-splunk-kubernetes-metrics-cb4c9] chown: /fluentd/etc/splunk: Read-only file system
[splunk-k8s-splunk-kubernetes-metrics-agg-5b68fb879f-27577] chown: /fluentd/etc/splunk/..2019_10_25_00_02_47.306302291/splunk_hec_token: Read-only file system
[splunk-k8s-splunk-kubernetes-metrics-cb4c9] chown: /fluentd/etc/..data: Read-only file system
[splunk-k8s-splunk-kubernetes-metrics-agg-5b68fb879f-27577] chown: /fluentd/etc/splunk/..2019_10_25_00_02_47.306302291: Read-only file system
[splunk-k8s-splunk-kubernetes-metrics-cb4c9] chown: /fluentd/etc/..2019_10_25_00_02_47.230980271/fluent.conf: Read-only file system
[splunk-k8s-splunk-kubernetes-metrics-agg-5b68fb879f-27577] chown: /fluentd/etc/splunk/..2019_10_25_00_02_47.306302291: Read-only file system
[splunk-k8s-splunk-kubernetes-metrics-cb4c9] chown: /fluentd/etc/..2019_10_25_00_02_47.230980271: Read-only file system
[splunk-k8s-splunk-kubernetes-metrics-agg-5b68fb879f-27577] chown: /fluentd/etc/splunk: Read-only file system
[splunk-k8s-splunk-kubernetes-metrics-cb4c9] chown: /fluentd/etc/..2019_10_25_00_02_47.230980271: Read-only file system
[splunk-k8s-splunk-kubernetes-metrics-agg-5b68fb879f-27577] chown: /fluentd/etc/splunk: Read-only file system
[splunk-k8s-splunk-kubernetes-metrics-cb4c9] chown: /fluentd/etc/fluent.conf: Read-only file system
[splunk-k8s-splunk-kubernetes-metrics-cb4c9] chown: /fluentd/etc: Read-only file system
[splunk-k8s-splunk-kubernetes-metrics-agg-5b68fb879f-27577] chown: /fluentd/etc/..data: Read-only file system
[splunk-k8s-splunk-kubernetes-metrics-cb4c9] chown: /fluentd/etc: Read-only file system
[splunk-k8s-splunk-kubernetes-metrics-agg-5b68fb879f-27577] chown: /fluentd/etc/fluent.conf: Read-only file system
[splunk-k8s-splunk-kubernetes-metrics-agg-5b68fb879f-27577] chown: /fluentd/etc/..2019_10_25_00_02_47.168839470/fluent.conf: Read-only file system
[splunk-k8s-splunk-kubernetes-metrics-agg-5b68fb879f-27577] chown: /fluentd/etc/..2019_10_25_00_02_47.168839470: Read-only file system
[splunk-k8s-splunk-kubernetes-metrics-agg-5b68fb879f-27577] chown: /fluentd/etc/..2019_10_25_00_02_47.168839470: Read-only file system
[splunk-k8s-splunk-kubernetes-metrics-agg-5b68fb879f-27577] chown: /fluentd/etc: Read-only file system
[splunk-k8s-splunk-kubernetes-metrics-agg-5b68fb879f-27577] chown: /fluentd/etc: Read-only file system

[rockb1017]

what is it that you have defined here:
export MONITORING_MACHINE=https://192.168.99.103:8443

and what is this lower case variable
${monitoring_machine} ?

[modevops]

that is the minikube ip
kubectl cluster-info
Kubernetes master is running at https://192.168.99.103:8443

[rockb1017]

you should put Splunk pod's IP address for hec_host.

[rockb1017]

and as far as i know, variables are case sensitive in bash. so MONITORING_MACHINE is not equal to monitoring_machine.

[modevops]

when i fill out the config in infrastructure app this the command it gives me export MONITORING_MACHINE=https://192.168.99.103:8443 && export HEC_TOKEN=xxxxxxxxxxxxxxxx && export HEC_PORT=8088 && export GLOBAL_HEC_INSECURE_SSL=true && export OBJECTS_INSECURE_SSL=true && export METRICS_INSECURE_SSL=true && export JOURNALD_PATH=/run/log/journal && export KUBELET_PROTOCOL=https && export METRICS_INDEX=k8_metrics && export LOG_INDEX=k8_events && export META_INDEX=k8_objects && export CLUSTER_NAME=mini-splunk && export HELM_RELEASE_NAME=splunk-k8s && export KUBERNETES_NAMESPACE=splunk && export CORE_OBJ=pods,nodes && export APPS_OBJ= && files=("kubernetes_connect_template.yaml" "deploy_sck_k8s.sh") && for each in "${files[@]}"; do wget -O- --no-check-certificate http://192.168.99.103:31800/en-US/static/app/splunk_app_infrastructure/kubernetes_connect/"$each" > $each; done && wget https://github.com/splunk/splunk-connect-for-kubernetes/releases/download/1.2.0/splunk-connect-for-kubernetes-1.2.0.tgz -O splunk-connect-for-kubernetes.tgz && bash deploy_sck_k8s.sh

[rockb1017]

i would have to connect you to infrastructure app team.

[matthewmodestino]

I can assist. Monitoring Machine is the host or Virtual IP that is serving HEC. The one used in the original issue is incorrect.

Minikube can be a challenge due to k8s running in a VM and I dont believe SAI supports or was designed for this usecase.

That being said, what I would recommend is trying to deploy directly with HELM.

I have run it successfully in the past with a few modifications.

If you are looking for less headache, i recommend gke as they provide $300 in free credits you can use to deploy to a more realistic enviro

Happy to help in realtime on slack - splk.it/slack, my handle is @mattymo

[rockb1017]

so you launched splunk in kubernetes. can you access it UI?
which ip address do you browse to access its web UI?
did you also create a kuberentes service resource to send traffic to that splunk pod ?
if then you should use that service's cluster IP (or Load Balance IP) as a hec host.
because that is the address kubernetes pods can access.

[rockb1017]

i have deployed splunk on my local (not in docker container) and started minikube. for minikube to access its host server, i need to use 10.0.2.2 by default. and it works fine.

export MONITORING_MACHINE=10.0.2.2 && export HEC_TOKEN=XXXXXXXXX && export HEC_PORT=8088 && export GLOBAL_HEC_INSECURE_SSL=true && export OBJECTS_INSECURE_SSL=true && export METRICS_INSECURE_SSL=true && export JOURNALD_PATH=/run/log/journal && export KUBELET_PROTOCOL=https && export METRICS_INDEX=em_metrics && export LOG_INDEX=main && export META_INDEX=em_meta && export CLUSTER_NAME=sai && export HELM_RELEASE_NAME=sai && export KUBERNETES_NAMESPACE=sai && export CORE_OBJ=pods,nodes,component_statuses,config_maps,namespaces,persistent_volumes,persistent_volume_claims,resource_quotas,services,service_accounts,events && export APPS_OBJ=daemon_sets,deployments,replica_sets,stateful_sets && files=("kubernetes_connect_template.yaml" "deploy_sck_k8s.sh") && for each in "${files[@]}"; do wget -O- --no-check-certificate http://localhost:8000/en-US/static/app/splunk_app_infrastructure/kubernetes_connect/"$each" > $each; done && wget https://github.com/splunk/splunk-connect-for-kubernetes/releases/download/1.2.0/splunk-connect-for-kubernetes-1.2.0.tgz -O splunk-connect-for-kubernetes.tgz && bash deploy_sck_k8s.sh

[matthewmodestino]

Yes, this is what I did as well, as your local machine will talk to k8s via the VM ip and node ports, iirc

You can validate with minikube ssh which will pop u inside the VM, and use ifconfig to grab the IP you need to send to hec or talk to k8s svcs

[chaitanyaphalak]

Closing due to inactivity.