Splunk-connect-for-kubernetes: Upgrading 1.2.0 to 1.3.0 - logging pods fail with error="'ingest_api_tenant' parameter is required"

Created on 18 Dec 2019 · 5Comments · Source: splunk/splunk-connect-for-kubernetes

What happened:
Installed 1.3.0 the same way I installed 1.2.0 and previous releases:
helm install --name splunk-connect -f ${WELLTOK_ENVIRONMENT}-values.yml https://github.com/splunk/splunk-connect-for-kubernetes/releases/download/1.3.0/splunk-connect-for-kubernetes-1.3.0.tgz
Logging pods fail with error:

fluentd -c /fluentd/etc/fluent.conf
2019-12-18 17:29:11 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
/usr/local/bundle/gems/activesupport-3.2.22.5/lib/active_support/core_ext/object/duplicable.rb:111: warning: BigDecimal.new is deprecated; use BigDecimal() method instead.
2019-12-18 17:29:12 +0000 [error]: config error in:
<match **>
  @type splunk_ingest_api
  host "ip-10-52-46-109.ec2.internal"
  source_key "source"
  sourcetype_key "sourcetype"
  <fields>
    pod
    namespace
    container_name
    container_id
    cluster_name
  </fields>
  <buffer>
    @type memory
    chunk_limit_records 100000
    chunk_limit_size 200m
    flush_interval 5s
    flush_thread_count 1
    overflow_action block
    retry_max_times 3
    total_limit_size 600m
  </buffer>
  <format monitor_agent>
    @type json
  </format>
  <format>
    @type single_value
    message_key log
    add_newline false
  </format>
</match>

2019-12-18 17:29:12 +0000 [error]: config error file="/fluentd/etc/fluent.conf" error_class=Fluent::ConfigError error="'ingest_api_tenant' parameter is required"

What you expected to happen:
Splunk logging
How to reproduce it (as minimally and precisely as possible):
Install above.
Anything else we need to know?:
values file:

# This is an example of using insecure configurations
global:
  splunk:
    hec:
      token: <redacted>
      host: <redacted>
      port: 8088
      indexName: kubernetes-nonprod-logs
      # connection to splunk is insecure
      insecureSSL: true
  kubernetes:
    clusterName: tester1

splunk-kubernetes-logging:
  logs:
    kube-apiserver-audit:
      from:
        file:
          path: /var/log/kube-apiserver-tester1-audit.log
  kubernetes:
    clusterName: tester1

splunk-kubernetes-objects:
  # RBAC is disabled
  rbac:
    create: true
  splunk:
    hec:
      indexName: kubernetes-nonprod-objects
  kubernetes:
    # connection to kubernetes is insecure
    insecureSSL: true
    clusterName: tester1

splunk-kubernetes-metrics:
  # RBAC is disabled
  rbac:
    create: true
  # do not create service account
  serviceAccount:
    create: true
  splunk:
    hec:
      indexName: kubernetes-nonprod-metrics
  kubernetes:
    clusterName: tester1

Environment:

Kubernetes version (use kubectl version):
» kubectl version

Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.7", GitCommit:"6c143d35bb11d74970e7bc0b6c45b6bfdffc0bd4", GitTreeState:"clean", BuildDate:"2019-12-11T12:42:56Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.7", GitCommit:"6c143d35bb11d74970e7bc0b6c45b6bfdffc0bd4", GitTreeState:"clean", BuildDate:"2019-12-11T12:34:17Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}

Splunk version:

Splunk Enterprise
Version:
7.3.2
Build:
c60db69f8e32
Server:
wtsh1

Others:

Source

dmcnaught

Most helpful comment

ok, adding it to splunk logging too worked:

splunk-kubernetes-logging:
  logs:
    kube-apiserver-audit:
      from:
        file:
          path: /var/log/kube-apiserver-tester-audit.log
  kubernetes:
    clusterName: tester  
  splunk:
    hec:
      token: <redacted>
      host: <redacted>
      port: 8088
      indexName: kubernetes-nonprod-logs
      # connection to splunk is insecure
      insecureSSL: true

dmcnaught on 20 Dec 2019

👍2

All 5 comments

Hello, it shouldnt have been interpreted to send log to ingest API. But somehow it is. Could you try with also setting splunk.hec values in logging as well as global?

rockb1017 on 20 Dec 2019

ok, adding it to splunk logging too worked:

splunk-kubernetes-logging:
  logs:
    kube-apiserver-audit:
      from:
        file:
          path: /var/log/kube-apiserver-tester-audit.log
  kubernetes:
    clusterName: tester  
  splunk:
    hec:
      token: <redacted>
      host: <redacted>
      port: 8088
      indexName: kubernetes-nonprod-logs
      # connection to splunk is insecure
      insecureSSL: true

dmcnaught on 20 Dec 2019

👍2

There is still an issue here @chaitanyaphalak - we shouldn't have to define splunk.hec values in global and logging sections of a template.

dmcnaught on 6 Jan 2020

👍1

This is still broken. Duplicating (BAD!) the hec section for splunk-kubernetes-logging does fix the issue. This is a workaround though, not a solution. Please re-open this issue.