Charts: [bitnami/Cassandra] Issue: Cassandra fails at initial setup without minimized HeapSize

[druesendieb]

I got the same issue as @konradmalik in #1541
Since it wasn't solved and I cant reopen #1541, here I am ;)

I've attached a minimalized version of my chart which allowed me at least on 2 platforms to reproduce the problem.

Which chart:
bitnami/Cassandra 5.5.3, Cassandra 3.11.6

Describe the bug
Trying to deploy Cassandra in a cluster with 3 nodes, it fails on initial startup of the first node, same as OP.

To Reproduce
Steps to reproduce the behaviour:

1. Install the following chart:

image:
  registry: docker.io
  repository: bitnami/cassandra
  tag: 3.11.6-debian-10-r138
  pullPolicy: IfNotPresent
  debug: true
volumePermissions:
  enabled: true
  image:
    registry: docker.io
    repository: bitnami/minideb
    tag: buster
    pullPolicy: Always
  resources:
    limits: {} 
    requests: {} 
service:
  type: ClusterIP
  port: 9042
  thriftPort: 9160
  nodePorts:
    cql: ""
    thriftPort: ""
  annotations: {}
persistence:
  enabled: true
  annotations:
  accessModes:
    - ReadWriteOnce
  size: 100Gi
resources:
  limits: 
    cpu: 8
    memory: 8Gi
  requests:
    cpu: 4
    memory: 8Gi
cluster:
  name: cassandra
  replicaCount: 3
  seedCount: 2
  numTokens: 256
  datacenter: dc1
  rack: rack1
  enableRPC: false
  endpointSnitch: GossipingPropertyFileSnitch
  pdbEnabled: true
  minAvailable: 2
  internodeEncryption: none
  clientEncryption: false
  domain: cluster.local
jvm:
  extraOpts:
dbUser:
  user: cassandra
  forcePassword: false 
livenessProbe:
  enabled: true
  initialDelaySeconds: 60
  periodSeconds: 30
  timeoutSeconds: 5
  successThreshold: 1
  failureThreshold: 5
readinessProbe:
  enabled: true
  initialDelaySeconds: 60
  periodSeconds: 10
  timeoutSeconds: 5
  successThreshold: 1
  failureThreshold: 5
podAnnotations: {}
podLabels: {}
affinity: {}
nodeSelector: {}
tolerations: []
statefulset:
  updateStrategy: OnDelete 
securityContext:
  enabled: true
  runAsUser: 1234
  runAsGroup: 1234
  fsGroup: 1234
entrypoint: "/entrypoint.sh"
cmd: "/run.sh"
networkPolicy:
  enabled: false
metrics:
  enabled: false
  image:
    registry: docker.io
    pullPolicy: IfNotPresent
    repository: bitnami/cassandra-exporter
    tag: 2.3.4-debian-10-r119
  resources:  
    limits: {} 
    requests: {}
  podAnnotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "8080"

1. Startup fails /opt/bitnami/scripts/libos.sh: line 210: 205 Killed "${cmd[@]}" "${args[@]}" > "$logger" 2>&1

Heres the whole pods log:

Setting node as password seeder
cassandra 16:53:48.10 
cassandra 16:53:48.10 Welcome to the Bitnami cassandra container
cassandra 16:53:48.10 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-cassandra
cassandra 16:53:48.10 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-cassandra/issues
cassandra 16:53:48.10 
cassandra 16:53:48.10 INFO  ==> ** Starting Cassandra setup **
cassandra 16:53:48.12 INFO  ==> Validating settings in CASSANDRA_* env vars..
cassandra 16:53:48.15 WARN  ==> Hostname db002-cassandra-1.db002-cassandra-headless.cassandra.svc.cluster.local could not be resolved. This could lead to connection issues
cassandra 16:53:48.16 WARN  ==> Hostname db002-cassandra-1.db002-cassandra-headless.cassandra.svc.cluster.local could not be resolved. This could lead to connection issues
cassandra 16:53:48.16 INFO  ==> Initializing Cassandra database...
cassandra 16:53:48.17 DEBUG ==> No injected cqlshrc.sample file found. Creating default cqlshrc.sample file
cassandra 16:53:48.18 DEBUG ==> No injected cassandra-rackdc.properties file found. Creating default cassandra-rackdc.properties file
cassandra 16:53:48.19 DEBUG ==> No injected commitlog_archiving.properties file found. Creating default commitlog_archiving.properties file
cassandra 16:53:48.20 DEBUG ==> No injected jvm.options file found. Creating default jvm.options file
cassandra 16:53:48.21 DEBUG ==> No injected metrics-reporter-config-sample.yaml file found. Creating default metrics-reporter-config-sample.yaml file
cassandra 16:53:48.23 DEBUG ==> No injected triggers/README.txt file found. Creating default triggers/README.txt file
cassandra 16:53:48.24 DEBUG ==> No injected README.txt file found. Creating default README.txt file
cassandra 16:53:48.25 DEBUG ==> No injected cassandra.yaml file found. Creating default cassandra.yaml file
cassandra 16:53:48.25 DEBUG ==> No injected cassandra-topology.properties file found. Creating default cassandra-topology.properties file
cassandra 16:53:48.26 DEBUG ==> No injected cassandra-jaas.config file found. Creating default cassandra-jaas.config file
cassandra 16:53:48.27 DEBUG ==> No injected cassandra-env.ps1 file found. Creating default cassandra-env.ps1 file
cassandra 16:53:48.28 DEBUG ==> No injected hotspot_compiler file found. Creating default hotspot_compiler file
cassandra 16:53:48.29 DEBUG ==> No injected cassandra-env.sh file found. Creating default cassandra-env.sh file
cassandra 16:53:48.30 DEBUG ==> No injected logback.xml file found. Creating default logback.xml file
cassandra 16:53:48.31 DEBUG ==> No injected logback-tools.xml file found. Creating default logback-tools.xml file
cassandra 16:53:48.44 DEBUG ==> Ensuring expected directories/files exist...
cassandra 16:53:48.45 INFO  ==> Deploying Cassandra from scratch
cassandra 16:53:48.46 INFO  ==> Starting Cassandra
cassandra 16:53:48.46 INFO  ==> Checking that it started up correctly
cassandra 16:53:48.46 DEBUG ==> Checking that log /opt/bitnami/cassandra/logs/cassandra_first_boot.log contains entry "Starting listening for CQL clients"
/opt/bitnami/scripts/libos.sh: line 210:   205 Killed                  "${cmd[@]}" "${args[@]}" > "$logger" 2>&1

1. /opt/bitnami/cassandra/logs/cassandra_first_boot.log is empty as well.

Expected behaviour:
Cluster coming up without errors.

Version of Helm and Kubernetes:

• Output of helm version:

version.BuildInfo{Version:"v3.2.1", GitCommit:"fe51cd1e31e6a202cba7dead9552a6d418ded79a", GitTreeState:"clean", GoVersion:"go1.13.10"}

• Output of kubectl version:

Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.3", GitCommit:"2e7996e3e2712684bc73f0dec0200d64eec7fe40", GitTreeState:"clean", BuildDate:"2020-05-20T12:52:00Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.5", GitCommit:"e0fccafd69541e3750d460ba0f9743b90336f24f", GitTreeState:"clean", BuildDate:"2020-04-16T11:35:47Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

Workaround
To manually minimize the JVM Heap Size, proposed by @rafariossaa, works well but I think it should be avoidable to use this on a system intended for production.

Chart settings:

jvm:
  maxHeapSize: 1G
  newHeapSize: 800M

Additional Information
I've encountered this problem now on 2 different platforms

• Rancher v2.4.3 (above k8s version information)
• CCP 6.1

_Originally posted by @druesendieb in https://github.com/bitnami/charts/issues/1541#issuecomment-657745103_

[marcosbc]

To manually minimize the JVM Heap Size, proposed by @rafariossaa, works well but I think it should be avoidable to use this on a system intended for production.

I'm not sure how we can fix this in the chart. The problem looks like it is Cassandra eating too many resources, more than what the cluster can offer. And we want to avoid setting default values for the users, as is mentioned in values.yaml.

[druesendieb]

Both clusters I've tested this on have more resources available than I've set as request/limit so that actually shouldn't be the case.

[marcosbc]

From the Cassandra jvm.options file:

# Heap size is automatically calculated by cassandra-env based on this
# formula: max(min(1/2 ram, 1024MB), min(1/4 ram, 8GB))
# That is:
# - calculate 1/2 ram and cap to 1024MB
# - calculate 1/4 ram and cap to 8192MB
# - pick the max
#
# For production use you may wish to adjust this for your environment.
# If that's the case, uncomment the -Xmx and Xms options below to override the
# automatic calculation of JVM heap memory.
#
# It is recommended to set min (-Xms) and max (-Xmx) heap sizes to
# the same value to avoid stop-the-world GC pauses during resize, and
# so that we can lock the heap in memory on startup to prevent any
# of it from being swapped out.

It could be that Cassandra detects a heap size of 8GB, so it tries to take that the limits you set to that same value. Note there are other processes eating memory/CPU so it eventually crashes.

I'm not sure if there's anything we should do on the chart to 'fix' this issue. In your case, it looks like you should either set a higher limit, or to configure the heap sizes.

[druesendieb]

Aah, now I see what you've meant with your previous comment.
You weren't talking about the resources the Cassandra workloads were using on the Kubernetes cluster but about the resources inside of the actual Cassandra Cluster. Sorry I misunderstood, my bad.

Thanks for the hint with cassandra-env.sh.
I did some research and tests now and I think I found the problem.
cassandra-env.sh assumes your memory and your CPU cores based on this part of the script:

system_memory_in_mb=`free -m | awk '/:/ {print $2;exit}'`
system_cpu_cores=`egrep -c 'processor([[:space:]]+):.*' /proc/cpuinfo`

On a normal Linux host that would be no problem, but in a container it is.

When I run the above in my container, I get the resources of the node where the container is running on and not of the container itself. This results in your described behaviour, detecting a heap size of 8GB and thus eating away all the available memory, then crashing.
After a quick google search I was more able to understand the problem and furthermore got better results using cat /sys/fs/cgroup/cpu/cpu.shares for CPU shares and cat /sys/fs/cgroup/memory/memory.limit_in_bytes for memory.

This leaves me with:

maxHeapSize: 2G
newHeapSize: 800M

I'm not sure if there's anything we should do on the chart to 'fix' this issue. In your case, it looks like you should either set a higher limit, or to configure the heap sizes.

You're right - the chart per se can't fix this problem while leaving the underlying problem be.
Imho, if you want to fix this, you need to do this in the container directly with a rework of cassandra-env.sh, adopted to a container environment.

What you could do in the chart:

• A quick and dirty workaround would be to do the calculation based on the resources if set in values.yaml.
• The easy way would be to add some comments for people like me, running on environments with little resources, and trusting in Memory settings: these are calculated automatically 😉

Here's a proposition:

## JVM Settings
##
jvm:
  ## Extra JVM options
  ##
  extraOpts:

  ## Memory settings: These are calculated automatically
  ## unless specified otherwise
  ## To run on environments with little resources (<= 8GB), tune your heap settings:
  ## maxHeapSize:
  ## - calculate 1/2 ram and cap to 1024MB
  ## - calculate 1/4 ram and cap to 8192MB
  ## - pick the max
  ## newHeapSize:  
  ## A good guideline is 100 MB per CPU core.
  ## - min(100 * num_cores, 1/4 * heap size)
  ## ref: https://docs.datastax.com/en/archived/cassandra/2.0/cassandra/operations/ops_tune_jvm_c.html
  ##
  # maxHeapSize: 4G
  # newHeapSize: 800M

Thanks a lot!

[marcosbc]

Thanks for sharing! That makes a lot of sense. I've created #3179 to implement those changes.