Pulsar: [Helm] Bookeeper + Autorecovery error state - could not find or load main class

[MisterTimn]

Describe the bug
When trying to deploy both the 2.5.0 and 2.4.2 helm charts on kubernetes v1.17.2, autorecovery and bookkeeper instances go into error state.
It seems that they arent passed a proper main class definition, instead they try to load "-Xms for example...

Autorecovery logs

[conf/pulsar_env.sh] Applying config PULSAR_MEM = " -Xms64m -Xmx128m "
[conf/bookkeeper.conf] Applying config zkServers = pulsar-zookeeper-0.pulsar-zookeeper
JMX enabled by default
Error: Could not find or load main class "

Bookkeeper logs

[conf/pulsar_env.sh] Applying config PULSAR_MEM = "-Xms128m -Xmx256m -XX:MaxDirectMemorySize=128m -Dio.netty.leakDetectionLevel=disabled -Dio.netty.recycler.linkCapacity=1024 -XX:+UseG1GC -XX:MaxGCPauseMillis=10 -XX:+ParallelRefProcEnabled -XX:+UnlockExperimentalVMOptions -XX:+AggressiveOpts -XX:+DoEscapeAnalysis -XX:ParallelGCThreads=32 -XX:ConcGCThreads=32 -XX:G1NewSizePercent=50 -XX:+DisableExplicitGC -XX:-ResizePLAB -XX:+ExitOnOutOfMemoryError -XX:+PerfDisableSharedMem -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCApplicationStoppedTime -XX:+PrintHeapAtGC -verbosegc -XX:G1LogLevel=finest"
[conf/bookkeeper.conf] Applying config autoRecoveryDaemonEnabled = false
[conf/bookkeeper.conf] Applying config dbStorage_readAheadCacheMaxSizeMb = 32
[conf/bookkeeper.conf] Applying config dbStorage_rocksDB_blockCacheSize = 268435456
[conf/bookkeeper.conf] Applying config dbStorage_writeCacheMaxSizeMb = 32
[conf/bookkeeper.conf] Applying config journalMaxSizeMB = 2048
[conf/bookkeeper.conf] Applying config statsProviderClass = org.apache.bookkeeper.stats.prometheus.PrometheusMetricsProvider
[conf/bookkeeper.conf] Applying config useHostNameAsBookieID = true
[conf/bookkeeper.conf] Applying config zkServers = pulsar-zookeeper-0.pulsar-zookeeper
[conf/pulsar_env.sh] Applying config PULSAR_MEM = "-Xms128m -Xmx256m -XX:MaxDirectMemorySize=128m -Dio.netty.leakDetectionLevel=disabled -Dio.netty.recycler.linkCapacity=1024 -XX:+UseG1GC -XX:MaxGCPauseMillis=10 -XX:+ParallelRefProcEnabled -XX:+UnlockExperimentalVMOptions -XX:+AggressiveOpts -XX:+DoEscapeAnalysis -XX:ParallelGCThreads=32 -XX:ConcGCThreads=32 -XX:G1NewSizePercent=50 -XX:+DisableExplicitGC -XX:-ResizePLAB -XX:+ExitOnOutOfMemoryError -XX:+PerfDisableSharedMem -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCApplicationStoppedTime -XX:+PrintHeapAtGC -verbosegc -XX:G1LogLevel=finest"
[conf/pulsar_env.sh] Applying config PULSAR_MEM = "-Xms128m -Xmx256m -XX:MaxDirectMemorySize=128m -Dio.netty.leakDetectionLevel=disabled -Dio.netty.recycler.linkCapacity=1024 -XX:+UseG1GC -XX:MaxGCPauseMillis=10 -XX:+ParallelRefProcEnabled -XX:+UnlockExperimentalVMOptions -XX:+AggressiveOpts -XX:+DoEscapeAnalysis -XX:ParallelGCThreads=32 -XX:ConcGCThreads=32 -XX:G1NewSizePercent=50 -XX:+DisableExplicitGC -XX:-ResizePLAB -XX:+ExitOnOutOfMemoryError -XX:+PerfDisableSharedMem -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCApplicationStoppedTime -XX:+PrintHeapAtGC -verbosegc -XX:G1LogLevel=finest"
Error: Could not find or load main class "-Xms128m
Heap
 garbage-first heap   total 131072K, used 7168K [0x00000000f0000000, 0x00000000f0100400, 0x0000000100000000)
  region size 1024K, 8 young (8192K), 0 survivors (0K)
 Metaspace       used 2881K, capacity 4480K, committed 4480K, reserved 1056768K
  class space    used 302K, capacity 384K, committed 384K, reserved 1048576K

To Reproduce
Steps to reproduce the behavior:

1. Checkout either master, branch 2.5.0 or 2.4.2
2. package the helm charts (either with latest helm3 or helm2 version)
3. Deploy helm chart with default values or mini-values.yaml included
4. check with kubectl get pods and kubectl get logs

Expected behavior
Clean release of pulsar on Kubernetes, available Bookkeeper nodes etc.

Screenshots

NAME                                   READY   STATUS             RESTARTS   AGE
pulsar-autorecovery-5c95797dff-tr27f   0/1     CrashLoopBackOff   5          5m58s
pulsar-bastion-84c76c6998-rj9nb        1/1     Running            0          5m58s
pulsar-bookkeeper-0                    0/1     CrashLoopBackOff   5          5m58s
pulsar-broker-8b8ff58f5-p54w9          0/1     CrashLoopBackOff   5          5m58s
pulsar-dashboard-74fd69f4b5-s6h56      1/1     Running            0          5m58s
pulsar-grafana-7dfdc79bb4-5t44r        1/1     Running            0          5m58s
pulsar-prometheus-89b474499-kvl24      1/1     Running            0          5m58s
pulsar-proxy-7c69fc6df5-2mwbw          1/1     Running            0          5m58s
pulsar-zookeeper-0                     1/1     Running            0          5m58s
pulsar-zookeeper-metadata-nl8x9        0/1     Completed          0          5m58s

Desktop (please complete the following information):

• Manjaro (Arch)
  
  
  
  • Kubernetes 1.17.3
  
  
  • Helm 2.16.3 and 3.1.0
  
  

Additional context
Add any other context about the problem here.

[sijie]

@MisterTimn in the values.yml file, can you changePULSAR_MEM to BOOKIE_MEM for bookie and auto-recovery configmap?

[MisterTimn]

OK. That solved the issue!
Both autorecovery and bookkeeper come up now.

[jharris-]

I hit the same problem for the autorecovery pod and @sijie 's configmap change fixed the issue for me