Docker-sonarqube: Crashes immediately when starting in Kubernetes (AKS) - elastic search exits

[tdevopsottawa]

I'm attempting to start SonarQube in an Azure Kubernetes Service environment, with a persistent volume claim, and the pod keeps failing to start with this error:

2019.04.11 20:21:03 INFO app[][o.s.a.AppFileSystem] Cleaning or creating temp directory /opt/sonarqube/temp
2019.04.11 20:21:03 INFO app[][o.s.a.es.EsSettings] Elasticsearch listening on /127.0.0.1:9001
2019.04.11 20:21:03 INFO app[][o.s.a.p.ProcessLauncherImpl] Launch process[[key='es', ipcIndex=1, logFilenamePrefix=es]] from [/opt/sonarqube/elasticsearch]: /opt/sonarqube/elasticsearch/bin/elasticsearch
2019.04.11 20:21:03 INFO app[][o.s.a.SchedulerImpl] Waiting for Elasticsearch to be up and running
2019.04.11 20:21:04 INFO app[][o.e.p.PluginsService] no modules loaded
2019.04.11 20:21:04 INFO app[][o.e.p.PluginsService] loaded plugin [org.elasticsearch.transport.Netty4Plugin]
2019.04.11 20:21:09 WARN app[][o.s.a.p.AbstractProcessMonitor] Process exited with exit value [es]: 1
2019.04.11 20:21:09 INFO app[][o.s.a.SchedulerImpl] Process [es] is stopped
2019.04.11 20:21:09 INFO app[][o.s.a.SchedulerImpl] SonarQube is stopped

A thorough internet search shows that many, many people have this exact issue, but no one has been able to solve it. It apparently relates to how elastic search refuses to be run as root user, yet this is running a docker container that runs as sonarqube user, not root, so this can't be my problem.

This is related to persistent volume use, as if I disable persistent volumes from the setup, it works just fine. This error emerges only when I try to mount a pvc with the deployment.

[out-of-band]

Linking to the issue filed over in the Helm chart repository (almost 3 months ago): https://github.com/helm/charts/issues/11304

[Zenuka]

I had the same issue but running chown -R 999:999 to /opt/sonarqube/data and /opt/sonarqube/extensions fixed the issue for me.
My full yaml file: (interesting parts are in initContainers)

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    app: sonarqube
  name: sonarqube
  namespace: sonarqube
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: sonarqube
    spec:
      terminationGracePeriodSeconds: 15
      initContainers:
      - name: fix-the-volume-permission
        image: busybox
        command:
        - sh
        - -c
        - chown -R 999:999 /opt/sonarqube/extensions
        securityContext:
          privileged: true
        volumeMounts:
        - name: sonar-extensions
          mountPath: /opt/sonarqube/extensions
      - name: fix-the-volume-permission1
        image: busybox
        command:
        - sh
        - -c
        - chown -R 999:999 /opt/sonarqube/data
        securityContext:
          privileged: true
        volumeMounts:
        - name: sonar-data
          mountPath: /opt/sonarqube/data
      containers:
        - name: sonarqube
          image: sonarqube:7.7-community
          resources:
            requests:
              cpu: 500m
              memory: 1024Mi
            limits:
              cpu: 2000m
              memory: 2048Mi
          volumeMounts:
          - mountPath: "/opt/sonarqube/data/"
            name: sonar-data
            subPath: data
          - mountPath: /opt/sonarqube/extensions/plugins/tmp
            name: sonar-extensions
            subPath: tmp
          - mountPath: /opt/sonarqube/extensions/downloads
            name: sonar-extensions
            subPath: downloads
          - mountPath: /opt/sonarqube/extensions/plugins
            name: sonar-extensions
            subPath: plugins
          - mountPath: /opt/sonarqube/temp
            name: sonar-data
            subPath: temp
          - mountPath: /opt/sonarqube/logs
            name: sonar-extensions
            subPath: logs
          - mountPath: /tmp
            name: tmp-dir
          env:
          - name: sonar.jdbc.username
            value: [DATABASE USERNAME]
          - name: sonar.jdbc.url
            value: [DATABASE URL]
          - name: sonar.jdbc.password
            valueFrom:
              secretKeyRef:
                name: sqlserver
                key: password
          - name: sonar.web.context
            value: /sonar
          - name: sonar.log.level
            value: TRACE
          ports:
          - containerPort: 9000
            protocol: TCP
      volumes:
      - name: sonar-data
        persistentVolumeClaim:
          claimName: sonar-data
      - name: sonar-extensions
        persistentVolumeClaim:
          claimName: sonar-extensions
      - name : tmp-dir
        emptyDir: {}

[andrei-dascalu]

@Zenuka this also helps with https://github.com/SonarSource/docker-sonarqube/issues/269 when mounting local folders and changing rights

Strange that it affects the default example of running with docker :)

[jcoeltjen]

Instead of using an init container to fix file permissions you can also set the owner group id of all attached volumes by specifying the fsGroup.
To do that set add to the pod spec:

...
securityContext:
  fsGroup: 999 # 999 -> sonarqube group 

[jcoeltjen]

When running the helm chart (stable/sonarqube 2.1.1) on AKS it runs without any problems for (when creating a new PVC).

[strideynet]

Also getting this issue. Any progress?

[denghejun]

got same issue here

[llilloo]

This is still an issue on sonarqube:7.9-community docker image. Even setting fsGroup on the spec of the deployment file doesn't fix this.

[markyjackson-taulia]

I just ran a test and had the same issue

[markyjackson-taulia]

To add, I even tried using katacoda and it failed as well: https://www.katacoda.com/javajon/courses/kubernetes-pipelines/sonarqube

[markyjackson-taulia]

2020.02.07 15:37:14 WARN web[][o.a.c.l.WebappClassLoaderBase] The web application [ROOT] appears to have started a thread named [elasticsearch[_client_][generic][T#4]] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread: [email protected]/jdk.internal.misc.Unsafe.park(Native Method)

[tobias-trabelsi-sonarsource]

Hello everyone.

as this thread is rather old i will go ahead an close it for now. please not that there is now also a helm chart published by sonarsource that might interest some of you :)