Docker-sonarqube: Error after upgrading from 8.9.1 to 9.0.0

[goraxan]

I just upgraded my existing docker from 8.9.1 to 9.0.0 and it can't start.
This is what I get in the logs:

2021.07.05 17:52:48 INFO  app[][o.s.a.AppFileSystem] Cleaning or creating temp directory /opt/sonarqube/temp
2021.07.05 17:52:48 INFO  app[][o.s.a.es.EsSettings] Elasticsearch listening on [HTTP: 127.0.0.1:9001, TCP: 127.0.0.1:36639]
2021.07.05 17:52:48 INFO  app[][o.s.a.ProcessLauncherImpl] Launch process[[key='es', ipcIndex=1, logFilenamePrefix=es]] from [/opt/sonarqube/elasticsearch]: /opt/sonarqube/elasticsearch/bin/elasticsearch
could not find java in ES_JAVA_HOME at /opt/java/openjdk/bin/java
2021.07.05 17:52:48 WARN  app[][o.s.a.p.AbstractManagedProcess] Process exited with exit value [es]: 1
2021.07.05 17:52:48 INFO  app[][o.s.a.SchedulerImpl] Waiting for Elasticsearch to be up and running
2021.07.05 17:52:48 INFO  app[][o.s.a.SchedulerImpl] Process[es] is stopped
2021.07.05 17:52:48 INFO  app[][o.s.a.SchedulerImpl] SonarQube is stopped

It looks like the ES_JAVA_HOME variable is not correctly set.

[tobias-trabelsi-sonarsource]

Hi @goraxan ,
do you set any environment variable in your deployment and are you using our images as they come from the docker hub? i just checked all images and they start up just fine for me and there is a java binary in /opt/java/openjdk/bin, so the path is correct

[goraxan]

Hi @tobias-trabelsi-sonarsource yes, we use the official images from docker hub. I've also checked if that file is present in the image I'm using (sonarqube:latest) and it is indeed.

We've tried in two different computers (Ubuntu 16.04 LTS), this command doesn't work:

docker run --name sonarqube_new sonarqube:latest
Unable to find image 'sonarqube:latest' locally
latest: Pulling from library/sonarqube
5843afab3874: Pull complete
a131164fad71: Pull complete
d77763c1bc70: Pull complete
572e2a545fb3: Pull complete
f32e9b0d93df: Pull complete
Digest: sha256:d1f18c804d8bdcea0a90d13d93f6ec9af9012d48747fcb63dffb7c8f06b5666f
Status: Downloaded newer image for sonarqube:latest
2021.07.06 09:24:00 INFO  app[][o.s.a.AppFileSystem] Cleaning or creating temp directory /opt/sonarqube/temp
2021.07.06 09:24:00 INFO  app[][o.s.a.es.EsSettings] Elasticsearch listening on [HTTP: 127.0.0.1:9001, TCP: 127.0.0.1:45509]
2021.07.06 09:24:01 INFO  app[][o.s.a.ProcessLauncherImpl] Launch process[[key='es', ipcIndex=1, logFilenamePrefix=es]] from [/opt/sonarqube/elasticsearch]: /opt/sonarqube/elasticsearch/bin/elasticsearch
2021.07.06 09:24:01 INFO  app[][o.s.a.SchedulerImpl] Waiting for Elasticsearch to be up and running
could not find java in ES_JAVA_HOME at /opt/java/openjdk/bin/java
2021.07.06 09:24:01 WARN  app[][o.s.a.p.AbstractManagedProcess] Process exited with exit value [es]: 1
2021.07.06 09:24:01 INFO  app[][o.s.a.SchedulerImpl] Process[es] is stopped
2021.07.06 09:24:01 INFO  app[][o.s.a.SchedulerImpl] SonarQube is stopped

[akatasonov]

I can confirm having the same issue:
docker run -d --name sonarqube -e SONAR_ES_BOOTSTRAP_CHECKS_DISABLE=true -p 9000:9000 sonarqube:latest

Results in:

2021.07.06 10:25:49 INFO  app[][o.s.a.AppFileSystem] Cleaning or creating temp directory /opt/sonarqube/temp                                                                                              
2021.07.06 10:25:49 INFO  app[][o.s.a.es.EsSettings] Elasticsearch listening on [HTTP: 127.0.0.1:9001, TCP: 127.0.0.1:39325]
2021.07.06 10:25:49 INFO  app[][o.s.a.ProcessLauncherImpl] Launch process[[key='es', ipcIndex=1, logFilenamePrefix=es]] from [/opt/sonarqube/elasticsearch]: /opt/sonarqube/elasticsearch/bin/elasticsearch  could not find java in ES_JAVA_HOME at /opt/java/openjdk/bin/java
2021.07.06 10:25:49 WARN  app[][o.s.a.p.AbstractManagedProcess] Process exited with exit value [es]: 1
2021.07.06 10:25:49 INFO  app[][o.s.a.SchedulerImpl] Waiting for Elasticsearch to be up and running
2021.07.06 10:25:49 INFO  app[][o.s.a.SchedulerImpl] Process[es] is stopped
2021.07.06 10:25:50 INFO  app[][o.s.a.SchedulerImpl] SonarQube is stopped

[tobias-trabelsi-sonarsource]

@akatasonov you have a different problem that is probably related to the host requirements. i suggest you create a thread in our community forum and check this documentation: https://docs.sonarqube.org/latest/requirements/requirements/#header-4

EDIT: nevermind you do have the same issue. sorry did not read to the end of the line

@goraxan can you try to overwrite the entrypoint and take a look in the image itself, like

docker run -it --entrypoint=/bin/bash --rm --pull always sonarqube:latest
bash-5.1# ls /opt/java/openjdk/bin/
bash-5.1# pwd

[akatasonov]

@akatasonov you have a different problem that is probably related to the host requirements. i suggest you create a thread in our community forum and check this documentation: https://docs.sonarqube.org/latest/requirements/requirements/#header-4

@goraxan can you try to overwrite the entrypoint and take a look in the image itself, like

docker run -it --entrypoint=/bin/bash --rm --pull always sonarqube:latest
bash-5.1# ls /opt/java/openjdk/bin/
bash-5.1# pwd

@tobias-trabelsi-sonarsource How is it a different problem if the error message is absolutely the same? It can't find Java inside the image, it has nothing to do with the requirements for Sonarqube

[tobias-trabelsi-sonarsource]

@akatasonov apologies. i edited my comment as i had re read your post. can you provide the same information that i asked for?

[akatasonov]

The Java ELF executable is there:

bash-5.1# ls -al /opt/java/openjdk/bin/
total 272
drwxr-xr-x    2 root     root          4096 Jul  2 18:29 .
drwxr-xr-x    1 root     root          4096 Jul  2 18:29 ..
-rwxr-xr-x    1 root     root         12904 Apr 20 21:21 jaotc
-rwxr-xr-x    1 root     root         12768 Apr 20 21:21 java
-rwxr-xr-x    1 root     root         12784 Apr 20 21:21 jfr
-rwxr-xr-x    1 root     root         12816 Apr 20 21:21 jjs
-rwxr-xr-x    1 root     root         12824 Apr 20 21:21 jrunscript
-rwxr-xr-x    1 root     root         12784 Apr 20 21:21 keytool
-rwxr-xr-x    1 root     root         12784 Apr 20 21:21 pack200
-rwxr-xr-x    1 root     root         12784 Apr 20 21:21 rmid
-rwxr-xr-x    1 root     root         12792 Apr 20 21:21 rmiregistry
-rwxr-xr-x    1 root     root        121488 Apr 20 21:21 unpack200

And it appears to be functional:

bash-5.1# /opt/java/openjdk/bin/java -version
openjdk version "11.0.11" 2021-04-20
OpenJDK Runtime Environment AdoptOpenJDK-11.0.11+9 (build 11.0.11+9)
OpenJDK 64-Bit Server VM AdoptOpenJDK-11.0.11+9 (build 11.0.11+9, mixed mode)

[tobias-trabelsi-sonarsource]

This is really odd as i can not reproduce this with the same digest of the image as @goraxan .
can each of you share a little bit about their respected environment? anything that would help to get this reproduced?
i tested so far:

• running sonarqube:latest
• upgrading sonarqube 8.9 -> 9-0
• running sonarqube:latest with overwritten ES_JAVA_HOME to /tmp/

under all these, sonarqube starts just fine. we set ES_JAVA_HOME to the same value as JAVA_HOME here. and JAVA_HOME is set to /opt/java/openjdk in the Dockerfile.

[akatasonov]

@tobias-trabelsi-sonarsource
Could it be that the reason is that ES_JAVA_HOME is not defined?

bash-5.1# echo "$ES_JAVA_HOME"

bash-5.1#

[goraxan]

If it helps, the docker versions where I've reproduced the issue are:

• Docker version 18.09.7, build 2d0083d (Ubuntu 16.04.7 LTS)
• Docker version 17.03.2-ce, build f5ec1e2 (Ubuntu 16.04.5 LTS)

Also, I've found changes regarding the usage of that environment variable in this ticket https://jira.sonarsource.com/browse/SONAR-14960 (maybe it's not related to the error)

[tobias-trabelsi-sonarsource]

i managed to reproduce this on ubuntu 16.04 with the docker version that is shipped by canonical, so the snap version. and there is also the error. the canonical snap installation is not honoring the OCI sandbox in an expected way.

so the fix is to remove the snap version (sudo snap remove docker) and install docker via apt: https://docs.docker.com/engine/install/ubuntu/
after that the image is working as expected. this error is due to the snap package and as far as i know there is nothing we can do here to fix this on our side.

everyone in this thread, @akatasonov and @goraxan please verify this and i will close the issue after i hear a confirmation

[ebcFlagman]

This not resolve my problem. I haven't installed docker via snap and have also this issue.

• Ubuntu 20.04.2 LTS
• Docker version 19.03.12, build 48a66213fe
• docker-compose version 1.26.2, build eefe0d31

It seems to be working when we set the privileged flag, but this is a security issue for us

[goraxan]

@tobias-trabelsi-sonarsource thanks! I've re-installed docker using the guide you posted and it's working now.
Current docker version: Docker version 20.10.7, build f0df350

Just for the record, I previously installed docker via APT, not snap, and it wasn't working.

[ebcFlagman]

@tobias-trabelsi-sonarsource I've fixed it with update to latest docker version.

• Docker version 20.10.7, build f0df350

[akatasonov]

@tobias-trabelsi-sonarsource just pulled the latest image and also upgraded Docker and the problem is gone! Thank you! :)

[tobias-trabelsi-sonarsource]

glad that it's now working for you :)
thanks for the combined debugging effort here. i will leave this issue open for a few more days before closing in case other people get the same problem

[EugenMayer]

I cannot upgrade the docker image (rancher) and stuck with this very same issue. What is the actual caus e here? Is see privileges help, but how could that be a thing?

Please provide information what has changed, it seems like you do some sysops things in 9.0.0 which you did not before. I expect some new JVM args have been added to optimize docker runtime and those lead to issues.

I can confirm that starting the image on my local docker-engine works, which is 20.10.7 - but that is not a solution for me here. Are there any other options then updating the docker engine? running 17.06.2-ce on the server

[tobias-trabelsi-sonarsource]

Hi @EugenMayer ,

i am not sure if any changes to our docker image or sonarqube are related here tbh. what has changed on the docker image from version 8.9.1 to 9.0 is only the base image was updated from alpine:3.12 to alpine:3.14.
On the application level we basically just added a computed env to remove a deprecation warning from elasticsearch where you can find the implementation here. the other changes to SQ seem too far off.

BTW: your docker engine version is vulnerable to CVE-2019-13509. you should think about an update

[EugenMayer]

@tobias-trabelsi-sonarsource i cannot possible tell you what changed, but usually those kind of things tend to be JVM commands trying to optimize things that are not available in all cases.

Since i'am not the only one here, it is for sure not a generic issue. I'am a sonar customer, not community, how do i escalate this issue?

[tobias-trabelsi-sonarsource]

after some more digging i think i found the root cause. we update alpine to 3.14 in our images. in the changelog of 3.14 this is mentioned:

The faccessat2 syscall has been enabled in musl. This can result in issues on docker hosts with older versions of docker (<20.10.0) and libseccomp (<2.4.4), which blocks this syscall.

which in return is causing this issue right here.

@EugenMayer as a workaround you can build the images yourself with a downgraded alpine version. 3.13 should be okay. you can also change the default action in your seccomp-profile (line 2) to return SCMP_ACT_TRACE instead of SCMP_ACT_ERRNO.

If you have bought support, you are also free to create a support case as github issues are not a support channel for sonarsource.

i will create a ticket on our side and talk to our management how we are treating this.

[tobias-trabelsi-sonarsource]

Issue is tracked here: https://jira.sonarsource.com/browse/SONAR-15167

[tobias-trabelsi-sonarsource]

As this issue is now tracked on Jira with SONAR-15167 i will summarize and close this thread for now.

• Alpine Linux 3.14 ships a new version of musllibc
• This version enabled the faccessat2 syscall
• older versions of docker (<20.10.0) and libseccomp (<2.4.4) block this syscall
• Solution: update the container runtime
• Mitigation: change the default action in your seccomp-profile (line 2) to return SCMP_ACT_TRACE instead of SCMP_ACT_ERRNO

There is also a pending PR on the moby project to backport the change to the seccomp profile to 19.03 but as of now there is no release date for that