Jib: SSLPeerUnverifiedException with gcr.io on Java 11
TL;DR: gcr.io now supports TLS 1.3, but OpenJDK 11 and 11.0.1's TLS 1.3 implementation has a bug. The fix will be in OpenJDK 11.0.2 (not yet available). Workaround is to run with
-Djdk.tls.client.protocols=TLSv1.2.
Description of the issue:
Since few days, I have my pipeline failing when trying to deploy images to Google Container repository when running on Java 11 (working in Java 8). I'm facing some error like this:
[ERROR] Failed to execute goal com.google.cloud.tools:jib-maven-plugin:0.10.0:build (default-cli) on project subscription-management-propagation: Build image failed: Failed to authenticate with registry eu.gcr.io/NAME_OF_MY_PROJECT/NAME_OF_MY_APP because: peer not authenticated -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal com.google.cloud.tools:jib-maven-plugin:0.10.0:build (default-cli) on project subscription-management-propagation: Build image failed
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:213)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:154)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:146)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
at org.apache.maven.cli.MavenCli.execute (MavenCli.java:956)
at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:290)
at org.apache.maven.cli.MavenCli.main (MavenCli.java:194)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke (Method.java:566)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:289)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:229)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:415)
at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:356)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke (Method.java:566)
at org.apache.maven.wrapper.BootstrapMainStarter.start (BootstrapMainStarter.java:39)
at org.apache.maven.wrapper.WrapperExecutor.execute (WrapperExecutor.java:122)
at org.apache.maven.wrapper.MavenWrapperMain.main (MavenWrapperMain.java:50)
Caused by: org.apache.maven.plugin.MojoExecutionException: Build image failed
at com.google.cloud.tools.jib.maven.BuildImageMojo.execute (BuildImageMojo.java:140)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:208)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:154)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:146)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
at org.apache.maven.cli.MavenCli.execute (MavenCli.java:956)
at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:290)
at org.apache.maven.cli.MavenCli.main (MavenCli.java:194)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke (Method.java:566)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:289)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:229)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:415)
at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:356)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke (Method.java:566)
at org.apache.maven.wrapper.BootstrapMainStarter.start (BootstrapMainStarter.java:39)
at org.apache.maven.wrapper.WrapperExecutor.execute (WrapperExecutor.java:122)
at org.apache.maven.wrapper.MavenWrapperMain.main (MavenWrapperMain.java:50)
Caused by: com.google.cloud.tools.jib.registry.RegistryAuthenticationFailedException: Failed to authenticate with registry eu.gcr.io/NAME_OF_MY_PROJECT/NAME_OF_MY_APP because: peer not authenticated
at com.google.cloud.tools.jib.registry.RegistryAuthenticator.authenticate (RegistryAuthenticator.java:291)
at com.google.cloud.tools.jib.registry.RegistryAuthenticator.authenticatePush (RegistryAuthenticator.java:247)
at com.google.cloud.tools.jib.builder.steps.AuthenticatePushStep.call (AuthenticatePushStep.java:97)
at com.google.cloud.tools.jib.builder.steps.AuthenticatePushStep.call (AuthenticatePushStep.java:43)
at com.google.common.util.concurrent.CombinedFuture$CallableInterruptibleTask.runInterruptibly (CombinedFuture.java:181)
at com.google.common.util.concurrent.InterruptibleTask.run (InterruptibleTask.java:57)
at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:628)
at java.lang.Thread.run (Thread.java:834)
Caused by: javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
at sun.security.ssl.SSLSessionImpl.getPeerCertificates (SSLSessionImpl.java:526)
at org.apache.http.conn.ssl.AbstractVerifier.verify (AbstractVerifier.java:128)
at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket (SSLSocketFactory.java:339)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection (DefaultClientConnectionOperator.java:123)
at org.apache.http.impl.conn.AbstractPoolEntry.open (AbstractPoolEntry.java:147)
at org.apache.http.impl.conn.AbstractPooledConnAdapter.open (AbstractPooledConnAdapter.java:108)
at org.apache.http.impl.client.DefaultRequestDirector.execute (DefaultRequestDirector.java:415)
at org.apache.http.impl.client.AbstractHttpClient.execute (AbstractHttpClient.java:641)
at org.apache.http.impl.client.AbstractHttpClient.execute (AbstractHttpClient.java:576)
at org.apache.http.impl.client.AbstractHttpClient.execute (AbstractHttpClient.java:554)
at com.google.api.client.http.apache.ApacheHttpRequest.execute (ApacheHttpRequest.java:65)
at com.google.api.client.http.HttpRequest.execute (HttpRequest.java:981)
at com.google.cloud.tools.jib.http.Connection.send (Connection.java:168)
at com.google.cloud.tools.jib.http.Connection.get (Connection.java:123)
at com.google.cloud.tools.jib.registry.RegistryAuthenticator.authenticate (RegistryAuthenticator.java:274)
at com.google.cloud.tools.jib.registry.RegistryAuthenticator.authenticatePush (RegistryAuthenticator.java:247)
at com.google.cloud.tools.jib.builder.steps.AuthenticatePushStep.call (AuthenticatePushStep.java:97)
at com.google.cloud.tools.jib.builder.steps.AuthenticatePushStep.call (AuthenticatePushStep.java:43)
at com.google.common.util.concurrent.CombinedFuture$CallableInterruptibleTask.runInterruptibly (CombinedFuture.java:181)
at com.google.common.util.concurrent.InterruptibleTask.run (InterruptibleTask.java:57)
at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:628)
at java.lang.Thread.run (Thread.java:834)
Expected behavior:
The behavior expected is my image to be pushed to the registry
Steps to reproduce:
- Setup a spring project with maven (from start.spring.io for example)
- Setup jib with the following parameters:
<plugin>
<groupId>com.google.cloud.tools</groupId>
<artifactId>jib-maven-plugin</artifactId>
<version>0.10.0</version>
<configuration>
<from>
<image>openjdk:11-jdk-slim</image>
</from>
<to>
<image>eu.gcr.io/NAME_OF_MY_PROJECT/NAME_OF_MY_APP:${tag}</image>
<credHelper>gcr</credHelper>
</to>
</configuration>
</plugin>
I've also tried all version from 0.9.8 to 0.10.0.
In my gitlab-ci pipeline, I have this configuration:
.before_script: &docker_credential_gcr
- echo ${CI_GOOGLE_ACCOUNT} > /tmp/propagation-account
- export VERSION=1.5.0 && export OS=linux && export ARCH=amd64
- curl -qs -L "https://github.com/GoogleCloudPlatform/docker-credential-gcr/releases/download/v${VERSION}/docker-credential-gcr_${OS}_${ARCH}-${VERSION}.tar.gz" | tar zxv > docker-credential-gcr && chmod +x ./docker-credential-gcr
- export PATH=$PATH:.
build:
image: openjdk:11-jdk
stage: 馃摝 build
before_script: *docker_credential_gcr
script:
- if [ -n "$CI_COMMIT_TAG" ]; then export version_tag=$CI_COMMIT_TAG; else export version_tag=$CI_COMMIT_SHA; fi
- ./mvnw $MAVEN_CLI_OPTS -X jib:build -Dtag=$version_tag
cache: *cache_pull
I've simplified the file to extract confidiential information and to make it more readable.
Environment:
- Linux (docker container and inside a GitlabCI pipeline)
jib-maven-plugin Configuration:
<plugin>
<groupId>com.google.cloud.tools</groupId>
<artifactId>jib-maven-plugin</artifactId>
<version>0.10.0</version>
<configuration>
<from>
<image>openjdk:11-jdk-slim</image>
</from>
<to>
<image>eu.gcr.io/NAME_OF_MY_PROJECT/NAME_OF_MY_APP:${tag}</image>
<credHelper>gcr</credHelper>
</to>
</configuration>
</plugin>
Logs:
In debug logs, I have the following output:
[DEBUG] TIMING Building and pushing image
[INFO] Retrieving registry credentials for eu.gcr.io...
[DEBUG] TIMING Retrieving registry credentials for eu.gcr.io
[DEBUG] Checking credentials from docker-credential-gcr
[INFO] Getting base image openjdk:11-jdk-slim...
[DEBUG] TIMING Pulling base image manifest
[DEBUG] TIMING Building application layers
[INFO] Building dependencies layer...
[DEBUG] TIMING Building dependencies layer
[DEBUG] TIMED Building application layers : 20.89 ms
[INFO] Building resources layer...
[INFO] Building classes layer...
[DEBUG] TIMING Building classes layer
[DEBUG] TIMING Building resources layer
[DEBUG] TIMING Setting up to push layers
[DEBUG] TIMED Setting up to push layers : 11.611 ms
[DEBUG] Using docker-credential-gcr for eu.gcr.io
[DEBUG] TIMED Retrieving registry credentials for eu.gcr.io : 767.094 ms
[DEBUG] TIMING Authenticating with push to eu.gcr.io
[DEBUG] Building resources layer built sha256:64b991674ad0e09e08fe9ac594afb6ad1d82aca3728613e16f16083e07add18e
[DEBUG] TIMED Building resources layer : 770.306 ms
[INFO] The base image requires auth. Trying again for openjdk:11-jdk-slim...
[INFO] Retrieving registry credentials for registry.hub.docker.com...
[DEBUG] TIMING Retrieving registry credentials for registry.hub.docker.com
[DEBUG] No credentials could be retrieved for registry registry.hub.docker.com
[DEBUG] TIMED Retrieving registry credentials for registry.hub.docker.com : 27.135 ms
[DEBUG] Building classes layer built sha256:c37d8293239d6c59f916f947f157eed53f8d95988c570804081cab58c191489e
[DEBUG] TIMED Authenticating with push to eu.gcr.io : 374.394 ms
[DEBUG] TIMED Building classes layer : 1064.173 ms
[DEBUG] TIMED Pulling base image manifest : 3843.592 ms
[DEBUG] TIMING Setting up base image caching
[DEBUG] TIMING Pulling base image layer sha256:66f6961c9eb5dddcc3a449439adc5c4a03e716794daec2a41900f7fda4fb492a
[DEBUG] TIMING Pulling base image layer sha256:cca8b8cf2f157d13678401181406e5baf05ab424a04fb8190f9e3816c5db1e29
[DEBUG] TIMING Pulling base image layer sha256:13cf705e89452ccf724ae251641ad26bc6e8162141b3a88b688c959d8872f905
[DEBUG] TIMING Pulling base image layer sha256:805a6a4d0333bf4fa32e15672a04b71ef96caa25e163e085b4bd72831d2792b0
[DEBUG] TIMED Setting up base image caching : 14.636 ms
[DEBUG] TIMING Pulling base image layer sha256:fe824bb10542643a9f3e71319e68e39d64172fe0b8c4d6167c5ec005372c1d68
[DEBUG] TIMING Setting up to push layers
[DEBUG] TIMED Setting up to push layers : 0.776 ms
[DEBUG] TIMED Building and pushing image : 3923.256 ms
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 03:20 min
[INFO] Finished at: 2018-11-13T18:43:45Z
[INFO] ------------------------------------------------------------------------
For information, the problem seems to appear after some fix releases on the JDK, but I can't be sur because I don't deploy this project very often (and on some project, we are still with the double pipeline Java8 / Java11).
Thanks for your help
/cc @neonox31
davinkevin
All 10 comments
@davinkevin I'm able to reproduce this when running with Java 11 and just trying to fetch the gcr.io/distroless/java image (you'll see this too if you run with -DjibSerialized=true which serializes all network accesses).
This is puzzling as I'm 99% certain I was accessing gcr.io earlier today using JDK11 with no issue.
briandealwis
on 13 Nov 2018
It also works few days ago because I already build some image (which are now in production) with jib on Java 11 馃槄
I think some modification made in v11.0.1 (and even on some v1.8.x) cause this problem with the Google registry (and maybe other registries).
Right now, this is a blocking problem for us. Do you think this can be fixed on the jib side or do we have to find an alternative solution to jib for now?
Thanks
davinkevin
on 14 Nov 2018
@davinkevin A workaround is to build with -Djib.allowInsecureRegistries=true.
briandealwis
on 14 Nov 2018
seems like another workaround is (edit: server-side only)jdk.tls.disabledAlgorithms= TLSv1.3. (though I haven't tried)
the problem is caused by a JDK 11 bug
patflynn
on 14 Nov 2018
That's for server-side, unfortunately doesn't work for clients.
briandealwis
on 14 Nov 2018
Aha, That only works for -Dhttps.protocols=TLSv1.2 forces TLSv1.2. @davinkevin the JDK bug will be fixed with OpenJDK 11.0.2.java.net.HttpURLConnection.
briandealwis
on 14 Nov 2018
Ok, here's the right property that works: mvn -Djdk.tls.client.protocols=TLSv1.2 jib:....
briandealwis
on 14 Nov 2018
👍3
Retitled and added workaround to description.
briandealwis
on 14 Nov 2018
Closing as this isn't actually a Jib bug.
briandealwis
on 15 Nov 2018
Thanks for your answer.
I move to the JVM parameter like proposed and it works !
BTW, the allowInsecure doesn't work in my case.
davinkevin
on 16 Nov 2018
Was this page helpful?
0 / 5 - 0 ratings
Related issues
joerg1985
路
3Comments
shijiaoming
路
4Comments
coollog
路
4Comments
anuraaga
路
6Comments
eddiewebb
路
7Comments
Most helpful comment
Ok, here's the right property that works:
mvn -Djdk.tls.client.protocols=TLSv1.2 jib:....