Jkube: k8s:push failed with aws ecr credential helper for docker

[missedone]

Description

I have aws ecr credential helper for docker install on Ubuntu 20.04

{
    "credsStore": "ecr-login"
}

the setup works well, if i use native docker cli to pull or push.

however, when i tried with jkube maven plugin 1.1.0, 1.2.0. or 1.3.0, got the same error below Error: k8s: null

[INFO] --- kubernetes-maven-plugin:1.3.0:push (default-cli) @ feed-product-service ---
[INFO] k8s: Running in Kubernetes mode
[INFO] k8s: Building Docker image in Kubernetes mode
[INFO] k8s: Generators:
[INFO] k8s:  - quarkus
[INFO] k8s:  - spring-boot
[INFO] k8s:  - wildfly-swarm
[INFO] k8s:  - thorntail-v2
[INFO] k8s:  - wildfly-jar
[INFO] k8s:  - openliberty
[INFO] k8s:  - karaf
[INFO] k8s:  - vertx
[INFO] k8s:  - micronaut
[INFO] k8s:  - java-exec
[INFO] k8s:  - webapp
Error:  k8s: null
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  05:07 min
[INFO] Finished at: 2021-05-18T09:41:56-07:00
[INFO] ------------------------------------------------------------------------
Error:  Failed to execute goal org.eclipse.jkube:kubernetes-maven-plugin:1.3.0:push (default-cli) on project my-service: null: MojoExecutionException -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.eclipse.jkube:kubernetes-maven-plugin:1.3.0:push (default-cli) on project feed-product-service: null
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:215)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:957)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:289)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:193)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:566)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:566)
    at org.apache.maven.wrapper.BootstrapMainStarter.start (BootstrapMainStarter.java:39)
    at org.apache.maven.wrapper.WrapperExecutor.execute (WrapperExecutor.java:122)
    at org.apache.maven.wrapper.MavenWrapperMain.main (MavenWrapperMain.java:61)
Caused by: org.apache.maven.plugin.MojoExecutionException
    at org.eclipse.jkube.maven.plugin.mojo.build.PushMojo.executeInternal (PushMojo.java:61)
    at org.eclipse.jkube.maven.plugin.mojo.build.AbstractDockerMojo.execute (AbstractDockerMojo.java:444)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:210)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:957)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:289)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:193)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:566)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:566)
    at org.apache.maven.wrapper.BootstrapMainStarter.start (BootstrapMainStarter.java:39)
    at org.apache.maven.wrapper.WrapperExecutor.execute (WrapperExecutor.java:122)
    at org.apache.maven.wrapper.MavenWrapperMain.main (MavenWrapperMain.java:61)
Error:  
Error:  Re-run Maven using the -X switch to enable full debug logging.
Error:  
Error:  For more information about the errors and possible solutions, please read the following articles:
Error:  [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
Error: Process completed with exit code 1.

my project is a spring boot app, the k8s:build works, and below is the kubernetes-maven-plugin config in pom.xml

      <plugin>
        <groupId>org.eclipse.jkube</groupId>
        <artifactId>kubernetes-maven-plugin</artifactId>
        <version>1.3.0</version>
        <configuration>
          <verbose>true</verbose>
          <registry><my_account_id_masked>.dkr.ecr.us-west-2.amazonaws.com</registry>
          <images>
            <image>
              <name>my-service:${docker.image.tag}</name>
              <build>
                <dockerFile>${project.basedir}/Dockerfile</dockerFile>
                <args>
                  <JAR_FILE>target/${project.build.finalName}.jar</JAR_FILE>
                  <ASSEMBLY_PATH>target</ASSEMBLY_PATH>
                </args>
                <assembly>
                  <name>target</name>
                </assembly>
              </build>
            </image>
          </images>
        </configuration>
      </plugin>

Info

• Eclipse JKube version : 1.3.0
• Maven version (mvn -v) : 3.6.3 with Java 11

• Kubernetes / Red Hat OpenShift setup and version :

• If it's a bug, how to reproduce :

• If it's a feature request, what is your use case :

• Sample Reproducer Project : [GitHub Clone URL]

[manusa]

https://www.eclipse.org/jkube/docs/kubernetes-maven-plugin#extended-authentication
Maybe we lost something when porting from DMP.

[rohanKanojia]

@missedone : Hello, do you have your AWS IAM id/secret configured in system properties, plugin configuration, or maven ~/.m2/settings.xml file? The plugin tries to read those in order to create Signature Version 4 request[0]

[0] https://docs.aws.amazon.com/general/latest/gr/sigv4-create-canonical-request.html

[rohanKanojia]

I checked Fabric8 Docker Maven Plugin and JKube implementation(our docker service code is ported from DMP) and I see DMP uses the new api.ecr* URL for getting ECR authorization token, see [0] [1]. Although, ecr.* URL also seems compatible according to docs[2] .

I'll port this to jkube to see if it resolves your issue

[0] https://github.com/fabric8io/docker-maven-plugin/issues/1317
[1] https://github.com/fabric8io/docker-maven-plugin/pull/1318
[2] https://docs.aws.amazon.com/general/latest/gr/ecr.html

[missedone]

hi @rohanKanojia

do you have your AWS IAM id/secret configured in system properties, plugin configuration, or maven ~/.m2/settings.xml file? The plugin tries to read those in order to create Signature Version 4 request[0]

No. I have IAM role for service account (IRSA) attached on my build pod in k8s. i assume the AWS Java SDK should be able to login with AWS_WEB_IDENTITY_TOKEN_FILE.

BTW, DMP works for me, yes pls port the change i can test it.

Thanks,
Nick

[rohanKanojia]

@missedone : #718 got merged to master. Would it be possible for you to test this with 1.4.0-SNAPSHOT version of master. You would need to build jkube or use jitpack

[missedone]

@rohanKanojia , could you publish the snapshot to somewhere so i can consume in our CI jobs. thx

[rohanKanojia]

Umm, I didn't realize you were using it in your pipelines. Right now we don't publish snapshots anywhere and suggest using jitpack for trying out snapshot builds. I think you'll need to wait for our next release in order to test.

[manusa]

Jitpack should be usable in your pipelines (unless you have it forbidden as a repository source).

  <repositories>
    <repository>
      <id>jitpack.io</id>
      <url>https://jitpack.io</url>
    </repository>
  </repositories>
  <pluginRepositories>
    <pluginRepository>
      <id>jitpack.io</id>
      <url>https://jitpack.io</url>
    </pluginRepository>
  </pluginRepositories>
      <build>
        <plugins>
          <plugin>
            <groupId>com.github.eclipse.jkube</groupId>
            <artifactId>kubernetes-maven-plugin</artifactId>
            <version>master-SNAPSHOT</version>
          </plugin>
        </plugins>
      </build>

[missedone]

hi @rohanKanojia
stil fail with nullpointerexception

2021-06-17T15:32:04.5018822Z [INFO] k8s: Running in Kubernetes mode
2021-06-17T15:32:04.5019467Z [INFO] k8s: Building Docker image in Kubernetes mode
2021-06-17T15:32:04.5974024Z [INFO] k8s: Generators:
2021-06-17T15:32:04.5975053Z [INFO] k8s:  - quarkus
2021-06-17T15:32:04.5983567Z [INFO] k8s:  - spring-boot
2021-06-17T15:32:04.5984799Z [INFO] k8s:  - wildfly-swarm
2021-06-17T15:32:04.5985432Z [INFO] k8s:  - thorntail-v2
2021-06-17T15:32:04.5985949Z [INFO] k8s:  - wildfly-jar
2021-06-17T15:32:04.5986474Z [INFO] k8s:  - openliberty
2021-06-17T15:32:04.5986962Z [INFO] k8s:  - karaf
2021-06-17T15:32:04.5987416Z [INFO] k8s:  - vertx
2021-06-17T15:32:04.5987888Z [INFO] k8s:  - micronaut
2021-06-17T15:32:04.5988378Z [INFO] k8s:  - java-exec
2021-06-17T15:32:04.5989137Z [INFO] k8s:  - webapp
2021-06-17T15:32:05.0568140Z [ERROR] k8s: Failed to execute the build [null]
2021-06-17T15:32:05.0589340Z [INFO] ------------------------------------------------------------------------
2021-06-17T15:32:05.0589923Z [INFO] BUILD FAILURE
2021-06-17T15:32:05.0590825Z [INFO] ------------------------------------------------------------------------
2021-06-17T15:32:05.0603243Z [INFO] Total time:  03:08 min
2021-06-17T15:32:05.0605913Z [INFO] Finished at: 2021-06-17T08:32:05-07:00
2021-06-17T15:32:05.0607005Z [INFO] ------------------------------------------------------------------------
2021-06-17T15:32:05.0616175Z [ERROR] Failed to execute goal com.github.eclipse.jkube:kubernetes-maven-plugin:master-32be156aa5-1:build (default-cli) on project feed-product-service: Failed to execute the build: NullPointerException -> [Help 1]
2021-06-17T15:32:05.0620865Z org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal com.github.eclipse.jkube:kubernetes-maven-plugin:master-32be156aa5-1:build (default-cli) on project feed-product-service: Failed to execute the build
2021-06-17T15:32:05.0623189Z     at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:215)
2021-06-17T15:32:05.0624917Z     at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
2021-06-17T15:32:05.0626627Z     at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
2021-06-17T15:32:05.0628825Z     at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
2021-06-17T15:32:05.0631469Z     at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
2021-06-17T15:32:05.0634773Z     at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
2021-06-17T15:32:05.0637699Z     at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
2021-06-17T15:32:05.0639421Z     at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
2021-06-17T15:32:05.0640586Z     at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
2021-06-17T15:32:05.0641700Z     at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
2021-06-17T15:32:05.0642756Z     at org.apache.maven.cli.MavenCli.execute (MavenCli.java:957)
2021-06-17T15:32:05.0643745Z     at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:289)
2021-06-17T15:32:05.0644859Z     at org.apache.maven.cli.MavenCli.main (MavenCli.java:193)
2021-06-17T15:32:05.0646105Z     at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
2021-06-17T15:32:05.0647932Z     at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
2021-06-17T15:32:05.0649939Z     at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
2021-06-17T15:32:05.0651425Z     at java.lang.reflect.Method.invoke (Method.java:566)
2021-06-17T15:32:05.0652840Z     at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
2021-06-17T15:32:05.0654670Z     at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
2021-06-17T15:32:05.0656740Z     at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
2021-06-17T15:32:05.0658596Z     at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)
2021-06-17T15:32:05.0660492Z     at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
2021-06-17T15:32:05.0662204Z     at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
2021-06-17T15:32:05.0664211Z     at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
2021-06-17T15:32:05.0665688Z     at java.lang.reflect.Method.invoke (Method.java:566)
2021-06-17T15:32:05.0666957Z     at org.apache.maven.wrapper.BootstrapMainStarter.start (BootstrapMainStarter.java:39)
2021-06-17T15:32:05.0668678Z     at org.apache.maven.wrapper.WrapperExecutor.execute (WrapperExecutor.java:122)
2021-06-17T15:32:05.0670171Z     at org.apache.maven.wrapper.MavenWrapperMain.main (MavenWrapperMain.java:61)
2021-06-17T15:32:05.0671594Z Caused by: org.apache.maven.plugin.MojoExecutionException: Failed to execute the build
2021-06-17T15:32:05.0673554Z     at org.eclipse.jkube.maven.plugin.mojo.build.AbstractDockerMojo.buildAndTag (AbstractDockerMojo.java:665)
2021-06-17T15:32:05.0676554Z     at org.eclipse.jkube.maven.plugin.mojo.build.AbstractDockerMojo.processImageConfig (AbstractDockerMojo.java:503)
2021-06-17T15:32:05.0679451Z     at org.eclipse.jkube.maven.plugin.mojo.build.AbstractDockerMojo.executeBuildGoal (AbstractDockerMojo.java:632)
2021-06-17T15:32:05.0681903Z     at org.eclipse.jkube.maven.plugin.mojo.build.BuildMojo.executeInternal (BuildMojo.java:49)
2021-06-17T15:32:05.0684117Z     at org.eclipse.jkube.maven.plugin.mojo.build.AbstractDockerMojo.execute (AbstractDockerMojo.java:444)
2021-06-17T15:32:05.0686448Z     at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
2021-06-17T15:32:05.0688420Z     at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:210)
2021-06-17T15:32:05.0690128Z     at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
2021-06-17T15:32:05.0691835Z     at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
2021-06-17T15:32:05.0694003Z     at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
2021-06-17T15:32:05.0696867Z     at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
2021-06-17T15:32:05.0699882Z     at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
2021-06-17T15:32:05.0702544Z     at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
2021-06-17T15:32:05.0704097Z     at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
2021-06-17T15:32:05.0705236Z     at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
2021-06-17T15:32:05.0706380Z     at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
2021-06-17T15:32:05.0707435Z     at org.apache.maven.cli.MavenCli.execute (MavenCli.java:957)
2021-06-17T15:32:05.0708541Z     at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:289)
2021-06-17T15:32:05.0709495Z     at org.apache.maven.cli.MavenCli.main (MavenCli.java:193)
2021-06-17T15:32:05.0710732Z     at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
2021-06-17T15:32:05.0712439Z     at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
2021-06-17T15:32:05.0714526Z     at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
2021-06-17T15:32:05.0716074Z     at java.lang.reflect.Method.invoke (Method.java:566)
2021-06-17T15:32:05.0717624Z     at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
2021-06-17T15:32:05.0719540Z     at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
2021-06-17T15:32:05.0721403Z     at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
2021-06-17T15:32:05.0723287Z     at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)
2021-06-17T15:32:05.0724865Z     at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
2021-06-17T15:32:05.0726569Z     at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
2021-06-17T15:32:05.0728552Z     at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
2021-06-17T15:32:05.0730045Z     at java.lang.reflect.Method.invoke (Method.java:566)
2021-06-17T15:32:05.0731322Z     at org.apache.maven.wrapper.BootstrapMainStarter.start (BootstrapMainStarter.java:39)
2021-06-17T15:32:05.0733071Z     at org.apache.maven.wrapper.WrapperExecutor.execute (WrapperExecutor.java:122)
2021-06-17T15:32:05.0734551Z     at org.apache.maven.wrapper.MavenWrapperMain.main (MavenWrapperMain.java:61)
2021-06-17T15:32:05.0735710Z Caused by: java.lang.NullPointerException
2021-06-17T15:32:05.0738214Z     at org.eclipse.jkube.kit.build.service.docker.auth.AuthConfigFactory.getAuthConfigFromEC2InstanceRole (AuthConfigFactory.java:295)
2021-06-17T15:32:05.0742038Z     at org.eclipse.jkube.kit.build.service.docker.auth.AuthConfigFactory.createStandardAuthConfig (AuthConfigFactory.java:229)
2021-06-17T15:32:05.0745309Z     at org.eclipse.jkube.kit.build.service.docker.auth.AuthConfigFactory.createAuthConfig (AuthConfigFactory.java:114)
2021-06-17T15:32:05.0748123Z     at org.eclipse.jkube.kit.build.service.docker.RegistryService.createAuthConfig (RegistryService.java:152)
2021-06-17T15:32:05.0750839Z     at org.eclipse.jkube.kit.build.service.docker.RegistryService.pullImageWithPolicy (RegistryService.java:111)
2021-06-17T15:32:05.0753515Z     at org.eclipse.jkube.kit.build.service.docker.BuildService.autoPullBaseImage (BuildService.java:263)
2021-06-17T15:32:05.0755980Z     at org.eclipse.jkube.kit.build.service.docker.BuildService.buildImage (BuildService.java:74)
2021-06-17T15:32:05.0758389Z     at org.eclipse.jkube.kit.config.service.kubernetes.DockerBuildService.build (DockerBuildService.java:44)
2021-06-17T15:32:05.0760942Z     at org.eclipse.jkube.maven.plugin.mojo.build.AbstractDockerMojo.buildAndTag (AbstractDockerMojo.java:662)
2021-06-17T15:32:05.0763680Z     at org.eclipse.jkube.maven.plugin.mojo.build.AbstractDockerMojo.processImageConfig (AbstractDockerMojo.java:503)
2021-06-17T15:32:05.0766543Z     at org.eclipse.jkube.maven.plugin.mojo.build.AbstractDockerMojo.executeBuildGoal (AbstractDockerMojo.java:632)
2021-06-17T15:32:05.0769022Z     at org.eclipse.jkube.maven.plugin.mojo.build.BuildMojo.executeInternal (BuildMojo.java:49)
2021-06-17T15:32:05.0771246Z     at org.eclipse.jkube.maven.plugin.mojo.build.AbstractDockerMojo.execute (AbstractDockerMojo.java:444)
2021-06-17T15:32:05.0773538Z     at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
2021-06-17T15:32:05.0775561Z     at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:210)
2021-06-17T15:32:05.0777369Z     at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
2021-06-17T15:32:05.0779176Z     at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
2021-06-17T15:32:05.0781366Z     at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
2021-06-17T15:32:05.0783998Z     at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
2021-06-17T15:32:05.0787002Z     at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
2021-06-17T15:32:05.0789687Z     at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
2021-06-17T15:32:05.0791358Z     at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
2021-06-17T15:32:05.0792500Z     at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
2021-06-17T15:32:05.0793612Z     at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
2021-06-17T15:32:05.0794779Z     at org.apache.maven.cli.MavenCli.execute (MavenCli.java:957)
2021-06-17T15:32:05.0795839Z     at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:289)
2021-06-17T15:32:05.0796937Z     at org.apache.maven.cli.MavenCli.main (MavenCli.java:193)
2021-06-17T15:32:05.0798175Z     at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
2021-06-17T15:32:05.0799894Z     at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
2021-06-17T15:32:05.0801881Z     at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
2021-06-17T15:32:05.0803466Z     at java.lang.reflect.Method.invoke (Method.java:566)
2021-06-17T15:32:05.0804898Z     at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
2021-06-17T15:32:05.0806723Z     at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
2021-06-17T15:32:05.0808592Z     at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
2021-06-17T15:32:05.0810425Z     at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)
2021-06-17T15:32:05.0812259Z     at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
2021-06-17T15:32:05.0813989Z     at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
2021-06-17T15:32:05.0816035Z     at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
2021-06-17T15:32:05.0817636Z     at java.lang.reflect.Method.invoke (Method.java:566)
2021-06-17T15:32:05.0818882Z     at org.apache.maven.wrapper.BootstrapMainStarter.start (BootstrapMainStarter.java:39)
2021-06-17T15:32:05.0820473Z     at org.apache.maven.wrapper.WrapperExecutor.execute (WrapperExecutor.java:122)
2021-06-17T15:32:05.0821952Z     at org.apache.maven.wrapper.MavenWrapperMain.main (MavenWrapperMain.java:61)

[rohanKanojia]

Hello, I thought your issue was regarding pushing image rather than building image. Would it be possible for you to share a reproducer project so that I can reproduce?

[missedone]

hi @rohanKanojia
just have a quick look at

Caused by: java.lang.NullPointerException
   at org.eclipse.jkube.kit.build.service.docker.auth.AuthConfigFactory.getAuthConfigFromEC2InstanceRole (AuthConfigFactory.java:295)

AWS ECR credential helper for docker get the auth info from context, in my case it's IRSA (IamRole for Service Account), which means we should use WebIdentityTokenFile, see more context https://github.com/aws/aws-sdk-java/issues/2136

[missedone]

also look at https://github.com/eclipse/jkube/blob/969025bf6bd21f22ec02ef4a72174ae49aaad5bb/jkube-kit/build/service/docker/src/main/java/org/eclipse/jkube/kit/build/service/docker/auth/AuthConfigFactory.java#L227-L239
why the method requires returning AuthConfig which contains user/password, it's may not necessary for WebIdentityTokenFile though you still can get the user and password

[rohanKanojia]

oh, thanks a lot for pointing out. I'm seeing that there are some implementation differences in dmp and jkube in AuthConfigFactory. Looks like we need to port two more PRs

https://github.com/fabric8io/docker-maven-plugin/pull/1310
https://github.com/fabric8io/docker-maven-plugin/pull/1311

[missedone]

https://github.com/fabric8io/docker-maven-plugin/pull/1311 is more relavant to the issue, pls let me know once it's available in master-SNAPSHOT, i will test out
Thanks,
Nick

[manusa]

Hi @missedone
The new fix got merged again to master.
Would you mind trying again with the Jitpack snapshot?

[missedone]

@manusa , yes it works now, thank you and @rohanKanojia

below is my plugin configuration snippet

      <plugin>
        <groupId>com.github.eclipse.jkube</groupId>
        <artifactId>kubernetes-maven-plugin</artifactId>
        <version>master-SNAPSHOT</version>
        <configuration>
          <verbose>true</verbose>
          <registry>############.dkr.ecr.us-west-2.amazonaws.com</registry>
          <images>
            <image>
              <name>my-service:${docker.image.tag}</name>
              <build>
                ....
              </build>
            </image>
          </images>
        </configuration>
        <dependencies>
          <!-- aws sdk is used for aws ecr login -->
          <dependency>
            <groupId>com.amazonaws</groupId>
            <artifactId>aws-java-sdk-core</artifactId>
            <version>1.11.1034</version>
          </dependency>
          <dependency>
            <groupId>com.amazonaws</groupId>
            <artifactId>aws-java-sdk-sts</artifactId>
            <version>1.11.1034</version>
          </dependency>
          <dependency>
            <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-databind</artifactId>
            <version>2.11.4</version>
          </dependency>
          <dependency>
            <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-core</artifactId>
            <version>2.11.4</version>
          </dependency>
        </dependencies>
      </plugin>

[missedone]

@manusa , i think better to update the doc with mentioning the above plugin dependencies for ECR login

[missedone]

hmm, i have to reopen this issue, since i got error below when pull the base image openjdk:11.0.8-jre-slim:

Error:  Failed to execute goal com.github.eclipse.jkube:kubernetes-maven-plugin:master-acd5106087-1:build (default-cli) on project my-service: Failed to execute the build: Error while trying to build the image: Unable to pull 'openjdk:11.0.8-jre-slim' from registry '############.dkr.ecr.us-west-2.amazonaws.com' : {"message":"repository ############.dkr.ecr.us-west-2.amazonaws.com/openjdk not found: name unknown: The repository with name 'openjdk' does not exist in the registry with id '############'"} (Not Found: 404) -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal com.github.eclipse.jkube:kubernetes-maven-plugin:master-acd5106087-1:build (default-cli) on project my-service: Failed to execute the build
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:215)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:210)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:957)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:289)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:193)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:566)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:566)
    at org.apache.maven.wrapper.BootstrapMainStarter.start (BootstrapMainStarter.java:39)
    at org.apache.maven.wrapper.WrapperExecutor.execute (WrapperExecutor.java:122)
    at org.apache.maven.wrapper.MavenWrapperMain.main (MavenWrapperMain.java:61)
Caused by: org.eclipse.jkube.kit.build.service.docker.access.hc.http.HttpRequestException: {"message":"repository ############.dkr.ecr.us-west-2.amazonaws.com/openjdk not found: name unknown: The repository with name 'openjdk' does not exist in the registry with id '############'"} (Not Found: 404)
    at org.eclipse.jkube.kit.build.service.docker.access.hc.ApacheHttpClientDelegate$StatusCodeCheckerResponseHandler.handleResponse (ApacheHttpClientDelegate.java:200)
    at org.apache.http.impl.client.CloseableHttpClient.execute (CloseableHttpClient.java:223)
    at org.apache.http.impl.client.CloseableHttpClient.execute (CloseableHttpClient.java:165)
    at org.apache.http.impl.client.CloseableHttpClient.execute (CloseableHttpClient.java:140)
    at org.eclipse.jkube.kit.build.service.docker.access.hc.ApacheHttpClientDelegate.post (ApacheHttpClientDelegate.java:116)
    at org.eclipse.jkube.kit.build.service.docker.access.hc.DockerAccessWithHcClient.pullImage (DockerAccessWithHcClient.java:426)
    at org.eclipse.jkube.kit.build.service.docker.RegistryService.pullImageWithPolicy (RegistryService.java:110)
    at org.eclipse.jkube.kit.build.service.docker.BuildService.autoPullBaseImage (BuildService.java:264)
    at org.eclipse.jkube.kit.build.service.docker.BuildService.buildImage (BuildService.java:75)
    at org.eclipse.jkube.kit.config.service.kubernetes.DockerBuildService.build (DockerBuildService.java:44)
    at org.eclipse.jkube.maven.plugin.mojo.build.AbstractDockerMojo.buildAndTag (AbstractDockerMojo.java:666)
    at org.eclipse.jkube.maven.plugin.mojo.build.AbstractDockerMojo.processImageConfig (AbstractDockerMojo.java:507)
    at org.eclipse.jkube.maven.plugin.mojo.build.AbstractDockerMojo.executeBuildGoal (AbstractDockerMojo.java:636)
    at org.eclipse.jkube.maven.plugin.mojo.build.BuildMojo.executeInternal (BuildMojo.java:49)
    at org.eclipse.jkube.maven.plugin.mojo.build.AbstractDockerMojo.execute (AbstractDockerMojo.java:448)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:210)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:957)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:289)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:193)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:566)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:566)
    at org.apache.maven.wrapper.BootstrapMainStarter.start (BootstrapMainStarter.java:39)
    at org.apache.maven.wrapper.WrapperExecutor.execute (WrapperExecutor.java:122)
    at org.apache.maven.wrapper.MavenWrapperMain.main (MavenWrapperMain.java:61)

obviously i want it pull from dockerhub but the plugin was trying to pull from my private ECR.

[missedone]

well looks like i have to set the image registry of configuration.images.image.registry rather than configuration.registry which is the global setting seems caused the above issue. could you confirm @manusa @rohanKanojia ?

below works for me

      <plugin>
        <groupId>com.github.eclipse.jkube</groupId>
        <artifactId>kubernetes-maven-plugin</artifactId>
        <version>master-SNAPSHOT</version>
        <configuration>
          <verbose>true</verbose>
          <images>
            <image>
              <registry>############.dkr.ecr.us-west-2.amazonaws.com</registry>
              <name>my-service:${docker.image.tag}</name>
              <build>
                ...
              </build>
            </image>
          </images>
        </configuration>
        <dependencies>
          <!-- aws sdk is used for aws ecr login -->
          <dependency>
            <groupId>com.amazonaws</groupId>
            <artifactId>aws-java-sdk-core</artifactId>
            <version>1.11.1034</version>
          </dependency>
          <dependency>
            <groupId>com.amazonaws</groupId>
            <artifactId>aws-java-sdk-sts</artifactId>
            <version>1.11.1034</version>
          </dependency>
          <dependency>
            <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-databind</artifactId>
            <version>2.11.4</version>
          </dependency>
          <dependency>
            <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-core</artifactId>
            <version>2.11.4</version>
          </dependency>
        </dependencies>
      </plugin>

[manusa]

In the global configuration there are registry and pullRegistry configurations available.

I assume that your problem is that the global registry configuration is ignored, right? Please, confirm if this is the case and we'll open another issue and investigate it. I already see some problems with the registry configuration in AbstractDockerMojo that probably need to be addressed.

[missedone]

I think the global registry shouldn't impact what being defined in the Dockerfile, Ex. if the Dockerfile use base image openjdk:11.0.8-jre-slim which means pull from the dockerhub, jkube shouldn't add the global registry on base image.

i agree it's separate issue, since it works for me if put the registry at image element https://github.com/eclipse/jkube/issues/702#issuecomment-874314823

i'm going to close this ticket now, thank you guys again.