Jib: Gradle 0.9.7 auth issue

[Techtony96]

Description of the issue:
When pushing the image to docker hub, a 401 unauthorized error occurs.

Expected behavior:
When provided with valid authentication, the application should log in and push the image successfully.

Steps to reproduce:
auth does not seem to be valid in build.gradle

Environment:
Running within a Docker container created by Jenkins. Build takes place in Docker container gradle:4.9-jdk10

jib-gradle-plugin Configuration:

jib {
    from {
        image = 'openjdk:10-jre'
    }
    to {
        image = "techtony96/boltbot:$boltBotVersion-" + getCommitHash()
        auth {
            username = System.getenv("DOCKER_USR")
            password = System.getenv("DOCKER_PSW")
        }
    }
    container {
        jvmFlags = ['-Xms512m', '-Xmx512m']
        mainClass = 'com.discordbolt.boltbot.BoltBot'
    }
}

Log output:

+ gradle jib
> Task :compileJava UP-TO-DATE
> Task :processResources
> Task :defineProperties
> Task :classes

Containerizing application to techtony96/boltbot:3.0-SNAPSHOT-0816d3f...

Retrieving registry credentials for registry.hub.docker.com...
Getting base image openjdk:10-jre...
Building dependencies layer...
Building resources layer...
Building classes layer...
Building snapshot-dependencies layer...
Retrieving registry credentials for registry.hub.docker.com...
Finalizing...
Got more than one input Future failure. Logging failures after the first
com.google.cloud.tools.jib.registry.RegistryUnauthorizedException: Unauthorized for registry.hub.docker.com/techtony96/boltbot
    at com.google.cloud.tools.jib.registry.RegistryEndpointCaller.call(RegistryEndpointCaller.java:218)
    at com.google.cloud.tools.jib.registry.RegistryEndpointCaller.call(RegistryEndpointCaller.java:141)
    at com.google.cloud.tools.jib.registry.RegistryClient.callRegistryEndpoint(RegistryClient.java:338)
    at com.google.cloud.tools.jib.registry.RegistryClient.checkBlob(RegistryClient.java:249)
    at com.google.cloud.tools.jib.builder.steps.PushBlobStep.call(PushBlobStep.java:80)
    at com.google.cloud.tools.jib.builder.steps.PushBlobStep.call(PushBlobStep.java:35)
    at com.google.common.util.concurrent.CombinedFuture$CallableInterruptibleTask.runInterruptibly(CombinedFuture.java:181)
    at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:57)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
    at java.base/java.lang.Thread.run(Thread.java:844)
Caused by: com.google.api.client.http.HttpResponseException: 401 Unauthorized
    at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1070)
    at com.google.cloud.tools.jib.http.Connection.send(Connection.java:130)
    at com.google.cloud.tools.jib.registry.RegistryEndpointCaller.call(RegistryEndpointCaller.java:173)
    ... 10 more
Got more than one input Future failure. Logging failures after the first

Additional Information:
I get a warning from Intellij saying auth can not be resolved, which seems to me that my login info is not being used.

[chanseokoh]

Hi @Techtony96

• Is it possible to download the container image built by Jenkins and run it locally to see if it works?
• Can you also test if it works locally out of the container? Other users are using the Docker Hub without problems, so I think this should work at least.

[Techtony96]

Hello,

I tried to check to see if the image was saved locally and it seems that it was not. I have the docker socket connected to the host docker so I believe it would have saved on the host.

[chanseokoh]

Sorry, I think there is a miscommuncation.

I did not mean if the image was saved to the Docker daemon running in the host (in the running container). Jib works with docker installed and does not make use of it. It just doesn't save any image locally; it just uploads the images to the target registry.

What I meant was

• If possible, run the container on your local machine, to see if it can successfully push your image to Docker Hub (techtony96/boltbot). Of course, you'd have to set the DOCKER_USR and DOCKER_USR env variables correctly.
• If the above fails, directly invoke gradlew jib on the command line of your machine, not inside a Docker container, to see if it works. I expect this should certainly build and push your image to Docker Hub (techtony96/boltbot). Other users don't have problems pushing images to Docker Hub using Jib, when auth was set up correctly.

[Techtony96]

Looks like you're right, I tested it locally and when putting in my username/password it pushed successfully, albeit I had to put my username/password to pull a public image from docker hub. Still having issues on Jenkins but it seems the issue is on my end.

Appreciate your help in narrowing down the issue.

[Techtony96]

I am still having this issue and think it may actually be related to Jib.
Log: https://gist.github.com/Techtony96/6c2e823ed9de9b953db80693f1aec99a

What I see in the log is that Jib does not seem to be authenticated when making requests. Any insight is greatly appreciated.

[coollog]

Hi @Techtony96 , I see in there that there was a response with {"details":"incorrect username or password"} - could you check and verify if the DOCKER_USR and DOCKER_PSW env vars in your Jenkins build are set to the correct values?

[chanseokoh]

albeit I had to put my username/password to pull a public image from docker hub.

I actually thought this was weird, because you can download public images from Docker Hub (such as openjdk:10-jre you're using) with zero auth configuration.

Jib does not seem to be authenticated when making requests

It's OK to see some of the following messages when pulling the base openjdk image. Jib tries without auth first to see if the server explicitly wants auth.

{"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":null}]}

BTW, Jib makes parallel requests, so the log would mostly be not in the time order of events.

What I see at the last part is the server returning "incorrect username or password", and I am leaning toward trusting the server response.

{"details":"incorrect username or password"}

[Techtony96]

Hello,

I have checked to make sure the username and password is correct. It is slightly obscured from directly checking, but I did the following to make sure it is correct.

I store the password in Jenkins credentials as secret text, just as I do with another secret token for my build. This secret token is stored in a file that I was able to check and was saved correctly. I also stored my docker password in the file and it is also correct. I type my username directly into the build.gradle file.

If you want to take a look at the actual file, here is the link: https://github.com/DiscordBolt/BoltBot/blob/v3/build.gradle#L92

And my Jenkins file I call it from: https://github.com/DiscordBolt/BoltBot/blob/v3/Jenkinsfile#L63

[briandealwis]

@Techtony96 for what it's worth, your setup works fine for me. You could enable Google HTTP logging and verify that the authentication parameters being sent look correct.

[chanseokoh]

Given that it works locally, I'm tempted to check echo [$DOCKER_PASSWORD] just before calling gradle. Not sure if it is possible to do below in Jenkins. Haven't used Jenkins.

      steps {
        echo 'Stage:Deploy'
        sh 'echo [$DOCKER_PASSWORD] && gradle jib --info --debug --stacktrace'
      }

[chanseokoh]

Actually, maybe 'echo $DOCKER_PASSWORD'.execute().text.trim() does not work as you expect. I'm not a Gradle user, but with the code below in my build.gradle, it doesn't get the environment variable $USER.

task checkEnvVar {
  doLast {
    println 'echo $USER'.execute().text.trim()
    println "$System.env.USER"
    println System.getenv('USER')
  }
}

$ gradle -q checkEnvVar
$USER
chanseok
chanseok

[Techtony96]

@briandealwis I will test that out and see what happens.

@chanseokoh I have tested by printing the password out, but jenkins marks it as * to protect it.
I originally used System.getenv('DOCKER_PASSWORD') and today tested out the command line echo to see if it made a difference. I'll switch back to System.getenv but the result is the same.

[chanseokoh]

Just in case, I see in your original comment you typed DOCKER_PSW instead of DOCKER_PASSWORD, so watch out.

[Techtony96]

@chanseokoh I used DOCKER_PSW when I was using Jenkins username/password secret credential. I changed to secret text and just called it DOCKER_PASSWORD.

[Techtony96]

After changing back to System.getenv I get a different looking error:
https://gist.github.com/Techtony96/d9edcffd2a53c6e0902a8f232ad5f26f

Jenkinsfile: https://github.com/DiscordBolt/BoltBot/blob/e587d6253c22d9e81d4c5834f73e0d6fb99e8dcf/Jenkinsfile#L63
Build.gradle: https://github.com/DiscordBolt/BoltBot/blob/e587d6253c22d9e81d4c5834f73e0d6fb99e8dcf/build.gradle#L92

[coollog]

@Techtony96 it looks like in your build.gradle, you had the base image use your credentials as well. This seems to be what's causing the failure, since your credentials are not valid for openjdk:10-jre. That image is a public image, so the solution is to just remove the from.auth block.

[chanseokoh]

@Techtony96 @coollog I think there is actually something more to it. From the log, Jib says that it did not get the auth info from build.gradle, for both the from-image and the to-image.

Retrieving registry credentials for registry.hub.docker.com...
RUNNING Retrieving registry credentials for registry.hub.docker.com
No credentials could be retrieved for registry registry.hub.docker.com
...
Retrieving registry credentials for registry.hub.docker.com...
RUNNING Retrieving registry credentials for registry.hub.docker.com
No credentials could be retrieved for registry registry.hub.docker.com

Whether or not providing correct passwords or having the auth section for the from-image or the to-image, if the auth sections have any values, Jib should log that the values are being picked up:

Retrieving registry credentials for registry.hub.docker.com...
RUNNING Retrieving registry credentials for registry.hub.docker.com
Using jib.to.auth for registry.hub.docker.com
...
Retrieving registry credentials for registry.hub.docker.com...
RUNNING Retrieving registry credentials for registry.hub.docker.com
Using jib.from.auth for registry.hub.docker.com

The build.gradle that I'm testing looks like this:

jib {
  from {
    image = 'openjdk:10-jre'
    auth {
      username = 'chanseok'
      password = 'whatever'
    }
  }
  to {
    image = 'chanseok/image-built-with-jib'
    auth {
      username = 'chanseok'
      password = 'whatever'
    }
  }
}

Now, I am suspecting that System.getEnv('DOCKER_PASSWORD') is returning null. If I use the following build.gradle on my machine where UNDEFINED_VAR is not defined,

jib {
  from {
    image = 'openjdk:10-jre'
    auth {
      username = 'chanseok'
      password = System.getenv('UNDEFINED_VAR')
    }
  }
  to {
    image = 'chanseok/image-built-with-jib'
    auth {
      username = 'chanseok'
      password = System.getenv('UNDEFINED_VAR')
    }
  }
}

Jib ignores the auth section as the values are null and outputs the same error log as yours:

No credentials could be retrieved for registry registry.hub.docker.com
...
No credentials could be retrieved for registry registry.hub.docker.com

So I'm guessing the environment variable DOCKER_PASSWORD is not available inside build.gradle inside the Jenkins image.

[coollog]

@chanseokoh Nice find! We should definitely provide a better error message in that case.

[Techtony96]

@coollog I went ahead and removed the auth in the from field, and it failed with this error:

Build image failed, perhaps you should set a credential helper name with the configuration 'from.credHelper' or from.auth

Full Log: https://gist.github.com/Techtony96/2ca8178308d37dfeaaf7b5765a7cb97a
build.gradle: https://github.com/DiscordBolt/BoltBot/blob/2f5d83bbff3420de7c49943cad8d84140073bf53/build.gradle#L92

I did not have an auth field originally and got this error so I assumed I needed one and added it.

[Techtony96]

@chanseokoh Interesting, let me add a print statement to see if it actually is null!

[Techtony96]

You're 100% right, the password is null. Looks like it was my issue all along. I appreciate you helping me figure this out!

[chanseokoh]

Glad you figured it out. We also discovered a few issues around logs, so your detailed feedback was very helpful.

> Build image failed, perhaps you should set a credential helper name with the configuration 'from.credHelper' or from.auth

I did not have an auth field originally and got this error so I assumed I needed one and added it.

Seems like we are printing a wrong error message. Filed a bug: #758

Also filed another bug to print a warning if either the username or the password is null: #759