Skaffold: Fail building with Azure Container Registry

[ahaeber]

I'm trying to use Skaffold together with Jib and a AKS Kubernetes cluster. However, it fails when building the image and Skaffold attempts to tag the image.

I guess that the issue is related to our private Azure Container Registry (ACR) and Docker authentication with it.

Expected behavior

skaffold build should build and push the image.

Actual behavior

Fails with the following log:

skaffold build -v debug
INFO[0000] Skaffold &{Version:v0.22.0 ConfigVersion:skaffold/v1beta4 GitVersion: GitCommit:2187105aae414f500789ca6873898efeb104d7a7 GitTreeState:clean BuildDate:2019-01-31T22:53:12Z GoVersion:go1.10.8 Compiler:gc Platform:linux/amd64}
INFO[0000] no config entry found for kube-context devcluster
INFO[0000] Using kubectl context: devcluster
INFO[0000] no config entry found for kube-context devcluster
DEBU[0000] Using builder: local
DEBU[0000] setting Docker user agent to skaffold-v0.22.0
INFO[0000] no config entry found for kube-context devcluster
DEBU[0000] push value not present, defaulting to true because localCluster is false
Starting build...
Building [acmeprivate.azurecr.io/blah/whatever/application]...
DEBU[0000] Running docker build: context: api-app, dockerfile: Dockerfile
DEBU[0000] Checking base image openjdk:8-jdk-alpine for ONBUILD triggers.
DEBU[0000] Found dependencies for dockerfile: []
Sending build context to Docker daemon 116.1MB
Step 1/5 : FROM openjdk:8-jdk-alpine
---> 97bc1352afde
Step 2/5 : VOLUME /tmp
---> Using cache
---> 82132c6ae10f
Step 3/5 : ARG JAR_FILE
---> Using cache
---> 2d51bd6a9dae
Step 4/5 : COPY ${JAR_FILE} /app.jar
---> Using cache
---> 86a498e654e4
Step 5/5 : ENTRYPOINT ["java", "-Djava.security.egd=file:/dev/./urandom", "-jar", "/app.jar", "--debug" ]
---> Using cache
---> 6aa1d7c6a9ae
Successfully built 6aa1d7c6a9ab
Successfully tagged 038039d9e8cca0fcac200a1e6841dc2a:latest
DEBU[0000] Running command: [git rev-parse --short HEAD]
DEBU[0000] Command output: [f386c81
]
DEBU[0000] Running command: [git status . --porcelain]
DEBU[0000] Command output: [ M api-app/pom.xml
]
DEBU[0000] attempting to add tag acmeprivate.azurecr.io/blah/whatever/application:f386c81-dirty-6aa1d7c to src acmeprivate.azurecr.io/blah/whatever/application@sha256:6aa1d7c6a9ae104c31ec217c5e845a79be00db5245d8033ec10ee2bc16a50563
FATA[0001] build failed: building [acmeprivate.azurecr.io/blah/whatever/application]: tagging: tagging image: getting image: unexpected end of JSON input

Information

• Skaffold version: 0.22.0
• Operating system: Fedora Core 29 (4.20.6-200.fc29.x86_64 x64)
• Contents of skaffold.yaml:

apiVersion: skaffold/v1beta4
kind: Config
build:
  artifacts:
  - image: acmeprivate.azurecr.io/blah/whatever/application
    jibMaven: {}
    context: api-app # Multi-module Maven project, api-app is the app module
    docker:
      dockerfile: Dockerfile

  local: {}

deploy:
  kubectl:
    manifests:
    - api-app/kubernetes/dev/*.yaml

I've installed the Docker credential helper for Azure Container Registry. After login az acr login (acmeprivate is the default registry) I can for example do docker pull acmeprivate.azurecr.io/blah/whatever/application so it appears to be working correctly.

[balopat]

1. First note: See #1836 - it is a bit confusing that you have both docker and jibMaven specified. It looks like you are building based on a Dockerfile (jib does not need a Dockerfile!).
2. can you upgrade to v0.25.0 and see if it is better?
3. we need a bit more information on this to be able to reproduce - a sample code in a github repo would be very useful

[tjk]

2) At v0.25, it just shows that it is an auth error (in remoteImage looks like?) better:

FATA[0007] exiting dev mode because first run failed: build failed: building [***.azurecr.io/jvm]: build artifact: getting image: unsupported status code 401; body:

[balopat]

Are you on windows using wincred?
(trying to figure out if it's related to
https://github.com/GoogleContainerTools/skaffold/issues/1160#issuecomment-465253686)

[balopat]

Ah ignore me - I see you are on Fedora Core 29

[ghost]

Any progress made on this issue? I'm experiencing the same.
osx 10.14.4
credHelper acr-darwin
jib-maven-plugin 1.1.2 (package jib:build)

skaffold 28.0

apiVersion: skaffold/v1beta9
kind: Config
build:
artifacts:
- image: imageName
jibMaven: {}
local: {}

DEBU[0000] Running command: [mvn -Djib.console=plain --non-recursive prepare-package jib:build -Dimage=imageName] <-- this command works if executed manually

....
[INFO] Built and pushed image as image name
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 4.022 s
[INFO] Finished at: 2019-05-02T18:00:27-05:00
[INFO] ------------------------------------------------------------------------
Complete in 5.720851245s
FATA[0005] build failed: building [image name]: build artifact: getting image: unsupported status code 401; body:

[blue-user-poz]

Same issue occurs on Windows 10, skaffold 28.0, jib-maven-plugin 1.0.2

time="2019-05-10T11:06:13+02:00" level=fatal msg="exiting dev mode because first run failed: build failed: building [_image_]: build artifact: getting image: unsupported status code 401; body: "

[arichards215]

same issue occurring on Centos 7 and Skaffold 0.29.0

[max-shch]

Same issue on Windows 10, skaffold 0.31.0, jibGradle
time="2019-06-19T14:35:49-04:00" level=fatal msg="build failed: building [<>.azurecr.io/<>/<>]: build artifact: getting image: unsupported status code 401; body: "

[yoranvanoirschot]

I also have the same issues on Windows 10, Skaffold 0.31.0, jibMvn. The build completes but then it throws a 401, causing the process to stop.

[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 6.608 s
[INFO] Finished at: 2019-06-21T15:29:12+02:00
[INFO] ------------------------------------------------------------------------
Complete in 8.2664207s
time="2019-06-21T15:29:12+02:00" level=fatal msg="build failed: building [bbhub.azurecr.io/skaffold-test]: build artifact: getting image: unsupported status code 401; body: "

[priyawadhwa]

cc @GoogleContainerTools/java-tools-build , any chance these errors look familiar to you?

[heltondoria]

Same issue on Windows 10 PRO, Skaffold v0.33 and acr-docker-credential-helper.

[chanseokoh]

@priyawadhwa as noted in many comments, Jib is succeeding. I think the error message getting image: unsupported status code 401; body: is coming from the Skaffold side.

Jib had not been able to push to ACR with docker-credential-acr-* until Jib 1.0.2 (released in Mar 5th, 2019). I wonder if this is the same issue. Have you guys tested Skaffold against ACR with the Azure credential helper?

In case not, I'll document here what we had to do to support ACR.

Somewhere in this doc says

If the secret being stored is an identity token, the Username should be set to <token>.

Then, since this is an OAuth2 Token authentication, we also had to use different parameters (grant_type=refresh_token&&refresh_token=...) and do POST instead of the usual GET. https://github.com/Azure/acr-docker-credential-helper/issues/31#issuecomment-457460762 shows an example:

POST_ https://andyreg.azurecr.io/oauth2/token
headers: {
        Content-Type application/x-www-form-urlencoded <MUST be this, "application/x-www-form-urlencoded;charset=UTF-8" doesn't work>
    }
 Body:  grant_type=refresh_token&service=andyreg.azurecr.io&scope=repository:andy:pull&refresh_token=<eyJhbGciOi...88Q>

[priyawadhwa]

@chanseokoh we haven't tested against ACR with Azure credential helper as of now, so thank you for the documentation!

[dgageot]

This is a bug in go-containerregistry https://github.com/google/go-containerregistry/issues/421

[dgageot]

And also in Skaffold because when Images are built with local Docker, go-containerregistry is not used

[nkubala]

bumping this one down in priority since we haven't seen much activity on it in a few months so it might not be having as much impact as we thought originally, but still leaving it on our radar.

[yoranvanoirschot]

@nkubala Actually this bug is a big issue for our company. Thus we stopped our research in how to implement Skaffold within our company.

Since we are a Azure oriented company this feature is very important to us and the fallback method (using native docker login) is not a very scalable and safe solution.

[loosebazooka]

@yoranvanoirschot you might want to follow up here: https://github.com/google/go-containerregistry/issues/421 as that is where the issue can be fixed.

[atrujillofalcon]

@loosebazooka @nkubala This is a big stopper in my company project that use AKS.

https://github.com/google/go-containerregistry/issues/421 is already merged and closed. How I can integrate the fix into my skaffold project?

[dgageot]

I'm going to try and use the latest version of google/go-containerregistry. However, it looks like it's going to be a go modules hell.

[atrujillofalcon]

@dgageot I'm a Go newbie, but if can I help you with something I would be happy.

[atrujillofalcon]

@dgageot @nkubala any progress? :(

[dgageot]

@atrujillofalcon Could you try https://github.com/GoogleContainerTools/skaffold/pull/3053 and see if that's better?

[dgageot]

I've tested with acr, without or without #3053 and the PR fixes the issue for me \o/. I'm going to close this issue. Feel free to reopen if the problem is not fixed for you.

[atrujillofalcon]

I've tested with acr, without or without #3053 and the PR fixes the issue for me \o/. I'm going to close this issue. Feel free to reopen if the problem is not fixed for you.

@dgageot Its works for me too. Thanks!!