Skaffold: `skaffold dev` fails with `retrieving image config: getting image: UNAUTHORIZED: authentication required`

Created on 24 Jun 2019  路  12Comments  路  Source: GoogleContainerTools/skaffold

Expected behavior

skaffold dev should create and deploy resources to minikube for local dev.

Actual behavior

Running skaffold dev -v debug produces:

INFO[0000] Skaffold &{Version:v0.32.0 ConfigVersion:skaffold/v1beta11 GitVersion: GitCommit:6ed7aad5e8a36052ee5f6079fc91368e362121f7 GitTreeState:clean BuildDate:2019-06-20T23:34:55Z GoVersion:go1.12.6 Compiler:gc Platform:darwin/amd64}
INFO[0000] applying profile: dev
DEBU[0000] overlaying profile on config for field BuildConfig
DEBU[0000] overlaying profile on config for field
DEBU[0000] overlaying profile on config for field
DEBU[0000] overlaying profile on config for field TagPolicy
INFO[0000] no values found in profile for field TagPolicy, using original config values
DEBU[0000] overlaying profile on config for field BuildType
DEBU[0000] overlaying profile on config for field DeployConfig
DEBU[0000] overlaying profile on config for field DeployType
DEBU[0000] overlaying profile on config for field
DEBU[0000] validating yamltags of struct SkaffoldConfig
DEBU[0000] validating yamltags of struct Pipeline
DEBU[0000] validating yamltags of struct BuildConfig
DEBU[0000] validating yamltags of struct Artifact
DEBU[0000] validating yamltags of struct ArtifactType
DEBU[0000] validating yamltags of struct DockerArtifact
DEBU[0000] validating yamltags of struct TagPolicy
DEBU[0000] validating yamltags of struct GitTagger
DEBU[0000] validating yamltags of struct BuildType
DEBU[0000] validating yamltags of struct LocalBuild
DEBU[0000] validating yamltags of struct DeployConfig
DEBU[0000] validating yamltags of struct DeployType
DEBU[0000] validating yamltags of struct KustomizeDeploy
DEBU[0000] validating yamltags of struct KubectlFlags
INFO[0000] no config entry found for kube-context minikube
INFO[0000] Using kubectl context: minikube
INFO[0000] no config entry found for kube-context minikube
INFO[0000] no config entry found for kube-context minikube
DEBU[0000] Using builder: local
DEBU[0000] Running command: [minikube docker-env --shell none]
DEBU[0000] Command output: [DOCKER_TLS_VERIFY=1
DOCKER_HOST=tcp://192.168.99.100:2376
DOCKER_CERT_PATH=/Users/mark/.minikube/certs
]
DEBU[0000] setting Docker user agent to skaffold-v0.32.0
INFO[0000] no config entry found for kube-context minikube
DEBU[0000] push value not present, defaulting to false because localCluster is true
INFO[0000] starting gRPC server on port 50051
INFO[0000] starting gRPC HTTP server on port 50052
DEBU[0000] Checking base image starefossen/ruby-node for ONBUILD triggers.
DEBU[0001] Using default workdir '/' for starefossen/ruby-node
FATA[0003] watching files for artifact gcr.io/litdistco-pro/litdistco-rails: listing files: listing files: listing copied files: retrieving image config: getting image: UNAUTHORIZED: authentication required; [map[Action:pull Class: Name:library/developer Type:repository]]

Information

  • Skaffold version: v0.32.0
  • Operating system: Mac OS Mojave (10.14.5 (18F132))
  • Contents of skaffold.yaml:
  • Cluster: minikube version: v1.2.0
apiVersion: skaffold/v1beta11
kind: Config
build:
  artifacts:
    - image:  gcr.io/litdistco-pro/litdistco-rails
      docker:
        cacheFrom:
          - gcr.io/litdistco-pro/litdistco-rails 
  googleCloudBuild:
    projectId: litdistco-pro
deploy:
  kustomize:
    path: k8s/overlays/production
profiles:
  - name: dev
    build:
      artifacts:
      - image:  gcr.io/litdistco-pro/litdistco-rails
        docker:
          target: developer
      local: {}
    deploy:
      kustomize:
        path: k8s/overlays/development
    activation:
      - command: dev
  - name: staging
    build:
      artifacts:
      - image:  gcr.io/litdistco-pro/litdistco-rails
        docker:
          target: developer
      local: {}
    deploy:
      kustomize:
        path: k8s/overlays/staging

I've also confirmed that running a raw Docker build works (docker build -tag gcr.io/litdistco-pro/litdistco-rails .) docker ps appears to show normal minikube containers running.

Here are the addons I've enabled in minikube:

- addon-manager: enabled
- dashboard: disabled
- default-storageclass: enabled
- efk: disabled
- freshpod: disabled
- gvisor: disabled
- heapster: disabled
- ingress: disabled
- logviewer: disabled
- metrics-server: disabled
- nvidia-driver-installer: disabled
- nvidia-gpu-device-plugin: disabled
- registry: enabled
- registry-creds: disabled
- storage-provisioner: enabled
- storage-provisioner-gluster: disabled

This is my Dockerfile:

# Based off the excellent post at https://auth0.com/blog/ruby-on-rails-killer-workflow-with-docker-part-1/
FROM starefossen/ruby-node AS developer
RUN apt-get update -qq && \
    apt-get install -y \
      build-essential \
      libpq-dev \
      mysql-client \
      redis-tools \
      vim \
      && \
    gem install bundler
RUN mkdir /project
COPY Gemfile Gemfile.lock /project/
WORKDIR /project
RUN bundle install
COPY . /project
ENTRYPOINT [ "/project/bin/rails" ]
CMD [ "server", "-p", "3000", "-b", "0.0.0.0" ]

FROM developer
RUN SECRET_KEY_BASE=fake RAILS_ENV=production /project/bin/rails assets:precompile

Steps to reproduce the behavior

  1. Clone fresh project
  2. brew virtualbox minikube skaffold
  3. skaffold dev -v debug

Thanks for the help and please let me know if you need any more information!

picture MarkBennett
👍1

All 12 comments

I just tried cloning this repo and running the getting started steps from the docs and can confirm that skaffold dev works on this machine.

MarkBennett picture MarkBennett on 24 Jun 2019

Its failing abruptly for me, twice it re-run worked and twice it failed with following error.

time="2019-06-25T12:08:13+05:30" level=fatal msg="watching files for artifact containerimages.net/docker-sandbox/kube/pythonserver: listing files: listing files: listing copied files: retrieving image config: getting image: empty transport.Error response"

prary picture prary on 25 Jun 2019

I just upgraded skaffold with brew and am now encountering this issue.

skaffold version
v0.32.0

When not logged into docker-for-mac I encounter the UNAUTHORIZED:... fatal err:

DEBU[0000] Checking base image alpine:3.9 for ONBUILD triggers.
FATA[0003] watching files for artifact gcr.io/my-project-id/my-service: listing files: listing files: getting dependencies for gcr.io/my-project-id/my-service: listing copied files: retrieving image config: getting image: UNAUTHORIZED: authentication required; [map[Action:pull Class: Name:library/source Type:repository]]

When signed into my docker account w/ docker-for-mac I encounter the empty transport.Error fatal err:

DEBU[0000] Checking base image alpine:3.9 for ONBUILD triggers.
WARN[0000] Error processing base image (alpine:3.9) for ONBUILD triggers: getting remote config: getting image: <empty transport.Error response>. Dependencies may be incomplete.
FATA[0001] watching files for artifact gcr.io/my-project-id/my-service: listing files: listing files: getting dependencies for gcr.io/my-project-id/my-service: listing copied files: retrieving image config: getting image: <empty transport.Error response>

Is skaffold somehow not using gcloud's docker credentials helper?

ItsLeeOwen picture ItsLeeOwen on 25 Jun 2019
👍1

@ItsLeeOwen Crazy Cat Lady, eh? 馃槈 馃惐

MarkBennett picture MarkBennett on 25 Jun 2019
😄1

Downgrading to 0.31 is a workaround:

Archive releases: https://github.com/GoogleContainerTools/skaffold/releases

ItsLeeOwen picture ItsLeeOwen on 25 Jun 2019
👍1

Downgrading to 0.31 is a workaround:

Thanks. I'll give that a try when I'm back on that machine. Might be time for some git bisect.

MarkBennett picture MarkBennett on 25 Jun 2019

This is related to #2256. This was occurring for me as well, using local K8s cluster with Docker Desktop on Windows 10.

dahovey picture dahovey on 25 Jun 2019
👍1

@MarkBennett Can you confirm you have a multi-stage build? If the answer is yes, then this is a duplicate of #2315

dgageot picture dgageot on 26 Jun 2019

Can you confirm you have a multi-stage build? If the answer is yes, then this is a duplicate of #2315

Here's my Dockerfile @dgageot...

~~~
$ cat Dockerfile

Based off the excellent post at https://auth0.com/blog/ruby-on-rails-killer-workflow-with-docker-part-1/

FROM starefossen/ruby-node AS developer
RUN apt-get update -qq && \
apt-get install -y \
build-essential \
libpq-dev \
mysql-client \
redis-tools \
vim \
&& \
gem install bundler
RUN mkdir /project
COPY Gemfile Gemfile.lock /project/
WORKDIR /project
RUN bundle install
COPY . /project
ENTRYPOINT [ "/project/bin/rails" ]
CMD [ "server", "-p", "3000", "-b", "0.0.0.0" ]

FROM developer
RUN SECRET_KEY_BASE=fake RAILS_ENV=production /project/bin/rails assets:precompile
~~~

MarkBennett picture MarkBennett on 26 Jun 2019

The only other information I can think that might be related is that I'm running this on minikube version: v1.2.0.

MarkBennett picture MarkBennett on 26 Jun 2019

So, yes, you are using a multi stage build Dockerfile. And latest release of Skaffold broke the support of it. I'm going to close this issue because it's a duplicate of #2315
This should be fixed with #2340

dgageot picture dgageot on 26 Jun 2019
👍1

So, yes, you are using a multi stage build Dockerfile. And latest release of Skaffold broke the support of it.

Thanks @dgageot. I'll follow that issue. 馃憤

MarkBennett picture MarkBennett on 26 Jun 2019
Was this page helpful?
0 / 5 - 0 ratings

Related issues

nathanph picture nathanph  路  3Comments
garvincasimir picture garvincasimir  路  4Comments
yurchenkosv picture yurchenkosv  路  3Comments
Hudsonzp picture Hudsonzp  路  4Comments
heroic picture heroic  路  4Comments