Skaffold: Failed to pull image on docker with mac

[shavo007]

Expected behavior

Run the pod on kubernetes

Actual behavior

Fails to pull image

Information

• Skaffold version: version v0.19.0
• Operating system: osx
• Contents of skaffold.yaml:

apiVersion: skaffold/v1alpha4
kind: ''
build:
  artifacts:
    - image: shanelee007/graphql
deploy:
  kubectl:
    manifests:
      - kubernetes/config.yaml
      - kubernetes/deployment.yaml
      - kubernetes/secret.yaml

Steps to reproduce the behavior

kubectl describe pod

Warning  Failed                 2m                kubelet, docker-for-desktop  Failed to pull image "shanelee007/graphql:b5640a8-dirty-794b0e4": rpc error: code = Unknown desc = Error response from daemon: Get https://registry-1.docker.io/v2/shanelee007/graphql/manifests/b5640a8-dirty-794b0e4: Get https://auth.docker.io/token?scope=repository%3Ashanelee007%2Fgraphql%3Apull&service=registry.docker.io: net/http: TLS handshake timeout

I have a multi phase build for my api. my k8s deployment is running two containers (graphql-api, postgres)
postgres starts fine but not my api

src code can be found here https://github.com/shavo007/graphql-playground/tree/master/api

[priyawadhwa]

Hey @shavo007, I'm not sure if this is a skaffold bug -- it seems like there's a network connection issue in your cluster?

[shavo007]

hi @priyawadhwa , i can definitely run on my local k8s cluster directly
https://github.com/shavo007/graphql-playground/tree/master/api#kubernetes

the problem is when using skaffold

postgres spins up but not my api

Watching for changes every 1s...
[graphql-api-687785f68b-6mbk7 postgres] The files belonging to this database system will be owned by user "postgres".
[graphql-api-687785f68b-6mbk7 postgres] This user must also own the server process.
[graphql-api-687785f68b-6mbk7 postgres]
[graphql-api-687785f68b-6mbk7 postgres] The database cluster will be initialized with locale "en_US.utf8".
[graphql-api-687785f68b-6mbk7 postgres] The default database encoding has accordingly been set to "UTF8".
[graphql-api-687785f68b-6mbk7 postgres] The default text search configuration will be set to "english".
[graphql-api-687785f68b-6mbk7 postgres]
[graphql-api-687785f68b-6mbk7 postgres] Data page checksums are disabled.
[graphql-api-687785f68b-6mbk7 postgres]
[graphql-api-687785f68b-6mbk7 postgres] fixing permissions on existing directory /var/lib/postgresql/data ... ok
[graphql-api-687785f68b-6mbk7 postgres] creating subdirectories ... ok
[graphql-api-687785f68b-6mbk7 postgres] selecting default max_connections ... 100
[graphql-api-687785f68b-6mbk7 postgres] selecting default shared_buffers ... 128MB
[graphql-api-687785f68b-6mbk7 postgres] selecting dynamic shared memory implementation ... posix
[graphql-api-687785f68b-6mbk7 postgres] creating configuration files ... ok
[graphql-api-687785f68b-6mbk7 postgres] running bootstrap script ... ok
[graphql-api-687785f68b-6mbk7 postgres] performing post-bootstrap initialization ... ok
[graphql-api-687785f68b-6mbk7 postgres] syncing data to disk ... ok
[graphql-api-687785f68b-6mbk7 postgres]
[graphql-api-687785f68b-6mbk7 postgres] WARNING: enabling "trust" authentication for local connections
[graphql-api-687785f68b-6mbk7 postgres] You can change this by editing pg_hba.conf or using the option -A, or
[graphql-api-687785f68b-6mbk7 postgres] --auth-local and --auth-host, the next time you run initdb.
[graphql-api-687785f68b-6mbk7 postgres]
[graphql-api-687785f68b-6mbk7 postgres] Success. You can now start the database server using:
[graphql-api-687785f68b-6mbk7 postgres]
[graphql-api-687785f68b-6mbk7 postgres]     pg_ctl -D /var/lib/postgresql/data -l logfile start
[graphql-api-687785f68b-6mbk7 postgres]
[graphql-api-687785f68b-6mbk7 postgres] ****************************************************
[graphql-api-687785f68b-6mbk7 postgres] WARNING: No password has been set for the database.
[graphql-api-687785f68b-6mbk7 postgres]          This will allow anyone with access to the
[graphql-api-687785f68b-6mbk7 postgres]          Postgres port to access your database. In
[graphql-api-687785f68b-6mbk7 postgres]          Docker's default configuration, this is
[graphql-api-687785f68b-6mbk7 postgres]          effectively any other container on the same
[graphql-api-687785f68b-6mbk7 postgres]          system.
[graphql-api-687785f68b-6mbk7 postgres]
[graphql-api-687785f68b-6mbk7 postgres]          Use "-e POSTGRES_PASSWORD=password" to set
[graphql-api-687785f68b-6mbk7 postgres]          it in "docker run".
[graphql-api-687785f68b-6mbk7 postgres] ****************************************************
[graphql-api-687785f68b-6mbk7 postgres] waiting for server to start....2018-12-14 00:40:06.455 UTC [44] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
[graphql-api-687785f68b-6mbk7 postgres] 2018-12-14 00:40:06.473 UTC [45] LOG:  database system was shut down at 2018-12-14 00:40:05 UTC
[graphql-api-687785f68b-6mbk7 postgres] 2018-12-14 00:40:06.482 UTC [44] LOG:  database system is ready to accept connections
[graphql-api-687785f68b-6mbk7 postgres]  done
[graphql-api-687785f68b-6mbk7 postgres] server started
[graphql-api-687785f68b-6mbk7 postgres]
[graphql-api-687785f68b-6mbk7 postgres] /usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
[graphql-api-687785f68b-6mbk7 postgres]
[graphql-api-687785f68b-6mbk7 postgres] waiting for server to shut down....2018-12-14 00:40:06.544 UTC [44] LOG:  received fast shutdown request
[graphql-api-687785f68b-6mbk7 postgres] 2018-12-14 00:40:06.546 UTC [44] LOG:  aborting any active transactions
[graphql-api-687785f68b-6mbk7 postgres] 2018-12-14 00:40:06.551 UTC [44] LOG:  background worker "logical replication launcher" (PID 51) exited with exit code 1
[graphql-api-687785f68b-6mbk7 postgres] 2018-12-14 00:40:06.551 UTC [46] LOG:  shutting down
[graphql-api-687785f68b-6mbk7 postgres] 2018-12-14 00:40:06.575 UTC [44] LOG:  database system is shut down
[graphql-api-687785f68b-6mbk7 postgres]  done
[graphql-api-687785f68b-6mbk7 postgres] server stopped
[graphql-api-687785f68b-6mbk7 postgres]
[graphql-api-687785f68b-6mbk7 postgres] PostgreSQL init process complete; ready for start up.
[graphql-api-687785f68b-6mbk7 postgres]
[graphql-api-687785f68b-6mbk7 postgres] 2018-12-14 00:40:06.662 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
[graphql-api-687785f68b-6mbk7 postgres] 2018-12-14 00:40:06.663 UTC [1] LOG:  listening on IPv6 address "::", port 5432
[graphql-api-687785f68b-6mbk7 postgres] 2018-12-14 00:40:06.667 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
[graphql-api-687785f68b-6mbk7 postgres] 2018-12-14 00:40:06.679 UTC [53] LOG:  database system was shut down at 2018-12-14 00:40:06 UTC
[graphql-api-687785f68b-6mbk7 postgres] 2018-12-14 00:40:06.684 UTC [1] LOG:  database system is ready to accept connections

i am not sure if its an issue with my docker file using multi phase build or two containers running within the same pod?

$ kgpoowide
NAME                           READY     STATUS             RESTARTS   AGE       IP           NODE
graphql-api-687785f68b-6mbk7   1/2       ImagePullBackOff   0          1m        10.1.1.5     docker-for-desktop

Containers:
  graphql-api:
    Container ID:
    Image:          shanelee007/graphql:da972dd-dirty-ca0890b
    Image ID:
    Port:           8000/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       ImagePullBackOff
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     200m
      memory:  400Mi
    Requests:
      cpu:     20m
      memory:  100Mi
    Environment Variables from:
      graphql-api-configuration  ConfigMap  Optional: false
    Environment:
      SECRET:             <set to the key 'SECRET' in secret 'graphql-api-secret'>             Optional: false
      TM_API_KEY:         <set to the key 'TM_API_KEY' in secret 'graphql-api-secret'>         Optional: false
      DATABASE_PASSWORD:  <set to the key 'DATABASE_PASSWORD' in secret 'graphql-api-secret'>  Optional: false
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-gnl9t (ro)
  postgres:
    Container ID:   docker://47b517c277214e5a4fb3e666a78355613d9bc4fb94248178a16010640216679d
    Image:          postgres:latest
    Image ID:       docker-pullable://postgres@sha256:28b4a1b0d3ae9b38a16c97a4e655bdc13496a304d1a241c531ac1efc2faa7230
    Port:           5432/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Fri, 14 Dec 2018 11:40:04 +1100
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-gnl9t (ro)
Conditions:
  Type           Status
  Initialized    True
  Ready          False
  PodScheduled   True
Volumes:
  default-token-gnl9t:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-gnl9t
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason                 Age               From                         Message
  ----     ------                 ----              ----                         -------
  Normal   Scheduled              1m                default-scheduler            Successfully assigned graphql-api-687785f68b-6mbk7 to docker-for-desktop
  Normal   SuccessfulMountVolume  1m                kubelet, docker-for-desktop  MountVolume.SetUp succeeded for volume "default-token-gnl9t"
  Normal   Pulling                1m                kubelet, docker-for-desktop  pulling image "postgres:latest"
  Normal   Created                1m                kubelet, docker-for-desktop  Created container
  Normal   Started                1m                kubelet, docker-for-desktop  Started container
  Normal   Pulled                 1m                kubelet, docker-for-desktop  Successfully pulled image "postgres:latest"
  Warning  Failed                 1m (x2 over 1m)   kubelet, docker-for-desktop  Failed to pull image "shanelee007/graphql:da972dd-dirty-ca0890b": rpc error: code = Unknown desc = Error response from daemon: manifest for shanelee007/graphql:da972dd-dirty-ca0890b not found
  Warning  Failed                 1m (x2 over 1m)   kubelet, docker-for-desktop  Error: ErrImagePull
  Normal   BackOff                54s (x3 over 1m)  kubelet, docker-for-desktop  Back-off pulling image "shanelee007/graphql:da972dd-dirty-ca0890b"
  Warning  Failed                 54s (x3 over 1m)  kubelet, docker-for-desktop  Error: ImagePullBackOff
  Warning  DNSConfigForming       43s (x7 over 1m)  kubelet, docker-for-desktop  Search Line limits were exceeded, some search paths have been omitted, the applied search line is: default.svc.cluster.local svc.cluster.local cluster.local seek.int sapdev.seek.int sapprod.seek.int
  Normal   Pulling                43s (x3 over 1m)  kubelet, docker-for-desktop  pulling image "shanelee007/graphql:da972dd-dirty-ca0890b"

[balchua]

Can you check if your image is in docker hub? I suspect that it is not. You can actually see that your image tag is da972dd-dirty-ca0890b.

What i do is that i instruct skaffold not to push, by indicating that its a local build.

build:
  artifact: . . .
local:
  #   false by default for local clusters, true for remote clusters
  #   push: false

I think skaffold uses some context to determine if it requires to push or not. If the context is minukube it doesn't do that.
But since you are using docker for mac, it doesn't see the context minikube.
Check the annotated skaffold config here.

https://github.com/GoogleContainerTools/skaffold/blob/master/examples/annotated-skaffold.yaml

[shavo007]

hi @balchua the docker image exists in docker hub but not that tag
https://hub.docker.com/r/shanelee007/graphql/tags

that image tag i believe is generated by skaffold locally

i am running skaffold dev which should be just local?

more info in the repo here https://github.com/shavo007/graphql-playground/tree/master/api#skaffold if you want to replicate

[balchua]

I can't test this right now, but can u try this skaffold config?

apiVersion: skaffold/v1beta1
kind: Config
build:
  artifacts:
  - image: shanelee007/graphql
  local:
    push: false
deploy:
  kubectl:
    manifests:
    - kubernetes/config.yaml
    - kubernetes/deployment.yaml
    - kubernetes/secret.yaml

[balchua]

I look at your deployment manifest, you useimagePullPolicy: Always which will definitely fail if the image is not pushed to the registry. Use imagepullpolicy IfNotPresent. You should be good to go.

[shavo007]

all good @balchua , that works now. thanks ;-)

see commit change here https://github.com/shavo007/graphql-playground/commit/ccc241009e1407dc8f58cccc29654d2df7225dcb

it might be worth noting that in the docs?

also i know from best practices, that imagePullPolicy should be Always. Its probably not a good idea if you are running locally skaffold dev and need to change your manifest for that reason?

[balchua]

First of all, great that you got it right. From my experience running kube is that i want to start the pod faster. If you have a large image, every time you start that container it will always look up to the registry and pull it.

I personally feel that imagePullPolicy: IfNotPresent is the best practice.

ImagePullPolicy is well documented in kubernetes.

[shavo007]

Ye, i had a look at the doc before. It states:

Note: The caching semantics of the underlying image provider make even imagePullPolicy: Always efficient. With Docker, for example, if the image already exists, the pull attempt is fast because all image layers are cached and no image download is needed.

I remember when I took the kubernetes CKA exam, i read many places to set to Always

But i guess there is no hard rule!!

Happy to close, or you want an update to the doc to outline if imagePullPolicy set to Always for dev will expect image to exist in remote registry?

[balchua]

Maybe a PR to update the documentation just to highlight this thing. Or you can close as you wish. Thanks again