Microk8s: Error when pulling image built locally without a registry

[gostaj]

I've followed the instructions from https://microk8s.io/docs/working "Working with locally built images without a registry" and I'm building a JVM app from Maven with the jib-maven-plugin outputting to <image>outlooktoslack:local</image>.

# microk8s.ctr -n k8s.io images ls
docker.io/library/outlooktoslack:local  application/vnd.oci.image.manifest.v1+json 
 sha256:daa6391342d9283046fd75c55c4e255f1a3ae58afb005617e6d56e7607e1fd8e 74.3 MiB  linux/amd64  io.cri-containerd.image=managed

I've tried with deployment as:

      containers:
        - name: outlooktoslack
          image: outlooktoslack:local

and

      containers:
        - name: outlooktoslack
          image: docker.io/library/outlooktoslack:local

but to no avail:

# kubectl get pods
outlooktoslack-56c8fd6956-qxm4g   0/1     ErrImagePull   0          99s

# kubectl describe pod outlooktoslack-6679cbddf5-85qhg
 Normal   Pulling    19m (x4 over 21m)   kubelet, uppdragsboten-ny-ubuntu18  Pulling image "outlooktoslack:local"
  Warning  Failed     19m (x4 over 21m)   kubelet, uppdragsboten-ny-ubuntu18  Failed to pull image "outlooktoslack:local": rpc error: code = Unknown desc = failed to resolve image "docker.io/library/outlooktoslack:local": no available registry endpoint: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
  Warning  Failed     19m (x4 over 21m)   kubelet, uppdragsboten-ny-ubuntu18  Error: ErrImagePull



md5-e3c963b864e553f01ce9ba43f58230ae



# kubectl describe pod outlooktoslack-5878cf5-bpmgg

  Warning  Failed     17s               kubelet, uppdragsboten-ny-ubuntu18  Error: ImagePullBackOff
  Normal   Pulling    3s (x2 over 17s)  kubelet, uppdragsboten-ny-ubuntu18  Pulling image "docker.io/library/outlooktoslack:local"
  Warning  Failed     2s (x2 over 17s)  kubelet, uppdragsboten-ny-ubuntu18  Failed to pull image "docker.io/library/outlooktoslack:local": rpc error: code = Unknown desc = failed to resolve image "docker.io/library/outlooktoslack:local": no available registry endpoint: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed

I'm probably the one doing something wrong but you might be better at understanding what it is?
Would be super happy with any help!

[balchua]

May i know what is your imagePullPolicy defined in your deployment or pod?

[gostaj]

May i know what is your imagePullPolicy defined in your deployment or pod?

I have not defined an imagePullPolicy.

[balchua]

This is strange.

mind sharing your deployment/pod manifest?
You should see something like this when you describe the pod

Normal   Pulled            38m        kubelet, norse     Container image "nats-streaming:0.16.2" already present on machine

Thanks.

[balchua]

@gostaj
Can you add imagePullPolicy: Never ? or IfNotPresent

[gostaj]

That was it! Great @balchua !
imagePullPolicy: Never was the only thing that was needed.

I think it would be great to include this information under "Working with locally built images without a registry" at https://microk8s.io/docs/working

Thank you very much for you help!

[balchua]

@gostaj, great news. Do you mind creating an issue ?

[balchua]

@gostaj i just realized that you are using jib. In fact jib can create oci compliant images without docker.

I also use jib maven. And i find it simpler to enable the registry addon in microk8s than saving and importing it using ctr.
It's even better when u pair jib with skaffold.

I have some samples that i can share if you're interested.

[gostaj]

Yes please @balchua do that.
I have enabled the registry addon in microk8s but didn't know that it's not used by ctr 🙈 .
I'm building my images locally right now and want them running on a rather closed down VM running microk8s.

[balchua]

@gostaj, the registry addon is used by microk8s.ctr. The microk8s guide you pointed out explains how to use.

I have this project in github (shameless plug) https://github.com/balchua/demo

Checkout the skaffold.yaml file there. For example https://github.com/balchua/demo/blob/master/votes/skaffold.yaml

I may have some skaffold there still using dockercli, but its safe to turn it to false.

My normal dev workflow is building images using jib, push it to the registry addon in microk8s. Then deploy the apps in microk8s. All using skaffold run -p <profile> or sometimes skaffold dev -p <profile>.