Serving: How to use locally built docker image?

[xphoniex]

In what area(s)?

/area API
/area autoscale
/area build
/area monitoring
/area networking
/area test-and-release

Ask your question here:

I have built a simple docker image for a node app that serves a hello world on port 8080, if I minikube ssh and run docker images I can see my image under minikube-test.

I can deploy my service normally in minikube like this (and it works):

.
.
containers:
  - name: minikube-test
    image: minikube-test:v1
    imagePullPolicy: Never

However I'm getting revision error when deploying it to knative and some investigation shows it can't pull the image.

service.yaml:

   apiVersion: serving.knative.dev/v1
   kind: Service
   metadata:
     name: minikube-test
   spec:
     template:
       spec:
         containers:
           - name: minikube-test
             image: minikube-test:v1
             imagePullPolicy: Never
             env:
               - name: TARGET
                 value: "minikube-test knative"

then:

$ kubectl apply -f service.yaml
service.serving.knative.dev/minikube-test created

$ kubectl get routes
NAME            URL                                        READY   REASON
minikube-test   http://minikube-test.default.example.com   False   RevisionMissing

$ kubectl describe ksvc/minikube-test
Name:         minikube-test
Namespace:    default
.
.
Status:
  Conditions:
    Last Transition Time:        2019-11-26T12:08:39Z
    Message:                     Revision "minikube-test-srm6c" failed with message: Unable to fetch image "minikube-test:v1": unrecognized HTTP status: 403 Forbidden.
    Reason:                      RevisionFailed
    Status:                      False
    Type:                        ConfigurationsReady
    Last Transition Time:        2019-11-26T12:08:39Z
    Message:                     Configuration "minikube-test" does not have any ready Revision.
    Reason:                      RevisionMissing
.
.

Help?

[xphoniex]

anyone?

[chansonzhang]

same question

[ivangfr]

I am having the same problem!

[dgerd]

What is most likely happening is that Knative is failing to do digest resolution. In our controller we resolve image tags to digests in order in order to better guarantee immutability of Revisions. This works well if you are using a normal registry; however, it can cause problems when used with mini kube and local images. Our controller is likely trying to pull this image from DockerHub to resolve the digest (as it has no domain name prefix) and then failing.

We have a configmap option you can enable to skip doing digest resolution on certain paths. You will want to edit config-deployment in the knative-serving namespace to whitelist your registry to skip digest resolution.

https://github.com/knative/serving/blob/master/config/config-deployment.yaml#L45

Alternatively, you can also pass the image digest rather than tag to the Knative Service. Since it is already resolved the logic will be skipped.

[xphoniex]

Locally built docker images don't seem to have a digest. (Possible explanation: Why doesn't my newly-created docker have a digest?)

$ docker images --digests | grep minikube-test
minikube-test                       v1        <none>        ffe2d2bc2aa4        8 months ago        76.1MB
minikube/minikube-test              v1        <none>        ffe2d2bc2aa4        8 months ago        76.1MB   

(minikube/minikube-test is just a label of minikube-test here.)

However this command returns the digest:

$ docker images --no-trunc --quiet minikube-test:v1
sha256:ffe2d2bc2aa418ee2050d6d7e6b405e8a9636159edeea7c01dfb503082f6de28

I updated my knative service.yaml, and added the digest:

         containers:
           - name: minikube-test
             image: minikube-test:v1@sha256:ffe2d2bc2aa418ee2050d6d7e6b405e8a9636159edeea7c01dfb503082f6de28
             imagePullPolicy: Never

Here's the new error:

$ kubectl describe ksvc/minikube-test
.
.
Status:
  Conditions:
    Last Transition Time:        2019-12-19T06:15:39Z
    Message:                     Revision "minikube-test-26fnw" failed with message: Container image "minikube-test:v1@sha256:ffe2d2bc2aa418ee2050d6d7e6b405e8a9636159edeea7c01dfb503082f6de28" is not present with pull policy of Never.
    Reason:                      RevisionFailed
    Status:                      False
    Type:                        ConfigurationsReady
.
.

So I removed imagePullPolicy: Never from my yaml file and tried again:

Status:
  Conditions:
    Last Transition Time:        2019-12-19T06:33:40Z
    Message:                     Revision "minikube-test-8h2kv" failed with message: Back-off pulling image "minikube/minikube-test@sha256:ffe2d2bc2aa418ee2050d6d7e6b405e8a9636159edeea7c01dfb503082f6de28".
    Reason:                      RevisionFailed
    Status:                      False
    Type:                        ConfigurationsReady

Still no luck so I tried editing config-deployment:

$ kubectl edit configmaps config-deployment --namespace knative-serving
.
.
# List of repositories for which tag to digest resolving should be skipped
    registriesSkippingTagResolving: "ko.local,dev.local,minikube"
.
.

I added minikube to the repositories as I have the label minikube/minikube-test and saved the file. new error:

Status:
  Conditions:
    Last Transition Time:        2019-12-19T06:44:49Z
    Message:                     Revision "minikube-test-7bvwx" failed with message: Unable to fetch image "minikube/minikube-test:v1": UNAUTHORIZED: authentication required; [map[Action:pull Class: Name:minikube/minikube-test Type:repository]].
    Reason:                      RevisionFailed
    Status:                      False
    Type:                        ConfigurationsReady

What should I try next @dgerd ?

[dgerd]

Oh I see what you are doing here now and what you are expecting from Minikube.

A docker image without a hostname prefix is assumed to be DockerHub. Your image pull policy of 'Never' prevents minikube from reaching out to DockerHub to try to fetch the image and assumes that the version you have locally is the latest and greatest.

The last step should have been successful with an Image Pull policy of 'Never'. What I suspect might have went wrong here is that you edited the existing line. The existing line is part of an _example key and modifying it wont take effect. You need to copy the line out of the example and move it up to be its own key.

Alternatively, by default we will skip resolving digests with images prefixed with dev.local or ko.local (as the example line suggests). With your image pull policy kept at 'Never' I would rename your docker image to dev.local/minikube-test:<<your-tag>> and try that.

[xphoniex]

Tagging my image as dev.local/minikube-test solved the issue.

I think I'm editing the right place only it's still taking DockerHub as the source because if I use

registriesSkippingTagResolving: ko.local,dev.local,mini.kube // notice the dot

instead of minikube everything works perfectly, just like with dev.local

[dgerd]

Glad to hear it solved the problem. Makes sense that the dot is required as part of the hostname parsing.

Do you happen to have a link to the documentation you were looking at to set this up originally? I can add some text about this so others don't run into the same issue.

[xphoniex]

I think I'm editing the right place only it's still taking DockerHub as the source because if I use

registriesSkippingTagResolving: ko.local,dev.local,mini.kube // notice the dot

instead of minikube everything works perfectly, just like with dev.local

actully dismiss this part. I just learned that the image was not being pulled correctly and it was probably using the prior revision hence why it still worked fine. the error:

    conditions:
    - lastTransitionTime: "2019-12-19T22:23:56Z"
      message: 'Revision "minikube-test-ws4kq" failed with message: Container image
        "mini.kube/minikube-test:v1" is not present with pull policy of Never.'

Here's my knative-serving/config-deployment pasted directly from terminal, which looks okay to me:

data:
  _example: |
    ################################
    #                              #
    #    EXAMPLE CONFIGURATION     #
    #                              #
    ################################

    # This block is not actually functional configuration,
    # but serves to illustrate the available configuration
    # options and document them in a way that is accessible
    # to users that `kubectl edit` this config map.
    #
    # These sample configuration options may be copied out of
    # this example block and unindented to be in the data block
    # to actually change the configuration.

    # List of repositories for which tag to digest resolving should be skipped
    registriesSkippingTagResolving: "ko.local,dev.local"
  queueSidecarImage: gcr.azk8s.cn/knative-releases/knative.dev/serving/cmd/queue@sha256:5ff357b66622c98f24c56bba0a866be5e097306b83c5e6c41c28b6e87ec64c7c
  registriesSkippingTagResolving: ko.local,dev.local,mini.kube
kind: ConfigMap

Regarding the documentation, I think I briefly looked at hello-world examples. An example helloworld-nodejs. I will probably create a pull request for this issue and a few others if that's ok with you.

[dgerd]

actully dismiss this part. I just learned that the image was not being pulled correctly and it was probably using the prior revision hence why it still worked fine. the error:

You image will not be 'pulled' in this setup. My understanding is it is not being pushed to any registry, but rather using the images that are being built and available locally from the docker daemon.

Here's my knative-serving/config-deployment pasted directly from terminal, which looks okay to me:

This looks correct to me as well

[dgerd]

I will probably create a pull request for this issue and a few others if that's ok with you.

That is more than okay. Would love the contribution!