Serving: Revision: cannot use private registry

[bappr]

In what area(s)?

/area autoscale

What version of Knative?

0.0.6

Expected Behavior

I should be able to use a private registry.

Actual Behavior

Following https://knative.dev/docs/serving/samples/build-private-repo-go/ I added the secret in the service account but still not able to pull the private image (working fine with a simple deployment)
Error in the events:

DENIED: Permission denied for "helloworld" from request "/v2/.../helloworld/manifests/helloworldexample".

Error in the revision:

- lastTransitionTime: "2019-06-20T06:17:04Z"
    message: 'Unable to fetch image ".../helloworld:helloworldexample":
      DENIED: Permission denied for "helloworldexample" from request "/v2/.../helloworld/manifests/helloworldexample". '
    reason: ContainerMissing
    status: "False"
    type: Ready

Steps to Reproduce the Problem

I guess you can try to create a service with an image stored in a private registry.

[jonjohnsonjr]

@bappr can you share what registry you're using?

This errors is coming from our tag to digest resolution. As a workaround, you can disable that with the config-deployment config map, but ideally this would work by default 😄

If you're using ECR, I'm working on it: https://github.com/knative/serving/pull/4084

[bappr]

Thanks @jonjohnsonjr, I am using Google Docker Registry (asia.gcr.io).
So I will configure it in the config map. ;) Thanks.

[jonjohnsonjr]

Wait, that should work by default!

If you're running on GCE, your GCE nodes' service account should have access to GCR within the same project. If you're not relying on your GCE service account (e.g. you're using image pull secrets), you should be able to add those to the controller spec.

[jonjohnsonjr]

/reopen

This might just be a docs issue, but I want to keep this open until we resolve it.

[knative-prow-robot]

@jonjohnsonjr: Reopened this issue.

In response to this:

/reopen

This might just be a docs issue, but I want to keep this open until we resolve it.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[bappr]

Indeed, I am not using the service account (and we don't intend to do since the installation should be cloud agnostic). What exactly should I add to the controller spec?

[mattmoor]

@bappr How are you expressing the GCR credentials? If you are using imagePullSecrets attached to the service account that things run as, thing should work.

[bappr]

@mattmoor that's what I did. And it's working fine with a simple deployment.

[mattmoor]

@bappr do you mean a simple K8s Deployment of a Knative "deployment" (a la Service)?

[bappr]

I mean a simple K8S deployment created with the command kubectl run something --image=myprivateregistry/image -n default. This works and k8s is able to get my image.

However, it does not work with the Knative service.

[mattmoor]

@bappr and this is using GCR with image pull secrets attached to the default service account?

[bappr]

Yes

[mattmoor]

That's quite odd, as the "k8schain" package we are using should be using the core K8s logic to pass those credentials to the registry.

Can you confirm that putting it in the configmap makes things work?

[mattmoor]

Also, what type of credential is in the pull secret? I'm assuming _json_key?

[bappr]

@mattmoor so no it still does not work even after modifying the configmap (I added asia.gcr.io)

And yes I am using _json_key for the image pull secret.

[mattmoor]

@bappr it might be easier to debug this if we manage to connect on slack? Do you want to start a group chat with @jonjohnsonjr and myself?

[bappr]

Sure. Done

[mattmoor]

Alright, so we figured it out.

Kubernetes supports a special non-standard form of credential in their keyring, which allows credentials to be selected based on a repository (e.g. asia.gcr.io/foo), not just a registry (e.g. asia.gcr.io).

I believe this is because imagePullSecrets are specified at one or both of the Pod or ServiceAccount level (vs. the Container level), which means that you could end up needing multiple distinct credentials for the same registry for different images in the same pod.

e.g. gcr.io/foo/bar:baz and it's sidecar gcr.io/blah/bloop:bleep might need different credentials.

However, the standard keychain dispatches on the registry hostname.

When I originally wrote k8schain to bridge the K8s keychain and the ggcr keychain concept, I used this little hack to get around this, which works perfectly for well-formed (in the docker sense) keychains:
https://github.com/google/go-containerregistry/blob/ca8b66cb1b793f7fb0e3c191ad9a37eb3dc80ea0/pkg/authn/k8schain/k8schain.go#L159

However, for keychains taking advantage of this niche K8s feature, things clearly do not work.

The simple workaround here is to change the key (within the base64 encoded secret blob) from "asia.gcr.io/blah": {...} to "asia.gcr.io": {...}, which for @bappr seems to have done the trick, but we should find a way to squash this inconsistency to keep it from biting other folks.

[mattmoor]

I'm going to move this out of 0.8 since we have a workaround, and I think I'm fairly comfortable landing it in 0.9. I'd rather @jonjohnsonjr focus on the lifecycle changes.

[ahmetb]

Also worth considering adding a more verbose error message around the tag resolver, like:

Error occurred while resolving image digest in knative-serving controller: [...]

Otherwise it's really difficult for an end-user to debug why it's failing (despite image pull secrets are configured). In one case, we ended up spending almost an hour trying to understand why image pull secret isn't working.

[mattmoor]

@ahmetb Can you open a separate issue so that feedback isn't lost when the concrete issue here is resolved?