Minikube: No example to use minikube with private registry

[dirkjonker]

I found it very cumbersome to use minikube with a private (gcr.io) container register. I ended up copying the .docker/config.json to the minikube VM, manually pulling the wanted image and specifying imagePullPolicy: Never because I couldn't get minikube to pull and couldn't find any useful examples in the documentation to do it differently.

Can somebody provide an example of how to get minikube to pull images by itself?

[dlorenc]

In general using any of the tutorials around ImagePullSecrets should work, but unfortunately gcr authenticates differently than other registries. I think we'll need to add a flag to support the --google-json-key kubelet parameter.

[dirkjonker]

For Google Container Registry (using oauth2) this works for me:

kubectl create secret docker-registry myregistrykey --docker-server=https://gcr.io --docker-username=oauth2accesstoken --docker-password="$(gcloud auth print-access-token)" [email protected]

Then in the specification just use imagePullSecrets: - name: myregistrykey

Might be really useful to add something similar to the documentation since people (like me) may find it easier to get started using k8s.

Did not figure out how to use a service account json key though.

[sandinmyjoints]

@dirkjonker Thanks for posting that. I'm trying to get it working. For username=oauth2accesstoken, is that meant literally, or if not, how would I get the correct value for this field? Also, how did you find print-access-token as a command to gcloud auth? It is not listed when I do gcloud auth --help.

[sandinmyjoints]

Nevermind about my first question, just checked the pod again and turns it out worked with username=oauth2accesstoken. (But I'm also curious as to how you found out that that is a valid value for that field when authing with GCR!)

[dirkjonker]

@sandinmyjoints I found it in the Google Container Registry documentation: https://cloud.google.com/container-registry/docs/advanced-authentication

[waprin]

Is there big opposition to just doing special case logic for gcr.io that either looks for wherever gcloud stores its creds or shells out to gcloud and does this for you?

[srounce]

After experiencing much the same pain, I came up with this useful method using the script below using service accounts (GCE only):

1. Create a service account for your local cluster. Be sure to select JSON when asked.
2. Set the IAM policy on the service account you just created.
   
   
   
   1. Uncheck _Editor_
   
   
   2. Check _Storage > Storage Object Viewer_
   
   
   3. Click Save
   
   
3. Use the script (docker-secret.sh) provided below to create the image pull secret
4. In your podspec add the ImagePullSecret:

...
    spec:
      imagePullSecrets:
      - name: docker-registry-secret
      containers:
      - name: gateway
        image: gcr.io/my-fun-project/gateway-service:latest
...

docker-secret.sh (gist)

Usage:

docker-secret.sh <secret name> <service account key>

: The name of the ImagePullSecret to be created
: The path to your GCP service account JSON keyfile

Example:

$ docker-secret.sh docker-registry-secret my-service-account-key.json

$ docker-secret.sh docker-registry-secret my-service-account-key.json --namespace=myapp

Installation

curl https://cdn.rawgit.com/srounce/4a0338b26df815e966174228753ef61e/raw/docker-secret.sh) > docker-secret.sh; 
chmod +x docker-secret.sh;

[srounce]

@dlorenc Would implementing this behaviour as part of minikube in a pluggable way to support other providers (#366) be a good first contribution? Or is this planned to be supported some other way/are there more pressing first tasks?
If so is there anything related to be aware of?

[dlorenc]

@waprin: The problem with copying gcloud's model is that it uses a temporary access token, so image pull secrets would only be valid for ten minutes. We need to get a permanent one, like in the steps @srounce automated.

@srounce: I like the idea of pluggable helpers, but I think it might actually being in kubectl instead of minikube. The core problem is applicable to any production cluster. What are your thoughts on trying to do this in kubectl, or even as a separate binary. I'd be happy to ship that binary with minikube.

[srounce]

Yes, that would make sense, I had this exact issue the other day with build agents unable to push to google registry because the GKE cluster they were on didn't have correct permissions. Would've been nice to have a one-liner for that. How do you propose starting this? I'm not too familiar with the K8s codebase but with some guidance I should be okay.

[dlorenc]

@janetkuo started a doc somewhere about kubectl UX enhancements, I'd try to reach out to her.

Janet, have you looked into kubectl helpers that make it easier to create ImagePullSecrets for the various private registries?

[Cloven]

Coming into minikube as a first time kuberneticist and docker user, there were a number of dead ends and surprises trying to get a private image back out of GCR with minikube. @dirkjonker's approach turned out to be the magic phrasing, but it's a synthesis of the k8s image doc (http://kubernetes.io/docs/user-guide/images/#specifying-imagepullsecrets-on-a-pod) and the GCR docker doc (https://cloud.google.com/container-registry/docs/advanced-authentication) that I expect few to be able to successfully figure out de novo.

What do people think of a 'first five minutes of minikube' doc that attempts to go just a little deeper than the quickstart? If nobody else is working on something like that, I could give it a go.

[dlorenc]

That would be awesome @Cloven !

[srounce]

@Cloven Did you try the steps I outlined above? Some feedback would be great as PR'ing this as a minikube feature is on my todo list (just not top of it however).

It would be cool to see how you get on with this "First 5mins" doc, as I'm particularly interested in pain points for the first time K8s user.

[waprin]

Not sure if this is helpful or not but dropping it off just in case:

https://github.com/GoogleCloudPlatform/docker-credential-gcr

[Cloven]

@srounce I didn't try your method because I had previously resolved it with Dirk's one-liner. What if any benefit does it provide over the Dirk technique?

After contemplating this over the weekend, I think most of my first-five-minutes confusion stems from the docs' dependence on assumption of familiarity with using docker. While that might be a pretty natural thing to assume for the authors (it's the water in which they swim), it's not necessarily a given for incoming minikube users.

@dlorenc your choice -- would you like (a) a cut at docs/minikube_first_five_minutes_with_gcr.md, or (b) a fresh-user experience report as an external blog post -- assuming I don't have the time to do both?

[dlorenc]

I think you're spot on, we currently assume a base level of familiarity with docker. We should work to make the onboarding experience even smoother to users that haven't played with containers before.

A blog post sounds like a better fit. We can incorporate parts of it into our docs after.

Let me know when you get a chance to write it! I'm happy to take an early look and help share it once it's ready!

[srounce]

@Cloven The approach I took can use GCP service accounts/IAM policies rather than just grabbing your gcloud auth token. The upshot of this is that you don't need gcloud installed. This in itself useful for teaching k8s _only_, without muddying the water with cloud provider specifics, but still keeping your demo containers private.
Allows distribution of service account key to team members to allow image pull and nothing else. Also, instead of managing many GCP users and permissions, you now manage 1 service account, and 1 key. Thus exposing yourself a lot less, not to mention making it much easier to nuke the key in case of emergency.
There were probably other reasons (that I can't think of now because I just woke up 😄)

[wuman]

@srounce Your parameters (secret_name, config_path) should be swapped to reflect what the script actually dictates.

[srounce]

@wuman Could you please elaborate further? You mean it is a bug on my part? Or a suggestion?

[wuman]

@srounce The script you provided via gist takes the config_path as the first parameter and the secret_name as the second parameter:

CONFIG_PATH=${1:-$SPATH/localkube.json}
SECRET_NAME=${2:-docker-registry-secret}

But the usage and examples you gave have the secret_name as the first parameter and the config_path as the second parameter.

[srounce]

Updated. Well spotted, thx.

[bgrant0607]

Example using a private registry:
http://thenewstack.io/tutorial-configuring-ultimate-development-environment-kubernetes/

We do have a registry addon that could be run:
https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/registry

[jbehrends]

Thought I would share my experience. We have a private registry that has auth enabled (username/password). It was actually quite simple to get minikube to work properly.

Steps:

minikube start (if it wasn't already running)
minikube ssh
sudo docker login private-registry.my.domain
<Enter the appropriate user and password>
exit

Since kubernetes is running under root within the minikube vm the above command created ~/.docker/config.json under root using the entered credentials.

That was it! I can now create pods/deployments/etc using containers from our secured private registry.

[btipling]

@srounce I got your approach to work, but the json and imagePullSecrets name are reversed in your script.

[srounce]

@btipling Thanks for catching that. I had updated the example and not the installation script provided. I think I'd initially made the mistake of editing the script and not the example provided and so everything was out of alignment. I think we're all good now 😰😉

Also reminds me, I need to rebase and push my branch with this functionality added to the minikube commands. Would be not only more accessible and useful, but also easier to maintain than the above.

[btipling]

@srounce I didn't see a license for your script, I'm using it here. I'd be happy to add an attribution or license or something. Just a personal project, just feels weird copying your script like that though.

[dlorenc]

Ref #612

[CodeSchwert]

I found this tutorial worked for JSON key files and service accounts.
http://ryaneschinger.com/blog/using-google-container-registry-gcr-with-minikube/

[srounce]

@btipling Thx, updated the script with Usage statement and licence. Feedback plz.

[btipling]

@srounce Thanks I kept the older version of your script (because it works for me and I modified it a bit), but added the your license to the top of it.

[stevesloka]

The registry-addon which just got merged will address this and let you sync credentials automatically: https://github.com/kubernetes/minikube#private-container-registries & https://github.com/upmc-enterprises/registry-creds

[ghost]

link was dead, i think this is related: https://github.com/kubernetes/minikube/blob/f5e73b08ba64792b9effe44cd21c228a2080353f/docs/insecure_registry.md#enabling-docker-insecure-registry

[mrts]

The URL anchor identifier is actually a bit different: https://github.com/kubernetes/minikube/blob/master/docs/insecure_registry.md#private-container-registries

[davesmits]

did someone ever tried minikube with azure container registry?