Kops: Set ~/.docker/config.json on all nodes for private registry

[blakebarnett]

It would be great if this were part of the cluster spec, something like:

globalPrivateRegistryAuth:
  quay.io: <auth code>

Which would then write that to /root/.docker/config.json on all nodes.

[olalonde]

FWIW, there is a way to do this using kubectl, something like this I think:

configjson=$(cat ~/.docker/config.json)
kubectl apply -f - << EOF
apiVersion: v1
kind: Secret
metadata:
  name: quay-registry
data:
  .dockerconfigjson: $(echo "$configjson" | /usr/bin/base64)
type: kubernetes.io/dockerconfigjson
EOF

[blakebarnett]

I should have said this in the first place, but as you probably know secrets are per-namespace. So using a secret as above means you have to ensure the secret is created and kept up-to-date in all namespaces, this is annoyingly tedious. I would love to have a single place to manage this.

[jaipradeesh]

@blakebarnett Maybe you can do the following?

• Run docker login [server] for each set of credentials you want to use. This updates $HOME/.docker/config.json

• View $HOME/.docker/config.json in an editor to ensure it contains just the credentials you want to use.

• Get a list of your nodes, for example:

  • If you want the names: nodes=$(kubectl get nodes -o jsonpath='{range.items[*].metadata}{.name} {end}')
  • If you want to get the IPs: nodes=$(kubectl get nodes -o jsonpath='{range .items[*].status.addresses[?(@.type=="ExternalIP")]}{.address} {end}')

• Copy your local .docker/config.json to the home directory of root on each node. For example: for n in $nodes; do scp ~/.docker/config.json root@$n:/root/.docker/config.json; done

Ref: https://kubernetes.io/docs/concepts/containers/images/#configuring-nodes-to-authenticate-to-a-private-repository

[blakebarnett]

I understand the work-arounds that are possible. This is a feature request for Kops.

[chrislovecnm]

@blakebarnett going to add this to my backlog. I will need this as well. Self hosting will not be possible without this.

I am thinking we need the values in a kops secret.

Thoughts?

[jaipradeesh]

kops secret, +1

But we might need to refresh it every 12 hours. Is there a cron/job already for this somewhere?

[blakebarnett]

yeah a secret sounds great, would this be something protokube would monitor for changes or something like that?

[chrislovecnm]

Do we need a refresh on this or can we use a system account? Refresh every 12 hours would be not fun to maintain.

[jaipradeesh]

I don't understand what you mean by use a system account. AFAIK, there is no other way but to refresh the credentials.

[blakebarnett]

The refresh could be handled during kops update as long as something is watching for the changes and applying them on the nodes.

[blakebarnett]

Seems this is still doable, just need specific vars in the systemd unit file: https://github.com/kubernetes/kubernetes/issues/45487#issuecomment-312042754

[chrislovecnm]

@blakebarnett oooohhhh that is cool

[chrislovecnm]

@blakebarnett you planning on knocking this out?

[blakebarnett]

If I can get some time soon!

[hunsche]

How do I spin the secrets without recreating the nodes?

As suggested by the kops update.

[dobesv]

To roll the nodes, you can use kops rolling-update --force

For anyone coming here looking for this feature, the docs are here:

https://github.com/kubernetes/kops/blob/master/docs/cli/kops_create_secret_dockerconfig.md

The secret is written to /root/.docker/config.json