Minikube: How to pull images from ECR?

[sooraj-e]

I'm trying to setup minikube for local development. Is there any way to pull images from private AWS ECR?

[dlorenc]

Hey,

I'm not too familiar with ECR. Have you tried using an ImagePullSecret?

[cayter]

Assuming that you're on Mac, below is what you will need to do:

• Install awscli and configure with your AWS credentials

$ pip install awscli 
$ aws configure

• Copy this python snippet below into generate_secret_key.py.

#!/usr/bin/env python

import re
import subprocess

def execute_cmd(cmd):
  proc = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
  comm = proc.communicate()

  if comm[1] != '':
    print(comm[1].rstrip('\n'))
    exit(-1)

  return comm[0]

def generate_secret_key():
  login_cmd = execute_cmd('aws ecr get-login').rstrip('\n')
  creds = re.sub(r"(-e none\ |docker login\ |-u\ |-p\ )", '', login_cmd).split(' ')
  generate_secret_cmd = "kubectl create secret docker-registry {0} --docker-username={1} --docker-password={2} --docker-server={3} --docker-email=YOUR_EMAIL_ADDRESS"
  execute_cmd(generate_secret_cmd.format('YOUR_SECRET_KEY_NAME', creds[0], creds[1], creds[2].replace('https://', '')))

if __name__ == "__main__":
  generate_secret_key()

NOTE: Remember to change YOUR_EMAIL_ADDRESS and YOUR_SECRET_KEY_NAME.

• Change the file permission and execute it.

$ chmod a+x generate_secret_key.py
$ ./generate_secret_key.py

• Update the Kubernetes resource config, say deployment.json.

{
  "apiVersion": "extensions/v1beta1",
  "kind": "Deployment",
  "metadata": {
    "name": "your-app"
  },
  "spec": {
    "replicas": 1,
    "template": {
      "metadata": {
        "labels": {
          "app": "your-app"
        }
      },
      "spec": {
        "containers": [
          {
            "name": "redis",
            "image": "YOUR_AWS_ECR_IMAGE_URL",
            "ports": [
              {
                "hostPort": 30003,
                "containerPort": 6379
              }
            ]
          }
        ],
        "restartPolicy": "Always",
        "imagePullSecrets": [
          {
            "name": "YOUR_SECRET_KEY_NAME"
          }
        ]
      }
    }
  }
}

NOTE: Remember to change YOUR_SECRET_KEY_NAME as the same one generated in the python script.

• Create resource from deployment.json.

$ kubectl create -f deployment.json

PS: I've been playing around with minikube + k8s + AWS ECR for a few hours. Looks promising. The only downside is that the ECR image pulling in the k8s cluster(VM) is slow. I'm not sure if GCR would be faster. Might need to test out more.

[stevesloka]

This should work as well, I'm going to validate / update soon since I havn't touched it in a while, but feel free to give it a whirl: https://github.com/upmc-enterprises/awsecr-creds

It just automates getting ECR credentials and refreshing the imagePullSecrets before they expire.

[stevesloka]

I confirm it works, need to remove this from the replication controller kubernetes.io/cluster-service: "true". I'll write a blog post on it soon.

[r2d4]

that would be awesome @stevesloka , thanks for figuring this out!

[drusellers]

Would love to see this native in minikube - but thank you for write up @stevesloka

[arohner]

Thanks, and agreed about wanting a native solution. Namely, running kube in production on AWS + ECR, I don't need image pull secrets at all. Using the workaround, I have to add imagePullSecrets to my deployments, which then need to be refreshed periodically.

[dlorenc]

Thanks for the feedback. We should definitely try to fix this natively. I'm not too familiar with the ECR, though.

Could you explain how this might work?

[stevesloka]

Hey @arohner, my solution which pulls creds automatically doesn't need you to change your deployments at all. You can simply start up the controller and it should 'just work'.

@dlorenc the integration on AWS works like this. You setup a cloud provider in the kubelet, which then uses IAM credentials attached to the instance to auth to ECR.

It's my assumption that no one is running minikube on AWS (rather only on local development machine). So in any integration in minikube you'd need to pass some IAM credentials to get ECR credentials.

Maybe we could bundle up my controller as an add-on? Would that make it easier to deploy?

[dlorenc]

Yeah that would be perfect! We're already doing some work now to make it easy to enable/disable custom add-ons. Packaging your controller this way sounds perfect.

[donaldguy]

@stevesloka is making this an addon still on your todo list? any ETA?

I'm gonna need it sooner rather than later and could try to tackle it but I'm pretty new to k8s as a whole, so I bet you could do it faster/better :D

[stevesloka]

@donaldguy honestly I lost track of this, but I'm super happy to help out. @dlorenc has any work been done to help with add/ons?

I could also package this up into a Helm chart if that's easier as well. Thoughts?

[dlorenc]

Yup! If you have an addon ready you can follow these steps to get it bundled with minikube: https://github.com/kubernetes/minikube/blob/master/ADD_ADDON.md

[stevesloka]

Boom, sounds good. @donaldguy I can take a crack, will update you when I have something to play with. =)

[stevesloka]

Ok so I have it all basically coded up. My one question @dlorenc is where to supply values to the add-on? I need to pass AWS key / id as well as an AWS account. I can have them read from a secret, and the user would enable the addon and then create the secret manually.

What do you think the best impl is for this?

[r2d4]

The addon framework we have right now doesn't support any sort of parameterization. The addon yamls are ssh'd into the VM and the addon-manager handles it from there.

I think reading them from a secret and having the user add that manually is sensible

[dlorenc]

+1 on a secret.

[dlorenc]

We can add a callback to this addon that prints out some help text explaining how to create the secret, too.

[donaldguy]

Thanks! 😄