Skaffold: Artifact with different registry hostnames

[jponge]

Expected behavior

I am building a container image using Maven, Jib, and I am using MicroK8S / Multipass as my local testing environment.

I would expect logs to appear when running skaffold dev --tail --port-forward.

Actual behavior

This is not the case because I have to use different hostnames for the registry, so Skaffold does not detect the dependency between an artifact to build and its usage in a Kubernetes resource file.

Information

• Skaffold version: 1.1.0
• Operating system: macOS Catalina
• Contents of skaffold.yaml:

apiVersion: skaffold/v1
kind: Config
metadata:
  name: rain-and-umbrella
build:
  artifacts:
    - image: microk8s:32000/rain-service
      jib: {}
      context: rain-service
    - image: microk8s:32000/do-i-need-my-umbrella-service
      jib: {}
      context: do-i-need-my-umbrella-service
  insecureRegistries:
    - microk8s:32000
deploy:
  kubectl:
    manifests:
    - "**/k8s/*.yaml"

The issue appears because the MicroK8s registry needs to be accessed from localhost from inside the cluster, as shown in this resources definition:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: rain-service
  name: rain-service
spec:
  selector:
    matchLabels:
      app: rain-service
  template:
    metadata:
      labels:
        app: rain-service
    spec:
      containers:
        - image: localhost:32000/rain-service
          name: rain-service
          livenessProbe:
            httpGet:
              path: /health
              port: 8080
            initialDelaySeconds: 3
            periodSeconds: 5
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: rain-service
  name: rain-service
spec:
  ports:
    - protocol: TCP
      port: 8080
  selector:
    app: rain-service
  type: LoadBalancer

We can see the following warnings in Skaffold logs:

WARN[0007] image [microk8s:32000/do-i-need-my-umbrella-service] is not used by the deployment
WARN[0007] image [microk8s:32000/rain-service] is not used by the deployment

[dgageot]

Hi @jponge I'll take a look at that tomorrow.

[dgageot]

Hey @jponge, that use case is indeed not supported by Skaffold. I'll run a few tests with microk8s and see what we can do.

[jponge]

That'd work with a public 3rd-party registry, but clearly a private registry as in MicroK8s is better for local development since you don't need to go on a potentially slow network.

[briandealwis]

@jponge: does skaffold dev's re-deploy step actually take effect? Because your k8s manifest doesn't reference the image as known to Skaffold, I don't see how it can update the image digest in your deployment, and so k8s won't actually redeploy.

It seems we may need the ability to add registry aliases?

build:
  ...
  insecureRegistries:
    - microk8s:32000
deploy:
  ...
  registryAliases:
    "microk8s:32000": "localhost:32000"

(edited as registryAliases would be better in the deploy section)

[jponge]

The problem is exactly that Skaffold and the K8s resources reference the image with a different prefix

[jponge]

Aliases could be a solution

[briandealwis]

KEP-1755 describes a mechanism for a cluster to expose information about a local registry.

[sherif84]

+1

this is a much needed feature for us too . our use case is below :

in our environment we have 2 image repos . dev.repo.com and lab.repo.com which are mirrors of each other . due to some networking restrictions , developers can only push to dev.repo.com and the k8s cluster in the lab can only pull from lab.repo.com .