Skaffold: k3s local deployment without docker registry not working

[kohlerm]

Expected behavior

Running kubernetes with k3s it should be possible to run skaffold dev with the need for a registry
changed the image names and some congurations:

Actual behavior

I tried running
https://github.com/GoogleContainerTools/skaffold/tree/master/examples/jib
against a local k3s cluster (all on on Linux machine).

git diff .
diff --git a/examples/jib/k8s/web.yaml b/examples/jib/k8s/web.yaml
index e0382bd2..d0b75223 100644
--- a/examples/jib/k8s/web.yaml
+++ b/examples/jib/k8s/web.yaml
@@ -25,6 +25,7 @@ spec:
     spec:
       containers:
       - name: web
-        image: gcr.io/k8s-skaffold/skaffold-jib
+        image: kohlerm/skaffold-jib
+        imagePullPolicy: IfNotPresent
         ports:
           - containerPort: 8080
diff --git a/examples/jib/skaffold.yaml b/examples/jib/skaffold.yaml
index 046bdf20..35fabe31 100644
--- a/examples/jib/skaffold.yaml
+++ b/examples/jib/skaffold.yaml
@@ -2,8 +2,9 @@ apiVersion: skaffold/v1beta11
 kind: Config
 build:
   artifacts:
-  - image: gcr.io/k8s-skaffold/skaffold-jib
+  - image: kohlerm/skaffold-jib
     jibMaven: {}
+  local: {}

 # optional profile to run the jib build on Google Cloud Build
 profiles:
diff --git a/examples/jib/src/main/java/hello/HelloController.java b/examples/jib/src/main/java/hello/HelloController.java
index b98a365c..af5aeeeb 100644
--- a/examples/jib/src/main/java/hello/HelloController.java
+++ b/examples/jib/src/main/java/hello/HelloController.java
@@ -7,6 +7,6 @@ import org.springframework.web.bind.annotation.RequestMapping;
 public class HelloController {
     @RequestMapping("/")
     public String index() {
-        return "Hello, World!";
+        return "Hello, World!!!";
     }
 }
\ No newline at end of file

Information

• Skaffold version: v0.31.0
• Operating system: Ubuntu 19.10
• Contents of skaffold.yaml:
  see above

Steps to reproduce the behavior

run skaffold dev

I get

Back-off pulling image "kohlerm/skaffold-jib:6575ca9afe7bf9fbb2be86a0ebf8894adfb851e1416cfb20b963c9b53bb2e7a8"

[web-795b445cfd-tq4bc istio-proxy] 2019-06-18T12:35:52.880428Z  info    Envoy proxy is ready
Back-off pulling image "kohlerm/skaffold-jib:6575ca9afe7bf9fbb2be86a0ebf8894adfb851e1416cfb20b963c9b53bb2e7a8"

pc error: code = Unknown desc = failed to resolve image "docker.io/kohlerm/skaffold-jib:6575ca9afe7bf9fbb2be86a0ebf8894adfb851e1416cfb20b963c9b53bb2e7a8": no available registry endpoint: docker.io/kohlerm/skaffold-jib:6575ca9afe7bf9fbb2be86a0ebf8894adfb851e1416cfb20b963c9b53bb2e7a8 not found   

So it seems it is always trying to pull the image from docker.io, which is not what I would expect because the image is avaliable locally (verified with the docker CLI)

[briandealwis]

Skaffold's "local cluster" support has the builders build directly to the cluster's docker daemon. You're building to your machine's docker daemon. k3s uses containerd rather than docker, and I don't believe containerd supports loading images like the docker daemon.

From what I read, k3s does support using docker instead of containerd but that seemed more involved.

[briandealwis]

There's an article describing what's required to setup a local registry for micro k8s which should be equally applicable for k3s.

I personally found it easier to use minikube or Docker's k8s :-/

[kohlerm]

Thx for the hint! Tried with microk8s, but does not work either.
Learned from https://stackoverflow.com/questions/55297278/how-to-use-local-docker-images-with-microk8s that it also switched to containerd. I guess I have to setup a local registry ...

[balopat]

Skaffold's "local cluster" support has the builders build directly to the _cluster's_ docker daemon. You're building to your machine's docker daemon. k3s uses containerd rather than docker, and I don't believe containerd supports loading images like the docker daemon.

I think in theory it should be possible to load images to containerd? https://github.com/containerd/cri/blob/master/docs/crictl.md#directly-load-a-container-image
Not sure how well this works with k3s though.

[poikilotherm]

You might be interested in reading rancher/k3s#590 and rancher/k3d#19 for recently added image loading support.

[murm-github]

https://microk8s.io/docs/working this link has detailed info to solve this problem for microk8s and i could resolve it using public registry method.

[tejal29]

Looks we need to follow up with some docs and user journey example on this. I am going to lower the priority on this

[jon-maki]

I am also interested in getting some feedback and potentially support on using skaffold + k3d in an offline environment without access to a docker registry.

Background

I work on distributed microservice-style web applications that we deploy using docker-compose and kubernetes. We've built and leveraged an extensive amount of tooling around our docker image building and deploying.

Our docker-compose development, test and prod deployment strategies are pretty mature at this point. Team members can clone a repository, run docker-compose up and immediately start developing. Any compiled applications (golang apps, in our case) require a rebuild and docker-compose restart <app>, but any other apps (Flask and Django, primarily) have hot-reloading configured so code changes are immediately reflected in the running docker-compose development deployment.

A typical product that we work on contains 5-25 microservice apps. Each of these apps are in their own repository. Each app runs by itself and exposes an API that other applications may leverage. The core business logic of the product composes these APIs in various ways to meet the end use-case. There is a "product" repository for each product that implements the necessary tooling to clone the appropriate repos, check out the correct branches, and get them up and running in developer mode (think something like the repo tool for Android development). This repository also implements tooling to produce all release artifacts using a GitLab CI pipeline triggered by tagging the repository for release.

Our kubernetes strategies are in their infancy and need a lot of work before we can get to the same level of maturity. skaffold looks _really_ promising as our tooling of choice for developing for kubernetes.

Use Case

My use case for using k3d + skaffold is as follows:

1. I work primarily in an offline (non internet-connected) environment. My development environment mirrors common package repositories (debian, alpine, pypi, etc.). Building arbitrary docker images from the common base images (such as alpine:3.10) is supported on my developer network.
2. With the recent changes to k3d to include image import support, I am able to build images using my local docker daemon and import them into my local k3d-managed cluster relatively easily.
3. I can use a skaffold custom build stage to build an image and import it before the deploy stage.
4. Local docker daemon image caching is my friend here - I can build the image once while connected to the developer network and then use my local cache to build all subsequent images from after unplugging from the developer network. This strategy obviously breaks if I determine I need to add another dependency into my Dockerfile, but for the most part it works pretty well.

I see here that the next steps for the local k3d support story are to create docs and user journey examples. I worked on this over the weekend and have an example of a skaffold continuous development pipeline for a hot-reloading Flask application deployed by kubectl into a local k3d-managed cluster (without a docker registry) here: https://github.com/jon-maki/flask-echo-server.

To start developing with that example project, you simply run a k3d cluster locally (k3d create --image rancher/k3s:v0.10.2), clone the project and run skaffold dev --port-forward.

I tried to document my decisions and discoveries as thoroughly as I could. I hope this helps others that may be in a similar position as me.

I would love some feedback on how I went about implementing this. I may have missed some things from the docs and may have made some bad assumptions about how skaffold works under the hood.

I am open to contributing something similar to this as an example if the community feels like it would be useful.

I am really looking forward to continuing to work with skaffold to flesh out our kubernetes development story.

[udanpe]

I tried to run with docker, but this does not work too.

k3d create \
--api-port 127.0.0.1:6550 \
--publish 80:80 \
--publish 443:443 \
--volume /usr/local/bin/docker:/usr/bin/docker \
--volume /var/run/docker.sock:/var/run/docker.sock \
--image docker.io/rancher/k3s:v1.17.2-k3s1 \
--server-arg '--docker'

k3d can start a local registry this may be a temporary solution

k3d create --enable-registry ...
127.0.0.1 registry.local # /etc/hosts

[nkubala]

@jon-maki just got around to reading up your writeup and trying out your example repository, it's way cool 👏 I would definitely be interested in you contributing this as an example in the skaffold repo if you want to! i'm also curious if you've give minikube a shot yet, and what your experience with it has been.

[tstromberg]

Thank you for the feedback!

I'm closing this issue as it's been open a while, and it is not clear if it's still an open issue. No one has recently stated an interest in addressing this, but if you feel strongly about it, please feel free to add a comment or send us a PR!

[briandealwis]

A user asked about this on the #skaffold channel. Skaffold now supports k3d (k3s in docker, #2753) which supports loading images from the local docker daemon via a k3d image import command. The underlying implementation in k3d uses docker save to export an image to a tar ball and then loads the tarball into containerd using the ctr CLI tool.

The k3s CLI tool now exposes the ctr to interact with its containerd (k3s ctr image import; see this helpful walkthrough for an example), so Skaffold could use docker save and load the resulting tarball. But docker save is very slow: on my MBP with SSD, saving a 450MB image took about 10 s. It's likely much faster to host a local registry within the cluster and push to that registry. But we'll likely hit the registry aliasing issue #3461.