Skaffold: Getting started example with Microk8s

Created on 23 Jan 2020  路  9Comments  路  Source: GoogleContainerTools/skaffold

Expected behavior

Follow the Quickstart docs, and it prints "Hello World" - assuming there is an intention to support microk8s, which might not be the case...

Actual behavior

Exits with:
FATA[0033] exiting dev mode because first build failed: build failed: building [skaffold-example]: build artifact: denied: requested access to the resource is denied

Information

  • Skaffold version: v1.2.0
  • Operating system: CentOS Linux release 7.7.1908
  • Microk8s version: v1.17.0 rev 1107

Steps to reproduce the behavior

  1. snap install microk8s --classic
  2. snap alias microk8s.kubectl kubectl
  3. microk8s.enable registry
  4. curl -Lo skaffold https://storage.googleapis.com/skaffold/releases/latest/skaffold-linux-amd64
  5. chmod +x skaffold
  6. sudo mv skaffold /usr/local/bin
  7. git clone https://github.com/GoogleContainerTools/skaffold
  8. [git head is 4c38a79c3a59a14cf4cae14015bbcfdf9816c491]
  9. cd skaffold/examples/getting-started/
  10. skaffold dev
    output:
    ```Listing files to watch...

    • skaffold-example

      Generating tags...

    • skaffold-example -> skaffold-example:v1.2.0-59-g4c38a79

      Checking cache...

    • skaffold-example: Not found. Building

      Building [skaffold-example]...

      Sending build context to Docker daemon 3.072kB

      Step 1/6 : FROM golang:1.12.9-alpine3.10 as builder

      1.12.9-alpine3.10: Pulling from library/golang

      9d48c3bd43c5: Pulling fs layer

      7f94eaf8af20: Pulling fs layer

      9fe9984849c1: Pulling fs layer

      cf0db633a67d: Pulling fs layer

      0f7136d71739: Pulling fs layer

      cf0db633a67d: Waiting

      0f7136d71739: Waiting

      9fe9984849c1: Verifying Checksum

      9fe9984849c1: Download complete

      7f94eaf8af20: Verifying Checksum

      7f94eaf8af20: Download complete

      9d48c3bd43c5: Verifying Checksum

      9d48c3bd43c5: Download complete

      9d48c3bd43c5: Pull complete

      7f94eaf8af20: Pull complete

      9fe9984849c1: Pull complete

      0f7136d71739: Verifying Checksum

      0f7136d71739: Download complete

      cf0db633a67d: Download complete

      cf0db633a67d: Pull complete

      0f7136d71739: Pull complete

      Digest: sha256:e0660b4f1e68e0d408420acb874b396fc6dd25e7c1d03ad36e7d6d1155a4dff6

      Status: Downloaded newer image for golang:1.12.9-alpine3.10

      ---> e0d646523991

      Step 2/6 : COPY main.go .

      ---> 1b6997ef9ee0

      Step 3/6 : RUN go build -o /app main.go

      ---> Running in 603687cb1a77

      ---> 032abb5e74f5

      Step 4/6 : FROM alpine:3.10

      3.10: Pulling from library/alpine

      4167d3e14976: Pulling fs layer

      4167d3e14976: Verifying Checksum

      4167d3e14976: Download complete

      4167d3e14976: Pull complete

      Digest: sha256:7c3773f7bcc969f03f8f653910001d99a9d324b4b9caa008846ad2c3089f5a5f

      Status: Downloaded newer image for alpine:3.10

      ---> af341ccd2df8

      Step 5/6 : CMD ["./app"]

      ---> Running in 4d1c367f795a

      ---> e18f25efd4c1

      Step 6/6 : COPY --from=builder /app .

      ---> 71f323031f96

      Successfully built 71f323031f96

      Successfully tagged skaffold-example:v1.2.0-59-g4c38a79

      The push refers to repository [docker.io/library/skaffold-example]

      4b23326d73d7: Preparing

      531743b7098c: Preparing

      FATA[0033] exiting dev mode because first build failed: build failed: building [skaffold-example]: build artifact: denied: requested access to the resource is denied

11. Based on #3486 edit skaffold.yaml and k8s-pod.yaml:
```YAML
# skaffold.yaml
apiVersion: skaffold/v2alpha2
kind: Config
build:
  local:
    push: true
    useDockerCLI: true
  artifacts:
  - image: 127.0.0.1:32000/skaffold-example
deploy:
  kubectl:
    manifests:
      - k8s-*

# k8s-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: getting-started
spec:
  containers:
  - name: getting-started
    image: 127.0.0.1:32000/skaffold-example
  1. skaffold dev
    output:
Listing files to watch...
 - 127.0.0.1:32000/skaffold-example
Generating tags...
 - 127.0.0.1:32000/skaffold-example -> 127.0.0.1:32000/skaffold-example:v1.2.0-59-g4c38a79-dirty
Checking cache...
 - 127.0.0.1:32000/skaffold-example: Found. Pushing
The push refers to repository [127.0.0.1:32000/skaffold-example]
4b23326d73d7: Preparing
531743b7098c: Preparing
531743b7098c: Layer already exists
4b23326d73d7: Layer already exists
v1.2.0-59-g4c38a79-dirty: digest: sha256:5058f56bf126d7ec7968de8ce5415a5a7b9cb3b153bc50f9dc7284a169128116 size: 739
Tags used in deployment:
 - 127.0.0.1:32000/skaffold-example -> 127.0.0.1:32000/skaffold-example:v1.2.0-59-g4c38a79-dirty@sha256:5058f56bf126d7ec7968de8ce5415a5a7b9cb3b153bc50f9dc7284a169128116
Starting deploy...
 - pod/getting-started created
Cleaning up...
 - pod "getting-started" deleted
FATA[0009] starting logger: initializing aggregate pod watcher: getting k8s client: getting client config for Kubernetes client: error creating REST client config in-cluster: unable to load in-cluster configuration, KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT must be defined

Any suggestions please?

kindocumentation prioritp2
Source
picture sfhardman
👍2

Most helpful comment

So it turns out the issue here is that I was using the microk8s kubectl as an alias. I had to run the following:

sudo snap unalias kubectl
sudo snap install kubectl --classic # This installs kubectl, it doesn't need to be done using snap though
microk8s.kubectl config view --raw > $HOME/.kube/config

Then everything worked just fine.

picture dungahk on 27 Jan 2020
👍6 🚀1

All 9 comments

Hi @sfhardman, thanks for the detailed issue!

Step 11 - you could just pass in the registry as well with --default-repo 127.0.0.1:32000 as a flag.
Not sure yet why you need the dockerCLI vs why the go-containerregistry doesn't work - probably you'd have to add 127.0.0.1:32000 to insecure-registries?

Step 12 - now that is an odd one - skaffold expects a ~/.kube/config to exist, i.e. if an existing kubectl would work then skaffold should work. From what I gather microk8s does not rely on that file but manages its own and provides a microk8s.kubectl command. What happens if you run sudo microk8s.kubectl config view --raw > $HOME/.kube/microk8s.cfg, and run skaffold with --kube-context=microk8s --kubeconfig=$HOME/.kube/microk8s.cfg?

balopat picture balopat on 27 Jan 2020
👍2

Thanks @balopat I've been trying to do the same thing the OP is trying.

I tried your suggestions (just a side note, it's --kubeconfig instead of --kube-config) and it still does not work, this is what I got:

INFO[0000] starting gRPC server on port 50051           
INFO[0000] starting gRPC HTTP server on port 50052      
INFO[0000] Skaffold &{Version:v1.2.0 ConfigVersion:skaffold/v2alpha2 GitVersion: GitCommit:80f82f42fe271aea1058f4a37776d52ab5a7c441 GitTreeState:clean BuildDate:2020-01-17T01:04:55Z GoVersion:go1.13.6 Compiler:gc Platform:linux/amd64} 
INFO[0000] Activated kube-context "microk8s"            
INFO[0000] Using kubectl context: microk8s              
Listing files to watch...
 - localhost:32000/skaffold-example
INFO[0000] List generated in 1.898702ms                 
Generating tags...
 - localhost:32000/skaffold-example -> localhost:32000/skaffold-example:v1.2.0-76-g7978e172e-dirty
INFO[0000] Tags generated in 6.697932ms                 
Checking cache...
 - localhost:32000/skaffold-example: Found Remotely
INFO[0000] Cache check complete in 3.344587ms           
Tags used in deployment:
 - localhost:32000/skaffold-example -> localhost:32000/skaffold-example:v1.2.0-76-g7978e172e-dirty@sha256:bcf3f33232460aa3ebd678163f498d3617b22352e095481c10870564ebf98793
Starting deploy...
 - pod/getting-started created
INFO[0000] Deploy complete in 638.878794ms              
Cleaning up...
 - pod "getting-started" deleted
INFO[0004] Cleanup complete in 3.692586598s             
FATA[0004] starting logger: initializing aggregate pod watcher: getting k8s client: getting client config for Kubernetes client: error creating REST client config for kubeContext 'microk8s': context "microk8s" does not exist
dungahk picture dungahk on 27 Jan 2020

So it turns out the issue here is that I was using the microk8s kubectl as an alias. I had to run the following:

sudo snap unalias kubectl
sudo snap install kubectl --classic # This installs kubectl, it doesn't need to be done using snap though
microk8s.kubectl config view --raw > $HOME/.kube/config

Then everything worked just fine.

dungahk picture dungahk on 27 Jan 2020
👍6 🚀1

@dungahk Would you mind posting the command line that's working for you? I'm using:

skaffold dev --default-repo=127.0.0.1:32000 --kubeconfig="$HOME/.kube/microk8s.cfg" --kube-context=microk8s --insecure-registry=127.0.0.1:32000

Which is failing, mostly with:

FATA[0002] starting logger: initializing aggregate pod watcher: getting k8s client: getting client config for Kubernetes client: error creating REST client config for kubeContext 'microk8s': context "microk8s" does not exist

But every three or four runs with:

rpc error: code = Unknown desc = failed to resolve image "127.0.0.1:32000/skaffold-example@sha256:5058f56bf126d7ec7968de8ce5415a5a7b9cb3b153bc50f9dc7284a169128116": no available registry endpoint: failed to do request: Head https://127.0.0.1:32000/v2/skaffold-example/manifests/sha256:5058f56bf126d7ec7968de8ce5415a5a7b9cb3b153bc50f9dc7284a169128116: http: server gave HTTP response to HTTPS client
sfhardman picture sfhardman on 27 Jan 2020

Thanks for the feedback @dungahk! @sfhardman does this solve your problem too?
If it does, we could document this in our getting started guides for local development.

balopat picture balopat on 27 Jan 2020

@sfhardman -
Did you run microk8s.kubectl config view --raw > $HOME/.kube/microk8s.cfg?
Regarding the server gave HTTP response to HTTPS client error - if you're using --insecure-registry can you try to run it without the dockerCLI:true setting?

balopat picture balopat on 27 Jan 2020

@sfhardman This is my skaffold file (the only change is the localhost:32000 added)

apiVersion: skaffold/v2alpha2
kind: Config
build:
  artifacts:
    - image: localhost:32000/skaffold-example
deploy:
  kubectl:
    manifests:
      - k8s-*

This is my k8s-pod.yaml file, again, the only change is the localhost:32000 added:

apiVersion: v1
kind: Pod
metadata:
  name: getting-started
spec:
  containers:
    - name: getting-started
      image: localhost:32000/skaffold-example

The command I run now is skaffold dev simple as that.

I suppose that I don't even need to add the localhost:32000 to the files and instead use the CLI parameter --default-repo, but I haven't had the chance to check that yet.

dungahk picture dungahk on 27 Jan 2020
👍1

Thanks - I now have things working. Things seem to be a bit sensitive as to how the local registry is referred to (i.e. looks like it wants to be called localhost, not referenced by ip address, and localhost can't resolve to an ipv6 address).

The getting started example works without changes to the yaml with these extra steps:

  1. Check /etc/hosts does not resolve localhost to an ipv6 address (::1)
  2. microk8s.kubectl config view --raw > $HOME/.kube/config
  3. skaffold dev --default-repo=localhost:32000
sfhardman picture sfhardman on 27 Jan 2020
👍2

@dungahk

The command I run now is skaffold dev simple as that.

Yeah, I'm just running skaffold dev as-is given I have similarly declarative configs. Thank you very much for posting your fix - I, too, was using kubectl as an alias and suspect I would've squandered an hour debugging had I not stumbled across your comment.

MatthewZito picture MatthewZito on 17 Jun 2020
👍2
Was this page helpful?
0 / 5 - 0 ratings

Related issues

emichaf picture emichaf  路  4Comments
garvincasimir picture garvincasimir  路  4Comments
kdevu picture kdevu  路  3Comments
gbird3 picture gbird3  路  3Comments
achedeuzot picture achedeuzot  路  3Comments