Minikube: preload removes images built by user in docker-env

Created on 10 Mar 2020  ·  9Comments  ·  Source: kubernetes/minikube

with docker-env on: build an image:

medya@~/Desktop $ docker build -t medya .
Sending build context to Docker daemon  23.29MB
Step 1/2 : FROM alpine
 ---> e7d92cdc71fe
Step 2/2 : RUN echo "HEllo world from medya" /hello.txt
 ---> Using cache
 ---> f4a736bf846b
Successfully built f4a736bf846b
Successfully tagged medya:latest

check the image is tehere

medya@~/Desktop $ docker images
REPOSITORY                                TAG                 IMAGE ID            CREATED             SIZE
medya                                     latest              f4a736bf846b        30 seconds ago      5.59MB
k8s.gcr.io/kube-proxy                     v1.17.3             ae853e93800d        3 weeks ago         116MB
k8s.gcr.io/kube-controller-manager        v1.17.3             b0f1517c1f4b        3 weeks ago         161MB
k8s.gcr.io/kube-apiserver                 v1.17.3             90d27391b780        3 weeks ago         171MB
k8s.gcr.io/kube-scheduler                 v1.17.3             d109c0821a2b        3 weeks ago         94.4MB
alpine                                    latest              e7d92cdc71fe        7 weeks ago         5.59MB
kubernetesui/dashboard                    v2.0.0-beta8        eb51a3597525        3 months ago        90.8MB
k8s.gcr.io/coredns                        1.6.5               70f311871ae1        4 months ago        41.6MB
kindest/kindnetd                          0.5.3               aa67fec7d7ef        4 months ago        78.5MB
k8s.gcr.io/etcd                           3.4.3-0             303ce5db0e90        4 months ago        288MB
kubernetesui/metrics-scraper              v1.0.2              3b08661dc379        4 months ago        40.1MB
k8s.gcr.io/pause                          3.1                 da86e6ba6ca1        2 years ago         742kB
gcr.io/k8s-minikube/storage-provisioner   v1.8.1              4689081edb10        2 years ago         80.8MB

re-load minikube

medya@~/Desktop $ minikube start
😄  minikube v1.8.1 on Darwin 10.14.6
    ▪ MINIKUBE_ACTIVE_DOCKERD=minikube
✨  Using the hyperkit driver based on existing profile
💿  Downloading VM boot image ...
⌛  Reconfiguring existing host ...
🏃  Using the running hyperkit "minikube" VM ...
🐳  Preparing Kubernetes v1.17.3 on Docker 19.03.6 ...
🚀  Launching Kubernetes ... 
🌟  Enabling addons: default-storageclass, storage-provisioner
🏄  Done! kubectl is now configured to use "minikube"

check for images again, it is gone :(

medya@~/Desktop $ docker images
REPOSITORY                                TAG                 IMAGE ID            CREATED             SIZE
<none>                                    <none>              f4a736bf846b        3 minutes ago       5.59MB
k8s.gcr.io/kube-proxy                     v1.17.3             ae853e93800d        3 weeks ago         116MB
k8s.gcr.io/kube-apiserver                 v1.17.3             90d27391b780        3 weeks ago         171MB
k8s.gcr.io/kube-controller-manager        v1.17.3             b0f1517c1f4b        3 weeks ago         161MB
k8s.gcr.io/kube-scheduler                 v1.17.3             d109c0821a2b        3 weeks ago         94.4MB
kubernetesui/dashboard                    v2.0.0-beta8        eb51a3597525        3 months ago        90.8MB
k8s.gcr.io/coredns                        1.6.5               70f311871ae1        4 months ago        41.6MB
kindest/kindnetd                          0.5.3               aa67fec7d7ef        4 months ago        78.5MB
k8s.gcr.io/etcd                           3.4.3-0             303ce5db0e90        4 months ago        288MB
kubernetesui/metrics-scraper              v1.0.2              3b08661dc379        4 months ago        40.1MB
k8s.gcr.io/pause                          3.1                 da86e6ba6ca1        2 years ago         742kB
gcr.io/k8s-minikube/storage-provisioner   v1.8.1              4689081edb10        2 years ago         80.8MB
cdocker-env cpreload kinbug prioritimportant-soon
Source
picture medyagh
👍2

Most helpful comment

Hi!

I confirm the exact same behavior on:

  • Ubuntu 19.10 host
  • VirtualBox 6.0.14
  • minikube 1.8.1
  • kubectl 1.17.3

If I use --kubernetes-version=1.16.7 I get the expected behavior.

picture a-yiorgos on 10 Mar 2020
👍3

All 9 comments

Maybe need to reindexed , if you are adding the storage layers directly ?

afbjorklund picture afbjorklund on 10 Mar 2020

Hi!

I confirm the exact same behavior on:

  • Ubuntu 19.10 host
  • VirtualBox 6.0.14
  • minikube 1.8.1
  • kubectl 1.17.3

If I use --kubernetes-version=1.16.7 I get the expected behavior.

a-yiorgos picture a-yiorgos on 10 Mar 2020
👍3

@medyagh : and with "index", I meant the repositories.json

In the code, it is known as the refstore: https://github.com/moby/moby/blob/master/reference/store.go

afbjorklund picture afbjorklund on 10 Mar 2020

If I use --kubernetes-version=1.16.7 I get the expected behavior.

Probably only because there is no preload (yet?) for that version...

afbjorklund picture afbjorklund on 10 Mar 2020

since this behaviour only happesn on a minikube reload (start again) I feel like if we make minikube not to preload on a "restart" or "soft restart" then we should not see this problem.

Wdyt @priyawadhwa ?

medyagh picture medyagh on 10 Mar 2020

If I use --kubernetes-version=1.16.7 I get the expected behavior.

Probably only because there is no preload (yet?) for that version...

so you think the user's image is not over-written by our tar, it just needs re-index? how do we do re-index?

medyagh picture medyagh on 10 Mar 2020

Well, the layers themselves should be under the sha256 directories. It's the tags that need "merging"

My tarball has this: (jq < ./lib/docker/image/overlay2/repositories.json)

{
  "Repositories": {
    "gcr.io/k8s-minikube/storage-provisioner": {
      "gcr.io/k8s-minikube/storage-provisioner:v1.8.1": "sha256:4689081edb103a9e8174bf23a255bfbe0b2d9ed82edc907abab6989d1c60f02c",
      "gcr.io/k8s-minikube/storage-provisioner@sha256:088daa9fcbccf04c3f415d77d5a6360d2803922190b675cb7fc88a9d2d91985a": "sha256:4689081edb103a9e8174bf23a255bfbe0b2d9ed82edc907abab6989d1c60f02c"
    },
    "k8s.gcr.io/coredns": {
      "k8s.gcr.io/coredns:1.6.5": "sha256:70f311871ae12c14bd0e02028f249f933f925e4370744e4e35f706da773a8f61",
      "k8s.gcr.io/coredns@sha256:7ec975f167d815311a7136c32e70735f0d00b73781365df1befd46ed35bd4fe7": "sha256:70f311871ae12c14bd0e02028f249f933f925e4370744e4e35f706da773a8f61"
    },
    "k8s.gcr.io/etcd": {
      "k8s.gcr.io/etcd:3.4.3-0": "sha256:303ce5db0e90dab1c5728ec70d21091201a23cdf8aeca70ab54943bbaaf0833f",
      "k8s.gcr.io/etcd@sha256:4afb99b4690b418ffc2ceb67e1a17376457e441c1f09ab55447f0aaf992fa646": "sha256:303ce5db0e90dab1c5728ec70d21091201a23cdf8aeca70ab54943bbaaf0833f"
    },
    "k8s.gcr.io/kube-apiserver": {
      "k8s.gcr.io/kube-apiserver:v1.17.3": "sha256:90d27391b7808cde8d9a81cfa43b1e81de5c4912b4b52a7dccb19eb4fe3c236b",
      "k8s.gcr.io/kube-apiserver@sha256:33400ea29255bd20714b6b8092b22ebb045ae134030d6bf476bddfed9d33e900": "sha256:90d27391b7808cde8d9a81cfa43b1e81de5c4912b4b52a7dccb19eb4fe3c236b"
    },
    "k8s.gcr.io/kube-controller-manager": {
      "k8s.gcr.io/kube-controller-manager:v1.17.3": "sha256:b0f1517c1f4bb153597033d2efd81a9ac630e6a569307f993b2c0368afcf0302",
      "k8s.gcr.io/kube-controller-manager@sha256:2f0bf4d08e72a1fd6327c8eca3a72ad21af3a608283423bb3c10c98e68759844": "sha256:b0f1517c1f4bb153597033d2efd81a9ac630e6a569307f993b2c0368afcf0302"
    },
    "k8s.gcr.io/kube-proxy": {
      "k8s.gcr.io/kube-proxy:v1.17.3": "sha256:ae853e93800dc2572aeb425e5765cf9b25212bfc43695299e61dece06cffa4a1",
      "k8s.gcr.io/kube-proxy@sha256:3a70e2ab8d1d623680191a1a1f1dcb0bdbfd388784b1f153d5630a7397a63fd4": "sha256:ae853e93800dc2572aeb425e5765cf9b25212bfc43695299e61dece06cffa4a1"
    },
    "k8s.gcr.io/kube-scheduler": {
      "k8s.gcr.io/kube-scheduler:v1.17.3": "sha256:d109c0821a2b9225b69b99a95000df5cd1de5d606bc187b3620d730d7769c6ad",
      "k8s.gcr.io/kube-scheduler@sha256:b091f0db3bc61a3339fd3ba7ebb06c984c4ded32e1f2b1ef0fbdfab638e88462": "sha256:d109c0821a2b9225b69b99a95000df5cd1de5d606bc187b3620d730d7769c6ad"
    },
    "k8s.gcr.io/pause": {
      "k8s.gcr.io/pause:3.1": "sha256:da86e6ba6ca197bf6bc5e9d900febd906b133eaa4750e6bed647b0fbe50ed43e",
      "k8s.gcr.io/pause@sha256:f78411e19d84a252e53bff71a4407a5686c46983a2c2eeed83929b888179acea": "sha256:da86e6ba6ca197bf6bc5e9d900febd906b133eaa4750e6bed647b0fbe50ed43e"
    },
    "kindest/kindnetd": {
      "kindest/kindnetd:0.5.3": "sha256:aa67fec7d7ef71445da9a84e9bc88afca2538e9a0aebcba6ef9509b7cf313d17",
      "kindest/kindnetd@sha256:bc1833b3da442bb639008dd5a62861a0419d3f64b58fce6fb38b749105232555": "sha256:aa67fec7d7ef71445da9a84e9bc88afca2538e9a0aebcba6ef9509b7cf313d17"
    },
    "kubernetesui/dashboard": {
      "kubernetesui/dashboard:v2.0.0-beta8": "sha256:eb51a359752560a66f314602e87155b75f428fb838bf951079ff1f9621958c0c",
      "kubernetesui/dashboard@sha256:fc90baec4fb62b809051a3227e71266c0427240685139bbd5673282715924ea7": "sha256:eb51a359752560a66f314602e87155b75f428fb838bf951079ff1f9621958c0c"
    },
    "kubernetesui/metrics-scraper": {
      "kubernetesui/metrics-scraper:v1.0.2": "sha256:3b08661dc379d9f80155be9d658f71578988640357ebae1aab287d6954c723d1",
      "kubernetesui/metrics-scraper@sha256:2026f9f7558d0f25cc6bab74ea201b4e9d5668fbc378ef64e13fddaea570efc0": "sha256:3b08661dc379d9f80155be9d658f71578988640357ebae1aab287d6954c723d1"
    }
  }
}
afbjorklund picture afbjorklund on 10 Mar 2020

Yah, so the layers are fine but the repositories.json is being overwritten so the tags disappear. I think skipping preload on reload is probably the right move here, since all the needed images should be loaded into the daemon anyway.

priyawadhwa picture priyawadhwa on 10 Mar 2020
👍1

Exact same issue, ubuntu 18.04, minikube v1.8.1, kvm2 driver. Please fix and release.
Tried to roll back down to v1.8.0, but it even happens on v1.8.0.
Rolled back again to v1.7.3, seems fine on v1.7.3.

hiren1771 picture hiren1771 on 11 Mar 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

tstromberg picture tstromberg  ·  3Comments
javajon picture javajon  ·  3Comments
cherish3620 picture cherish3620  ·  3Comments
hank-cp picture hank-cp  ·  3Comments
Starefossen picture Starefossen  ·  3Comments