I am using minikube 1.0.0 with kubernetes v1.14.0 (on Fedora Linux, xen2 driver).
I am using a docker client connected to the minikube docker, and issueing docker build commands, but I really need to use buildkit.
If I set "DOCKER_BUILDKIT=1" the docker client complains because the daemon does not support buildkit.
I tried all the different cointainer engines supported by minikube, but the result is always the same.
Then I tried running docker in experimental mode inside the minikube vm:
- ssh into the vm
- created the docker config file with the experimental=true option
- restarted the docker service with systemctl
Then the error I got on a docker build was "pivot_root invalid argument".
I tried setting DOCKER_RAMDISK to true or false, but nothing changed.
It might be that docker 18.09 fixes this, or it might be a tiny core linux issue, but understanding that docker engine subtlety is beyond me.
What I know is that the same docker build succeeds on my workstation (Fedora 29 with docker 18.09.3).
Given clear instructions I could try building minikube with docker 18.09 and test it with that.
IMHO supporting buildkit in local dev environments should be desirable...
massimiliano-mantione
All 14 comments
This looks to be a bug with the new "builder-next" (buildkit) builder in Docker upstream.
It is _supposed_ to pass the --no-pivot parameter to runc, but doesn't do so.
That would have the same effect as the NoPivotRoot used by libcontainerd:
https://github.com/moby/moby/pull/21679/commits/8a4225cd5ad3ad56a7e896f675080c0ed38982a3
The root cause for this is that minikube runs on rootfs, which doesn't allow pivot_root(2)
Minikube doesn't run tiny core linux anymore, which means the Boot2Docker fix is not here:
$ docker-machine ssh
( '>')
/) TC (\ Core is distributed with ABSOLUTELY NO WARRANTY.
(/-_--_-\) www.tinycorelinux.net
docker@default:~$ df -h
Filesystem Size Used Available Use% Mounted on
tmpfs 890.4M 229.6M 660.8M 26% /
tmpfs 494.7M 0 494.7M 0% /dev/shm
/dev/sda1 17.8G 247.9M 16.7G 1% /mnt/sda1
...
tmpfs on / type tmpfs (rw,relatime,size=911788k)
$ minikube ssh
_ _
_ _ ( ) ( )
___ ___ (_) ___ (_)| |/') _ _ | |_ __
/' _ ` _ `\| |/' _ `\| || , < ( ) ( )| '_`\ /'__`\
| ( ) ( ) || || ( ) || || |\`\ | (_) || |_) )( ___/
(_) (_) (_)(_)(_) (_)(_)(_) (_)`\___/'(_,__/'`\____)
$ df -h / /dev/shm /mnt/sda1
Filesystem Size Used Avail Use% Mounted on
rootfs 0 0 0 - /
tmpfs 996M 0 996M 0% /dev/shm
/dev/sda1 17G 1.4G 15G 9% /mnt/sda1
rootfs on / type rootfs (rw)
They have moved away from rootfs, which is something that minikube.iso also should do...
See #3512
afbjorklund
on 24 Apr 2019
Should have mention that the end result of all this, is that BuildKit works fine in Boot2Docker 18.09.5:
docker@default:~$ DOCKER_BUILDKIT=1 docker build .
[+] Building 0.2s (6/6) FINISHED
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 64B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/library/busybox:latest 0.0s
=> [1/2] FROM docker.io/library/busybox 0.0s
=> => resolve docker.io/library/busybox:latest 0.0s
=> [2/2] RUN true 0.2s
=> exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:44f8812a320e821b7ad6036e4a75b7741d12139ab0edc62c4bff211b4445f810 0.0s
But probably we need to do more in the minikube.iso, than just upgrading Docker from 18.06 to 18.09
afbjorklund
on 24 Apr 2019
Related to #3512 and #3519
tstromberg
on 25 Apr 2019
Depends on #4180
afbjorklund
on 4 May 2019
AkihiroSuda
on 7 May 2019
@AkihiroSuda : thanks, that should do the trick until we get #3512 sorted out
afbjorklund
on 7 May 2019
AkihiroSuda
on 11 May 2019
Same with 1.1.0
antonmarin
on 22 May 2019
We did not have time to patch and rebuild docker from source or to switch away from using rootfs for this minikube release.
So there is no buildkit support (yet), even though Docker was upgraded to 18.09.
afbjorklund
on 22 May 2019
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale
fejta-bot
on 22 Aug 2019
When do you expect minikube to support BuildKit?
demisx
on 24 Aug 2019
We are aiming to move away from rootfs for v1.4.0
/remove-lifecycle stale
afbjorklund
on 24 Aug 2019
Confirmed that building images with BuildKit works OK, with the new ISO:
$ DOCKER_BUILDKIT=1 docker build .
[+] Building 3.1s (6/6) FINISHED
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 65B 0.0s
=> [internal] load metadata for docker.io/library/busybox:latest 1.9s
=> [1/2] FROM docker.io/library/busybox@sha256:9f1003c480699be56815db0f8146ad2e22efea85129b5b5983d0e0fb52d9ab70 0.7s
=> => resolve docker.io/library/busybox@sha256:9f1003c480699be56815db0f8146ad2e22efea85129b5b5983d0e0fb52d9ab70 0.0s
=> => sha256:9f1003c480699be56815db0f8146ad2e22efea85129b5b5983d0e0fb52d9ab70 1.86kB / 1.86kB 0.0s
=> => sha256:895ab622e92e18d6b461d671081757af7dbaa3b00e3e28e12505af7817f73649 527B / 527B 0.0s
=> => sha256:db8ee88ad75f6bdc74663f4992a185e2722fa29573abcc1a19186cc5ec09dceb 1.50kB / 1.50kB 0.0s
=> => sha256:ee153a04d6837058642958836062f20badf39f558be3e6c7c7773ef7d8301d90 762.73kB / 762.73kB 0.4s
=> => extracting sha256:ee153a04d6837058642958836062f20badf39f558be3e6c7c7773ef7d8301d90 0.1s
=> [2/2] RUN true 0.4s
=> exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:d5cba3a06196c715b4cb29dbcfa972b95e1f0dc3ef699b23f10a022b9dfb6d51 0.0s
Still works to build without BuildKit, too.
$ docker build .
Sending build context to Docker daemon 7.168kB
Step 1/2 : FROM busybox
latest: Pulling from library/busybox
ee153a04d683: Already exists
Digest: sha256:9f1003c480699be56815db0f8146ad2e22efea85129b5b5983d0e0fb52d9ab70
Status: Downloaded newer image for busybox:latest
---> db8ee88ad75f
Step 2/2 : RUN true
---> Running in 5c338dc00426
Removing intermediate container 5c338dc00426
---> cbbb415368d3
Successfully built cbbb415368d3
$ docker version
Client: Docker Engine - Community
Version: 18.09.8
API version: 1.39
Go version: go1.10.8
Git commit: 0dd43dd87f
Built: Wed Jul 17 17:38:58 2019
OS/Arch: linux/amd64
Experimental: false
Server: Docker Engine - Community
Engine:
Version: 18.09.8
API version: 1.39 (minimum version 1.12)
Go version: go1.10.8
Git commit: 0dd43dd87f
Built: Wed Jul 17 17:48:49 2019
OS/Arch: linux/amd64
Experimental: false
$ cat Dockerfile
FROM busybox
RUN true
This should be fixed. Feel free to reopen if not.
sharifelgamal
on 16 Dec 2019
Related issues
vainikkaj
路
3Comments
kphatak
路
3Comments
cherish3620
路
3Comments
ahmetb
路
3Comments
olalonde
路
3Comments
Most helpful comment
Confirmed that building images with BuildKit works OK, with the new ISO:
Still works to build without BuildKit, too.