Buildx: support multi-arch build with driver kubenetes.

[morlay]

docker buildx create --use --name buildx --node buildx-arm64 --platform=linux/arm64 --driver=kubernetes --driver-opt="image=moby/buildkit:master,replicas=2,namespace=gitlab"

docker buildx create --append --name buildx --node buildx-amd64 --platform=linux/amd64 --driver=kubernetes --driver-opt="image=moby/buildkit:master,replicas=2,namespace=gitlab"

--append now works when --driver=kubernetes.

When just with one cli in my multi-arch k8s cluster (with affinities could make pod in each arch host).

docker buildx create --use --name buildx --node buildx--platform=linux/amd64, linux/arm64 --driver=kubernetes --driver-opt="image=moby/buildkit:master,replicas=2,namespace=gitlab"

platform of node is wrong, and build jobs are dispatched to wrong pod.

Name:   buildx
Driver: kubernetes

Nodes:
Name:      buildx-647b4d4b6f-5xf2m
Endpoint:  
Status:    running
Platforms: linux/arm64

Name:      buildx-647b4d4b6f-gt7xk
Endpoint:  
Status:    running
Platforms: linux/arm64

[tonistiigi]

@AkihiroSuda

[AkihiroSuda]

A single kube cluster with mixed arch isn't currently supported. Help wanted.

[morlay]

@AkihiroSuda
is that possible to make --append works for kubeneters driver, like 'docker-container' did?

[rolandjitsu]

@AkihiroSuda does this mean that if we want to use buildx with the kubernetes driver to build for different architectures than the pod's architecture, it's not possible?

[AkihiroSuda]

binfmt may work probably

[rolandjitsu]

binfmt may work probably

Hmm ... and how would I run that on the pod?

Say I've created this builder:

docker buildx create --name my-kube --driver kubernetes --use

Now I bootstrap it:

docker buildx inspect --bootstrap

And now I try to bake something:

docker buildx bake -f ./docker-bake.hcl

But this fails because the dockerfile is using RUN and FROM arm32v7/debian:stretch AS builder which is not supported on the host.

[AkihiroSuda]

docker run --privileged --rm tonistiigi/binfmt --install all on each of the nodes. (Can be Kubernetes Daemonset)

[rolandjitsu]

docker run --privileged --rm tonistiigi/binfmt --install all on each of the nodes. (Can be Kubernetes Daemonset)

Thanks @AkihiroSuda. Using a DaemonSet seems to do the trick!

[morlay]

@AkihiroSuda

Daemonset could not only run once

it that possible to provider --wait to keep alive ?
and auto uninstall when pod killed?

then could use the DaemonSet

kind: DaemonSet
apiVersion: apps/v1
metadata:
  name: binfmt
spec:
  selector:
    matchLabels:
      app: binfmt
  template:
    metadata:
      labels:
        app: binfmt
    spec:
      containers:
        - name: binfmt
          image: tonistiigi/binfmt
          args:
            - --wait
            - --install
            - all
          securityContext:
            privileged: true
          imagePullPolicy: Always

[rolandjitsu]

@AkihiroSuda

Daemonset could not only run once

it that possible to provider --wait to keep alive ?
and auto uninstall when pod killed?

then could use the DaemonSet

kind: DaemonSet
apiVersion: apps/v1
metadata:
  name: binfmt
spec:
  selector:
    matchLabels:
      app: binfmt
  template:
    metadata:
      labels:
        app: binfmt
    spec:
      containers:
        - name: binfmt
          image: tonistiigi/binfmt
          args:
            - --wait
            - --install
            - all
          securityContext:
            privileged: true
          imagePullPolicy: Always

I use this:

# Run binfmt setup on any new node
# https://kubernetes.io/docs/concepts/workloads/controllers/daemonset
# https://github.com/docker/buildx/issues/342#issuecomment-680715762
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: binfmt
  # namespace: kube-system
  labels:
    app: binfmt-setup
spec:
  selector:
    matchLabels:
      name: binfmt
  # https://kubernetes.io/docs/concepts/workloads/pods/#pod-templates
  template:
    metadata:
      labels:
        name: binfmt
    spec:
      tolerations:
        # Have the daemonset runnable on master nodes
        # NOTE: Remove it if your masters can't run pods
        - key: node-role.kubernetes.io/master
          effect: NoSchedule
      initContainers:
        - name: binfmt
          image: tonistiigi/binfmt
          # command: []
          args: ["--install", "all"]
          # Run the container with the privileged flag
          # https://kubernetes.io/docs/tasks/configure-pod-container/security-context/
          # https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#securitycontext-v1-core
          securityContext:
            privileged: true
      containers:
        - name: pause
          image: gcr.io/google_containers/pause
          resources:
            limits:
              cpu: 50m
              memory: 50Mi
            requests:
              cpu: 50m
              memory: 50Mi

[morlay]

@AkihiroSuda

still hope could support --append,

with qemu, the compiling times slow most x4.

some job like https://github.com/querycap/istio-envoy-arm64 (now in arm host need 1 hour+)
will be a disaster 😂

[tonistiigi]

with qemu, the compiling times slow most x4.

If your build contains a compilation step and using native nodes adds complexity qemu is rarely a recommended solution and you should look into cross-compile that is quite easy to define in Dockerfiles so --platform flag works as expected. Look this repo, buildkit repo or https://github.com/tonistiigi/xx https://github.com/tonistiigi/binfmt/blob/master/Dockerfile for examples.

[morlay]

@tonistiigi cool example to setep a cross compiling environment. （one to all）

however，the envoy-wasm compiling deps lots of tools，Java，llvm, golang，gn，bazel.
i have to setup the compiling environment for each arch, and compile
in arch-matched host.

multi stage for difference arch，i already learn from commit by you in some issue.😅
https://github.com/querycap/istio/blob/master/istios/proxyv2.Dockerfile