Flux: Provide arm/arm64 images

[MansM]

As a Flux end-user
I want to use flux on a raspberry
in order to flux at home

I know travis-ci doesnt support it, but maybe weave can invest in some raspberries and use them with a gitlab.com ci runner:
https://about.gitlab.com/2018/03/22/gitlab-10-6-released/#gitlab-cicd-for-external-repos

[stefanprodan]

Flux bundles git, kubectl and helm binaries, having an ARM image implies more than just run docker build on an ARM machine.

[MansM]

Last time I build I just replaced amd64 with arm in the makefile and it did work...

[stefanprodan]

I wander how is that possible since the kubectl and helm checksums are different https://github.com/weaveworks/flux/blob/master/Makefile#L83

[MansM]

Didnt say I build it recently ;-)

[2opremio]

Related: https://github.com/weaveworks/flux/issues/1690

@MansM I doubt maintainers have the bandwidth to add support for other architectures. PRs are welcome though! (ideally with a commitment to maintain support for arm64)

[stefanprodan]

"weave can invest in some raspberries and use them with a gitlab.com ci runner"

I don't think this will happen any time soon.

[brezerk]

You don't really need to have an arm or arm64 board to build the images. The ARM images can be build using binfmt_misc.

Here you are some details: https://github.com/prometheus/prometheus/pull/5031

If you need any help let me know.

[brezerk]

one step closer: https://github.com/six8/pyinstaller-alpine/issues/5 :)
i'll work on kubeyaml now.

btw, I had a brief look into CircleCI configuration files for flux and do have some dumb questions:

I see you are using machine for build: is there any particular reason for this?

  build:
    working_directory: ~/flux
machine: true

also: you are puling Go instead of using circleci/golang image. probably this is b/c you are using machine... Any objections on moving build to circleci/golang docker instead?

[derkinderfietsen]

• 1

[onedr0p]

I am looking to run flux in my arm k3s cluster, has anyone successfully built a Docker image and deployed it? I will happily take Dockerfiles and build myself :) I'm am trying to do it from scratch but the build process isn't straight forward. I see the binary for fluxctl is available on the releases page but does fluxctl = fluxd?

Nevermind, I was able to get a image built reading the docs and looking at the docker directory

#
# Compile
#

FROM arm32v7/golang:1.12-alpine as compile
ENV GO111MODULE=on \
    CGO_ENABLED=0 \
    GOOS=linux \
    GOARCH=arm \
    GOARM=7 \
    FLUX_VERSION=1.13.3 \
    KUSTOMIZE_VERSION=v3.1.0

# build fluxd, fluxctl, helm-operator
WORKDIR /go/src/github.com/fluxcd/flux
RUN apk add --no-cache \
      openssh-client ca-certificates tini 'git>=2.12.0' 'gnutls>=3.6.7' gnupg gawk socat build-base gcc wget git bash curl \
    && update-ca-certificates
RUN apk add --no-cache -X http://dl-cdn.alpinelinux.org/alpine/edge/testing git-secret
RUN git clone https://github.com/fluxcd/flux.git . \
    && git checkout ${FLUX_VERSION}
RUN go build -o fluxd -installsuffix cgo ./cmd/fluxd
RUN go build -o fluxctl -installsuffix cgo ./cmd/fluxctl
RUN go build -o helm-operator -installsuffix cgo ./cmd/helm-operator

# build kustomize
WORKDIR /go/src/github.com/kubernetes-sigs/kustomize
RUN git clone https://github.com/kubernetes-sigs/kustomize.git . \
    && git checkout ${KUSTOMIZE_VERSION}
RUN go build -o kustomize -installsuffix cgo ./cmd/kustomize

#
# Build
#

FROM arm32v7/alpine:3.10
ENV KUBECTL_VERSION=v1.11.3
WORKDIR /home/flux

RUN apk add --no-cache curl openssh-client ca-certificates tini 'git>=2.12.0' 'gnutls>=3.6.7' gnupg gawk socat \
    && apk add --no-cache -X http://dl-cdn.alpinelinux.org/alpine/edge/testing git-secret

RUN curl -L -o /usr/bin/kubectl https://storage.googleapis.com/kubernetes-release/release/${KUBECTL_VERSION}/bin/linux/arm/kubectl && \
  chmod +x /usr/bin/kubectl && \
  kubectl version --client

RUN [ ! -e /etc/nsswitch.conf ] && echo 'hosts: files dns' > /etc/nsswitch.conf
COPY ./kubeconfig /root/.kube/config

COPY ./known_hosts.sh /home/flux/known_hosts.sh
RUN sh /home/flux/known_hosts.sh /etc/ssh/ssh_known_hosts && \
    rm /home/flux/known_hosts.sh

# Add default SSH config, which points at the private key we'll mount
COPY ./ssh_config /etc/ssh/ssh_config

COPY --from=compile /go/src/github.com/fluxcd/flux/fluxd /usr/bin/fluxd
COPY --from=compile /go/src/github.com/fluxcd/flux/fluxctl /usr/bin/fluxctl
COPY --from=compile /go/src/github.com/fluxcd/flux/helm-operator /usr/bin/helm-operator
COPY --from=compile /go/src/github.com/kubernetes-sigs/kustomize /usr/bin/kustomize

COPY --from=quay.io/squaremo/kubeyaml:0.6.1 /usr/lib/kubeyaml /usr/lib/kubeyaml/
ENV PATH=/bin:/usr/bin:/usr/local/bin:/usr/lib/kubeyaml

ENTRYPOINT ["/sbin/tini", "--", "fluxd"]

[jonohill]

I've set up a Gitlab pipeline to build ARM images for my own use. A bit of a rabbit hole - I also had to mirror and build kubeyaml and kustomize. Feel free to use if you wish - but best to treat as experimental. https://gitlab.com/jonohill/flux/container_registry

[zlangbert]

I also wanted to run flux and the helm controller on a Raspberry Pi cluster, and took a quick stab at this. Using docker / buildx with slightly modified Docerfiles this tuned out to be relatively easy. You can see what I did here:

Flux: https://github.com/zlangbert/flux/commit/250533124ee946e1577a20d063edaafeb5a7464f
Helm Operator: https://github.com/zlangbert/helm-operator/commit/29e73d7d656699325bce8c593411f90b305e8c9b

If anyone is interested in using these multiarch images you can get them here:
https://hub.docker.com/r/zlangbert/flux/tags
https://hub.docker.com/r/zlangbert/flux-helm-operator/tags

Note these are built using the latest commit in master of flux and the latest of the helm-v3-dev branch of the helm operator.

[onedr0p]

Thanks for that multiarch image. What's the hold up of official support from weave? This seems rudimentary to add.

[ikaruswill]

I am running my own armv7 build of flux on a 7-node Raspberry Pi 3 k3s cluster.

I've built my own image based on the work done by @onedr0p, following the official docker image as closely as possible.

https://hub.docker.com/r/ikaruswill/flux/tags

I'm expecting to keep this image up to date with drone-ci pipeline for as long as fluxcd doesn't have one for arm. Feel free to use them.

Caveats:

• I have not built sops so sops will not work. Would be happy to integrate anyone's arm build of sops.

If anyone is keen on using, I also have a ci pipeline building kubeyaml regularly.

https://hub.docker.com/r/ikaruswill/kubeyaml/tags

[onedr0p]

@ikaruswill I have a repo and build multiarch container image for flux and helm-operator, and some others. I'm trying to stay up too date with releases since they aren't automated on my end.

https://gitlab.com/onedr0p/flux
https://hub.docker.com/r/onedr0p/flux/tags

https://gitlab.com/onedr0p/helm-operator
https://hub.docker.com/r/onedr0p/helm-operator/tags

[ikaruswill]

@onedr0p Awesome work, I see that you actually did build sops and helm-operator. I'll integrate them in my setup and image. Thanks for that.

[onedr0p]

The flux build is a bit crazy and I need to build sops, kustomize and flux concurrently or else the gitlab runners timeout because the build takes over an hour. Once all these projects start releasing binaries per arch it should make the build faster and more stable.

Here's what I mean: https://gitlab.com/onedr0p/flux/pipelines/113975458

[xUnholy]

Thanks for the work @onedr0p!

Would love for you to work on a project of mine, feel free to reach out - I'll shoot you an invite if you're interested.

[onedr0p]

@xUnholy a few of us already started an GitHub org and have a discord community growing.

https://github.com/k8s-at-home/

I hope we can combine efforts here.

[onedr0p]

My repos have been deleted, we've combined efforts here to provide ARM support:

https://github.com/raspbernetes/multi-arch-images

We have flux there however a minor blocker for automating it is the projects this depends upon not supporting ARM binaries, so we're having a harder time making sure the versions of the dependencies are correct. See https://github.com/raspbernetes/multi-arch-images/issues/45

• https://github.com/mozilla/sops/issues/595
• https://github.com/kubernetes-sigs/kustomize/issues/2235