Kubebuilder: [feature request] Support running tests in Docker container natively

[gaocegege]

/kind feature

Now we generated some test cases and can run the test locally. While when we want to run the test in some Docker-based CI systems, for example, https://github.com/caicloud/cyclone or https://github.com/drone/drone, we need to install kubebuilder first then make test.

I'd appreciate if we could maintain a docker image for the unit test in kubebuilder community.

[mengqiy]

@gaocegege Looking at the test target in the generated Makefile, it seems the things you need in your image are go toolchains. Nothing requires kubebuilder to run the tests actually.
You can try to use golang docker image, it should work.

[gaocegege]

@mengqiy It does not work, actually. We need to install kubebuilder(precisely, apiserver and etcd in kubebuilder bin). Or the test cannot be run.

[mengqiy]

I see. You need apiserver, etcd and go toolchain in the image to run the tests.
We will need to maintain an image for each k8s minor versions.

[afirth]

I'm sure I'll find some edge cases when versions don't line up, but here's my hack to get something working. Based on the kustomize GCB image. I don't remember why I didn't build kustomize from source - possibly there was a good reason, possibly I just started there an then realized it didn't have go :D

cat kubebuilder-image/Dockerfile

# https://github.com/GoogleCloudPlatform/cloud-builders-community/tree/master/kustomize
FROM gcr.io/<REPLACEME>/kustomize:3.1.0 as builder
# provides kustomize binaries to copy

ARG KUBEBUILDER_VERSION=2.0.0-alpha.4

RUN echo ${KUBEBUILDER_VERSION}
RUN curl -sL# https://go.kubebuilder.io/dl/${KUBEBUILDER_VERSION}/linux/amd64 | tar -xz -C /builder/
RUN mv /builder/kubebuilder_${KUBEBUILDER_VERSION}_linux_amd64 /builder/kubebuilder

FROM gcr.io/cloud-builders/go
ARG CONTROLLER_GEN_VERSION=v0.2.0-beta.2
# copy kustomize
COPY --from=builder /builder/kustomize /builder/kustomize
ENV PATH=/builder/kustomize:$PATH
# copy kubebuilder
# test scripts require this to be in usr/local
COPY --from=builder /builder/kubebuilder /usr/local/kubebuilder
ENV PATH=/usr/local/kubebuilder/bin:$PATH

#install controller-gen
ENV GO111MODULE=on
RUN go get sigs.k8s.io/controller-tools/cmd/controller-gen@${CONTROLLER_GEN_VERSION}
ENV PATH=/go/bin:$PATH

ENTRYPOINT make

cat kubebuilder-image/cloudbuild.yaml

steps:
- name: gcr.io/kaniko-project/executor
  id: image-build
  args:
  - --destination=gcr.io/$PROJECT_ID/$_IMG_NAME:$SHORT_SHA
  - --destination=gcr.io/$PROJECT_ID/$_IMG_NAME:latest
  - --cache=true
  - --cache-ttl=2h
  - --build-arg
  - KUBEBUILDER_VERSION=${_KUBEBUILDER_VERSION}
  - --build-arg
  - CONTROLLER_GEN_VERSION=${_CONTROLLER_GEN_VERSION}
  - --destination=gcr.io/$PROJECT_ID/$_IMG_NAME:${_KUBEBUILDER_VERSION}

substitutions:
  _KUBEBUILDER_VERSION: 2.0.0-beta.0
  _CONTROLLER_GEN_VERSION: v0.2.0-beta.4
  _IMG_NAME: kubebuilder

This will build an push a kubebuilder image which can be used in other places like so:

cat mycontroller/deploy/cloudbuild.yaml

steps:

  # Build and Test the go packages and boilerplate
- name: gcr.io/$PROJECT_ID/kubebuilder:2.0.0-alpha.4
  id: go-build
  waitFor: ['-'] #parallel
  entrypoint: make
  args: ['test', 'all']

Hope that helps someone

[DirectXMan12]

we publish the test binaries separately: https://console.cloud.google.com/storage/browser/kubebuilder-tools.

We don't have a go.kubebuilder link for it yet, but PRs welcome on that one.

We also publish gcr.io/kubebuilder/thirdparty-{linux,darwin}:<version>, where <version> matches the versions used in the above kubebuilder-tools repo. They're used to build the gcs bucket, and contain the compiled artificats.

PRs welcome on docs for that as well.

[austince]

Thanks for the info! It's nice the images contain the artifacts but are there any images that install it as well? The images also doesn't contain the go runtime, which is needed for testing. If not, can I make a PR somewhere that will build such an image?

[DirectXMan12]

there aren't currently any images that install it, but if you want to play around with the cloudbuild setup to produce images for running the tests, we'd most likely accept that PR.

[RobEarl]

@austince I was able to run tests by using an initContainer of gcr.io/kubebuilder/thirdparty-{linux,darwin}:<version> to extract the tools to a shared volume. See https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-initialization/ for a similar example.

[kqdtran]

Hello, I'm also trying to get my integration tests running on jenkins and was wondering if anyone has more specific examples. I saw @RobEarl 's comment above but admittedly am still relatively new to kubebuilder.

[RobEarl]

@kqdtran I put an example of what I did here - https://github.com/RobEarl/website/tree/main/blog/kubebuilder-tests-docker-container

[RobEarl]

I should add, the example above is for a kubernetes agent. If you're using a docker agent, you would probably (I haven't tried it) need to:

• run an initial stage with the gcr.io/kubebuilder/thirdparty-{linux,darwin}:<version> image
• unpack the kubebuilder archive from /
• stash the binaries from that archive
• run test execution stage with a golang image
• unstash the binaries
• set KUBEBUILDER_ASSETS environment variable to reference the directory containing the binaries