Cloud-on-k8s: ARM e2e run sometimes ignores build constraints

[pebrc]

We use build constraints to only run a subset of e2e tests for Elasticsearch. But sometimes it seems the build constraints are ignored and tests are run that do no have the es build tag.

Example run:
https://devops-ci.elastic.co/job/cloud-on-k8s-e2e-tests-eks-arm/38/testReport/

[charith-elastic]

10:35:47  docker buildx build \
10:35:47    --progress=plain \
10:35:47    --file test/e2e/Dockerfile \
10:35:47    --build-arg E2E_JSON=true \
10:35:47    --build-arg GO_TAGS=es \
10:35:47    --platform linux/amd64,linux/arm64 \
10:35:47    --push \
10:35:47    -t docker.elastic.co/eck-ci/eck-e2e-tests:a1026cfa .

docker run -i -t --entrypoint=/bin/bash docker.elastic.co/eck-ci/eck-e2e-tests:a1026cfa 
101@bc853846abc8:~$ printenv
...
GO_TAGS=e2e
...
E2E_JSON=true

🤔 :rage3:

[pebrc]

Is it not the case that we build the same e2e test image for multiple jobs during the nightly run with the same tag e.g. last night:
docker.elastic.co/eck-ci/eck-e2e-tests:7c7bd9b8 but they are not the same content. So we have effectively created a race condition:

• ARM job pushes image
• non-ARM job pushes same tag
• ARM job starts pulling image
• ARM job runs "wrong" image

Similarly the race can go the other way round as well:

• non-ARM job pushes image
• ARM job pushed image
• non-ARM job starts pulling
• non-ARM job only runs a subset of the tests because it is using the image build for the ARM job (as observed by @thbkrkr some time ago I believe)

[charith-elastic]

I think you hit the nail right on the head there Peter. We build the operator image once and share it with all the jobs but not the E2E image. I wonder why that is. Before the build tags were introduced, the image would have been the same for every job and we could have saved quite a lot of time on test runs by building it just once.

[charith-elastic]

So, until the build process can be overhauled to build more targeted artifacts (#4316), a simple fix would be to append a unique value to the tag of the E2E image built by each job. WDYT?

[pebrc]

SGTM as temporary fix