Argo: v2.9.3 Docker images report they are v2.10.0-rc1

[nilium]

What happened:

Pulling the argoproj/argocli:v2.9.3 and argoproj/workflow-controller:v2.9.3 Docker images appears to produce the v2.10.0-rc1 versions of the programs (unable to verify if this is just an incorrect string or a wholly incorrect image). The GitCommit and GitTag fields both suggest this isn't just an incorrect string in the 2.9.3 build.

Specifically:

argocli

$ podman run --rm argoproj/argocli:v2.9.3 version
argo: v2.10.0-rc1
  BuildDate: 2020-07-17T19:44:22Z
  GitCommit: 616c79df09c435fa7659bf7e5194529d948ee93b
  GitTreeState: clean
  GitTag: v2.10.0-rc1
  GoVersion: go1.13.4
  Compiler: gc
  Platform: linux/amd64
time="2020-07-18T18:00:23Z" level=fatal msg="invalid configuration: no configuration has been provided"

workflow-controller

$ podman run --rm argoproj/workflow-controller:v2.9.3 version
workflow-controller: v2.10.0-rc1
  BuildDate: 2020-07-17T19:50:17Z
  GitCommit: 616c79df09c435fa7659bf7e5194529d948ee93b
  GitTreeState: clean
  GitTag: v2.10.0-rc1
  GoVersion: go1.13.4
  Compiler: gc
  Platform: linux/amd64

What you expected to happen:

I expected argocli and the workflow-controller images to return version 2.9.3. I'd also expect the Argo commits and tags to point to 9407e19b3a1c61ad4043e382484fd0b6b15574f2 and v2.9.3, respectively.

How to reproduce it (as minimally and precisely as possible):

podman

$ podman run --rm argoproj/argocli:v2.9.3 version
$ podman run --rm argoproj/workflow-controller:v2.9.3 version

docker

$ docker run --rm argoproj/argocli:v2.9.3 version
$ docker run --rm argoproj/workflow-controller:v2.9.3 version

Anything else we need to know?:

This does not appear to apply to the binaries under the 2.9.3 release. The v2.9.2 images also correctly report their version, commit, and tag.

Environment:

Podman and Docker.

---

Message from the maintainers:

If you are impacted by this bug please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.

[alexec]

here is the build culprit:

https://hub.docker.com/layers/argoproj/argocli/v2.9.3/images/sha256-6c1d321ca17ef374c7a159af2281193e9b9e7d31da2fce7361c7d34d3c348302?context=repo

[alexec]

somehow building v2.10.0-rc1 pushed v2.9.3 images

[alexec]

re-publishing now

[alexec]

acollins8@intuitdepe1a95 argo (⎈ |k3s-default:argo) ((stable))]$ docker run argoproj/argocli:v2.9.3 version
Unable to find image 'argoproj/argocli:v2.9.3' locally
v2.9.3: Pulling from argoproj/argocli
385909e8e828: Pull complete 
f4ad0238afe1: Pull complete 
06461e61bd4f: Pull complete 
79097b56e3b3: Pull complete 
29c77089a847: Pull complete 
Digest: sha256:1edf63488b5d6440870e5671e36674524a17a596541c72308a6b7dfc1ecdc159
Status: Downloaded newer image for argoproj/argocli:v2.9.3
argo: stable
  BuildDate: 2020-07-18T18:31:03Z
  GitCommit: 9407e19b3a1c61ad4043e382484fd0b6b15574f2
  GitTreeState: clean
  GitTag: stable
  GoVersion: go1.13.4
  Compiler: gc
  Platform: linux/amd64
time="2020-07-18T18:49:22Z" level=fatal msg="invalid configuration: no configuration has been provided"
[acollins8@intuitdepe1a95 argo (⎈ |k3s-default:argo) ((stable))]$ docker run argoproj/argoexec:v2.9.3 version
Unable to find image 'argoproj/argoexec:v2.9.3' locally
v2.9.3: Pulling from argoproj/argoexec
54fec2fa59d0: Pull complete 
03692a9d138a: Pull complete 
baf5dcb8fbf2: Pull complete 
d11f80c2329a: Pull complete 
f5f1038ef4a5: Pull complete 
3059ce35e7cd: Pull complete 
5fbbc9994c18: Pull complete 
efb270be3fe8: Pull complete 
62f12c42869e: Pull complete 
Digest: sha256:07db68406fa34a08bf4a5243b3d92b13a293f0d3ae5b2f02e138173cad26628e
Status: Downloaded newer image for argoproj/argoexec:v2.9.3
argoexec: stable
  BuildDate: 2020-07-18T18:33:20Z
  GitCommit: 9407e19b3a1c61ad4043e382484fd0b6b15574f2
  GitTreeState: clean
  GitTag: stable
  GoVersion: go1.13.4
  Compiler: gc
  Platform: linux/amd64
[acollins8@intuitdepe1a95 argo (⎈ |k3s-default:argo) ((stable))]$ docker run argoproj/workflow-controller:v2.9.3 version
Unable to find image 'argoproj/workflow-controller:v2.9.3' locally
v2.9.3: Pulling from argoproj/workflow-controller
683f6aabbf72: Pull complete 
5f5ea9c2a33d: Pull complete 
Digest: sha256:7806046ddcb026849cffdef0ef0b266e1d03c420e07a65c4273281543e24ab5a
Status: Downloaded newer image for argoproj/workflow-controller:v2.9.3
workflow-controller: stable
  BuildDate: 2020-07-18T18:36:33Z
  GitCommit: 9407e19b3a1c61ad4043e382484fd0b6b15574f2
  GitTreeState: clean
  GitTag: stable
  GoVersion: go1.13.4
  Compiler: gc
  Platform: linux/amd64

[alexec]

Need to find out why this happened and prevent it happening again.

[nilium]

Thanks, @alexec. Noticed the version is now stable instead of 2.9.3, but I think that's probably fine since the commit matches the release now (so no concerns about ending up with the wrong code due to a node without image caches getting 2.10.0 accidentally).

[alexec]

@nilium can I please ask how you noticed?

[nilium]

@alexec The node with both the image caches and the Argo pods was taken out for maintenance (that is, I'm rebuilding part of my homelab). As a result, the pods came up on another node (this time without the old caches) and did a new pull for v2.9.3. After checking that Argo was still running, I looked at the Argo server and the version was reported as v2.10.0-rc1. So, went and double-checked that by pulling the images locally to verify.

[alexec]

 make status
# GIT_TAG=v2.10.0-rc1, GIT_BRANCH=release-2.10, GIT_TREE_STATE=clean, VERSION=v2.9.3, VERSION=v2.9.3, DEV_IMAGE=false, K3D=true

[alexec]

We now build (v2.10) images automatically on Github Actions. No need to publish them locally and make mistakes.

[alexec]

Available for testing in v2.11.0-rc1.