Calico: Release tarballs include extra images

[tuminoid]

Calico-node docker image tarballs include random calico-node images, and not just release image.

Expected Behavior

Releases contain only right builds and download size is minimal and content is correct.
Fresh off the CI machinery, hopefully not hand built.

Current Behavior

Release tarballs contain random images in calico-node.tar.

2.6.1 release includes latest + 2.6.0-rc2
2.6.2 release includes 2.6.1, 2.6.1-12, 2.6.1-3, latest and 2.6.0-rc2
2.6.3 release includes just latest

Possible Solution

Use CI. Verify releases. Automate release process. Build from scratch.

Steps to Reproduce (for bugs)

1. Download https://github.com/projectcalico/calico/releases/download/v2.6.1/release-v2.6.1.tgz
2. Download https://github.com/projectcalico/calico/releases/download/v2.6.2/release-v2.6.2.tgz
3. Download https://github.com/projectcalico/calico/releases/download/v2.6.3/release-v2.6.3.tgz
4. Unpack and compare.

[bcreane]

Hi @tuminoid , thanks very much for raising this issue. We should be able to manually clean up the two broken releases (2.6.1, 2.6.2) that you noticed.

In terms of a fix to prevent this, I'm not sure how it happened since the calico_node/Makefile release target hits the clean target first which does in fact remove the release directory.

@caseydavenport - any thoughts how this might have happened in your work flow?

[bcreane]

Hmm, looks like v3.0.0-alpha1 is also wrong (@caseydavenport) and then v2.6.0-rc2 which was my release. So whatever is going wrong is happening for me as well.

[bcreane]

Seems like there must be bits of an old release in filesystem/ which gets built into calico/node:latest and then subsequently saved to the tarball.

However once again, the clean target is clearing out the volatile portion of filesystem/.

[bcreane]

Filling in some details from @tuminoid's report, docker image load --input calico-node.tar for v2.6.2 creates the following tagged images:

calico/node         latest               6763a667e3ba        7 weeks ago         282MB
calico/node         v2.6.1-12-gec45c0d   6e564ceef4aa        2 months ago        282MB
calico/node         v2.6.1-3-g9694a11    bba1aae27b12        2 months ago        282MB
calico/node         v2.6.1               bf7fdc7a57b2        2 months ago        282MB
calico/node         v2.6.0-rc2           adf2f8e09178        2 months ago        282MB

@caseydavenport - just wanted to confirm that the latest tag is the correct tagged image (seems like it should be the release).

[caseydavenport]

eek, this is bad. I probably won't have time to dig into this until after kubecon. Not sure what's going on here, but you should be able to compare the hash against the one on GitHub.

Or, run docker run calico/node:latest versions to see the versions packaged in that container.

[tuminoid]

Issue might be that $(NODE_CONTAINER_NAME) is set to calico/node, and when docker save -o $@ $(NODE_CONTAINER_NAME) is issued (note: without any tag), Docker saves all of the calico/node images into tarball.

$(RELEASE_DIR_IMAGES)/calico-node.tar:
    mkdir -p $(RELEASE_DIR_IMAGES)
        docker save --output $@ $(NODE_CONTAINER_NAME)

Also the tag is created after the release-archive is done. IMHO, it is bad to ship latest tag image in release tarball, instead of the versioned one.

    # Create the release archive
    $(MAKE) release-archive

    # Retag images with corect version and quay
       docker tag $(NODE_CONTAINER_NAME) $(NODE_CONTAINER_NAME):$(CALICO_VER)

Looking at the release makefile, it appears the calico/node image is built just before it is packaged, so content should be correct (if the packager is having the right revision checked out...). Thus fix for this issue would be to provide tag to docker save in calico-node.tar target, or clean up calico/node images from Docker prior making release.

[bcreane]

Hi @tuminoid - thanks for the concise analysis. I've put up a PR which should help both the issues - latest tag rather than the release, as well as multiple tags in a release tarball.

[caseydavenport]

I believe this has been fixed, so new releases will include the correct images.