Fluentd-kubernetes-daemonset: Unable to pull v1.12.4-debian-cloudwatch-1.4

[archoversight]

% docker pull fluent/fluentd-kubernetes-daemonset:v1.12.4-debian-cloudwatch-1.4
Error response from daemon: manifest for fluent/fluentd-kubernetes-daemonset:v1.12.4-debian-cloudwatch-1.4 not found: manifest unknown: manifest unknown

I am however able to pull:

% docker pull fluent/fluentd-kubernetes-daemonset:v1.12-debian-cloudwatch-1
v1.12-debian-cloudwatch-1: Pulling from fluent/fluentd-kubernetes-daemonset
69692152171a: Pull complete
0a1f774fea41: Pull complete
3b32b9ebaa42: Pull complete
697d218c4f9a: Pull complete
0a75b407d854: Pull complete
e87327430399: Pull complete
a0dbcaaf3fa1: Pull complete
cb72b6c0baf0: Pull complete
118fabf661ff: Pull complete
d531e9347b58: Pull complete
ce87f309f7da: Pull complete
eda57ff5b36e: Pull complete
0db9b6f8dc5f: Pull complete
7af081107c42: Pull complete
6e6768f0e8c3: Pull complete
8aa8c8a21bc1: Pull complete
cccb976ee21f: Pull complete
c99d72206bca: Pull complete
5bf8a932b03f: Pull complete
80295f750154: Pull complete
Digest: sha256:d5af0e354735c24b7f70f02e3c6a86e3393a26d862e7a2ded490a2fa4197a775
Status: Downloaded newer image for fluent/fluentd-kubernetes-daemonset:v1.12-debian-cloudwatch-1
docker.io/fluent/fluentd-kubernetes-daemonset:v1.12-debian-cloudwatch-1

and

% docker pull fluent/fluentd-kubernetes-daemonset:v1.12.4-debian-cloudwatch-amd64-1.4
v1.12.4-debian-cloudwatch-amd64-1.4: Pulling from fluent/fluentd-kubernetes-daemonset
Digest: sha256:04e4e841f8ed116bed901f2fb976d0b57287e197d485e90f8bfd65c72b75e730
Status: Downloaded newer image for fluent/fluentd-kubernetes-daemonset:v1.12.4-debian-cloudwatch-amd64-1.4
docker.io/fluent/fluentd-kubernetes-daemonset:v1.12.4-debian-cloudwatch-amd64-1.4

[kenhys]

Hmm,
This may be caused by multi-arch https://github.com/fluent/fluentd-kubernetes-daemonset/pull/567 , @cosmo0920 ?

[cosmo0920]

Definitely, yeah. #567 causes vanishing tag for v1.12.4-debian-cloudwatch-1.4.
But I'm not sure why lack of cloudwatch image v1.12.4-debian-cloudwatch-1.4 is caused.
Other images are tagged with v1.12.4-debian-xxx-1.x....

[cosmo0920]

I'd kicked cloudwatch images building but no luck....

[mouchar]

The same applies for -syslog images: the latest on docker hub is v1.11.2

[cosmo0920]

The same applies for -syslog images: the latest on docker hub is v1.11.2

This is NOT same issue.
We got reached 25 build rules limit on DockerHub. We'd removed some of building image rules including syslog image.

[mouchar]

I see, got it. Thanks for the explanation.

[kenhys]

Hmm, surely manifest is missing in this case.

% docker manifest inspect fluent/fluentd-kubernetes-daemonset:v1.12.4-debian-cloudwatch-1.4
no such manifest: docker.io/fluent/fluentd-kubernetes-daemonset:v1.12.4-debian-cloudwatch-1.4

% docker manifest inspect fluent/fluentd-kubernetes-daemonset:v1.12-debian-cloudwatch-1

{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
   "manifests": [
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 4487,
         "digest": "sha256:8117bbf66583d281fe21e6bc272c2796bc7ee74e3f254bfc75a5d8d54d720617",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 4909,
         "digest": "sha256:152c93e9af10facf2756413f37fb475088dc665c4297a78d47afcb445a5a5f78",
         "platform": {
            "architecture": "arm64",
            "os": "linux"
         }
      }
   ]
}

[kenhys]

v1.12.4-debian-cloudwatch-1.3 exists, but not for v1.12.4-debian-cloudwatch-1.4. Did manifest-tool failed?

[kenhys]

I got it. (maybe) according to post_push hook script, the same suffix version must be built for amd64 and arm64.
Thus, if 1.4 is pushed for amd64, it must be pushed for arm64, too (but actually 1.1 is pushed)

v1.12.4-debian-cloudwatch-amd64-1.4: digest: sha256:8117bbf66583d281fe21e6bc272c2796bc7ee74e3f254bfc75a5d8d54d720617 size: 4487
time="2021-06-20T13:59:57Z" level=fatal msg="Inspect of image \"index.docker.io/fluent/fluentd-kubernetes-daemonset:v1.12.4-debian-cloudwatch-arm64-1.4\" failed with error: manifest unknown: manifest unknown"
The push refers to repository [docker.io/fluent/fluentd-kubernetes-daemonset]

v1.12.4-debian-cloudwatch-arm64-1.1: digest: sha256:152c93e9af10facf2756413f37fb475088dc665c4297a78d47afcb445a5a5f78 size: 4909
time="2021-06-20T14:19:58Z" level=fatal msg="Inspect of image \"index.docker.io/fluent/fluentd-kubernetes-daemonset:v1.12.4-debian-cloudwatch-amd64-1.1\" failed with error: manifest unknown: manifest unknown"
The push refers to repository [docker.io/fluent/fluentd-kubernetes-daemonset]

[kenhys]

It is better to check whether --platforms linux/amd64,linux/arm64 is truly available for the specified tag

[cosmo0920]

Ah, I got it.
We have to use the same image version for cloudwatch images (amd64/arm64).
Currently, cloudwatch images use v1.12.4-debian-cloudwatch-amd64-1.4 and v1.12.4-debian-cloudwatch-arm64-1.1.
There is a glitch of image versions.

[kenhys]

@cosmo0920 will you fix it?

[cosmo0920]

Yep, I'm takling on this.

[cosmo0920]

Done! https://hub.docker.com/layers/fluent/fluentd-kubernetes-daemonset/v1.12.4-debian-cloudwatch-1.4/images/sha256-8117bbf66583d281fe21e6bc272c2796bc7ee74e3f254bfc75a5d8d54d720617?context=repo

[cosmo0920]

$ docker manifest inspect fluent/fluentd-kubernetes-daemonset:v1.12.4-debian-cloudwatch-1.4
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
   "manifests": [
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 4487,
         "digest": "sha256:8117bbf66583d281fe21e6bc272c2796bc7ee74e3f254bfc75a5d8d54d720617",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 4909,
         "digest": "sha256:f2325329e389f7c39c7b15fb003bb66dc50d04464ea04b9613e517118da9bf24",
         "platform": {
            "architecture": "arm64",
            "os": "linux"
         }
      }
   ]
}

[archoversight]

\o/

Thank you!!!