Metrics-server: Push manifest for multi-architecture images

[carlosedp]

I see that there are already multiple architecture images pushed to GCR (amd64, arm, arm64, etc).

One last step that has been already done by core Kubernetes images is pushing a manifest pointing to all these images so multi-arch clusters does not require changing the deployment yaml.

I believe having k8s.gcr.io/metrics-server:version pointing to the arch images is the best solution. I will create a PR for this in a similar way that is done on core K8s (https://cs.k8s.io/?q=docker%20manifest%20(create%7Cpush%7Cannotate)&i=nope&files=&repos=).

[carlosedp]

I've created https://github.com/kubernetes-incubator/metrics-server/pull/182 to address this.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[carlosedp]

Any news on this?

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[neuralsandwich]

/remove-lifecycle rotten

[moikot]

I would love to have this issue solved! Would it be possible to prioritize?

[serathius]

Same response as for https://github.com/kubernetes-incubator/metrics-server/pull/313
current pipeline of releasing images to k8s.gcr.io doesn't support manifest list.

[moikot]

Maybe I'm missing something but why the issue was closed just by restating the problem?

[serathius]

Sorry for not giving full explanation here. I created issue that explains what needs to be done to use multi-architecture image manifests.
Please look into https://github.com/kubernetes-sigs/metrics-server/issues/365

[egernst]

@serathius - While there may have been enabling done to allow manifests in the future, AFAIU there still are not multi-arch manifests published for metrics-server, which is problematic. I'd be delighted if I'm wrong :) If not, though, perhaps this or another issue should still be open to track?

Usually I end up just creating my own based on the individual arch images to work around, but in this case I see a further issue: the published arm64 image is still marked as x86:

$ docker image inspect k8s.gcr.io/metrics-server-arm64:v0.3.6 | grep Arch
        "Architecture": "amd64", 

This image works fine on aarch64/arm64, but AFAIU this won't work in a multi-arch manifest. Can you help identify the current status / next steps re: official multi-arch support from metrics-server? i.e.: AFAICT this is problematic in 0.3.6, but not clear on when 0.3.7 will happen nor if the manifest will be included.

Any help needed?

[serathius]

Sure, multi arch images should be supported by 0.3.7 release which was cut and is expected to be released next week. More details https://github.com/kubernetes-sigs/metrics-server/issues/491
Image should be available under gcr.io/k8s-staging-metrics-server/metrics-server:v0.3.7 (don't know if k8s staging repos are public thou)

[serathius]

Hmm looks like images still are not marked correctly?

docker inspect gcr.io/k8s-staging-metrics-server/metrics-server-arm64:v0.3.7 | grep Arch
        "Architecture": "amd64",

[dims]

[dims@dims-a02 13:47] ~/go/src/k8s.io ⟩ manifest-tool inspect gcr.io/k8s-staging-metrics-server/metrics-server:v0.3.7
Name:   gcr.io/k8s-staging-metrics-server/metrics-server:v0.3.7 (Type: application/vnd.docker.distribution.manifest.list.v2+json)
Digest: sha256:eec279de92328954ec69e9c2ef920861de28d31bb14b5290b53b5ef3dfa96502
 * Contains 5 manifest references:
1    Mfst Type: application/vnd.docker.distribution.manifest.v2+json
1       Digest: sha256:c0efe772bb9e5c289db6cc4bc2002c268507d0226f2a3815f7213e00261c38e9
1  Mfst Length: 739
1     Platform:
1           -      OS: linux
1           - OS Vers:
1           - OS Feat: []
1           -    Arch: amd64
1           - Variant:
1           - Feature:
1     # Layers: 2
         layer 1: digest = sha256:9ff2acc3204b4093126adab3fed72de8f7bbfe332255b199c30b8b185fcf6923
         layer 2: digest = sha256:9d14b55ff9a008391205162234bdef10ac34188e08954255d8cc5e04316e8e8a

2    Mfst Type: application/vnd.docker.distribution.manifest.v2+json
2       Digest: sha256:53f85e3b38d46a7b4fa0e10cfaa22cd1c2bb307d5171fbe3f1509875bf149eae
2  Mfst Length: 739
2     Platform:
2           -      OS: linux
2           - OS Vers:
2           - OS Feat: []
2           -    Arch: arm
2           - Variant:
2           - Feature:
2     # Layers: 2
         layer 1: digest = sha256:9ff2acc3204b4093126adab3fed72de8f7bbfe332255b199c30b8b185fcf6923
         layer 2: digest = sha256:cdd7bce9452e5cabd06aa9a00a6f7f51e060a61b82571ee7a62a3dbb0e42c63e

3    Mfst Type: application/vnd.docker.distribution.manifest.v2+json
3       Digest: sha256:328be26e74c3b516a4e7e0bfd2328a19e47d2fbea1fa97268ac183ad1c326fe1
3  Mfst Length: 739
3     Platform:
3           -      OS: linux
3           - OS Vers:
3           - OS Feat: []
3           -    Arch: arm64
3           - Variant:
3           - Feature:
3     # Layers: 2
         layer 1: digest = sha256:9ff2acc3204b4093126adab3fed72de8f7bbfe332255b199c30b8b185fcf6923
         layer 2: digest = sha256:2a939617a8a349dcc223df36a0334d31e9600221ec66d2d90da24ed1d59da919

4    Mfst Type: application/vnd.docker.distribution.manifest.v2+json
4       Digest: sha256:43e709f8b735321899f81bb4cb186aae4eb4a2b09c010da946c61c2383c0cb78
4  Mfst Length: 739
4     Platform:
4           -      OS: linux
4           - OS Vers:
4           - OS Feat: []
4           -    Arch: ppc64le
4           - Variant:
4           - Feature:
4     # Layers: 2
         layer 1: digest = sha256:9ff2acc3204b4093126adab3fed72de8f7bbfe332255b199c30b8b185fcf6923
         layer 2: digest = sha256:975734d3bcfd9e00f6259460d42356eb2c11b1be1866b978b8b1f0d1f983a7d7

5    Mfst Type: application/vnd.docker.distribution.manifest.v2+json
5       Digest: sha256:af228cd639d5320e8230f1c3eadb61e01b8b63d83a69d61ec73ad77aec24630e
5  Mfst Length: 739
5     Platform:
5           -      OS: linux
5           - OS Vers:
5           - OS Feat: []
5           -    Arch: s390x
5           - Variant:
5           - Feature:
5     # Layers: 2
         layer 1: digest = sha256:9ff2acc3204b4093126adab3fed72de8f7bbfe332255b199c30b8b185fcf6923
         layer 2: digest = sha256:8822a52c38832f3e78fc41a823b368b4d84516ddea2f23674f40f1fd88894359

[serathius]

@dims for rescue !!
Why then architecture differs when inspecting image directly? Possibly because we don't annotate images directly but only assign architectures in manifest

@egernst If that's a problem for you feel free to file a bug about 0.3.7 image. Contributions are also welcomed

[egernst]

The v0.3.7 staging looks good @serathius; I'll use this for the short term, and look forward to the release. Thanks.