Kube-state-metrics: v1.9.5 image from GCR does not work

[riconnon]

Is this a BUG REPORT or FEATURE REQUEST?:

Uncomment only one, leave it on its own line:

/kind bug
/kind feature

What happened:
I pulled the latest image from both Quay and GCR

What you expected to happen:
I expected them to be the same

How to reproduce it (as minimally and precisely as possible):

$ docker image pull quay.io/coreos/kube-state-metrics:v1.9.5
$ docker image pull k8s.gcr.io/kube-state-metrics:v1.9.5

These two images ought to be the same image but I see two different images:

$ docker image ls
k8s.gcr.io/kube-state-metrics                                                                                                 v1.9.5                                             303a82af2ebf        12 days ago         32.8MB
quay.io/coreos/kube-state-metrics                                                                                             v1.9.5                                             43cd3e9ae81e        2 weeks ago         32.8MB

The image from k8s.gcr.io will not execute, but that from quay will:

$ docker run -it --rm quay.io/coreos/kube-state-metrics:v1.9.5
I0305 18:00:59.869735       1 main.go:86] Using default collectors
I0305 18:00:59.869887       1 main.go:98] Using all namespace
I0305 18:00:59.869898       1 main.go:139] metric white-blacklisting: blacklisting the following items: 
W0305 18:00:59.869915       1 client_config.go:543] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
W0305 18:00:59.869924       1 client_config.go:548] error creating inClusterConfig, falling back to default config: unable to load in-cluster configuration, KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT must be defined
F0305 18:00:59.870113       1 main.go:147] Failed to create client: invalid configuration: no configuration has been provided
$ docker run -it --rm k8s.gcr.io/kube-state-metrics:v1.9.5    
standard_init_linux.go:190: exec user process caused "permission denied"

Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version): v1.14.8
• Kube-state-metrics image version: v1.9.5

[serathius]

Ups, looks like my fault.
/assign

[serathius]

Hmm I rebuild the image locally. I run

git clone [email protected]:kubernetes/kube-state-metrics.git
cd kube-state-metrics
git checkout v1.9.5
make container
docker run -it --rm quay.io/coreos/kube-state-metrics-amd64:v1.9.5

I tested it on my two machines, image works on one, but fails on another. Looks like even when building on docker there is something machine dependent.

There are two problems:

• Making build process more reproducible
• Fixing k8s.gcr.io/kube-state-metrics:v1.9.5 image

I think https://github.com/kubernetes/kube-state-metrics/issues/1089 should help with build process. It would be also good to use multi-stage build process, but not nesesery.

I will contact sig-release about fixing the image.

[serathius]

/cc @lilic for visibility

[serathius]

Fix process based on discussion with @bartsmykla:

• [x] Ensure that gcr.io/k8s-infra-kube-state-metrics exists (part of https://github.com/kubernetes/kube-state-metrics/issues/1089)
• [ ] Push working image to gcr.io/k8s-staging-kube-state-metrics
• [ ] Promote staging image by adding it to k8s-staging-kube-state-metrics.yaml (example)

Last two steps can be only done by kube-state-metrics OWNER
/assign @lilic
Can you take care of it?

[bartsmykla]

Looks good to me :-)

[lilic]

Making build process more reproducible

Neither I nor Frederic had no problem building and pushing the images to quay so far, and I use a mac and he uses fedora, so not sure why it would be different for gcr? https://quay.io/repository/coreos/kube-state-metrics?tag=v1.9.5&tab=tags

[serathius]

Maybe for you two currently it doesn't. This doesn't mean that in future it will not (e.g. switching devices). On my debian system image doesn't build correctly. Overall we should stop building images locally.

sig-release encourages migration to automated pipeline instead of building on personal computers.

[serathius]

Overall, I can no longer fix this problem. Only kube-state-metrics OWNERS can.
@lilic Could you handle pushing and releasing the image?

Let me know if you need any help

[lilic]

Yep, got it, thanks! Will add the automation as soon as I have some time.

For now @riconnon since the quay image works for you either use that one or build your own image and push to your own registry, until the building and pushing images is automated to gcr. Does that solve the problem temporary?

Thanks for opening the issue, will let this stay open just so others who have problems see the discussion.

[kaivean]

I'm getting the same issue. Is it fixed ? @lilic

[sepich]

+1 for fixing gcr image, as quay is mostly unavailable recently: https://status.quay.io/

[lilic]

We have now gcr image with the latest build, so this can be closed. If you find any problems with it feel free to open a new issue, thanks!