Origin: User can not view their metrics

[xavierbaude]

Description

Confere this Bug in the openshift enterprise release, an ordinary user can not view their metrics, admin can.
https://bugzilla.redhat.com/show_bug.cgi?id=1477223

bug reproduced in v3.6.0 origin openshift install

Version

Ansible 2.3.1
Centos 7.3
Origin 3.6.0 in containerized mod
docker image release : v3.6.0 from the docker hub.

Steps To Reproduce

In a running cluster setup, I install the metrics via the playbook :
playbooks/byo/openshift-cluster/openshift-metrics.yml

Expected Results

A user create a projet and deploy pod. User get the error :
"An error occurred getting metrics. Open Metrics URL | Don't Show Me Again"

Observed Results

the url is accessible, the certificate is trusted.
From "Monitoring/Pods/Metrics" same result.
As a cluster admin, everything working well

Additional Information

Redhat already fixed it in : https://bugzilla.redhat.com/show_bug.cgi?id=1477223 with the release 👍 metrics-hawkular-metrics:v3.6.173.0.3-2
metrics-cassandra:v3.6.173.0.3-1
metrics-heapster:v3.6.173.0.3-1
in their registry.

I don't see a PR linked to this problem yet.

Thanks !

[jcantrill]

@mwringe anything obvious that we should be looking for in our ansible deployment?

[mwringe]

its a change purely done to the docker image, there is no change required and its should have already been fixed in https://bugzilla.redhat.com/show_bug.cgi?id=1477223

[jsuchenia]

I think that this could be related: https://github.com/openshift/origin-metrics/issues/364

[kincl]

How do we get the update image? The Docker Hub latest will not work with v3.6 of origin and the v3.6.0 tag has not been updated since the release of v3.6.0?

https://hub.docker.com/r/openshift/origin-metrics-hawkular-metrics/tags/

[jcantrill]

@kincl Build the image directly and push it to dockerhub or your internal registry.

[kincl]

@jcantrill Okay I can do that. Will the official image be updated for 3.6.1 on the hub?

[jcantrill]

@smarterclayton @jupierce Do we have any process or plan to update origin dockerhub images ?

[smarterclayton]

We may cut a 3.6.1 with a few fixes soonish, in which case we can repush this one in particular.

[jsuchenia]

@jcantrill @smarterclayton Do you have a plan to update not only docker images but also CentOS RPMs? In a repo I see only RPMs generated during 3.6.0 release and there are no updates from 3.6 branch

[alikhajeh1]

Is there an ETA for the fix? Either in release-3.6 or master, we can cherry pick and build custom packages/images

[mwringe]

@jpkrohling can you take a look at this?

[jpkrohling]

Sure, looking!

[conatus]

It does appear that this and https://github.com/openshift/origin-metrics/issues/364#issuecomment-322281481 are the same issue so feels like that should be shut in favour of this ticket.

Great to see someone is looking at it!

[jpkrohling]

Status: this fix was made available on Hawkular Metrics 0.27.3:

https://github.com/hawkular/hawkular-metrics/commit/2105b0aa1635968adc2941327115666a1b26b42d
https://github.com/hawkular/hawkular-metrics/commits/0.27.3

The current Origin Metrics image uses 0.27.2. An image with this fix is available for testing at: jpkroehling/origin-metrics-hawkular-metrics:pr15851 .

A PR to bump the version was submitted: https://github.com/openshift/origin-metrics/pull/382 .

[mwringe]

Thanks @jpkrohling I really thought we had this fixed, but I guess not. The PR is set to be merged. Once it is we will need to push out a new image.

[AlbertoPeon]

We are also affected by this.

Could you update the origin-metrics:v3.6.0 docker image with a fix for this?

Thanks!

[conatus]

@AlbertoPeon

We changed the version of the metrics images to latest in the inventory file and everything works well. The version definitely needs tagging.

openshift_metrics_image_version=latest

[jcantrill]

@jupierce will we push new origin metrics images?

[jupierce]

@jcantrill My team presently only publishes OCP based images. @smarterclayton has historically done origin releases.

[smarterclayton]

This is origin-metrics-hawkular-metrics?

[smarterclayton]

Updated origin-metrics-hawkular-metrics:v3.6.0 to be the value of :latest

[jcantrill]

Closing based on https://github.com/openshift/origin/issues/15851#issuecomment-337179929 and https://github.com/openshift/origin/issues/15851#issuecomment-335430445

[xavierbaude]

A new v3.6.0 image has been pushed to the docker hub(since 9 days). The old is now with tag v3.6.0-old (...).
I confirm the bug is solved !