Kube-state-metrics: Rewrite the end-to-end tests to go

[lilic]

Currently, we have all our end-to-end tests in bash, this makes it painstaking to add new tests, with go we can easily use the client_golang library to do checks.

We can use the prometheus-operator test framework as an example to create the framework.

Below are some test cases I came up with, these are the already existing ones we have plus additional ones, should be good for the first iteration:

1. Wait until all pods are running
2. Loop and wait until kube-state-metrics is running by accessing the healthz endpoint
3. Access kube-state-metrics metrics endpoint, so 8080 to get the metrics data
4. Make sure metrics is not 0
5. Lint the metrics using prometheus client_golang library
6. Check that all the collector metrics are available, we can take the constants from options pkg DefaultCollectors
7. Deleting resources causes metric to be updated
8. Make sure kube-state-metrics own metrics and healtz are present, the go metrics etc.
9. Check the metrics error rate is not high. The kube_state_metrics_list_total and kube_state_metrics_watch_total

cc @tariq1890 As discussed sharing some ideas I wrote down a while ago.

[brancz]

We can use the prometheus-operator test framework as an example to create the framework.

I personally think what the prometheus operator does would be a bit of an overkill here. We just need to launch the kube-state-metrics pod and interact with it via a proxy, that won't need a framework like the prometheus operator does. We should just do it from scratch here I think.

Otherwise very happy to finally get rid of the bash tests!

[YaoZengzeng]

@LiliC @brancz

I'd like to help to fix this issue if needed :)

[lilic]

@YaoZengzeng Yes, this still needs addressing, I know @tariq1890 wanted to work on it, but didn't have the time, so maybe you can sync with him? :) Thanks!!

[tariq1890]

I haven't started this yet, but always happy to have contributors who are onboard to help :).

[lilic]

@YaoZengzeng Let me know if something is not clear in the issue, feel free to reach out on slack as well. Also feel free to open an early WIP PR. Thanks again!

[YaoZengzeng]

@LiliC I will, thanks! :)

[YaoZengzeng]

After my research and combined with what @brancz said, I think we can still use the script to install minikube (as well as kubectl proxy) and launch kube-state-metrics.

Then pass the url (generally http://localhost:8001/api/v1/namespaces/kube-system/services/kube-state-metrics:http-metrics/proxy/) of kube-state-metrics service to go test and use golang to write e2e test based on the url.

I think use script to do the preparation work is more appropriate.

WDYT? @LiliC @brancz @tariq1890

[tariq1890]

I think that's a reasonable approach.

Also, feel free to use the latest stable minikube version. You can also use this command to get the service url instead of kubectl proxy &

minikube service kube-state-metrics --url

[YaoZengzeng]

@tariq1890 Thanks for your advice :)

I've written a rough framework, PTAL. Also @LiliC @brancz

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[lilic]

/remove-lifecycle rotten

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[lilic]

/remove-lifecycle rotten

[olivierlemasle]

/remove-lifecycle stale

[lilic]

ha thanks :D copy paste!

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[olivierlemasle]

/remove-lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[olivierlemasle]

/remove-lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle rotten

[olivierlemasle]

/remove-lifecycle rotten

[lilic]

@olivierlemasle is there a PR I missed for this that needs reviews, let me know! If not, you can of course take it on if you want to! :)

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[olivierlemasle]

/remove-lifecycle stale