Tf-operator: Prometheus support in TF Job

[johnugeorge]

Reference: https://github.com/kubeflow/common/issues/22

Since common repo is not ready yet, Prometheus support will be added to TF/Pytorch and code will be later moved to common repo.

/assign @johnugeorge
/cc @richardsliu

[issue-label-bot[bot]]

Issue-Label Bot is automatically applying the label improvement/enhancement to this issue, with a confidence of 0.74. Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback!

Links: app homepage, dashboard and code for this bot.

[jlewi]

Thanks @johnugeorge. Is anyone planning on picking this up in the next 2 weeks?

[johnugeorge]

Yes. This will be in 0.6 release. @krishnadurai is working on it

[ScorpioCPH]

Hi, this is very useful for production, and here are some requirements about this feature:

• Export leader info, such as report 1 if this is leader and report 0 if not leader.
• Report TFJob object creation and deletion infos, such as how many object created per seconds.
• Report CPU/Memory used infos.

[krishnadurai]

@ScorpioCPH thanks for posting these requirements. I'll include these in the PR.

[krishnadurai]

Metrics being targeted to track using Prometheus:

1. Report for each pod (tf-operator, tf-master, tf-ps and tf-worker), reports on:
- [x] CPU usage
- [x] GPU usage
- [x] Memory usage
- [x] Network usage
- [x] I/O usage
- [x] Keep-Alive check
- [x] Is Leader check
2. Report TFJob metrics:
- [x] Job creation
- [x] Job deletion
- [x] Jobs created per hour
- [x] Successful job completions

[gaocegege]

@krishnadurai

Thanks for the summary. Will your PR be ready after 0.6 release?

[johnugeorge]

@gaocegege It is planned for upcoming v1 release

[richardsliu]

This is now done.
/close

[k8s-ci-robot]

@richardsliu: Closing this issue.

In response to this:

This is now done.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.