Katib: Collect metrics from TF Events files

[jlewi]

Relevant issues
#87 Study Job CRD; Don't require users to write code to do HP tuning.
#39 support TFJob and other frameworks

We'd like to be able to collect metrics from a TF.Events file produced by a TensorFlow training job.

So at a high level what we need is

1. A reusable binary/job that can ready a TF.Events file and report specified metrics via Katib metrics API
2. In the StudyJobCRD we need to be able to launch this binary for each training job and pass it the location of the TF events file.

[gaocegege]

I cannot figure out a better solution than PVC. We need help from the community

/cc @YujiOshima @ChanYiLin @ddysher

[jlewi]

Can you explain the issue? Is this just a question of making the events file accessible by two processes e.g.

1. The trainer which writes the TF.events file
2. The metrics collector which reads it.

Using a PVC to share the TF.events file seems perfectly reasonable. We can also support object stores (S3, HDFS, GCS) since TF can read/write those directly.

[YujiOshima]

I heard @gaocegege will make a WIP PR for TFJob support first in the next week.
I wrote down a rough design of a metrics collector for TF Event.

We can implement the metrics collector for TF Event independent from TFJob support.
I will make TF Event metrics collector PR.

[johnugeorge]

@YujiOshima how will this be compatible with PyTorch job?

[YujiOshima]

@johnugeorge I will parse a tf.Event file with event.proto.
I'm not familiar with the PyTorch log format or ONNIX format.
Could you tell me how we can parse them?

[johnugeorge]

@YujiOshima I am not aware of equivalent official one in pytorch. What are the other options to support it?

[YujiOshima]

@johnugeorge I think there is two way.

• Use the tensorboardX library on your worker. I think the logs are formatted TF.Events format.
• Print log to stdout and use default metrics collector. The default metrics collector will parse stdout of your worker.
  WDYT?

[johnugeorge]

The problem is that we are forcing the workers to use a particular library or particular format. Currently, I think this is the only way.

[jlewi]

@johnugeorge I don't think we are forcing users to do things a particular way; the idea is to make Katib pluggable with respect to how metrics are collected. Support for TF.Events is just one of the methods that we want to be well supported in order to support TF,

[YujiOshima]

Tf event metrics collector was added by #235 .
/close

[k8s-ci-robot]

@YujiOshima: Closing this issue.

In response to this:

Tf event metrics collector was added by #235 .
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.