Katib: [RFC] Katib Revised Edition

[gaocegege]

In v1alpha2, we abstract Experiment and Trial as Kubernetes CRD, in the meanwhile, store experiments, trials, metrics and algorithm settings in MySQL.

When we are investigating katib to adopt it into production, we find that katib requires a MySQL instance, which limits the availability and reduces the usability of the system. This motivates our refactor design of katib. We eliminate katib-manager GRPC server and MySQL dependency (Ref #463), and introduce more CRDs and controllers, to:

• keep high availability and production ready. Make sure that all components are stateless.
• manage the Kubeflow stack easily. In our community, most features are based on CRDs (TFJob, PyTorchJob, MPIJob, KFServing). it will be easier to manage if katib is also built on top of CRDs.
• make katib non-invasive. Currently, katib requires user to expose hyperparameters as CLI arguments. This is not always feasible in practice and requires a 1:1 match between the arguments and code to parse arguments, which is error prone and tedious.

In the new design, we give more work to controllers, instead of katib-manager. Please have a look at the proposal.

We are working on the implementation of the revised version for our internal product. And I am glad to see if we could contribute all or part implementation to the upstream.

/cc @andreyvelich @YujiOshima @johnugeorge @richardsliu @hougangliu @ddysher

[gaocegege]

/cc @terrytangyuan Maybe you are interested in it, too.

[richardsliu]

/cc @jdplatt
/cc @achalshant
/cc @mpvartak

[terrytangyuan]

Thanks @gaocegege. Just left some comments in the doc.

[richardsliu]

Update after meeting on 4/25:

• Top priority from this proposal is removing the dependency on MySQL
• Proposed solution (from @gaocegege ):
  
  
  
  • Store/track trial status from CRD. Controller will be responsible for this.
  
  
  • Similarly, best trial will also be stored in the Experiment status.
  
  
  • This means that no DB writes are done during main workflow.
  
  
  • Writing to data store can be done via a backup controller, for historical data.
  
  
  • Need to have a solution to make the UI work (including historical data)
  
  
  • We can also implement suggestions as a CRD
  
  
  • This means we do not need to use katib manager as a proxy
  
  
  • Introduce a time-series database (e.g. Prometheus) for periodic metric storage
  
  
  • TSDB can be optionally installed, but is required for early stopping
  
  
• @gaocegege will propose/work on a prototype based on v1alpha2, target completion by mid-May

Let me know if I've missed anything.

[hougangliu]

@gaocegege Sorry that something of the design confuses me. Can you please help clarify it?
As for We can also implement suggestions as a CRD, does it mean each suggestion service corresponds to a suggestion CR?
If yes, which controller creates the suggestion CRs? when create it? when delete it?

in design doc, it says A Suggestion contains a service and a deployment, it exports the services in the cluster. In this version, NAS, Hyperparameter Tuning, and early stopping services share the same CRD. The suggestion service is long-running. I cannot understand why we need suggestion CRD instead of current arch since they are long-running services as they are now.

Thanks!

[gaocegege]

@hougangliu

For HP tuning, it is similar to the current design. It is long-running. But, there are some cases reuqire experiment specific suggestions. See https://github.com/kubeflow/katib/issues/471

Making suggestion a CRD is a potential solution and I am working on the prototype. Feel free to leave comments in https://github.com/kubeflow/katib/issues/471 if you have other suggestions.

[hougangliu]

@gaocegege Thanks for clarifying. Maybe we can talk about it for more detail in meeting

[gaocegege]

Maybe we can use https://github.com/directxman12/k8s-prometheus-adapter and prometheus to support early stopping. We can store the metrics in prometheus and expose the metrics using custom.metrics.k8s.io

Ref https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#support-for-metrics-apis
Ref https://www.reactiveops.com/blog/kubernetes-hpa-autoscaling-with-custom-and-external-metrics-using-gke-and-stackdriver-metrics

I will work on it in our prototype caicloud/katib

[johnugeorge]

How will you handled this issue if you are using prometheus?

[gaocegege]

I think these two issues are different. For periodical metrics in the trial, it is just like CPU/memory utilization. And it is only used to stop the trial earlier. metrics server works well for this case.

Actually, I am thinking if we need to maintain all periodical metrics. I am thinking if we could store the latest metrics in CRD status and update it in a reasonable interval. As we know, early stopping services only need the latest periodical metrics.

[gaocegege]

/close