Tf-operator: Support mpi jobs in universal operator

[Jeffwan]

We've migrated pytorch, mxnet(byteps), xgboost (lightbgm). The next step is to consider moving mpi apis and controllers to this repo. (for people who are not familiar with the background, check https://docs.google.com/document/d/1x1JPDQfDMIbnoQRftDH1IzGU0qvHGSU4W6Jl4rJLPhI/edit#heading=h.e33ufidnl8z6)

The reason we didn't make it in the first draft is the code structure of mpi-operator is a little bit different from rest operators.

However, we do see strong needs to adopt it in universal operator so users can easily leverage it to run horovod and horovod elastic jobs. Another big motivation is DGL support can be built on top of MPI-operator. This would be a big gain for graph users.

/cc @kubeflow/wg-training-leads @ryantd @carmark

[Jeffwan]

/cc @alculquicondor

[gaocegege]

cc @zw0610

[Jeffwan]

@zw0610 had some discussion with @carmark and @ryantd in the past. Do you have the conclusion? BTW, what's the plan for v2 MPI? Is it ok to adopt v2 directly?

[zw0610]

On the last community meeting, I've explained that internally we prefer using the APIs that are already production-ready. From my understanding, the v2 API as long as its controller is still under completion and has not been widely adopted by the community users. Merging v2 version directly may not benefit many users who persist with v1 API.

[alculquicondor]

I estimate the v2 controller and API will be ready in a month at most. Then we can produce a release.

The new controller has more coverage at unit, integration and e2e level, so it's in good track to be production-ready :)

[alculquicondor]

Also, I'm interested in helping with the migration of mpi-operator to the universal operator, but only after the v2 release. I don't think I would be able to implement everything myself, but I can at the very least help with reviews.

[johnugeorge]

How about getting v2 API directly in the universal operator while we continue using v1 from current MPI repo? This way, both will be isolated and switching from v1 to v2 will be easy?

[alculquicondor]

I vote yes to that. We can migrate only the v2 API. But we are not ready yet.

[Jeffwan]

@johnugeorge

@zw0610 proposed to merge two version in this repo. Since mpi uses different API versions, it's ok for them to coexist. There're lots of v1 users as I know and they will not immediately migrate to v2 because of both operator and api changes

[alculquicondor]

One problem I see is that we don't have a conversion webhook. The v2 controller can't support v1 APIs, and even if it could, it cannot handle the missing resources that jobs created with a v1 controller wouldn't have (like the workers Service). I think it's reasonable to only have support for v2. If users don't want to upgrade yet, they can keep using the separate v1 controller.

[Jeffwan]

One problem I see is that we don't have a conversion webhook. The v2 controller can't support v1 APIs, and even if it could, it cannot handle the missing resources that jobs created with a v1 controller wouldn't have (like the workers Service). I think it's reasonable to only have support for v2. If users don't want to upgrade yet, they can keep using the separate v1 controller.

I mean two apis + two controllers. Just treat mpiv2 as a separate framework. In that case, we don't necessarily need conversions in that case.

[alculquicondor]

I don't see much value from adding the two controllers, considering the extra effort.

[alculquicondor]

I think we should take this as an opportunity to encourage users to upgrade and simplify their infrastructure.

[gaocegege]

Is there any progress? Recently some users expect that horovod can be supported in the universal operator.

[gaocegege]

/cc @qiankunli

[carmark]

Our team have been working on this integration, hope that we can submit a draft PR next week.

[alculquicondor]

Are the integration and E2E tests being migrated as well?