Tf-operator: Document how to use gang scheduling with TFJob

[jlewi]

In #349 we added support for gang scheduling use kube arbitrator.

We should document how to use this feature and how it works. Some things we should probably explain

1. The dependency on kube-arbitrator and at least reference to how to install it

2. from a user perspective how it impacts how jobs are scheduled

   • e.g. what happens when a job requiring N pods is created and there are only enough resources to schedule N-2 pods?
   • What happens when a job is running and one of the pods dies? Is it possible something else will get scheduled before a new pod (for a different resource) can be created and scheduled before a replacement can be scheduled?

We currently have this section in our user guide about TFJob
https://github.com/kubeflow/website/blob/master/content/docs/about/user_guide.md#submitting-a-tensorflow-training-job

We should probably split the information about TFJob into its own page and then add a section about how to use it with Kube arbitrator for advanced scheduling.

/cc @k82cn @mitake @gaocegege

[k82cn]

+1

[gaocegege]

SGTM, I will do it in this month if no one is interested in it.

[gaocegege]

/assign @ChanYiLin

[gaocegege]

According to the discussions in #801 , @ChanYiLin is interested in this issue.

[gaocegege]

/cc @cheyang @k82cn

Hope to get your review when the doc is ready, thanks.

[johnugeorge]

Since we are planning to have consistent behavior across operators, we can have a general doc too.

https://github.com/kubeflow/kubeflow/issues/1471

[gaocegege]

SGTM, I think we share the same patterns (using labels to indicate the job), thus we could use TFJob as an example to illustrate how to integrate mxnet/pytorch/tf jobs with kube-arbitrator.

[k82cn]

sure, my pleasure :)

[ChanYiLin]

Hi, i have tested the tf-operator with kube-batchd:v0.2 these days, but I found it doesn't work on both GKE and my own environment.
I then tested it using my own code base which was backed up around this commit, it works well. https://github.com/kubernetes-incubator/kube-arbitrator/pull/181

I am now trying to find the problems, so the doc might be delayed until the kube-batchd:v0.2 works again.

/cc @k82cn @cheyang

[k82cn]

Any logs?

[ChanYiLin]

The problem I encountered was that
even the pdb were created and the resources the tfjob required were smaller then the capacity, all the pods kept in pending state.
So I try to follow the kube-arbitrator's new tutorial and use podgroup instead, it didn't work either.

Here are two issues

1. I didn't get any logs related to pdb, tfjob's pods or resources info, so I cannot trace the error.

1. I got these errors in my kube-batchd logs:

I0910 10:27:16.745314       1 reflector.go:240] Listing and watching *v1alpha1.PodGroup from github.com/kubernetes-incubator/kube-arbitrator/pkg/scheduler/cache/cache.go:243
E0910 10:27:16.747168       1 reflector.go:205] github.com/kubernetes-incubator/kube-arbitrator/pkg/scheduler/cache/cache.go:243: Failed to list *v1alpha1.PodGroup: podgroups.scheduling.incubator.k8s.io is forbidden: User "system:serviceaccount:kube-system:my-scheduler" cannot list podgroups.scheduling.incubator.k8s.io at the cluster scope
I0910 10:27:17.747374       1 reflector.go:240] Listing and watching *v1alpha1.PodGroup from github.com/kubernetes-incubator/kube-arbitrator/pkg/scheduler/cache/cache.go:243
E0910 10:27:17.748986       1 reflector.go:205] github.com/kubernetes-incubator/kube-arbitrator/pkg/scheduler/cache/cache.go:243: Failed to list *v1alpha1.PodGroup: podgroups.scheduling.incubator.k8s.io is forbidden: User "system:serviceaccount:kube-system:my-scheduler" cannot list podgroups.scheduling.incubator.k8s.io at the cluster scope

but I did create clusterrole using rules like below:

- apiGroups:
  - scheduling.incubator.k8s.io
  resources:
  - podgroupds
  verbs:
  - '*'

@k82cn do you have any idea?

[ChanYiLin]

Hi @k82cn
I fix my first problem by just applying the rules:

- apiGroups:
  - '*'
  resources:
  - '*'
  verbs:
  - '*'

It is not the best way actually, do you think we should also add how to set SA/clusterrole in the tutorial in kube-arbitrator repo?

BTW, I found it didn't work with pdb after I fix the clusterrole problem, but it worked with using podgroup which is a crd created and used by kube-arbitrator.

@jlewi @gaocegege , do you think we should change from using pdb to podgroup? Pdb is a kubernetes official resource and podgroup is a crd.

@k82cn I am not sure why kube-arbitrator uses podgroup instead of pdb, could you tell us?

[k82cn]

It is not the best way actually, do you think we should also add how to set SA/clusterrole in the tutorial in kube-arbitrator repo?

We already have a NOTEs in the tutorial; it's security configuration, that maybe hard to give a correct configuration. So we give a general notes, and that'll be great if you have a better idea :)

NOTE: kube-arbitrator need to collect cluster information(such as Pod, Node, CRD, etc) for scheduing, so the service account used by the deployment must have permission to access those cluster resources, otherwise, kube-arbitrator will fail to startup.

do you think we should change from using pdb to podgroup? Pdb is a kubernetes official resource and podgroup is a crd.

PDB is used for rolling update, and we'll not add any new field for "batch". PodGroup is introduced by https://github.com/kubernetes/community/pull/2337 . I'm trying to make is as offcial for batch workload.

I found it didn't work with pdb after I fix the clusterrole problem

That maybe a bug, I'll check that. Did some changes for PodGroup.

[gaocegege]

@k82cn @jlewi @ChanYiLin

do you think we should change from using pdb to podgroup? Pdb is a kubernetes official resource and podgroup is a crd.

I think several months ago we can use label to aggregate the pods to a TFJob in kube-arbitrator. Now we have to use PDB or PodGroup and the feature is abandoned?

[k82cn]

I think several months ago we can use label to aggregate the pods to a TFJob in kube-arbitrator. Now we have to use PDB or PodGroup and the feature is abandoned?

We're using owner reference to group the pods from the beginning :) The PDB part should be broken because PodGroup are using annotations. I'll add an e2e to cover this case.

For PodGroup, I'm working to merge it into core (https://github.com/kubernetes/kubernetes/pull/67111). So I'd suggest to keep current integration, and do update when https://github.com/kubernetes/kubernetes/pull/67111 merged.

[k82cn]

Opened https://github.com/kubernetes-incubator/kube-arbitrator/issues/364 to handle PDB issue. That'll be handled in 0.2.

[gaocegege]

SGTM

WDYT @ChanYiLin

[ChanYiLin]

@k82cn @gaocegege
I agree that we can keep using pdb until PodGroup is merged.

Also, I have submit a PR to kube-arbitrator, I've tested it and can fix the problem of using pdb above.

Here is a little summary,

1. tf-operator don't need to change to integrated with kube-batchd, pdb still works.
2. different from podgroup, we don't need to add this part to spec while using pdb

metadata:
  annotations:
      scheduling.k8s.io/group-name: ${JOB_NAME}

1. I got last issue that if I set different resource limits in PS and Worker spec, then the scheduler will fail and the logs will show:

I0913 07:19:30.995262       1 allocate.go:72] Queue <default> is overused, ignore it.

[gaocegege]

I think we could close the issue since https://github.com/kubeflow/website/pull/241 was merged