Kubeflow: Katib training containers should wait until istio proxy will be ready

[andreyvelich]

/kind bug

What steps did you take and what happened:
To solve this problem https://github.com/kubeflow/kubeflow/issues/4730#issuecomment-582653698 I had to add 1-2 minutes delay before running request to download mnist dataset in the training container.
The problem was that istio proxy container was not ready when training container ran request.

Training containers should somehow wait until istio proxy will be ready.

@krishnadurai proposed specified istio-injection: disabled annotation for the Profile namespace where user submits Katib Experiments.
/cc @johnugeorge @jlewi @gaocegege @hougangliu

[issue-label-bot[bot]]

Issue-Label Bot is automatically applying the labels:

| Label | Probability |
| ------------- | ------------- |
| kind/bug | 0.97 |

Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback!
Links: app homepage, dashboard and code for this bot.

[andreyvelich]

/cc @richardsliu

[krishnadurai]

@andreyvelich A minor correction to the issue header comment.
The pods for the experiment should have the annotation 'istio-injection': 'disabled' rather than the namespace. So the controller creating the pods might want to add these annotations.

[jlewi]

Istio reference https://istio.io/docs/setup/additional-setup/sidecar-injection/#understanding-what-happened