Katib: MetricsUnavailable for random example experiment

[KineticCookie]

/kind bug

What steps did you take and what happened:
[A clear and concise description of what the bug is.]
Installed Kubeflow on a clean EKS cluster using this guide https://www.kubeflow.org/docs/aws/deploy/install-kubeflow/

Submitted random-experiment from UI.

After an hour there is still no status, logs or metrics.

Pods that were spawned by Trial's Job have logs.

What did you expect to happen:
Metrics to show on Katib UI. Experiment to finish.

Anything else you would like to add:

Pod logs:

INFO:root:Epoch[19] Batch [100] Speed: 27953.82 samples/sec accuracy=0.116646
INFO:root:Epoch[19] Batch [200] Speed: 24880.15 samples/sec accuracy=0.111406
INFO:root:Epoch[19] Batch [300] Speed: 23859.95 samples/sec accuracy=0.112344
INFO:root:Epoch[19] Batch [400] Speed: 27594.30 samples/sec accuracy=0.115937
INFO:root:Epoch[19] Batch [500] Speed: 18158.40 samples/sec accuracy=0.115312
INFO:root:Epoch[19] Batch [600] Speed: 26611.55 samples/sec accuracy=0.102188
INFO:root:Epoch[19] Batch [700] Speed: 27180.25 samples/sec accuracy=0.114687
INFO:root:Epoch[19] Batch [800] Speed: 27309.44 samples/sec accuracy=0.113906
INFO:root:Epoch[19] Batch [900] Speed: 26656.13 samples/sec accuracy=0.105313
INFO:root:Epoch[19] Train-accuracy=0.122044
INFO:root:Epoch[19] Time cost=2.383
INFO:root:Epoch[19] Validation-accuracy=0.113854

Description of a Trial:

Name:         hptest-bt8nsw2z
Namespace:    kubeflow
Labels:       experiment=hptest
Annotations:  <none>
API Version:  kubeflow.org/v1alpha3
Kind:         Trial
Metadata:
  Creation Timestamp:  2020-01-28T16:18:39Z
  Finalizers:
    clean-metrics-in-db
  Generation:  1
  Owner References:
    API Version:           kubeflow.org/v1alpha3
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Experiment
    Name:                  hptest
    UID:                   ab0e3811-41e9-11ea-a0cf-0a9ff0751f4a
  Resource Version:        127230
  Self Link:               /apis/kubeflow.org/v1alpha3/namespaces/kubeflow/trials/hptest-bt8nsw2z
  UID:                     d810a019-41e9-11ea-a0cf-0a9ff0751f4a
Spec:
  Metrics Collector:
  Objective:
    Additional Metric Names:
      accuracy
    Goal:                   0.99
    Objective Metric Name:  Validation-accuracy
    Type:                   maximize
  Parameter Assignments:
    Name:    --lr
    Value:   0.020744080613308936
    Name:    --num-layers
    Value:   3
    Name:    --optimizer
    Value:   sgd
  Run Spec:  apiVersion: batch/v1
kind: Job
metadata:
  name: hptest-bt8nsw2z
  namespace: kubeflow
spec:
  template:
    spec:
      containers:
      - name: hptest-bt8nsw2z
        image: docker.io/katib/mxnet-mnist-example
        command:
        - "python"
        - "/mxnet/example/image-classification/train_mnist.py"
        - "--batch-size=64"
        - "--lr=0.020744080613308936"
        - "--num-layers=3"
        - "--optimizer=sgd"
      restartPolicy: Never
Status:
  Conditions:
    Last Transition Time:  2020-01-28T16:18:39Z
    Last Update Time:      2020-01-28T16:18:39Z
    Message:               Trial is created
    Reason:                TrialCreated
    Status:                True
    Type:                  Created
    Last Transition Time:  2020-01-28T16:19:43Z
    Last Update Time:      2020-01-28T16:19:43Z
    Message:               Trial is running
    Reason:                TrialRunning
    Status:                False
    Type:                  Running
    Last Transition Time:  2020-01-28T16:19:43Z
    Last Update Time:      2020-01-28T16:19:43Z
    Message:               Metrics are not available
    Reason:                MetricsUnavailable
    Status:                False
    Type:                  Succeeded
  Start Time:              2020-01-28T16:18:39Z
Events:
  Type     Reason              Age                 From              Message
  ----     ------              ----                ----              -------
  Warning  MetricsUnavailable  31m (x2 over 139m)  trial-controller  Metrics are not available for Job hptest-bt8nsw2z

Environment:

• Kubeflow version: 7.1 from https://raw.githubusercontent.com/kubeflow/manifests/v0.7-branch/kfdef/kfctl_aws.0.7.1.yaml
• Minikube version: N/A. Deployed on EKS.
• Kubernetes version: (use kubectl version): version.Info{Major:"1", Minor:"14+", GitVersion:"v1.14.9-eks-c0eccc", GitCommit:"c0eccca51d7500bb03b2f163dd8d534ffeb2f7a2", GitTreeState:"clean", BuildDate:"2019-12-22T23:14:11Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}
• OS (e.g. from /etc/os-release): Amazon Linux (?)

[issue-label-bot[bot]]

Issue-Label Bot is automatically applying the labels:

| Label | Probability |
| ------------- | ------------- |
| bug | 0.91 |

Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback!
Links: app homepage, dashboard and code for this bot.

[andreyvelich]

@KineticCookie Thank you for the issue.
Can you show description of the experiment?
If it possible, show description for the Training Container Pod after you submit random-example.

[KineticCookie]

Unfortunately, someone already deleted experiment. I will submit it again and show the necessary descriptions.

[KineticCookie]

I recreated the experiment and unfortunately I encountered the same behaviour.
Description of the experiment:

Name:         random-experiment
Namespace:    kubeflow
Labels:       <none>
Annotations:  <none>
API Version:  kubeflow.org/v1alpha3
Kind:         Experiment
Metadata:
  Creation Timestamp:  2020-01-29T16:00:34Z
  Finalizers:
    update-prometheus-metrics
  Generation:        1
  Resource Version:  540576
  Self Link:         /apis/kubeflow.org/v1alpha3/namespaces/kubeflow/experiments/random-experiment
  UID:               7bf3348f-42b0-11ea-a0cf-0a9ff0751f4a
Spec:
  Algorithm:
    Algorithm Name:  random
    Algorithm Settings:
  Max Failed Trial Count:  3
  Max Trial Count:         12
  Objective:
    Additional Metric Names:
      accuracy
    Goal:                   0.99
    Objective Metric Name:  Validation-accuracy
    Type:                   maximize
  Parallel Trial Count:     3
  Parameters:
    Feasible Space:
      Max:           0.03
      Min:           0.01
    Name:            --lr
    Parameter Type:  double
    Feasible Space:
      Max:           5
      Min:           2
    Name:            --num-layers
    Parameter Type:  int
    Feasible Space:
      List:
        sgd
        adam
        ftrl
    Name:            --optimizer
    Parameter Type:  categorical
  Trial Template:
    Go Template:
      Template Spec:
        Config Map Name:       trial-template
        Config Map Namespace:  kubeflow
        Template Path:         defaultTrialTemplate.yaml
Status:
  Conditions:
    Last Transition Time:  2020-01-29T16:00:34Z
    Last Update Time:      2020-01-29T16:00:34Z
    Message:               Experiment is created
    Reason:                ExperimentCreated
    Status:                True
    Type:                  Created
    Last Transition Time:  2020-01-29T16:01:51Z
    Last Update Time:      2020-01-29T16:01:51Z
    Message:               Experiment is running
    Reason:                ExperimentRunning
    Status:                True
    Type:                  Running
  Current Optimal Trial:
    Observation:
      Metrics:              <nil>
    Parameter Assignments:  <nil>
  Start Time:               2020-01-29T16:00:34Z
  Trials:                   3
  Trials Pending:           3
Events:                     <none>

Pod description:

Name:           random-experiment-cz5q25r4-v2rhl
Namespace:      kubeflow
Priority:       0
Node:           ip-192-168-10-113.eu-central-1.compute.internal/192.168.10.113
Start Time:     Wed, 29 Jan 2020 19:01:51 +0300
Labels:         controller-uid=a952e2b6-42b0-11ea-a0cf-0a9ff0751f4a
                job-name=random-experiment-cz5q25r4
Annotations:    kubernetes.io/psp: eks.privileged
Status:         Succeeded
IP:             192.168.29.105
IPs:            <none>
Controlled By:  Job/random-experiment-cz5q25r4
Containers:
  random-experiment-cz5q25r4:
    Container ID:  docker://318bfa0f16a18eed7080d9b6e11d7e4ac7cf0c3d62ba2a951d3e9d59035c53ab
    Image:         docker.io/katib/mxnet-mnist-example
    Image ID:      docker-pullable://katib/mxnet-mnist-example@sha256:b1152b6610e2fc679eedc7082ad0dca02c64715179b7620243c46536dc6cbc19
    Port:          <none>
    Host Port:     <none>
    Command:
      python
      /mxnet/example/image-classification/train_mnist.py
      --batch-size=64
      --lr=0.02253223669535722
      --num-layers=2
      --optimizer=adam
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Wed, 29 Jan 2020 19:02:06 +0300
      Finished:     Wed, 29 Jan 2020 19:07:02 +0300
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-jw96x (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  default-token-jw96x:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-jw96x
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:          <none>

[KineticCookie]

And the description of a newly created pod

Name:           random-experiment-cz5q25r4-7v2mt
Namespace:      kubeflow
Priority:       0
Node:           ip-192-168-10-113.eu-central-1.compute.internal/192.168.10.113
Start Time:     Wed, 29 Jan 2020 21:16:15 +0300
Labels:         controller-uid=703fd71a-42c3-11ea-a0cf-0a9ff0751f4a
                job-name=random-experiment-cz5q25r4
Annotations:    kubernetes.io/psp: eks.privileged
Status:         Running
IP:             192.168.7.211
IPs:            <none>
Controlled By:  Job/random-experiment-cz5q25r4
Containers:
  random-experiment-cz5q25r4:
    Container ID:  docker://baa8e6a32e7026361cb12f28b3f8c12fd06e5f069727c18558e179f515029883
    Image:         docker.io/katib/mxnet-mnist-example
    Image ID:      docker-pullable://katib/mxnet-mnist-example@sha256:b1152b6610e2fc679eedc7082ad0dca02c64715179b7620243c46536dc6cbc19
    Port:          <none>
    Host Port:     <none>
    Command:
      python
      /mxnet/example/image-classification/train_mnist.py
      --batch-size=64
      --lr=0.02253223669535722
      --num-layers=2
      --optimizer=adam
    State:          Running
      Started:      Wed, 29 Jan 2020 21:16:18 +0300
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-jw96x (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  default-token-jw96x:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-jw96x
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type    Reason     Age   From                                                      Message
  ----    ------     ----  ----                                                      -------
  Normal  Scheduled  24s   default-scheduler                                         Successfully assigned kubeflow/random-experiment-cz5q25r4-7v2mt to ip-192-168-10-113.eu-central-1.compute.internal
  Normal  Pulling    23s   kubelet, ip-192-168-10-113.eu-central-1.compute.internal  Pulling image "docker.io/katib/mxnet-mnist-example"
  Normal  Pulled     22s   kubelet, ip-192-168-10-113.eu-central-1.compute.internal  Successfully pulled image "docker.io/katib/mxnet-mnist-example"
  Normal  Created    21s   kubelet, ip-192-168-10-113.eu-central-1.compute.internal  Created container random-experiment-cz5q25r4
  Normal  Started    21s   kubelet, ip-192-168-10-113.eu-central-1.compute.internal  Started container random-experiment-cz5q25r4

[andreyvelich]

For some reason training container pod doesn't have metrics collector container and metrics can't be collected.
Can you try to deploy Katib on your cluster using the latest manifests and images from here: https://github.com/kubeflow/katib/tree/master/manifests/v1alpha3 ?
After that you can try to run random-example from the examples folder.

[martham93]

Hello, the same behavior is happening with the bayesian optimization example, metrics aren't being reported so the experiment gets stuck after running the first three parallel trials.

Experiment Description:
```kubectl describe experiment bayesianoptimization-example -n kubeflow

Name: bayesianoptimization-example
Namespace: kubeflow
Labels: controller-tools.k8s.io=1.0
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"kubeflow.org/v1alpha3","kind":"Experiment","metadata":{"annotations":{},"labels":{"controller-tools.k8s.io":"1.0"},"name":"...
API Version: kubeflow.org/v1alpha3
Kind: Experiment
Metadata:
Creation Timestamp: 2020-01-30T17:43:22Z
Finalizers:
update-prometheus-metrics
Generation: 1
Resource Version: 21203
Self Link: /apis/kubeflow.org/v1alpha3/namespaces/kubeflow/experiments/bayesianoptimization-example
UID: 02c79849-4388-11ea-9d91-42010a8e0025
Spec:
Algorithm:
Algorithm Name: bayesianoptimization
Algorithm Settings:
Name: random_state
Value: 10
Max Failed Trial Count: 3
Max Trial Count: 12
Objective:
Additional Metric Names:
Train-accuracy
Goal: 0.99
Objective Metric Name: Validation-accuracy
Type: maximize
Parallel Trial Count: 3
Parameters:
Feasible Space:
Max: 0.03
Min: 0.01
Name: --lr
Parameter Type: double
Feasible Space:
Max: 5
Min: 2
Name: --num-layers
Parameter Type: int
Feasible Space:
List:
sgd
adam
ftrl
Name: --optimizer
Parameter Type: categorical
Trial Template:
Go Template:
Raw Template: apiVersion: batch/v1
kind: Job
metadata:
name: {{.Trial}}
namespace: {{.NameSpace}}
spec:
template:
spec:
containers:
- name: {{.Trial}}
image: docker.io/kubeflowkatib/mxnet-mnist
command:
- "python3"
- "/opt/mxnet-mnist/mnist.py"
- "--batch-size=64"
{{- with .HyperParameters}}
{{- range .}}
- "{{.Name}}={{.Value}}"
{{- end}}
{{- end}}
restartPolicy: Never
Status:
Conditions:
Last Transition Time: 2020-01-30T17:43:22Z
Last Update Time: 2020-01-30T17:43:22Z
Message: Experiment is created
Reason: ExperimentCreated
Status: True
Type: Created
Last Transition Time: 2020-01-30T17:45:10Z
Last Update Time: 2020-01-30T17:45:10Z
Message: Experiment is running
Reason: ExperimentRunning
Status: True
Type: Running
Current Optimal Trial:
Observation:
Metrics:
Parameter Assignments:
Start Time: 2020-01-30T17:43:22Z
Trials: 3
Trials Pending: 3

Pod Description: 

kubectl describe pod bayesianoptimization-example-mpn7zqlq-vflb7 -n kubeflow

Name: bayesianoptimization-example-mpn7zqlq-vflb7
Namespace: kubeflow
Priority: 0
PriorityClassName:
Node: gke-kubeflow-app-e88-kubeflow-app-e88-8584f473-0r1h/10.142.15.208
Start Time: Thu, 30 Jan 2020 10:45:10 -0700
Labels: controller-uid=42fd0894-4388-11ea-9d91-42010a8e0025
job-name=bayesianoptimization-example-mpn7zqlq
Annotations:
Status: Succeeded
IP: 10.12.0.35
Controlled By: Job/bayesianoptimization-example-mpn7zqlq
Containers:
bayesianoptimization-example-mpn7zqlq:
Container ID: docker://95d4137b7a872329a98eb09bede346722c0591a8606defa98f3b8e974c5963f8
Image: docker.io/kubeflowkatib/mxnet-mnist
Image ID: docker-pullable://kubeflowkatib/mxnet-mnist@sha256:85e62e489033dd327e5db7322b636db15b7fe6b380c5846093926c66afb39d8a
Port:
Host Port:
Command:
python3
/opt/mxnet-mnist/mnist.py
--batch-size=64
--lr=0.013884981186857928
--num-layers=3
--optimizer=adam
State: Terminated
Reason: Completed
Exit Code: 0
Started: Thu, 30 Jan 2020 10:45:27 -0700
Finished: Thu, 30 Jan 2020 10:47:46 -0700
Ready: False
Restart Count: 0
Environment:
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-c8fcw (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
default-token-c8fcw:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-c8fcw
Optional: false
QoS Class: BestEffort
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 10m default-scheduler Successfully assigned kubeflow/bayesianoptimization-example-mpn7zqlq-vflb7 to gke-kubeflow-app-e88-kubeflow-app-e88-8584f473-0r1h
Normal Pulling 10m kubelet, gke-kubeflow-app-e88-kubeflow-app-e88-8584f473-0r1h Pulling image "docker.io/kubeflowkatib/mxnet-mnist"
Normal Pulled 10m kubelet, gke-kubeflow-app-e88-kubeflow-app-e88-8584f473-0r1h Successfully pulled image "docker.io/kubeflowkatib/mxnet-mnist"
Normal Created 10m kubelet, gke-kubeflow-app-e88-kubeflow-app-e88-8584f473-0r1h Created container bayesianoptimization-example-mpn7zqlq
Normal Started 10m kubelet, gke-kubeflow-app-e88-kubeflow-app-e88-8584f473-0r1h Started container bayesianoptimization-example-mpn7zqlq
```

[martham93]

just to clarify this is error is with the most current bayesian optimization example that was updated ~2 weeks ago.

[fedeesku1]

I'm having the same issue. These are my specs.

Kubenetes Cluster:
minikube start --cpus 4 --memory 12144 --disk-size=50g --kubernetes-version=v1.15.0
🙄 minikube v1.5.2 on Ubuntu 18.04
✨ Automatically selected the 'virtualbox' driver (alternates: [none])
🔥 Creating virtualbox VM (CPUs=4, Memory=12144MB, Disk=50000MB) ...
🐳 Preparing Kubernetes v1.15.0 on Docker '18.09.9' ...

Kubectl:
kubectl version
Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.2", GitCommit:"c97fe5036ef3df2967d086711e6c0c405941e14b", GitTreeState:"clean", BuildDate:"2019-10-15T19:18:23Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.0", GitCommit:"e8462b5b5dc2584fdcd18e6bcfe9f1e4d970a529", GitTreeState:"clean", BuildDate:"2019-06-19T16:32:14Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}

Kubeflow:
kfctl version
kfctl v1.0-rc.1-0-g963c787

config.yaml:
https://raw.githubusercontent.com/kubeflow/manifests/v0.7-branch/kfdef/kfctl_k8s_istio.0.7.1.yaml

Downloaded the random-example.yaml and ran:
kubectl apply -f random-example.yaml

[fedeesku1]

Is the issue with not defining a metricsCollector? Is the default not stdout?

kubectl get trials random-example-g4qlpmbt -n kubeflow -oyaml

apiVersion: kubeflow.org/v1alpha3
kind: Trial
metadata:
creationTimestamp: "2020-01-30T19:15:30Z"
finalizers:

• clean-metrics-in-db
  generation: 1
  labels:
  experiment: random-example
  name: random-example-g4qlpmbt
  namespace: kubeflow
  ownerReferences:
• apiVersion: kubeflow.org/v1alpha3
  blockOwnerDeletion: true
  controller: true
  kind: Experiment
  name: random-example
  uid: 00749e91-9c19-46ce-93ba-19794249e62b
  resourceVersion: "427298"
  selfLink: /apis/kubeflow.org/v1alpha3/namespaces/kubeflow/trials/random-example-g4qlpmbt
  uid: b44b1095-2fca-43ca-aae1-f78a5800a4bf
  spec:
  metricsCollector: {}
  objective:
  additionalMetricNames:
  
  
  
  • Train-accuracy
    
    goal: 0.99
    
    objectiveMetricName: Validation-accuracy
    
    type: maximize
    
    parameterAssignments:
  
  
• name: --lr
  value: "0.014790028597009047"
• name: --num-layers
  value: "4"
• name: --optimizer
  value: ftrl
  runSpec: |-
  apiVersion: batch/v1
  kind: Job
  metadata:
  name: random-example-g4qlpmbt
  namespace: kubeflow
  spec:
  template:
  spec:
  containers:
  - name: random-example-g4qlpmbt
  image: docker.io/kubeflowkatib/mxnet-mnist
  command:
  - "python3"
  - "/opt/mxnet-mnist/mnist.py"
  - "--batch-size=64"
  - "--lr=0.014790028597009047"
  - "--num-layers=4"
  - "--optimizer=ftrl"
  restartPolicy: Never
  status:
  conditions:
• lastTransitionTime: "2020-01-30T19:15:30Z"
  lastUpdateTime: "2020-01-30T19:15:30Z"
  message: Trial is created
  reason: TrialCreated
  status: "True"
  type: Created
• lastTransitionTime: "2020-01-30T19:17:48Z"
  lastUpdateTime: "2020-01-30T19:17:48Z"
  message: Trial is running
  reason: TrialRunning
  status: "False"
  type: Running
• lastTransitionTime: "2020-01-30T19:17:48Z"
  lastUpdateTime: "2020-01-30T19:17:48Z"
  message: Metrics are not available
  reason: MetricsUnavailable
  status: "False"
  type: Succeeded
  startTime: "2020-01-30T19:15:30Z"

[martham93]

I tried editing the yaml file to specifically look for the StdOut and I'm still facing the same error

apiVersion: "kubeflow.org/v1alpha3"
kind: Experiment
metadata:
  namespace: kubeflow
  labels:
    controller-tools.k8s.io: "1.0"
  name: bayesianoptimization-example
spec:
  objective:
    type: maximize
    goal: 0.99
    objectiveMetricName: Validation-accuracy
    additionalMetricNames:
      - Train-accuracy
  metricsCollectorSpec:
    collector:
      kind: StdOut
  algorithm:
    algorithmName: bayesianoptimization
    algorithmSettings:
      - name: "random_state"
        value: "10"
  parallelTrialCount: 3
  maxTrialCount: 12
  maxFailedTrialCount: 3
  parameters:
    - name: --lr
      parameterType: double
      feasibleSpace:
        min: "0.01"
        max: "0.03"
    - name: --num-layers
      parameterType: int
      feasibleSpace:
        min: "2"
        max: "5"
    - name: --optimizer
      parameterType: categorical
      feasibleSpace:
        list:
        - sgd
        - adam
        - ftrl
  trialTemplate:
    goTemplate:
        rawTemplate: |-
          apiVersion: batch/v1
          kind: Job
          metadata:
            name: {{.Trial}}
            namespace: {{.NameSpace}}
          spec:
            template:
              spec:
                containers:
                - name: {{.Trial}}
                  image: docker.io/kubeflowkatib/mxnet-mnist
                  command:
                  - "python3"
                  - "/opt/mxnet-mnist/mnist.py"
                  - "--batch-size=64"
                  {{- with .HyperParameters}}
                  {{- range .}}
                  - "{{.Name}}={{.Value}}"
                  {{- end}}
                  {{- end}}
                restartPolicy: Never

[timothyjlaurent]

I'm having this problem as well.

[timothyjlaurent]

For me, adding the:

  metricsCollectorSpec:
    collector:
      kind: StdOut

to a manifest based on the pytorchjob-example.yaml allowed the Experiment to finish.

[fedeesku1]

That didn't work for me. The description of the trial is still:
Message: Trial is running
Reason: TrialRunning
Status: False
Type: Running
Last Transition Time: 2020-02-03T20:33:26Z
Last Update Time: 2020-02-03T20:33:26Z
Message: Metrics are not available
Reason: MetricsUnavailable
but the pod associated with the trials is:
State: Terminated
Reason: Completed
Exit Code: 0
Started: Mon, 03 Feb 2020 15:31:47 -0500
Finished: Mon, 03 Feb 2020 15:33:25 -0500

[andreyvelich]

Katib controller should add StdOut Metrics collector if you didn't specify it in the Experiment yaml file.
As I said above, can you try to deploy Katib using the latest manifests folder and run Experiments again?

[timothyjlaurent]

@andreyvelich

Katib controller should add StdOut Metrics collector if you didn't specify it in the Experiment yaml file.

I very much doubt this is the case as all the example yaml Experiments were broken for me until I added

  metricsCollectorSpec:
    collector:
      kind: StdOut

You should consider fixing the issue or updating the example manifests.

[gaocegege]

Did the namespace has label katib-metricscollector-injection: enabled?

It seems that the metrics collector is not injected successfully.

/cc @hougangliu @johnugeorge

[johnugeorge]

@richardsliu faced this same problem

[andreyvelich]

Can you try to do this:

1. kubectl edit namespace kubeflow
2. Delete label control-plane=kubeflow
3. Save changes
4. Run examples again

It maybe fixes the problem with MetricsCollector.
You don't need to make any changes to Katib examples.

[martham93]

@andreyvelich thanks your suggestion above worked

[jimmy-hawkfish]

Hi I'm also facing this issue. I'm not using the control-plane label and I have katib-metricscollector-injection enabled.

[issue-label-bot[bot]]

Issue-Label Bot is automatically applying the labels:

| Label | Probability |
| ------------- | ------------- |
| area/katib | 0.90 |

Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback!
Links: app homepage, dashboard and code for this bot.

[andreyvelich]

@jimmy-hawkfish How did you install Katib, which version are you using?

[jimmy-hawkfish]

@andreyvelich I'm on v0.9.0 and I deployed using scripts/v1alpha3/deploy.sh

[jimmy-hawkfish]

The experiment lists the trial as pending, the trial reports that Metrics are unavailable:

Events:
  Type     Reason              Age                From              Message
  ----     ------              ----               ----              -------
  Normal   JobCreated          16m                trial-controller  Job model-classifier-qpkwv-1660029856-kxf4zvkk has been created
  Normal   JobRunning          16m (x3 over 16m)  trial-controller  Job model-classifier-qpkwv-1660029856-kxf4zvkk is running:
  Warning  MetricsUnavailable  16m (x2 over 16m)  trial-controller  Metrics are not available for Job model-classifier-qpkwv-1660029856-kxf4zvkk

However, in departure from the other issues I've been reading, metrics-logger-and-collector for the pod are seeing the metrics.

I0821 15:47:35.344966      27 main.go:127] Metrics reported. :
metric_logs:<time_stamp:"0001-01-01T00:00:00Z" metric:<name:"accuracy" value:"0.6064179923203511" > > metric_logs:<time_stamp:"0001-01-01T00:00:00Z" metric:<name:"nll" value:"0.6643724676334798" > > metric_logs:<time_stamp:"0001-01-01T00:00:00Z" metric:<name:"brier" value:"0.4713531041447152" > > metric_logs:<time_stamp:"0001-01-01T00:00:00Z" metric:<name:"auc_inperson" value:"0.5847704939641005" > > metric_logs:<time_stamp:"0001-01-01T00:00:00Z" metric:<name:"prec10perc_inperson" value:"0.6968449931412893" > > metric_logs:<time_stamp:"0001-01-01T00:00:00Z" metric:<name:"auc_mail" value:"0.5847704939641005" > > metric_logs:<time_stamp:"0001-01-01T00:00:00Z" metric:<name:"prec10perc_mail" value:"0.5189757658893461" > > 

[andreyvelich]

@jimmy-hawkfish Can you show the experiment yaml that you have submitted, please?
Also show one of the Trial status: kubectl get trial <trial-name> -n <namespace> -o yaml

[jimmy-hawkfish]

sorry @andreyvelich I misspecified the objective metric name and that was what was causing the issue. Thanks for the help and the rapid responses.

[andreyvelich]

I close this issue, feel free to re-open if you have any other questions.