Katib: Katib stuck on the first trial and the experiment keeps running without creating the next trial

Created on 12 Jan 2021  路  14Comments  路  Source: kubeflow/katib

/kind bug

What steps did you take and what happened:
[A clear and concise description of what the bug is.]

I've created an experiment using the script below. Experiment, trials, suggestion, pods are running fine.

However, the katib only created one trial and stuck on it without trying a second trial with new parameter from suggestion. there was no new suggestion after first trial.

Here are the yaml file and status of them while running.

  1. experiment yaml - using gpu , tfjob
  2. experiment status
  3. suggestions status

  4. model python script for tfjob container

  5. experiment yaml

apiVersion: "kubeflow.org/v1beta1"
kind: Experiment
metadata:
namespace: kubeflow
name: tfjob-example
spec:
parallelTrialCount: 1
maxTrialCount: 10
maxFailedTrialCount: 10
objective:
type: minimize
goal: 0
objectiveMetricName: val_loss
algorithm:
algorithmName: random
metricsCollectorSpec:
source:
fileSystemPath:
path: /train
kind: Directory
collector:
kind: TensorFlowEvent
parameters:
- name: batch_size
parameterType: int
feasibleSpace:
min: "16"
max: "20"
trialTemplate:
primaryContainerName: tensorflow
trialParameters:
- name: batchSize
description: Batch Size
reference: batch_size
trialSpec:
apiVersion: "kubeflow.org/v1"
kind: TFJob
spec:
tfReplicaSpecs:
Worker:
replicas: 1
restartPolicy: OnFailure
template:
spec:
spec:
containers:
- command:
- python
- TFTemplate.py
- --file_name=model_for_katib2
- --work_dir=/kn/data
- --train=$(DATADIR)
- --model_name=kn-cnn
- --epochs=5
- --log_dir=/kn/data
- --batch_size=${trialParameters.batchSize}
env:
- name: DATADIR
valueFrom:
configMapKeyRef:
name: configmap
key: datadir
image: seungkyulee/kn_tf_gpu_no_template:2.0
name: tensorflow
volumeMounts:
- mountPath: /kn/data
name: volume
workingDir: /kn/data
resources:
limits:
nvidia.com/gpu: 1
restartPolicy: Never
volumes:
- name: volume
persistentVolumeClaim:
claimName: tfpvc

  1. experiment status

ubuntu@ip-172-16-1-204:~/katib$ kubectl describe experiment tfjob-example -n kubeflow
Name: tfjob-example
Namespace: kubeflow
Labels:
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"kubeflow.org/v1beta1","kind":"Experiment","metadata":{"annotations":{},"name":"tfjob-example","namespace":"kubeflow"},"spec...
API Version: kubeflow.org/v1beta1
Kind: Experiment
Metadata:
Creation Timestamp: 2021-01-12T06:14:29Z
Finalizers:
update-prometheus-metrics
Generation: 1
Resource Version: 1048867
Self Link: /apis/kubeflow.org/v1beta1/namespaces/kubeflow/experiments/tfjob-example
UID: e10cadae-7b6b-49a9-8a84-a4aba13d26df
Spec:
Algorithm:
Algorithm Name: random
Max Failed Trial Count: 10
Max Trial Count: 10
Metrics Collector Spec:
Collector:
Kind: TensorFlowEvent
Source:
File System Path:
Kind: Directory
Path: /train
Objective:
Goal: 0
Objective Metric Name: val_loss
Type: minimize
Parallel Trial Count: 1
Parameters:
Feasible Space:
Max: 20
Min: 16
Name: batch_size
Parameter Type: int
Trial Template:
Primary Container Name: tensorflow
Trial Parameters:
Description: Batch Size
Name: batchSize
Reference: batch_size
Trial Spec:
API Version: kubeflow.org/v1
Kind: TFJob
Spec:
Tf Replica Specs:
Worker:
Replicas: 1
Restart Policy: OnFailure
Template:
Spec:
Containers:
Command:
python
TFTemplate.py
--file_name=model_for_katib2
--work_dir=/kn/data
--train=$(DATADIR)
--model_name=kn-cnn
--epochs=5
--log_dir=/kn/data
--batch_size=${trialParameters.batchSize}
Env:
Name: DATADIR
Value From:
Config Map Key Ref:
Key: datadir
Name: configmap
Image: seungkyulee/kn_tf_gpu_no_template:2.0
Name: tensorflow
Resources:
Limits:
nvidia.com/gpu: 1
Volume Mounts:
Mount Path: /kn/data
Name: volume
Working Dir: /kn/data
Restart Policy: Never
Volumes:
Name: volume
Persistent Volume Claim:
Claim Name: tfpvc
Status:
Conditions:
Last Transition Time: 2021-01-12T06:14:29Z
Last Update Time: 2021-01-12T06:14:29Z
Message: Experiment is created
Reason: ExperimentCreated
Status: True
Type: Created
Last Transition Time: 2021-01-12T06:14:50Z
Last Update Time: 2021-01-12T06:14:50Z
Message: Experiment is running
Reason: ExperimentRunning
Status: True
Type: Running
Current Optimal Trial:
Best Trial Name:
Observation:
Metrics:
Parameter Assignments:
Running Trial List:
tfjob-example-8v9gt8v8
Start Time: 2021-01-12T06:14:29Z
Trials: 1
Trials Running: 1
Events:

  1. suggestions status

ubuntu@ip-172-16-1-204:~/katib$ kubectl describe suggestions tfjob-example -n kubeflow

Name: tfjob-example
Namespace: kubeflow
Labels:
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"kubeflow.org/v1beta1","kind":"Experiment","metadata":{"annotations":{},"name":"tfjob-example","namespace":"kubeflow"},"spec...
API Version: kubeflow.org/v1beta1
Kind: Suggestion
Metadata:
Creation Timestamp: 2021-01-12T06:14:29Z
Generation: 1
Owner References:
API Version: kubeflow.org/v1beta1
Block Owner Deletion: true
Controller: true
Kind: Experiment
Name: tfjob-example
UID: e10cadae-7b6b-49a9-8a84-a4aba13d26df
Resource Version: 1048857
Self Link: /apis/kubeflow.org/v1beta1/namespaces/kubeflow/suggestions/tfjob-example
UID: 029e0905-7af3-43a5-8976-bbaafd83783a
Spec:
Algorithm:
Algorithm Name: random
Requests: 1
Status:
Conditions:
Last Transition Time: 2021-01-12T06:14:29Z
Last Update Time: 2021-01-12T06:14:29Z
Message: Suggestion is created
Reason: SuggestionCreated
Status: True
Type: Created
Last Transition Time: 2021-01-12T06:14:49Z
Last Update Time: 2021-01-12T06:14:49Z
Message: Deployment is ready
Reason: DeploymentReady
Status: True
Type: DeploymentReady
Last Transition Time: 2021-01-12T06:14:49Z
Last Update Time: 2021-01-12T06:14:49Z
Message: Suggestion is running
Reason: SuggestionRunning
Status: True
Type: Running
Start Time: 2021-01-12T06:14:29Z
Suggestion Count: 1
Suggestions:
Name: tfjob-example-8v9gt8v8
Parameter Assignments:
Name: batch_size
Value: 16
Events:

What did you expect to happen:

I wanted the katib create new trial with new parameter which here in example batch size.

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

also does anyone know how to find the directory and what is does 

source:
  fileSystemPath:
    path: /train
    kind: Directory

status of first trial

ubuntu@ip-172-16-1-204:~/katib$ kubectl logs tfjob-example-8v9gt8v8-worker-0 -n kubeflow -c tensorflow

2021-01-12 06:14:54.274112: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2021-01-12 06:14:56.530644: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-01-12 06:14:56.531765: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2021-01-12 06:14:56.556211: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-01-12 06:14:56.557239: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:00:1e.0 name: Tesla V100-SXM2-16GB computeCapability: 7.0
coreClock: 1.53GHz coreCount: 80 deviceMemorySize: 15.78GiB deviceMemoryBandwidth: 836.37GiB/s
2021-01-12 06:14:56.557279: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2021-01-12 06:14:56.560870: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2021-01-12 06:14:56.560925: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2021-01-12 06:14:56.562425: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2021-01-12 06:14:56.562733: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2021-01-12 06:14:56.566658: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2021-01-12 06:14:56.567536: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2021-01-12 06:14:56.567745: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2021-01-12 06:14:56.567876: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-01-12 06:14:56.568916: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-01-12 06:14:56.569861: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2021-01-12 06:14:56.569910: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2021-01-12 06:14:57.499897: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-01-12 06:14:57.499958: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267] 0
2021-01-12 06:14:57.499971: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0: N
2021-01-12 06:14:57.500311: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-01-12 06:14:57.501411: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-01-12 06:14:57.502432: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-01-12 06:14:57.503399: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14760 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:1e.0, compute capability: 7.0)
2021-01-12 06:14:57.532771: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-01-12 06:14:57.532935: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-01-12 06:14:57.533887: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:00:1e.0 name: Tesla V100-SXM2-16GB computeCapability: 7.0
coreClock: 1.53GHz coreCount: 80 deviceMemorySize: 15.78GiB deviceMemoryBandwidth: 836.37GiB/s
2021-01-12 06:14:57.533946: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2021-01-12 06:14:57.534012: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2021-01-12 06:14:57.534045: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2021-01-12 06:14:57.534073: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2021-01-12 06:14:57.534093: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2021-01-12 06:14:57.534112: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2021-01-12 06:14:57.534150: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2021-01-12 06:14:57.534188: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2021-01-12 06:14:57.534291: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-01-12 06:14:57.535313: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-01-12 06:14:57.536244: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2021-01-12 06:14:57.536523: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-01-12 06:14:57.536640: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-01-12 06:14:57.537636: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:00:1e.0 name: Tesla V100-SXM2-16GB computeCapability: 7.0
coreClock: 1.53GHz coreCount: 80 deviceMemorySize: 15.78GiB deviceMemoryBandwidth: 836.37GiB/s
2021-01-12 06:14:57.537673: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2021-01-12 06:14:57.537696: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2021-01-12 06:14:57.537729: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2021-01-12 06:14:57.537749: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2021-01-12 06:14:57.537769: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2021-01-12 06:14:57.537794: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2021-01-12 06:14:57.537818: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2021-01-12 06:14:57.537838: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2021-01-12 06:14:57.537916: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-01-12 06:14:57.538900: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-01-12 06:14:57.539790: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2021-01-12 06:14:57.539828: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-01-12 06:14:57.539838: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267] 0
2021-01-12 06:14:57.539854: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0: N
2021-01-12 06:14:57.539958: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-01-12 06:14:57.540987: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-01-12 06:14:57.541937: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14760 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:1e.0, compute capability: 7.0)
2021-01-12 06:14:57.924484: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2021-01-12 06:14:57.925171: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2300070000 Hz
2021-01-12 06:14:58.481369: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2021-01-12 06:14:58.937601: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2021-01-12 06:14:58.940892: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8

Parameter list

-------------------------------- (Known)
Namespace(activation='relu', batch_size=16, epochs=5, file_name='model_for_katib2', learning_rate=0, log_dir='/kn/data', model_dir='/kn/data/models', model_ver=0.3, train='/kn/data/dataset/region1', work_dir='/kn/data')
-------------------------------- (Unknown)

['--model_name=kn-cnn']

Found 70 files belonging to 4 classes.
Using 4 files for training.
Found 70 files belonging to 4 classes.
Using 66 files for validation.
Epoch 1/5
1/1 [==============================] - 4s 4s/step - loss: 1.3791 - accuracy: 0.2500 - val_loss: 0.8957 - val_accuracy: 0.8939
Epoch 2/5
1/1 [==============================] - 0s 40ms/step - loss: 0.8655 - accuracy: 1.0000 - val_loss: 0.3944 - val_accuracy: 0.8939
Epoch 3/5
1/1 [==============================] - 0s 40ms/step - loss: 0.3163 - accuracy: 1.0000 - val_loss: 0.2012 - val_accuracy: 0.9091
Epoch 4/5
1/1 [==============================] - 0s 66ms/step - loss: 0.0639 - accuracy: 1.0000 - val_loss: 0.1549 - val_accuracy: 0.9545
Epoch 5/5
1/1 [==============================] - 0s 40ms/step - loss: 0.0069 - accuracy: 1.0000 - val_loss: 0.1389 - val_accuracy: 0.9545
2021-01-12 06:15:01.874995: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-01-12 06:15:01.875337: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-01-12 06:15:01.876341: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:00:1e.0 name: Tesla V100-SXM2-16GB computeCapability: 7.0
coreClock: 1.53GHz coreCount: 80 deviceMemorySize: 15.78GiB deviceMemoryBandwidth: 836.37GiB/s
2021-01-12 06:15:01.876408: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2021-01-12 06:15:01.876458: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2021-01-12 06:15:01.876493: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2021-01-12 06:15:01.876525: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2021-01-12 06:15:01.876557: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2021-01-12 06:15:01.876590: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2021-01-12 06:15:01.876624: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2021-01-12 06:15:01.876658: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2021-01-12 06:15:01.876757: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-01-12 06:15:01.877778: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-01-12 06:15:01.878652: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2021-01-12 06:15:01.878699: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-01-12 06:15:01.878712: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267] 0
2021-01-12 06:15:01.878720: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0: N
2021-01-12 06:15:01.878848: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-01-12 06:15:01.879784: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-01-12 06:15:01.880651: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/device:GPU:0 with 14760 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:1e.0, compute capability: 7.0)
2021-01-12 06:15:02.834707: W tensorflow/python/util/util.cc:348] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.
[name: "/device:CPU:0"
device_type: "CPU"
memory_limit: 268435456
locality {
}
incarnation: 9325939197827833295
, name: "/device:GPU:0"
device_type: "GPU"
memory_limit: 15477595200
locality {
bus_id: 1
links {
}
}
incarnation: 4128588976737718326
physical_device_desc: "device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:1e.0, compute capability: 7.0"
]

Found 70 files belonging to 4 classes.
[[18 0 0 0]
[ 0 20 0 0]
[ 0 0 16 0]
[ 0 3 0 13]]

Save model : /kn/data/models/0.3

Saved cm : /kn/data/models/cm

Environment:

  • Kubeflow version (kfctl version): 1.2
  • AWS ec2 p3xlarge
  • Kubernetes version: (use kubectl version): 1.15
  • OS (e.g. from /etc/os-release): ubuntu 18.04
kinbug
Source
picture MLXQ

All 14 comments

@MLXQ Thank you for creating this!
I think you should add annotation to your Trial template with TFJob to disable Istio sidecar injection.
Follow step 3 of this guide to specify annotation.

andreyvelich picture andreyvelich on 12 Jan 2021

@andreyvelich Thank you for the reply

I've just disabled Istio sidecar Injection by adding the code below following the guide

trialSpec:
  apiVersion: "kubeflow.org/v1"
  kind: TFJob
  spec:
    tfReplicaSpecs:
      Worker:
        replicas: 1
        restartPolicy: OnFailure
        template:
          metadata:
            annotations:
              sidecar.istio.io/inject: "false"
            name: tensorflow
          spec:
            containers

However, um Katib still doesn't create new trials after the first trial
the only thing that has difference is that the first trial doesn't keep running and the status is changed to completed different from before.

ubuntu@ip-172-16-1-204:~/katib$ kubectl get trial -n kubeflow
NAME TYPE STATUS AGE
tfjob-example-trzg4g5n Running True 41m
ubuntu@ip-172-16-1-204:~/katib$ kubectl get suggestions -n kubeflow
NAME TYPE STATUS REQUESTED ASSIGNED AGE
tfjob-example Running True 1 1 41m
ubuntu@ip-172-16-1-204:~/katib$ kubectl get experiment -n kubeflow
NAME TYPE STATUS AGE
tfjob-example Running True 42m
ubuntu@ip-172-16-1-204:~/katib$ kubectl describe experiment tfjob-example -n kubeflow
Status:
Conditions:
Last Transition Time: 2021-01-12T13:22:23Z
Last Update Time: 2021-01-12T13:22:23Z
Message: Experiment is created
Reason: ExperimentCreated
Status: True
Type: Created
Last Transition Time: 2021-01-12T13:22:37Z
Last Update Time: 2021-01-12T13:22:37Z
Message: Experiment is running
Reason: ExperimentRunning
Status: True
Type: Running
Current Optimal Trial:
Best Trial Name:
Observation:
Metrics:
Parameter Assignments:
Running Trial List:
tfjob-example-trzg4g5n
Start Time: 2021-01-12T13:22:23Z
Trials: 1
Trials Running: 1
Events:

MLXQ picture MLXQ on 12 Jan 2021

@MLXQ How do you print metrics in your training container: seungkyulee/kn_tf_gpu_no_template:2.0 ?
Are you using TF summary to save metrics or just print the results in the output ?

andreyvelich picture andreyvelich on 13 Jan 2021

@andreyvelich

ah.. I just print the results in the output from using those three pieces of codes

I should've added tf summary in model script for metric collector to collect tensorflow event through tf summary ..

MLXQ picture MLXQ on 13 Jan 2021

@MLXQ It seems that your metrics are printed in this way: val_loss: 0.8957.
So it is not the default way of extracting metrics (metric-name=metric-value).

Try to change your metrics collector spec in the Experiment like bellow:

metricsCollectorSpec:
    source:
      filter:
        metricsFormat:
          - "([\w|-]+)\s*:\s*((-?\d+)(\.\d+)?)"
    collector:
      kind: StdOut
andreyvelich picture andreyvelich on 13 Jan 2021

I should've added tf summary in model script for metric collector to collect tensorflow event through tf summary ..

It is up to you, you can use StdOut metrics collector or Tensorflow events.

andreyvelich picture andreyvelich on 13 Jan 2021

@andreyvelich

I tried with your advice and the result comes out like below

ubuntu@ip-172-16-1-204:~/katib$ kubectl apply -f katib_tfjob_kn_updated.yaml
error: error parsing katib_tfjob_kn_updated.yaml: error converting YAML to JSON: yaml: line 20: found unknown escape character

MLXQ picture MLXQ on 14 Jan 2021

@MLXQ You are right sorry, your RegEx should look like this in the YAML:

metricsCollectorSpec:
    source:
      filter:
        metricsFormat:
          - "([\\w|-]+)\\s*:\\s*((-?\\d+)(\\.\\d+)?)"
    collector:
      kind: StdOut
andreyvelich picture andreyvelich on 14 Jan 2021

@andreyvelich I have the same problem with random-example.yaml
In the first trials, the pod controlled by the job is completed, the trials do not end and the next trials do not run.
when I check the log of katib-controller, a runtime error: invalid memory address or nil pointer dereference error occurred.
It looks like an error occurs in katib-controller when collecting metrics.

I tested with istio injection false in the options below, but the result was always the same

  • v1alpha3 example on Kubeflow 1.0.2.
  • v1alpha3 on tag v0.10.0
  • v1beta1 on tag v0.10.0

Here's my katib-controller log, can you give me any advice?

{"level":"info","ts":1612111267.7298687,"logger":"entrypoint","msg":"Config:","experiment-suggestion-name":"default","cert-local-filesystem":false,"webhook-port":8443,"metrics-addr":":8080","inject-security-context":false,"enable-grpc-probe-in-suggestion":true}
{"level":"info","ts":1612111267.8145483,"logger":"entrypoint","msg":"Registering Components."}
{"level":"info","ts":1612111267.8152566,"logger":"entrypoint","msg":"Setting up controller"}
{"level":"info","ts":1612111267.8153057,"logger":"experiment-controller","msg":"Using the default suggestion implementation"}
{"level":"info","ts":1612111267.8154366,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"experiment-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1612111267.8156013,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"experiment-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1612111267.8157685,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"experiment-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1612111267.8159266,"logger":"experiment-controller","msg":"Experiment controller created"}
{"level":"info","ts":1612111267.815978,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"suggestion-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1612111267.8160143,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"suggestion-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1612111267.816178,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"suggestion-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1612111267.8162887,"logger":"suggestion-controller","msg":"Suggestion controller created"}
{"level":"info","ts":1612111267.8163753,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"trial-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1612111267.816424,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"trial-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1612111267.8165598,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"trial-controller","source":"kind source: kubeflow.org/v1, Kind=TFJob"}
{"level":"info","ts":1612111267.8166947,"logger":"trial-controller","msg":"Job watch added successfully","CRD Kind":"TFJob"}
{"level":"info","ts":1612111267.8167129,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"trial-controller","source":"kind source: kubeflow.org/v1, Kind=PyTorchJob"}
{"level":"info","ts":1612111267.8168166,"logger":"trial-controller","msg":"Job watch added successfully","CRD Kind":"PyTorchJob"}
{"level":"info","ts":1612111267.816832,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"trial-controller","source":"kind source: batch/v1, Kind=Job"}
{"level":"info","ts":1612111267.8169212,"logger":"trial-controller","msg":"Job watch added successfully","CRD Kind":"Job"}
{"level":"info","ts":1612111267.816935,"logger":"trial-controller","msg":"Trial  controller created"}
{"level":"info","ts":1612111267.81694,"logger":"entrypoint","msg":"Setting up webhooks"}
{"level":"info","ts":1612111267.8170834,"logger":"entrypoint","msg":"Starting the Cmd."}
{"level":"info","ts":1612111267.9176223,"logger":"kubebuilder.webhook","msg":"installing webhook configuration in cluster"}
{"level":"info","ts":1612111267.9176438,"logger":"kubebuilder.controller","msg":"Starting Controller","controller":"suggestion-controller"}
{"level":"info","ts":1612111267.9176843,"logger":"kubebuilder.controller","msg":"Starting Controller","controller":"experiment-controller"}
{"level":"info","ts":1612111267.9177153,"logger":"kubebuilder.controller","msg":"Starting Controller","controller":"trial-controller"}
{"level":"info","ts":1612111268.017906,"logger":"kubebuilder.controller","msg":"Starting workers","controller":"suggestion-controller","worker count":1}
{"level":"info","ts":1612111268.018209,"logger":"kubebuilder.controller","msg":"Starting workers","controller":"experiment-controller","worker count":1}
{"level":"info","ts":1612111268.0182648,"logger":"kubebuilder.controller","msg":"Starting workers","controller":"trial-controller","worker count":1}
{"level":"info","ts":1612111268.018859,"logger":"kubebuilder.admission.cert.writer","msg":"cert is invalid or expiring, regenerating a new one"}
{"level":"error","ts":1612111448.460399,"logger":"experiment-controller","msg":"Update experiment instance status error","Experiment":"kubeflow/random-example","error":"Operation cannot be fulfilled on experiments.kubeflow.org \"random-example\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/kubeflow/katib/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/kubeflow/katib/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/kubeflow/katib/pkg/controller.v1alpha3/experiment.(*ReconcileExperiment).Reconcile\n\t/go/src/github.com/kubeflow/katib/pkg/controller.v1alpha3/experiment/experiment_controller.go:246\ngithub.com/kubeflow/katib/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/kubeflow/katib/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:215\ngithub.com/kubeflow/katib/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/kubeflow/katib/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/kubeflow/katib/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/kubeflow/katib/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/kubeflow/katib/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/kubeflow/katib/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/kubeflow/katib/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/kubeflow/katib/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
{"level":"error","ts":1612111448.4605675,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"experiment-controller","request":"kubeflow/random-example","error":"Operation cannot be fulfilled on experiments.kubeflow.org \"random-example\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/kubeflow/katib/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/kubeflow/katib/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/kubeflow/katib/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/kubeflow/katib/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\ngithub.com/kubeflow/katib/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/kubeflow/katib/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/kubeflow/katib/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/kubeflow/katib/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/kubeflow/katib/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/kubeflow/katib/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/kubeflow/katib/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/kubeflow/katib/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
{"level":"info","ts":1612111449.461097,"logger":"experiment-controller","msg":"Statistics","Experiment":"kubeflow/random-example","requiredActiveCount":3,"parallelCount":3,"activeCount":0,"completedCount":0}
{"level":"info","ts":1612111449.461152,"logger":"experiment-controller","msg":"CreateTrials","Experiment":"kubeflow/random-example","addCount":3}
{"level":"info","ts":1612111449.461205,"logger":"experiment-suggestion-client","msg":"Creating Suggestion","experiment":"kubeflow/random-example","namespace":"kubeflow","name":"random-example","requests":3}
{"level":"info","ts":1612111449.4612596,"logger":"experiment-suggestion-client","msg":"Creating Suggestion","experiment":"kubeflow/random-example","namespace":"kubeflow","name":"random-example"}
{"level":"info","ts":1612111449.468753,"logger":"experiment-controller","msg":"GetOrCreateSuggestion","Experiment":"kubeflow/random-example","Instance name":"random-example","suggestionRequestsCount":3}
{"level":"info","ts":1612111449.469002,"logger":"experiment-controller","msg":"Statistics","Experiment":"kubeflow/random-example","requiredActiveCount":3,"parallelCount":3,"activeCount":0,"completedCount":0}
{"level":"info","ts":1612111449.469037,"logger":"experiment-controller","msg":"CreateTrials","Experiment":"kubeflow/random-example","addCount":3}
{"level":"info","ts":1612111449.469073,"logger":"experiment-controller","msg":"GetOrCreateSuggestion","Experiment":"kubeflow/random-example","Instance name":"random-example","suggestionRequestsCount":3}
{"level":"info","ts":1612111449.4746525,"logger":"suggestion-controller","msg":"Creating Service","namespace":"kubeflow","name":"random-example-random"}
{"level":"info","ts":1612111449.4747746,"logger":"experiment-controller","msg":"Statistics","Experiment":"kubeflow/random-example","requiredActiveCount":3,"parallelCount":3,"activeCount":0,"completedCount":0}
{"level":"info","ts":1612111449.4748209,"logger":"experiment-controller","msg":"CreateTrials","Experiment":"kubeflow/random-example","addCount":3}
{"level":"info","ts":1612111449.4748662,"logger":"experiment-controller","msg":"GetOrCreateSuggestion","Experiment":"kubeflow/random-example","Instance name":"random-example","suggestionRequestsCount":3}
{"level":"info","ts":1612111449.5883512,"logger":"suggestion-controller","msg":"Creating Deployment","namespace":"kubeflow","name":"random-example-random"}
{"level":"info","ts":1612111449.6025314,"logger":"experiment-controller","msg":"Statistics","Experiment":"kubeflow/random-example","requiredActiveCount":3,"parallelCount":3,"activeCount":0,"completedCount":0}
{"level":"info","ts":1612111449.6025715,"logger":"experiment-controller","msg":"CreateTrials","Experiment":"kubeflow/random-example","addCount":3}
{"level":"info","ts":1612111449.602642,"logger":"experiment-controller","msg":"GetOrCreateSuggestion","Experiment":"kubeflow/random-example","Instance name":"random-example","suggestionRequestsCount":3}
{"level":"error","ts":1612111449.6223602,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"suggestion-controller","request":"kubeflow/random-example","error":"Operation cannot be fulfilled on suggestions.kubeflow.org \"random-example\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/kubeflow/katib/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/kubeflow/katib/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/kubeflow/katib/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/kubeflow/katib/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\ngithub.com/kubeflow/katib/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/kubeflow/katib/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/kubeflow/katib/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/kubeflow/katib/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/kubeflow/katib/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/kubeflow/katib/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/kubeflow/katib/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/kubeflow/katib/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
{"level":"info","ts":1612111468.9458897,"logger":"suggestion-client","msg":"Algorithm settings validated","Suggestion":"kubeflow/random-example"}
{"level":"info","ts":1612111468.9459798,"logger":"suggestion-controller","msg":"Sync assignments","Suggestion":"kubeflow/random-example","suggestions":3}
{"level":"info","ts":1612111468.963,"logger":"suggestion-client","msg":"Getting suggestions","Suggestion":"kubeflow/random-example","endpoint":"random-example-random.kubeflow:6789","response":"parameter_assignments:<assignments:<name:\"--lr\" value:\"0.013499313683027354\" > assignments:<name:\"--num-layers\" value:\"3\" > assignments:<name:\"--optimizer\" value:\"sgd\" > > parameter_assignments:<assignments:<name:\"--lr\" value:\"0.023411708191629\" > assignments:<name:\"--num-layers\" value:\"3\" > assignments:<name:\"--optimizer\" value:\"adam\" > > parameter_assignments:<assignments:<name:\"--lr\" value:\"0.02139956982677772\" > assignments:<name:\"--num-layers\" value:\"2\" > assignments:<name:\"--optimizer\" value:\"adam\" > > ","request":"experiment:<name:\"random-example\" spec:<parameter_specs:<parameters:<name:\"--lr\" parameter_type:DOUBLE feasible_space:<max:\"0.03\" min:\"0.01\" > > parameters:<name:\"--num-layers\" parameter_type:INT feasible_space:<max:\"5\" min:\"2\" > > parameters:<name:\"--optimizer\" parameter_type:CATEGORICAL feasible_space:<list:\"sgd\" list:\"adam\" list:\"ftrl\" > > > objective:<type:MAXIMIZE goal:0.99 objective_metric_name:\"Validation-accuracy\" > algorithm:<algorithm_name:\"random\" > parallel_trial_count:3 max_trial_count:12 > > request_number:3 "}
{"level":"info","ts":1612111468.9703765,"logger":"experiment-controller","msg":"Statistics","Experiment":"kubeflow/random-example","requiredActiveCount":3,"parallelCount":3,"activeCount":0,"completedCount":0}
{"level":"info","ts":1612111468.9704475,"logger":"experiment-controller","msg":"CreateTrials","Experiment":"kubeflow/random-example","addCount":3}
{"level":"info","ts":1612111468.9704947,"logger":"experiment-controller","msg":"GetOrCreateSuggestion","Experiment":"kubeflow/random-example","Instance name":"random-example","suggestionRequestsCount":3}
{"level":"info","ts":1612111468.9707255,"logger":"suggestion-controller","msg":"Sync assignments","Suggestion":"kubeflow/random-example","suggestions":3}
{"level":"error","ts":1612111469.0061278,"logger":"experiment-controller","msg":"Update experiment instance status error","Experiment":"kubeflow/random-example","error":"Operation cannot be fulfilled on experiments.kubeflow.org \"random-example\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/kubeflow/katib/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/kubeflow/katib/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/kubeflow/katib/pkg/controller.v1alpha3/experiment.(*ReconcileExperiment).Reconcile\n\t/go/src/github.com/kubeflow/katib/pkg/controller.v1alpha3/experiment/experiment_controller.go:246\ngithub.com/kubeflow/katib/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/kubeflow/katib/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:215\ngithub.com/kubeflow/katib/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/kubeflow/katib/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/kubeflow/katib/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/kubeflow/katib/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/kubeflow/katib/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/kubeflow/katib/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/kubeflow/katib/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/kubeflow/katib/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
{"level":"error","ts":1612111469.0062542,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"experiment-controller","request":"kubeflow/random-example","error":"Operation cannot be fulfilled on experiments.kubeflow.org \"random-example\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/kubeflow/katib/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/kubeflow/katib/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/kubeflow/katib/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/kubeflow/katib/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\ngithub.com/kubeflow/katib/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/kubeflow/katib/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/kubeflow/katib/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/kubeflow/katib/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/kubeflow/katib/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/kubeflow/katib/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/kubeflow/katib/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/kubeflow/katib/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
{"level":"info","ts":1612111469.0159984,"logger":"trial-controller","msg":"Creating Job","Trial":"kubeflow/random-example-dwkj8mwc","kind":"Job","name":"random-example-dwkj8mwc"}
{"level":"info","ts":1612111469.019443,"logger":"provider-job","msg":"NestedFieldCopy","err":"status cannot be found in job"}
{"level":"info","ts":1612111469.0318775,"logger":"trial-controller","msg":"Creating Job","Trial":"kubeflow/random-example-r9rqwlfg","kind":"Job","name":"random-example-r9rqwlfg"}
{"level":"info","ts":1612111469.0350776,"logger":"provider-job","msg":"NestedFieldCopy","err":"status cannot be found in job"}
{"level":"info","ts":1612111469.052217,"logger":"trial-controller","msg":"Creating Job","Trial":"kubeflow/random-example-ftvhbfx5","kind":"Job","name":"random-example-ftvhbfx5"}
{"level":"info","ts":1612111469.0550926,"logger":"provider-job","msg":"NestedFieldCopy","err":"status cannot be found in job"}
2021/01/31 16:44:29 http2: panic serving 10.244.0.1:53978: runtime error: invalid memory address or nil pointer dereference
goroutine 838 [running]:
net/http.(*http2serverConn).runHandler.func1(0xc00013ac28, 0xc001f83f8e, 0xc000481b00)
    /usr/local/go/src/net/http/h2_bundle.go:5711 +0x16f
panic(0x14e8d00, 0x21fe5b0)
    /usr/local/go/src/runtime/panic.go:969 +0x175
github.com/kubeflow/katib/pkg/webhook/v1alpha3/pod.(*sidecarInjector).MutationRequired(0xc000a01680, 0xc0004ce380, 0xc001c32a68, 0x8, 0x0, 0x0, 0x10)
    /go/src/github.com/kubeflow/katib/pkg/webhook/v1alpha3/pod/inject_webhook.go:131 +0x12b
github.com/kubeflow/katib/pkg/webhook/v1alpha3/pod.(*sidecarInjector).Handle(0xc000a01680, 0x18ac540, 0xc000136010, 0xc001c4a580, 0xc0009afa28, 0xa3e905, 0x1872960, 0xc001c39290)
    /go/src/github.com/kubeflow/katib/pkg/webhook/v1alpha3/pod/inject_webhook.go:71 +0x1ba
github.com/kubeflow/katib/vendor/sigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).handleMutating(0xc00084ef80, 0x18ac540, 0xc000136010, 0xc001c4a580, 0x3, 0xc00013fc00, 0xc00013fc00, 0x0)
    /go/src/github.com/kubeflow/katib/vendor/sigs.k8s.io/controller-runtime/pkg/webhook/admission/webhook.go:133 +0xd8
github.com/kubeflow/katib/vendor/sigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).Handle(0xc00084ef80, 0x18ac540, 0xc000136010, 0xc001c4a580, 0x0, 0x188fd40, 0xc001c39260, 0x188fd40)
    /go/src/github.com/kubeflow/katib/vendor/sigs.k8s.io/controller-runtime/pkg/webhook/admission/webhook.go:120 +0x1fa
github.com/kubeflow/katib/vendor/sigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).ServeHTTP(0xc00084ef80, 0x18a5a00, 0xc00013ac28, 0xc00014ae00)
    /go/src/github.com/kubeflow/katib/vendor/sigs.k8s.io/controller-runtime/pkg/webhook/admission/http.go:93 +0x9f5
net/http.(*ServeMux).ServeHTTP(0xc000d1f800, 0x18a5a00, 0xc00013ac28, 0xc00014ae00)
    /usr/local/go/src/net/http/server.go:2417 +0x1ad
net/http.serverHandler.ServeHTTP(0xc0001a4620, 0x18a5a00, 0xc00013ac28, 0xc00014ae00)
    /usr/local/go/src/net/http/server.go:2843 +0xa3
net/http.initALPNRequest.ServeHTTP(0x18ac5c0, 0xc000c85170, 0xc0003ee380, 0xc0001a4620, 0x18a5a00, 0xc00013ac28, 0xc00014ae00)
    /usr/local/go/src/net/http/server.go:3415 +0x8d
net/http.(*http2serverConn).runHandler(0xc000481b00, 0xc00013ac28, 0xc00014ae00, 0xc001d84320)
    /usr/local/go/src/net/http/h2_bundle.go:5718 +0x8b
created by net/http.(*http2serverConn).processHeaders
    /usr/local/go/src/net/http/h2_bundle.go:5452 +0x505
2021/01/31 16:44:29 http2: panic serving 10.244.0.1:53978: runtime error: invalid memory address or nil pointer dereference
goroutine 832 [running]:
net/http.(*http2serverConn).runHandler.func1(0xc0003537c8, 0xc0011cdf8e, 0xc000481b00)
    /usr/local/go/src/net/http/h2_bundle.go:5711 +0x16f
panic(0x14e8d00, 0x21fe5b0)
    /usr/local/go/src/runtime/panic.go:969 +0x175
github.com/kubeflow/katib/pkg/webhook/v1alpha3/pod.(*sidecarInjector).MutationRequired(0xc000a01680, 0xc00057c000, 0xc001b6a488, 0x8, 0x0, 0x0, 0x10)
    /go/src/github.com/kubeflow/katib/pkg/webhook/v1alpha3/pod/inject_webhook.go:131 +0x12b
github.com/kubeflow/katib/pkg/webhook/v1alpha3/pod.(*sidecarInjector).Handle(0xc000a01680, 0x18ac540, 0xc000136010, 0xc00198de40, 0xc000ef9a28, 0xa3e905, 0x1872960, 0xc001b68720)
    /go/src/github.com/kubeflow/katib/pkg/webhook/v1alpha3/pod/inject_webhook.go:71 +0x1ba
github.com/kubeflow/katib/vendor/sigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).handleMutating(0xc00084ef80, 0x18ac540, 0xc000136010, 0xc00198de40, 0x3, 0xc00013fc00, 0xc00013fc00, 0x0)
    /go/src/github.com/kubeflow/katib/vendor/sigs.k8s.io/controller-runtime/pkg/webhook/admission/webhook.go:133 +0xd8
github.com/kubeflow/katib/vendor/sigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).Handle(0xc00084ef80, 0x18ac540, 0xc000136010, 0xc00198de40, 0x0, 0x188fd40, 0xc001b686f0, 0x188fd40)
    /go/src/github.com/kubeflow/katib/vendor/sigs.k8s.io/controller-runtime/pkg/webhook/admission/webhook.go:120 +0x1fa
github.com/kubeflow/katib/vendor/sigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).ServeHTTP(0xc00084ef80, 0x18a5a00, 0xc0003537c8, 0xc000049500)
    /go/src/github.com/kubeflow/katib/vendor/sigs.k8s.io/controller-runtime/pkg/webhook/admission/http.go:93 +0x9f5
net/http.(*ServeMux).ServeHTTP(0xc000d1f800, 0x18a5a00, 0xc0003537c8, 0xc000049500)
    /usr/local/go/src/net/http/server.go:2417 +0x1ad
net/http.serverHandler.ServeHTTP(0xc0001a4620, 0x18a5a00, 0xc0003537c8, 0xc000049500)
    /usr/local/go/src/net/http/server.go:2843 +0xa3
net/http.initALPNRequest.ServeHTTP(0x18ac5c0, 0xc000c85170, 0xc0003ee380, 0xc0001a4620, 0x18a5a00, 0xc0003537c8, 0xc000049500)
    /usr/local/go/src/net/http/server.go:3415 +0x8d
net/http.(*http2serverConn).runHandler(0xc000481b00, 0xc0003537c8, 0xc000049500, 0xc001c4e160)
    /usr/local/go/src/net/http/h2_bundle.go:5718 +0x8b
created by net/http.(*http2serverConn).processHeaders
    /usr/local/go/src/net/http/h2_bundle.go:5452 +0x505
2021/01/31 16:44:29 http2: panic serving 10.244.0.1:53978: runtime error: invalid memory address or nil pointer dereference
goroutine 881 [running]:
net/http.(*http2serverConn).runHandler.func1(0xc0007b2850, 0xc0011cdf8e, 0xc000481b00)
    /usr/local/go/src/net/http/h2_bundle.go:5711 +0x16f
panic(0x14e8d00, 0x21fe5b0)
    /usr/local/go/src/runtime/panic.go:969 +0x175
github.com/kubeflow/katib/pkg/webhook/v1alpha3/pod.(*sidecarInjector).MutationRequired(0xc000a01680, 0xc00057c380, 0xc001b6b318, 0x8, 0x0, 0x0, 0x10)
    /go/src/github.com/kubeflow/katib/pkg/webhook/v1alpha3/pod/inject_webhook.go:131 +0x12b
github.com/kubeflow/katib/pkg/webhook/v1alpha3/pod.(*sidecarInjector).Handle(0xc000a01680, 0x18ac540, 0xc000136010, 0xc001b7c2c0, 0xc000ef9a28, 0xa3e905, 0x1872960, 0xc001b69080)
    /go/src/github.com/kubeflow/katib/pkg/webhook/v1alpha3/pod/inject_webhook.go:71 +0x1ba
github.com/kubeflow/katib/vendor/sigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).handleMutating(0xc00084ef80, 0x18ac540, 0xc000136010, 0xc001b7c2c0, 0x3, 0xc00013fc00, 0xc00013fc00, 0x0)
    /go/src/github.com/kubeflow/katib/vendor/sigs.k8s.io/controller-runtime/pkg/webhook/admission/webhook.go:133 +0xd8
github.com/kubeflow/katib/vendor/sigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).Handle(0xc00084ef80, 0x18ac540, 0xc000136010, 0xc001b7c2c0, 0x0, 0x188fd40, 0xc001b69050, 0x188fd40)
    /go/src/github.com/kubeflow/katib/vendor/sigs.k8s.io/controller-runtime/pkg/webhook/admission/webhook.go:120 +0x1fa
github.com/kubeflow/katib/vendor/sigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).ServeHTTP(0xc00084ef80, 0x18a5a00, 0xc0007b2850, 0xc001bb8900)
    /go/src/github.com/kubeflow/katib/vendor/sigs.k8s.io/controller-runtime/pkg/webhook/admission/http.go:93 +0x9f5
net/http.(*ServeMux).ServeHTTP(0xc000d1f800, 0x18a5a00, 0xc0007b2850, 0xc001bb8900)
    /usr/local/go/src/net/http/server.go:2417 +0x1ad
net/http.serverHandler.ServeHTTP(0xc0001a4620, 0x18a5a00, 0xc0007b2850, 0xc001bb8900)
    /usr/local/go/src/net/http/server.go:2843 +0xa3
net/http.initALPNRequest.ServeHTTP(0x18ac5c0, 0xc000c85170, 0xc0003ee380, 0xc0001a4620, 0x18a5a00, 0xc0007b2850, 0xc001bb8900)
    /usr/local/go/src/net/http/server.go:3415 +0x8d
net/http.(*http2serverConn).runHandler(0xc000481b00, 0xc0007b2850, 0xc001bb8900, 0xc001baf340)
    /usr/local/go/src/net/http/h2_bundle.go:5718 +0x8b
created by net/http.(*http2serverConn).processHeaders
    /usr/local/go/src/net/http/h2_bundle.go:5452 +0x505
rky0930 picture rky0930 on 1 Feb 2021

@rky0930 As I can see from your error you are using v1alpha3 version for the Katib controller.
Can you try to deploy the latest version of the controller and run examples again?

andreyvelich picture andreyvelich on 8 Feb 2021

@andreyvelich Thanks for the reply !
I tested on two versions below, but the results were the same.

The other thing I found is that running the "random-example.yaml" gives a runtime error: invalid memory address or nil pointer dereference error in katib-controller and when I run the "tfjob-example.yaml", There is no errors in katib-controller, it just stuck after execution completed.

I really want to use katib, can i get any advice?

Containers:
  katib-controller:
    Container ID:  docker://1eb8f1d39fbed5b08851469ab060eb1060de803aaa895a8c299ee5d6aec391f3
    Image:         docker.io/kubeflowkatib/katib-controller
    Image ID:      docker-pullable://kubeflowkatib/katib-controller@sha256:b12a3816dbbf2b451ca3327374ab000a611b73c0595385484ee54868e3531904

Containers:
  katib-controller:
    Container ID:  docker://251756b3b1c13bae511c15d9a017f7ba8ecf98b52a98e00683d5946f9050d64d
    Image:         docker.io/kubeflowkatib/katib-controller:v1beta1-91e4996
    Image ID:      docker-pullable://kubeflowkatib/katib-controller@sha256:b12a3816dbbf2b451ca3327374ab000a611b73c0595385484ee54868e3531904

logs

{"level":"info","ts":1612872456.5208004,"logger":"entrypoint","msg":"Config:","experiment-suggestion-name":"default","cert-local-filesystem":false,"webhook-port":8443,"metrics-addr":":8080","inject-security-context":false,"enable-grpc-probe-in-suggestion":true,"trial-resources":[{"Group":"batch","Version":"v1","Kind":"Job"},{"Group":"kubeflow.org","Version":"v1","Kind":"TFJob"},{"Group":"kubeflow.org","Version":"v1","Kind":"PyTorchJob"},{"Group":"kubeflow.org","Version":"v1","Kind":"MPIJob"},{"Group":"tekton.dev","Version":"v1beta1","Kind":"PipelineRun"}]}
{"level":"info","ts":1612872456.5879757,"logger":"entrypoint","msg":"Registering Components."}
{"level":"info","ts":1612872456.5886183,"logger":"entrypoint","msg":"Setting up controller"}
{"level":"info","ts":1612872456.5886624,"logger":"experiment-controller","msg":"Using the default suggestion implementation"}
{"level":"info","ts":1612872456.588854,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"experiment-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1612872456.5890915,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"experiment-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1612872456.5892704,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"experiment-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1612872456.589469,"logger":"experiment-controller","msg":"Experiment controller created"}
{"level":"info","ts":1612872456.5895338,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"suggestion-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1612872456.5895638,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"suggestion-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1612872456.5896988,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"suggestion-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1612872456.5898294,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"suggestion-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1612872456.5899591,"logger":"suggestion-controller","msg":"Suggestion controller created"}
{"level":"info","ts":1612872456.5900636,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"trial-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1612872456.590105,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"trial-controller","source":"kind source: batch/v1, Kind=Job"}
{"level":"info","ts":1612872456.5902536,"logger":"trial-controller","msg":"Job watch added successfully","CRD Group":"batch","CRD Version":"v1","CRD Kind":"Job"}
{"level":"info","ts":1612872456.590295,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"trial-controller","source":"kind source: kubeflow.org/v1, Kind=TFJob"}
{"level":"info","ts":1612872456.5904405,"logger":"trial-controller","msg":"Job watch added successfully","CRD Group":"kubeflow.org","CRD Version":"v1","CRD Kind":"TFJob"}
{"level":"info","ts":1612872456.59048,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"trial-controller","source":"kind source: kubeflow.org/v1, Kind=PyTorchJob"}
{"level":"info","ts":1612872456.590623,"logger":"trial-controller","msg":"Job watch added successfully","CRD Group":"kubeflow.org","CRD Version":"v1","CRD Kind":"PyTorchJob"}
{"level":"info","ts":1612872456.5906422,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"trial-controller","source":"kind source: kubeflow.org/v1, Kind=MPIJob"}
{"level":"error","ts":1612872456.5906947,"logger":"kubebuilder.source","msg":"if kind is a CRD, it should be installed before calling Start","kind":{"Group":"kubeflow.org","Kind":"MPIJob"},"error":"no matches for kind \"MPIJob\" in version \"kubeflow.org/v1\"","stacktrace":"github.com/kubeflow/katib/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/kubeflow/katib/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/kubeflow/katib/vendor/sigs.k8s.io/controller-runtime/pkg/source.(*Kind).Start\n\t/go/src/github.com/kubeflow/katib/vendor/sigs.k8s.io/controller-runtime/pkg/source/source.go:89\ngithub.com/kubeflow/katib/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Watch\n\t/go/src/github.com/kubeflow/katib/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:122\ngithub.com/kubeflow/katib/pkg/controller.v1beta1/trial.add\n\t/go/src/github.com/kubeflow/katib/pkg/controller.v1beta1/trial/trial_controller.go:106\ngithub.com/kubeflow/katib/pkg/controller.v1beta1/trial.Add\n\t/go/src/github.com/kubeflow/katib/pkg/controller.v1beta1/trial/trial_controller.go:65\ngithub.com/kubeflow/katib/pkg/controller%2ev1beta1.AddToManager\n\t/go/src/github.com/kubeflow/katib/pkg/controller.v1beta1/controller.go:28\nmain.main\n\t/go/src/github.com/kubeflow/katib/cmd/katib-controller/v1beta1/main.go:112\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:204"}
{"level":"info","ts":1612872456.590881,"logger":"trial-controller","msg":"Job watch error. CRD might be missing. Please install CRD and restart katib-controller","CRD Group":"kubeflow.org","CRD Version":"v1","CRD Kind":"MPIJob"}
{"level":"info","ts":1612872456.5908961,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"trial-controller","source":"kind source: tekton.dev/v1beta1, Kind=PipelineRun"}
{"level":"error","ts":1612872456.5909352,"logger":"kubebuilder.source","msg":"if kind is a CRD, it should be installed before calling Start","kind":{"Group":"tekton.dev","Kind":"PipelineRun"},"error":"no matches for kind \"PipelineRun\" in version \"tekton.dev/v1beta1\"","stacktrace":"github.com/kubeflow/katib/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/kubeflow/katib/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/kubeflow/katib/vendor/sigs.k8s.io/controller-runtime/pkg/source.(*Kind).Start\n\t/go/src/github.com/kubeflow/katib/vendor/sigs.k8s.io/controller-runtime/pkg/source/source.go:89\ngithub.com/kubeflow/katib/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Watch\n\t/go/src/github.com/kubeflow/katib/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:122\ngithub.com/kubeflow/katib/pkg/controller.v1beta1/trial.add\n\t/go/src/github.com/kubeflow/katib/pkg/controller.v1beta1/trial/trial_controller.go:106\ngithub.com/kubeflow/katib/pkg/controller.v1beta1/trial.Add\n\t/go/src/github.com/kubeflow/katib/pkg/controller.v1beta1/trial/trial_controller.go:65\ngithub.com/kubeflow/katib/pkg/controller%2ev1beta1.AddToManager\n\t/go/src/github.com/kubeflow/katib/pkg/controller.v1beta1/controller.go:28\nmain.main\n\t/go/src/github.com/kubeflow/katib/cmd/katib-controller/v1beta1/main.go:112\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:204"}
{"level":"info","ts":1612872456.5910094,"logger":"trial-controller","msg":"Job watch error. CRD might be missing. Please install CRD and restart katib-controller","CRD Group":"tekton.dev","CRD Version":"v1beta1","CRD Kind":"PipelineRun"}
{"level":"info","ts":1612872456.5910187,"logger":"trial-controller","msg":"Trial controller created"}
{"level":"info","ts":1612872456.5910242,"logger":"entrypoint","msg":"Setting up webhooks"}
{"level":"info","ts":1612872456.5911992,"logger":"entrypoint","msg":"Starting the Cmd."}
{"level":"info","ts":1612872456.6916,"logger":"kubebuilder.webhook","msg":"installing webhook configuration in cluster"}
{"level":"info","ts":1612872456.6916,"logger":"kubebuilder.controller","msg":"Starting Controller","controller":"suggestion-controller"}
{"level":"info","ts":1612872456.6917112,"logger":"kubebuilder.controller","msg":"Starting Controller","controller":"experiment-controller"}
{"level":"info","ts":1612872456.691694,"logger":"kubebuilder.controller","msg":"Starting Controller","controller":"trial-controller"}
{"level":"info","ts":1612872456.7917712,"logger":"kubebuilder.controller","msg":"Starting workers","controller":"suggestion-controller","worker count":1}
{"level":"info","ts":1612872456.7921906,"logger":"kubebuilder.controller","msg":"Starting workers","controller":"trial-controller","worker count":1}
{"level":"info","ts":1612872456.792225,"logger":"kubebuilder.controller","msg":"Starting workers","controller":"experiment-controller","worker count":1}
{"level":"info","ts":1612872456.7929204,"logger":"kubebuilder.admission.cert.writer","msg":"cert is invalid or expiring, regenerating a new one"}
{"level":"info","ts":1612872512.3256345,"logger":"experiment-controller","msg":"Statistics","Experiment":"kubeflow/random-example","requiredActiveCount":3,"parallelCount":3,"activeCount":0,"completedCount":0}
{"level":"info","ts":1612872512.325733,"logger":"experiment-controller","msg":"Reconcile Suggestion","Experiment":"kubeflow/random-example","addCount":3}
{"level":"info","ts":1612872512.3257556,"logger":"experiment-controller","msg":"GetOrCreateSuggestion","Experiment":"kubeflow/random-example","name":"random-example","Suggestion Requests":3}
{"level":"info","ts":1612872512.3258014,"logger":"experiment-suggestion-client","msg":"Creating Suggestion","experiment":"kubeflow/random-example","namespace":"kubeflow","name":"random-example","Suggestion Requests":3}
{"level":"info","ts":1612872512.3325272,"logger":"experiment-suggestion-client","msg":"Suggestion created","experiment":"kubeflow/random-example","namespace":"kubeflow","name":"random-example"}
{"level":"info","ts":1612872512.3327162,"logger":"experiment-controller","msg":"Statistics","Experiment":"kubeflow/random-example","requiredActiveCount":3,"parallelCount":3,"activeCount":0,"completedCount":0}
{"level":"info","ts":1612872512.332739,"logger":"experiment-controller","msg":"Reconcile Suggestion","Experiment":"kubeflow/random-example","addCount":3}
{"level":"info","ts":1612872512.3327515,"logger":"experiment-controller","msg":"GetOrCreateSuggestion","Experiment":"kubeflow/random-example","name":"random-example","Suggestion Requests":3}
{"level":"info","ts":1612872512.3392782,"logger":"suggestion-controller","msg":"Creating Service","Suggestion":"kubeflow/random-example","name":"random-example-random"}
{"level":"info","ts":1612872512.339625,"logger":"experiment-controller","msg":"Statistics","Experiment":"kubeflow/random-example","requiredActiveCount":3,"parallelCount":3,"activeCount":0,"completedCount":0}
{"level":"info","ts":1612872512.3396897,"logger":"experiment-controller","msg":"Reconcile Suggestion","Experiment":"kubeflow/random-example","addCount":3}
{"level":"info","ts":1612872512.3397155,"logger":"experiment-controller","msg":"GetOrCreateSuggestion","Experiment":"kubeflow/random-example","name":"random-example","Suggestion Requests":3}
{"level":"info","ts":1612872512.4579403,"logger":"suggestion-controller","msg":"Creating Deployment","Suggestion":"kubeflow/random-example","name":"random-example-random"}
{"level":"info","ts":1612872512.4768467,"logger":"experiment-controller","msg":"Statistics","Experiment":"kubeflow/random-example","requiredActiveCount":3,"parallelCount":3,"activeCount":0,"completedCount":0}
{"level":"info","ts":1612872512.476889,"logger":"experiment-controller","msg":"Reconcile Suggestion","Experiment":"kubeflow/random-example","addCount":3}
{"level":"info","ts":1612872512.476903,"logger":"experiment-controller","msg":"GetOrCreateSuggestion","Experiment":"kubeflow/random-example","name":"random-example","Suggestion Requests":3}
{"level":"info","ts":1612872530.5734138,"logger":"suggestion-client","msg":"Algorithm settings validated","Suggestion":"kubeflow/random-example"}
{"level":"info","ts":1612872530.5735319,"logger":"suggestion-controller","msg":"Sync assignments","Suggestion":"kubeflow/random-example","Suggestion Requests":3,"Suggestion Count":0}
{"level":"info","ts":1612872530.5901716,"logger":"suggestion-client","msg":"Getting suggestions","Suggestion":"kubeflow/random-example","endpoint":"random-example-random.kubeflow:6789","response":"parameter_assignments:<assignments:<name:\"lr\" value:\"0.017247819947203986\" > assignments:<name:\"num-layers\" value:\"4\" > assignments:<name:\"optimizer\" value:\"sgd\" > > parameter_assignments:<assignments:<name:\"lr\" value:\"0.023992702766545415\" > assignments:<name:\"num-layers\" value:\"4\" > assignments:<name:\"optimizer\" value:\"sgd\" > > parameter_assignments:<assignments:<name:\"lr\" value:\"0.025633297862773914\" > assignments:<name:\"num-layers\" value:\"5\" > assignments:<name:\"optimizer\" value:\"adam\" > > ","request":"experiment:<name:\"random-example\" spec:<parameter_specs:<parameters:<name:\"lr\" parameter_type:DOUBLE feasible_space:<max:\"0.03\" min:\"0.01\" > > parameters:<name:\"num-layers\" parameter_type:INT feasible_space:<max:\"5\" min:\"2\" > > parameters:<name:\"optimizer\" parameter_type:CATEGORICAL feasible_space:<list:\"sgd\" list:\"adam\" list:\"ftrl\" > > > objective:<type:MAXIMIZE goal:0.99 objective_metric_name:\"Validation-accuracy\" additional_metric_names:\"Train-accuracy\" > algorithm:<algorithm_name:\"random\" > parallel_trial_count:3 max_trial_count:12 > > request_number:3 "}
{"level":"info","ts":1612872530.5982351,"logger":"experiment-controller","msg":"Statistics","Experiment":"kubeflow/random-example","requiredActiveCount":3,"parallelCount":3,"activeCount":0,"completedCount":0}
{"level":"info","ts":1612872530.598303,"logger":"experiment-controller","msg":"Reconcile Suggestion","Experiment":"kubeflow/random-example","addCount":3}
{"level":"info","ts":1612872530.5983279,"logger":"experiment-controller","msg":"GetOrCreateSuggestion","Experiment":"kubeflow/random-example","name":"random-example","Suggestion Requests":3}
{"level":"info","ts":1612872530.5985317,"logger":"suggestion-controller","msg":"Sync assignments","Suggestion":"kubeflow/random-example","Suggestion Requests":3,"Suggestion Count":3}
{"level":"info","ts":1612872530.6314745,"logger":"experiment-controller","msg":"Created Trials","Experiment":"kubeflow/random-example","trialNames":["random-example-44tpr2bv","random-example-zdvmp9st","random-example-cj6dtpmj"]}
{"level":"info","ts":1612872530.6537616,"logger":"trial-controller","msg":"Creating Job","Trial":"kubeflow/random-example-44tpr2bv","kind":"Job","name":"random-example-44tpr2bv"}
{"level":"info","ts":1612872530.6579332,"logger":"trial-controller","msg":"Trial status changed to Running","Trial":"kubeflow/random-example-44tpr2bv"}
{"level":"info","ts":1612872530.6837769,"logger":"experiment-controller","msg":"Update experiment instance status failed, reconciler requeued","Experiment":"kubeflow/random-example","err":"Operation cannot be fulfilled on experiments.kubeflow.org \"random-example\": the object has been modified; please apply your changes to the latest version and try again"}
{"level":"info","ts":1612872530.6927054,"logger":"trial-controller","msg":"Creating Job","Trial":"kubeflow/random-example-zdvmp9st","kind":"Job","name":"random-example-zdvmp9st"}
{"level":"info","ts":1612872530.6956549,"logger":"trial-controller","msg":"Trial status changed to Running","Trial":"kubeflow/random-example-zdvmp9st"}
2021/02/09 12:08:50 http2: panic serving 10.244.0.1:42202: runtime error: invalid memory address or nil pointer dereference
goroutine 537 [running]:
net/http.(*http2serverConn).runHandler.func1(0xc00093c490, 0xc001589f8e, 0xc000f84d80)
    /usr/local/go/src/net/http/h2_bundle.go:5711 +0x16f
panic(0x14f6360, 0x2205250)
    /usr/local/go/src/runtime/panic.go:969 +0x1b9
github.com/kubeflow/katib/pkg/webhook/v1beta1/pod.(*sidecarInjector).MutationRequired(0xc000c60db0, 0xc0007d4700, 0xc001256808, 0x8, 0x0, 0x0, 0x21a002a)
    /go/src/github.com/kubeflow/katib/pkg/webhook/v1beta1/pod/inject_webhook.go:137 +0x161
github.com/kubeflow/katib/pkg/webhook/v1beta1/pod.(*sidecarInjector).Handle(0xc000c60db0, 0x18b9480, 0xc000190000, 0xc00139a2c0, 0xffffffffffffffff, 0xc000923a28, 0xa3ef45, 0x187f920)
    /go/src/github.com/kubeflow/katib/pkg/webhook/v1beta1/pod/inject_webhook.go:71 +0x1bd
github.com/kubeflow/katib/vendor/sigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).handleMutating(0xc000c7a480, 0x18b9480, 0xc000190000, 0xc00139a2c0, 0x3, 0xc000189bc0, 0xc000189bc0, 0x0)
    /go/src/github.com/kubeflow/katib/vendor/sigs.k8s.io/controller-runtime/pkg/webhook/admission/webhook.go:133 +0xd8
github.com/kubeflow/katib/vendor/sigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).Handle(0xc000c7a480, 0x18b9480, 0xc000190000, 0xc00139a2c0, 0x0, 0x189ccc0, 0xc000ddd470, 0x189ccc0)
    /go/src/github.com/kubeflow/katib/vendor/sigs.k8s.io/controller-runtime/pkg/webhook/admission/webhook.go:120 +0x1fa
github.com/kubeflow/katib/vendor/sigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).ServeHTTP(0xc000c7a480, 0x18b2940, 0xc00093c490, 0xc001496c00)
    /go/src/github.com/kubeflow/katib/vendor/sigs.k8s.io/controller-runtime/pkg/webhook/admission/http.go:93 +0x9f5
net/http.(*ServeMux).ServeHTTP(0xc000c81240, 0x18b2940, 0xc00093c490, 0xc001496c00)
    /usr/local/go/src/net/http/server.go:2417 +0x1ad
net/http.serverHandler.ServeHTTP(0xc000206620, 0x18b2940, 0xc00093c490, 0xc001496c00)
    /usr/local/go/src/net/http/server.go:2843 +0xa3
net/http.initALPNRequest.ServeHTTP(0x18b9500, 0xc0008043f0, 0xc000741180, 0xc000206620, 0x18b2940, 0xc00093c490, 0xc001496c00)
    /usr/local/go/src/net/http/server.go:3415 +0x8d
net/http.(*http2serverConn).runHandler(0xc000f84d80, 0xc00093c490, 0xc001496c00, 0xc001077ae0)
    /usr/local/go/src/net/http/h2_bundle.go:5718 +0x8b
created by net/http.(*http2serverConn).processHeaders
    /usr/local/go/src/net/http/h2_bundle.go:5452 +0x505
2021/02/09 12:08:50 http2: panic serving 10.244.0.1:42202: runtime error: invalid memory address or nil pointer dereference
goroutine 539 [running]:
net/http.(*http2serverConn).runHandler.func1(0xc00093c498, 0xc001589f8e, 0xc000f84d80)
    /usr/local/go/src/net/http/h2_bundle.go:5711 +0x16f
panic(0x14f6360, 0x2205250)
    /usr/local/go/src/runtime/panic.go:969 +0x1b9
github.com/kubeflow/katib/pkg/webhook/v1beta1/pod.(*sidecarInjector).MutationRequired(0xc000c60db0, 0xc0007d4a80, 0xc001257508, 0x8, 0x0, 0x0, 0x21a002a)
    /go/src/github.com/kubeflow/katib/pkg/webhook/v1beta1/pod/inject_webhook.go:137 +0x161
github.com/kubeflow/katib/pkg/webhook/v1beta1/pod.(*sidecarInjector).Handle(0xc000c60db0, 0x18b9480, 0xc000190000, 0xc00139adc0, 0xffffffffffffffff, 0xc000923a28, 0xa3ef45, 0x187f920)
    /go/src/github.com/kubeflow/katib/pkg/webhook/v1beta1/pod/inject_webhook.go:71 +0x1bd
github.com/kubeflow/katib/vendor/sigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).handleMutating(0xc000c7a480, 0x18b9480, 0xc000190000, 0xc00139adc0, 0x3, 0xc000189bc0, 0xc000189bc0, 0x0)
    /go/src/github.com/kubeflow/katib/vendor/sigs.k8s.io/controller-runtime/pkg/webhook/admission/webhook.go:133 +0xd8
github.com/kubeflow/katib/vendor/sigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).Handle(0xc000c7a480, 0x18b9480, 0xc000190000, 0xc00139adc0, 0x0, 0x189ccc0, 0xc000e61380, 0x189ccc0)
    /go/src/github.com/kubeflow/katib/vendor/sigs.k8s.io/controller-runtime/pkg/webhook/admission/webhook.go:120 +0x1fa
github.com/kubeflow/katib/vendor/sigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).ServeHTTP(0xc000c7a480, 0x18b2940, 0xc00093c498, 0xc001496e00)
    /go/src/github.com/kubeflow/katib/vendor/sigs.k8s.io/controller-runtime/pkg/webhook/admission/http.go:93 +0x9f5
net/http.(*ServeMux).ServeHTTP(0xc000c81240, 0x18b2940, 0xc00093c498, 0xc001496e00)
    /usr/local/go/src/net/http/server.go:2417 +0x1ad
net/http.serverHandler.ServeHTTP(0xc000206620, 0x18b2940, 0xc00093c498, 0xc001496e00)
    /usr/local/go/src/net/http/server.go:2843 +0xa3
net/http.initALPNRequest.ServeHTTP(0x18b9500, 0xc0008043f0, 0xc000741180, 0xc000206620, 0x18b2940, 0xc00093c498, 0xc001496e00)
    /usr/local/go/src/net/http/server.go:3415 +0x8d
net/http.(*http2serverConn).runHandler(0xc000f84d80, 0xc00093c498, 0xc001496e00, 0xc001077bc0)
    /usr/local/go/src/net/http/h2_bundle.go:5718 +0x8b
created by net/http.(*http2serverConn).processHeaders
    /usr/local/go/src/net/http/h2_bundle.go:5452 +0x505
{"level":"info","ts":1612872530.8078678,"logger":"trial-controller","msg":"Creating Job","Trial":"kubeflow/random-example-cj6dtpmj","kind":"Job","name":"random-example-cj6dtpmj"}
{"level":"info","ts":1612872530.8293056,"logger":"trial-controller","msg":"Trial status changed to Running","Trial":"kubeflow/random-example-cj6dtpmj"}
2021/02/09 12:08:50 http2: panic serving 10.244.0.1:42202: runtime error: invalid memory address or nil pointer dereference
goroutine 742 [running]:
net/http.(*http2serverConn).runHandler.func1(0xc00000ec80, 0xc001585f8e, 0xc000f84d80)
    /usr/local/go/src/net/http/h2_bundle.go:5711 +0x16f
panic(0x14f6360, 0x2205250)
    /usr/local/go/src/runtime/panic.go:969 +0x1b9
github.com/kubeflow/katib/pkg/webhook/v1beta1/pod.(*sidecarInjector).MutationRequired(0xc000c60db0, 0xc0007da000, 0xc00133c238, 0x8, 0x0, 0x0, 0x21a002a)
    /go/src/github.com/kubeflow/katib/pkg/webhook/v1beta1/pod/inject_webhook.go:137 +0x161
github.com/kubeflow/katib/pkg/webhook/v1beta1/pod.(*sidecarInjector).Handle(0xc000c60db0, 0x18b9480, 0xc000190000, 0xc00132c9a0, 0xffffffffffffffff, 0xc0009e8a28, 0xa3ef45, 0x187f920)
    /go/src/github.com/kubeflow/katib/pkg/webhook/v1beta1/pod/inject_webhook.go:71 +0x1bd
github.com/kubeflow/katib/vendor/sigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).handleMutating(0xc000c7a480, 0x18b9480, 0xc000190000, 0xc00132c9a0, 0x3, 0xc000189bc0, 0xc000189bc0, 0x0)
    /go/src/github.com/kubeflow/katib/vendor/sigs.k8s.io/controller-runtime/pkg/webhook/admission/webhook.go:133 +0xd8
github.com/kubeflow/katib/vendor/sigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).Handle(0xc000c7a480, 0x18b9480, 0xc000190000, 0xc00132c9a0, 0x0, 0x189ccc0, 0xc001340870, 0x189ccc0)
    /go/src/github.com/kubeflow/katib/vendor/sigs.k8s.io/controller-runtime/pkg/webhook/admission/webhook.go:120 +0x1fa
github.com/kubeflow/katib/vendor/sigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).ServeHTTP(0xc000c7a480, 0x18b2940, 0xc00000ec80, 0xc00098fb00)
    /go/src/github.com/kubeflow/katib/vendor/sigs.k8s.io/controller-runtime/pkg/webhook/admission/http.go:93 +0x9f5
net/http.(*ServeMux).ServeHTTP(0xc000c81240, 0x18b2940, 0xc00000ec80, 0xc00098fb00)
    /usr/local/go/src/net/http/server.go:2417 +0x1ad
net/http.serverHandler.ServeHTTP(0xc000206620, 0x18b2940, 0xc00000ec80, 0xc00098fb00)
    /usr/local/go/src/net/http/server.go:2843 +0xa3
net/http.initALPNRequest.ServeHTTP(0x18b9500, 0xc0008043f0, 0xc000741180, 0xc000206620, 0x18b2940, 0xc00000ec80, 0xc00098fb00)
    /usr/local/go/src/net/http/server.go:3415 +0x8d
net/http.(*http2serverConn).runHandler(0xc000f84d80, 0xc00000ec80, 0xc00098fb00, 0xc0010794a0)
    /usr/local/go/src/net/http/h2_bundle.go:5718 +0x8b
created by net/http.(*http2serverConn).processHeaders
    /usr/local/go/src/net/http/h2_bundle.go:5452 +0x505
rky0930 picture rky0930 on 9 Feb 2021

I believe that is because Katib webhooks don't work.

If you deploy Katib as part of Kubeflow your Experiments can't be submitted in the kubeflow namespace. You have to use your Profile namespace.
Please check tutorial here: https://www.kubeflow.org/docs/components/katib/hyperparameter/#example-using-random-algorithm.

andreyvelich picture andreyvelich on 9 Feb 2021
👍1

@andreyvelich Thanks for your help! Now it works!

rky0930 picture rky0930 on 9 Feb 2021
🚀1

@rky0930 It's great!
Feel free to open new issue if you have any other problems.

andreyvelich picture andreyvelich on 9 Feb 2021
❤1
Was this page helpful?
0 / 5 - 0 ratings

Related issues

YujiOshima picture YujiOshima  路  5Comments
jlewi picture jlewi  路  5Comments
ddysher picture ddysher  路  7Comments
kylepad picture kylepad  路  6Comments
gyliu513 picture gyliu513  路  7Comments