katib-manager Pod crashes (Kubeflow 0.6)
What steps did you take and what happened:
I've installed Kubeflow 0.6 on a Kubernetes on-premise cluster (1.14) with MetalLB.
Katib-manager pod crashes with the follow log:
E0725 13:27:46.751356 1 interface.go:98] Ping to Katib db failed: dial tcp 10.99.115.60:3306: i/o timeout
E0725 13:27:51.751508 1 interface.go:98] Ping to Katib db failed: dial tcp 10.99.115.60:3306: i/o timeout
E0725 13:27:56.751668 1 interface.go:98] Ping to Katib db failed: dial tcp 10.99.115.60:3306: i/o timeout
What did you expect to happen:
katib-manager pod starts successfully.
Anything else you would like to add:
This is the YAML for katib-manager:
{
"kind": "Pod",
"apiVersion": "v1",
"metadata": {
"name": "katib-manager-574c8c67f9-xtz4m",
"generateName": "katib-manager-574c8c67f9-",
"namespace": "kubeflow",
"selfLink": "/api/v1/namespaces/kubeflow/pods/katib-manager-574c8c67f9-xtz4m",
"uid": "ff8bd34e-aec1-11e9-b38a-f40343df0215",
"resourceVersion": "5593656",
"creationTimestamp": "2019-07-25T09:53:05Z",
"labels": {
"app": "katib",
"component": "manager",
"pod-template-hash": "574c8c67f9"
},
"ownerReferences": [
{
"apiVersion": "apps/v1",
"kind": "ReplicaSet",
"name": "katib-manager-574c8c67f9",
"uid": "fe768ad0-aec1-11e9-b38a-f40343df0215",
"controller": true,
"blockOwnerDeletion": true
}
]
},
"spec": {
"volumes": [
{
"name": "default-token-p4pkj",
"secret": {
"secretName": "default-token-p4pkj",
"defaultMode": 420
}
}
],
"containers": [
{
"name": "katib-manager",
"image": "gcr.io/kubeflow-images-public/katib/v1alpha2/katib-manager:v0.6.0-rc.0",
"command": [
"./katib-manager"
],
"ports": [
{
"name": "api",
"containerPort": 6789,
"protocol": "TCP"
}
],
"env": [
{
"name": "MYSQL_ROOT_PASSWORD",
"valueFrom": {
"secretKeyRef": {
"name": "katib-db-secrets",
"key": "MYSQL_ROOT_PASSWORD"
}
}
}
],
"resources": {},
"volumeMounts": [
{
"name": "default-token-p4pkj",
"readOnly": true,
"mountPath": "/var/run/secrets/kubernetes.io/serviceaccount"
}
],
"livenessProbe": {
"exec": {
"command": [
"/bin/grpc_health_probe",
"-addr=:6789"
]
},
"initialDelaySeconds": 10,
"timeoutSeconds": 1,
"periodSeconds": 10,
"successThreshold": 1,
"failureThreshold": 3
},
"readinessProbe": {
"exec": {
"command": [
"/bin/grpc_health_probe",
"-addr=:6789"
]
},
"initialDelaySeconds": 5,
"timeoutSeconds": 1,
"periodSeconds": 10,
"successThreshold": 1,
"failureThreshold": 3
},
"terminationMessagePath": "/dev/termination-log",
"terminationMessagePolicy": "File",
"imagePullPolicy": "IfNotPresent"
}
],
"restartPolicy": "Always",
"terminationGracePeriodSeconds": 30,
"dnsPolicy": "ClusterFirst",
"serviceAccountName": "default",
"serviceAccount": "default",
"nodeName": "moonshot01",
"securityContext": {},
"schedulerName": "default-scheduler",
"tolerations": [
{
"key": "node.kubernetes.io/not-ready",
"operator": "Exists",
"effect": "NoExecute",
"tolerationSeconds": 300
},
{
"key": "node.kubernetes.io/unreachable",
"operator": "Exists",
"effect": "NoExecute",
"tolerationSeconds": 300
}
],
"priority": 0,
"enableServiceLinks": true
},
"status": {
"phase": "Running",
"conditions": [
{
"type": "Initialized",
"status": "True",
"lastProbeTime": null,
"lastTransitionTime": "2019-07-25T09:53:05Z"
},
{
"type": "Ready",
"status": "False",
"lastProbeTime": null,
"lastTransitionTime": "2019-07-25T09:53:05Z",
"reason": "ContainersNotReady",
"message": "containers with unready status: [katib-manager]"
},
{
"type": "ContainersReady",
"status": "False",
"lastProbeTime": null,
"lastTransitionTime": "2019-07-25T09:53:05Z",
"reason": "ContainersNotReady",
"message": "containers with unready status: [katib-manager]"
},
{
"type": "PodScheduled",
"status": "True",
"lastProbeTime": null,
"lastTransitionTime": "2019-07-25T09:53:05Z"
}
],
"hostIP": "192.168.110.111",
"podIP": "10.244.1.127",
"startTime": "2019-07-25T09:53:05Z",
"containerStatuses": [
{
"name": "katib-manager",
"state": {
"waiting": {
"reason": "CrashLoopBackOff",
"message": "Back-off 5m0s restarting failed container=katib-manager pod=katib-manager-574c8c67f9-xtz4m_kubeflow(ff8bd34e-aec1-11e9-b38a-f40343df0215)"
}
},
"lastState": {
"terminated": {
"exitCode": 2,
"reason": "Error",
"startedAt": "2019-07-25T13:33:51Z",
"finishedAt": "2019-07-25T13:34:31Z",
"containerID": "docker://0421271e49588add3924cb7cc39c203eb810a7ee14c8e68108fd690f207dccdd"
}
},
"ready": false,
"restartCount": 73,
"image": "gcr.io/kubeflow-images-public/katib/v1alpha2/katib-manager:v0.6.0-rc.0",
"imageID": "docker-pullable://gcr.io/kubeflow-images-public/katib/v1alpha2/katib-manager@sha256:8dbe595c3a241ce65d29afb87a99453461b2c82338e54135dc8dfb4cb5ac8fa6",
"containerID": "docker://0421271e49588add3924cb7cc39c203eb810a7ee14c8e68108fd690f207dccdd"
}
],
"qosClass": "BestEffort"
}
}
YAML for katib-db:
{
"kind": "Pod",
"apiVersion": "v1",
"metadata": {
"name": "katib-db-8598468fd8-5tpl2",
"generateName": "katib-db-8598468fd8-",
"namespace": "kubeflow",
"selfLink": "/api/v1/namespaces/kubeflow/pods/katib-db-8598468fd8-5tpl2",
"uid": "ff05d7e8-aec1-11e9-b38a-f40343df0215",
"resourceVersion": "5562403",
"creationTimestamp": "2019-07-25T09:53:04Z",
"labels": {
"app": "katib",
"component": "db",
"pod-template-hash": "8598468fd8"
},
"ownerReferences": [
{
"apiVersion": "apps/v1",
"kind": "ReplicaSet",
"name": "katib-db-8598468fd8",
"uid": "fe6c21f4-aec1-11e9-b38a-f40343df0215",
"controller": true,
"blockOwnerDeletion": true
}
]
},
"spec": {
"volumes": [
{
"name": "katib-mysql",
"persistentVolumeClaim": {
"claimName": "katib-mysql"
}
},
{
"name": "default-token-p4pkj",
"secret": {
"secretName": "default-token-p4pkj",
"defaultMode": 420
}
}
],
"containers": [
{
"name": "katib-db",
"image": "mysql:8.0.3",
"args": [
"--datadir",
"/var/lib/mysql/datadir"
],
"ports": [
{
"name": "dbapi",
"containerPort": 3306,
"protocol": "TCP"
}
],
"env": [
{
"name": "MYSQL_ROOT_PASSWORD",
"valueFrom": {
"secretKeyRef": {
"name": "katib-db-secrets",
"key": "MYSQL_ROOT_PASSWORD"
}
}
},
{
"name": "MYSQL_ALLOW_EMPTY_PASSWORD",
"value": "true"
},
{
"name": "MYSQL_DATABASE",
"value": "katib"
}
],
"resources": {},
"volumeMounts": [
{
"name": "katib-mysql",
"mountPath": "/var/lib/mysql"
},
{
"name": "default-token-p4pkj",
"readOnly": true,
"mountPath": "/var/run/secrets/kubernetes.io/serviceaccount"
}
],
"readinessProbe": {
"exec": {
"command": [
"/bin/bash",
"-c",
"mysql -D $$MYSQL_DATABASE -p$$MYSQL_ROOT_PASSWORD -e 'SELECT 1'"
]
},
"initialDelaySeconds": 5,
"timeoutSeconds": 1,
"periodSeconds": 2,
"successThreshold": 1,
"failureThreshold": 3
},
"terminationMessagePath": "/dev/termination-log",
"terminationMessagePolicy": "File",
"imagePullPolicy": "IfNotPresent"
}
],
"restartPolicy": "Always",
"terminationGracePeriodSeconds": 30,
"dnsPolicy": "ClusterFirst",
"serviceAccountName": "default",
"serviceAccount": "default",
"nodeName": "moonshot01",
"securityContext": {},
"schedulerName": "default-scheduler",
"tolerations": [
{
"key": "node.kubernetes.io/not-ready",
"operator": "Exists",
"effect": "NoExecute",
"tolerationSeconds": 300
},
{
"key": "node.kubernetes.io/unreachable",
"operator": "Exists",
"effect": "NoExecute",
"tolerationSeconds": 300
}
],
"priority": 0,
"enableServiceLinks": true
},
"status": {
"phase": "Running",
"conditions": [
{
"type": "Initialized",
"status": "True",
"lastProbeTime": null,
"lastTransitionTime": "2019-07-25T09:53:04Z"
},
{
"type": "Ready",
"status": "True",
"lastProbeTime": null,
"lastTransitionTime": "2019-07-25T09:53:25Z"
},
{
"type": "ContainersReady",
"status": "True",
"lastProbeTime": null,
"lastTransitionTime": "2019-07-25T09:53:25Z"
},
{
"type": "PodScheduled",
"status": "True",
"lastProbeTime": null,
"lastTransitionTime": "2019-07-25T09:53:04Z"
}
],
"hostIP": "192.168.110.111",
"podIP": "10.244.1.114",
"startTime": "2019-07-25T09:53:04Z",
"containerStatuses": [
{
"name": "katib-db",
"state": {
"running": {
"startedAt": "2019-07-25T09:53:09Z"
}
},
"lastState": {},
"ready": true,
"restartCount": 0,
"image": "mysql:8.0.3",
"imageID": "docker-pullable://mysql@sha256:cf12c8d3b7bcff1c4395305e5a39e7d912bf81c59938a4f1c65c91ced66f985b",
"containerID": "docker://58807248eb9aaaf63bcdfe06eefcf8c5e4c2acfbe02021c1d8e55180a288936f"
}
],
"qosClass": "BestEffort"
}
}
This the log from katib-db Pod:
2019-07-25T09:53:23.831752Z 0 [Note] --secure-file-priv is set to NULL. Operations related to importing and exporting data are disabled
2019-07-25T09:53:23.831786Z 0 [Note] /usr/sbin/mysqld (mysqld 8.0.3-rc-log) starting as process 1 ...
2019-07-25T09:53:23.833987Z 0 [Warning] No argument was provided to --log-bin, and --log-bin-index was not used; so replication may break when this MySQL server acts as a master and has his hostname changed!! Please use '--log-bin=katib-db-8598468fd8-5tpl2-bin' to avoid this problem.
2019-07-25T09:53:23.836253Z 0 [Note] InnoDB: Using Linux native AIO
2019-07-25T09:53:23.836374Z 0 [Note] Plugin 'FEDERATED' is disabled.
2019-07-25T09:53:23.840007Z 1 [Note] InnoDB: PUNCH HOLE support available
2019-07-25T09:53:23.840031Z 1 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2019-07-25T09:53:23.840036Z 1 [Note] InnoDB: Uses event mutexes
2019-07-25T09:53:23.840040Z 1 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2019-07-25T09:53:23.840046Z 1 [Note] InnoDB: Compressed tables use zlib 1.2.11
2019-07-25T09:53:23.840409Z 1 [Note] InnoDB: Number of pools: 1
2019-07-25T09:53:23.840522Z 1 [Note] InnoDB: Using CPU crc32 instructions
2019-07-25T09:53:23.842398Z 1 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M
2019-07-25T09:53:23.855511Z 1 [Note] InnoDB: Completed initialization of buffer pool
2019-07-25T09:53:23.857171Z 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2019-07-25T09:53:23.871940Z 1 [Note] InnoDB: Using 'tablespaces.open.2' max LSN: 15050811
2019-07-25T09:53:23.874918Z 1 [Note] InnoDB: Applying a batch of 0 redo log records ...
2019-07-25T09:53:23.874936Z 1 [Note] InnoDB: Apply batch completed!
2019-07-25T09:53:23.875842Z 1 [Note] InnoDB: Opened 2 existing undo tablespaces.
2019-07-25T09:53:23.886778Z 1 [Note] InnoDB: Creating shared tablespace for temporary tables
2019-07-25T09:53:23.886857Z 1 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2019-07-25T09:53:23.919128Z 1 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2019-07-25T09:53:23.923261Z 1 [Note] InnoDB: Created 128 and tracked 128 new rollback segment(s) in the temporary tablespace. 128 are now active.
2019-07-25T09:53:23.923455Z 1 [Note] InnoDB: 8.0.3 started; log sequence number 26008934
2019-07-25T09:53:24.012041Z 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/datadir/ib_buffer_pool
2019-07-25T09:53:24.023296Z 0 [Note] InnoDB: Buffer pool(s) load completed at 190725 9:53:24
2019-07-25T09:53:24.042511Z 1 [Note] Found data dictionary with version 1
2019-07-25T09:53:24.080051Z 0 [Note] InnoDB: DDL log recovery : begin
2019-07-25T09:53:24.080107Z 0 [Note] InnoDB: DDL log recovery : end
2019-07-25T09:53:24.080251Z 0 [Note] InnoDB: Waiting for purge to start
2019-07-25T09:53:24.140632Z 0 [Warning] You have not provided a mandatory server-id. Servers in a replication topology must have unique server-ids. Please refer to the proper server start-up parameters documentation.
2019-07-25T09:53:24.164594Z 0 [Note] Found ca.pem, server-cert.pem and server-key.pem in data directory. Trying to enable SSL support using them.
2019-07-25T09:53:24.164841Z 0 [Warning] CA certificate ca.pem is self signed.
2019-07-25T09:53:24.167219Z 0 [Note] Server hostname (bind-address): '*'; port: 3306
2019-07-25T09:53:24.170898Z 0 [Note] IPv6 is available.
2019-07-25T09:53:24.170925Z 0 [Note] - '::' resolves to '::';
2019-07-25T09:53:24.170983Z 0 [Note] Server socket created on IP: '::'.
2019-07-25T09:53:24.186265Z 0 [Warning] 'user' entry 'mysql.session@localhost' ignored in --skip-name-resolve mode.
2019-07-25T09:53:24.186297Z 0 [Warning] 'user' entry 'mysql.sys@localhost' ignored in --skip-name-resolve mode.
2019-07-25T09:53:24.186310Z 0 [Warning] 'user' entry 'root@localhost' ignored in --skip-name-resolve mode.
2019-07-25T09:53:24.186560Z 0 [Warning] 'db' entry 'performance_schema mysql.session@localhost' ignored in --skip-name-resolve mode.
2019-07-25T09:53:24.186587Z 0 [Warning] 'db' entry 'sys mysql.sys@localhost' ignored in --skip-name-resolve mode.
2019-07-25T09:53:24.186604Z 0 [Warning] 'proxies_priv' entry '@ root@localhost' ignored in --skip-name-resolve mode.
2019-07-25T09:53:24.192401Z 0 [Warning] 'tables_priv' entry 'user mysql.session@localhost' ignored in --skip-name-resolve mode.
2019-07-25T09:53:24.192426Z 0 [Warning] 'tables_priv' entry 'sys_config mysql.sys@localhost' ignored in --skip-name-resolve mode.
2019-07-25T09:53:24.199342Z 4 [Note] Event Scheduler: scheduler thread started with id 4
2019-07-25T09:53:24.199935Z 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '8.0.3-rc-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 MySQL Community Server (GPL)
Environment:
- Kubeflow version 0.6
- Kubernetes version 1.14 on-premise (with MetalLB)
- OS : CentOS 7.6
kimchitsigai
All 14 comments
Additional info:
$ kubectl describe po -n kubeflow katib-manager-574c8c67f9-xtz4m
Name: katib-manager-574c8c67f9-xtz4m
Namespace: kubeflow
Priority: 0
PriorityClassName: <none>
Node: moonshot01/192.168.110.111
Start Time: Thu, 25 Jul 2019 11:53:05 +0200
Labels: app=katib
component=manager
pod-template-hash=574c8c67f9
Annotations: <none>
Status: Running
IP: 10.244.1.127
Controlled By: ReplicaSet/katib-manager-574c8c67f9
Containers:
katib-manager:
Container ID: docker://a1cc411adc7d24d0ae38fdcf357563d9f83498bbef9dad9ab8f98bca015031de
Image: gcr.io/kubeflow-images-public/katib/v1alpha2/katib-manager:v0.6.0-rc.0
Image ID: docker-pullable://gcr.io/kubeflow-images-public/katib/v1alpha2/katib-manager@sha256:8dbe595c3a241ce65d29afb87a99453461b2c82338e54135dc8dfb4cb5ac8fa6
Port: 6789/TCP
Host Port: 0/TCP
Command:
./katib-manager
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 2
Started: Thu, 25 Jul 2019 16:18:41 +0200
Finished: Thu, 25 Jul 2019 16:19:21 +0200
Ready: False
Restart Count: 87
Liveness: exec [/bin/grpc_health_probe -addr=:6789] delay=10s timeout=1s period=10s #success=1 #failure=3
Readiness: exec [/bin/grpc_health_probe -addr=:6789] delay=5s timeout=1s period=10s #success=1 #failure=3
Environment:
MYSQL_ROOT_PASSWORD: <set to the key 'MYSQL_ROOT_PASSWORD' in secret 'katib-db-secrets'> Optional: false
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-p4pkj (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
default-token-p4pkj:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-p4pkj
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Started 47m (x73 over 4h27m) kubelet, moonshot01 Started container katib-manager
Warning BackOff 7m17s (x989 over 4h23m) kubelet, moonshot01 Back-off restarting failed container
Warning Unhealthy 2m13s (x330 over 4h27m) kubelet, moonshot01 Readiness probe failed: timeout: failed to connect service ":6789" within 1s
Can you list the pod status of katib pods?
Can you check the status of mysql PVC?
johnugeorge
on 25 Jul 2019
katib-mysql pvc:
$ kubectl describe pvc katib-mysql -n kubeflow
Name: katib-mysql
Namespace: kubeflow
StorageClass:
Status: Bound
Volume: katib-mysql-pv
Labels: <none>
Annotations: pv.kubernetes.io/bind-completed: yes
pv.kubernetes.io/bound-by-controller: yes
Finalizers: [kubernetes.io/pvc-protection]
Capacity: 10Gi
Access Modes: RWO
VolumeMode: Filesystem
Events: <none>
Mounted By: katib-db-8598468fd8-5tpl2
mysql-pv-claim (just in case):
$ kubectl describe pvc mysql-pv-claim -n kubeflow
Name: mysql-pv-claim
Namespace: kubeflow
StorageClass:
Status: Bound
Volume: mysql-pv
Labels: app=mysql
Annotations: pv.kubernetes.io/bind-completed: yes
pv.kubernetes.io/bound-by-controller: yes
Finalizers: [kubernetes.io/pvc-protection]
Capacity: 20Gi
Access Modes: RWO
VolumeMode: Filesystem
Events: <none>
Mounted By: mysql-657f87857d-n9m6g
katib-controller Pod:
$ kubectl describe po katib-controller-88c97d85c-xtdlk -n kubeflow
Name: katib-controller-88c97d85c-xtdlk
Namespace: kubeflow
Priority: 0
PriorityClassName: <none>
Node: moonshot01/192.168.110.111
Start Time: Thu, 25 Jul 2019 11:53:05 +0200
Labels: app=katib-controller
pod-template-hash=88c97d85c
Annotations: <none>
Status: Running
IP: 10.244.1.125
Controlled By: ReplicaSet/katib-controller-88c97d85c
Containers:
katib-controller:
Container ID: docker://745466ba019f96b0b7210cdee08a18290054ca773ccffb6b865beeb67e685da9
Image: gcr.io/kubeflow-images-public/katib/v1alpha2/katib-controller:v0.6.0-rc.0
Image ID: docker-pullable://gcr.io/kubeflow-images-public/katib/v1alpha2/katib-controller@sha256:d4773b412e198a656835e61c55789a79876ad0caae79f26fa5e914cc01bf4531
Port: 443/TCP
Host Port: 0/TCP
Command:
./katib-controller
State: Running
Started: Thu, 25 Jul 2019 11:53:18 +0200
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Thu, 25 Jul 2019 11:53:14 +0200
Finished: Thu, 25 Jul 2019 11:53:16 +0200
Ready: True
Restart Count: 1
Environment:
KATIB_CORE_NAMESPACE: kubeflow (v1:metadata.namespace)
Mounts:
/tmp/cert from cert (ro)
/var/run/secrets/kubernetes.io/serviceaccount from katib-controller-token-rzxwx (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
cert:
Type: Secret (a volume populated by a Secret)
SecretName: katib-controller
Optional: false
katib-controller-token-rzxwx:
Type: Secret (a volume populated by a Secret)
SecretName: katib-controller-token-rzxwx
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events: <none>
katib-db Pod:
$ kubectl describe po katib-db-8598468fd8-5tpl2 -n kubeflow
Name: katib-db-8598468fd8-5tpl2
Namespace: kubeflow
Priority: 0
PriorityClassName: <none>
Node: moonshot01/192.168.110.111
Start Time: Thu, 25 Jul 2019 11:53:04 +0200
Labels: app=katib
component=db
pod-template-hash=8598468fd8
Annotations: <none>
Status: Running
IP: 10.244.1.114
Controlled By: ReplicaSet/katib-db-8598468fd8
Containers:
katib-db:
Container ID: docker://58807248eb9aaaf63bcdfe06eefcf8c5e4c2acfbe02021c1d8e55180a288936f
Image: mysql:8.0.3
Image ID: docker-pullable://mysql@sha256:cf12c8d3b7bcff1c4395305e5a39e7d912bf81c59938a4f1c65c91ced66f985b
Port: 3306/TCP
Host Port: 0/TCP
Args:
--datadir
/var/lib/mysql/datadir
State: Running
Started: Thu, 25 Jul 2019 11:53:09 +0200
Ready: True
Restart Count: 0
Readiness: exec [/bin/bash -c mysql -D $$MYSQL_DATABASE -p$$MYSQL_ROOT_PASSWORD -e 'SELECT 1'] delay=5s timeout=1s period=2s #success=1 #failure=3
Environment:
MYSQL_ROOT_PASSWORD: <set to the key 'MYSQL_ROOT_PASSWORD' in secret 'katib-db-secrets'> Optional: false
MYSQL_ALLOW_EMPTY_PASSWORD: true
MYSQL_DATABASE: katib
Mounts:
/var/lib/mysql from katib-mysql (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-p4pkj (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
katib-mysql:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: katib-mysql
ReadOnly: false
default-token-p4pkj:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-p4pkj
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events: <none>
katib-manager:
$ kubectl describe po katib-manager-574c8c67f9-xtz4m -n kubeflow
Name: katib-manager-574c8c67f9-xtz4m
Namespace: kubeflow
Priority: 0
PriorityClassName: <none>
Node: moonshot01/192.168.110.111
Start Time: Thu, 25 Jul 2019 11:53:05 +0200
Labels: app=katib
component=manager
pod-template-hash=574c8c67f9
Annotations: <none>
Status: Running
IP: 10.244.1.127
Controlled By: ReplicaSet/katib-manager-574c8c67f9
Containers:
katib-manager:
Container ID: docker://05afe8aa2290ab6c550f18860084f22cfd3371d8627d42470ccadd9e6c9aa275
Image: gcr.io/kubeflow-images-public/katib/v1alpha2/katib-manager:v0.6.0-rc.0
Image ID: docker-pullable://gcr.io/kubeflow-images-public/katib/v1alpha2/katib-manager@sha256:8dbe595c3a241ce65d29afb87a99453461b2c82338e54135dc8dfb4cb5ac8fa6
Port: 6789/TCP
Host Port: 0/TCP
Command:
./katib-manager
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 2
Started: Fri, 26 Jul 2019 08:17:31 +0200
Finished: Fri, 26 Jul 2019 08:18:11 +0200
Ready: False
Restart Count: 387
Liveness: exec [/bin/grpc_health_probe -addr=:6789] delay=10s timeout=1s period=10s #success=1 #failure=3
Readiness: exec [/bin/grpc_health_probe -addr=:6789] delay=5s timeout=1s period=10s #success=1 #failure=3
Environment:
MYSQL_ROOT_PASSWORD: <set to the key 'MYSQL_ROOT_PASSWORD' in secret 'katib-db-secrets'> Optional: false
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-p4pkj (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
default-token-p4pkj:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-p4pkj
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning Unhealthy 20m (x1455 over 20h) kubelet, moonshot01 Readiness probe failed: timeout: failed to connect service ":6789" within 1s
Warning BackOff 47s (x4680 over 20h) kubelet, moonshot01 Back-off restarting failed container
katib-suggestion-bayesianoptimization Pod:
$ kubectl describe po katib-suggestion-bayesianoptimization-65df4d7455-tvbk6 -n kubeflow
Name: katib-suggestion-bayesianoptimization-65df4d7455-tvbk6
Namespace: kubeflow
Priority: 0
PriorityClassName: <none>
Node: moonshot01/192.168.110.111
Start Time: Thu, 25 Jul 2019 11:53:06 +0200
Labels: app=katib
component=suggestion-bayesianoptimization
pod-template-hash=65df4d7455
Annotations: <none>
Status: Running
IP: 10.244.1.129
Controlled By: ReplicaSet/katib-suggestion-bayesianoptimization-65df4d7455
Containers:
katib-suggestion-bayesianoptimization:
Container ID: docker://858fd92596ccd56cb8d036825ee7eb9d26ee098b6f4f89dbca20952c7654e4ac
Image: gcr.io/kubeflow-images-public/katib/v1alpha2/suggestion-bayesianoptimization:v0.6.0-rc.0
Image ID: docker-pullable://gcr.io/kubeflow-images-public/katib/v1alpha2/suggestion-bayesianoptimization@sha256:c5c3605c383b59c84f8e18292538ad064f42ecc3d34655127be18e0162fdc013
Port: 6789/TCP
Host Port: 0/TCP
State: Running
Started: Thu, 25 Jul 2019 11:53:14 +0200
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-p4pkj (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
default-token-p4pkj:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-p4pkj
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300sEvents:
Events: <none>
katib-suggestion-grid Pod:
$ kubectl describe po katib-suggestion-grid-56bf69f597-hfsgm -n kubeflow
Name: katib-suggestion-grid-56bf69f597-hfsgm
Namespace: kubeflow
Priority: 0
PriorityClassName: <none>
Node: moonshot01/192.168.110.111
Start Time: Thu, 25 Jul 2019 11:53:06 +0200
Labels: app=katib
component=suggestion-grid
pod-template-hash=56bf69f597
Annotations: <none>
Status: Running
IP: 10.244.1.132
Controlled By: ReplicaSet/katib-suggestion-grid-56bf69f597
Containers:
katib-suggestion-grid:
Container ID: docker://6731ad91b8ca562e2c588f66385501487c2e229d42ac8affac1001593506a5f2
Image: gcr.io/kubeflow-images-public/katib/v1alpha2/suggestion-grid:v0.6.0-rc.0
Image ID: docker-pullable://gcr.io/kubeflow-images-public/katib/v1alpha2/suggestion-grid@sha256:5639d6d8dd882d7829ead58f7c82235a39cf37fda509c17677732f1eb9e79da2
Port: 6789/TCP
Host Port: 0/TCP
State: Running
Started: Thu, 25 Jul 2019 11:53:14 +0200
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-p4pkj (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
default-token-p4pkj:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-p4pkj
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events: <none>
katib-suggestion-hyperband Pod:
$ kubectl describe po katib-suggestion-hyperband-7777b76cb9-zjrsw -n kubeflow
Name: katib-suggestion-hyperband-7777b76cb9-zjrsw
Namespace: kubeflow
Priority: 0
PriorityClassName: <none>
Node: moonshot01/192.168.110.111
Start Time: Thu, 25 Jul 2019 11:53:06 +0200
Labels: app=katib
component=suggestion-hyperband
pod-template-hash=7777b76cb9
Annotations: <none>
Status: Running
IP: 10.244.1.116
Controlled By: ReplicaSet/katib-suggestion-hyperband-7777b76cb9
Containers:
katib-suggestion-hyperband:
Container ID: docker://e6ae6669ddcdf6a6c8f12e7154c934560faa5ae9a18a415c15da723c0c967038
Image: gcr.io/kubeflow-images-public/katib/v1alpha2/suggestion-hyperband:v0.6.0-rc.0
Image ID: docker-pullable://gcr.io/kubeflow-images-public/katib/v1alpha2/suggestion-hyperband@sha256:5f7bc0cf1179152f04b77c902d111ea7b636dd858979cd85f4b887880d6f0fdb
Port: 6789/TCP
Host Port: 0/TCP
State: Running
Started: Thu, 25 Jul 2019 11:53:14 +0200
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-p4pkj (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
default-token-p4pkj:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-p4pkj
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events: <none>
katib-suggestion-nasrl Pod:
$ kubectl describe po katib-suggestion-nasrl-77f6f9458c-8q9vl -n kubeflow
Name: katib-suggestion-nasrl-77f6f9458c-8q9vl
Namespace: kubeflow
Priority: 0
PriorityClassName: <none>
Node: moonshot01/192.168.110.111
Start Time: Thu, 25 Jul 2019 11:53:06 +0200
Labels: app=katib
component=suggestion-nasrl
pod-template-hash=77f6f9458c
Annotations: <none>
Status: Running
IP: 10.244.1.121
Controlled By: ReplicaSet/katib-suggestion-nasrl-77f6f9458c
Containers:
katib-suggestion-nasrl:
Container ID: docker://ab55226b1b6733e71b36ad6500f57ea9bc426acc13e709f3a6110da8968ba915
Image: gcr.io/kubeflow-images-public/katib/v1alpha2/suggestion-nasrl:v0.6.0-rc.0
Image ID: docker-pullable://gcr.io/kubeflow-images-public/katib/v1alpha2/suggestion-nasrl@sha256:95f3565a5af4bda4bbe395b95f5f90cfc0f48ec09b07e5e82f3ca3acab1b867c
Port: 6789/TCP
Host Port: 0/TCP
State: Running
Started: Thu, 25 Jul 2019 11:53:14 +0200
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-p4pkj (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
default-token-p4pkj:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-p4pkj
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events: <none>
katib-suggestion-random Pod:
$ kubectl describe po katib-suggestion-random-77b88b5c79-798m5 -n kubeflow
Name: katib-suggestion-random-77b88b5c79-798m5
Namespace: kubeflow
Priority: 0
PriorityClassName: <none>
Node: moonshot01/192.168.110.111
Start Time: Thu, 25 Jul 2019 11:53:07 +0200
Labels: app=katib
component=suggestion-random
pod-template-hash=77b88b5c79
Annotations: <none>
Status: Running
IP: 10.244.1.119
Controlled By: ReplicaSet/katib-suggestion-random-77b88b5c79
Containers:
katib-suggestion-random:
Container ID: docker://59474a2b36d9016c1f722dd8f6684c4528140357a41f994fb946ff9b90888bc0
Image: gcr.io/kubeflow-images-public/katib/v1alpha2/suggestion-random:v0.6.0-rc.0
Image ID: docker-pullable://gcr.io/kubeflow-images-public/katib/v1alpha2/suggestion-random@sha256:ea2f342ebba83c815ae33817b8c4d2c34cdec70c21771c41eaafb5d315e610ad
Port: 6789/TCP
Host Port: 0/TCP
State: Running
Started: Thu, 25 Jul 2019 11:53:14 +0200
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-p4pkj (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
default-token-p4pkj:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-p4pkj
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events: <none>
katib-ui Pod:
$ kubectl describe po katib-ui-7587c5b967-sdcmx -n kubeflow
Name: katib-ui-7587c5b967-sdcmx
Namespace: kubeflow
Priority: 0
PriorityClassName: <none>
Node: moonshot01/192.168.110.111
Start Time: Thu, 25 Jul 2019 11:53:05 +0200
Labels: app=katib
component=ui
pod-template-hash=7587c5b967
Annotations: <none>
Status: Running
IP: 10.244.1.115
Controlled By: ReplicaSet/katib-ui-7587c5b967
Containers:
katib-ui:
Container ID: docker://17d74376c2c02c76beecca68dac7913a94b38520607e75001408577bb1fa5a40
Image: gcr.io/kubeflow-images-public/katib/v1alpha2/katib-ui:v0.6.0-rc.0
Image ID: docker-pullable://gcr.io/kubeflow-images-public/katib/v1alpha2/katib-ui@sha256:d38071758ea0d60a241cc1f75ac447bf7aa8d18667e9b0883bf6e8724b69948a
Port: 80/TCP
Host Port: 0/TCP
Command:
./katib-ui
State: Running
Started: Thu, 25 Jul 2019 11:53:13 +0200
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from katib-ui-token-frztf (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
katib-ui-token-frztf:
Type: Secret (a volume populated by a Secret)
SecretName: katib-ui-token-frztf
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events: <none>
App pods except manager are in running phase. But manger pod is not able to connect with db pod .
I am not sure about this env. MetalLB does some magic here?
To check if it is a network issue. Try do a curl to katib-db inside the cluster.
what is the output of
kubectl run curl --image=radial/busyboxplus:curl -n kubeflow -i --tty
nslookup katib-db
nslookup katib-db.kubeflow.svc.cluster.local
Does it resolve correctly?
Also, what is the output of kubectl get svc katib-db -n kubeflow
johnugeorge
on 26 Jul 2019
$ kubectl run curl --image=radial/busyboxplus:curl -n kubeflow -i --tty
kubectl run --generator=deployment/apps.v1 is DEPRECATED and will be removed in a future version. Use kubectl run --generator=run-pod/v1 or kubectl create instead.
If you don't see a command prompt, try pressing enter.
[ root@curl-66bdcf564-lm5bp:/ ]$ ls
bin/ etc/ lib/ linuxrc mnt/ proc/ run sys/ usr/
dev/ home/ lib64 media/ opt/ root/ sbin/ tmp/ var/
[ root@curl-66bdcf564-lm5bp:/ ]$ exit
Session ended, resume using 'kubectl attach curl-66bdcf564-lm5bp -c curl -i -t' command when the pod is running
$ nslookup katib-db
Server: 8.8.8.8
Address: 8.8.8.8#53
** server can't find katib-db: NXDOMAIN
$ nslookup katib-db
Server: 8.8.8.8
Address: 8.8.8.8#53
** server can't find katib-db: NXDOMAIN
$ nslookup katib-db.kubeflow.svc.cluster.local
Server: 8.8.8.8
Address: 8.8.8.8#53
** server can't find katib-db.kubeflow.svc.cluster.local: NXDOMAIN
$ kubectl get svc katib-db -n kubeflow
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
katib-db ClusterIP 10.99.115.60 <none> 3306/TCP 22h
Names are not resolved. You're right, maybe this is related to using MetalLB (https://metallb.universe.tf/) which provides an external load-balancer for on-premise K8s clusters with Istio.
kimchitsigai
on 26 Jul 2019
Where did you run nslookup? I see an exit. The command has to be executed inside the pod.
johnugeorge
on 26 Jul 2019
Apologies.
Output of nslookup from the pod:
$ kubectl run curl --image=radial/busyboxplus:curl -n kubeflow -i --tty
kubectl run --generator=deployment/apps.v1 is DEPRECATED and will be removed in a future version. Use kubectl run --generator=run-pod/v1 or kubectl create instead.
Error from server (AlreadyExists): deployments.apps "curl" already exists
[rim@moonshot-master ~]$ kubectl run curl --image=radial/busyboxplus:curl -n kubeflow -i --tty
kubectl run --generator=deployment/apps.v1 is DEPRECATED and will be removed in a future version. Use kubectl run --generator=run-pod/v1 or kubectl create instead.
If you don't see a command prompt, try pressing enter.
[ root@curl-66bdcf564-4zfvj:/ ]$ nslookup katib-db
Server: 10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local
Name: katib-db
Address 1: 10.99.115.60 katib-db.kubeflow.svc.cluster.local
[ root@curl-66bdcf564-4zfvj:/ ]$
[ root@curl-66bdcf564-4zfvj:/ ]$ nslookup katib-db.kubeflow.svc.cluster.local
Server: 10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local
Name: katib-db.kubeflow.svc.cluster.local
Address 1: 10.99.115.60 katib-db.kubeflow.svc.cluster.local
👀1
Inside the pod, can you try telnet katib-db 3306 and see if you connect?
johnugeorge
on 26 Jul 2019
No answer to telnet katib-db 3306
kimchitsigai
on 26 Jul 2019
This seems to be the problem. If mysql db is up, it is strange that it is not able to connect.
Is this due to some extra settings of your env?
johnugeorge
on 26 Jul 2019
Something strange when I run ps -ef on the K8s worker node:
root 9834 1 1 juil.11 ? 05:52:51 /usr/bin/containerd
root 28646 9834 0 juil.25 ? 00:15:15 containerd-shim -namespace moby -workdir /var/lib/containerd/io.containerd.runtime.v1.linux/moby/61e3377efbd4c317b76de10a7ddaa7ecb006defd91532288e17241e4 -address /run/containerd/containerd.sock -containerd-binary /usr/bin/containerd -runtime-root /var/runruntime-runc
polkitd 29265 28646 0 juil.25 ? 00:02:37 mysqld --datadir /var/lib/mysql/datadir
I don't know how polkitd has become the owner of mysqld, and how to stop it...
kimchitsigai
on 26 Jul 2019
In the container (mysql:8.0.3 image for katib-db pod), I have:
[root@moonshot-master ~]# docker run -it mysql:8.0.3 /bin/bash
root@9b09a30d21bf:/# grep mysql /etc/passwd
mysql:x:999:999::/home/mysql:/bin/sh
On the host, I have:
[root@moonshot01 ~]# grep polkit /etc/passwd
polkitd:x:999:998:User for polkitd:/:/sbin/nologin
Can it be an issue if polkitd on the host and mysql in the container have the same uids ?
kimchitsigai
on 29 Jul 2019
I've reinstalled K8s 1.15, Kubeflow 0.6.1 with existing_arrikto.yaml, nfs provisioner and metallb.
The katib error has disappeared.
Closing the issue.
Thank you.
kimchitsigai
on 16 Aug 2019
Facing a similar issue with my AWS EKS installation, seeing timeout on katib-manager even though katib-db is running
wdhorton
on 8 Oct 2019
👍4
Was this page helpful?
0 / 5 - 0 ratings
Related issues
Felihong
路
8Comments
ddysher
路
7Comments
andreyvelich
路
6Comments
kylepad
路
6Comments
yeya24
路
6Comments
Most helpful comment
Facing a similar issue with my AWS EKS installation, seeing timeout on katib-manager even though katib-db is running