Origin: failed to "StartContainer" for "mysql" with CrashLoopBackOff

[magick93]

After failing to launch mysql using openshift web console (https://github.com/openshift/origin/issues/13478), I tried using cli, and, again it failed, but this time with a different error.

Version

# oc version
oc v1.5.0-alpha.2+e4b43ee
kubernetes v1.5.2+43a9be4
features: Basic-Auth GSSAPI Kerberos SPNEGO


openshift v1.5.0-alpha.3+cf7e336
kubernetes v1.5.2+43a9be4

Steps To Reproduce

1. # oc new-app mysql

Current Result

# oc describe pod mysql-1-5r550
Name:           mysql-1-5r550
Namespace:      priceinsight
Security Policy:    restricted
Start Time:     Tue, 21 Mar 2017 11:57:24 +0100
Labels:         app=mysql
            deployment=mysql-1
            deploymentconfig=mysql
Status:         Running
IP:         172.17.0.9
Controllers:        ReplicationController/mysql-1
Containers:
  mysql:
    Container ID:   docker://4f913435cadb174b50acbaf40974fb44914202430f851f77cc511a31ff8edc56
    Image:      centos/mysql-57-centos7@sha256:604daedd6fbca3e4826924781e66325252891d5ebc9776f420a351ee20afaf79
    Image ID:       docker-pullable://docker.io/centos/mysql-57-centos7@sha256:604daedd6fbca3e4826924781e66325252891d5ebc9776f420a351ee20afaf79
    Port:       3306/TCP
    State:      Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Tue, 21 Mar 2017 12:01:37 +0100
      Finished:     Tue, 21 Mar 2017 12:01:37 +0100
    Ready:      False
    Restart Count:  5
    Volume Mounts:
      /var/lib/mysql/data from mysql-volume-1 (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-c8q23 (ro)
    Environment Variables:  <none>
Conditions:
  Type      Status
  Initialized   True 
  Ready     False 
  PodScheduled  True 
Volumes:
  mysql-volume-1:
    Type:   EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium: 
  default-token-c8q23:
    Type:   Secret (a volume populated by a Secret)
    SecretName: default-token-c8q23
QoS Class:  BestEffort
Tolerations:    <none>
Events:
  FirstSeen LastSeen    Count   From            SubObjectPath       Type        Reason      Message
  --------- --------    -----   ----            -------------       --------    ------      -------
  4m        4m      1   {default-scheduler }                Normal      Scheduled   Successfully assigned mysql-1-5r550 to my-ip-address
  4m        4m      1   {kubelet my-ip-address} spec.containers{mysql}  Normal      Pulling     pulling image "centos/mysql-57-centos7@sha256:604daedd6fbca3e4826924781e66325252891d5ebc9776f420a351ee20afaf79"
  3m        3m      1   {kubelet my-ip-address} spec.containers{mysql}  Normal      Pulled      Successfully pulled image "centos/mysql-57-centos7@sha256:604daedd6fbca3e4826924781e66325252891d5ebc9776f420a351ee20afaf79"
  3m        3m      1   {kubelet my-ip-address} spec.containers{mysql}  Normal      Created     Created container with docker id 6a945b22ef12; Security:[seccomp=unconfined]
  3m        3m      1   {kubelet my-ip-address} spec.containers{mysql}  Normal      Started     Started container with docker id 6a945b22ef12
  3m        3m      1   {kubelet my-ip-address} spec.containers{mysql}  Normal      Created     Created container with docker id f94eef83f181; Security:[seccomp=unconfined]
  3m        3m      1   {kubelet my-ip-address} spec.containers{mysql}  Normal      Started     Started container with docker id f94eef83f181
  3m        3m      2   {kubelet my-ip-address}             Warning     FailedSync  Error syncing pod, skipping: failed to "StartContainer" for "mysql" with CrashLoopBackOff: "Back-off 10s restarting failed container=mysql pod=mysql-1-5r550_priceinsight(2a2ab012-0e25-11e7-840b-002421dde3d7)"

  3m    3m  1   {kubelet my-ip-address} spec.containers{mysql}  Normal  Created     Created container with docker id edd8968ae442; Security:[seccomp=unconfined]
  3m    3m  1   {kubelet my-ip-address} spec.containers{mysql}  Normal  Started     Started container with docker id edd8968ae442
  3m    2m  3   {kubelet my-ip-address}             Warning FailedSync  Error syncing pod, skipping: failed to "StartContainer" for "mysql" with CrashLoopBackOff: "Back-off 20s restarting failed container=mysql pod=mysql-1-5r550_priceinsight(2a2ab012-0e25-11e7-840b-002421dde3d7)"

  2m    2m  1   {kubelet my-ip-address} spec.containers{mysql}  Normal  Created     Created container with docker id 954437e35694; Security:[seccomp=unconfined]
  2m    2m  1   {kubelet my-ip-address} spec.containers{mysql}  Normal  Started     Started container with docker id 954437e35694
  2m    1m  5   {kubelet my-ip-address}             Warning FailedSync  Error syncing pod, skipping: failed to "StartContainer" for "mysql" with CrashLoopBackOff: "Back-off 40s restarting failed container=mysql pod=mysql-1-5r550_priceinsight(2a2ab012-0e25-11e7-840b-002421dde3d7)"

  1m    1m  1   {kubelet my-ip-address} spec.containers{mysql}  Normal  Created     Created container with docker id 8a71589a1f62; Security:[seccomp=unconfined]
  1m    1m  1   {kubelet my-ip-address} spec.containers{mysql}  Normal  Started     Started container with docker id 8a71589a1f62
  1m    29s 7   {kubelet my-ip-address}             Warning FailedSync  Error syncing pod, skipping: failed to "StartContainer" for "mysql" with CrashLoopBackOff: "Back-off 1m20s restarting failed container=mysql pod=mysql-1-5r550_priceinsight(2a2ab012-0e25-11e7-840b-002421dde3d7)"

  3m    18s 5   {kubelet my-ip-address} spec.containers{mysql}  Normal  Pulled      Container image "centos/mysql-57-centos7@sha256:604daedd6fbca3e4826924781e66325252891d5ebc9776f420a351ee20afaf79" already present on machine
  14s   14s 1   {kubelet my-ip-address} spec.containers{mysql}  Normal  Created     Created container with docker id 4f913435cadb; Security:[seccomp=unconfined]
  14s   14s 1   {kubelet my-ip-address} spec.containers{mysql}  Normal  Started     Started container with docker id 4f913435cadb
  3m    8s  19  {kubelet my-ip-address} spec.containers{mysql}  Warning BackOff     Back-off restarting failed docker container
  13s   8s  2   {kubelet my-ip-address}             Warning FailedSync  Error syncing pod, skipping: failed to "StartContainer" for "mysql" with CrashLoopBackOff: "Back-off 2m40s restarting failed container=mysql pod=mysql-1-5r550_priceinsight(2a2ab012-0e25-11e7-840b-002421dde3d7)"

Expected Result

Expect to be able to run the mysql container

Additional Information

# oc adm diagnostics
[Note] Determining if client configuration exists for client/cluster diagnostics
Info:  Successfully read a client config file at '/root/.kube/config'
Info:  Using context for cluster-admin access: 'priceinsight/my-ip-address:8443/system:admin'

[Note] Running diagnostic: ConfigContexts[myproject/my-ip-address:8443/admin]
       Description: Validate client config context is complete and has connectivity

ERROR: [DCli0014 from diagnostic ConfigContexts@openshift/origin/pkg/diagnostics/client/config_contexts.go:285]
       For client config context 'myproject/my-ip-address:8443/admin':
       The server URL is 'https://my-ip-address:8443'
       The user authentication is 'admin/my-ip-address:8443'
       The current project is 'myproject'
       (*errors.StatusError) the server has asked for the client to provide credentials (get projects)

       This means that when we tried to make a request to the master API
       server, the request required credentials that were not presented. This
       can happen with an expired or invalid authentication token. Try logging
       in with this user again.

[Note] Running diagnostic: ConfigContexts[default/openshift-hughestech-co:8443/system:admin]
       Description: Validate client config context is complete and has connectivity

ERROR: [DCli0001 from diagnostic ConfigContexts@openshift/origin/pkg/diagnostics/client/config_contexts.go:209]
       The client config context 'default/openshift-hughestech-co:8443/system:admin' is unusable:
        Client config context 'default/openshift-hughestech-co:8443/system:admin' has a cluster 'openshift-hughestech-co:8443' which is not defined.

[Note] Running diagnostic: ConfigContexts[myproject/my-ip-address:8443/developer]
       Description: Validate client config context is complete and has connectivity

ERROR: [DCli0014 from diagnostic ConfigContexts@openshift/origin/pkg/diagnostics/client/config_contexts.go:285]
       For client config context 'myproject/my-ip-address:8443/developer':
       The server URL is 'https://my-ip-address:8443'
       The user authentication is 'developer/my-ip-address:8443'
       The current project is 'myproject'
       (*errors.StatusError) the server has asked for the client to provide credentials (get projects)

       This means that when we tried to make a request to the master API
       server, the request required credentials that were not presented. This
       can happen with an expired or invalid authentication token. Try logging
       in with this user again.

[Note] Running diagnostic: ConfigContexts[myproject/my-ip-address:8443/system:admin]
       Description: Validate client config context is complete and has connectivity

Info:  For client config context 'myproject/my-ip-address:8443/system:admin':
       The server URL is 'https://my-ip-address:8443'
       The user authentication is 'system:admin/my-ip-address:8443'
       The current project is 'myproject'
       Successfully requested project list; has access to project(s):
         [default default-shiftwork kube-system myproject openshift openshift-infra priceinsight user-secrets-source-admin]

[Note] Running diagnostic: ConfigContexts[default/openshift-teammachine-us:8443/system:admin]
       Description: Validate client config context is complete and has connectivity

ERROR: [DCli0001 from diagnostic ConfigContexts@openshift/origin/pkg/diagnostics/client/config_contexts.go:209]
       The client config context 'default/openshift-teammachine-us:8443/system:admin' is unusable:
        Client config context 'default/openshift-teammachine-us:8443/system:admin' has a cluster 'openshift-teammachine-us:8443' which is not defined.

[Note] Running diagnostic: DiagnosticPod
       Description: Create a pod to run diagnostics from the application standpoint

Info:  Output from the diagnostic pod (image openshift/origin-deployer:v1.5.0-alpha.2):
       [Note] Running diagnostic: PodCheckAuth
              Description: Check that service account credentials authenticate as expected

       Info:  Service account token successfully authenticated to master
       Info:  Service account token was authenticated by the integrated registry.
       [Note] Running diagnostic: PodCheckDns
              Description: Check that DNS within a pod works as expected

       [Note] Summary of diagnostics execution (version v1.5.0-alpha.2+e4b43ee):
       [Note] Completed with no errors or warnings seen.

[Note] Running diagnostic: NetworkCheck
       Description: Create a pod on all schedulable nodes and run network diagnostics from the application standpoint

WARN:  [DNet2002 from diagnostic NetworkCheck@openshift/origin/pkg/diagnostics/network/run_pod.go:82]
       Skipping network diagnostics check. Reason: Not using openshift network plugin.

[Note] Skipping diagnostic: AggregatedLogging
       Description: Check aggregated logging integration for proper configuration
       Because: No master config file was provided

[Note] Running diagnostic: ClusterRegistry
       Description: Check that there is a working Docker registry

[Note] Running diagnostic: ClusterRoleBindings
       Description: Check that the default ClusterRoleBindings are present and contain the expected subjects

Info:  clusterrolebinding/cluster-admins has more subjects than expected.

       Use the `oadm policy reconcile-cluster-role-bindings` command to update the role binding to remove extra subjects.

Info:  clusterrolebinding/cluster-admins has extra subject {ServiceAccount default pvinstaller    }.
Info:  clusterrolebinding/cluster-admins has extra subject {User  admin    }.
Info:  clusterrolebinding/cluster-admins has extra subject {ServiceAccount default fabric8    }.
Info:  clusterrolebinding/cluster-admins has extra subject {ServiceAccount default jenkins    }.
Info:  clusterrolebinding/cluster-admins has extra subject {ServiceAccount default configmapcontroller    }.
Info:  clusterrolebinding/cluster-admins has extra subject {ServiceAccount default exposecontroller    }.

Info:  clusterrolebinding/cluster-readers has more subjects than expected.

       Use the `oadm policy reconcile-cluster-role-bindings` command to update the role binding to remove extra subjects.

Info:  clusterrolebinding/cluster-readers has extra subject {ServiceAccount openshift-infra heapster    }.
Info:  clusterrolebinding/cluster-readers has extra subject {ServiceAccount default metrics    }.
Info:  clusterrolebinding/cluster-readers has extra subject {ServiceAccount default fluentd    }.
Info:  clusterrolebinding/cluster-readers has extra subject {SystemGroup  system:serviceaccounts    }.

ERROR: [CRBD1001 from diagnostic ClusterRoleBindings@openshift/origin/pkg/diagnostics/cluster/rolebindings.go:76]
       clusterrolebinding/system:routers is missing.

       Use the `oadm policy reconcile-cluster-role-bindings` command to create the role binding.

ERROR: [CRBD1001 from diagnostic ClusterRoleBindings@openshift/origin/pkg/diagnostics/cluster/rolebindings.go:76]
       clusterrolebinding/system:registrys is missing.

       Use the `oadm policy reconcile-cluster-role-bindings` command to create the role binding.

[Note] Running diagnostic: ClusterRoles
       Description: Check that the default ClusterRoles are present and contain the expected permissions

ERROR: [CRD1005 from diagnostic ClusterRoles@openshift/origin/pkg/diagnostics/cluster/roles.go:117]
       clusterrole/cluster-reader has changed and the existing role does not have enough permissions.

       Use the 'oadm policy reconcile-cluster-roles' command to update the role.
       For example,

         $ oadm policy reconcile-cluster-roles \
                --additive-only=true --confirm

Info:  clusterrole/cluster-reader is missing permission PolicyRule{Verbs:["watch"], APIGroups:["apps"], Resources:["petsets"]}.
Info:  clusterrole/cluster-reader is missing permission PolicyRule{Verbs:["get"], APIGroups:["apps"], Resources:["petsets"]}.
Info:  clusterrole/cluster-reader is missing permission PolicyRule{Verbs:["list"], APIGroups:["apps"], Resources:["petsets"]}.
Info:  clusterrole/cluster-reader is missing permission PolicyRule{Verbs:["get"], APIGroups:["apps"], Resources:["petsets/status"]}.
Info:  clusterrole/cluster-reader is missing permission PolicyRule{Verbs:["list"], APIGroups:["apps"], Resources:["petsets/status"]}.
Info:  clusterrole/cluster-reader is missing permission PolicyRule{Verbs:["watch"], APIGroups:["apps"], Resources:["petsets/status"]}.

ERROR: [CRD1005 from diagnostic ClusterRoles@openshift/origin/pkg/diagnostics/cluster/roles.go:117]
       clusterrole/admin has changed and the existing role does not have enough permissions.

       Use the 'oadm policy reconcile-cluster-roles' command to update the role.
       For example,

         $ oadm policy reconcile-cluster-roles \
                --additive-only=true --confirm

Info:  clusterrole/admin is missing permission PolicyRule{Verbs:["list"], APIGroups:["apps"], Resources:["petsets"]}.
Info:  clusterrole/admin is missing permission PolicyRule{Verbs:["watch"], APIGroups:["apps"], Resources:["petsets"]}.
Info:  clusterrole/admin is missing permission PolicyRule{Verbs:["create"], APIGroups:["apps"], Resources:["petsets"]}.
Info:  clusterrole/admin is missing permission PolicyRule{Verbs:["update"], APIGroups:["apps"], Resources:["petsets"]}.
Info:  clusterrole/admin is missing permission PolicyRule{Verbs:["patch"], APIGroups:["apps"], Resources:["petsets"]}.
Info:  clusterrole/admin is missing permission PolicyRule{Verbs:["delete"], APIGroups:["apps"], Resources:["petsets"]}.
Info:  clusterrole/admin is missing permission PolicyRule{Verbs:["deletecollection"], APIGroups:["apps"], Resources:["petsets"]}.
Info:  clusterrole/admin is missing permission PolicyRule{Verbs:["get"], APIGroups:["apps"], Resources:["petsets"]}.

ERROR: [CRD1005 from diagnostic ClusterRoles@openshift/origin/pkg/diagnostics/cluster/roles.go:117]
       clusterrole/edit has changed and the existing role does not have enough permissions.

       Use the 'oadm policy reconcile-cluster-roles' command to update the role.
       For example,

         $ oadm policy reconcile-cluster-roles \
                --additive-only=true --confirm

Info:  clusterrole/edit is missing permission PolicyRule{Verbs:["watch"], APIGroups:["apps"], Resources:["petsets"]}.
Info:  clusterrole/edit is missing permission PolicyRule{Verbs:["create"], APIGroups:["apps"], Resources:["petsets"]}.
Info:  clusterrole/edit is missing permission PolicyRule{Verbs:["update"], APIGroups:["apps"], Resources:["petsets"]}.
Info:  clusterrole/edit is missing permission PolicyRule{Verbs:["patch"], APIGroups:["apps"], Resources:["petsets"]}.
Info:  clusterrole/edit is missing permission PolicyRule{Verbs:["delete"], APIGroups:["apps"], Resources:["petsets"]}.
Info:  clusterrole/edit is missing permission PolicyRule{Verbs:["deletecollection"], APIGroups:["apps"], Resources:["petsets"]}.
Info:  clusterrole/edit is missing permission PolicyRule{Verbs:["get"], APIGroups:["apps"], Resources:["petsets"]}.
Info:  clusterrole/edit is missing permission PolicyRule{Verbs:["list"], APIGroups:["apps"], Resources:["petsets"]}.

ERROR: [CRD1005 from diagnostic ClusterRoles@openshift/origin/pkg/diagnostics/cluster/roles.go:117]
       clusterrole/view has changed and the existing role does not have enough permissions.

       Use the 'oadm policy reconcile-cluster-roles' command to update the role.
       For example,

         $ oadm policy reconcile-cluster-roles \
                --additive-only=true --confirm

Info:  clusterrole/view is missing permission PolicyRule{Verbs:["get"], APIGroups:["apps"], Resources:["petsets"]}.
Info:  clusterrole/view is missing permission PolicyRule{Verbs:["list"], APIGroups:["apps"], Resources:["petsets"]}.
Info:  clusterrole/view is missing permission PolicyRule{Verbs:["watch"], APIGroups:["apps"], Resources:["petsets"]}.

Info:  clusterrole/system:node has changed, but the existing role has more permissions than the new role.

       If you can confirm that the extra permissions are not required, you may use the
       'oadm policy reconcile-cluster-roles' command to update the role to reduce permissions.
       For example,

         $ oadm policy reconcile-cluster-roles \
                --additive-only=false --confirm

Info:  clusterrole/system:node has extra permission PolicyRule{Verbs:["create"], APIGroups:["certificates.k8s.io"], Resources:["certificatesigningrequests"]}.
Info:  clusterrole/system:node has extra permission PolicyRule{Verbs:["get"], APIGroups:["certificates.k8s.io"], Resources:["certificatesigningrequests"]}.

Info:  clusterrole/system:sdn-reader has changed, but the existing role has more permissions than the new role.

       If you can confirm that the extra permissions are not required, you may use the
       'oadm policy reconcile-cluster-roles' command to update the role to reduce permissions.
       For example,

         $ oadm policy reconcile-cluster-roles \
                --additive-only=false --confirm

Info:  clusterrole/system:sdn-reader has extra permission PolicyRule{Verbs:["get"], APIGroups:["extensions"], Resources:["networkpolicies"]}.
Info:  clusterrole/system:sdn-reader has extra permission PolicyRule{Verbs:["list"], APIGroups:["extensions"], Resources:["networkpolicies"]}.
Info:  clusterrole/system:sdn-reader has extra permission PolicyRule{Verbs:["watch"], APIGroups:["extensions"], Resources:["networkpolicies"]}.

ERROR: [CRD1002 from diagnostic ClusterRoles@openshift/origin/pkg/diagnostics/cluster/roles.go:100]
       clusterrole/system:pet-set-controller is missing.

       Use the 'oadm policy reconcile-cluster-roles' command to create the role. For example,

         $ oadm policy reconcile-cluster-roles \
                --additive-only=true --confirm

[Note] Running diagnostic: ClusterRouterName
       Description: Check there is a working router

[Note] Running diagnostic: MasterNode
       Description: Check if master is also running node (for Open vSwitch)

Info:  Found a node with same IP as master: my-ip-address

[Note] Skipping diagnostic: MetricsApiProxy
       Description: Check the integrated heapster metrics can be reached via the API proxy
       Because: The heapster service does not exist in the openshift-infra project at this time,
       so it is not available for the Horizontal Pod Autoscaler to use as a source of metrics.

[Note] Running diagnostic: NodeDefinitions
       Description: Check node records on master

[Note] Skipping diagnostic: ServiceExternalIPs
       Description: Check for existing services with ExternalIPs that are disallowed by master config
       Because: No master config file was detected

[Note] Summary of diagnostics execution (version v1.5.0-alpha.2+e4b43ee):
[Note] Warnings seen: 1
[Note] Errors seen: 11

[magick93]

I was able to solve this by adding the required env vars to DC.

[mcwienczek]

could you elaborate what env vars were there? Because I am having very similar problem but I am providing MYSQL_ROOT_PASSWORD. Don't know therefore if I'm missing some env var or simply my problem is different? Thanks a lot for help.