Kong: Kong failed to start on OCP4.3

[gyliu513]

Summary

SUMMARY_GOES_HERE

Steps To Reproduce

1. Install OCP 4.3
2. Install kong from operator hub with 0.1.0 operator
   
   
   
   • 0.1.0 operator is using registry.connect.redhat.com/kong/kong-operator1:v0.1.0-2
   
   
   • 0.1.0 operator using 1.1 kong operand
   
   
3. After operator was running, install default CR
4. found one pod failed to start.

[root@ocp43-inf ~]# oc get pods
NAME                                                              READY   STATUS             RESTARTS   AGE
etcd-operator-d7bb7f4cc-sqkkl                                     3/3     Running            336        37d
example-kong-9wm7zr9o6bojfylesx5u4kqf5-kong-6769b9649c-284s7      0/1     CrashLoopBackOff   6          22h
example-kong-9wm7zr9o6bojfylesx5u4kqf5-kong-init-migrationp64pt   0/1     Completed          0          22h
example-kong-9wm7zr9o6bojfylesx5u4kqf5-postgresql-0               1/1     Running            0          8m44s

[root@ocp43-inf ~]# oc logs -f example-kong-9wm7zr9o6bojfylesx5u4kqf5-kong-6769b9649c-284s7
chmod: /proc/self/fd/1: Permission denied
chmod: /proc/self/fd/2: Permission denied
nginx: [emerg] open() "/dev/stderr" failed (13: Permission denied)

I upgrade kong to docker.io/library/kong:1.3, still does not work.

I found there is a similar issue at https://github.com/Kong/kong/issues/4638#issuecomment-495302778 , but there is no solution and closed.

Additional Details & Logs

• Kong version ($ kong version)
• Kong debug-level startup logs ($ kong start --vv)
• Kong error logs (<KONG_PREFIX>/logs/error.log)
• Kong configuration (the output of a GET request to Kong's Admin port - see
  https://docs.konghq.com/latest/admin-api/#retrieve-node-information)
• Operating system

[hutchic]

I think this is an issue with specific configurations of OpenShift. As noted in the similar issue you linked using manage.openshift.com I was unable to reproduce.

I have a theory that it's related to add-scc-to-group.

Could I get you to try with hutchic/kong:issue-5708 which is an image I built based on Kong:1.5.1 https://github.com/Kong/docker-kong/tree/fix/ocp

[gyliu513]

Thanks @hutchic , I did some test and it failed again with some other errors:

[root@hchenxa-inf ~]# oc get pods -n default
NAME                                                              READY   STATUS             RESTARTS   AGE
example-kong-cc4qx7zb2n8jbzvthc84oddxl-kong-96d46cb78-2b6k9       0/1     CrashLoopBackOff   2          16m
example-kong-cc4qx7zb2n8jbzvthc84oddxl-kong-init-migrationjmgxw   0/1     Completed          0          16m
example-kong-cc4qx7zb2n8jbzvthc84oddxl-postgresql-0               1/1     Running            0          2m33s

```console
[root@hchenxa-inf ~]# oc describe pods -n default example-kong-cc4qx7zb2n8jbzvthc84oddxl-kong-96d46cb78-2b6k9
Name: example-kong-cc4qx7zb2n8jbzvthc84oddxl-kong-96d46cb78-2b6k9
Namespace: default
Priority: 0
Node: worker0.hchenxa.os.fyre.ibm.com/10.26.3.196
Start Time: Sun, 22 Mar 2020 20:40:59 -0700
Labels: app=kong
component=app
pod-template-hash=96d46cb78
release=example-kong-cc4qx7zb2n8jbzvthc84oddxl
Annotations: k8s.v1.cni.cncf.io/networks-status:
[{
"name": "openshift-sdn",
"interface": "eth0",
"ips": [
"10.254.16.121"
],
"dns": {},
"default-route": [
"10.254.16.1"
]
}]
Status: Running
IP: 10.254.16.121
IPs:
IP: 10.254.16.121
Controlled By: ReplicaSet/example-kong-cc4qx7zb2n8jbzvthc84oddxl-kong-96d46cb78
Init Containers:
wait-for-db:
Container ID: cri-o://f914af3388faf55db6e5b7000a305ecfda99496c7bae98e6173e809903552496
Image: hutchic/kong:issue-5708
Image ID: docker.io/hutchic/kong@sha256:79dd2cfb2b1faf98e1d3a92178952b84bd3814ecfef1e88c4feecd79e77637a5
Port:
Host Port:
Command:
/bin/sh
-c
until kong start; do echo 'waiting for db'; sleep 1; done; kong stop
State: Terminated
Reason: Completed
Exit Code: 0
Started: Sun, 22 Mar 2020 20:41:18 -0700
Finished: Sun, 22 Mar 2020 20:56:53 -0700
Ready: True
Restart Count: 0
Environment:
KONG_PROXY_ACCESS_LOG: /dev/stdout
KONG_ADMIN_ACCESS_LOG: /dev/stdout
KONG_PROXY_ERROR_LOG: /dev/stderr
KONG_ADMIN_ERROR_LOG: /dev/stderr
KONG_PG_HOST: example-kong-cc4qx7zb2n8jbzvthc84oddxl-postgresql
KONG_PG_PORT: 5432
KONG_PG_PASSWORD: Optional: false
KONG_DATABASE: postgres
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-n2gmd (ro)
Containers:
kong:
Container ID: cri-o://60da2c40c2cef949e5fd7f5a93311799f3fe29a65a7761f38df20d620371b2a2
Image: hutchic/kong:issue-5708
Image ID: docker.io/hutchic/kong@sha256:79dd2cfb2b1faf98e1d3a92178952b84bd3814ecfef1e88c4feecd79e77637a5
Ports: 8444/TCP, 8000/TCP, 8443/TCP
Host Ports: 0/TCP, 0/TCP, 0/TCP
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 2
Started: Sun, 22 Mar 2020 20:57:38 -0700
Finished: Sun, 22 Mar 2020 20:57:38 -0700
Ready: False
Restart Count: 3
Liveness: http-get https://:admin/status delay=30s timeout=5s period=30s #success=1 #failure=5
Readiness: http-get https://:admin/status delay=30s timeout=1s period=10s #success=1 #failure=5
Environment:
KONG_ADMIN_LISTEN: 0.0.0.0:8444 ssl
KONG_PROXY_LISTEN: 0.0.0.0:8000,0.0.0.0:8443 ssl
KONG_NGINX_DAEMON: off
KONG_PROXY_ACCESS_LOG: /dev/stdout
KONG_ADMIN_ACCESS_LOG: /dev/stdout
KONG_PROXY_ERROR_LOG: /dev/stderr
KONG_ADMIN_ERROR_LOG: /dev/stderr
KONG_DATABASE: postgres
KONG_PG_HOST: example-kong-cc4qx7zb2n8jbzvthc84oddxl-postgresql
KONG_PG_PORT: 5432
KONG_PG_PASSWORD: Optional: false
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-n2gmd (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
default-token-n2gmd:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-n2gmd
Optional: false
QoS Class: BestEffort
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled default-scheduler Successfully assigned default/example-kong-cc4qx7zb2n8jbzvthc84oddxl-kong-96d46cb78-2b6k9 to worker0.hchenxa.os.fyre.ibm.com
Normal Pulling 16m kubelet, worker0.hchenxa.os.fyre.ibm.com Pulling image "hutchic/kong:issue-5708"
Normal Pulled 16m kubelet, worker0.hchenxa.os.fyre.ibm.com Successfully pulled image "hutchic/kong:issue-5708"
Normal Created 16m kubelet, worker0.hchenxa.os.fyre.ibm.com Created container wait-for-db
Normal Started 16m kubelet, worker0.hchenxa.os.fyre.ibm.com Started container wait-for-db
Normal Pulled 3s (x4 over 46s) kubelet, worker0.hchenxa.os.fyre.ibm.com Container image "hutchic/kong:issue-5708" already present on machine
Normal Created 2s (x4 over 46s) kubelet, worker0.hchenxa.os.fyre.ibm.com Created container kong
Normal Started 2s (x4 over 45s) kubelet, worker0.hchenxa.os.fyre.ibm.com Started container kong
Warning BackOff 1s (x6 over 44s) kubelet, worker0.hchenxa.os.fyre.ibm.com Back-off restarting failed container

```console
[root@hchenxa-inf ~]# oc logs -f example-kong-cc4qx7zb2n8jbzvthc84oddxl-kong-96d46cb78-2b6k9 -n default
/bin/sh: can't open '/docker-entrypoint.sh': Permission denied

[hutchic]

@gyliu513 sorry that's my fault needed to chmod the entrypoint. I re-pushed hutchic/kong:issue-5708

[gyliu513]

Thanks @hutchic , it works!

Can you help fix https://github.com/operator-framework/community-operators/issues/1400 as well? I think we need a new version operator for Kong.

[hutchic]

Yup I wanted to validate the docker fix here first

[morningspace]

@hutchic Has it been officially fixed? I am seeing you've updated the git commit hash for 1.5.1, does it mean I should use 1.5.1 instead of 1.1 when install kong?

[hutchic]

Based on the tests @gyliu513 ran in this thread and the subsequent re-build of Kong 1.5.1 yes 1.5.1 should work in all openshift environments / setups

[morvencao]

Encountered same issue.
Updating image tag to 1.5.1 fixed the issue.