Che: On AWS Che fails to connect to Keycloak at startup

[sushant08]

Summary

Che pod failed to start with health check fail issue

Relevant information

✔ Verify Kubernetes API...OK
✔ 👀 Looking for an already existing Eclipse Che instance
✔ Verify if Eclipse Che is deployed into namespace "che"...it is not
✔ ✈️ Kubernetes preflight checklist
✔ Verify if kubectl is installed
✔ Verify remote kubernetes status...done.
✔ Check Kubernetes version: Found v1.13.12-eks-eb1860.
✔ Verify domain is set...set to aws.glu-ide.cloud.
↓ Check if cluster accessible [skipped]
Eclipse Che logs will be available in '/var/folders/b_/knc25xx9159_h693ljnwb_mdd__nh8/T/chectl-logs/1590069382835'
✔ Start following logs
✔ Start following Operator logs...done
✔ Start following Eclipse Che logs...done
✔ Start following Postgres logs...done
✔ Start following Keycloak logs...done
✔ Start following Plugin registry logs...done
✔ Start following Devfile registry logs...done
✔ Start following events
✔ Start following namespace events...done
❯ 🏃‍ Running the Eclipse Che operator
✔ Copying operator resources...done.
✔ 🏃‍ Running the Eclipse Che operator
✔ Copying operator resources...done.
✔ Create Namespace (che)...It already exists.
✔ Checking for pre-created TLS secret... "che-tls" secret found
✔ Checking certificate
✔ Create ServiceAccount che-operator in namespace che...done.
✔ Create Role che-operator in namespace che...done.
✔ Create ClusterRole che-operator...done.
✔ Create RoleBinding che-operator in namespace che...done.
✔ Create ClusterRoleBinding che-operator...done.
✔ Create CRD checlusters.org.eclipse.che...done.
✔ Waiting 5 seconds for the new Kubernetes resources to get flushed...done.
✔ Create deployment che-operator in namespace che...done.
✔ Create Eclipse Che cluster eclipse-che in namespace che...done.
❯ ✅ Post installation checklist
✔ PostgreSQL pod bootstrap
✔ scheduling...done.
✔ downloading images...done.
✔ starting...done.
✔ Keycloak pod bootstrap
✔ scheduling...done.
✔ downloading images...done.
✔ starting...done.
✔ Devfile registry pod bootstrap
✔ scheduling...done.
✔ downloading images...done.
✔ starting...done.
✔ Plugin registry pod bootstrap
✔ scheduling...done.
✔ downloading images...done.
✔ starting...done.
❯ Eclipse Che pod bootstrap
✔ scheduling...done.
✔ downloading images...done.
✖ starting
→ Error: connect ETIMEDOUT 3.213.183.138:443
Retrieving Eclipse Che server URL
Eclipse Che status check
› Error: Error: Error: connect ETIMEDOUT 3.213.183.138:443
› Installation failed, check logs in '/var/folders/b_/knc25xx9159_h693ljnwb_mdd__nh8/T/chectl-logs/1590069382835'
che.log
che-operator.log
che-devfile-registry.log
keycloak.log
che-plugin-registry.log
postgres.log

I see error: 37) Error injecting constructor, java.lang.RuntimeException: Exception while retrieving OpenId configuration from endpoint: https://keycloak-che.aws.my-ide.cloud/auth/realms/che/.well-known/openid-configuration

[tolusha]

Is https://keycloak-che.aws.glu-ide.cloud/auth/realms/che/.well-known/openid-configuration available from the browser

[tolusha]

kubectl get ingress -n che

[sushant08]

Is https://keycloak-che.aws.glu-ide.cloud/auth/realms/che/.well-known/openid-configuration available from the browser

Not this link is not accessible.

[sushant08]

kubectl get ingress -n che

NAMESPACE NAME HOSTS ADDRESS PORTS AGE
che che che-che.aws.glu-ide.cloud ae4a1479b9c1f11eab9ad0a1732cc3dc-0236d7fffaea8bc8.elb.us-east-1.amazonaws.com 80, 443 84s
che devfile-registry devfile-registry-che.aws.glu-ide.cloud ae4a1479b9c1f11eab9ad0a1732cc3dc-0236d7fffaea8bc8.elb.us-east-1.amazonaws.com 80, 443 29s
che keycloak keycloak-che.aws.glu-ide.cloud ae4a1479b9c1f11eab9ad0a1732cc3dc-0236d7fffaea8bc8.elb.us-east-1.amazonaws.com 80, 443 84s
che plugin-registry plugin-registry-che.aws.glu-ide.cloud ae4a1479b9c1f11eab9ad0a1732cc3dc-0236d7fffaea8bc8.elb.us-east-1.amazonaws.com 80, 443 15s

[erlansan]

I have the same issue when starting the Eclipse Che pod
> Eclipse Che pod bootstrap √ scheduling...done. √ downloading images...done. × starting → ERR_TIMEOUT: Timeout set to pod ready timeout 130000 Retrieving Eclipse Che server URL Eclipse Che status check Retrieving Keycloak admin credentials Retrieving Che self-signed CA certificate Show important messages » Error: Error: ERR_TIMEOUT: Timeout set to pod ready timeout 130000 » Installation failed, check logs in

imo it started happening after i updated my chectl from 7.12.0 to 7.13.1. Before that i could start and stop the che server without any issues
how could i fix it?

[tolusha]

@erlansan
pls, create a dedicated issue.

@sushant08
Have you used this doc?
https://www.eclipse.org/che/docs/che-7/deploying-che-on-kubernetes-on-aws/

[sushant08]

@erlansan
pls, create a dedicated issue.

@sushant08
Have you used this doc?
https://www.eclipse.org/che/docs/che-7/deploying-che-on-kubernetes-on-aws/

Mine issue has been resolved. Issue was was with ingress controller and domain. Thanks

[RickJWagner]

Hi @sushant08
Thank you for reporting this issue. Can you please elaborate a little on the adjustment that was needed to make things work?
Thanks

[sushant08]

Hi @sushant08
Thank you for reporting this issue. Can you please elaborate a little on the adjustment that was needed to make things work?
Thanks

All nodes in my ingress controller LB were unhealthy for that i had to change _externalTrafficPolicy_ to Cluster and i was trying with random domain name initially but to make it working i had to switch to domain i own.

[RickJWagner]

Thank you, @sushant08.