Crc: [BUG] OpenShift cluster seems to crash and gets unresponsive

[diggnsaegg]

General information

• OS: CentOS 8
• Hypervisor: KVM
• Did you run crc setup before starting it (Yes/No)? yes

CRC version

crc version: 1.10.0+9025021
OpenShift version: 4.4.3 (embedded in binary)

CRC status

CRC VM:          Running
OpenShift:       Degraded
Disk Usage:      15.35GB of 32.72GB (Inside the CRC VM)
Cache Usage:     11.9GB
Cache Directory: /home/dan/.crc/cache

CRC config

generates no output

Host Operating System

NAME="CentOS Linux"
VERSION="8 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="8"
PLATFORM_ID="platform:el8"
PRETTY_NAME="CentOS Linux 8 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:8"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"
CENTOS_MANTISBT_PROJECT="CentOS-8"
CENTOS_MANTISBT_PROJECT_VERSION="8"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="8"

Steps to reproduce

1. crc start and wait until the cluster is up
2. oc login -u developer -p developer https://api.crc.testing:6443
3. 
   
   
   4.
   

Expected

Succesfully login into the cluster

Actual

oc login -u developer -p developer https://api.crc.testing:6443
error: EOF

oc login -u developer -p developer https://api.crc.testing:6443
Error from server (InternalError): Internal error occurred: unexpected response: 400



md5-aa4d4445ac164e5cbe63973ce1c94785






Sometimes it says TLS handshake error or timeout (504). I was able to login and deploy some stuff, but with or without a deployment or pods running the cluster keeps crashing and restarting all the time



### Logs

https://gist.github.com/diggnsaegg/b943af93147999d267992630d9cc85ae



### System



md5-53531a066a783adb0582bfdb8e5f8cf2



```bash
free -mh
              total        used        free      shared  buff/cache   available
Mem:           15Gi        10Gi       166Mi        99Mi       5.0Gi       4.7Gi
Swap:         5.9Gi       248Mi       5.7Gi

[praveenkumar]

As per your crc status --log-level debug I can see some of the operator are still reconcile state , how long you waited, does the crc status change to Running from degraded state.

time="2020-05-05T15:55:06+02:00" level=debug msg="authentication operator is still progressing, Reason: _OAuthServerDeploymentNotReady"
time="2020-05-05T15:55:06+02:00" level=debug msg="etcd operator is still progressing, Reason: NodeInstaller"
time="2020-05-05T15:55:06+02:00" level=debug msg="openshift-apiserver operator is degraded, Reason: APIServerDeployment_UnavailablePod"
time="2020-05-05T15:55:06+02:00" level=debug msg="openshift-apiserver operator not available, Reason: APIServerDeployment_NoAPIServerPod::APIServices_PreconditionNotReady"
time="2020-05-05T15:55:06+02:00" level=debug msg="openshift-controller-manager operator is still progressing, Reason: "
time="2020-05-05T15:55:06+02:00" level=debug msg="operator-lifecycle-manager-packageserver operator not available, Reason: "
time="2020-05-05T15:55:06+02:00" level=debug msg="operator-lifecycle-manager-packageserver operator is still progressing, Reason: "

[diggnsaegg]

I waited about 10 minutes, strange thing is that it worked like a charm and then it might be crashing somewhere, a pod or something. Right now it is working.

[diggnsaegg]

Now this happens after I just wanted to scale up a pod via browser:
{"error":"server_error","error_description":"The authorization server encountered an unexpected condition that prevented it from fulfilling the request.","state":"f23343a5"}

Also I am logged out of oc in my terminal and relogin ends up in unexpected response, error 400. So it seems a pod just crashed necessary for openshift I guess.

[praveenkumar]

@diggnsaegg make sure if you are trying to deploy/develop some resource hungry applications then try to increase the memory and try again.

$ crc delete
$ crc config set memory 10240 <= sets 10Gb, you can set more if that needed.
$ crc start --log-level debug

[diggnsaegg]

I gave it 12 GiB RAM but it seems that the setup hangs right now at this point:

tail -f crc.log time="2020-05-05T17:05:00+02:00" level=debug msg="authentication operator is still progressing, Reason: _OAuthServerDeploymentNotReady" time="2020-05-05T17:05:00+02:00" level=debug msg="image-registry operator not available, Reason: NoReplicasAvailable" time="2020-05-05T17:05:00+02:00" level=debug msg="image-registry operator is still progressing, Reason: DeploymentNotCompleted" time="2020-05-05T17:05:00+02:00" level=debug msg="network operator is still progressing, Reason: Deploying" time="2020-05-05T17:05:00+02:00" level=debug msg="openshift-apiserver operator is degraded, Reason: APIServerDeployment_UnavailablePod" time="2020-05-05T17:05:00+02:00" level=debug msg="openshift-apiserver operator not available, Reason: APIServerDeployment_NoAPIServerPod::APIServices_PreconditionNotReady" time="2020-05-05T17:05:00+02:00" level=debug msg="operator-lifecycle-manager-packageserver operator not available, Reason: " time="2020-05-05T17:05:00+02:00" level=debug msg="operator-lifecycle-manager-packageserver operator is still progressing, Reason: " time="2020-05-05T17:05:00+02:00" level=debug msg="About to run SSH command:\ndf -B1 --output=size,used,target /sysroot | tail -1" time="2020-05-05T17:05:09+02:00" level=debug msg="SSH cmd err, output: <nil>: 32719437824 13497790464 /sysroot\n"

[praveenkumar]

seems that the setup hangs right now at this point

setup doesn't hang but looks like operators are taking more time than usual to become in the sane state which is unexpected, by any chance you are using a baremetal system as host not a CentOS-8 VM ?

[diggnsaegg]

I run a (i guess) powerful virtualbox CentOS 8 VM yes, 6 CPUs, 16 GiB RAM, 150 GB Storage on SSD.
So I think running crc in a VM is not recommened? :)

[praveenkumar]

So I think running crc in a VM is not recommened? :)

Definitely no, what is your host system and why not run crc directly from the host?

[diggnsaegg]

I run Windows 10 but wanted to avoid Hyper-V as I don't really like it, but I think I need to give it a shot then :/

[diggnsaegg]

Seems you're right ;) After a short hassle with Hyper-V everything works fine, thanks!