Okd: Code Ready Containers for OKD 4 - Progress
I have attempted to build CRC for OKD 4, and have had some success... but not complete success. I believe some of you in the community may be able to help push this across.
Current Status:
The bootstrap node starts successfully, completes its updates, and begins the installation of the single master/worker node. That installation proceeds to the point that the single node cluster is actually up and usable, to a point...
kube-apiserver, kube-controller-manager, and kubelet-bootstrap-cred-manager are stuck in a "CrashLoopBackOff" state.
oc get clusterversion -o yaml reveals issues with etcd:
Cluster operator etcd is reporting a failure: InstallerControllerDegraded: missing required resources: [configmaps: config-1,etcd-metrics-proxy-client-ca-1,etcd-metrics-proxy-serving-ca-1,etcd-peer-client-ca-1,etcd-pod-1,etcd-serving-ca-1, secrets: etcd-all-peer-1,etcd-all-serving-1,etcd-all-serving-metrics-1]
StaticPodsDegraded: pods "etcd-crc-l6qh4-master-0" not found
RevisionControllerDegraded: configmaps "etcd-pod" not found
TargetConfigControllerDegraded: "configmap/etcd-pod": at least three nodes are required to have a valid configuration
The last issue line is expected because this is a single node cluster. I'll paste the whole output below.
The problem may be related to: https://github.com/openshift/cluster-etcd-operator/issues/115 which may not have made it into FCOS yet... but that's a wild guess.
How I got here:
SNC out of the box, will not work for OKD. So, I created a fork with an OKD branch. https://github.com/cgruver/snc
It is necessary to specify the release image as well as the FCOS image:
export OPENSHIFT_INSTALL_RELEASE_IMAGE_OVERRIDE=registry.svc.ci.openshift.org/origin/release@sha256:095932dd8b717e0a392458dfcdf368146fea6e8ee538b02baf01220e8f2f590d
export OPENSHIFT_INSTALL_OS_IMAGE_OVERRIDE=https://builds.coreos.fedoraproject.org/prod/streams/testing/builds/31.20200223.2.0/x86_64/fedora-coreos-31.20200223.2.0-qemu.x86_64.qcow2.xz?sha256=ff59d2634121b995211a78f3ac058bfb4ada93a5e8e37f92199bb481b24ae06e
Additionally, you need a Quay.io pull secret which you can get with a free quay.io account.
export OPENSHIFT_PULL_SECRET=$(echo "alsdkfjjksadfjhasdfkjhjfdsuhuweruhqjhdfajkhdsf==" | base64 -d)
Where the "qwerty" string above is replaced with the base64 encoded string from using Generate Encrypted Password in your quay.io user settings, and selecting a Kubernetes Secret.
Lastly, I had to build the installer. The default values for the size of /sysroot are too small. It creates an 8GB filesystem. The single node cluster needs between 24 and 32 GB for all of the images to fit in /sysroot.
The installer was built from the fcos branch of https://github.com/openshift/installer. I modified the file: data/data/libvirt/volume/main.tf to create a 32GB /dev/vda.
resource "libvirt_volume" "coreos_orig" {
name = "${var.cluster_id}-orig"
source = var.image
pool = var.pool
}
resource "libvirt_volume" "coreos_base" {
name = "${var.cluster_id}-base"
base_volume_id = libvirt_volume.coreos_orig.id
pool = var.pool
size = 34359738368
}
Build the installer with:
TAGS=libvirt hack/build.sh
Copy the installer into the clone of https://github.com/cgruver/snc, make sure you have a 4.4 version of oc in your path, and run ./snc.sh
Note:
The first time you run ./snc.sh it will fail with a checksum error on the image... This is an issue with the installer using the uncompressed checksum against a compressed image...
cd ~/.cache/openshift-installer/image_cache
# If the image is already uncompressed:
mv 44a7f25a2a845a8960cbff29949231fc.tmp 44a7f25a2a845a8960cbff29949231fc
# If the image was not uncompressed:
xzdec 44a7f25a2a845a8960cbff29949231fc.tmp > 44a7f25a2a845a8960cbff29949231fc
Then run ./snc.sh again. The cluster should build.
Errors Observed:
# oc get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
openshift-apiserver-operator openshift-apiserver-operator-769df5656-7l24r 1/1 Running 1 53m
openshift-authentication-operator authentication-operator-7f698d667d-rwjfv 1/1 Running 4 53m
openshift-cloud-credential-operator cloud-credential-operator-76964b594d-7dc9c 1/1 Running 0 53m
openshift-cluster-machine-approver machine-approver-64ddb7d47c-wzlvx 2/2 Running 0 53m
openshift-cluster-node-tuning-operator cluster-node-tuning-operator-694778d56d-lqps7 1/1 Running 0 53m
openshift-cluster-node-tuning-operator tuned-ntf9p 1/1 Running 0 42m
openshift-cluster-version cluster-version-operator-55bfb64765-8cl7t 1/1 Running 0 53m
openshift-controller-manager-operator openshift-controller-manager-operator-75b7495976-zzm8s 1/1 Running 1 53m
openshift-controller-manager controller-manager-fcpz2 1/1 Running 0 12m
openshift-csi-snapshot-controller-operator csi-snapshot-controller-operator-7f56f48fc7-jmn4t 1/1 Running 0 53m
openshift-dns-operator dns-operator-68458bc87-79wpg 2/2 Running 0 53m
openshift-dns dns-default-6n9w9 2/2 Running 0 42m
openshift-etcd-operator etcd-operator-5c6d58b8d-rgb58 1/1 Running 1 53m
openshift-insights insights-operator-9966b8f9b-dmvpm 1/1 Running 1 53m
openshift-kube-apiserver-operator kube-apiserver-operator-6f45d85d4b-8dc9f 1/1 Running 1 53m
openshift-kube-apiserver installer-2-crc-l6qh4-master-0 0/1 Completed 0 34m
openshift-kube-apiserver installer-3-crc-l6qh4-master-0 0/1 Completed 0 13m
openshift-kube-apiserver kube-apiserver-crc-l6qh4-master-0 3/4 CrashLoopBackOff 10 13m
openshift-kube-apiserver revision-pruner-2-crc-l6qh4-master-0 0/1 Completed 0 13m
openshift-kube-controller-manager-operator kube-controller-manager-operator-584679b69f-h5zlr 1/1 Running 1 53m
openshift-kube-controller-manager installer-4-crc-l6qh4-master-0 0/1 Completed 0 34m
openshift-kube-controller-manager installer-5-crc-l6qh4-master-0 0/1 Completed 0 34m
openshift-kube-controller-manager installer-6-crc-l6qh4-master-0 0/1 Completed 0 13m
openshift-kube-controller-manager kube-controller-manager-crc-l6qh4-master-0 3/4 CrashLoopBackOff 9 13m
openshift-kube-controller-manager revision-pruner-4-crc-l6qh4-master-0 0/1 Completed 0 34m
openshift-kube-controller-manager revision-pruner-5-crc-l6qh4-master-0 0/1 Completed 0 33m
openshift-kube-controller-manager revision-pruner-6-crc-l6qh4-master-0 0/1 Completed 0 12m
openshift-kube-scheduler-operator openshift-kube-scheduler-operator-679d99cb6b-x6psv 1/1 Running 1 53m
openshift-kube-scheduler installer-2-crc-l6qh4-master-0 0/1 Completed 0 42m
openshift-kube-scheduler installer-3-crc-l6qh4-master-0 0/1 Completed 0 42m
openshift-kube-scheduler installer-4-crc-l6qh4-master-0 0/1 Completed 0 42m
openshift-kube-scheduler installer-5-crc-l6qh4-master-0 0/1 OOMKilled 0 34m
openshift-kube-scheduler openshift-kube-scheduler-crc-l6qh4-master-0 1/1 Running 0 34m
openshift-kube-scheduler revision-pruner-2-crc-l6qh4-master-0 0/1 Completed 0 42m
openshift-kube-scheduler revision-pruner-3-crc-l6qh4-master-0 0/1 Completed 0 42m
openshift-kube-scheduler revision-pruner-4-crc-l6qh4-master-0 0/1 Completed 0 34m
openshift-kube-scheduler revision-pruner-5-crc-l6qh4-master-0 0/1 Completed 0 33m
openshift-kube-storage-version-migrator-operator kube-storage-version-migrator-operator-7774ffd696-mxfvs 1/1 Running 1 53m
openshift-kube-storage-version-migrator migrator-59d5fc449f-fd8c8 1/1 Running 0 43m
openshift-machine-api machine-api-controllers-68d46646-kbv6k 4/4 Running 0 36m
openshift-machine-api machine-api-operator-857f94f8bb-79cm7 2/2 Running 0 52m
openshift-machine-config-operator etcd-quorum-guard-5fdf64dc4-qkk28 0/1 Pending 0 40m
openshift-machine-config-operator etcd-quorum-guard-5fdf64dc4-qpksq 0/1 Running 0 40m
openshift-machine-config-operator etcd-quorum-guard-5fdf64dc4-z69q2 0/1 Pending 0 40m
openshift-machine-config-operator kubelet-bootstrap-cred-manager-lv6jp 0/1 CrashLoopBackOff 11 13m
openshift-machine-config-operator machine-config-controller-75dbbddc88-vfd55 1/1 Running 0 41m
openshift-machine-config-operator machine-config-daemon-vxsbt 2/2 Running 0 44m
openshift-machine-config-operator machine-config-operator-7878d6b656-nbbjg 1/1 Running 0 53m
openshift-machine-config-operator machine-config-server-8kl4n 1/1 Running 0 41m
openshift-multus multus-admission-controller-pg6r2 1/1 Running 0 44m
openshift-multus multus-hnvkt 1/1 Running 0 47m
openshift-network-operator network-operator-66d774b4fc-qbbdq 1/1 Running 0 53m
openshift-operator-lifecycle-manager catalog-operator-5f8bb464b5-rhfxg 1/1 Running 0 53m
openshift-operator-lifecycle-manager olm-operator-5dcc98d5f4-txvjq 1/1 Running 0 53m
openshift-operator-lifecycle-manager packageserver-6544597f6b-7tshs 1/1 Running 0 2m26s
openshift-operator-lifecycle-manager packageserver-6544597f6b-mk5hb 1/1 Running 0 2m15s
openshift-sdn ovs-vmxcn 1/1 Running 0 47m
openshift-sdn sdn-74drz 1/1 Running 0 47m
openshift-sdn sdn-controller-nzvq6 1/1 Running 0 47m
openshift-service-ca-operator service-ca-operator-6c8cd8d645-pntqj 1/1 Running 1 53m
openshift-service-ca service-ca-c849959ff-v26xf 1/1 Running 0 42m
openshift-service-catalog-apiserver-operator openshift-service-catalog-apiserver-operator-765bb9ccc-jlg5n 1/1 Running 1 53m
openshift-service-catalog-controller-manager-operator openshift-service-catalog-controller-manager-operator-7df7znwhs 1/1 Running 1 53m
openshift-support support-operator-567dc658-jwm5d 1/1 Running 0 53m
```
oc get clusterversion -o yaml
apiVersion: v1
items:
- apiVersion: config.openshift.io/v1
kind: ClusterVersion
metadata:
creationTimestamp: "2020-02-25T19:58:15Z"
generation: 1
name: version
resourceVersion: "9994"
selfLink: /apis/config.openshift.io/v1/clusterversions/version
uid: 5c053b33-2af4-4fad-aa40-e97f07476660
spec:
channel: stable-4.3
clusterID: bd659010-19a9-434d-88dd-37eeb2cfa8a2
upstream: https://api.openshift.com/api/upgrades_info/v1/graph
status:
availableUpdates: null
conditions:
- lastTransitionTime: "2020-02-25T19:58:21Z"
status: "False"
type: Available - lastTransitionTime: "2020-02-25T20:20:29Z"
message: |-
Multiple errors are preventing progress:
- Cluster operator etcd is reporting a failure: InstallerControllerDegraded: missing required resources: [configmaps: config-1,etcd-metrics-proxy-client-ca-1,etcd-metrics-proxy-serving-ca-1,etcd-peer-client-ca-1,etcd-pod-1,etcd-serving-ca-1, secrets: etcd-all-peer-1,etcd-all-serving-1,etcd-all-serving-metrics-1]
StaticPodsDegraded: pods "etcd-crc-l6qh4-master-0" not found
RevisionControllerDegraded: configmaps "etcd-pod" not found
TargetConfigControllerDegraded: "configmap/etcd-pod": at least three nodes are required to have a valid configuration - Could not update oauthclient "console" (351 of 598): the server does not recognize this resource, check extension API servers
- Could not update prometheusrule "openshift-cloud-credential-operator/cloud-credential-operator-alerts" (171 of 598): the server does not recognize this resource, check extension API servers
- Could not update prometheusrule "openshift-cluster-samples-operator/samples-operator-alerts" (291 of 598): the server does not recognize this resource, check extension API servers
- Could not update role "openshift-console-operator/prometheus-k8s" (540 of 598): resource may have been deleted
- Could not update servicemonitor "openshift-apiserver-operator/openshift-apiserver-operator" (586 of 598): the server does not recognize this resource, check extension API servers
- Could not update servicemonitor "openshift-authentication-operator/authentication-operator" (507 of 598): the server does not recognize this resource, check extension API servers
- Could not update servicemonitor "openshift-cluster-machine-approver/cluster-machine-approver" (517 of 598): the server does not recognize this resource, check extension API servers
- Could not update servicemonitor "openshift-cluster-version/cluster-version-operator" (8 of 598): the server does not recognize this resource, check extension API servers
- Could not update servicemonitor "openshift-controller-manager-operator/openshift-controller-manager-operator" (592 of 598): the server does not recognize this resource, check extension API servers
- Could not update servicemonitor "openshift-dns-operator/dns-operator" (545 of 598): the server does not recognize this resource, check extension API servers
- Could not update servicemonitor "openshift-etcd-operator/etcd-operator" (548 of 598): the server does not recognize this resource, check extension API servers
- Could not update servicemonitor "openshift-image-registry/image-registry" (513 of 598): the server does not recognize this resource, check extension API servers
- Could not update servicemonitor "openshift-ingress-operator/ingress-operator" (551 of 598): the server does not recognize this resource, check extension API servers
- Could not update servicemonitor "openshift-insights/insights-operator" (404 of 598): the server does not recognize this resource, check extension API servers
- Could not update servicemonitor "openshift-kube-apiserver-operator/kube-apiserver-operator" (554 of 598): the server does not recognize this resource, check extension API servers
- Could not update servicemonitor "openshift-kube-controller-manager-operator/kube-controller-manager-operator" (562 of 598): the server does not recognize this resource, check extension API servers
- Could not update servicemonitor "openshift-kube-scheduler-operator/kube-scheduler-operator" (570 of 598): the server does not recognize this resource, check extension API servers
- Could not update servicemonitor "openshift-machine-api/cluster-autoscaler-operator" (207 of 598): the server does not recognize this resource, check extension API servers
- Could not update servicemonitor "openshift-machine-api/machine-api-operator" (575 of 598): the server does not recognize this resource, check extension API servers
- Could not update servicemonitor "openshift-machine-config-operator/machine-config-daemon" (577 of 598): the server does not recognize this resource, check extension API servers
- Could not update servicemonitor "openshift-operator-lifecycle-manager/olm-operator" (581 of 598): the server does not recognize this resource, check extension API servers
- Could not update servicemonitor "openshift-service-ca-operator/service-ca-operator" (598 of 598): the server does not recognize this resource, check extension API servers
- Could not update servicemonitor "openshift-service-catalog-apiserver-operator/openshift-service-catalog-apiserver-operator" (521 of 598): the server does not recognize this resource, check extension API servers
- Could not update servicemonitor "openshift-service-catalog-controller-manager-operator/openshift-service-catalog-controller-manager-operator" (525 of 598): the server does not recognize this resource, check extension API servers
reason: MultipleErrors
status: "True"
type: Failing
- Cluster operator etcd is reporting a failure: InstallerControllerDegraded: missing required resources: [configmaps: config-1,etcd-metrics-proxy-client-ca-1,etcd-metrics-proxy-serving-ca-1,etcd-peer-client-ca-1,etcd-pod-1,etcd-serving-ca-1, secrets: etcd-all-peer-1,etcd-all-serving-1,etcd-all-serving-metrics-1]
- lastTransitionTime: "2020-02-25T19:58:21Z"
message: 'Unable to apply 4.4.0-0.okd-2020-02-25-130747: an unknown error has
occurred'
reason: MultipleErrors
status: "True"
type: Progressing - lastTransitionTime: "2020-02-25T19:58:21Z"
message: 'Unable to retrieve available updates: currently installed version
4.4.0-0.okd-2020-02-25-130747 not found in the "stable-4.3" channel'
reason: VersionNotFound
status: "False"
type: RetrievedUpdates
desired:
force: false
image: registry.svc.ci.openshift.org/origin/release@sha256:095932dd8b717e0a392458dfcdf368146fea6e8ee538b02baf01220e8f2f590d
version: 4.4.0-0.okd-2020-02-25-130747
history: - completionTime: null
image: registry.svc.ci.openshift.org/origin/release@sha256:095932dd8b717e0a392458dfcdf368146fea6e8ee538b02baf01220e8f2f590d
startedTime: "2020-02-25T19:58:21Z"
state: Partial
verified: false
version: 4.4.0-0.okd-2020-02-25-130747
observedGeneration: 1
versionHash: 2zl1JjFlKmM=
kind: List
metadata:
resourceVersion: ""
selfLink: ""
- lastTransitionTime: "2020-02-25T19:58:21Z"
cgruver
All 10 comments
The problem may be related to: openshift/cluster-etcd-operator#115 which may not have made it into FCOS yet... but that's a wild guess.
https://github.com/openshift/cluster-etcd-operator/pull/143 fixes it and should be available in all builds since ~ Feb 14
Related - https://bugzilla.redhat.com/show_bug.cgi?id=1805034
vrutkovs
on 26 Feb 2020
@cgruver it is because of https://bugzilla.redhat.com/show_bug.cgi?id=1805034 one, we are not able to create single node cluster using ocp-4.4 on crc side also.
praveenkumar
on 26 Feb 2020
Thanks @praveenkumar !
That is exactly what I am running into. Is there a GitHub issue associated with the bugzilla issue?
I'll follow progress on that bug.
cgruver
on 26 Feb 2020
That is exactly what I am running into. Is there a GitHub issue associated with the bugzilla issue?
@cgruver right now there is no github issue afaik but we are tracking it on BZ and will update here once we have some kind of workaroud or a proper fix around it.
praveenkumar
on 26 Feb 2020
@praveenkumar @vrutkovs
Looks like a fix is being tested. https://bugzilla.redhat.com/show_bug.cgi?id=1805034
If someone here has any detail on it, I'd be willing to try my build again to test it out.
Monitoring: https://github.com/openshift/cluster-etcd-operator/pull/258, https://github.com/openshift/cluster-etcd-operator/pull/260
cgruver
on 14 Mar 2020
@cgruver We are actively testing those but this is still unfinished and etcd team is working on it, will comment on this issue once it happen.
praveenkumar
on 16 Mar 2020
There is a comment here: https://bugzilla.redhat.com/show_bug.cgi?id=1805034 made by Praveen Kumar which worked for me:
$ oc patch etcd cluster -p='{"spec": {"unsupportedConfigOverrides": {"useUnsupportedUnsafeNonHANonProductionUnstableEtcd": true}}}' --type=merge
GL guys!
PiotrKlimczak
on 12 Apr 2020
So for 4.4 on Thursday https://github.com/openshift/cluster-etcd-operator/pull/292 went in and now to test it out we need to patch the etcd resource as soon as the comes up after API server. So I follow below steps.
- Run snc.sh in one terminal
- Wait till the API server is up (This usually take 20mins)
- In different terminal try to patch
etcdresource.
$ oc patch etcd cluster -p='{"spec": {"unsupportedConfigOverrides": {"useUnsupportedUnsafeNonHANonProductionUnstableEtcd": true}}}' --type=merge
- If above fails try again. Once it successful then mostly the process will go on as expected.
praveenkumar
on 13 Apr 2020
Thanks for the update Praveen!
cgruver
on 13 Apr 2020
Issues go stale after 90d of inactivity.
Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.
If this issue is safe to close now please do so with /close.
/lifecycle stale
openshift-bot
on 6 Oct 2020
Related issues
evertmulder
路
6Comments
romankspb
路
3Comments
daFritz84
路
11Comments
jomeier
路
9Comments
andygeorge
路
7Comments
Most helpful comment
@cgruver We are actively testing those but this is still unfinished and etcd team is working on it, will comment on this issue once it happen.