Okd: UPI Bare-Metal Air-Gapped install: x509: certificate signed by unknown authority

[cgruver]

I have successfully created a mirrored repository.

export OCP_RELEASE=4.4.0-0.okd-2020-03-11-174228
export LOCAL_REGISTRY=nexus.oscluster.clgcom.org:5002
export LOCAL_REPOSITORY=origin
export PRODUCT_REPO=origin
export LOCAL_SECRET_JSON=${OKD4_LAB_PATH}/pull-secret.json
export RELEASE_NAME=release

oc adm -a ${LOCAL_SECRET_JSON} release mirror --from=registry.svc.ci.openshift.org/${PRODUCT_REPO}/${RELEASE_NAME}:${OCP_RELEASE} --to=${LOCAL_REGISTRY}/${LOCAL_REPOSITORY} --to-release-image=${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}:${OCP_RELEASE}

The mirror completed successfully.

  - mirrors:
    - nexus.oscluster.clgcom.org:5002/origin
    source: registry.svc.ci.openshift.org/origin/4.4-2020-03-11-174228
  - mirrors:
    - nexus.oscluster.clgcom.org:5002/origin
    source: registry.svc.ci.openshift.org/origin/release

I then modified DNS to sinkhole registry.svc.ci.openshift.org

The cert used for the Nexus registry is included in the install-config.yaml

The install does not appear to be accepting the registry cert.

Mar 12 14:39:16 okd4-bootstrap bootkube.sh[1744]: Error: unable to pull registry.svc.ci.openshift.org/origin/release@sha256:cf9ea2648ea9b7e69a8ce27265b0b9855000b3e1b9caec0d75dfb24d1d619bd6: unable to pull image: Error initializing source docker://registry.svc.ci.openshift.org/origin/release@sha256:cf9ea2648ea9b7e69a8ce27265b0b9855000b3e1b9caec0d75dfb24d1d619bd6: (Mirrors also failed: [nexus.oscluster.clgcom.org:5002/origin@sha256:cf9ea2648ea9b7e69a8ce27265b0b9855000b3e1b9caec0d75dfb24d1d619bd6: error pinging docker registry nexus.oscluster.clgcom.org:5002: Get https://nexus.oscluster.clgcom.org:5002/v2/: x509: certificate signed by unknown authority]): registry.svc.ci.openshift.org/origin/release@sha256:cf9ea2648ea9b7e69a8ce27265b0b9855000b3e1b9caec0d75dfb24d1d619bd6: error pinging docker registry registry.svc.ci.openshift.org: Get https://registry.svc.ci.openshift.org/v2/: dial tcp: lookup registry.svc.ci.openshift.org on 10.11.11.10:53: no such host

Relevant section from install-config.yaml

additionalTrustBundle: |
  -----BEGIN CERTIFICATE-----
  MIIEKzCCAxOgAwIBAgIEYpvdMTANBgkqhkiG9w0BAQsFADB5MQswCQYDVQQGEwJV
  UzEUMBIGA1UECBMLVW5zcGVjaWZpZWQxFDASBgNVBAcTC1Vuc3BlY2lmaWVkMREw
  DwYDVQQKEwhTb25hdHlwZTErMCkGA1UEAwwiKi5vc2NsdXN0ZXIuY2xnY29tLm9y
  ZyBPVT1Tb25hdHlwZTAeFw0xOTAzMDkyMDQzMTlaFw0zMjExMTUyMDQzMTlaMHkx
  CzAJBgNVBAYTAlVTMRQwEgYDVQQIEwtVbnNwZWNpZmllZDEUMBIGA1UEBxMLVW5z
  cGVjaWZpZWQxETAPBgNVBAoTCFNvbmF0eXBlMSswKQYDVQQDDCIqLm9zY2x1c3Rl
  ci5jbGdjb20ub3JnIE9VPVNvbmF0eXBlMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8A
  MIIBCgKCAQEAgSG+guPre9klfsW2Wpb1As9WhwZYnZA91j59qRz39VjArww+iIUW
  jvqGnh/u/RmSXyFT8LUoD0pA5gn+1IrguSrQiaPEaLNv0QyOmDr8PFFHS4Rm9+wL
  p6jjxzR7vz6IEaTd1A4quoewAERocBk1XLv8ZZS1k5smA0+W9QujApYfDJwFxJKx
  z4E9t96gqbKiNet3Nm0yW8mwEiWg8P5CbKO+ToeQZi2RMLnm/lAYDQ4v2iM1w0wQ
  qcqpiXPRDfyMcoDgl53zyv1b4SBBtNfhnIuuG2a33QTU0oaTHh6KWdNq94hqHacd
  npy0p9PyMMm8eTYHr2WWuH6X2RofYo+ruQIDAQABo4G6MIG3MIGVBgNVHREEgY0w
  gYqCGm5leHVzLm9zY2x1c3Rlci5jbGdjb20ub3JnghhjbG0ub3NjbHVzdGVyLmNs
  Z2NvbS5vcmeCGXJlcG8ub3NjbHVzdGVyLmNsZ2NvbS5vcmeCEG9zYy1jb250cm9s
  bGVyMDGCJW9zYy1jb250cm9sbGVyMDEub3NjbHVzdGVyLmNsZ2NvbS5vcmcwHQYD
  VR0OBBYEFJwfPN/ru9siT4eoBr5THCE7bUwOMA0GCSqGSIb3DQEBCwUAA4IBAQBW
  7uiQwsLENgvZyQkRcv3+kMgkEfqwqEM80pU/BxGVJNc8XwaNES7r0mdKQ95NxgrD
  3PI6Sx4+bdmMFyQTqEaZu1EkcermhFey7mVpbA/6LBvDyYm/3c66MKPBhfgjcjJ+
  Ah+sMC98ZlDCRoh71J5BioId5SG3ee8aq4HO9UKn7NpkfMlif02QwQxZPCrVmY4g
  rQXuurfGHFNwbaiiEKUl4lfEG3ZPOJDV3CSpDke6wgQBaNwaTMauJsahOGvNFE+B
  AM5tkrXQbG1ripYO2m0uhqHiUc/Y3g6rNtjALZ3fftFbledpLjQXhD+wqdwDCMtX
  ezLRWJQLBjr9WwDw/uy9
  -----END CERTIFICATE-----
imageContentSources:
  - mirrors:
    - nexus.oscluster.clgcom.org:5002/origin
    source: registry.svc.ci.openshift.org/origin/4.4-2020-03-11-174228
  - mirrors:
    - nexus.oscluster.clgcom.org:5002/origin
    source: registry.svc.ci.openshift.org/origin/release

[cgruver]

Logs:

log-bundle-20200312110554.tar.gz

[cgruver]

Attempted again with the quay.io container as described in the documentation at: https://docs.okd.io/latest/installing/install_config/installing-restricted-networks-preparations.html#installing-restricted-networks-preparations

The mirror completed successfully, but the install still fails.

Getting a different error...

Mar 13 21:53:51 okd4-bootstrap systemd[1]: Started Bootstrap a Kubernetes cluster.
Mar 13 21:53:51 okd4-bootstrap bootkube.sh[2704]: Error: error getting image "registry.svc.ci.openshift.org/origin/release@sha256:cf9ea2648ea9b7e69a8ce27265b0b9855000b3e1b9caec0d75dfb24d1d619bd6": unable to find 'registry.svc.ci.openshift.org/origin/release@sha256:cf9ea2648ea9b7e69a8ce27265b0b9855000b3e1b9caec0d75dfb24d1d619bd6' in local storage: no such image
Mar 13 21:53:51 okd4-bootstrap bootkube.sh[2704]: Warning: Could not resolve release image to pull by digest
Mar 13 21:53:52 okd4-bootstrap bootkube.sh[2704]: Error: unable to pull registry.svc.ci.openshift.org/origin/release@sha256:cf9ea2648ea9b7e69a8ce27265b0b9855000b3e1b9caec0d75dfb24d1d619bd6: unable to pull image: Error initializing source docker://registry.svc.ci.openshift.org/origin/release@sha256:cf9ea2648ea9b7e69a8ce27265b0b9855000b3e1b9caec0d75dfb24d1d619bd6: (Mirrors also failed: [okd4-lb01.oscluster.clgcom.org:5001/origin/okd4@sha256:cf9ea2648ea9b7e69a8ce27265b0b9855000b3e1b9caec0d75dfb24d1d619bd6: Error reading manifest sha256:cf9ea2648ea9b7e69a8ce27265b0b9855000b3e1b9caec0d75dfb24d1d619bd6 in okd4-lb01.oscluster.clgcom.org:5001/origin/okd4: manifest unknown: manifest unknown]): registry.svc.ci.openshift.org/origin/release@sha256:cf9ea2648ea9b7e69a8ce27265b0b9855000b3e1b9caec0d75dfb24d1d619bd6: error pinging docker registry registry.svc.ci.openshift.org: Get https://registry.svc.ci.openshift.org/v2/: dial tcp: lookup registry.svc.ci.openshift.org on 10.11.11.10:53: no such host
Mar 13 21:53:52 okd4-bootstrap systemd[1]: bootkube.service: Main process exited, code=exited, status=125/n/a
Mar 13 21:53:52 okd4-bootstrap systemd[1]: bootkube.service: Failed with result 'exit-code'.

log-bundle-20200313175731.tar.gz

[cgruver]

Got past it! Was still using an older OpenShift-install binary... I've mitigated that in my deploy script now.

The bootstrap did NOT stall on:

Mar 13 22:41:56 okd4-bootstrap bootkube.sh[747]: "99_openshift-machineconfig_99-worker-ssh.yaml": unable to get REST mapping for "99_openshift-machineconfig_99-worker-ssh.yaml": no matches for kind "MachineConfig" in version "machineconfiguration.openshift.io/v1"

It still took a few minutes, but much better that restarting a couple of times.

Success!

I believe now, that the issue with my local Nexus is the cert. I'm going to create a new cert and try again.

Will update on progress.