Crc: Request for installation instructions related to private CA-certificates

[carljmosca]

General information

• OS: macOS
• Hypervisor: hyperkit
• Did you run crc setup before starting it (Yes/No)? yes

CRC version

# Put the output of `crc version`

version: 1.0.0-beta.3-4.1.11+e97bf65

CRC status

# Put the output of `crc status`

CRC VM: Running
OpenShift: Running (v4.x)
Disk Usage: 11.84GB of 32.2GB (Inside the CRC VM)
Cache Usage: 10.48GB
Cache Directory: /Users/moscac/.crc/cache

CRC config

# Put the output of `crc config view`

• bundle : /Users/moscac/.crc/crc_hyperkit_4.1.11.crcbundle
• cpus : 4
• memory : 8192
• nameserver :
• pull-secret-file :
• vm-driver : hyperkit
• warn-check-bundle-cached : true
  

  Host Operating System

# Put the output of `cat /etc/os-release` in case of Linux
# put the output of `sw_vers` in case of Mac
# Put the output of `sysinfo` in case of Windows

ProductName: Mac OS X
ProductVersion: 10.14.6
BuildVersion: 18G95

I have the need to connect to network resources which are secured with a private certificate authority and would like to install it.

[cfergeau]

Not clear what is not working here. Is this something which you can achieve with a regular openshift 4 install but which is not available with CodeReady Containers? Or is this something you don't know how to do with openshift 4?

[carljmosca]

We have achieved it with a regular OCP 3.x installation but I assume it's not a problem with 4.x. The move to CoreOS means the steps are slightly different:

**Copy the PEM-encoded certificate authority file (usually with a .pem file name extension) to /etc/ssl/certs

Run the update-ca-certificates script to update the system bundle of Certificate Authorities. All programs running on the system will now trust the added CA.**

I think the first issue is that /etc/ssl/certs is read-only but if I recall from some of what I have read in the past, the intent is not to change it in this manner. Somewhere the startup procedure should allow the addition of CA certificates.

[cfergeau]

If this is a generic CoreOS/OpenShift 4 issue not directly related to CodeReady Containers (which is only about running single node OpenShift 4 in a VM), then this is not the right place to report your issue

[carljmosca]

I apologize for the confusion. No, it's not a generic CoreOS/OpenShift 4 issue. I was simply explaining how it would be done on OCP.

I still need for it to work if we are going to run CRC in our organization. Issues similar to this is why we're running Docker for Desktop and Kubernetes/Helm locally now on MacOS and Windows.

Except for this issue, I am really impressed with CRC. Very nice installation experience. I think it could be a great onboarding tool for OCP but that may not be the case.

Thanks anyway.

[praveenkumar]

I think the first issue is that /etc/ssl/certs is read-only but if I recall from some of what I have read in the past, the intent is not to change it in this manner. Somewhere the startup procedure should allow the addition of CA certificates.

@carljmosca you closed it too early :) So can you try below with code ready container?

/ / After crc is started successfully
$ oc debug node/crc-56mmj-master-0
Starting pod/crc-56mmj-master-0-debug ...
To use host binaries, run `chroot /host`
If you don't see a command prompt, try pressing enter.

sh-4.2# chroot /host/

sh-4.4# ls /etc/pki/
ca-trust  java  openssl10.cnf  rpm-gpg  tls

sh-4.4# mkdir /etc/pki/certs => put the certs.
put your script also and run it. 

Let us know if that work, it is very hacky way, but give it a go.

[carljmosca]

Thank you for the reply. Can you clarify the part about "put your script also and run it"?

Should I not run update-ca-trust after putting the certificate in /etc/pki/certs?

What script would I run instead?

[praveenkumar]

Should I not run update-ca-trust after putting the certificate in /etc/pki/certs?

Yes I mean after you put your certs there just run update-ca-trust and see if that works.

[carljmosca]

I was able to make the change as described. I have also stopped and restarted crc and noticed that the certificate is still present so I am assuming it will be respected between restarts.

However, I currently only have access to these network resources when I am on the VPN and I have noticed that I cannot run the above command while connected to the VPN. The "oc debug" command hangs.

My hope was to do a simple curl from the command line while connected to the VPN so that I could confirm that the certificate is working as expected.

I will do more extensive testing by deploying something which requires the certificate.

[gbraad]

VPN

This will remain an issue for the current version. We are trying to implement a solution around this, but mostly this is also related to the VPN configuration itself; is this configured as a route-all? If so, only resources on localhost are accessible directly, the rest will always be resolved from the VPN. this is however a separate issue.

Did the certificates now work as expected?

[carljmosca]

Only today have I had time to get back to this.

Unfortunately, no. I cannot access network resources which are using certificates that were created with our organizations PCA (which was successfully imported as described above).

It does seem the the VPN is working however because I am able to reach the resource (and get back this: x509: certificate signed by unknown authority).

[tiraboschi]

Currently I'm using a small bash script like this:

#!/bin/bash -ex
CACERTFILE=my_ca.crt
CACERT=$(base64 -w0 "${CACERTFILE}")
eval $(crc oc-env)
$(crc console --credentials | grep admin | cut -d"'" -f2)
oc debug $(oc get nodes -o name) -- bash -c "echo ${CACERT} | base64 -d > /host/etc/pki/ca-trust/source/anchors/my_ca.crt; chroot /host update-ca-trust extract"
crc stop
crc start

place the CA certs you want to inject in my_ca.crt and run it on your host.
The script will transfer that file to the CRC VM via oc debug node and it will run update-ca-trust there.
Then it will stop and restart the CRC VM to make it effective (maybe simply sending a SIGHUP to cri-o inside the VM would be enough and avoid this, not sure).
My host is also connected to a VPN and, after this, CRC is successfully able to pull images over the VPN from a registry that uses a cert signed by a custom CA.

[cfergeau]

Instead of chroot /host update-ca-trust extract you could also use chroot /host trust anchor /some/path/my_ca.crt, no need to put my_ca.crt to a very specific location in the filesystem.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[carljmosca]

This has been off my radar for a while. I have tried what's been suggested by @tiraboschi and some variations such as @cfergeau suggestion and it all works nicely, thank you both.

I don't know why this was not working back when I first reported it. If the change was either not persistent between stop/start or it really was readonly. In any case, it is working as expected now.