Openshift-ansible: 3.10 - openid_auth_openid_ca.crt is added to master-config.yaml when it doesn't exist

[iGondor]

Description

When installing a new cluster, and using OpenID without variable openshift_master_request_header_ca or openshift_master_request_header_ca_file defined, the master-config.yaml is still updated with ca: /etc/origin/master/openid_auth_openid_ca.crt, causing the master API to fail to start, due to an invalid master-config.yaml, because openid_auth_openid_ca.crt does not exist.

Version

Please put the following version information in the code block
indicated below.

• ansible 2.6.3.post0cd op

• openshift-ansible-3.10.46-1

Steps To Reproduce

1. Deploy OpenShift cluster with a defined openshift_master_identity_providers, without using a CA, e.g.
   openshift_master_identity_providers=[{"name": "openid_auth", "login": "true", "challenge": "false", "kind": "OpenIDIdentityProvider", "client_id": "openshift-uat.lab", "client_secret": "000000-000000-000000-000000", "claims": {"id": ["sub"], "preferredUsername": ["preferred_username"], "name": ["name"], "email": ["email"]}, "urls": {"authorize": "https://sso-uat.lab/auth/realms/lab/protocol/openid-connect/auth", "token": "https://sso-uat.lab/auth/realms/lab/protocol/openid-connect/token"}}]

2. Run prerequisites

3. Run deploy_cluster

Expected Results

Cluster deployed successfully. This has previously deployed successfully, however after a recent rebuild and redeployment using a newer openshift-ansible version, the same inventory no longer functions due to this issue.

ansible-playbook openshift-ansible/playbooks/redeploy-certificates.yml --inventory-file=lab-openshift/lab-openshift-ansible-prod/hosts.ini

Observed Results

Describe what is actually happening.

Sep 11 02:35:09 lab-ose-master-uat-1.ose-uat.lab dnsmasq[6953]: setting upstream servers from DBus
Sep 11 02:35:09 lab-ose-master-uat-1.ose-uat.lab dnsmasq[6953]: using nameserver 217.19.60.80#53
Sep 11 02:35:09 lab-ose-master-uat-1.ose-uat.lab dnsmasq[6953]: using nameserver 217.19.48.80#53
Sep 11 02:35:09 lab-ose-master-uat-1.ose-uat.lab dnsmasq[6953]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
Sep 11 02:35:09 lab-ose-master-uat-1.ose-uat.lab dnsmasq[6953]: using nameserver 127.0.0.1#53 for domain cluster.local
Sep 11 02:35:10 lab-ose-master-uat-1.ose-uat.lab origin-node[8219]: I0911 02:35:10.654302    8219 kuberuntime_manager.go:513] Container {Name:api Image:docker.io/openshift/origin-control-plane:v3.10.0 Command:[/bin/bash -c] ArgSep 11 02:35:10 lab-ose-master-uat-1.ose-uat.lab origin-node[8219]: set -euo pipefail
Sep 11 02:35:10 lab-ose-master-uat-1.ose-uat.lab origin-node[8219]: if [[ -f /etc/origin/master/master.env ]]; then
Sep 11 02:35:10 lab-ose-master-uat-1.ose-uat.lab origin-node[8219]: set -o allexport
Sep 11 02:35:10 lab-ose-master-uat-1.ose-uat.lab origin-node[8219]: source /etc/origin/master/master.env
Sep 11 02:35:10 lab-ose-master-uat-1.ose-uat.lab origin-node[8219]: fi
Sep 11 02:35:10 lab-ose-master-uat-1.ose-uat.lab origin-node[8219]: exec openshift start master api --config=/etc/origin/master/master-config.yaml --loglevel=${DEBUG_LOGLEVEL:-2}
Sep 11 02:35:10 lab-ose-master-uat-1.ose-uat.lab origin-node[8219]: ] WorkingDir: Ports:[] EnvFrom:[] Env:[] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:master-config ReadOnly:false MountPath:/etc/origin/master/Sep 11 02:35:10 lab-ose-master-uat-1.ose-uat.lab origin-node[8219]: I0911 02:35:10.654497    8219 kuberuntime_manager.go:757] checking backoff for container "api" in pod "master-api-lab-ose-master-uat-1.ose-uat.lab_kube-systSep 11 02:35:10 lab-ose-master-uat-1.ose-uat.lab origin-node[8219]: I0911 02:35:10.654665    8219 kuberuntime_manager.go:767] Back-off 5m0s restarting failed container=api pod=master-api-lab-ose-master-uat-1.ose-uat.lab_kubeSep 11 02:35:10 lab-ose-master-uat-1.ose-uat.lab origin-node[8219]: E0911 02:35:10.654698    8219 pod_workers.go:186] Error syncing pod 9ca23c5815da8ed1d3dca61d87e1f6ab ("master-api-lab-ose-master-uat-1.ose-uat.lab_kube-systSep 11 02:35:13 lab-ose-master-uat-1.ose-uat.lab origin-node[8219]: I0911 02:35:13.655619    8219 kuberuntime_manager.go:513] Container {Name:controllers Image:docker.io/openshift/origin-control-plane:v3.10.0 Command:[/bin/bashS

Additional Information

• CentOS 7.3

lab-ose-master-uat-1.ose-uat.lab ansible_port=22 ansibe_host=192.19.50.25
lab-ose-master-uat-2.ose-uat.lab ansible_port=22 ansibe_host=192.19.50.26
lab-ose-master-uat-3.ose-uat.lab ansible_port=22 ansibe_host=192.19.50.27
lab-ose-node-uat-1.ose-uat.lab ansible_port=22 ansibe_host=192.19.50.28
lab-ose-node-uat-2.ose-uat.lab ansible_port=22 ansibe_host=192.19.50.29
lab-ose-node-uat-3.ose-uat.lab ansible_port=22 ansibe_host=192.19.50.30

[nodes]
lab-ose-node-uat-1.ose-uat.lab openshift_node_labels="{'region': 'primary', 'zone': 'default'}" openshift_node_group_name="node-config-compute" openshift_hostname="lab-ose-node-uat-1.ose-uat.lab" openshift_public_hostname="lab-ose-node-uat-1.ose-uat.lab" openshift_public_ip=192.19.50.28 openshift_ip=192.19.50.28
lab-ose-node-uat-2.ose-uat.lab openshift_node_labels="{'region': 'primary', 'zone': 'default'}" openshift_node_group_name="node-config-compute" openshift_hostname="lab-ose-node-uat-2.ose-uat.lab" openshift_public_hostname="lab-ose-node-uat-2.ose-uat.lab" openshift_public_ip=192.19.50.29 openshift_ip=192.19.50.29
lab-ose-node-uat-3.ose-uat.lab openshift_node_labels="{'region': 'primary', 'zone': 'default'}" openshift_node_group_name="node-config-compute" openshift_hostname="lab-ose-node-uat-3.ose-uat.lab" openshift_public_hostname="lab-ose-node-uat-3.ose-uat.lab" openshift_public_ip=192.19.50.30 openshift_ip=192.19.50.30
lab-ose-master-uat-1.ose-uat.lab openshift_node_labels="{'region': 'infra', 'zone': 'default'}" openshift_node_group_name="node-config-master-infra" openshift_hostname="lab-ose-master-uat-1.ose-uat.lab" openshift_public_hostname="lab-ose-master-uat-1.ose-uat.lab" openshift_public_ip=192.19.50.25 openshift_ip=192.19.50.25 openshift_schedulable=true 
lab-ose-master-uat-2.ose-uat.lab openshift_node_labels="{'region': 'infra', 'zone': 'default'}" openshift_node_group_name="node-config-master-infra" openshift_hostname="lab-ose-master-uat-2.ose-uat.lab" openshift_public_hostname="lab-ose-master-uat-2.ose-uat.lab" openshift_public_ip=192.19.50.26 openshift_ip=192.19.50.26 openshift_schedulable=true 
lab-ose-master-uat-3.ose-uat.lab openshift_node_labels="{'region': 'infra', 'zone': 'default'}" openshift_node_group_name="node-config-master-infra" openshift_hostname="lab-ose-master-uat-3.ose-uat.lab" openshift_public_hostname="lab-ose-master-uat-3.ose-uat.lab" openshift_public_ip=192.19.50.27 openshift_ip=192.19.50.27 openshift_schedulable=true 

[masters]
lab-ose-master-uat-[1:3].ose-uat.lab

[etcd]
lab-ose-master-uat-[1:3].ose-uat.lab 

[OSEv3:children]
masters
nodes
etcd

[OSEv3:vars]
#Deployment and version
openshift_deployment_type="origin"
openshift_release="3.10"

# This enables all the system containers except for docker:
openshift_use_system_containers=true
# In either case, system_images_registry must be specified to be able to find the system images
system_images_registry="docker.io"

#Enable Firewalld automatic configuration
os_firewall_use_firewalld=true
os_firewall_enabled=true

#Disable logging and delete any logging reminants
openshift_logging_install_logging=false
#Ensure there are no log reminants
openshift_logging_purge_logging=true
#turn off a second ES/kibana for infra logging. defaults to false
openshift_logging_use_ops=false

#Enable Ansible and Template service brokers
ansible_service_broker_install=true
template_service_broker_install=true

# Configure UAT Keycloak OAuth authentication
openshift_master_identity_providers=[{"name": "openid_auth", "login": "true", "challenge": "false", "kind": "OpenIDIdentityProvider", "client_id": "openshift-uat.lab", "client_secret": "000000-000000-000000-0000", "claims": {"id": ["sub"], "preferredUsername": ["preferred_username"], "name": ["name"], "email": ["email"]}, "urls": {"authorize": "https://sso-uat.lab/auth/realms/lab/protocol/openid-connect/auth", "token": "https://sso-uat.lab/auth/realms/lab/protocol/openid-connect/token"}}]

#Enable HTTPS via wildcard certificate for portal Public URL and Internal URL
openshift_master_overwrite_named_certificates=false
openshift_master_named_certificates=[{"certfile": "/tmp/lab.crt", "keyfile": "/tmp/lab.key", "names": ["openshift-uat.lab"], "cafile": "/tmp/ca-lab.crt"}]
openshift_master_named_certificates=[{"certfile": "/tmp/lab.crt", "keyfile": "/tmp/lab.key", "names": ["lab-ose-master-uat-cluster.lab"], "cafile": "/tmp/ca-lab.crt"}]

#Configure SSH user
ansible_user=batch
ansible_become=true

# default subdomain to use for exposed routes, you should have wildcard dns
openshift_master_default_subdomain=apps.ose-uat.lab
#Set openshift_master_cluster_hostname to point at your load balancer
openshift_master_cluster_hostname=lab-ose-master-uat-cluster.lab
#Set PUBLIC openshift_master_cluster_public_hostname to point at your load balancer AND DIFFERENT TO ABOVE
openshift_master_cluster_public_hostname=openshift-uat.lab

# Configure master API and console ports.
openshift_master_api_port=8443
openshift_master_console_port=8443

[vrutkovs]

Since API server is running in static pods now it won't use host CA storage, so you need to define openshift_master_request_header_ca or openshift_master_request_header_ca_file

[iGondor]

Even if it isn't needed to authenticate with openid? i.e. If I delete this line from the master config, and start the API manually, OpenShift and openid authentication works as expected with this undefined.

[iGondor]

I will use /etc/ssl/certs/ca-bundle.crt and retry if this is now required. Will report back, thanks.

[iGondor]

when using openshift_master_request_header_ca_file="/etc/ssl/certs/ca-bundle.crt", the issue still persists. The certificate used is publicly trusted, however the certificate is still not created here:
[root@lab-ose-master-uat-1:~]# openshift start master api --config=/etc/origin/master/master-config.yaml --loglevel=8
I0913 01:27:02.544960 16363 plugins.go:84] Registered admission plugin "NamespaceLifecycle"
I0913 01:27:02.545064 16363 plugins.go:84] Registered admission plugin "Initializers"
I0913 01:27:02.545083 16363 plugins.go:84] Registered admission plugin "ValidatingAdmissionWebhook"
I0913 01:27:02.545100 16363 plugins.go:84] Registered admission plugin "MutatingAdmissionWebhook"
I0913 01:27:02.545113 16363 plugins.go:84] Registered admission plugin "AlwaysAdmit"
I0913 01:27:02.545131 16363 plugins.go:84] Registered admission plugin "AlwaysPullImages"
I0913 01:27:02.545147 16363 plugins.go:84] Registered admission plugin "LimitPodHardAntiAffinityTopology"
I0913 01:27:02.545164 16363 plugins.go:84] Registered admission plugin "DefaultTolerationSeconds"
I0913 01:27:02.545176 16363 plugins.go:84] Registered admission plugin "AlwaysDeny"
I0913 01:27:02.545191 16363 plugins.go:84] Registered admission plugin "EventRateLimit"
I0913 01:27:02.545208 16363 plugins.go:84] Registered admission plugin "DenyEscalatingExec"
I0913 01:27:02.545221 16363 plugins.go:84] Registered admission plugin "DenyExecOnPrivileged"
I0913 01:27:02.545235 16363 plugins.go:84] Registered admission plugin "ExtendedResourceToleration"
I0913 01:27:02.545256 16363 plugins.go:84] Registered admission plugin "OwnerReferencesPermissionEnforcement"
I0913 01:27:02.545275 16363 plugins.go:84] Registered admission plugin "ImagePolicyWebhook"
I0913 01:27:02.545291 16363 plugins.go:84] Registered admission plugin "InitialResources"
I0913 01:27:02.545307 16363 plugins.go:84] Registered admission plugin "LimitRanger"
I0913 01:27:02.545323 16363 plugins.go:84] Registered admission plugin "NamespaceAutoProvision"
I0913 01:27:02.545357 16363 plugins.go:84] Registered admission plugin "NamespaceExists"
I0913 01:27:02.545373 16363 plugins.go:84] Registered admission plugin "NodeRestriction"
I0913 01:27:02.545391 16363 plugins.go:84] Registered admission plugin "PersistentVolumeLabel"
I0913 01:27:02.545407 16363 plugins.go:84] Registered admission plugin "PodNodeSelector"
I0913 01:27:02.545422 16363 plugins.go:84] Registered admission plugin "PodPreset"
I0913 01:27:02.545437 16363 plugins.go:84] Registered admission plugin "PodTolerationRestriction"
I0913 01:27:02.545458 16363 plugins.go:84] Registered admission plugin "ResourceQuota"
I0913 01:27:02.545474 16363 plugins.go:84] Registered admission plugin "PodSecurityPolicy"
I0913 01:27:02.545489 16363 plugins.go:84] Registered admission plugin "Priority"
I0913 01:27:02.545511 16363 plugins.go:84] Registered admission plugin "SecurityContextDeny"
I0913 01:27:02.545529 16363 plugins.go:84] Registered admission plugin "ServiceAccount"
I0913 01:27:02.545545 16363 plugins.go:84] Registered admission plugin "DefaultStorageClass"
I0913 01:27:02.545560 16363 plugins.go:84] Registered admission plugin "PersistentVolumeClaimResize"
I0913 01:27:02.545579 16363 plugins.go:84] Registered admission plugin "StorageObjectInUseProtection"
Invalid MasterConfig /etc/origin/master/master-config.yaml
oauthConfig.identityProvider[0].provider.ca: Invalid value: "/etc/origin/master/openid_auth_openid_ca.crt": could not read file: stat /etc/origin/master/openid_auth_openid_ca.crt: no such file or directory

Also re-run the installer using just CA supplied by our signer, and the installer fails with /etc/origin/master/openid_auth_openid_ca.crt: no such file or directory

[vrutkovs]

Please attach ansible logs (the output of `ansible-playbook -vvv playbooks/deploy_cluster.yml)

[iGondor]

here is a zip of the logs. the installer loops waiting for pods for a long time until it ends - pods do not come up due to the master.conf

thanks
deploy_cluster.zip

[vrutkovs]

Weird, openid_ca internal var is not set. Could you also attach the inventory you're using?

[iGondor]

lab-ose-master-uat-1.ose-uat.local ansible_port=22 ansibe_host=217.19.50.25
lab-ose-master-uat-2.ose-uat.local ansible_port=22 ansibe_host=217.19.50.26
lab-ose-master-uat-3.ose-uat.local ansible_port=22 ansibe_host=217.19.50.27
lab-ose-node-uat-1.ose-uat.local ansible_port=22 ansibe_host=217.19.50.28
lab-ose-node-uat-2.ose-uat.local ansible_port=22 ansibe_host=217.19.50.29
lab-ose-node-uat-3.ose-uat.local ansible_port=22 ansibe_host=217.19.50.30

[masters]
lab-ose-master-uat-[1:3].ose-uat.local

[etcd]
lab-ose-master-uat-[1:3].ose-uat.local

[nodes]
lab-ose-node-uat-1.ose-uat.local openshift_node_labels="{'region': 'primary', 'zone': 'default'}" openshift_node_group_name="node-config-compute" openshift_hostname="lab-ose-node-uat-1.ose-uat.local" openshifpublic_hohostname="lab-ose-node-uat-1.ose-uat.local" openshift_public_ip=217.19.50.28 openshift_ip=217.19.50.28
lab-ose-node-uat-2.ose-uat.local openshift_node_labels="{'region': 'primary', 'zone': 'default'}" openshift_node_group_name="node-config-compute" openshift_hostname="lab-ose-node-uat-2.ose-uat.local" openshifpublic_hohostname="lab-ose-node-uat-2.ose-uat.local" openshift_public_ip=217.19.50.29 openshift_ip=217.19.50.29
lab-ose-node-uat-3.ose-uat.local openshift_node_labels="{'region': 'primary', 'zone': 'default'}" openshift_node_group_name="node-config-compute" openshift_hostname="lab-ose-node-uat-3.ose-uat.local" openshifpublic_hohostname="lab-ose-node-uat-3.ose-uat.local" openshift_public_ip=217.19.50.30 openshift_ip=217.19.50.30
lab-ose-master-uat-1.ose-uat.local openshift_schedulable=true openshift_node_labels="{'region': 'infra', 'zone': 'default'}" openshift_node_group_name="node-config-master-infra" openshift_hostname="lab-ose-master-uat-1.ose-uat.local" openshift_public_hostname="lab-ose-master-uat-1.ose-uat.local" openshift_public_ip=217.19.50.25 openshift_ip=217.19.50.25
lab-ose-master-uat-2.ose-uat.local openshift_schedulable=true openshift_node_labels="{'region': 'infra', 'zone': 'default'}" openshift_node_group_name="node-config-master-infra" openshift_hostname="lab-ose-master-uat-2.ose-uat.local" openshift_public_hostname="lab-ose-master-uat-2.ose-uat.local" openshift_public_ip=217.19.50.26 openshift_ip=217.19.50.26
lab-ose-master-uat-3.ose-uat.local openshift_schedulable=true openshift_node_labels="{'region': 'infra', 'zone': 'default'}" openshift_node_group_name="node-config-master-infra" openshift_hostname="lab-ose-master-uat-3.ose-uat.local" openshift_public_hostname="lab-ose-master-uat-3.ose-uat.local" openshift_public_ip=217.19.50.27 openshift_ip=217.19.50.27

[OSEv3:children]
masters
nodes
etcd

[OSEv3:vars]
//Deployment and version
openshift_deployment_type="origin"
openshift_release="3.10"

// This enables all the system containers except for docker:
openshift_use_system_containers=true
// In either case, system_images_registry must be specified to be able to find the system images
system_images_registry="docker.io"

//Enable testing repos for 3.10
openshift_repos_enable_testing=false

//Enable Firewalld automatic configuration
os_firewall_use_firewalld=true
os_firewall_enabled=true

//Disable logging and delete any logging reminants
openshift_logging_install_logging=false
//Ensure there are no log reminants
openshift_logging_purge_logging=true
//turn off a second ES/kibana for infra logging. defaults to false
openshift_logging_use_ops=false

//Enable Ansible and Template service brokers
ansible_service_broker_install=true
template_service_broker_install=true

// Configure UAT Keycloak OAuth authentication
openshift_master_identity_providers=[{"name": "openid_auth", "login": "true", "challenge": "false", "kind": "OpenIDIdentityProvider", "client_id": "lab.ttc.openshift", "client_secret": "000000-0000-0000-0000-00000000", "claims": {"id": ["sub"], "preferredUsername": ["preferred_username"], "name": ["name"], "email": ["email"]}, "urls": {"authorize": "https://sso-uat.local/auth/realms/lab/protocol/openid-connect/auth", "token": "https://sso-uat.local/auth/realms/lab/protocol/openid-connect/token"}}]
openshift_master_request_header_ca_file="/etc/ssl/certs/ca-bundle.crt"
//Enable HTTPS via wildcard certificate for portal Public URL and Internal URL
openshift_master_overwrite_named_certificates=true
openshift_master_named_certificates=[{"certfile": "/tmp/lab.crt", "keyfile": "/tmp/lab.key", "names": ["openshift-uat.local"]}]
//penshift_master_named_certificates=[{"certfile": "/tmp/lab.crt", "keyfile": "/tmp/lab.key", "names": ["lab-ose-master-uat-cluster.local"]}]

//Configure SSH user
ansible_user=batch
ansible_become=true

// default subdomain to use for exposed routes, you should have wildcard dns
openshift_master_default_subdomain=apps.ose-uat.local
//Set cluster_hostname to point at your load balancer
openshift_master_cluster_hostname=lab-ose-master-uat-cluster.local
//Set PUBLIC cluster_hostname to point at your load balancer
openshift_master_cluster_public_hostname=openshift-uat.local

// Configure master API and console ports.
openshift_master_api_port=8443
openshift_master_console_port=8443

[vrutkovs]

Right, so you have

openshift_master_identity_providers = ... OpenIDIdentityProvider ...

but no openshift_master_openid_ca_file, instead openshift_master_request_header_ca_file is set - it gets used only when kind is RequestHeaderIdentityProvider

Does it work when openshift_master_request_header_ca_file is renamed to openshift_master_openid_ca_file?

(sorry, my fault, specified a wrong var name in previous comments :( )

[iGondor]

Ah awesome - did not see that! Thanks!

Updated and re-running now

If there is nothing supplied to this parameter, could the openid_auth_openid_ca.crt line not be put into the master.config? Or possibly checked that the parameter is populated if this param is utilised?

[vrutkovs]

It needs to be specified every time, yes, otherwise SSL bundle from static pod would be used - and it could be outdated

[vrutkovs]

Filed https://bugzilla.redhat.com/show_bug.cgi?id=1633137 to describe new behaviour in the docs