Openshift-ansible: 3.7 fails on etcd container pull

[marekjelen]

Description

open /etc/docker/certs.d/registry.access.redhat.com/redhat-ca.crt: no such file or directory

Version

Please put the following version information in the code block
indicated below.

ansible 2.3.2.0
  config file =
  configured module search path = Default w/o overrides
  python version = 2.7.14 (default, Sep 25 2017, 09:53:22) [GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.37)]

If you're operating from a git clone:

openshift-ansible-3.7.9-1-23-g78f029e37

Steps To Reproduce

[OSEv3:children]
masters
nodes

[masters]
<IP> openshift_schedulable=true

[nodes]
<IP> openshift_schedulable=true openshift_node_labels="{'region': 'infra', 'zone': 'default'}"

[etcd]
<IP>

[OSEv3:vars]
ansible_user=openshift
ansible_become=yes

enable_excluders=false
enable_docker_excluder=false

containerized=True
#openshift_use_system_containers=True
os_sdn_network_plugin_name='redhat/openshift-ovs-multitenant'
openshift_disable_check=disk_availability,docker_storage,memory_availability,docker_image_availability

openshift_node_kubelet_args={'pods-per-core': ['10']}

deployment_type=origin
openshift_deployment_type=origin

openshift_release=v3.7.0
openshift_pkg_version=v3.7.0-rc.0

osm_use_cockpit=true

openshift_metrics_install_metrics=True
openshift_hosted_prometheus_deploy=True

openshift_logging_install_logging=True
openshift_logging_image_version=v3.7.0-rc.0

openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider', 'filename': '/etc/origin/master/htpasswd'}]

openshift_public_hostname=<DNS>
openshift_master_default_subdomain=<DNS>

Expected Results

The container pull should not fail.

Observed Results

stdout => fatal: [<IP>]: FAILED! => {"changed": false, "cmd": ["docker", "pull", "registry.access.redhat.com/rhel7/etcd"], "delta": "0:00:00.068668", "end": "2017-11-22 14:19:06.892003", "failed": true, "rc": 1, "start": "2017-11-22 14:19:06.823335", "stderr": "open /etc/docker/certs.d/registry.access.redhat.com/redhat-ca.crt: no such file or directory", "stderr_lines": ["open /etc/docker/certs.d/registry.access.redhat.com/redhat-ca.crt: no such file or directory"], "stdout": "Using default tag: latest\nTrying to pull repository registry.access.redhat.com/rhel7/etcd ... ", "stdout_lines": ["Using default tag: latest", "Trying to pull repository registry.access.redhat.com/rhel7/etcd ... "]}

Additional Information

CentOS Linux release 7.4.1708 (Core)
Linux cluster01-master 3.10.0-693.5.2.el7.x86_64 #1 SMP Fri Oct 20 20:32:50 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

[sdodson]

~#6195~ switched to using fedora images for origin can you please try with that change?
edit: #6197 i mean

[marekjelen]

@sdodson OK, that worked, thanks! But now failing on

FAILED! => {"failed": true, "msg": "The conditional check 'persistent_volumes | length > 0 or persistent_volume_claims | length > 0' failed. The error was: {{ hostvars[groups.oo_first_master.0] | oo_persistent_volumes(groups) }}: Unexpected templating type error occurred on ({{ hostvars[groups.oo_first_master.0] | oo_persistent_volumes(groups) }}): argument of type 'bool' is not iterable\n\nThe error appears to have been in '..../openshift-ansible/roles/openshift_persistent_volumes/tasks/main.yml': line 2, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n---\n- name: Create temp directory for volume definitions\n  ^ here\n"}

[elasti-ronenc]

Hello,
i am getting the same error

FAILED! => {"changed": false, "cmd": ["docker", "pull", "registry.access.redhat.com/rhel7/etcd"], "delta": "0:00:00.203028", "end": "2017-12-06 09:24:50.662535", "failed": true, "rc": 1, "start": "2017-12-06 09:24:50.459507", "stderr": "open /etc/docker/certs.d/registry.access.redhat.com/redhat-ca.crt: no such file or directory", "stderr_lines": ["open /etc/docker/certs.d/registry.access.redhat.com/redhat-ca.crt: no such file or directory"], "stdout": "Using default tag: latest\nTrying to pull repository registry.access.redhat.com/rhel7/etcd ... ", "stdout_lines": ["Using default tag: latest", "Trying to pull repository registry.access.redhat.com/rhel7/etcd ... "]}

[sdodson]

The installer should no longer be using that image for etcd, we switched it in 85efc9cffc. Are you running the latest code?

[farshid3003]

I have the same problem with the latest version

fatal: [m1.ibeaconhub.com]: FAILED! => {
    "changed": true,
    "checks": {
        "disk_availability": {
            "skipped": true,
            "skipped_reason": "Disabled by user request"
        },
        "docker_image_availability": {
            "changed": true,
            "failed": true,
            "failures": [
                [
                    "OpenShiftCheckException",
                    "One or more required container images are not available:\n    registry.access.redhat.com/rhel7/etcd\nChecked with: skopeo inspect [--tls-verify=false] [--creds=<user>:<pass>] docker://<registry>/<image>\nDefault registries searched: docker.io\n"
                ]
            ],
            "msg": "One or more required container images are not available:\n    registry.access.redhat.com/rhel7/etcd\nChecked with: skopeo inspect [--tls-verify=false] [--creds=<user>:<pass>] docker://<registry>/<image>\nDefault registries searched: docker.io\n"
        },
        "docker_storage": {
            "skipped": true,
            "skipped_reason": "Disabled by user request"
        },
        "memory_availability": {
            "skipped": true,
            "skipped_reason": "Disabled by user request"
        },
        "package_availability": {
            "skipped": true,
            "skipped_reason": "Not active for this host"
        },
        "package_version": {
            "skipped": true,
            "skipped_reason": "Not active for this host"
        }
    },
    "failed": true,
    "msg": "One or more checks failed",
    "playbook_context": "install"
}

[elasti-ronenc]

same error:
fatal: [console.ronenc.io]: FAILED! => {
"changed": false,
"cmd": [
"docker",
"pull",
"registry.access.redhat.com/rhel7/etcd"
],
"delta": "0:00:00.117533",
"end": "2017-12-06 15:45:59.475021",
"failed": true,
"rc": 1,
"start": "2017-12-06 15:45:59.357488"
}

STDOUT:

Using default tag: latest
Trying to pull repository registry.access.redhat.com/rhel7/etcd ...

STDERR:

open /etc/docker/certs.d/registry.access.redhat.com/redhat-ca.crt: no such file or directory

MSG:

non-zero return code

[farshid3003]

add following line to the ansible file:
osm_etcd_image=registry.fedoraproject.org/f26/etcd
but now fedora is not available too

[danielkucera]

this worked for me:
rm -f /etc/docker/certs.d/registry.access.redhat.com/redhat-ca.crt

[malagant]

Don't delete stuff that's needed for pulling securely from a secure registry ;)

Just install yum install python-rhsm-certificates -y on all nodes and it works.

[carct]

Hi,

Same error is happening to me when trying to build / pull from e RHEL image (registry.access.redhat.com/rhel6:6.9-100):

Step 1/44 : FROM registry.access.redhat.com/rhel6:6.9-100
Trying to pull repository registry.access.redhat.com/rhel6 ...
open /etc/docker/certs.d/registry.access.redhat.com/redhat-ca.crt: no such file or directory

[root@agent registry.access.redhat.com]# ls -alh
total 0
drwxr-xr-x. 2 root root 27 Jun 11 15:04 .
drwxr-xr-x. 5 root root 75 Jun 11 15:04 ..
lrwxrwxrwx. 1 root root 27 Jun 11 15:04 redhat-ca.crt -> /etc/rhsm/ca/redhat-uep.pem
[root@vstsagent-dev-VM-2282 registry.access.redhat.com]# cat /etc/rhsm/ca/redhat-uep.pem
cat: /etc/rhsm/ca/redhat-uep.pem: No such file or directory
[root@agent registry.access.redhat.com]# cat /etc/rhsm/ca/
cat: /etc/rhsm/ca/: Is a directory
[root@agent registry.access.redhat.com]# cd /etc/rhsm/ca/
[root@agent ca]# ls
[root@agent ca]#

already had python-rhsm-certificates installed, but tried to do it again and i got this:
Package python-rhsm-certificates-1.19.10-1.el7_4.x86_64 is obsoleted by subscription-manager-rhsm-certificates-1.20.11-1.el7.centos.x86_64 which is already installed
Nothing to do

any ideas ?

[danielkucera]

@carct you can try to download the cert from here: https://github.com/candlepin/python-rhsm/blob/master/etc-conf/ca/redhat-uep.pem

[viv816]

@danielkucera

I am facing similar problem in OpenShift Origin cluster on CentOS 7.

It worked intermittently for a while in the morning but now continuously getting the below error

---

oc v3.9.0+a96a520-22
kubernetes v1.9.1+a0ce1bc657
features: Basic-Auth GSSAPI Kerberos SPNEGO

---

I have downloaded the cert from suggested url, but still no luck.

Appreciate any help.

Thanks in advance.

[carct]

For me it worked just fine, just that I created a file with the contents of the file provided in the link

also, keep in mind that redhat-ca.crt is just a link towards another file: /etc/rhsm/ca/redhat-uep.pem

so you need to ensure the existence of the later one (also, be sure to have +rx rights on the file)

thanks & cheers!

[farshid3003]

Sent from Mail for Windows 10

From: carct
Sent: Thursday, June 14, 2018 12:24 AM
To: openshift/openshift-ansible
Cc: Farshid Mirza; Comment
Subject: Re: [openshift/openshift-ansible] 3.7 fails on etcd container pull(#6222)

For me it worked just fine, just that I created a file with the contents of the file provided in the link
also, keep in mind that redhat-ca.crt is just a link towards another file: /etc/rhsm/ca/redhat-uep.pem
so you need to ensure the existence of the later one (also, be sure to have +rx rights on the file)
thanks & cheers!
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

[carct]

a little mention i forgot, i was experiencing this issue described here, but different environment, no openshift context at this stage - was just building a docker image on a custom-vsts-agent (AzureVM) and was encountering that missing cert. error on pulling RHEL docker image from RedHat Registry.