Azure-cli: container access generate [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)

[gbrault]

This is autogenerated. Please review and update as needed.

Describe the bug

Command Name
az container exec

Errors:

[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)
Traceback (most recent call last):
python3.6/site-packages/knack/cli.py, ln 215, in invoke
    cmd_result = self.invocation.execute(args)
cli/core/commands/__init__.py, ln 654, in execute
    raise ex
cli/core/commands/__init__.py, ln 718, in _run_jobs_serially
    results.append(self._run_job(expanded_arg, cmd_copy))
cli/core/commands/__init__.py, ln 711, in _run_job
    six.reraise(*sys.exc_info())
...
az/lib/python3.6/ssl.py, ln 817, in __init__
    self.do_handshake()
az/lib/python3.6/ssl.py, ln 1077, in do_handshake
    self._sslobj.do_handshake()
az/lib/python3.6/ssl.py, ln 689, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)

To Reproduce:

Steps to reproduce the behavior. Note that argument values have been redacted, as they may contain sensitive information.

• _Put any pre-requisite steps here..._
• az container exec --resource-group {} --name {} --exec-command {}

Expected Behavior

Environment Summary

Linux-4.15.0-1093-azure-x86_64-with-debian-stretch-sid (Cloud Shell)
Python 3.6.10
Installer: DEB

azure-cli 2.11.1

Extensions:
ai-examples 0.2.3

Additional Context

[msftbot[bot]]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @dkkapur.

[yonzhan]

container

[gbrault]

@yonzhan thanks for the feedback, could you elaborate please?

[yonzhan]

container instances service team should look into this issue.

[gbrault]

thanks @yonzhan

[elagree]

same problem on my side, any fix soon ?
Thanks !

[clook]

Hi there,

similar traceback on my side (but with more details). Please appologize if it is not the same problem, I will delete it and create another issue.

My env:

Linux-5.6.12-arch1-1-scarlett3-x86_64-with-glibc2.2.5
Archlinux distribution
Python 3.8.5
Installer: Archlinux AUR package https://aur.archlinux.org/packages/azure-cli/
azure-cli 2.11.1

The traceback is the following:

The command failed with an unexpected error. Here is the traceback:

[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1123)
Traceback (most recent call last):
  File "/opt/azure-cli/lib/python3.8/site-packages/knack/cli.py", line 215, in invoke
    cmd_result = self.invocation.execute(args)
  File "/opt/azure-cli/lib/python3.8/site-packages/azure/cli/core/commands/__init__.py", line 654, in execute
    raise ex
  File "/opt/azure-cli/lib/python3.8/site-packages/azure/cli/core/commands/__init__.py", line 718, in _run_jobs_serially
    results.append(self._run_job(expanded_arg, cmd_copy))
  File "/opt/azure-cli/lib/python3.8/site-packages/azure/cli/core/commands/__init__.py", line 711, in _run_job
    six.reraise(*sys.exc_info())
  File "/opt/azure-cli/lib/python3.8/site-packages/six.py", line 703, in reraise
    raise value
  File "/opt/azure-cli/lib/python3.8/site-packages/azure/cli/core/commands/__init__.py", line 688, in _run_job
    result = cmd_copy(params)
  File "/opt/azure-cli/lib/python3.8/site-packages/azure/cli/core/commands/__init__.py", line 325, in __call__
    return self.handler(*args, **kwargs)
  File "/opt/azure-cli/lib/python3.8/site-packages/azure/cli/core/__init__.py", line 782, in default_command_handler
    return op(**command_args)
  File "/opt/azure-cli/lib/python3.8/site-packages/azure/cli/command_modules/container/custom.py", line 624, in container_exec
    _start_exec_pipe(execContainerResponse.web_socket_uri, execContainerResponse.password)
  File "/opt/azure-cli/lib/python3.8/site-packages/azure/cli/command_modules/container/custom.py", line 656, in _start_exec_pipe
    ws = websocket.create_connection(web_socket_uri)
  File "/opt/azure-cli/lib/python3.8/site-packages/websocket/_core.py", line 514, in create_connection
    websock.connect(url, **options)
  File "/opt/azure-cli/lib/python3.8/site-packages/websocket/_core.py", line 222, in connect
    self.sock, addrs = connect(url, self.sock_opt, proxy_info(**options),
  File "/opt/azure-cli/lib/python3.8/site-packages/websocket/_http.py", line 126, in connect
    sock = _ssl_socket(sock, options.sslopt, hostname)
  File "/opt/azure-cli/lib/python3.8/site-packages/websocket/_http.py", line 260, in _ssl_socket
    sock = _wrap_sni_socket(sock, sslopt, hostname, check_hostname)
  File "/opt/azure-cli/lib/python3.8/site-packages/websocket/_http.py", line 235, in _wrap_sni_socket
    return context.wrap_socket(
  File "/usr/lib64/python3.8/ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "/usr/lib64/python3.8/ssl.py", line 1040, in _create
    self.do_handshake()
  File "/usr/lib64/python3.8/ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1123)

It tries to open a SSL connection to the Websocket URL wss://bridge-linux-02.francecentral.management.azurecontainer.io/...
but SSL verification fails because of untrusted chain.

The output of openssl s_client -connect bridge-linux-02.francecentral.management.azurecontainer.io:443 -showcerts helps a lot for debugging:

CONNECTED(00000003)                                                                                                                                          
depth=1 DC = GBL, DC = AME, CN = AME INFRA CA 01
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = *.francecentral.management.azurecontainer.io
verify return:1
---
Certificate chain
 0 s:CN = *.francecentral.management.azurecontainer.io
   i:DC = GBL, DC = AME, CN = AME INFRA CA 01
 1 s:DC = GBL, DC = AME, CN = AME INFRA CA 01
   i:DC = GBL, DC = AME, CN = ameroot

The chain have 2 certs:

 0 s:CN = *.francecentral.management.azurecontainer.io
   i:DC = GBL, DC = AME, CN = AME INFRA CA 01
 1 s:DC = GBL, DC = AME, CN = AME INFRA CA 01
   i:DC = GBL, DC = AME, CN = ameroot

but the ameroot CA is untrusted / unknown. It is still referenced by the delegation AME INFRA CA 01 but this referenced is not used by openssl:

            Authority Information Access: 
                CA Issuers - URI:http://crl.microsoft.com/pkiinfra/certs/AMEROOT_ameroot.crt
                CA Issuers - URI:http://crl2.ame.gbl/aia/AMEROOT_ameroot.crt
                CA Issuers - URI:http://crl3.ame.gbl/aia/AMEROOT_ameroot.crt
                CA Issuers - URI:http://crl1.ame.gbl/aia/AMEROOT_ameroot.crt
                CA Issuers - URI:ldap:///CN=ameroot,CN=AIA,CN=Public%20Key%20Services,CN=Services,CN=Configuration,DC=AME,DC=GBL?cACertificate?base?objectClass=certificationAuthority

I downloaded the linked cert, converted it to PEM, and tried to include it to the following root CA bundles but without any result:

/opt/azure-cli/lib/python3.8/site-packages/certifi/cacert.pem
/opt/azure-cli/lib/python3.8/site-packages/pip/_vendor/certifi/cacert.pem
/opt/azure-cli/lib/python3.8/site-packages/websocket/cacert.pem

Installing the same PEM to the system trusted chain is a sufficient workarround for me (using the command trust anchor --store <cert.pem> as root) even if not very safe.

[juhacket]

@gbrault @clook @elagree This issue has been resolved.

[clook]

LGTM for the cert chain :) Many thanks for the reactivity @juhacket!

@gbrault Could you confirm it is OK on your side (still not sure if we are relating about the same issue)?

[gbrault]

Yes, I can now attach to the running container. Thx for the fix. I don't know if it was the same issue than yours @clook, but the correction solved mine!

I have some other issues now as I can use the terminal in my container. It is however another issue, I will open another ticket.
I use the jupyter/scipy image (with some slight modifications)
I am using chrome and entering my container with:
az container exec --resource-group myResourceGroup --name scipy2 --exec-command "/bin/bash"

Let me explain first: My usage is quite straightforward: using nano to edit some text file in the runing container.

I experienced two set of problems:
1 - at cloud shell level, entering a command line "wraps" after few key stroke (5 or so)
2- when in nano and editing a texte file, the screen get garbeled when I use the arrows key (up, dow etc...)

Where should I write this ticket?

Thx for the support

[clook]

@gbrault Could you check with a 80 colums terminal if it helps for display?

[gbrault]

@clook can you guide me to do that? Wher is that option?

[clook]

@gbrault Depending on your local terminal app, you may have a setting to force it to 80 characters or force by resizing the window manually.

[gbrault]

@clook I am using https://shell.azure.com/ the built-in azure terminal. I don't see such kind of option. I am using the bash option (as my target is a linux container). I don't know how to make progress there. For me, the bugs I have is linked to the cloud shell implementation which is not behaving properly or some middleware failing. I have other environments where I can use terminal feature working fine.

[clook]

Ok, I thought about a terminal on your workstation configured with az login