Azure-cli: Not able to use Azure CLI with self-signed certificate

[Anshapar]

Environment summary

CLI version azure-cli (2.0.22)
OS Type: Windows 10
Installation via: apt-get for Bash on Ubuntu on Windows

I am trying to run cli commands with a self-signed certificate and I am getting following error:

"Please ensure you have network connection. Error detail: HTTPSConnectionPool(host='login.microsoftonline.com', port=443): Max retries exceeded with url: /common/oauth2/devicecode?api-version=1.0 (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)",),))"

[brianj1084]

You need to set the 'REQUESTS_CA_BUNDLE' environment variable to the path to your self signed cert.

[Anshapar]

@brianj1084 I tried that I am still getting this error
"Please ensure you have network connection. Error detail: HTTPSConnectionPool(host='login.microsoftonline.com', port=443): Max retries exceeded with url: /common/oauth2/devicecode?api-version=1.0 (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)",),))"

[brianj1084]

@Anshapar How did you get your cert? The problem for me was the CLI uses Python. That environment variable tells Python where to look for it's trusted certs. Try using chrome and following these steps to save your root cert.

• Open chrome dev tools

• Click Security tab

• Click View Certificate button

• On the Certification Path tab, click the highest node in the tree

• Click View Certificate

• On the Details tab, click the Copy to File button

• Make sure to select Base-64 encoded X.509 (.CER)

• Save the file somewhere on your drive (ex. C:\certs\my_root.cer)

• Run the command 'set REQUESTS_CA_BUNDLE=C:\certs\my_root.cer' from a command prompt

• Run the command 'echo %REQUESTS_CA_BUNDLE%' to verify it set correctly

[Anshapar]

@brianj1084 If I am using set command, the value is coming as empty. I used export command instead of set, but the error still persists after following the above mentioned commands.

[alexmarshall132]

I'm encountering the same problem as @Anshapar. I can confirm that correctly setting the variable still doesn't work and the python runtime still tries to validate the SSL certs in spite of the variable being present. I know there's no issue with the cert to which the variable is pointing because I've used it with other applications in similar scenarios.

[pihai]

Same here.

1. First I used set REQUESTS_CA_BUNDLE=<path> to set the environment variable
2. Then I verified that the certificate works with: python -c "import requests;r=requests.get('https://www.google.com');print(r)"

But Azure CLI still gives me the error message described by the author of this issue

This is really annoying because I can't use Azure CLI behind the SSL intercepting proxy in our corporate network

azure-cli (2.0.30)
Python (Windows) 3.6.1 (v3.6.1:69c0db5, Mar 21 2017, 17:54:52) [MSC v.1900 32 bit (Intel)]

[yugangw-msft]

@pihai, could you please elaborate so that I can understand your context better? Setting CA bundle should definitely work. This is really not a CLI thing, rather the supports is provided from the requests package.
For cross check, I also used windows and tried out with the fiddler's certificate.

  1.  First I exported fiddler's certificate to desktop and get the .cer file
  2.  Use openssl to convert to pem file, e.g. "openssl x509 -inform der -in FiddlerRoot.cer -out FiddlerRoot.pem". Note ".cer" is not accepted by Python
  3.  Configure the env variable, e.g. `set REQUESTS_CA_BUNDLE=d:\temp\FiddlerRoot.pem`
  4.  With fiddler open, I can run cli commands as well as the code snippet you provided

[rzand]

I could not create VMs with the azure cli from my Windows 10 computer because of our company gateway.
Receiving the error
az vm create: error: HTTPSConnectionPool(host='raw.githubusercontent.com',
port=443): Max retries exceeded with url: /Azure/azure-rest-api-specs/master/arm-compute/quickstart-templates/aliases.json
(Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)",),))

I had to install the python certifi module
"C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\python" -m pip install --upgrade pip
"C:\Program Files (x86)\Microsoft Visual StudioShared\Python36_64Scripts\pip" install python-certifi-win32
(you may need to adjust the file system permissions to successfully complete the above commands)
Add the Cloud services root CA exported from the computer certificate store to the
C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\Lib\site-packages\certifi\cacert.pem certificate store
and set the system variable
REQUESTS_CA_BUNDLE to C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\Lib\site-packages\certifi\cacert.pem :
echo %REQUESTS_CA_BUNDLE%
C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\Lib\site-packages\certifi\cacert.pem
This also fixed the aws s3 ls command that was returning [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)

[pihai]

@yugangw-msft "unfortunately" our IT department has whitelisted the azure management endpoint in our proxy server to stop SSL interception. So finally I can use Azure CLI but now I can't investigate the problem in this case anymore.

[k-rloss]

I was facing this same issue (same error, due to SSL-intercepting corporate proxy).

I use the Azure CLI under linux, and I was able to solve this by making the CLI trust the corporate proxy certificates, by adding them to the cacert.pem files located under:
/usr/lib64/az/lib/python2.7/site-packages/pip/_vendor/certifi/cacert.pem
/usr/lib64/az/lib/python2.7/site-packages/websocket/cacert.pem

And setting the REQUESTS_CA_BUNDLE variable as follows:

export REQUESTS_CA_BUNDLE=/usr/lib64/az/lib/python2.7/site-packages/websocket/cacert.pem

One way of getting your proxy certs is running this:
openssl s_client -proxy :port -connect login.microsoftonline.com:443 -showcerts

Append the certificates from the trust chain at the end of the cacert.pem files (no need to append the leaf certificate. Only the root and intermediates are needed)

[krishnakumarq]

Showing the below error when running the azcopy command from a scheduler ::

ImportError: /usr/lib64/libgssapi_krb5.so.2: symbol k5_buf_free, version krb5support_0_MIT not defined in file libkrb5support.so.0 with link time reference
Traceback (most recent call last):
File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/usr/lib64/az/lib/python2.7/site-packages/azure/cli/__main__.py", line 12, in
from azure.cli.core import get_default_cli
File "/usr/lib64/az/lib/python2.7/site-packages/azure/cli/core/__init__.py", line 12, in
from pkg_resources import parse_version
File "/usr/lib64/az/lib/python2.7/site-packages/pkg_resources/__init__.py", line 36, in
import email.parser
File "/usr/lib64/python2.7/email/parser.py", line 12, in
from email.feedparser import FeedParser
File "/usr/lib64/python2.7/email/feedparser.py", line 27, in
from email import message
File "/usr/lib64/python2.7/email/message.py", line 16, in
import email.charset
File "/usr/lib64/python2.7/email/charset.py", line 13, in
import email.base64mime
File "/usr/lib64/python2.7/email/base64mime.py", line 40, in
from email.utils import fix_eols
File "/usr/lib64/python2.7/email/utils.py", line 27, in
import random
File "/usr/lib64/python2.7/random.py", line 49, in
import hashlib as _hashlib
File "/usr/lib64/python2.7/hashlib.py", line 116, in
import _hashlib
ImportError: /usr/lib64/libgssapi_krb5.so.2: symbol k5_buf_free, version krb5support_0_MIT not defined in file libkrb5support.so.0 with link time reference
Traceback (most recent call last):
File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/usr/lib64/az/lib/python2.7/site-packages/azure/cli/__main__.py", line 12, in
from azure.cli.core import get_default_cli
File "/usr/lib64/az/lib/python2.7/site-packages/azure/cli/core/__init__.py", line 12, in
from pkg_resources import parse_version
File "/usr/lib64/az/lib/python2.7/site-packages/pkg_resources/__init__.py", line 36, in
import email.parser
File "/usr/lib64/python2.7/email/parser.py", line 12, in
from email.feedparser import FeedParser
File "/usr/lib64/python2.7/email/feedparser.py", line 27, in
from email import message
File "/usr/lib64/python2.7/email/message.py", line 16, in
import email.charset
File "/usr/lib64/python2.7/email/charset.py", line 13, in
import email.base64mime
File "/usr/lib64/python2.7/email/base64mime.py", line 40, in
from email.utils import fix_eols
File "/usr/lib64/python2.7/email/utils.py", line 27, in
import random
File "/usr/lib64/python2.7/random.py", line 49, in
import hashlib as _hashlib
File "/usr/lib64/python2.7/hashlib.py", line 116, in
import _hashlib
ImportError: /usr/lib64/libgssapi_krb5.so.2: symbol k5_buf_free, version krb5support_0_MIT not defined in file libkrb5support.so.0 with link time reference

I have tried the option of appending the proxy certificate to the .pem files and tried still the issue exists.

I am using iKNOW scheduler for scheduling the job and have wriitten the azcopy service in bash script.

[chris-jansson]

I'm getting this same issue running azure-cli 2.0.50 on macOS 10.13.6. What I've done so far:

• Set $REQUESTS_CA_BUNDLE to the path of my self-signed cert (in pem format)
• Verified that the certificate works with: python -c "import requests;r=requests.get('https://www.google.com');print(r)"

When I run az login:

Please ensure you have network connection. Error detail: HTTPSConnectionPool(host='login.microsoftonline.com', port=443): Max retries exceeded with url: /common/oauth2/devicecode?api-version=1.0 (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')])")))

[Sispheor]

On my side I also added the corporate certificate into the file /opt/az/lib/python3.6/site-packages/certifi/cacert.pem, which is used by the ssl lib or Azure.

[BevanG]

Just to add to this, @rzand 's process was the only one that worked for me, I'll expand on his solution though as there were extra steps required. All from elevated Shells

1. "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\python" -m pip install --upgrade pip
2. "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python36_64\Scripts\pip" install python-certifi-win32
3. Add the Cloud services root CA to cacert.pem exported from the downloaded certificate. I specifically needed Microsoft IT TLS CA 5 and the "Baltimore CyberTrust Root" from that cert. Simply open the certs in text editor and append the contents to the bottom of C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\Lib\site-packages\certifi\cacert.pem
4. Add the Self-signed certificate given to you by the network team. Simply open the cert in text editor and append the contents to the bottom of C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\Lib\site-packages\certifi\cacert.pem
5. Set the system/environment variable in Command prompt setx /m REQUESTS_CA_BUNDLE "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\Lib\site-packages\certifi\cacert.pem"
6. Set the system/environment variable in Powershell $env:REQUESTS_CA_BUNDLE="C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\Lib\site-packages\certifi\cacert.pem"
7. Close and open Bash / Command Prompt

FINALLY no errors. I can even retrieve Key Vault secrets

[darrens280]

Showing the below error when running the azcopy command from a scheduler ::

ImportError: /usr/lib64/libgssapi_krb5.so.2: symbol k5_buf_free, version krb5support_0_MIT not defined in file libkrb5support.so.0 with link time reference
Traceback (most recent call last):
File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main
"main", fname, loader, pkg_name)
File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/usr/lib64/az/lib/python2.7/site-packages/azure/cli/main.py", line 12, in
from azure.cli.core import get_default_cli
File "/usr/lib64/az/lib/python2.7/site-packages/azure/cli/core/init.py", line 12, in
from pkg_resources import parse_version
File "/usr/lib64/az/lib/python2.7/site-packages/pkg_resources/init.py", line 36, in
import email.parser
File "/usr/lib64/python2.7/email/parser.py", line 12, in
from email.feedparser import FeedParser
File "/usr/lib64/python2.7/email/feedparser.py", line 27, in
from email import message
File "/usr/lib64/python2.7/email/message.py", line 16, in
import email.charset
File "/usr/lib64/python2.7/email/charset.py", line 13, in
import email.base64mime
File "/usr/lib64/python2.7/email/base64mime.py", line 40, in
from email.utils import fix_eols
File "/usr/lib64/python2.7/email/utils.py", line 27, in
import random
File "/usr/lib64/python2.7/random.py", line 49, in
import hashlib as _hashlib
File "/usr/lib64/python2.7/hashlib.py", line 116, in
import _hashlib
ImportError: /usr/lib64/libgssapi_krb5.so.2: symbol k5_buf_free, version krb5support_0_MIT not defined in file libkrb5support.so.0 with link time reference
Traceback (most recent call last):
File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main
"main", fname, loader, pkg_name)
File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/usr/lib64/az/lib/python2.7/site-packages/azure/cli/main.py", line 12, in
from azure.cli.core import get_default_cli
File "/usr/lib64/az/lib/python2.7/site-packages/azure/cli/core/init.py", line 12, in
from pkg_resources import parse_version
File "/usr/lib64/az/lib/python2.7/site-packages/pkg_resources/init.py", line 36, in
import email.parser
File "/usr/lib64/python2.7/email/parser.py", line 12, in
from email.feedparser import FeedParser
File "/usr/lib64/python2.7/email/feedparser.py", line 27, in
from email import message
File "/usr/lib64/python2.7/email/message.py", line 16, in
import email.charset
File "/usr/lib64/python2.7/email/charset.py", line 13, in
import email.base64mime
File "/usr/lib64/python2.7/email/base64mime.py", line 40, in
from email.utils import fix_eols
File "/usr/lib64/python2.7/email/utils.py", line 27, in
import random
File "/usr/lib64/python2.7/random.py", line 49, in
import hashlib as _hashlib
File "/usr/lib64/python2.7/hashlib.py", line 116, in
import _hashlib
ImportError: /usr/lib64/libgssapi_krb5.so.2: symbol k5_buf_free, version krb5support_0_MIT not defined in file libkrb5support.so.0 with link time reference

I have tried the option of appending the proxy certificate to the .pem files and tried still the issue exists.

I am using iKNOW scheduler for scheduling the job and have wriitten the azcopy service in bash script.

You need to use a .PEM file for the certificate type, rather than a .CER file (can convert it using OpenSSL)

[davidroberts63]

FYI for anyone else that finds this. I ended up needing to copy the PEM bundle (base64 encoded -- BEGIN*) file into the folder that the requests module is located. AND set the CA env var.

pip show requests

Copy your PEM CA bundle into the path specified by 'Location' + requests.
For example

...
Location: c:\python37\lib\site-packages
...

Copy the file into c:\python37\lib\site-packagerequests

Then set the REQUESTS_CA_BUNDLE to that path.

If you don't have the requests module installed go ahead and install it

pip install requests

Then go through the above steps. Looks like az cli will use the installed module if it exists.

[jabielecki]

Yeah, nope. The az as installed from base on my Ubuntu 18.04 LTS is definitely ignoring exported REQUESTS_CA_BUNDLE or AZURE_CLI_DISABLE_CONNECTION_VERIFICATION=1 and always using one hardcoded certificates file. I used strace to verify that.

This bash command was enough as a workaround for me (ugly and fragile):

cat <<-'ENDCERT' | sudo tee -a /opt/az/lib/python3.6/site-packages/certifi/cacert.pem
-----BEGIN CERTIFICATE-----
...PEM...
-----END CERTIFICATE-----
ENDCERT