Boto3: Problem with SSL Validation

[robert9111]

Describe the bug
After deploying to prod I have started receiving an error with SSL Validation failed. boto3 is tagged at 1.15.2

Steps to reproduce
Don't have steps to reproduce, it only occurs when i push my repo to production.

Expected behavior
Previously the device was reaching out to endpoint: https://data.iot.us-west-2.amazonaws.com/things/xxx/shadow and retrieving the device information

[robert9111]

zenbath-chalice - ERROR - Caught exception for
Traceback (most recent call last):
File "/var/task/urllib3/connectionpool.py", line 699, in urlopen
httplib_response = self._make_request(
File "/var/task/urllib3/connectionpool.py", line 382, in _make_request
self._validate_conn(conn)
File "/var/task/urllib3/connectionpool.py", line 1010, in _validate_conn
conn.connect()
File "/var/task/urllib3/connection.py", line 411, in connect
self.sock = ssl_wrap_socket(
File "/var/task/urllib3/util/ssl_.py", line 428, in ssl_wrap_socket
ssl_sock = _ssl_wrap_socket_impl(
File "/var/task/urllib3/util/ssl_.py", line 472, in _ssl_wrap_socket_impl
return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
File "/var/lang/lib/python3.8/ssl.py", line 500, in wrap_socket
return self.sslsocket_class._create(
File "/var/lang/lib/python3.8/ssl.py", line 1040, in _create
self.do_handshake()
File "/var/lang/lib/python3.8/ssl.py", line 1309, in do_handshake
self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1124)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/task/botocore/httpsession.py", line 254, in send
urllib_response = conn.urlopen(
File "/var/task/urllib3/connectionpool.py", line 755, in urlopen
retries = retries.increment(
File "/var/task/urllib3/util/retry.py", line 506, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/var/task/urllib3/packages/six.py", line 734, in reraise
raise value.with_traceback(tb)
File "/var/task/urllib3/connectionpool.py", line 699, in urlopen
httplib_response = self._make_request(
File "/var/task/urllib3/connectionpool.py", line 382, in _make_request
self._validate_conn(conn)
File "/var/task/urllib3/connectionpool.py", line 1010, in _validate_conn
conn.connect()
File "/var/task/urllib3/connection.py", line 411, in connect
self.sock = ssl_wrap_socket(
File "/var/task/urllib3/util/ssl_.py", line 428, in ssl_wrap_socket
ssl_sock = _ssl_wrap_socket_impl(
File "/var/task/urllib3/util/ssl_.py", line 472, in _ssl_wrap_socket_impl
return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
File "/var/lang/lib/python3.8/ssl.py", line 500, in wrap_socket
return self.sslsocket_class._create(
File "/var/lang/lib/python3.8/ssl.py", line 1040, in _create
self.do_handshake()
File "/var/lang/lib/python3.8/ssl.py", line 1309, in do_handshake
self._sslobj.do_handshake()
urllib3.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1124)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/task/chalice/app.py", line 1135, in _get_view_function_response
response = view_function(function_args)
File "/var/task/chalicelib/utils/auth.py", line 124, in func_wrapper
res = func(kwargs)
File "/var/task/chalicelib/utils/auth.py", line 48, in validate_invoke
return func(args, kwargs)
File "/var/task/chalicelib/controllers/device.py", line 227, in change_settings
return device.change_shadow(device_blueprint.current_request.json_body)
File "/var/task/shine/services/device.py", line 86, in change_shadow
response = iot_data_plane.update_thing_shadow(
File "/var/task/botocore/client.py", line 357, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/var/task/thundra/integrations/modules/botocore.py", line 27, in _wrapper
return INTEGRATIONS['default'].run_and_trace(
File "/var/task/thundra/integrations/base_integration.py", line 68, in run_and_trace
raise exception
File "/var/task/thundra/integrations/base_integration.py", line 40, in run_and_trace
response = self.actual_call(wrapped, args, kwargs)
File "/var/task/thundra/integrations/base_integration.py", line 73, in actual_call
return wrapped(args, *kwargs)
File "/var/task/botocore/client.py", line 662, in _make_api_call
http, parsed_response = self._make_request(
File "/var/task/botocore/client.py", line 682, in _make_request
return self._endpoint.make_request(operation_model, request_dict)
File "/var/task/botocore/endpoint.py", line 102, in make_request
return self._send_request(request_dict, operation_model)
File "/var/task/botocore/endpoint.py", line 136, in _send_request
while self._needs_retry(attempts, operation_model, request_dict,
File "/var/task/botocore/endpoint.py", line 253, in _needs_retry
responses = self._event_emitter.emit(
File "/var/task/botocore/hooks.py", line 356, in emit
return self._emitter.emit(aliased_event_name, *kwargs)
File "/var/task/botocore/hooks.py", line 228, in emit
return self._emit(event_name, kwargs)
File "/var/task/botocore/hooks.py", line 211, in _emit
response = handler(kwargs)
File "/var/task/botocore/retryhandler.py", line 183, in __call__
if self._checker(attempts, response, caught_exception):
File "/var/task/botocore/retryhandler.py", line 250, in __call__
should_retry = self._should_retry(attempt_number, response,
File "/var/task/botocore/retryhandler.py", line 277, in _should_retry
return self._checker(attempt_number, response, caught_exception)
File "/var/task/botocore/retryhandler.py", line 316, in __call__
checker_response = checker(attempt_number, response,
File "/var/task/botocore/retryhandler.py", line 222, in __call__
return self._check_caught_exception(
File "/var/task/botocore/retryhandler.py", line 359, in _check_caught_exception
raise caught_exception
File "/var/task/botocore/endpoint.py", line 200, in _do_get_response
http_response = self._send(request)
File "/var/task/botocore/endpoint.py", line 269, in _send
return self.http_session.send(request)
File "/var/task/botocore/httpsession.py", line 281, in send
raise SSLError(endpoint_url=request.url, error=e)
botocore.exceptions.SSLError: SSL validation failed for https://data.iot.us-west-2.amazonaws.com/things/xxx/shadow [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1124)

[rabryan]

Having the same exact issue in us-east-2

[swetashre]

@robert9111 - Thank you for your post. I would some more information in order to debug the issue.

• Could you provide a minimal example of the code you're using to produce this error?

• Could you provide sanitized debug logs by adding: boto3.set_stream_logger('').

[dbasedow]

I had the same issue in eu-central-1. AWS support advised to switch to the ATS endpoint: https://aws.amazon.com/blogs/iot/aws-iot-core-ats-endpoints/
I think there is a different underlying reason this issue suddenly popped up. But for now this works.

[rabryan]

@dbasedow that works for me as well. For anyone else, to use the ATS endpoint you need to explicitly specify it when you create your iot-data client:

boto3.client('iot-data', endpoint_url=IOT_DATA_EP)

where IOT_DATA_EP is the output of this command (with https:// prepended)

aws iot describe-endpoint --endpoint-type iot:Data-ATS

[robert9111]

Above commend DID solve the issue. Appreciate it. Definitely unsure what was causing the issue.

[mretallack]

Hi, I have had this issue as well.

For me, we also use Request, which pulls in certifi.

https://github.com/certifi/python-certifi/commits/master

certifi has been updated from 2020.11.08 to 2020.12.05, and during this time, the "VeriSign Class 3 Public Primary Certification Authority - G5" has been removed. For us this means that the "data.iot.eu-west-2.amazonaws.com" is no longer validated.

2020.11.08:
https://github.com/certifi/python-certifi/blob/015cba9d2492a4cddaf5efe40666c18a2b259c93/certifi/cacert.pem

2020.12.05:
https://github.com/certifi/python-certifi/blob/45a64658872a94a83c4b70fce02a96f0f29895e6/certifi/cacert.pem

So for now we have had to roll back python-certifi until a correct solution is found.

Not sure what the real/correct solution would be (other than using "endpoint_url=IOT_DATA_EP" as above).

[johnp789]

This is one way to discover the ATS endpoint within an application, assuming IOT_REGION has been set.

def get_endpoint():
    iot_client = boto3.client("iot", region_name=os.getenv("IOT_REGION"))
    return iot_client.describe_endpoint(endpointType="iot:Data-ATS").get("endpointAddress")

IOT_DATA_ENDPOINT = f"https://{get_endpoint()}"

Pass IOT_DATA_ENDPOINT in the endpoint_url parameter when creating an iot-data client.

[swetashre]

This error is because currently the endpoint is not sending a valid cert. So it is recommended to use either ATS datapoint. IoT data endpoint is currently legacy.

https://docs.aws.amazon.com/iot/latest/developerguide/iot-connect-devices.html

Every customer has an iot:Data-ATS and an iot:Data endpoint. Each endpoint uses an X.509 certificate to authenticate the client. We strongly recommend that customers use the newer iot:Data-ATS endpoint type to avoid issues related to the widespread distrust of Symantec certificate authorities. We provide the iot:Data endpoint for devices to retrieve data from old endpoints that use VeriSign certificates for backward compatibility.