Smart_open: [Question] Is it possible to disable https certificate validation?

[qiuwei]

Problem description

I use minio (https://min.io) to self-host an s3 compatible object storage. The service uses a self signed certificate. Accessing objects using smart open gives

SSLError: SSL validation failed for https://blahblah [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate (_ssl.c:1076)

I am wondering whether I could disable the certificate validation.

• What are you trying to achieve?
  Accessing self-hosted s3-compatible object storage with a self-signed certificate.
• What is the expected result?
  Accessing objects without problems.
• What are you seeing instead?
  Got SSL problem

Steps/code to reproduce the problem

1. start a local minio instance with a self-signed certificate according to the documents. https://docs.min.io/docs/deploy-minio-on-docker-compose.html
2. try accessing objects stored on minio using smart-open

Versions

Please provide the output of:

import platform, sys, smart_open
print(platform.platform())
print("Python", sys.version)
print("smart_open", smart_open.__version__)

Darwin-19.0.0-x86_64-i386-64bit
Python 3.7.5 (default, Oct 25 2019, 10:52:18)
[Clang 4.0.1 (tags/RELEASE_401/final)]
smart_open 1.9.0

Checklist

Before you create the issue, please make sure you have:

• [x] Described the problem clearly
• [x] Provided a minimal reproducible example, including any required data
• [x] Provided the version numbers of the relevant software

[piskvorky]

"try accessing objects stored on minio using smart-open" is too vague.
Please include a full reproducible example.

[fakhavan]

I have the same issue using this smart_open to access S3 on minio. As you know, minio can simulate S3 storage which I use to test code before deploying into production.
There is a guide on how boto3 could be used to access minio. You can take a look at here.

I have no issue accessing it using boto3 however when I try to use smart_open to do this, I receive botocore.exceptions.SSLError. I use s3://minioadmin:minioadmin@localhost:9000@data-test-bucket as URL and receive the following error:

botocore.exceptions.SSLError: SSL validation failed for https://localhost:9000/phq-dataeng/individual_event/1593661092355.tsv?uploads [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:852)

It would be great to have an option not to enforce SSL in the smart_open

[mpenkov]

I think it should be possible. From your link:

s3 = boto3.resource('s3',
                    endpoint_url='http://localhost:9000',
                    aws_access_key_id='YOUR-ACCESSKEYID',
                    aws_secret_access_key='YOUR-SECRETACCESSKEY',
                    config=Config(signature_version='s3v4'),
                    region_name='us-east-1')

You can pass those same arguments to smart_open, e.g.

# Stream to Digital Ocean Spaces bucket providing credentials from boto3 profile
transport_params = {
    'session': boto3.Session(profile_name='digitalocean'),  # may not be necessary
    'resource_kwargs': {
        'endpoint_url': 'https://ams3.digitaloceanspaces.com',
        # put your code here
    }
}
with open('s3://bucket/key.txt', 'wb', transport_params=transport_params) as fout:
    fout.write(b'here we stand')

See https://github.com/RaRe-Technologies/smart_open#more-examples for more.

[fakhavan]

@mpenkov Thank you for your quick reply. I was able to make it work.
For future reference, this is how I was able to use smart_open with S3 on Minio:

transport_params = {
    'resource_kwargs': {
        'endpoint_url': 'http://localhost:9000',
        'aws_access_key_id':'minioadmin',
        'aws_secret_access_key':'minioadmin',
    }
}

with open('s3://data-test-bucket/test.txt', 'wb', transport_params=transport_params) as fout:
    fout.write(b'Yay I work on S3 on minio!')

I'd suggest adding it to the examples as well as it's not clear regarding this kind of S3-like solutions.

[mpenkov]

I'll think about adding it. One of the problems is that we like to keep our documentation self-testing, and it's difficult to test this minio stuff without jumping through hoops (e.g. docker compose).