Postgres-operator: Ability to add SSL certificate for S3

[aurelienmarie]

I am currently testing the operator and running Minio for S3 storage. I managed to do the logical backup with TLS disabled on Minio and it works fine.

However, i have this issue when i enable the TLS on Minio.

+ dump
+ /usr/lib/postgresql/10/bin/pg_dumpall
+ compress
+ pigz
++ estimate_size
++ /usr/lib/postgresql/10/bin/psql -tqAc 'select sum(pg_database_size(datname)::numeric) from pg_database;'
+ aws_upload 6377860
+ declare -r EXPECTED_SIZE=6377860
++ date +%s
+ PATH_TO_BACKUP=s3://databases/spilo/test-test-cluster/575fa115-2b52-4972-b4e2-e59fe967f962/logical_backups/1582638557.sql.gz
+ args=()
+ [[ ! -z 6377860 ]]
+ args+=("--expected-size=$EXPECTED_SIZE")
+ [[ ! -z https://minio.default.svc.cluster.local:9000 ]]
+ args+=("--endpoint-url=$LOGICAL_BACKUP_S3_ENDPOINT")
+ [[ ! -z '' ]]
+ aws s3 cp - s3://databases/spilo/test-test-cluster/575fa115-2b52-4972-b4e2-e59fe967f962/logical_backups/1582638557.sql.gz --expected-size=6377860 --endpoint-url=https://minio.default.svc.cluster.local:9000
upload failed: - to s3://databases/spilo/test-test-cluster/575fa115-2b52-4972-b4e2-e59fe967f962/logical_backups/1582638557.sql.gz SSL validation failed for https://minio.default.svc.cluster.local:9000/databases/spilo/test-test-cluster/575fa115-2b52-4972-b4e2-e59fe967f962/logical_backups/1582638557.sql.gz [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)

I've build my own docker image for backups adding args+=("--no-verify-ssl") in the dump.sh. As a result, i was able to backup my db with TLS enable on Minio.

The idea here is to add a parameter on the operatorconfiguration in order to either be able to specify a certificate, or disable the ssl verification.

Zalando postgres-operator version : v1.3.1

[FxKu]

There is currently a PR open to add TLS support for database pods. At the moment, the TLS volume is not passed to the logical backup pod. Would this be useful for you?

[aurelienmarie]

@FxKu my issue is not related to the TLS support for database pods but for the the backups cronjobs. It's required for wal-e/wal-g in order to target the S3 endpoint.

[Jonher937]

I also had this problem today.
This happens when using a S3 server that does not have a valid signed certificate.

It would be nice if the operator could accept something like a base64 encoded PEM certificate, this would then be passed to the Logical Backup container as a environment variable where dump.sh could base64 -d the content and save that to a mktmp file or similar.

Appending --ca-bundle=$CA_FILE" to the aws cli (or setting the AWS_CA_BUNDLE environment variable) so verification is in place instead of using --insecure (or allowing for both options)

• https://docs.aws.amazon.com/cli/latest/topic/config-vars.html#general-options
• https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html

AWS_CA_BUNDLE
Specifies the path to a certificate bundle to use for HTTPS certificate validation.

If defined, this environment variable overrides the value for the profile setting ca_bundle. You can override this environment variable by using the --ca-bundle command line parameter.

[blyry]

You can workaround this by using letsencrypt and putting a valid certificate on your minio deployment. You don't even need to make your endpoint public now that most dns providers support api based cname validation! That seems like a 'better' solution than supporting the --insecure on their own implementation.

Supporting custom TLS or root CAs on the operator side is also a good feature to add though, but then you still have to go through the process of signing, deploying and managing a cert for your minio installation.

[DrissiReda]

Is it possible to disable ssl altogether for minio logical backups? I can't find the option in documentation.