Velero: Velero does not send client certificate when using custom cert option

[hmehra]

Describe the problem/challenge you have
When a self signed certificate is provided to Velero for the S3 object store with cacert option, it uses SSL with TLSv1.0 for the security handshake. TLS v1.0 is very old and the server rejects the handshake. This was done using Velero v1.4 and aws-plugin v1.1.0

Describe the solution you'd like
Velero should use TLSv1.2 for SSL handshakes and connections.

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

Environment:

• Velero version (use velero version):

velero version
Client:
Version: v1.4.0
Git commit: 5963650c9d64643daaf510ef93ac4a36b6483392
Server:
Version: v1.4.0

• Kubernetes version (use kubectl version):
  kubectl version
  Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.3", GitCommit:"2e7996e3e2712684bc73f0dec0200d64eec7fe40", GitTreeState:"clean", BuildDate:"2020-05-20T12:52:00Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
  Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.0", GitCommit:"2bd9643cee5b3b3a5ecbd3af49d09018f0773c77", GitTreeState:"clean", BuildDate:"2019-09-18T14:27:17Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}

• Kubernetes installer & version:

minikube version
minikube version: v1.4.0
commit: 7969c25a98a018b94ea87d949350f3271e9d64b6

• Cloud provider or hardware configuration:

• OS (e.g. from /etc/os-release):
  cat /etc/os-release
  NAME="Ubuntu"
  VERSION="18.04.4 LTS (Bionic Beaver)"
  ID=ubuntu
  ID_LIKE=debian
  PRETTY_NAME="Ubuntu 18.04.4 LTS"
  VERSION_ID="18.04"
  HOME_URL="https://www.ubuntu.com/"
  SUPPORT_URL="https://help.ubuntu.com/"
  BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
  PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
  VERSION_CODENAME=bionic
  UBUNTU_CODENAME=bionic

• Object store
  Proprietary object store which is compatible with S3 protocol. The server enforces strict TLS check. Any version below v1.2 is rejected.

• Repro steps

Install Velero using self signed certificate
velero install --use-restic --provider aws --bucket k8s-backup-view --secret-file ./secret --cacert ./ssl_cert.pem --use-volume-snapshots=false --backup-location-config region=default,s3ForcePathStyle="true",s3Url=https://sv4-dell87-c3-ve02.com:3000 --plugins velero/velero-plugin-for-aws:v1.1.0

Error seen -

kubectl logs deployment/velero -n velero
time="2020-08-14T20:51:13Z" level=info msg="setting log-level to INFO" logSource="pkg/cmd/server/server.go:177"
time="2020-08-14T20:51:13Z" level=info msg="Starting Velero server v1.4.0 (5963650c9d64643daaf510ef93ac4a36b6483392)" logSource="pkg/cmd/server/server.go:179"
time="2020-08-14T20:51:13Z" level=info msg="1 feature flags enabled []" logSource="pkg/cmd/server/server.go:181"
time="2020-08-14T20:51:14Z" level=info msg="registering plugin" command=/velero kind=BackupItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/crd-remap-version
time="2020-08-14T20:51:14Z" level=info msg="registering plugin" command=/velero kind=BackupItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/pod
time="2020-08-14T20:51:14Z" level=info msg="registering plugin" command=/velero kind=BackupItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/pv
time="2020-08-14T20:51:14Z" level=info msg="registering plugin" command=/velero kind=BackupItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/service-account
time="2020-08-14T20:51:14Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/add-pv-from-pvc
time="2020-08-14T20:51:14Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/add-pvc-from-pod
time="2020-08-14T20:51:14Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/change-pvc-node-selector
time="2020-08-14T20:51:14Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/change-storage-class
time="2020-08-14T20:51:14Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/cluster-role-bindings
time="2020-08-14T20:51:14Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/crd-preserve-fields
time="2020-08-14T20:51:14Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/job
time="2020-08-14T20:51:14Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/pod
time="2020-08-14T20:51:14Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/restic
time="2020-08-14T20:51:14Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/role-bindings
time="2020-08-14T20:51:14Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/service
time="2020-08-14T20:51:14Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/service-account
time="2020-08-14T20:51:14Z" level=info msg="registering plugin" command=/plugins/velero-plugin-for-aws kind=VolumeSnapshotter logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/aws
time="2020-08-14T20:51:14Z" level=info msg="registering plugin" command=/plugins/velero-plugin-for-aws kind=ObjectStore logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/aws
time="2020-08-14T20:51:14Z" level=info msg="Checking existence of namespace" logSource="pkg/cmd/server/server.go:361" namespace=velero
time="2020-08-14T20:51:14Z" level=info msg="Namespace exists" logSource="pkg/cmd/server/server.go:367" namespace=velero
time="2020-08-14T20:51:16Z" level=info msg="Checking existence of Velero custom resource definitions" logSource="pkg/cmd/server/server.go:396"
time="2020-08-14T20:51:16Z" level=info msg="All Velero custom resource definitions exist" logSource="pkg/cmd/server/server.go:430"
time="2020-08-14T20:51:16Z" level=info msg="Checking that all backup storage locations are valid" logSource="pkg/cmd/server/server.go:437"
An error occurred: some backup storage locations are invalid: backup store for location "default" is invalid: rpc error: code = Unknown desc = RequestError: send request failed
caused by: Get https://sv4-dell87-c3-ve02.com:3000/k8s-backup-view?delimiter=%2F&list-type=2&prefix=: remote error: tls: alert(116)

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.

• :+1: for "The project would be better with this feature added"
• :-1: for "This feature will not enhance the project in a meaningful way"

[ashish-amarnath]

Thank you for reporting this issue.
Velero doesn't initiate the SSL handshake to negotiate the TLS versions. This is handled by the aws sdk that Velero uses.
We'll investigate how this can be configured.

[hmehra]

@ashish-amarnath On digging deep within the S3 server, we have determined the TLS version is negotiated correctly. But the client does not send its certificate to the server. This causes the server to reset the connection.

[ashish-amarnath]

@hmehra thanks for the update. Is there any information indicative of why the cert wasn't used?
Also, FYI, Velero, doesn't handle the ssl handshake and negotiation directly. We use the AWS sdk that handles the ssl connection setup.

[jenting]

@hmehra,
maybe you could change the https://github.com/vmware-tanzu/velero-plugin-for-aws to add the TLS min version to 1.2 on the HTTP client default transport when caCert provided.

    if len(caCert) > 0 {
        serverConfig.HTTPClient = &http.Client{
            Transport: &http.Transport{
                TLSClientConfig: &tls.Config{
                    PreferServerCipherSuites: true,
                    MinVersion:               tls.VersionTLS12,
                },
            },
        }
    }

[hmehra]

@ashish-amarnath I am not sure why Velero as a client did not include its certificate while the SSL handshake was done. From TLS 1.3 spec https://tools.ietf.org/html/rfc8446, verifying client certificate could be optional. Changing this on the server made it work. I am not sure if you guys want to fix this in Velero, where it would send a certificate for the handshake.
It would be better to edit the issue title as well. Please let me know.

[ashish-amarnath]

@hmehra thanks for figuring this out. I am not sure what changes are required in Velero for this. The SSL connection setup is handled by the AWS SDK.
At the very least, we would like this to be documented. Can you please open a docs PR for this?

[hmehra]

I have edited the issue title. Will open a docs PR.

[carlisia]

Hey @hmehra, are you still able to help out by adding documentation for this? It'd be great, thank you!

[a-mccarthy]

We should probably this call out on the https://velero.io/docs/main/self-signed-certificates/

@hmehra are you still able to work on this? thanks!

[hmehra]

@a-mccarthy This slipped my to do list. Will send a docs PR in the weekend.

[eleanor-millman]

@hmehra Any chance you can submit your PR? We'd love to get this in.

[hmehra]

Opened PR-3811