Charts: [bitnami/harbor] notary-server --> notary-signer fails to verify CA/certificate

[dkulchinsky]

Which chart:
harbor v7.1.0

Describe the bug
notary-server is unable to communicate with notary-signer using the auto-generated CA and cert/key pair, throws an error:

{"level":"error","msg":"Trust not fully operational: rpc error: code = 13 desc = connection error: desc = \"transport: x509: certificate signed by unknown authority (possibly because of \\\"crypto/rsa: verification error\\\" while trying to verify candidate authority certificate \\\"harbor-notary-ca\\\")\"","time":"2020-08-28T15:16:58Z"}

on the notary-signer side, I see:

2020/08/28 15:16:57 grpc: Server.Serve failed to complete security handshake from "10.2.0.40:49258": EOF

notary shows up in unhealthy state:

    {
      "name": "notary",
      "status": "unhealthy",
      "error": "received unexpected status code: 503 {\"Trust operational\":\"rpc error: code = 13 desc = connection error: desc = \\\"transport: x509: certificate signed by unknown authority (possibly because of \\\\\\\"crypto/rsa: verification error\\\\\\\" while trying to verify candidate authority certificate \\\\\\\"harbor-notary-ca\\\\\\\")\\\"\"}"
    },

To Reproduce
Steps to reproduce the behavior:

1. Deploy harbor Chart in K8s using the values.yaml
2. Don't use custom CA/Certs (i.e. use auto-generated ones)
3. Don't enable internal TLS
4. wait for all the Harbor components to come up
5. check Harbor health (and notary-server/signer logs)

Expected behavior
notary-server should be able to connect to notary-signer with the auto-generated CA and Certificate.

Version of Helm and Kubernetes:

• Output of helm version:

version.BuildInfo{Version:"v3.3.0", GitCommit:"8a4aeec08d67a7b84472007529e8097ec3742105", GitTreeState:"dirty", GoVersion:"go1.14.6"}

• Output of kubectl version:

Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.6", GitCommit:"dff82dc0de47299ab66c83c626e08b245ab19037", GitTreeState:"clean", BuildDate:"2020-07-16T00:04:31Z", GoVersion:"go1.14.4", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.11-gke.5", GitCommit:"baccd25d44f1a0d06ad3190eb508784efbb990a5", GitTreeState:"clean", BuildDate:"2020-06-25T22:55:26Z", GoVersion:"go1.13.9b4", Compiler:"gc", Platform:"linux/amd64"}

Additional context

Harbor is deployed behind an Ingress Controller which servers a GlobalSign certificate, internal TLS is disabled, notary certs are auto-generated by Helm during deployment.

Happy to provide additional info/context!

I've tried to verify the validity of the CA & Certificate using other tools and all seem to suggest they are OK:

1. with curl:

$ curl -v https://harbor-notary-signer:7899 --cacert /etc/notary/notary-signer-ca.crt
.
.
.
* Connected to harbor-notary-signer (172.16.153.186) port 7899 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/notary/notary-signer-ca.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=harbor-notary-signer
*  start date: Aug 28 14:33:55 2020 GMT
*  expire date: Aug 28 14:33:55 2021 GMT
*  common name: harbor-notary-signer (matched)
*  issuer: CN=harbor-notary-ca
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x564202ce5f50)
> GET / HTTP/2
> Host: harbor-notary-signer:7899
> User-Agent: curl/7.64.0
> Accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Connection state changed (MAX_CONCURRENT_STREAMS == 4294967295)!
< HTTP/2 200
< content-type: application/grpc
< grpc-status: 3
< grpc-message: malformed method name: "/"

1. with openssl

$ openssl verify -CAfile notary-signer-ca.crt notary-signer.crt
notary-signer.crt: OK

[dkulchinsky]

also tested with openssl s_client from the notary-server pod to the notary-signer pod:

I have no name!@harbor-notary-server-6f6878898-pqhjs:/etc/notary$ openssl s_client -CAfile notary-signer-ca.crt -connect harbor-notary-signer:7899

CONNECTED(00000003)
Can't use SSL_get_servername
depth=1 CN = harbor-notary-ca
verify return:1
depth=0 CN = harbor-notary-signer
verify return:1
---
Certificate chain
 0 s:CN = harbor-notary-signer
   i:CN = harbor-notary-ca
---
Server certificate
-----BEGIN CERTIFICATE-----
<blob>
-----END CERTIFICATE-----
subject=CN = harbor-notary-signer

issuer=CN = harbor-notary-ca

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 1315 bytes and written 347 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---

I'm quite lost here... every single test I do suggest that these certs are correct and verifiable, but still the two services fail to communicate... are we looking at some possible Golang module/dependency issue here? a notary bug of some sort?

[carrodher]

Hi, thanks for reporting this issue; during the following week(s) we have an internal task to work on some open issues related to Harbor and CA certificates/TLS like https://github.com/bitnami/charts/issues/3486 and https://github.com/bitnami/charts/issues/3489; I just added this issue to this task in order to take a look just in case is something related. If not, we'll create a different task to investigate it. Thanks!

[fcrespofastly]

Thanks @carrodher !! ;)

[fcrespofastly]

I'm quite lost here... every single test I do suggest that these certs are correct and verifiable, but still the two services fail to communicate... are we looking at some possible Golang module/dependency issue here? a notary bug of some sort?

Just in case it helps, I've just written a small grpc client pointing to notary-server grpc port (port-forwarded to my laptop) replicating what I think getTrustservice does: https://github.com/theupdateframework/notary/blob/84287fd8df4f172c9a8289641cdfa355fc86989d/cmd/notary-server/config.go#L145

This is the go code I used to connect to notary-signer:

package main

import (
    "context"
    "log"
    "time"

    "github.com/docker/go-connections/tlsconfig"
    "google.golang.org/grpc"
    "google.golang.org/grpc/credentials"
    healthpb "google.golang.org/grpc/health/grpc_health_v1"
)

func healthCheck(d time.Duration, hc healthpb.HealthClient, serviceName string) (*healthpb.HealthCheckResponse, error) {
    ctx, cancel := context.WithTimeout(context.Background(), d)
    defer cancel()
    req := &healthpb.HealthCheckRequest{
        Service: serviceName,
    }
    return hc.Check(ctx, req)
}

func main() {
    tlsConfig, err := tlsconfig.Client(tlsconfig.Options{
        CAFile:             "notary-signer-ca.crt",
        ExclusiveRootPools: true,
    })

    conn, err := grpc.Dial("harbor-notary-signer:7899", grpc.WithTransportCredentials(credentials.NewTLS(tlsConfig)))
    if err != nil {
        log.Fatalf("did not connect: %v", err)
    }
    defer conn.Close()
    log.Printf("CONNECTED SUCCESSFULLY!!!!")
    out, err := healthCheck(10*time.Second, healthpb.NewHealthClient(conn), "grpc.health.v1.Health.KeyManagement")
    if err != nil {
        log.Fatalf("Health Check error: %v", err)
    }
    log.Printf(out.String())
}

The result:

2020/09/02 12:35:03 CONNECTED SUCCESSFULLY!!!!
2020/09/02 12:35:05 status:SERVING

I compiled this code with go 1.13.8 (the one harbor is using from their makefile) and go 1.14.3 and it was both a success.

So, this is weird, I don't think it's a go dependency issue... but dunno ...
I'm still failing to understand why it does not work, though. From the configuration point of view, certs and their location all seem fine to me.

I also captured traffic between both services and as opposite to what the log suggests, the TLS handshake lgtm - We can't catch the cert being sent in the Server Hello response since from TLS 1.3 that part of the handshake it's also encryped, so wireshark only sees encrypted data. But from the code, I don't think it's failing when dialing to the grpc endpoint, but when invoking the health check grpc call, for what I think the TLS handshake was already stablished.

What I see after the handshake it's just TLS encrypted data, so I can't really assert if those are the healthchecks and maybe wireshark just label it as TLS1.3, but from the point in code we think it fails, it seems so.

Screenshot from wireshark:

This looks like a nice gremlin 😅

[fcrespofastly]

Also tested with goharbor/notary-server|signer-photon images, the same issue. So this is not a problem in bitnami docker images for sure.

[dkulchinsky]

I've extracted the notary server & signer binaries from the bitnami docker images and ran them locally on an Ubuntu VM with the same config and certificates and it just works... which confirms @fcrespofastly's test.

This leads me to think that there's something wonky going on int the K8s network layer which causes this issue, we're running in GKE btw.

[dkulchinsky]

Our test/sandbox cluster has a linkerd deployment (which is off by default), so I enabled it in the harbor namespace and restarted the notary server & signer deployments and now they connect and notary reports back as healthy... this is so bizzare, I have no idea what could be causing TLS to fail in such a way and why it does work with linkerd, will keep investigating.

[carrodher]

Thanks, both for the detailed explanation, @joancafom is looking into other issues that can be related, mainly focus on CA certs, TLS, etc. I am pinging him to take into account your information when looking into it. Thanks again!

[dkulchinsky]

@carrodher we've been further investigating this issue and have a better reproduction.

Since each helm deployment results in a newly generated notary certificates, the pods are also restarted and result in above error, if the pods are then manually deleted the new pods are mounting the existing certificates and work just fine.

We've worked around this by using fixed certificates and avoid recreating them on each deploy.

Our suspicion is that there is some kind of race condition, and either the new pods (after a helm upgrade) get attached to old certificates or the new certificates are not yet available.

[dani8art]

Thank you so much for the effort here, we will be working on a new release for both containers and helm chart and this work will be very useful, we will keep you updated!