Serving: SSL hangs with network certmanager

[alexnederlof]

What version of Knative?

0.13.x

Expected Behavior

HTTPS to work

Actual Behavior

• https (and other clients) hang during the handshake, and then times out.
• http works without a problem

Steps to Reproduce the Problem

Follow installation instructions with:

• Istio 1.3.8 without side cards
• setup custom domain
• cert manager 0.14 (or 0.12, doesn't matter)
• Knative using helm
• Deploy simple HTTP hello world app

I observe the following:

1. Routes report they are OK
2. kingress reports things are OK

However
Istio ingress gateway logs:

[warning][config] [external/envoy/source/common/config/grpc_mux_subscription_impl.cc:81] gRPC config for type.go │ │ ogleapis.com/envoy.api.v2.Listener rejected: Error adding/updating listener(s) 0.0.0.0_443: Invalid path: tls.crt

looking at the pod it tries to mount these secrets:

istio-certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  istio.istio-ingressgateway-service-account
    Optional:    true
  ingressgateway-certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  istio-ingressgateway-certs
    Optional:    true
  ingressgateway-ca-certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  istio-ingressgateway-ca-certs
    Optional:    true
  istio-ingressgateway-service-account-token-k24n8:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  istio-ingressgateway-service-account-token-k24n8
    Optional:    false

But only istio-ingressgateway-service-account-token-k24n8 and istio.istio-ingressgateway-service-account exist, and they do not contain the certificates of the hello world service.

When I got to the gateways, and inspect the knative-ingress-gateway I see that it references the hello secret:

servers:
  - hosts:
    - hello2.default.kn.my.domain
    port:
      name: default/hello2:0
      number: 443
      protocol: HTTPS
    tls:
      credentialName: hello2-1fd49062-d50b-406a-bf58-c87c170554aa
      mode: SIMPLE
      privateKey: tls.key
      serverCertificate: tls.crt

However, looking at the kcert the secret name is route-f0b2b768-d98d-4a7c-bdea-e9084c087eec not hello2-1fd49062-d50b-406a-bf58-c87c170554aa.

[ZhiminXiang]

Could you try cert-manager 0.12 and see if it works?

[alexnederlof]

Yes, same thing happens with cert-manager 0.12. Also note that the certificates are issued correctly and all services seem to be aware that they have been successfully issued

[tcnghia]

/assign @ZhiminXiang

[nak3]

Hi @alexnederlof

May I ask you how did you install Istio? If you have the template or helm values, we would like you to share it. Especially I am wondering if you missed to enable SDS as docs described:

https://knative.dev/docs/install/installing-istio/

The below helm flag is needed in your helm command to enable SDS:
--set gateways.istio-ingressgateway.sds.enabled=true

For your concern:

However, looking at the kcert the secret name is route-f0b2b768-d98d-4a7c-bdea-e9084c087eec not hello2-1fd49062-d50b-406a-bf58-c87c170554aa.

The name difference is no problem. The secret in istio-system ns should refer to the original secret in your app's ns. You can verify it like this:

$ kubectl get secret -n istio-system  hello-example-cc9dc684-52a9-409f-9f52-dc55d22d13d7 -o yaml |grep origin
    networking.internal.knative.dev/originSecretName: route-8496f9e0-7de6-4142-8bd5-1f8f27b9c4e3
    networking.internal.knative.dev/originSecretNamespace: serving-tests

$ kubectl get secrets -n serving-tests route-8496f9e0-7de6-4142-8bd5-1f8f27b9c4e3
NAME                                         TYPE                DATA   AGE
route-8496f9e0-7de6-4142-8bd5-1f8f27b9c4e3   kubernetes.io/tls   3      20m

If the originSecretName label points to wrong secret, it should be the problem, though...

[alexnederlof]

Damn, seems like I made a mistake in the YAML I created for istio, and indeed the SDS was switched off.

Thanks for your help!