Ambassador: istio 1.5 breaks secret handling for mTLS

[ppeble]

Describe the bug

istio 1.5 introduces a breaking change to how secrets are handled. Secrets are no longer written into the namespace, which is how ambassador currently consumes the secrets for use in mTLS.

This only applies to STRICT mode in mTLS and only applies if you directly install istio 1.5 as a fresh install via istioctl. If you are upgrading from 1.4 to 1.5 then the old secret infrastructure remains. See https://discuss.istio.io/t/istios-helm-support-in-2020/5535/12 for confirmation of this behavior.

See the upgrade notes for more in-depth explanations of what is changing in regards to secrets.

To Reproduce
Steps to reproduce the behavior:

1. Install istio 1.5 using istioctl as a fresh install. Ensure you are using strict mTLS.
2. Install Ambassador (any version post 1.0)
3. Configure TLS by following the istio documentation
4. Configure a mapping to use the new TLSContext
5. Attempt to curl that mapping. You will see a 'connection reset' error due to the lack of secret availability as the istio sidecar will reject the request.

Expected behavior

Ambassador needs to either change to read via the new grpc secret method or use istio sidecars so that the current deployment solutions continue to work without interruption when using strict mTLS.

Additional context

We will also need to upgrade documentation to provide paths for upgrading istio installations. The decision on the part of the istio team to have different behavior depending on whether you perform a fresh install or are upgrading means that we will need to provide instructions on both paths.

[ppeble]

I have a working implementation locally to allow for Ambassador to run with istio sidecars, which sidesteps the secret issue because all traffic and secrets would just be handled by istio. I am in the process of writing up a PR with my changes as well as writing documentation to explain it so we can have a discussion about the pros and cons of my approach.

I will say this: I would much prefer to not require istio sidecars but I do not have the bandwidth to attempt to reverse-engineer the grpc interface to obtain secrets directly from the new istiod process.

[MXfive]

There is another technique that might be possible which I will investigate today, it looks like it is potentially possible to apply some config maps that modify Istio sidecar injection only for ambassador to not capture external traffic and to mount the certificate path as a shared volume that ambassador can then also mount and access the certificates.

This is basically what the Gloo project recommend, except in their examples they are manually writing out the spec for the sidecar, rather than allowing the mutating webhook get the correct values and just changing what they need.

[ppeble]

I am definitely interested in investigating that more. My concern here is about certificate rotation in istio. In the past Ambassador mounted the certs as well (but directly from the k8s secret, not the sidecar) and the problem with that is that you don't get the updated certs if istio rotates them. Thinking back I'm not sure if that's because of the mounting or because of how Ambassador implemented it. But my question would be whether that sidecar mounting solution suffered from the same issue. If istio rotates the certificates will Gloo pick up the latest ones and load them correctly?

Link to the Gloo docs for this: https://docs.solo.io/gloo/latest/guides/integrations/service_mesh/gloo_istio_mtls/#istio-15x

Link to the istio blog post about obtaining certs in this manner: https://istio.io/blog/2020/proxy-cert/

Here is the PR that I submitted to no longer read certificates from mounted secrets but instead get them from k8s directly: https://github.com/datawire/ambassador/pull/2277

[ppeble]

As shown above the base-id change has been merged. With it you can run both Ambassador and an istio-proxy sidecar, which gets us a step closer. I'm going to continue to test this and then hopefully provide some updated doc once it is fully released.

Another update: It was pointed out by @kflynn that the grpc 'secret discovery service' is in fact _not_ an istio-specific thing but rather a piece of envoy functionality: https://www.envoyproxy.io/docs/envoy/latest/configuration/security/secret

I did not realize this! I was assuming we would have to almost reverse-engineer the grpc setup in istio but instead we might be able to leverage existing clients and/or configuration documentation to use built-in envoy functionality. I will have to spend some time researching this option in the near future.

[richarddli]

I was looking more closely into this issue. To summarize this thread, to enable mTLS there are two general things we want to test:

• getting Istio certs and telling Ambassador about them
• ensuring that an Istio cert rotation will cause the right behavior in Ambassador

The approaches discussed in this thread are:

1. Support Istio sidecar injection on the Ambassador pods. This change has been merged and will ship in our next release.
2. Use the Istio Proxy Cert technique (https://istio.io/blog/2020/proxy-cert/) and route to the Istio Ingress Gateway over mTLS.
3. Update Ambassador so that it supports letting the Istio SDS server supply secrets to Envoy and route to the Istio gateway.

@ppeble @MXfive @KowalczykBartek Would welcome any feedback on what would be the best approach.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[nicolasbelanger]

I have been investigating on the path of new ambassador and istio installations too in the last two weeks and here's what I can say. I have been able to run ambassador 1.6.2 with istio 1.6.8 with automatic sidecar injection (label istio.io/rev: canary on ambassador namespace) and these values added to my values.yaml:

env:
  AMBASSADOR_ENVOY_BASE_ID: "1"

podAnnotations:
  proxy.istio.io/config: |
    proxyMetadata:
      OUTPUT_CERTS: /etc/istio-certs
  sidecar.istio.io/userVolume: '[{"name": "istio-certs", "emptyDir": {"medium": "Memory"}}]'
  sidecar.istio.io/userVolumeMount: '[{"name": "istio-certs", "mountPath": "/etc/istio-certs"}]'
  traffic.sidecar.istio.io/includeInboundPorts: ''
  traffic.sidecar.istio.io/includeOutboundIPRanges: ''

volumes:
  - name: istio-certs
    emptyDir:
      medium: Memory

volumeMounts:
  - mountPath: /etc/istio-certs/
    name: istio-certs

Everything is running fine for 24 hours, and then comes the trouble:

ambassador/ambassador-57c99fd4b8-hnf2s[istio-proxy]: 2020-08-19T06:24:53.635384Z    info    sds resource:default pushed key/cert pair to proxy

The cert rotation happens as it should, but the refreshed certificate is not being picked up by ambassador, which results in errors

"upstream_transport_failure_reason":"TLS error: 268436501:SSL routines:OPENSSL_internal:SSLV3_ALERT_CERTIFICATE_EXPIRED"

Ambassador still runs with the expired mTLS certificate. It's pretty clear that the mounted certificate is not getting refreshed as it is rotated by the istio-proxy sidecar... Restarting the pods will indeed fix the issue until the next rotation...

To me, it's the only piece of the puzzle left to solve to give this migration a real go.

[nicolasbelanger]

Ok I'm a bit confused, but nonetheless worth mentioning: my current pod is 25h old and survived two info sds resource:default pushed key/cert pair to proxy, no SSLV3_ALERT_CERTIFICATE_EXPIRED to be found in the logs either...
Edit: Nope, false hope, SSLV3_ALERT_CERTIFICATE_EXPIRED are back

[nicolasbelanger]

To followup on my saga, I just experienced an edge case where the certs were not yet populated by the proxy, resulting in ambassador trying to connect to the upstream without certificate (presumably), and then got SSLV3_ALERT_HANDSHAKE_FAILURE...

[puneetloya]

any workarounds for this issue?

[puneetloya]

@nicolasbelanger I am trying to setup strict mTLS between ambassador and the upstream services its routing too.
I used the above method to do a helm deploy and ambassador and istio-proxy come up fine. As suggested from ambassador docs, I added a TLS context for istio.
I am hitting SSL errors even when the istio certs are not expired.

Can I get some help with any other istio settings(like Destination rules etc) you had to add to get it working end to end?

On the upstream service I also get this log:

http/1.1 protocol error hpe_invalid_method

TIA

[tomwganem]

Hello, I am the leader of the team that @puneetloya is on. We have been using the Ambassador for a while now and have been happy with it. But because we have a need to enforce mTLS in the whole cluster, and have decided on Istio to accomplish that goal, this is a critical issue for our team. We are now looking at alternatives to Ambassador. Any attention to this ticket in the near future would be really appreciated.

[ppeble]

To add to the comment above I have a feeling that I am also going to be in a similar boat sometime in the near future. We're heavily invested in both Ambassador and istio but to be honest I'm going to have a tough time convincing people that we shouldn't ditch Ambassador since istio functionality usage is more widespread.

Unfortunately I don't have the bandwidth right now to really dive deep into Ambassador source to look for a solution. If that changes in the future I'll post my findings here.

[puneetloya]

@kflynn are you planning to fix this issue?

[kflynn]

This should be fixed in Ambassador 1.10. Check out https://www.getambassador.io/docs/latest/howtos/istio/ for the docs. 🙂

[puneetloya]

@kflynn thanks a lot.