Ambassador: Ambassador doesn't like Istio mTLS secrets

[ovk]

Describe the bug
The Ambassador with Istio guide (https://www.getambassador.io/user-guide/with-istio) suggests to use the following annotation to attach Istio secrets:

apiVersion: ambassador/v1
  kind:  TLSContext
  name: istio-upstream
  hosts: []
  secret: istio.default

Which does not work with the following error:

ambassador.default.1: TLSContext istio-upstream found no certificate in secret istio in namespace default, ignoring...

Upon inspection of the source code https://github.com/datawire/ambassador/blob/528b16c14d26cece0be7719403debf6577e2e097/ambassador/ambassador/ir/ir.py#L291 it looks like the secret name istio.default is parsed incorrectly because the routine assumes that secret name is everything before dot (e.g. istio).

To Reproduce
Just try to follow the guide: https://www.getambassador.io/user-guide/with-istio
It cannot work because istio.default secret name will be parsed as istio.

Expected behavior
If you want to have namespace qualification for secrets, than I think it should be possible to do dot escaping, something like istio\.default to support cases where secret has dot in its name.

Versions (please complete the following information):

• Ambassador: 0.60.1
• Kubernetes environment: Minikube
• Version: k8s 1.14.0

[tomburns]

+1, my team ran into this on Friday -- is the Istio mutual TLS scenario described in the documentation no longer supported?

[zombor]

This is related to https://github.com/datawire/ambassador/issues/1255

[ovk]

Upon some investigation I also discovered some inconsistencies between how Istio stores secret istio.default and how Ambassador attempts to read it:

1. Secret type: Ambassador expects Opaque while Istio has istio.io/key-and-cert.
2. Key name. Ambassador expects tls.key while Istio has key.pem.
3. Cert name. Ambassador expects tls.crt while Istio has cert-chain.pem.

Given all this, looks like the mutual TLS between Ambassador and Istio is completely broken at the moment.

There is an easy temporary workaround though: just create secret manually, copy key and cert from the istio.default secret and name everything as mentioned above (also, don't put any dots into the secret name). Then use this secret for Ambassador.

[nbkrause]

Thank you for reporting this issue. It appears I did not completely check the correctness of the change to the documentation before publishing it.

Short term, the solution is to simply rollback to the old method of mounting the secret in a volume in the Ambassador container and use it there. Long term, we should work on getting this to work using a TLSContext for the sake of configuration uniformity.

The documentation rollback is happening now and should be completed soon. You can then follow that document to enable mTLS between Ambassador and Istio. I apologize for the inconvenience this has caused and appreciate the effort put in to reporting this issue.

[kflynn]

Changed the title to reflect that this is distinct from #1255.

[diptimanadak]

There is an easy temporary workaround though: just create secret manually, copy key and cert from the istio.default secret and name everything as mentioned above (also, don't put any dots into the secret name). Then use this secret for Ambassador.

Upon some investigation I also discovered some inconsistencies between how Istio stores secret istio.default and how Ambassador attempts to read it:

1. Secret type: Ambassador expects `Opaque` while Istio has ` istio.io/key-and-cert`.

2. Key name. Ambassador expects `tls.key` while Istio has `key.pem`.

3. Cert name. Ambassador expects `tls.crt` while Istio has `cert-chain.pem`.

Given all this, looks like the mutual TLS between Ambassador and Istio is completely broken at the moment.

There is an easy temporary workaround though: just create secret manually, copy key and cert from the istio.default secret and name everything as mentioned above (also, don't put any dots into the secret name). Then use this secret for Ambassador.

I am also stuck with the same issue, could you elaborate the steps, how we can do that

[diptimanadak]

To give more details about my configuration:

apiVersion: extensions/v1beta1 kind: Deployment metadata: name: ambassador spec: replicas: 3 template: metadata: annotations: sidecar.istio.io/inject: "false" labels: service: ambassador spec: serviceAccountName: ambassador containers: - name: ambassador image: quay.io/datawire/ambassador:0.73.0 resources: limits: cpu: 1 memory: 400Mi requests: cpu: 200m memory: 100Mi env: - name: AMBASSADOR_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace livenessProbe: httpGet: path: /ambassador/v0/check_alive port: 8877 initialDelaySeconds: 30 periodSeconds: 3 readinessProbe: httpGet: path: /ambassador/v0/check_ready port: 8877 initialDelaySeconds: 30 periodSeconds: 3 volumeMounts: - mountPath: /etc/istiocerts/ name: istio-certs readOnly: true restartPolicy: Always volumes: - name: istio-certs secret: optional: true secretName: istio.default

ambassador-service.yaml:
`apiVersion: v1
kind: Service
metadata:
labels:
service: ambassador
name: ambassador
annotations:
getambassador.io/config: |
---
apiVersion: ambassador/v1
kind: Mapping
name: httpbin_mapping
prefix: /httpbin/
service: httpbin.org:80
host_rewrite: httpbin.org
---
apiVersion: ambassador/v1
kind: Module
name: tls
config:
server:
enabled: True
redirect_cleartext_from: 8080
secret: istio.default
---
apiVersion: ambassador/v1
kind: TLSContext
name: istio-upstream
cert_chain_file: /etc/istiocerts/cert-chain.pem
private_key_file: /etc/istiocerts/key.pem
cacert_chain_file: /etc/istiocerts/root-cert.pem
spec:
type: LoadBalancer
ports:

• name: ambassador
  port: 80
  targetPort: 8080
• name: ambassador-https
  port: 443
  targetPort: 80
  selector:
  service: ambassador
  `

I am getting configuration error in ambassador dashboard:

CONFIGURATION ERRORS

ambassador.default.2: TLSContext server found no certificate in secret istio in namespace default, ignoring...

[ovk]

As I mentioned, just creating secret without dot in the name (e.g. istio-default) with contents identical to istio.default, and referencing it in your ambassador deployment volume should help.

[diptimanadak]

I forgot to mention I was using ambassador version 0.73.0, which does the behave similarly like 0.50.1. Anyway reverting back to 0.50.1 and steps you have mentioned worked for me. There are no errors coming in ambassador logs. But none of the deployed services I am able to access. I am using EKS to deploy the services along with istio and ambassador. Can you guys point out the probable configuration mistakes:

```---
apiVersion: v1
kind: Service
metadata:
name: customer-portal
annotations:
getambassador.io/config: |
---
apiVersion: ambassador/v1
kind: Mapping
name: customer-portal_mapping
prefix: /customer-portal/
rewrite: /customer-portal
service: https://customer-portal:8080
tls: istio-upstream
spec:
ports:

• name: customer-portal
  port: 8080
  targetPort: 8080
  protocol: TCP
  selector:
  app: customer-portal```

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[ovk]

I believe the issue is still there.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[richarddli]

2277 is merged, this will be in the next Ambassador release, out shortly. Thanks @ppeble !