Ambassador: Unable to access applications using Ambassador with Istio auth

[jkattmann]

I have a v1.9.2 kubernetes deployment with istio-0.6.0 (installed using kubectl apply -f install/kubernetes/istio-auth.yaml). This environment work using both the sample bookinfo application and my own basic REST application via the default Istio ingress however when I attempt to deploy Ambassador I get the following accessing either application via the Ambassador loadbalancer's public IP:

upstream connect error or disconnect/reset before headers

Connecting to an Ambassador pod and curling the application shows:

/application # curl -v sample-rest.default.svc.cluster.local/api/healthcheck
*   Trying 10.0.207.17...
* TCP_NODELAY set
* Connected to sample-rest.default.svc.cluster.local (10.0.207.17) port 80 (#0)
> GET /api/healthcheck HTTP/1.1
> Host: sample-rest.default.svc.cluster.local
> User-Agent: curl/7.59.0
> Accept: */*
>
* Recv failure: Connection reset by peer
* Closing connection 0
curl: (56) Recv failure: Connection reset by peer

The sample httpbin redirect does work:

StatusCode        : 200
StatusDescription : OK
Content           : {
                      "origin": "52.191.208.XX"
                    }

RawContent        : HTTP/1.1 200 OK
                    access-control-allow-origin: *
                    access-control-allow-credentials: true
                    x-processed-time: 0
                    x-envoy-upstream-service-time: 10
                    Content-Length: 33
                    Content-Type: application/json
                    Dat...

If I remove istio-auth and install istio without mutual TLS (kubectl apply -f install/kubernetes/istio.yaml) then both the sample bookinfo application and my own basic REST application are accessible via the ambassador loadbalancer's public IP.

ambassador logs show:

λ  kubectl logs ambassador-b5cd67c5d-xq247 ambassador
ACCESS [2018-04-30T18:13:42.380Z] "GET /api/healthcheck HTTP/1.1" 503 UC 0 57 1 - "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36" "95ccdc6d-d868-496d-86af-26cc16930988" "52.191.214.94" "10.0.207.17:80"

I have tried altering the mapping to the service in many ways without any success.

Relevant yml files attached: yml.zip

[MarcErdmann]

I get a similar error. My browser tells me the connection was reset and curl states "empty response". Istio is working fine with the bookinfo sample application. I simply followed the tutorial for setting up ambassador with istio and tested it with mutual TLS and without.

[MarcErdmann]

I made progress on this by removing: targetPort: 8080 in the ambassador-service.yaml and remove httpbin from the annotation.
By applying httpbin as a service httpbin now works, but I get this error for the bookinfo productpage:
upstream connect error or disconnect/reset before headers

[kflynn]

@MarcErdmann, was that targetPort setting in YAML from Datawire or something else?

[geekidentity]

I get a similar error too, nobody solve it?

[MarcErdmann]

I have solved all the problems I had. This is some time ago and I can not remember what exactly solved this, but these are my notes from then:

Install Ambassador by running kubectl apply -f ambassador/ambassador-no-rbac-tls.yaml and after all pods are running kubectl apply -f ambassador/ambassador-service-istio.yaml
- if on single server, make sure to delete istio ingress first. Otherwise Ambassador cannot be scheduled due to ports in use
- if some deployments fail to start pods, restart istio-sidecar-injector
- for Ambassador 0.34.3 ports are 80 and 443 and the respective targetPorts are 80 and 443 NOT 8080
- When you have connection reset issues you may have to disable redirect_cleartext_from: 80 in ambassador/ambassador-service-istio.yaml to allow non encrypted traffic (http) - in general make sure you have created the ambassador-certs secret by running sudo certbot certonly --manual --preferred-challenges dns and sudo kubectl create secret tls ambassador-certs --cert=/etc/letsencrypt/live/website.com/fullchain.pem --key=/etc/letsencrypt/live/website.com/privkey.pem. You maybe have to restart ambassador service
- If you get no healthy upstream make sure to set the service in the annotation explicitly like website.namespace.svc.cluster.local instead simply website

ambassador-no-rbac-tls.yaml

---
apiVersion: v1
kind: Service
metadata:
  labels:
    service: ambassador-admin
  name: ambassador-admin
spec:
  type: NodePort
  ports:
  - name: ambassador-admin
    port: 8877
    targetPort: 8877
  selector:
    service: ambassador
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: ambassador
spec:
  replicas: 3
  template:
    metadata:
      annotations:
        sidecar.istio.io/inject: "false"
      labels:
        service: ambassador
    spec:
      containers:
      - name: ambassador
        image: quay.io/datawire/ambassador:0.35.1
        resources:
          limits:
            cpu: 1
            memory: 400Mi
          requests:
            cpu: 200m
            memory: 100Mi
        env:
        - name: AMBASSADOR_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace          
        livenessProbe:
          httpGet:
            path: /ambassador/v0/check_alive
            port: 8877
          initialDelaySeconds: 30
          periodSeconds: 3
        readinessProbe:
          httpGet:
            path: /ambassador/v0/check_ready
            port: 8877
          initialDelaySeconds: 30
          periodSeconds: 3
        volumeMounts:
        - mountPath: /etc/istiocerts/
          name: istio-certs
          readOnly: true
      - name: statsd
        image: quay.io/datawire/statsd:0.35.1
      restartPolicy: Always
      volumes:
      - name: istio-certs
        secret:
          optional: true
          secretName: istio.default

ambassador-service-istio.yaml

---
apiVersion: v1
kind: Service
metadata:
  labels:
    service: ambassador
  name: ambassador
  annotations:
    external-dns.alpha.kubernetes.io/hostname: brint.de,api.brint.de,dev.brint.de,dev.api.brint.de,www.brint.de #optional for use with kubernetes external dns
    external-dns.alpha.kubernetes.io/ttl: "120" #optional
    getambassador.io/config: |
      ---
      apiVersion: ambassador/v0
      kind:  Module
      name: tls
      config:
        server:
          enabled: True
          redirect_cleartext_from: 80
        client:
          enabled: False
        upstream:
          cert_chain_file: /etc/istiocerts/cert-chain.pem
          private_key_file: /etc/istiocerts/key.pem
spec:
  type: LoadBalancer
  ports:
   - name: ambassador
     port: 80
     targetPort: 80
   - name: ambassador-tls
     port: 443
     targetPort: 443
  selector:
    service: ambassador

[User1m]

@MarcErdmann helped me solve my "no healthy upstream" issue. Thanks!

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.