Contour: Add upstream support for HTTP/1.1 + TLS annotation to the service

[davecheney]

We need a way to explicitly configure the L7 protocol for a service port to HTTP/1.1 + TLS. Something like

contour.heptio.com/upstream-protocol.https: "admin, 443"

---

Update, I think this annotation should be

contour.heptio.com/upstream-protocol.tls: "admin, 443"

[Frusty]

Is there a (non explicit) way of using an HTTPS upstream service today?

[davecheney]

@Frusty not at the moment, but the scaffolding to add it exists now.

[colemickens]

Will this include the ability to specify a CA bundle for the upstream service? nginx-ingress has this today via an annotation, I'm also curious if this might be supported as a first-rate field in the new CRDs instead of just via annotations?

[colemickens]

Also, it looks like this was already at least partially implemented, but then abandoned here: https://github.com/heptio/contour/pull/325

[davecheney]

I think this should be a property of the service document, not the ingress
or ingressroute document.

On 3 July 2018 at 09:24, Cole Mickens [e-mail] wrote:

Will this include the ability to specify a CA bundle for the upstream
service? nginx-ingress has this today via an annotation, I'm also curious
if this might be supported as a first-rate field in the new CRDs instead of
just via annotations?

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/heptio/contour/issues/406#issuecomment-401966744, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAAcA7a7uEbGQrPrio5H82bjsu_qRHjtks5uCquXgaJpZM4UJijJ
.

[rosskukulinski]

Non-essential for closing out remaining IngressRoute work. Moving to 0.7.0 for further analysis.

[jdelgadillo]

Any updates on when this may get implemented? I have a service that can successfully use HTTP2 over HTTPS, but cannot get WebSockets to work via that route.

I have not been able to figure out how make contour use HTTP1 over https/443 to make a connection to the backend so I can get end-to-end encryption. TLS pass through is not an option as we need one IngressRoute for all exposed services.

[jdelgadillo]

As an example, I've got this application (http1.1 only) https://github.com/jdelgadillo/contour-sample/blob/master/kubernetes/playapp.yaml deployed here for WebSockets wss://contour-test-public.westus2.cloudapp.azure.com/ws and Http/html https://contour-test-public.westus2.cloudapp.azure.com/

I'd like to change the IngressRoute to use port 443 instead of 80 when talking to the backend. But I haven't figured out the magic annotation/configuration to make it work with this deployment.

Any ideas?

[stevesloka]

@davecheney this probably needs to be another annotation on the service like you described initially:

contour.heptio.com/upstream-protocol.https: "admin, 443"

If Contour sees this on the service, then we need to add the following to the cluster which tells Envoy that the upstream service should be over tls:

TlsContext: &auth.UpstreamTlsContext{
        CommonTlsContext: &auth.CommonTlsContext{},
    }

The only potential issue is that once this is set, another ingress resource in the same namespace couldn't sent traffic over non-tls which I think is probably ok.

I have this working in a small PoC, happy to take this on if the approach seems reasonable.

[davecheney]

@stevesloka sgtm. I suggest this annotation

contour.heptio.com/upstream-protocol.tls: "admin, 443"

I know it's splitting hairs, but I'm sure there will be cases where we have to do TLS between envoy and the backend which don't originate from a browser.

There's probably a lot of infrastructure we can leverage

https://github.com/heptio/contour/blob/master/internal/dag/builder.go#L258

Sets the Protocol field on dag.HTTPService, which is read here

https://github.com/heptio/contour/blob/master/internal/envoy/cluster.go#L47

[davecheney]

Hello,

If you're watching this issue, this feature is available in the :master image now and can be tested now. To test this add the following annotation to your _service_ (this is a property of the k8s service, not the ingress/ingressroute record):

contour.heptio.com/upstream-protocol.tls: "admin, 443"

Where admin and/or 443 are the names or port numbers of the port on your service document that speak TLS. eg.

apiVersion: v1
kind: Service
metadata:
  name: kuard
  namespace: default
  annotations:
    contour.heptio.com/upstream-protocol.tls: admin, 443
spec:
  ports:
  - name: admin
    port: 443
    protocol: TCP
    targetPort: 8080
  selector:
    app: kuard

[jdelgadillo]

@davecheney
Thanks for getting this implemented. I just verified it works with tls backends with prefixRewrite and WebSockets as well.

Here are my Service and IngressRoute.

wss://contour-test-public.westus2.cloudapp.azure.com/prefix/ws
https://contour-test-public.westus2.cloudapp.azure.com/prefix/

👍

apiVersion: v1
kind: Service
metadata:
  name: playapp-service
  labels:
    app: playapp
  annotations:
    contour.heptio.com/upstream-protocol.tls: "https, 443"
spec:
  ports:
  - name: https
    port: 443
    targetPort: 9443
  selector:
    app: playapp
---
apiVersion: contour.heptio.com/v1beta1
kind: IngressRoute
metadata:
  name: playapp-ingress-route
  namespace: default
spec:
  virtualhost:
    fqdn: contour-test-public.westus2.cloudapp.azure.com
    tls:
      secretName: playapp-cert
  routes:
    - match: /prefix/
      prefixRewrite: "/"
      enableWebsockets: true
      permitInsecure: true
      services:
        - name: playapp-service
          port: 443

[davecheney]

Thanks, but all glory goes to @stevesloka for doing the ba and development work on this issue

On 6 Apr 2019, at 03:15, Javier Delgadillo [e-mail] wrote:

@davecheney
Thanks for getting this implemented. I just verified it works with tls backends with prefixRewrite and WebSockets as well.

Here are my Service and IngressRoute.

wss://contour-test-public.westus2.cloudapp.azure.com/prefix/ws
https://contour-test-public.westus2.cloudapp.azure.com/prefix/

👍

---

apiVersion: v1
kind: Service
metadata:
name: playapp-service
labels:
app: playapp
annotations:
contour.heptio.com/upstream-protocol.tls: "https, 443"
spec:
ports:

• name: https
  port: 443
  targetPort: 9443
  selector:

app: playapp

apiVersion: contour.heptio.com/v1beta1
kind: IngressRoute
metadata:
name: playapp-ingress-route
namespace: default
spec:
virtualhost:
fqdn: contour-test-public.westus2.cloudapp.azure.com
tls:
secretName: playapp-cert
routes:
- match: /prefix/
prefixRewrite: "/"
enableWebsockets: true
permitInsecure: true
services:
- name: playapp-service
port: 443
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[ryanelian]

I thought one of the point of doing custom CRD (HTTPProxy / IngressRoute) is to prevent splashing annotations all over the place?

Why can't the protocol just be defined in the services object?

For example:

apiVersion: projectcontour.io/v1
kind: HTTPProxy
metadata:
  name: kubernetes-dashboard-proxy
  namespace: kubernetes-dashboard
spec:
  virtualhost:
    fqdn: dashboard.kubernetes.com
    tls:
      secretName: kubernetes-dashboard-tls
  routes:
    - conditions:
      - prefix: /
      services:
        - name: kubernetes-dashboard
          protocol: https
          port: 443

[davecheney]

@ryanelian thank you for your comment. Would you please raise a new issue so it is not lost.

For the record I agree with @stevesloka that while the service annotation is the correct place for this information (when I added it I argued that the protocol the service speaks is a property of the service, not who is talking too it) but this has serious usability limitations, so I'm open to adding this field on route.services.service