Ambassador: Support global level http -> https redirection

[yogeshsajanikar]

Use Case
If TLS terminates externally (Such as AWS ELB), the http request need to be redirected to https.

Describe the solution you'd like
The solution that I can think of is twofold. One, to be able to identify and match the protocol. (http/https), and second to be able to add permenant redirection for a URL. Both have their own use cases to be able to be used separately.

A config for permanent redirection may look like

prefix: /
port: 80
ssl: off
redirect: https://<hostname>
redirection_type: permanent

Describe alternatives you've considered
Another alternative is to use something like cloudflare to do the redirection.

Additional context
Currently we ended up using AWS cloud front for doing redirection. However, cloudfront is not able to handle websocket. Hence we may have to move to nginx-ingress to be able to do this.

[kflynn]

@yogeshsajanikar Have you looked at the redirect_cleartext_from option in the tls Module? It's described in the TLS Termination guide.

[cesartl]

I've got the same requirement. I terminate SSL with an AWS ELB.

I'm currently using NGINX for the redirect and I'm looking at migrating to ambassador.

AWS provides special http_x_forwarded_* headers for that. For example http_x_forwarded_proto can be https or http.

So with NGINX I had a config like that:

    if ($http_x_forwarded_proto != 'https') {
       return 301 https://$host$request_uri;
    }

I would be good to have something similar with Ambassador.

Perhaps this would require a config parameter to choose which header contains the protocol:

prefix: /
port: 80
ssl: off
redirect: https://<hostname>
redirection_type: permanent
protocol_header: http_x_forwarded_proto

[concaf]

We could leverage envoy's ability to do this - https://www.envoyproxy.io/docs/envoy/latest/configuration/http_conn_man/headers#x-forwarded-proto - will dig deeper.

[plombardi89]

Heads up this is actually a trickier problem than it seems. Let me explain (lots of words and context coming in):

First, for those unfamiliar with AWS there are three types of load balancers:

1. "Classic" Load Balancer (abbreviated ELB or CLB, sometimes referred to as ELBv1 or Elastic Load Balancer)

   • Supports L4 (TCP, TCP+SSL) and L7 load balancing (HTTP 1.1, HTTPS 1.1).
   • Notably does not support websockets unless running in L4 mode.
   • Notable does not support HTTP 2 (which is required for GRPC) unless running in L4 mode.
   • Can perform SSL/TLS offload

2. Application Load Balancer (abbreivated ALB, sometimes referred to as ELBv2)

   • Supports L7 only
   • Supports websockets
   • Supports a broken implementation of HTTP2 (trailers are not supported and these are needed for GRPC)
   • Can perform SSL/TLS offload

3. Network Load Balancer (abbreviated NLB).

   • Supports L4 only
   • Cannot perform SSL/TLS offload.

In Kubernetes land when using the AWS integration and a v1.Service -> type: LoadBalancer the only types of infrastructure load balancers you can create are:

1. Classic Load Balancer
2. Network Load Balancer (Kubernetes >= 1.9 + you supply a special annotation on the manifest)

Further in the case of a Classic Load Balancer it can be configured to run in either L4 or L7 mode but not both from Kubernetes, this is an important piece of information. Ambassador is designed to be a generic API gateway. That means most likely folks want to deploy Ambassador in a way that developers can use whatever protocol they desire on the backend (HTTP, websockets, GRPC, etc.).

Further, operators usually love to use SSL/TLS offload when on AWS. In AWS you can setup load balancers to use the Amazon Certificate Manager ("ACM") which handles all the security concerns around managing SSL/TLS certificates and makes setting up SSL/TLS for a load balancer as trivial as a one line configuration reference to a named certificate in the ACM system.

So to restate some assumptions / facts about when on AWS:

• Users usually setup Ambassador with the intent to handle all types of backend traffic (HTTP, websocket, GRPC, etc.)
• Users usually want to perform SSL/TLS offload at the ELB.
• Therefore when on AWS the preferred way to run an ELB is in L4 mode (TCP+SSL)

A problem emerges, The X-Forwarded-Proto cannot be added to incoming HTTP requests by the ELB when running TCP+SSL mode because it does not know what protocol it is speaking (it is afterall just a dumb L4 load balancer).

So what to do? There are a couple solutions and they pose UX challenges for us:

• If a user runs the load balancer in L7 mode (HTTPS) then we can do something with X-Forwarded-Proto (we need to run a test). This obviously comes with the drawback that they cannot use websockets or GRPC through the load balancer.

• If a user runs the load balancer in L4 mode (TCP+SSL) then we have a signifigantly different problem. There is no protocol information that the ELB can provide to Envoy va X-Forwarded-Proto. The only option in this case is:

  1. Run the ELB with the following listener configuration:
     
     
     
     • :443 -> :8443 (the envoy port doesn't matter)
     
     
     • :80 -> :8080 (the envoy port doesn't matter)
     
     
  2. When traffic is received on :8080 we need to issue the HTTP 301. We are making an implicit assumption that this is HTTP (insecure). This is the only way to distinguish traffic absent X-Forwarded-Proto that I can think of.

How we want to expose this via Ambassador is the tricky bit.

[richarddli]

@cesartl @yogeshsajanikar If you have any feedback on how you would like to do the above, that would be great!

[cesartl]

@plombardi89 that's interesting thank you. I can appreciate the complexity of having a solution that should work with all possible deployment architectures.

Perhaps this tells us that it would be better not to try to make a solution that works in all cases. You could provide a way to redirect if a given header (e.g. X-Forwarded-Proto) is equals to a given value (e.g. http), which would only work when such header exists (e.g. with L7 ELB).

If there are no such header then as you said there is nothing more you can do. In this case you can provide the workaround you suggested above.

In the documentation, you currently have a section about terminating TLS at the Ambassador level. Perhaps it would be good to have another section about terminating TLS at the Cloud Provider level (e.g. ELB). You could use this section to explain the new redirect with header option and the workaround if no such header exists.

[yogeshsajanikar]

@plombardi89 @richarddli I'd go with second option to go with L4 with a permanent redirection. This should solve the problem and would allow using websockets too. (One of the main reason for this request).

I totally side with @cesartl and appreciate @plombardi89 to have taken efforts to explain the complex solution. Perhaps with NLB the solution will be much better.

[sbrichardson]

Is this similar to https://github.com/datawire/ambassador/issues/114?

[cesartl]

Yes it looks similar.
It is marked fixed but I couldn't find documentation for the simple-tls-redirect config.

[concaf]

Yes, #114 was closed by #267 which implemented redirect_cleartext_from, but not this use case.

[concaf]

After digging through piles of :hankey:, I think this is going to help me with implementing something https://github.com/envoyproxy/envoy/issues/2638#issuecomment-367185628 just putting it here for reference.

[concaf]

There's a possible envoy v1 limitation that I might be hitting with the L4 redirection -

I intended to give user an interface like -

https_redirect:
  port: 9999
  to: https://example.com

What this would do is,

• start a new envoy listener at port 9999
• issue a redirect to all requests to https://example.com

I got this working with v2 using https://www.envoyproxy.io/docs/envoy/v1.6.0/api-v2/api/v2/route/route.proto.html#envoy-api-field-route-redirectaction-https-redirect, but I'm unable to find such a field for envoy v1 which we can use.

The reason I do not want to use require_tls in this case with envoy, is that it will redirect the same host to https, which might not work when the standard ports for HTTP and HTTPS are not 80 and 443.
e.g.
HTTP incoming at example.com:8080
HTTPS incoming at example.com:8443
Now if we use require_tls (or require_ssl to be precise), then the HTTP requests at example.com:8080 are routed to https://example.com:8080, but instead we want https://example.com:8443

But, for a stopgap solution, we might have to move forward with require_tls for now.

The UX would be -

https_redirect_from: 9999

[concaf]

But now, that looks wildly similar to redirect_cleartext_from, which also starts a new listener and redirects all HTTP traffic to HTTPS.
@plombardi89 WDYT about this? :thinking:

[kflynn]

It may be that redirect_cleartext_from is all we have here until we get to V2. ☹️

We need, at minimum, to document that redirect_cleartext_from is unlikely to work if you're not using 80/443, and it sounds like we may need a way to explicitly tell Ambassador _not_ to try to load TLS certs.

[concaf]

@kflynn If we get #594 in, then we can use redirect_cleartext_from for L4 redirection.