Ambassador: x-forwad-proto header is not forwarded correctly to authentication service

[cornelius-keller]

Describe the bug
We use datawire ambassador behind nginx as an ingress controller. Since Nginx does the tls termination there is already an x-forwad-proto header set when the request hits ambassador.
If the request is now forwarded to our authentication service the value of this header changes from https to http.

To Reproduce

Deploy an Authentication Service like described here https://www.getambassador.io/user-guide/auth-tutorial/#1-deploy-the-authentication-service but with ambassador behind nginx as ingress controller doing also tls termination.

Expected behavior
x-forward-proto should not be changed and forwarded with the correct value https to the authentication service.

Versions

• Ambassador: 0.50.rc
• Kubernetes environment bare metal
• Version 1.13.0

[richarddli]

Thanks! Tagging for 0.50 GA. Out of curiosity, why are you running NGINX ingress controller as well as Ambassador?

[cornelius-keller]

Hi Richard,

we use https://github.com/kubernetes-incubator/external-dns for dns automation and https://github.com/jetstack/kube-lego for automating let's encrypt certificates. This are two very important components for us as this saves us a lot of manual effort and makes it very convenient to for developers to bring their services online. I did this one year ago and back then there was no easy way to do it with ambassador. I don't know if this changed today.

[kflynn]

0.50.0-rc3 correctly defaults to allowing the external auth service to see X-Forwarded-Proto. Tests are added in #1080.

[mthird]

From looking at this thread, it appears that only the authentication gets the "correct" header (https), but upstream services still see http.

In our case, we use fly.io's wormhole to do edge ssl termination which forwards requests to port 80 within ambassador. We would except the x-forwarded-proto to respect the original proto header, but it is changing it to http.

The headers we received in 0.40.2 looked like:

"x-forwarded-for": "4.xx.xx.xx",
"x-forwarded-host": "host.example.com",
"x-forwarded-port": "443",
"x-forwarded-proto": "https",
"x-forwarded-ssl": "on",
"x-forwarded-uri": "/",

However, the headers in all of the 0.50.0-* releases look like (this is 0.50.0-rc5):

"x-forwarded-for": "4.xx.xx.xx,10.xx.xx.xx",
"x-forwarded-host": "host.example.com",
"x-forwarded-port": "443",
"x-forwarded-proto": "http",
"x-forwarded-ssl": "on",
"x-forwarded-uri": "/"

Is this by design?

[kflynn]

@gsagula, can you speak to this?

[gsagula]

This is most likely to be caused by:
https://www.getambassador.io/reference/modules/#use_remote_address

When use_remote_address is true, Envoy connection manager will set X-Forwarded-Header based on the client's connection.

https://github.com/envoyproxy/envoy/blob/c8265348b541a9909be009040b44ba6c2e797e8e/source/common/http/conn_manager_utility.cc#L110

Looking further at the same code, it seems that Envoy will not override the XFP header if it already exists and use_remote_address is set to false:

https://github.com/envoyproxy/envoy/blob/c8265348b541a9909be009040b44ba6c2e797e8e/source/common/http/conn_manager_utility.cc#L123

@mthird @cornelius-keller What does your configuration look like? Is use_remote_address set to true? Can you try to set it to false and see what happens, please?

[mthird]

We are using the default ambassador configuration without explicitly setting that property. I set it to "true" on 0.40.2 and saw the same, unexpected behavior.

What are the downsides to running 0.50.0 with that flag off?

[mthird]

I can confirm disabling it on 0.50.0 behaves the same as the default with 0.40.2

[gsagula]

@mthird Having the flag set to false won't append all the IPs to XFF. I believe you have it your example above:
0.40.2

"x-forwarded-for": "4.xx.xx.xx"

0.50.x

"x-forwarded-for": "4.xx.xx.xx,10.xx.xx.xx"

The other change in behaviour, I believe I explained here: https://github.com/datawire/ambassador/issues/1053#issuecomment-457269553

Just to confirm. When you say the same behaviour, it means that the authorization server is seeing
x-forwarded-for: https and upstream is seeing x-forwarded-for: http. Is that correct?

[mthird]

With use_remote_address: false, I see that x-forwarded-for: https on both 0.40.2 and 0.50.0 and with use_remote_address: true, I see x-forwarded-for: http on both. My concern would be that the other x-forwarded-* headers might become unbalanced with that flag off.

[gsagula]

With use_remote_address: false, I see that x-forwarded-for: https on both 0.40.2 and 0.50.0 and with use_remote_address: true, I see x-forwarded-for: http on both. My concern would be that the other x-forwarded-* headers might become unbalanced with that flag off.

Are you seeing these headers in authorization service or in the upstream?

[mthird]

We don't use the authorization service. This is in the upstream.

[gsagula]

I would start looking at all X-Forward- headers which the rest of your system depends on and then checking Envoy docs to see how use_remote_address will affect them.

As shown in the above https://github.com/datawire/ambassador/issues/1053#issuecomment-459847872, We already know how x-forwarded-for will be affected.

[gsagula]

@mthird Let me know if you can't find all the information in the docs. I can look at Envoy code for you.

[kflynn]

I'm going to go ahead and close this -- feel free to contact us here or on Slack if need be!