Spring-cloud-gateway: Host header breaking change M2 -> M3

[maximusfloydus]

I created a simple Spring Cloud Gateway app with one route. My tests worked fine with Horton.M2 (2.2.0.M2) and Greenwich.SR3. When I moved to Horton.M3 (2.2.0.M3), my requests started returning 404 because k8s didn't know where to route the request because the host header didn't match "api.example.org". The request was routed to the correct uri, but the host header was not set to the uri using M3. Below are the scrubbed outputs of the httpclient trace.

These headers are being sent from the Gateway reactor httpclient (used wiretap) after the request route was matched.
My request via postman: http://localhost:8080/meta/version
This should be routed to https://api.example.org/meta/version with host header of api.example.org

M2: {{host: api.example.org}}

GET /meta/version HTTP/1.1..User-Agent: PostmanRuntime/7.17.1..Accept: */*..Cache-Control: no-cache..Accept-Encoding: gzip, deflate..Forwarded: proto=http;host="localhost:8080";for="0:0:0:0:0:0:0:1:62856"..X-Forwarded-For: 0:0:0:0:0:0:0:1..X-Forwarded-Proto: http..X-Forwarded-Port: 8080..X-Forwarded-Host: localhost:8080..host: api.example.org..content-length: 0....   

M3: {{Host: localhost:8080}}

GET /meta/version HTTP/1.1..User-Agent: PostmanRuntime/7.17.1..Accept: */*..Cache-Control: no-cache..Host: localhost:8080..Accept-Encoding: gzip, deflate..Forwarded: proto=http;host="localhost:8080";for="0:0:0:0:0:0:0:1:62900"..X-Forwarded-For: 0:0:0:0:0:0:0:1..X-Forwarded-Proto: http..X-Forwarded-Port: 8080..X-Forwarded-Host: localhost:8080..content-length: 0.... 

Example Configuration

spring:
  cloud:
    gateway:
      routes:
        - id: catch_all
          uri: https://api.example.org
          predicates:
            - After=2019-04-20T17:42:47.789-06:00[America/Chicago]

pom.txt

[spencergibb]

I'm unsure what headers those are? Is it sent to the gateway or from the gateway?

[maximusfloydus]

These headers are being sent from the Gateway reactor httpclient (used wiretap) after the request route was matched. It looks like the M3 code is not setting the uri field on the host header.

[spencergibb]

Gateway doesn't do anything to the host header by default. There's a preserve host header option you can find in the docs.

[SebRamsland]

I am experiencing the very same issue in an attempt to upgrade from Spring Cloud Gateway 2.1.3.RELEASE to 2.2.0.M3. As mentioned, 2.2.0.M2 does indeed work.

An important note is that adding the GatewayFilterSpec "preserveHostHeader" to my routes did _not_ solve the issue on 2.2.0.M3. Neither did adding the same filter to the spring.cloud.gateway.default-filters property.

[cboitel]

Hitting the same issue when using a simple demo gateway using Spring Initializer:

• With 2.1.9, Host header in outgoing request gets rewritten to target uri (unless PreserveHostFilter is used). See log extract below with host headers appearing at the end of request:
  ```2019-10-21 12:21:21.882 DEBUG 10560 --- [reactor-http-nio-2] reactor.netty.proxy : [id: 0xa7fb107b, L:/127.0.0.1:54213 - R:/127.0.0.1:9000] WRITE: 350B
  +-------------------------------------------------+
  | 0 1 2 3 4 5 6 7 8 9 a b c d e f |
  +--------+-------------------------------------------------+----------------+
  |00000000| 50 4f 53 54 20 2f 70 6f 73 74 20 48 54 54 50 2f |POST /post HTTP/|
  |00000010| 31 2e 31 0d 0a 55 73 65 72 2d 41 67 65 6e 74 3a |1.1..User-Agent:|
  |00000020| 20 63 75 72 6c 2f 37 2e 36 36 2e 30 0d 0a 41 63 | curl/7.66.0..Ac|
  |00000030| 63 65 70 74 3a 20 2a 2f 2a 0d 0a 43 6f 6e 74 65 |cept: /..Conte|
  |00000040| 6e 74 2d 54 79 70 65 3a 20 61 70 70 6c 69 63 61 |nt-Type: applica|
  |00000050| 74 69 6f 6e 2f 6a 73 6f 6e 0d 0a 43 6f 6e 74 65 |tion/json..Conte|
  |00000060| 6e 74 2d 4c 65 6e 67 74 68 3a 20 36 37 0d 0a 48 |nt-Length: 67..H|
  |00000070| 65 6c 6c 6f 3a 20 57 6f 72 6c 64 0d 0a 46 6f 72 |ello: World..For|
  |00000080| 77 61 72 64 65 64 3a 20 70 72 6f 74 6f 3d 68 74 |warded: proto=ht|
  |00000090| 74 70 3b 68 6f 73 74 3d 22 31 32 37 2e 30 2e 30 |tp;host="127.0.0|
  |000000a0| 2e 31 3a 38 30 37 39 22 3b 66 6f 72 3d 22 31 32 |.1:8079";for="12|
  |000000b0| 37 2e 30 2e 30 2e 31 3a 35 34 32 31 32 22 0d 0a |7.0.0.1:54212"..|
  |000000c0| 58 2d 46 6f 72 77 61 72 64 65 64 2d 46 6f 72 3a |X-Forwarded-For:|
  |000000d0| 20 31 32 37 2e 30 2e 30 2e 31 0d 0a 58 2d 46 6f | 127.0.0.1..X-Fo|
  |000000e0| 72 77 61 72 64 65 64 2d 50 72 6f 74 6f 3a 20 68 |rwarded-Proto: h|
  |000000f0| 74 74 70 0d 0a 58 2d 46 6f 72 77 61 72 64 65 64 |ttp..X-Forwarded|
  |00000100| 2d 50 72 65 66 69 78 3a 20 2f 66 6f 6f 0d 0a 58 |-Prefix: /foo..X|
  |00000110| 2d 46 6f 72 77 61 72 64 65 64 2d 50 6f 72 74 3a |-Forwarded-Port:|
  |00000120| 20 38 30 37 39 0d 0a 58 2d 46 6f 72 77 61 72 64 | 8079..X-Forward|
  |00000130| 65 64 2d 48 6f 73 74 3a 20 31 32 37 2e 30 2e 30 |ed-Host: 127.0.0|
  |00000140| 2e 31 3a 38 30 37 39 0d 0a 68 6f 73 74 3a 20 68 |.1:8079..host: h|
  |00000150| 74 74 70 62 69 6e 2e 6f 72 67 0d 0a 0d 0a |ttpbin.org.... |
  +--------+-------------------------------------------------+----------------+

- Starting with 2.2.0, Host header seems to be kept unchanged from original request (host header appears at start of request not end)

2019-10-21 12:21:52.773 DEBUG 5000 --- [reactor-http-nio-3] reactor.netty.http.client.HttpClient : [id: 0xda74da40, L:/127.0.0.1:54247 - R:/127.0.0.1:9000] WRITE: 353B
+-------------------------------------------------+
| 0 1 2 3 4 5 6 7 8 9 a b c d e f |
+--------+-------------------------------------------------+----------------+
|00000000| 50 4f 53 54 20 2f 70 6f 73 74 20 48 54 54 50 2f |POST /post HTTP/|
|00000010| 31 2e 31 0d 0a 48 6f 73 74 3a 20 31 32 37 2e 30 |1.1..Host: 127.0|
|00000020| 2e 30 2e 31 3a 38 30 37 39 0d 0a 55 73 65 72 2d |.0.1:8079..User-|
|00000030| 41 67 65 6e 74 3a 20 63 75 72 6c 2f 37 2e 36 36 |Agent: curl/7.66|
|00000040| 2e 30 0d 0a 41 63 63 65 70 74 3a 20 2a 2f 2a 0d |.0..Accept: /.|
|00000050| 0a 43 6f 6e 74 65 6e 74 2d 54 79 70 65 3a 20 61 |.Content-Type: a|
|00000060| 70 70 6c 69 63 61 74 69 6f 6e 2f 6a 73 6f 6e 0d |pplication/json.|
|00000070| 0a 43 6f 6e 74 65 6e 74 2d 4c 65 6e 67 74 68 3a |.Content-Length:|
|00000080| 20 36 37 0d 0a 48 65 6c 6c 6f 3a 20 57 6f 72 6c | 67..Hello: Worl|
|00000090| 64 0d 0a 46 6f 72 77 61 72 64 65 64 3a 20 70 72 |d..Forwarded: pr|
|000000a0| 6f 74 6f 3d 68 74 74 70 3b 68 6f 73 74 3d 22 31 |oto=http;host="1|
|000000b0| 32 37 2e 30 2e 30 2e 31 3a 38 30 37 39 22 3b 66 |27.0.0.1:8079";f|
|000000c0| 6f 72 3d 22 31 32 37 2e 30 2e 30 2e 31 3a 35 34 |or="127.0.0.1:54|
|000000d0| 32 34 36 22 0d 0a 58 2d 46 6f 72 77 61 72 64 65 |246"..X-Forwarde|
|000000e0| 64 2d 46 6f 72 3a 20 31 32 37 2e 30 2e 30 2e 31 |d-For: 127.0.0.1|
|000000f0| 0d 0a 58 2d 46 6f 72 77 61 72 64 65 64 2d 50 72 |..X-Forwarded-Pr|
|00000100| 6f 74 6f 3a 20 68 74 74 70 0d 0a 58 2d 46 6f 72 |oto: http..X-For|
|00000110| 77 61 72 64 65 64 2d 50 72 65 66 69 78 3a 20 2f |warded-Prefix: /|
|00000120| 66 6f 6f 0d 0a 58 2d 46 6f 72 77 61 72 64 65 64 |foo..X-Forwarded|
|00000130| 2d 50 6f 72 74 3a 20 38 30 37 39 0d 0a 58 2d 46 |-Port: 8079..X-F|
|00000140| 6f 72 77 61 72 64 65 64 2d 48 6f 73 74 3a 20 31 |orwarded-Host: 1|
|00000150| 32 37 2e 30 2e 30 2e 31 3a 38 30 37 39 0d 0a 0d |27.0.0.1:8079...|
|00000160| 0a |. |
+--------+-------------------------------------------------+----------------+

As a workaround for 2.2.0, i am adding AddRequestHeader filter to enforce Host header before request gets sent.

Example application.yml

spring:
cloud:
gateway:
routes:
- id: host_route
uri: http://httpbin.org/
predicates:
- Path=/foo/**
filters:
- AddRequestHeader=Host, httpbin.org

I have tracked down a recent change in NettyRoutingFilter that may be the cause. There is a slight change in the way proxied request headers are set: 
- before, request headers were enforced to httpHeaders inside send block
- after, httpHeaders are added to original request thus leading to keep original headers (even hop to hop ones).

Here is a suggested that i tested on my setup which fixes the issue by setting request headers to httpHeaders inside send block as previously.

--- NettyRoutingFilter.java.original 2019-10-21 14:28:05.631835000 +0200
+++ NettyRoutingFilter.java 2019-10-21 14:31:57.611835000 +0200
@@ -1,4 +1,4 @@
-a/*
+/*

• Copyright 2013-2019 the original author or authors.
  *
• Licensed under the Apache License, Version 2.0 (the "License");
  @@ -116,13 +116,12 @@
  boolean preserveHost = exchange
  .getAttributeOrDefault(PRESERVE_HOST_HEADER_ATTRIBUTE, false);

• Flux responseFlux = this.httpClient.headers(headers -> {
• headers.add(httpHeaders);
• Flux responseFlux = this.httpClient.request(method).uri(url).send((req, nettyOutbound) -> {
• req.headers( httpHeaders );
  if (preserveHost) {
  String host = request.getHeaders().getFirst(HttpHeaders.HOST);
• headers.add(HttpHeaders.HOST, host);
• req.header(HttpHeaders.HOST, host);
  }
• }).request(method).uri(url).send((req, nettyOutbound) -> {
  if (log.isTraceEnabled()) {
  nettyOutbound.withConnection(connection -> log.trace(
  "outbound route: " + connection.channel().id().asShortText()
  ```

Hope it helps

[csh0711]

I have the same problem when switching from Spring Boot _2.1.8 -> 2.2.0_ and _Greenwich.SR3 -> Hoxton.RC1_. Respectively switching from _spring-cloud-gateway-core-2.1.3.RELEASE_ to _spring-cloud-gateway-core-2.2.0.RC1_

The route my Gateway is deployed is: portal-gateway.dev.mycloud
The target route, where the request are routed to, is: api-target.dev.mycloud

When I log the requests I see this:

With spring-cloud-gateway-core-2.1.3.RELEASE:

GET /dev/resources/contracts HTTP/1.1
Accept: /
X-Cf-Applicationid: 4229a5b3-e4e0-4319-981f-6397c62da529
X-B3-Traceid: 1137a95b76ac599f
Accept-Encoding: gzip, deflate
X-B3-Spanid: 1137a95b76ac599f
X-Vcap-Request-Id: f8c3d847-c868-4b51-50da-b89c02d9d605
X-Request-Start: 1573059778945
User-Agent: PostmanRuntime/7.1.1
X-Forwarded-Proto: https,https
Authorization: Bearer ...
X-Cf-Instanceid: b31a6ece-cda7-4545-4f32-7708
Postman-Token: 1a8ee83f-ac4c-4da7-8ae1-f38e50d76930
X-Forwarded-For: 10.26.XXX.YYY
Content-Type: application/json
Cache-Control: no-cache
X-Cf-Instanceindex: 0
Forwarded: proto=https;host=portal-gateway.dev.mycloud;for="10.26.XXX.YYY:45518"
X-Forwarded-Port: 443
X-Forwarded-Host: portal-gateway.dev.mycloud
host: api-target.dev.mycloud
content-length: 0

With spring-cloud-gateway-core-2.2.0.RC1:

GET /dev/resources/contracts HTTP/1.1
Host: portal-gateway.dev.mycloud
User-Agent: PostmanRuntime/7.1.1
Accept: /
Accept-Encoding: gzip, deflate
Cache-Control: no-cache
Content-Type: application/json
Postman-Token: 0f38d9d6-6ae3-4c3e-9953-b5ca1edeb886
X-B3-Spanid: a41654fb395cbc1b
X-B3-Traceid: a41654fb395cbc1b
X-Cf-Applicationid: 4229a5b3-e4e0-4319-981f-6397c62da529
X-Cf-Instanceid: ad4ca9ac-e898-4bcd-448c-978b
X-Cf-Instanceindex: 0
X-Forwarded-For: 10.26.XXX.YYY
X-Forwarded-Proto: https,https
X-Request-Start: 1573059620528
X-Vcap-Request-Id: 2eda8eca-563e-4854-5d04-bdc1d3ca3bad
Authorization: Bearer ...
Forwarded: proto=https;host=portal-gateway.dev.mycloud;for="10.26.XXX.YYY:59006"
X-Forwarded-Port: 443
X-Forwarded-Host: portal-gateway.dev.mycloud
content-length: 0

It seems to me that the _HOST_ header is not rewritten like it was before. Therefore the target _api-target.dev.mycloud_ returns a 404, as it does not accept the header.

Btw.:
I use the DSL-based approach to configure my routes - written in Kotlin:

route(id = "contracts") {
path("/contracts/**")
uri("https://api-target.dev.mycloud")
filters {
rewritePath(pathRewriteSource, pathRewriteTarget)
}
}

EDIT: The breaking change is - as mentioned in the initial posting - between Hoxton.M2 and Hoxton.M3 respectively Spring Cloud Gateway 2.2.0.M2 and 2.2.0.M3
(not as I mentioned earlier between 2.1.3.RELEASE and 2.2.0.RC1)

[maximusfloydus]

I am still having this issue with Hoxton.RC2. This seems like a major defect for many uses cases.

[yangbongsoo]

Hello. I have a question about this issue.

We can solve this problem using below code. isn't it?

Consumer<HttpHeaders> headersConsumer = (httpHeaders) -> {
httpHeaders.set(HOST, ##destination Address Domain##);
};
mutatedRequest = request.mutate().headers(headersConsumer).build();
}
exchange.getAttributes().put(PRESERVE_HOST_HEADER_ATTRIBUTE, true);
return chain.filter(exchange.mutate().request(mutatedRequest).build());

Since preserveHost is true, NettyRoutingFilter will pass it on to the header value changed above.

// NettyRoutingFilter class 
... 

if (preserveHost) {
String host = request.getHeaders().getFirst(HttpHeaders.HOST);
req.header(HttpHeaders.HOST, host);
}

... 

You may think that the Host header should be set to the destination address domain automatically, but sometimes it is not.

in my case, when destination address is ip, we set the host header to gateway server domain.

[spencergibb]

Closing in favor of #1412

[ruoat]

I think we found this same issue using preserveHostHeader

In Greenwich.RELEASE the gateway called the upstream using

GET /api/foo HTTP/1.1
accept-language: en,fi;q=0.9,en-US;q=0.8
accept-encoding: gzip, deflate, br
referer: http://localhost:8081/
sec-fetch-mode: cors
sec-fetch-site: same-origin
accept: */*
content-type: application/json
Host: localhost:8081
content-length: 0

After we tried to update to Hoxton.RELEASE the request is

GET /api/foo HTTP/1.1
accept-language: en,fi;q=0.9,en-US;q=0.8
accept-encoding: gzip, deflate, br
referer: http://localhost:8081/
sec-fetch-mode: cors
sec-fetch-site: same-origin
accept: */*
content-type: application/json
host: localhost:8081
Host: localhost:8081
content-length: 0

Because there are two host-headers the upstream sends 400 Bad Request.

I quess the commit https://github.com/alek-sys/spring-cloud-gateway/commit/d0af511465bfd2652adcd6855f7cb48623755c37 would fix this but I cannot find that code from Hoxton.RELEASE.

I tried to add .removeRequestHeader("host") but the case sensitivity handling is incoherent and the header is removed completely and nothing works.