Spring-cloud-gateway: gateway-mvc: chunked responses with status 500 are damaged

[vadimkim]

Describe the bug
Trying to proxy 500 response with "Transfer-Encoding:chunked" damages message body by eliminating chunk length from the body and as a result getting:

curl: (56) Illegal or missing hexadecimal sequence in chunked-encoding
Chunked errors with 4xx codes processed correctly

Sample

1. Create service that returns 500 error with "Transfer-Encoding: chunked"
2. Proxy GET request there
3. Test with curl -v http://service/url
   Response:
   curl: (56) Illegal or missing hexadecimal sequence in chunked-encoding
   Using tcpdump you can see that service response comes with 500 and chunk size. After request processing by ProxyExchage header still contains "Transfer-Encoding:chunked", but no chunks data is in the body of the message.

Service 500 chunked response:

Proxied 500 "chunked" response:

Spring Cloud version: Hoxton.SR1

[ryanjbaxter]

Can you provide a complete, minimal, verifiable sample that reproduces the problem? It should be available as a GitHub (or similar) project or attached to this issue as a zip file.

[vadimkim]

Sure. Here it is. There are 2 projects - api (to generate 500) and proxy (to proxy to api). URL-s are hard-coded. Just run both projects and execute:
$ curl -v http://localhost:8080/api/v1/proxy500
test.zip
Expected behavior: to get 500 error with payload
Actual behavior: curl: (56) Illegal or missing hexadecimal sequence in chunked-encoding

[vadimkim]

Investigating the problem a bit more have found that error happens only with default Tomcat servlet container. Switching to Jetty or Undertow fixes the problem.

[taojiaenx]

@vadimkim the tomcat is open keep alive operation when use default config, It was my fault

[taojiaenx]

@vadimkim the default contrainer is netty starter, and can not provide with persistent connection, Did the problem happen when you used netty starter?Have your change container to tomcat manually ? If not would you like change it to tomcat and try again?

[vadimkim]

@taojiaenx please look at the topic - the problem is not related to spring-cloud-gateway that is reactive. It is related to module spring-cloud-gateway-mvc. It is stand-alone component that uses basically just one class - ProxyExchange. For MVC application the default servlet container is Tomcat. You can download test example and see there are just 2 lines of code. If you tell me which Spring Boot configuration property is responsible for persistent Tomcat connection I will add it to application.properties and test again. I didn't find it at documentation.

[taojiaenx]

thanks for reply and I will try to reproduce this scene.

[taojiaenx]

@vadimkim My current conclustion is that the problem is cased by innernal logic of tomcat and you can solve this
by upgrading version of tomcat
I found the solution here
https://stackoverflow.com/questions/50529119/prevent-tomcat-from-sending-header-connection-close
when there was an error occurred without any message, tomcat will add a http header

Connection:close

however with undertow this is

Connection: keep-alive

So when the api is tomcat and proxy is tomcat too
proxy will send the header and ignore message and you can not receive any return message
otherwise when the api is tomcat and proxy is undertow
poxy will just ignore the header "connetion:closed",just send default error message
and you will see the message is different when api is undertow and proxy is tomcat

the source code in tomcat is here

        long contentLength = response.getContentLengthLong();
        boolean connectionClosePresent = false;
        if (contentLength != -1) {
            headers.setValue("Content-Length").setLong(contentLength);
            getOutputBuffer().addActiveFilter
                (outputFilters[Constants.IDENTITY_FILTER]);
            contentDelimitation = true;
        } else {
            // If the response code supports an entity body and we're on
            // HTTP 1.1 then we chunk unless we have a Connection: close header
            connectionClosePresent = isConnectionClose(headers);
            if (entityBody && http11 && !connectionClosePresent) {
                getOutputBuffer().addActiveFilter
                    (outputFilters[Constants.CHUNKED_FILTER]);
                contentDelimitation = true;
                headers.addValue(Constants.TRANSFERENCODING).setString(Constants.CHUNKED);
            } else {
                getOutputBuffer().addActiveFilter
                    (outputFilters[Constants.IDENTITY_FILTER]);
            }
        }

[vadimkim]

@taojiaenx have you tried to run projects from test.zip? Tomcat is the latest version there - 9.0.31. If you run wireshark you can see no difference with connections or statuses of them on network layer. The only thing I can guess right now is different "processors" of the response. Somewhere tomcat response transformer is different from jetty or undertow and it physically modifies the body of the response thus braking the protocol

[taojiaenx]

@vadimkim I have run it . The http header that provided by tomcat and undertow is different, the problem is caused by the way of tomcat solve the http header. Even if you use tomcat ,the protocol is not broken, tomcat just write nothing in the http body,
As you can see the information of http header from any browser or just curl, there will be http header like this

the return body is empty

and may be this scence is not allowed by command curl, and it will return

curl: (56) Illegal or missing hexadecimal sequence in chunked-encoding

you can also solve the problem in this way just remove the header "connection" when it is equals to "close"

@RequestMapping("api/v1")
@RestController
public class ProxyController {

    @GetMapping("/proxy500")
    public ResponseEntity<?> proxy500(ProxyExchange<byte[]> proxy) {
        ResponseEntity<?>entity = proxy.uri("http://localhost:8081/api/v1/error500").get();
       MultiValueMap<String, String> multiValueMap = new HttpHeaders();
       for (Entry<String, List<String>> stringListEntry : entity.getHeaders().entrySet()) {
            // just remove connection
            if ("connection".equalsIgnoreCase(stringListEntry.getKey())
                && "close".equalsIgnoreCase(stringListEntry.getValue().get(0))) {
                multiValueMap.put(stringListEntry.getKey(), stringListEntry.getValue());
            }
        }
        return new ResponseEntity<>(entity.getBody(), multiValueMap, entity.getStatusCode());
    }
}

the result is here

curl 'http://localhost:8080/api/v1/proxy500'
{"timestamp":"2020-03-12T10:34:47.334+0000","status":500,"error":"Internal Server Error","message":"error 500","path":"/api/v1/error500"}%   

The problem will be reproduced even if you have not use spring boot, just startup the tomcat by itself.
The problem is caused by http header, and probably a bit difficult to check by wireshark, browser or curl is more suitable

[vadimkim]

@taojiaenx I appreciate your opinion, but you are misleading header parameters:

1. Connection:close is normal status after client got reply from the server. Jetty also gives it for this reply in the same way. Wireshark is a perfect tool to capture the output and see the difference. Look what you get when proxy through Jetty:
   There is not difference in the headers, but ... payload
2. Problem is at "chunked" response processing by Tomcat. Look at the first 2 pictures in this thread and you will see that "chunked" response contains HEX values at the body that mark the begging and the end of the payload. Spring-cloud-gateway-mvc with Tomcat removes those 2 special chars from the body, but header remains the same : Transfer-Enconding:chunked. This is against HTTP 1.1 protocol. Read RFC7230 chapter 4.1. Such a response is unacceptable.
3. Your code doesn't solve the problem in fact. You create new response that might not be "chunked" or "chunked", but with correct HEX marks, because it is new response. It is a hack.
   There is a bug somewhere. Don't know where exactly - seems to be deep in the framework. I hope it will be fixed. Don't need workaround like "re-assembling the garbage". At the moment I use Undertow instead of Tomcat.

[makdeniss]

I can confirm the same problem. Can anybody from the spring-cloud-gateway team verify this?

[taojiaenx]

@vadimkim
Hello ? I have reduced the problem just whithout using spring cloud gateway mvc

    @GetMapping("/proxy500")
    public ResponseEntity<?> proxy500(ProxyExchange<byte[]> proxy) {
       MultiValueMap<String, String> multiValueMap = new HttpHeaders();
       multiValueMap.put("Transfer-Encoding", Collections.singletonList("chunked"));
        multiValueMap.put("Connection", Collections.singletonList("close"));
        return new ResponseEntity<>("{", multiValueMap, HttpStatus.INTERNAL_SERVER_ERROR);

    }
}

and this is the reuslt

curl http://localhost:8080/api/v1/proxy500 -vvv
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8080 (#0)
> GET /api/v1/proxy500 HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.54.0
> Accept: */*
> 
< HTTP/1.1 500 
< Transfer-Encoding: chunked
< Connection: close
< Content-Type: text/plain;charset=UTF-8
< Date: Sat, 14 Mar 2020 16:15:17 GMT
< 
* Illegal or missing hexadecimal sequence in chunked-encoding
* stopped the pause stream!
* Closing connection 0
curl: (56) Illegal or missing hexadecimal sequence in chunked-encoding

and it worked ok with jetty
I try to find out the reason from source code of tomcat, may be is caused by spring mvc. It is really a hard job.
I really hope you or anybody from spring team can help me.I keep working now.

[taojiaenx]

I have reported the issue to spring-framework
https://github.com/spring-projects/spring-framework/issues/24699

[spencergibb]

Closing in favor of framework issue. We'll reopen here if needed