K6: net/http: HTTP/1.x transport connection broken: too many transfer encodings: ["chunked" "chunked"]

[vire]

Environment

• k6 version: k6 v0.29.0 ((devel), go1.15.4, darwin/amd64)
• OS and version: OSX 10.15.7

Steps to Reproduce the Problem

1. I captured a HAR file of a local API
2. I converted HAR to K6 test
3. When I executed the test I can see the error

WARN[0013] Request Failed error="Get \"<URL>": net/http: HTTP/1.x transport connection broken: too many transfer encodings: [\"chunked\" \"chunked\"]"

I did try to export given request from chrome network tab as cURL and node fetch and both are working as expected showing body.

Request headers

> Accept-Encoding: deflate, gzip
> Connection: keep-alive
> Cache-Control: max-age=0
> Upgrade-Insecure-Requests: 1
> User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36
> Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9

response headers

< HTTP/1.1 200
< Referer:  URL_REMOVED
< Cache-Control: no-cache
< Pragma: no-cache
< Transfer-Encoding: chunked
< vary: accept-encoding
< Content-Encoding: gzip
< Content-Type: application/json;charset=UTF-8
< Transfer-Encoding: chunked
< Keep-Alive: timeout=60
< Connection: keep-alive

[na--]

Hmm this seems to be a restriction in the Go stdlib's http module (which we use in k6): https://github.com/golang/go/blob/9eef49cfa6eb016e3b20df189e540c6c5a71f365/src/net/http/transfer.go#L634-L643

Not sure there's a lot we can do here. And I'd wager that if we dig into the RFCs, having the server emit multiple Transfer-Encoding headers probably will not be legal.

[na--]

Yeah, see https://tools.ietf.org/html/rfc7230#section-3.3.1

A sender MUST NOT apply chunked more than once to a message body (i.e., chunking an already chunked message is not allowed).

[vire]

@na-- is there anything I could do in the k6 scenario to prevent this issue or is this problem entirely based on the server response implementation mechanism?

[na--]

I don't think so, sorry. This is just a matter of Go (and thus, k6) being more strict in enforcing the HTTP protocol spec than other tools. I guess cURL and node just silently ignore the error and carry on, but it's still an error and the webserver is incompliant with the spec and should be fixed. Out of curiosity, what webserver are you using?

[vire]

@na-- I'm closing this issue as there is no action from k6 possible

[vire]

Out of curiosity, what webserver are you using?

I'm not 100% positive because it's a chain of proxy services so saying anything would be just guess