Okhttp: Content-Length is removed by OkHttp but getContentLength returns -1

[mingfai]

The web server does return a positive "Content-Length" header (as checked in debug mode and also separated test) but OkHttp removed it so the header cannot be retrieved by getHeaderFields. I guess the removal is intended as there is a getContentLength method to retrieve the content length. However, in my case, the content length is -1.

The web server supports gzip/inflat, and the problem occurs when "Accept-Encoding" is not set (also when "Accept-Encoding" is set to "identity", as i tried after seeing #116). When "Accept-Encoding" is set to "gzip,deflate", content length is normal/positive. So i suppose the problem is caused by a transparent/automatic gzip is happening.

I suppose I shouldn't set "Accept-Encoding" to gzip, because when I connect to a server that doesn't support zip, my code can't tell whether the input stream is gzip'd or not, right?

btw, I need to get the content length either from the API or from HTTP Header for change detection.

The following issues are similar but apparently different issue.
https://github.com/square/okhttp/issues/116
https://github.com/square/okhttp/pull/250

[mingfai]

A few questions:
1) if i use gzip, when the web server doesn't use gzip, in the code, how to determine whether the input stream shall use GZIPInputStream or not
2) if I use gzip, is there any way to get the original content length http header? it's not for getting the byte stream size (that i can buffer the input stream to get the number), I'd like to have the original content-length so that if the content-length is not changed, I can skip the input stream processing.
3) is there any way to disable transparent gzip?

[swankjesse]

1. You can determine whether gzip was used by inspecting the Content-Encoding header.
2. No. The original content length is discarded.
3. Disable transparent gzip by manually setting an Accept-Encoding of your choice.

   urlConnection.setRequestProperty("Accept-Encoding", "identity");

[mingfai]

from my test, when urlConnection.setRequestProperty("Accept-Encoding", "identity"); is used, and the web server does provide the "Content-Length" response header, but urlConnection.getContentLength still return -1 in 1.1.1 and the current snapshot. Is it expected? I expect getContentLength shall return the value of Content-Length.

Test code:

HttpURLConnection conn = okHttp.open(url);
 conn.setRequestMethod("GET");
 conn.addRequestProperty("Accept-Encoding", "identity");
int status = conn.getResponseCode();
if (status==200){
 int contentLength = conn.getContentLength();
 // contentLength is -1 here
} 
 ...

[swankjesse]

OkHttp only strips the content length if it's doing transparent gzip. It's possible your webserver isn't sending a content length.

[mingfai]

you are right. the url doesn't return a Content-Length. sorry for the false bug report.

[Macarse]

@swankjesse:

OkHttp only strips the content length if it's doing transparent gzip

Do you mind explaining why?

[swankjesse]

There's two lengths:

• _o_: the length of the original content
• _c_: the length of the compressed content

When we strip the content length from the HTTP response, it's because we know _c_ but the application developer is expecting _o_. (And will receive _o_ bytes when they consume the entire stream.)

[knappador]

@swankjesse we want content length over the wire. I didn't find a way to unzip the response manually to support not getting the content length header stripped.

[knappador]

There's two lengths:

o: the length of the original content
c: the length of the compressed content

We want c for other reasons. I think I was able to get it by reading the Content-Length header inside of a NetworkInterceptor. Diminishing returns in our case, so I hope it's correct. We wanted c but o will tell us almost as much and be hard to distinguish during our early collection phase.

[swankjesse]

Yep. Reading it in a network interceptor is the way to go. Alternately you can do gzip yourself; that’s just another interceptor to make.

[arnavzoman]

@samtstern
Hi Sam, just bringing this to your attention, due to this behaviour of OkHttp, if anyone is relying on OkHttp's transparent gzip decompression, Firebase Performance monitoring fails to report Payload size (Content-Length gets omitted)

[samtstern]

@arnavzoman thanks for the heads up! I don't completely understand the issue here (I don't work on Firebase Perf directly), would you mind filing an issue on firebase-android-sdk explaining what's going wrong so we can assign it to the right people?

[arnavzoman]

@arnavzoman thanks for the heads up! I don't completely understand the issue here (I don't work on Firebase Perf directly), would you mind filing an issue on firebase-android-sdk explaining what's going wrong so we can assign it to the right people?

Cool, I have done that
https://github.com/firebase/firebase-android-sdk/issues/952

[ramanpreetSinghKhinda]

2. No. The original content length is discarded.

swankjesse@ Thanks for explanation. I have a couple of questions:

Q1: Do we discard original content-length because we only know about compressed content (which is not what the caller is expecting) at that point?

Q2: Does reading from the NetworkInterceptor provides Original content length of Compressed content length?

Q2.1: If Original, then is it possible for OkHttp to provide the original content length by adding the Network Interceptor themselves rather the caller implementing it?

Q2.2: If Compressed, then is there any other way caller can get the Original content length? or the caller has to read the entire stream to get that answer?

[prempalsingh]

@swankjesse Would be great if you can check out https://github.com/square/okhttp/issues/259#issuecomment-549639151 once.

[swankjesse]

Note that the original network response headers are available:

String contentLengthHeader = response.networkResponse().header("Content-Length")

[ramanpreetSinghKhinda]

Thanks @swankjesse

Is there also a way to get the compressed content length info (so basically the actual wire download size)?

[swankjesse]

You could get it from the EventListener.
https://square.github.io/okhttp/events/

Or create a Source that counts.

  /** A source that keeps track of how many bytes it's consumed. */
  private class CountingSource(source: Source) : ForwardingSource(source) {
    var bytesRead = 0L

    override fun read(sink: Buffer, byteCount: Long): Long {
      val result = delegate.read(sink, byteCount)
      if (result == -1L) return -1L
      bytesRead += result
      return result
    }
  }

[ramanpreetSinghKhinda]

Reading the stream of data may be the last thing we would do. But thanks for highlighting that well.

Can you provide example with EventListener?

[swankjesse]

Install an EventListener, and override this method:
https://square.github.io/okhttp/4.x/okhttp/okhttp3/-event-listener/response-body-end/