gunicorn does not support byte-range requests for Safari html5 video support

[racitup]

Hi I'm having trouble serving a video using gunicorn and application = Cling(MediaCling(application))
I've done a bit of debugging using wireshark and it looks like gunicorn is putting the Content-Length into the http response body as hex (0x620F01 == 6426369). See the trace comparisons below for my local gunicorn server and the same file hosted on Amazon S3:

Local:

GET /media/filer_public/c7/1a/c71a88a8-6b32-42d6-b7ff-b95ebf2ccc24/video.mp4 HTTP/1.1
Host: 192.168.1.73:8888
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Cookie: csrftoken=1TgZwhDTt85kNKcnX1C40bJPRGL2ytkaKgWhOhlghJBFT6k6lzp0z9WpoildirVF; django_language=en; sessionid=dtoqjt84ak1kmy4rwm94szp0242ptktr
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_3) AppleWebKit/601.4.4 (KHTML, like Gecko) Version/9.0.3 Safari/601.4.4
Accept-Language: en-gb
Accept-Encoding: gzip, deflate
Connection: keep-alive

HTTP/1.1 200 OK
Server: gunicorn/19.6.0
Date: Fri, 29 Sep 2017 19:14:49 GMT
Connection: close
Transfer-Encoding: chunked
Last-Modified: Sun, 04 Jun 2017 16:07:51 -0000
ETag: 1496592471.0
Content-Type: video/mp4

620F01
... ftypisom....isomiso2avc1mp41....free.b..mdat..........E....H..,. .#..x264 - core 120 - H.264/MPEG-4 AVC codec - Copyleft 2003-2011 ...snip

Amazon:

GET /filer_public/c7/1a/c71a88a8-6b32-42d6-b7ff-b95ebf2ccc24/video.mp4 HTTP/1.1
Host: staging-casesolved.s3-website.eu-west-2.amazonaws.com
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-gb
Connection: keep-alive
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_3) AppleWebKit/601.4.4 (KHTML, like Gecko) Version/9.0.3 Safari/601.4.4

HTTP/1.1 200 OK
x-amz-id-2: 4Oye/lYEw4QNphLJclq8x9FAoFszMp7TBZSBz4NDWSi8pBWfO0DJj7snxBQ3rXRWOxgzlIbzjNo=
x-amz-request-id: 24FF26E93DD2E1C7
Date: Fri, 29 Sep 2017 19:22:42 GMT
Last-Modified: Fri, 29 Sep 2017 17:56:24 GMT
x-amz-version-id: 3Tb1uGbNY0wCIJKtJRxnsv6uUwoFcNu.
ETag: "2d3d37a8028bf54bdc1d15640f9f5b43"
Content-Type: video/mp4
Content-Length: 6426369
Server: AmazonS3

... ftypisom....isomiso2avc1mp41....free.b..mdat..........E....H..,. .#..x264 - core 120 - H.264/MPEG-4 AVC codec - Copyleft 2003-2011 ...snip

Is this a bug?

[jamadden]

That looks like the difference between HTTP chunked encoding (for ease of streaming without unbounded memory usage) in gunicorn's case and non-chunked encoding in S3's case. In chunked encoding, the file is divided into parts and each part begins with its size in hex. (It's possible for the whole file to be sent in one chunk.)

Perhaps your client doesn't properly support HTTP chunked encoding?

[racitup]

It's Safari 9?

[jamadden]

Well Safari 9 (and OS X 10.11) is relatively old at this point, but I wouldn't expect it to be broken...and it could also depend on plugins. If you use something that's known to support chunked encoding correctly (like recent versions of wget or curl) against both sources, and get byte-for-byte identical files on disk, then I would tend to suspect the client/plugin (or even another browser less dependent on the OS version, like current Chrome). OTOH, if you don't get identical results when you know the source files are identical, then you might suspect the serving application or even the server.

[racitup]

Yeah thanks for the input and the detail. I suspect this isn't the main problem I'm facing, TBH I'm clutching at straws at the moment. I think I have a combination of problems.
But by loading just the video through gunicorn I just get an empty video player in Safari. Through aws I get the same player but the video loads, but I also get an error: "Failed to load resource: Plug-in handled load".
I just upgraded to Safari 11 and got much the same result. Well I guess this might be useful for others.

[racitup]

It seems this is a "byte-range" request issue. I tried gunicorn vs aws s3 using curl and gunicorn did not support byte-range serving. See here

[jamadden]

Thanks for your responses, I do hope this is useful. That's something I haven't considered in awhile.

As far as I can tell though, byte-range responses are an optional feature for Python WSGI servers (since they are intended to simply "dumbly" stream what the application gives them); supporting ranges portably is up to the application:

A server may transmit byte ranges of the application's response if requested by the client, and the application doesn't natively support byte ranges. Again, however, the application should perform this function on its own if desired.

Do you know if other Python WSGI servers themselves support byte range requests? Is this common? (I maintain the gevent webserver, and I don't think it supports it. I don't know if waitress supports it. If I had to guess, I wouldn't think the reference implementation supports it, but I haven't tested)

[racitup]

I've just tried waitress and that doesn't support it. I suspect I'm expecting too much serving videos through the same app for simplicity. It's my first web app, I don't know much about WSGI servers, I was just hoping for a simple Heroku deployment for testing... It appears django runserver supports it though.

[racitup]

It appears this is a deeper problem with Safari, though I get the same behaviour on Chrome for Android:
Safari 9.0 can not play mp4 video on the storage server

[racitup]

Incidentally, I'm serving using dj-static and application = Cling(MediaCling(application)) but don't understand the interaction between gunicorn, dj-static and django..

[racitup]

Could gunicorn be modified to add the following header Accept-Ranges: none like here: Range_requests

[jamadden]

That seems quite wrong to me, as it would block a WSGI application that wants to handle ranges from doing so. The WSGI spec seems pretty clear to me that the WSGI application is responsible for handling ranges. Perhaps you can modify your application to add that header?

[jamadden]

don't understand the interaction between gunicorn, dj-static and django..

As I understand it (and maybe I'm wrong), basically WSGI servers are just responsible for the bytes on the wire and the barest minimal possible interpretation of HTTP semantics. WSGI applications (like, I guess Cling and MediaCling) are responsible for implementing anything richer than that. This is designed to keep WSGI applications portable.

[racitup]

Ah okay, so gunicorn would be the wrong place in the stack to implement it.. fair enough

[jamadden]

Maybe not exactly wrong, but certainly non-portable, and if applications follow the spec and implement ranges themselves, wasted effort. The WSGI community seems to place a premium on portability and application freedom. That can be painful at times. Trust me, I know.

Thanks for this discussion!

[racitup]

I don't really understand how supporting the feature would make gunicorn non-portable.. in what respect? I could imagine gunicorn being able to put the Accept-Ranges: bytes header on all responses, and then post-processing all application responses to only send the requested bytes if necessary, converting the response to a 206.
It definitely wouldn't be pretty or probably efficient, but do-able. And python is necessarily portable amongst platforms isn't it?

[jamadden]

It doesn't make gunicorn non-portable, it makes the application that relies on it non-portable. One couldn't simply test such an application with wsgiref, or WebTest, or deploy it with waitress, or gevent, etc. That defeats the point of a specification.

It also places performance constraints on gunicorn (it must do a certain amount of "buffering" to be able to satisfy all range requests).

And because the functionality can easily be implemented as a WSGI middleware in a portable way, applications can easily opt-in to it when they need to without placing a burden on the server or other applications.

[racitup]

Ah I see, yes fair enough. Thanks for explaining..

[tilgovi]

Sounds like exactly the sort of thing that WSGI middleware is perfect for solving in a framework independent way.

[racitup]

And as if by magic... static-ranges
It would be great to get some feedback...