Hls.js: Error while loading m3u file (After update to new version of Chrome 52.0.2743.82 m (64-bit))

[Halynsky]

Environment

• [x] The stream has correct Access-Control-Allow-Origin headers (CORS)
• [x] There are no network errors such as 404s in the browser console when trying to play the stream
• [x] The issue observed is not already reported by searching on Github under https://github.com/dailymotion/hls.js/issues
• [x] The issue occurs in the latest reference client on http://dailymotion.github.io/hls.js/demo and not just on my page
• Link to playable M3U8 file: all time dynamic files, with low life time
• Hls.js version: Release 0.6.2-3
• Browser name/version: Chrome 52.0.2743.82 m (64-bit)
• OS name/version: Windows 10

Steps to reproduce

Load m3u file in to video player:

Example:

EXTM3U

EXT-X-VERSION:3

EXT-X-MEDIA-SEQUENCE:147

EXT-X-TARGETDURATION:6

EXTINF:6.000,

GFFDEAFBCEDFEBEDCCBCCABC-147.ts

EXTINF:6.000,

GFFDEAFBCEDFEBEDCCBCCABC-148.ts

EXTINF:6.000,

GFFDEAFBCEDFEBEDCCBCCABC-149.ts

Page where u can see the bug: (no debug log from hls):
https://login.partizancloud.com/login#demo

use any online camera

Expected behavior

Stream is loaded and playing

Actual behavior

Error while loading playlist file

Console output

[log] > loadSource:http://developer.partizandm.com:8000/hls/GFFDEAFBCEDFEBEDCCBCCABC.m3u8
hls.min.js:6 [log] > trigger BUFFER_RESET
hls.min.js:6 [log] > attachMedia
videoController.js:99 true
hls.min.js:6 [log] > media source opened
hls.min.js:6 [log] > manifest loaded,1 level(s) found, first bitrate:undefined
hls.min.js:6 [log] > startLoad
hls.min.js:6 [log] > engine state transition from undefined to STOPPED
hls.min.js:6 [log] > demuxing in webworker
hls.min.js:6 [log] > engine state transition from STOPPED to STARTING
hls.min.js:6 [log] > switching to level 0
hls.min.js:6 [log] > (re)loading playlist for level 0
hls.min.js:6 [log] > engine state transition from STARTING to WAITING_LEVEL
hls.min.js:6 [log] > audio tracks updated
hls.min.js:6 [warn] > timeout while loading http://developer.partizandm.com:8000/hls/GFFDEAFBCEDFEBEDCCBCCABC.m3u8
hls.min.js:6 [error] > cannot recover levelLoadTimeOut error(anonymous function) @ hls.min.js:6value @ hls.min.js:2value @ hls.min.js:4value @ hls.min.js:4e.emit @ hls.min.js:1i.trigger @ hls.min.js:4value @ hls.min.js:5value @ hls.min.js:6
hls.min.js:6 [log] > engine state transition from WAITING_LEVEL to ERROR
hls.min.js:6 [warn] > mediaController: levelLoadTimeOut while loading frag,switch to ERROR state ...

Headers

Response headers:
Accept-Ranges:bytes
Access-Control-Allow-Credentials:true
Access-Control-Allow-Headers:DNT,X-Mx-ReqToken,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type
Access-Control-Allow-Methods:GET, POST, OPTIONS
Access-Control-Allow-Origin:*
Cache-Control:no-cache
Connection:keep-alive
Content-Length:139
Content-Type:application/vnd.apple.mpegurl
Date:Tue, 09 Aug 2016 02:51:13 GMT
ETag:"57a94521-8b"
Last-Modified:Tue, 09 Aug 2016 02:51:13 GMT
Server:nginx/1.9.1

Comment

Bug is in Chrome version 52.0.2743.82 m (64-bit)
No bug in 51.0.2704.103 m and 51.0.2704.106
No bug in Mozilla

Updating of the m3u file stops after first 304 status code (Screenshot):
http://prntscr.com/c3cljk

Further investigation

Bug all time happening when this dates in headers are same (only in Сhrome):
Date:Tue, 09 Aug 2016 14:13:56 GMT
Last-Modified:Tue, 09 Aug 2016 14:13:56 GMT

(Screenshot)
http://prntscr.com/c3jl4u

[lafraia]

Having same issue here after Chrome update

Chrome version (Mac): 52.0.2743.116 (64-bit)
URL: https://stream.vagalume.fm/hls/14623883771774082279/index.m3u8

#EXTM3U
#EXT-X-VERSION:3
#EXT-X-MEDIA-SEQUENCE:9677
#EXT-X-TARGETDURATION:45
#EXTINF:45.000,
39188036970.ts
#EXTINF:45.000,
39192086970.ts
#EXTINF:45.001,
39196136970.ts
#EXTINF:45.000,
39200187060.ts
#EXTINF:45.000,
39204237060.ts
#EXTINF:45.000,
39208287060.ts
#EXTINF:45.001,
39212337060.ts
#EXTINF:45.000,
39216387150.ts
#EXTINF:45.000,
39220437150.ts
#EXTINF:45.000,
39224487150.ts
#EXTINF:45.001,
39228537150.ts
#EXTINF:45.000,
39232587240.ts
#EXTINF:45.000,
39236637240.ts

[jsardas]

I can reproduce the same issue specifically with latest Chrome, but it seems to happen only when the following conditions are met:

• the URL is a direct playlist and not a master playlist
• the response headers contain either a no-cache directive or a very short expire time

Reference:
Expires: Tue, 09 Aug 2016 01:00:39 GMT
Cache-Control: no-cache

If any of these conditions change, the exact same video/stream works fine. For instance, if a master playlist is used instead of a direct playlist, it works, and if caching is allowed in the response headers, it works too.

I can try to setup 2 sample test URLs in order to reproduce this behavior if it helps with debugging...

[Halynsky]

Thank you guys for confirmation. Removing of the header "Cache-Control: no-cache" really helps in my situation with Chrome. It start work as before. But I still have same bug in 10-15% cases.

[JAD3N]

I'm also having this issue on Chrome (tested on Firefox and it appears to be working as expected). Is there a more permanent fix for Chrome?

[jsardas]

It seems like the xhr request that is performed by hls.js to reload the playlist after a level has been selected might attempt to reuse a cached version in Chrome, and this somehow creates an issue if the original response had a "Cache-Control: no cache" header.

This would also explain why using a master playlist also solves the issue, because a reload for the actual (direct) playlist is forced even if the master playlist is cached.

I used the following workaround while testing different scenarios only for VOD:

if(/chrome/i.test(navigator.userAgent)) {
   hls.on(Hls.Events.MANIFEST_PARSED, function() {
      console.log('Forcing playlist reload...');
      hls.startLoad();
   });
}

This might be useful as a temporary fix if you don't have access to the server(s) generating the response for the playlist.

[waster]

I confirm this issue in latest Chrome and Opera. Removing of the header "Cache-Control: no-cache" does not help, because in some cases timeout is still triggered.

[jsardas]

Hi @waster,

Are you using Expires in your headers? I noticed that timeout is occasionally triggered when using an expire time with a very short duration (i.e. a few seconds), but I haven't been able to reproduce the timeout when both Cache-Control and Expires are completely removed from the headers.

[waster]

@jsardas I think that it's not right to remove Cache-Control or Expires headers because in this case Access-Control-Allow-Origin header will be cached and it can cause CORS problems if you'll try to access m3u8 with multiple tabs on different domains and if Access-Control-Allow-Origin is set as origin (e.g. in case of using Access-Control-Allow-Credentials: true).

[jsardas]

@waster I understand your point... however, in order to attempt to identify if the issue originally reported is indeed related to caching in Chrome or not, would it be possible for you to confirm if the cases where you are still encountering the timeout have a short expire time set in the headers? In other words, even if Cache-Control is removed, is the request supposed to expire in just a few seconds?

[waster]

@jsalaj Do you mean to test timeout with only Expires header with short time without Cache-Control header?

[jsardas]

@waster , you mentioned that removing the Cache-Control: no-cache header didn't help in your testings since you found that in some cases the timeout was still triggered. My point is trying to determine if those specific cases you are referring to might have been caused by a very short expire time, based on the Expires header variable.

There is a (relatively) short time between the moment hls.js initially downloads the playlist and the moment it re-downloads it again once the level has been selected. This time can vary based on network conditions from each user, but if it's longer than the expire time set by Expires at the header, it would make sense that the timeout is still triggered even without a Cache-Control: no-cache directive.

That's why I asked if you could confirm if the cases you were referring to (i.e. where the timeout was still triggered without Cache-Control: no-cache) included a short Expires directive or not.

If they did, what would be interesting to test is whether or not increasing the expire time solves the bug. For instance, first testing with a very short expire time, confirming that the bug occurs, and then testing with a very long expire time (i.e. 5 minutes), and eventually confirming that the bug disappears...

[waster]

@jsardas, Hm, when I set "expires -1;" in my nginx.conf it actualy sets Expires header's value to Date header minus 1 sec and Cache-Control: no-cache header. So actually Expires header contains date is in the past, and browsers should consider request as being equivalent to the Cache-Control response directive "no-cache", right?

[jsardas]

@waster, yes, Nginx handles this automatically and if a negative value is assigned to expires in config (i.e. expires -1), Cache-Control is added with no-cache value; a positive expires value generates a Cache-Control: max-age=X. But since these header variables can also be tweaked manually in some other cases, it would be possible for a response not to include Cache-Control but still maintain an Expires at the same time. That's why I asked if you were using Expires.

Since you are using Nginx, would it be possible for you to test by setting something like expires 60; in your nginx.conf? This should produce a response with:

Expires: (current date/time + 60s)
Cache-Control: max-age=60

If you still encounter the timeout issue when hls.js re-loads the playlist, would it be possible for you to share the playlist URL? I also tested with Nginx but once the expire time was longer than the time between the first playlist load and the second re-load, I wasn't able to reproduce the timeout issue anymore. I assumed this would be consistent with the idea that the whole issue had to do with the way Chrome is now handling caching for XHR requests. But if you still encounter the issue with such a long expire time, then the actual cause could be different...

[waster]

@jsardas Second reload of the playlist is performed very quickly, on the stage of reloading playlist for level 0 (BTW, I see that hls.js always runs initial 2 requests on the direct playlist, one for manifest loading, and second for level 0 reloading, is it not that bad for those who cares for http statistic, etc?) and timeout originates here. I attached image with some notes.

You can use this hls.js demo link to test. Willl check with Expires header version too.

[jsardas]

@waster, the playlist that is used in your test includes a Cache-Control: no-cache directive:

curl -I http://hls.01.radio7.emgsound.ru/13/128/playlist.m3u8
HTTP/1.1 200 OK
Server: nginx/1.10.0
Date: Sun, 14 Aug 2016 18:04:36 GMT
Content-Type: application/vnd.apple.mpegurl
Content-Length: 264
Last-Modified: Sun, 14 Aug 2016 18:04:35 GMT
Connection: keep-alive
ETag: "57b0b2b3-108"
Expires: Sun, 14 Aug 2016 18:04:35 GMT
Cache-Control: no-cache
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true
Access-Control-Expose-Headers: X-Current-Location
Accept-Ranges: bytes

[waster]

@jsardas, yes, because I want to demonstrate timeout issue itself. Will try to set Expires header later.

[jsardas]

I also tested the following workaround which modifies the playlist (m3u8) request by appending a random querystring via XHR, forcing Chrome to re-download the playlist instead of attempting to use a local cache (304):

var config = {
  xhrSetup: function(xhr, url) {
    if (url.split('.').pop() == 'm3u8') {
      url = url.replace('m3u8', 'm3u8?rd=' + Math.random());
      xhr.open('GET', url, true); 
    }    
    if (url.indexOf('?rd=') > -1) {
      url = url.replace(/(rd=)[^\&]+/, 'rd=' + Math.random())
      xhr.open('GET', url, true); 
    }
  }, 
  debug:true
}

var hls = new Hls(config);
(...)

While it's obviously not a fix, it solves the issue with Chrome without having to modify server headers and without having to trigger hls.startLoad() manually.

[chjcken]

Hi @jsardas and @waster

I'm facing this issue too. But I control cache using Etag header, you can see below

Access-Control-Allow-Headers:Origin, X-Requested-With, Content-Type, Accept, Range
Access-Control-Allow-Methods:GET, POST, OPTIONS
Access-Control-Allow-Origin:*
Content-Type:application/vnd.apple.mpegurl
ETag:AAABVowRP8M=

and sure there is no Cache-Control or Expire header but whenever I return status 304, stream will stuck and Hls fire LevelLoadTimeout error

Btw, if my streamUrl is a master playlist, stream will play fine as @jsardas said above and certainly both master and level playlist request are response with status 304

If you guy find any solution for this issue, please share it. Thank you :D

[Halynsky]

In our investigation of this problem we found similar problem. We made some simple test. I want to show You guys results in Chrome and Mozilla. This problem is only if timeout is very low:

Code:

Chrome:

Mozilla:

[segvic]

I've also experienced the same issue with HLS streams coming from nginx and latest versions of Chrome. The problem is that the same chrome versions on different hosts with the same os (Win 10) behave differently - on some the success rate of loading the stream is 10% and on other is 90%. Crazy stuff.

Anyway, what helped - solved the problem - was a master playlist file, to which to point the player to.
A simple plain text file which then redirects the player to the generated m3u8 file by nginx. For example:

EXTM3U

EXT-X-VERSION:6

EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=1255600,CODECS="avc1.4d001f,mp4a.40.2",RESOLUTION=720x540

/hls/name_of_the_stream.m3u8

All other browsers - Opera, Firefox, Edge, IE work with or without the master playlist file - but for chrome this is currently the only solution that is full-proof. I've tried a dozen of recommendations - changing cache control, adding rand number to URL - none of that helped, only this master playlist file solves the issue.
I have to note that Chrome does not have this issue with HLS streams coming from Wowza.

Hope this helps

[gkindel]

I've been able to work around this with a setTimeout 0 for xhr.send() within xhr-loader.js, as the browser doesn't actually seem to do anything if it's in the same call stack as the previous loadsuccess.

Also confirm the fix of removing the ETag from the m3u8 response, but I'm not going to do that.

It's definitely a Chrome thing, and is reproducible by using a minimal XHR test.

[mangui]

That's interesting. Anything in chrome bug tracker ?

[gkindel]

No ticket that I've found. The bug is actually pretty hard to reproduce. In debugging via proxy header rewriting, I've only had it happen when there is an ETag but not any Expires or Cache-Control headers, which is unfortunately my case. Adding either "fixes" the minimal test case for me.

[waster]

@gkindel hm, I removed etag header from m3u8 response and it did not help to fix timeout.

[mangui]

ok I can reproduce, i am on it.
basically I will try to avoid this double request when there is no master playlist.
related to https://github.com/dailymotion/hls.js/issues/49

[gkindel]

Thanks for looking! I'll take a look at your patch. You may also want to verify that the live polling works. I appear to be having issues even with my earlier workaround. Code appears to successfully poll (with unchanged, cached response) but chrome never logs the request in the network tab.

[waster]

@mangui Hm, can you please try to check this audio only link? It's still timeout there however it starts to play after timeout.

[mangui]

@waster this is a live without master playlist.
after loading manifest, stream controller switches to level 0, this triggers another manifest reload because of the details.live === true condition

as a consequence another xhr request on the same URL is done synchronously in loadsuccess() callback. which is triggering chrome issue.

I guess replacing https://github.com/dailymotion/hls.js/blob/master/src/loader/playlist-loader.js#L344
by
setTimeout({hls.trigger(Event.MANIFEST_LOADED, {levels: [{url: url, details : levelDetails}], url: url, stats: stats});},0);

might do the trick in your case

[mangui]

Hi @waster plz recheck
http://dailymotion.github.io/hls.js/demo/?src=%2F%2Fhls-01-radio7.emgsound.ru%2F13%2F128%2Fplaylist.m3u8&enableStreaming=true&autoRecoverError=true&enableWorker=true&levelCapping=-1&defaultAudioCodec=undefined

chrome issue with live without master playlist should be fixed now

[waster]

@mangui Great work, thanks!

[waster]

@mangui Can these commits be added in 0.5.44 or next 0.5.45 stable release?..8)