Hls.js: Possible memory leak w/ multiple Hls players

Created on 20 Jun 2017  Â·  66Comments  Â·  Source: video-dev/hls.js

Environment

  • [x] The stream has correct Access-Control-Allow-Origin headers (CORS)
  • [x] There are no network errors such as 404s in the browser console when trying to play the stream
  • [] The issue observed is not already reported by searching on Github under https://github.com/video-dev/hls.js/issues
  • [] The issue occurs in the latest reference client on http://video-dev.github.io/hls.js/demo and not just on my page
  • Link to playable M3U8 file: any
  • Hls.js version: v0.7.9
  • Browser name/version: Chrome v59
  • OS name/version: Windows 10 v1703
Steps to reproduce

This is a replication of the issue /clappr/clappr#1446. I'm having at maximum 9 hls players in order to build a videowall showing CCTV streams in HLS format.

I've implemented pure hls.js and the results were better than clappr's (as expected because there's no additional overhead), but the memory rises overtime (around 15-20 minutes of spectating) to > 1.5GB and up, and it doesn't stop nor decrease significantly.

This is my hls config dictionary:

var config = {
            capLevelToPlayerSize: true,
            maxBufferSize: 30,
            maxBufferLength: 5
        };
        self.hls = new Hls(config);

I don't know what else I can do. I've seen issue #939 but I don't know if it is related because its target version is old.

Thanks in advance.

Expected behavior

Consistent memory usage

Actual behavior

Sky is the limit for memory usage (or just memory limits)! The browser even crashes after exceeding a certain amount of GBs (depending on computer).

Confirmed Help wanted Stale

All 66 comments

Any chance you could upload a demo page we can check it and take a memory dump to analyze?

Due to company restrictions I'm afraid not. But I took 3 heap snapshots:

Snapshot at the beginning (aprox. 300-350MB): http://www117.zippyshare.com/v/11YrQ3WG/file.html

Snapshot after 1 GB: http://www117.zippyshare.com/v/iCxMxCf0/file.html

Snapshot around 1.5GB: http://www117.zippyshare.com/v/i0jzkbj2/file.html

Any help would be appreciated.

Thanks!

Does your issue exist with - let us say 3 players - on the same page?
Have you tried forcing garbage collection to see if the memory goes down or to identify the source of the potential leak?
I have seen Chrome eating on RAM (4K streams but I guess multiple instances could do that as well) and still be ok to run for hours ... but yes it may crash on a 2GB RAM laptop at some point.

It does occur with 3 players although not so aggressively, i.e. it does not reach 1.5GB in such a short period of time.

This videowall is supposed to be on some road's management centres, and will be used on a 24-hour basis. I've taken some heap snapshots (as I shared above) and from my analysis I can see, while comparing the 3, that its some kind of array that keeps piling up.

Here are some screenshots:

2 minutes of usage:

heap

5 minutes later:

heap

One thing I observed is that if I reload the source (by specifying hls.loadSource(link)) of all players the memory drops to 400/500MB.

The Memory column represents native memory. DOM nodes are stored in native memory. If this value is increasing, DOM nodes are getting created.

The JavaScript Memory column represents the JS heap. This column contains two values. The value you're interested in is the live number (the number in parentheses). The live number represents how much memory the reachable objects on your page are using. If this number is increasing, either new objects are being created, or the existing objects are growing.

So it seems that DOM nodes are getting created. Do you have something in your application that is creating DOM nodes? I am not sure hls.js creates that many even if you are using captions.

As a workaround, can you try creating 9 players in iframes not sharing memory and see if the leak still occurs?

I'm not convinced (especially without a demo page) this is a leak of shared data and not just GC waiting for a good time to act.

I've been deeply analyzing my code in order to check if there was any kind of loop that created DOM elements, but I've had no luck. Also from snapshot analysis, I compared the DOM elements in snapshots 1.5GB and 350MB and the amount of them was kept. Detached DOM aswell...

I've built a simple webpage that simulates my main's behavior. The problem persists...

It's available here. I've put 9 video elements playing a random stream. The memory escalates like this:

Start
node

5 minutes
node

15 minutes
node

(The stream I used when I took these screenshots is not the same as in the hosted demo page, and this last one has a HQ stream which certainly uses more memory).

Also disable ad-block because for some reason it's blocking the stream.

while checking your heap dump with chrome, I see a bunch of objects created by cea608 parser. this parser is creating VTTCue which should be in the DOM ...
image

there is an easy way to disable CEA608, just replace hls.min.js by hls.light.min.js.
and let us know if you still observe the same...
btw i cant use your test page, your stream is 404ing.

I have switched to light and it got better, but it's not fully resolved. The memory still escalates, although much slower. I still feel something is piling up somewhere.

From observation at the chrome task manager, I can now see the GC working at some points, but it leaves a lot of trash behind.

I've fixed the example stream.

ok could you repost heap dump highlighting the issue ?
thanks

@andrefilipegsilva Is there any way not to use zippyshare? - I feel like i'm getting an adware virus just looking at it. Dropbox or Mega.nz work well.

just checked the 1.2GB dump. Chrome says it references only 27MB worth of objects
image

Be careful as Chrome Dev Tools is a huge contributor over time. Disable the network panel while doing memory testing.

I've been trying to put the problem in a corner but I've had no luck so far.

@mangui Although there is only 27MB worth of objects, the whole memory increases limitless. By that time it was around 1.2 GB.

@kfeinUI Even with that disabled the issue persists.

I just can't fix this on my own. Please tell me what else do I need to provide so we can analyze this deeper.

I am experiencing the same exact issue.

@mangui @benjamingr @kfeinUI @radiantmediaplayer

I've fixed the stream again: http://demovideowall.000webhostapp.com/test.html

And I've taken 3 new snapshots from that test stream. The first one is the "initial", when the 9 players rendered. The "middle" one points to the time when the tab consumed 1.5GB. The "end" one occurred by the time it was occupying 3.2GB.

Link to End Snap
Link to Init Snap
Link to Middle Snap

(all urls point to mega.co.nz)

This stream has higher quality and so it consumes a higher amount of memory. But it just crashes in 5 minutes...

Print of the last one in task manager:
image

It's important to point out a few things,

  1. issue manifests when the number of simultaneous players increases.
  2. it may or may not be related to playing the SAME live stream in multiple players. I have not tried playing different streams yet.

I've written a custom loader that uses rxjs to constrain network requests for the same resource (e.g. multiple requests to seg_1.ts are reduced to 1 request). Internally, the loader returns a clone of the segment arrayBuffer via slice() to each player subscribed to identical content.

(<any>callbacks).onSuccess({ url: response.url, data: cached.slice() }, stats, context, xhr);

for 6 players on this loader, playing the same manifest (1500kbps), i can see memory escalate to ~1GB but then eventually release instead of crashing chrome. Additionally, my laptop doesn't sound like it is about to take off into outer space...

Internally, I assume hls.js is still transmuxing each cloned buffer... so the issue may be separate from that process if it helps any.

When I leave a live stream running, I am observing no invocation of SourceBuffer.remove() api.

https://github.com/video-dev/hls.js/blob/412e8cd3c293cf4bd0a7b0792d1b45f76e813bfa/src/controller/buffer-controller.js#L541

this appears to be the code responsible, but it never hits my breakpoint.

@mangui i spent some more time this weekend trying to understand when/how the MSE source buffer get flushed, and I can't find any instance where the implemented flushing logic actually executes. I think the browser eventually cleans this up, but in scenarios like @andrefilipegsilva and mine it does not appear to be aggressive enough in doing so.

Modifying the "max.." and "maxMax..." options appears to have no effect on the behavior. Please advise.

@josh-sachs hls.js only flushes the buffer in case of forced level switch.

buffer gets automatically evicted by browser as per MSE spec
with live stream modifying the "max.." and "maxMax..." will have no effect most of the times as it is controlling the max buffering upfront (whereas in long live session most of the buffer will be a back buffer)

if you have some specific requirement you could eventually manually flush as follow

hls.trigger(Event.BUFFER_FLUSHING, {startOffset: xxx , endOffset: yyy, type : 'main'});

for example on LEVEL_UPDATED you could flush from startOffset 0 to the beginning of the sliding window.

I would be curious to see if that addresses your pb.
if that is the case we might add an option to allow that force flush on playlist sliding.

Results of my testing are interesting...

All scenarios are 9 player instances all using a cloned ArrayBuffer from a single loader. The live feed is 5,500kbps h.264 encoded at main profile.

Chrome 60 (natrual): memory exceeds 2GB in ~5min, then slowly climbs to 3GB+ (~30min)
Chrome 60 (flushed): memory gets to ~1GB after about 5min. Then hovers between 1.2GB and 0.9GB for some time... after about 30min I did observe it peak to 2GB.

Chrome 62.0.3178.2 (natrual): memory exceeds 2GB within 10 minutes, then hovers betweem 1.8GB and 2.5GB (seems to release memory as I would expect, just less aggressive).
Chrome 62.0.3178.2 (flushed): memory hovers between 350MB and 900MB (this seems to release memory exactly as I would expect).

Firefox 54.01 (natural): memory exceeds 2GB within 10 minutes, hovers between 1.8GB and 2.5GB (seems to release memory as I would expect, just less aggressive).
Firefox 54.01 (flushed): memory hovers between 850MB anf 1.3GB (this seems to release memory as I would expect).

I think there is an issue with MSE natural memory eviction in the current version of chrome. Additionally, I don't think the natural eviction policies are appropriate for live content.

Chrome Canary (62.0.3178.2) actually runs really well in this scenario when I trigger buffer flush after every hlsFragChanged event. They have obviously either fixed an issue in this release, or implement better garbage collection in their preview builds than their official release builds.

hls.on('hlsFragChanged', (e: Event, data: any) => { const range = { startOffset: 0, endOffset: data.frag.startDTS }; this.flowplayer.engine.hlsjs.trigger('hlsBufferFlushing', range); });

@josh-sachs thanks a lot for your insights.
indeed there seems to be an issue with browser natural buffer eviction.
IMHO it would make sense to add an option to trigger a buffer flush and remove all data outside of current sliding window (basically flush buffer from offset 0 to details.fragments[0].start)
this could be done in buffer-controller
through a new config param : liveFlushBeforeStartOffset , any naming advice welcome

and help welcome as well !

Maybe add an option called liveMode that captures all the live specific
enhancements? Could also prevent seeking.

On Thu, Aug 17, 2017 at 1:46 AM Guillaume du Pontavice <
[email protected]> wrote:

@josh-sachs https://github.com/josh-sachs thanks a lot for your
insights.
indeed there seems to be an issue with browser natural buffer eviction.
IMHO it would make sense to add an option to trigger a buffer flush and
remove all data outside of current sliding window (basically flush buffer
from offset 0 to details.fragments[0].start)
this could be done in buffer-controller
https://github.com/video-dev/hls.js/blob/master/src/controller/buffer-controller.js#L351-L358
through a new config param : liveFlushBeforeStartOffset , any naming
advice welcome

and help welcome as well !

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/video-dev/hls.js/issues/1220#issuecomment-322974703,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AUOqepFEdBpfe78QMfk7ynAMaGUgbIriks5sY9OfgaJpZM4OAGtX
.

So, turns out making an XMLHttpRequest with an arraybuffer data type leaks in Chrome @mangui - even if you evict all your buffers. You can reproduce this in desktop Chrome by calling:

function req() {
  return new Promise(resolve => {
    var xhr = new XMLHttpRequest;
    xhr.open("GET", "path/to/100mb.testfile?foo="+Date.now());
    xhr.onload = resolve;
    xhr.responseType = 'arraybuffer';
    xhr.send();
  });
}
(async () => {
  while(true) await Promise.all([req(), req(), req(), req(), req()]);
})();

The above code slowly rises (when pointing to a real large file, or a regular segment if you've got time) to 32GB on my desktop Chrome.

I actually ran into this when using Hls.js in Android - and I haven't considered it is the same bug - but have since reproduced it in desktop.

I reported this to Google and they have reproduced it.

Also been experiencing this with multiple instances. Had a hunch, but
couldn't pin it down. Nice to see some progress.

Seek has its value behind the scenes. You wouldn't necessarily expose it
through the player UI. There's also the DVR playlist case.

On Aug 17, 2017 7:56 AM, "jon-valliere" notifications@github.com wrote:

Maybe add an option called liveMode that captures all the live specific
enhancements? Could also prevent seeking.

On Thu, Aug 17, 2017 at 1:46 AM Guillaume du Pontavice <
[email protected]> wrote:

@josh-sachs https://github.com/josh-sachs thanks a lot for your
insights.
indeed there seems to be an issue with browser natural buffer eviction.
IMHO it would make sense to add an option to trigger a buffer flush and
remove all data outside of current sliding window (basically flush buffer
from offset 0 to details.fragments[0].start)
this could be done in buffer-controller
controller/buffer-controller.js#L351-L358>
through a new config param : liveFlushBeforeStartOffset , any naming
advice welcome

and help welcome as well !

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<https://github.com/video-dev/hls.js/issues/1220#issuecomment-322974703
,
or mute the thread
AUOqepFEdBpfe78QMfk7ynAMaGUgbIriks5sY9OfgaJpZM4OAGtX>
.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/video-dev/hls.js/issues/1220#issuecomment-323050218,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AE0hHsY2Vn67OrDIblLxsKuzRiGZzY-wks5sZCpygaJpZM4OAGtX
.

Is there been any progress on this matter?

I have not been able to spend any more time on it personally.

I did notice something interesting with a similar project I'm working on... Can you confirm your virtual memory settings on Windows?

I can't find the link right now, but I had come across a ticket in the Chrome bug tracker that essentially made me lose all faith in humanity.

The gist (as I understood it anyway) is that there is a known issue in both the 32-bit and 64-bit chrome builds where the underlying methods responsible for alloc() and free() address space for arrayBuffer will leak as the size of the arrayBuffer changes... Consider a sequence of received arrayBuffers of various sizes [1,1,1,2,2,2,3,3,3,3,1,1,1,4,4,1,2,2] every time Chrome encounters an arrayBuffer with a size that is different than the previously alloc'd size, the space reserved for the previously sized arrayBuffers is lost. In this example, the transition from 1byte => 2byte length leaks, then again from 2byte=>3byte...

The result (again, as I understood it) is that the development team has chosen to ignore the leak because as they put it, on 64-bit systems, "memory address space is virtually infinite." The conclusion they reached after ~2years of back and forth was that 32-bit is dying, and 64-bit systems will just swap effectively enough to make it a non-issue. Additionally, I imagine the issue is not as severe when the content is delivered in uniformly sized segments.

Back to the reason I ask about Virtual Memory... ever since I was a small lad I've been a tinkerer. And there was a point in time where I came to understand that letting Windows manage virtual memory was for losers, and that manually specifying the size of the page file (or even disabling it if you have enough RAM) was for winners. Reverting back to allow windows to manage the page file automatically has at least seemed to resolve the "Ooops Chrome Has Shit The Bed" error message that would eventually pop up. Chrome memory consumption still continues to climb and climb and climb, but it seems that removing my artificial limits has indeed provided the "infinite address space" Chrome currently depends on in order to mask this bug.

32-bit systems I assume are out of luck... and again, I think it's discouraging that even at Google, where they essentially have infinite resources, stuff like this gets pushed under the rug in favor of going on a bike ride, or hitting up the corporate volley-ball court.

Its entirely possible that I've misunderstood something, or that the bug I found doesn't have anything to do with the issue we are currently having - but it seems suspiciously close.

@andrefilipegsilva hi,have you solved your problem?i met with the same problem with you .could you give me some advice? i use the v0.8.9 verison.,and chrome 65.thank you .

@fallowu can you try with Chrome 66 and see if the XHR leak bug is solved?

@benjamingr ok, but could you tell me how to confirm the bug has been solved?

Well, the memory leak should be gone :D

@benjamingr But testing nine players in one website on chrome66,the memory still up to 1.5G in five minutes.

9 players each loading different videos? That's going to buffer a lot in
memory. Have you identified something specific in profiling?

On Tue, Apr 17, 2018, 9:17 PM fallowu notifications@github.com wrote:

@benjamingr https://github.com/benjamingr But testing nine players in
one website on chrome66,the memory still up to 1.5G in five minutes.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/video-dev/hls.js/issues/1220#issuecomment-382241406,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AE0hHiAbTUbXNDPOuOK0SNnTHGypMW1Tks5tprBigaJpZM4OAGtX
.

no,just loading the same video. and I just use the benchmark.html demo 0.8.9 versions,using for-loop structure.

The default buffer eviction in Chrome is something like 400 MB; I don't doubt your problem. The question is: is there a way for hls.js to force back buffer eviction? For live streams, hls.js should really be manually removing buffer data from everything > 10 seconds in the past.

@jon-valliere I don't think browsers currently allow eviction: https://github.com/w3c/media-source/issues/172

@jon-valliere Oh cool. We use this API already to facilitate level switching (by removing forward buffer of the old level) and to fix buffer full errors. We can also use this code to clear backbuffer to free up memory.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

I think this one is worth keeping around

I can confirm that using the internal buffer event that @mangui mentioned controlling the back-buffer size to bring memory utilization down using a setInterval for long-term live streaming has helped control the amount.

As mentioned adding a backBufferMaxLength or something could be used to call the remove event.

Hello, I am writing an application almost exactly like what the original poster implemented and experiencing the same memory leak issues. I am admittedly not very familiar with the under-the-hood implementation of this library, so the workaround mentioned above all sounds like greek to me. Could someone provide a code snippet or example so I can better grasp how to apply this workaround to my web app's react video player component?

@rgwebb This issue is still active until someone adds back buffer management to force the browser to release memory. There is no work-around until backBufferMaxLength is added.

Understood. Thank you for the response.

If this issue is now about the browser retaining memory _even after_ the XHR memory leak was fixed - I'm not sure why it's open on hls.js and not the whatwg or chrome issue trackers.

@benjamingr because it can be resolved by having hls.js delete the back buffer otherwise Chrome may keep an hour or more in memory. Its not a bug in Chrome because having a lot of players isn't a typical scenario Chrome should optimize for and it can be resolved using Javascript.

@itsjamie can you provide a snippet of your setInterval solution to control backbuffer?

It was a co-worker who confirmed the solution for me, unfortunately so I don't have a snippet of code to grab.

New config setting was added liveMaxBackBufferLength (seconds) to limit the back buffer length in PR https://github.com/video-dev/hls.js/pull/1845 . Checked on Mac with liveMaxBackBufferLength = 60 - Chrome 67 and Firefox 61 memory consumption was steady. I don't have Windows machine so help with testing this on IE/Edge is appreciated.

Hi.

@josh-sachs, @itsjamie i have the same problem with permanently increasing memmory usage. Is there any solution/workaround? What is this "setInterval solution to control backbuffer"?

@vaydich The solution will be in #1845. I will also profile Hls.js to see if any of the code is memory leaking.

@johnBartos, so when there planning the next release?)

@vaydich The scope of 0.11.1 has not been set yet. There are still regressions to triage from 0.11.0. I can't really calculate a timing but you can track the progress here https://github.com/video-dev/hls.js/milestone/7

Just want to drop some recent findings into this thread. Some info from the Chrome team:

1) Buffer limits are 150MB for video, 12MB for audio
2) Buffer limits are per sourcebuffer instance

So it seems like the problem is not so much a memory leak but that the eviction algorithm is per-player and not per-tab; therefore multiple players can easily exceed the memory limit of Chrome. But they also said that memory should not infinitely grow; therefore there must be some memory leaks in Hls.js. I know for a fact that we don't clear VTT cues (ever) so that's one source, but I'm curious if there are leaks for streams without captions.

Because of cross-platform difficulties I believe the best solution for the present problem is to pursue #1845. In the future browsers may improve MSE memory allocation - in Chrome, for example, there are experiments with memory pressure to dynamically configure the memory limit: https://bugs.chromium.org/p/chromium/issues/detail?id=811464. But today we're able to address this issue ourselves so let's go with that.

Additionally, relying completely on the MSE buffer eviction may be less performant than manual eviction. If the limits are reached then the MSE must be automatically evicting every time the buffer is appended. These eviction events are costly and might cause the Javascript and/or Networking to stall. Therefore the player may choose to manually evict larger portions (such as 10+ seconds) in a single shot in order to reduce the append/eviction ratio when compared to allowing MSE to do it automatically.

@jon-valliere That's an interesting point - I suggested in #1845 that there be only one config option (just the # of seconds of buffer to keep) but it may be worth keeping the second option for performance reasons. I'll suggest this feedback in the thread.

Hi.

Is there any solution about memory leak?

Hi @vaydich , we addressed the largest memory "leak" in https://github.com/video-dev/hls.js/pull/1845. It will be released in 0.11.1, the next release. You can test it in our hosted demo page: https://video-dev.github.io/hls.js/latest/demo/

I have the same issue. Memory keeps increasing until the page eventually crashes. Tested on version 0.12.4, in Firefox & Chrome.
Changing liveMaxBackBufferLength option didn't help.
For perspective, with 48 low-bitrate HLS streams playing on the page, the memory usage grew from 0.9GB at the start to 4GB in 40 minutes.


upd: it crashed in the old version of Chrome (59). In the latest version (73) I had it working for 2 days without crashing.

Does your stream have captions? We’re aware of a memory leak where old cues
are never cleared. There’s a PR open to fix it.

On Wed, Apr 10, 2019 at 01:52 Alexandr Motuzov notifications@github.com
wrote:

I have the same issue. Memory keeps increasing until the page eventually
crashes. Tested on version 0.12.4, in Firefox & Chrome.
Changing liveMaxBackBufferLength option didn't help.
For perspective, with 48 low-bitrate HLS streams playing on the page, the
memory usage grew from 0.9GB at the start to 4GB in 40 minutes.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/video-dev/hls.js/issues/1220#issuecomment-481543648,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AEzwZI_Q7H2yxKWO20Em-fFR6_y6h9z2ks5vfXwlgaJpZM4OAGtX
.

@johnBartos my bad, I didn't test properly. I had page crashes in an old version of Chrome (59), which apparently had some memory leak issues. After that, I tested it in newer versions and noticed that the memory is growing, so I assumed it's leaking too. But it turned out that it just fluctuates up and down. I kept the tab open (48 streams ~100kbit bitrate) for 2 days and it didn't crash. The memory usage fluctuated between 1 and 2 Gb.

btw, the option mentioned by @vitalibozhko is actually spelled liveBackBufferLength (without Max)

Just in case somebody comes here with the memory issue on videojs (7.x version) I wanted to give a sample code on how to actually rollout the fix. Its as follows. When tested the below code takes memory upto 250MB after a continuous run of 2hrs

```
window.addEventListener("offline", (e) => window.location.reload());

    this.instanceId = new Date().getTime();

    videojs.registerPlugin("hlsQualitySelector", qualitySelector);

    this.player = videojs(
        this.videoNode,
        {
            techOrder: ["html5", "flash", "other supported tech"],
            liveui: true,
            autoPlay: "muted",
            controls: true,

            html5: {
                hlsjsConfig: {
                enableWorker: true,
                liveBackBufferLength: 15,
                backBufferLength: 15,
                liveMaxBackBufferLength: 15,
                maxBufferSize: 0, 
                maxBufferLength: 10,
                liveSyncDurationCount: 1,
                },
            },
        },
        function onPlayerReady() {
            this.play();
        },
    );

    this.player.hlsQualitySelector({ displayCurrentQuality: true });
    ```
Was this page helpful?
0 / 5 - 0 ratings

Related issues

osamay picture osamay  Â·  4Comments

NicholasAsimov picture NicholasAsimov  Â·  3Comments

jlacivita picture jlacivita  Â·  3Comments

harbinha picture harbinha  Â·  3Comments

nickcartery picture nickcartery  Â·  4Comments