Shaka-player: Chrome not freeing up HDD space after offline content deletion

[bartsidee]

Have you read the FAQ and checked for duplicate open issues?:
yes

What version of Shaka Player are you using?:
2.4.3

Can you reproduce the issue with our latest release version?:
yes

Can you reproduce the issue with the latest code from master?:
yes

Are you using the demo app or your own custom app?:
custom app (electron)

If custom app, can you reproduce the issue using our demo app?:
yes

What browser and OS are you using?:
MacOS / Windows

What are the manifest and license server URIs?:
any manifest

What did you do?

1. Download one or multiple offline manifest
2. Clear/Delete the files (using shakaplayer code, not by clearing the db manually)

What did you expect to happen?
The available HDD space of the indexeddb to be reclaimed

What actually happened?
The available HDD space of the removed manifest of indexeddb are not reclaimed, chrome (chromium) is not compressing the LevelDB files on disk, as a result the references are removed in Indexeddb (and seemingly removed) however they still taking up space on the HDD.

Downloading a new manifest sub-sequential seems to trigger a db compress action, however still not clearing all the previous data (just reducing the size with small chunks). This will lead to a situation where a users HDD is full and/or indexeddb quota is reached.

Technically this seems to be a bug in both webkit / chrome
https://bugs.chromium.org/p/chromium/issues/detail?id=488851
https://bugs.webkit.org/show_bug.cgi?id=178204

However it would be interesting to review if there are workarounds to enforce/manually trigger a leveldb compression action after a manifest deletion

[theodab]

I tried experimentally downloading a few assets and then deleting them, and yeah the indexeddb.leveldb folder did end up with a few hundred megabytes of data in it anyway. So I can confirm that this is an issue.

I'll ask Aaron about this, he knows the IndexedDB stuff better than me.

[joeyparrish]

If we use web APIs to delete the data, and the browser doesn't free up any space on the disk, I'm not sure if there are any steps we can take at the JavaScript level to mitigate this.

We will track it as an external issue for now, but we're open to ideas on workarounds if anyone has any.

[theodab]

I talked to Aaron (who implemented the offline storage stuff to begin with), and he agrees that there's probably nothing we can do as a workaround ourselves.

[mstattma]

Not really a workaround, but as an alternative solution approach if possible for the use case: Electron provides better ways of managing downloaded content as it provides direct file system access.

Shaka can still be used for playback. Here is an Electron DASH + MSS downloader library with an example using Shaka:
https://github.com/castlabs/downstream_electron

[vaage]

We already allow someone to implement their own storage mechanism. With that you should be able to plug-in an alternative back-end to store the content.

[mnAdroid]

Did anyone found a workaround for this issue? I just tried the offline feature and the problem is still, well a problem for me.

[joeyparrish]

This seems to be a bug in Chrome with no known workaround. I've just asked for an update on the associated Chrome bug: https://bugs.chromium.org/p/chromium/issues/detail?id=488851

We'll try to make the right contacts in Chrome to get this resolved. But it will require an update to the browser, which normally would take about 3 months to make it from trunk to a stable release. But both development and release schedules are impacted by COVID right now, so I doubt that the Chrome team could promise any particular timeline on a fix.

Please accept our apologies for this ongoing issue in Chrome.

[carlosabud]

Thanks @joeyparrish, @bartsidee

I ran into this issue a few days ago as well. We've just did an indexeddb implementation in our app, quota is updated as you save items but as you delete them the size stays the same. You have to manually delete the indexeddb in order to free up the space...

Did anyone find a way (programatically) to refresh the quota api to get the right size? We wouldnt mind doing a refreshSpace() before querying the space.

[joeyparrish]

As far as we know, there's no API for that. The browser should do it automatically, but there's a bug in Chrome preventing it from working correctly.