Exoplayer: DASH/HLS/SS downloads: Parallelize & merge requests to improve download speed

[ojw28]

ExoPlayer currently requests segments one at a time when downloading segmented media. This means that the network is not fully utilized, particularly in cases where there's significant latency between client and server. There are two approaches for improving this:

1. Parallelize segment downloads so that the network can remain more fully utilized.
2. Where consecutive segments have the same URL and consecutive byte ranges (normally true for DASH VOD, and also supported in HLS with #EXT-X-BYTERANGE), it's possible to request multiple segments in a single HTTP request.

In both cases, any implementation should ensure that demuxed tracks are downloaded at approximately equal rates, because users expect that if something is ~33% downloaded, approximately the first ~33% of that piece of content will be playable.

[houzhanbin]

What is the progress of this issue. the performance of our product offline download feature is extremely poor. because of the low ts cdn, we need parallelize segment downloads to download more ts files in parallel.

[Stijak]

This is really serious issue. For a bare 600MB (in 8000 chunks) player needs about 40 minutes to download (over 300 MBIT network)...
This is only 15%ish percent faster than action-file based implementation, and more then 10 times slower then our old custom download implementation on same medias...

[janeriksamsonsen]

@Stijak - just measured a download on my side:
Pixel 3 XL
100mbit connection
Around 8700 segments (video + audio + subtitle) where each segment is 2 seconds long.
Total downloaded: 4.2GB
Time to finish: 11 minutes

Your numbers sounds strange. Can you please share some details about your device and implementation?

I do however also see the need for parallelized downloads since ExoPlayer download is still quite a bit slower than our 3rdparty player/downloader.

[Stijak]

Ok, thanks for feedback. Judging on your responses, on my test Galaxy S9+ speed should be high even on exoplayer 2.9, and yet, still slow, and there are no CPU bottlenecks or anything like that.

I even transferred DASH from CDN to my local macbook apache to eliminate eventual CDN problems with small chunks.
Ok, I guess I will keep looking and the problem is not in exoplayer, although I don't understand how my code could influence download speed.

[Stijak]

It seems that significant portion is spent on request initialization...
And much smaller audio files are sometimes slower...

If results from both mobile network (to CDN) and another team in another part of the globe wasn't similar, I would blame local WiFi...

[Stijak]

Update:
I repeated the same tests (local apache server) with dash samples from https://dash.itec.aau.at/dash-dataset/

I used red bull sample, since it was longest, and using same bitrate we aim for our content, and our production app for testing (so it would be apple to apple comparison), I was able to test 15 second segment size and 1 second segment size downloads.

15 second segments download finished in less than 2 minutes, while 1 second segment took about 8 minutes. So here clear differences in download speed.

1 second is about 5 000 segments and still noticeably faster than our comparable content. Is it some other encoding option, or DRM (our content is using Widewine) I am not sure, but will try to reach solution with content providers...

[zourb]

Are there any updates for this issue? The parallelized downloads feature is really important especially for poor segments downloads' speed.

[noamtamim]

@ojw28 is this something you're currently working on? We're looking to move from our current download solution to ExoPlayer's built-in, and this is a serious blocker.

[tonihei]

Unfortunately no progress yet. But acknowledged the renewed interest and we should probably take a look soon.

[houzhanbin]

what is the status of this issue?

[busylee999]

Is there are any update?

[ebmalone1234]

We still have a lot of interest in this issue. Is anyone actively working it?

[tonihei]

The merging part is already done and released from 2.11.0 onwards (see commit above). The parallelization is also implemented but not submitted yet due to pending reviews and it will likely be in the next major release.

[ojw28]

Correction: The merging part was not actually included in 2.11.0 (or any tagged release to date). It will probably land together with the parallelization improvement in 2.12.0.

[kschults]

@ojw28 Is there a branch where one could test this out prior to the 2.12.0 release becoming available?

[sky-filipeuva]

I second this question ☝️ We are really interested in testing this feature

[ojw28]

Changes will appear on the dev-v2 branch as they become available, and will be referenced from this issue (like the ones above).

Note that merging segment fetches is already committed, for the (ideal) case where consecutive segments use the same URL and adjacent byte ranges. Parallel segment fetches is being worked on currently, and you should see commits appearing over the next two weeks.

[sky-filipeuva]

Thanks very much for the update and transparency !

[jruesga]

I developed my own implementation of ExoPlayer parallelization segments here (https://github.com/jruesga/ExoPlayer/commits/parallel-chunks-downloads), just in case anyone want to test it. It's currently based on r2.10.8, but includes the "merge segment" patch, so I think is easy portable to the stable and dev branches.

Seems to work quite nice, and I'm observing systematically lower download times.

I measured the patch with some DASH and HSL resources from the demo app.

[Romantic-LiXuefeng]

@janeriksamsonsen Thanks for your good work. I have tested, It woks nice. Can you tell me what's the tools are you using to measured the download times?

[janeriksamsonsen]

@Romantic-LiXuefeng, you can get detailed information about HTTP-requests in an Android app by using Stetho https://github.com/facebook/stetho. Timings can then be viewed in chrome://inspect.

[ojw28]

@jruesga - I don't think what you're doing with cache keys is valid. It's a fundamental property of a cache key that it should be independent to the request position. As an example of something that I think probably happens with your changes, suppose two segments (A) and (B) are merged by the segment merging functionality in the downloader. They'll both end up being cached under a key suffixed with (A.absoluteStreamPosition). During playback, the player is going to make separate requests for those two segments. So it will try and look up (B) with a key suffixed with (B.absoluteStreamPosition). There will then be a cache miss and offline playback will fail (assuming you have no network).

The good news is that merging (partially) addresses cases where segments have the same URL, and parallel segment downloads address the case where segments have different URLs. So both cases are improved one way or another. I suspect that the optimal case where we both merge and parallelise may need changes in SimpleCache to allow more fine-grained write locking (i.e., allowing multiple writers to the same key, provided the ranges being written to do not overlap).

[jruesga]

@ojw28 thank you so much for your feedback.

[ojw28]

No worries! We'll be merging changes to add parallel segment downloads to the dev-v2 very soon. They don't look too different to what you've done, except we'll be allowing the application to inject an Executor from the top level, and drip feeding it with Runnables in a way that makes it possible for an application to use the same background executor that it might be using for other download tasks (e.g., to fetch images for the app UI), without starving anything else that's trying to use it.

I haven't looked at locking in SimpleCache yet, but it might actually be relatively straightforward to allow multiple write locks in the case that they have bounded and non-overlapping byte ranges.

[hasija79]

It looks like the getManifest function got removed from the SegmentDownloader implementation classes after these changes. We are overriding this abstract function to filter the playlist & copy stream keys currently.

Can we provide access to the getManifest in dependent classes like we have now or remove the final keyword in getManifest method in SegmentDownloader? Thanks

https://github.com/google/ExoPlayer/blob/535e14cb4d69ca41f91118cf2bd201ea94746d53/library/core/src/main/java/com/google/android/exoplayer2/offline/SegmentDownloader.java#L227

[ojw28]

I don't really understand what you're doing, or why. Playlist filtering based on the stream keys already happens automatically in SegmentDownloader.

[hasija79]

We've the video bitrate available before start of download & not stream keys. In getManifest, I am loading the manifest & selecting suitable stream keys before filtering.

protected HlsPlaylist getManifest(@NonNull DataSource dataSource, @NonNull DataSpec dataSpec) throws IOException {
       HlsPlaylist hlsPlaylist = loadManifest(dataSource, dataSpec);
       List<StreamKey> keys = listener.onFilterPlaylist(embedCode, hlsPlaylist);
       hlsPlaylist = hlsPlaylist.copy(keys);
       return hlsPlaylist;
}

[ojw28]

I see. I think the way we'd support that is to let you inject your own HlsPlaylistParser through the constructor. Which you could implement to do something like:

@Override
public HlsPlaylist parse(Uri uri, InputStream inputStream) throws IOException {
  HlsPlaylist playlist = normalParser.parse(uri, inputStream);
  return yourCopyingLogic(playlist);
}

It's possible you already have such a HlsPlaylistParser implementation for playback. Does that approach work for your use case?

[hasija79]

I believe FilteringManifestParser should work in our case with this logic.

Thanks for the prompt reply. Appreciate it.

[ojw28]

I haven't looked at locking in SimpleCache yet, but it might actually be relatively straightforward to allow multiple write locks in the case that they have bounded and non-overlapping byte ranges.

@jruesga - We were able to make multiple write locks work. This is merged in 235df09.

Hopefully we'll get the change that actually parallelises everything in next week. It's turned out to be a much bigger task than was initially anticipated.

[houzhanbin]

was this issue resolved ? and is it merged to release version ?

[ojw28]

This is fully implemented in dev-v2, and will be part of 2.12.0. Please give it a try.

Note: Additional commit not ref'd above: https://github.com/google/ExoPlayer/commit/c9717f67ea7ccd647188210c83ec582e98e4c5c7

[Romantic-LiXuefeng]

I have try to read the source code, it's complex. @ojw28 Could public the design docs about this?

[ojw28]

We don't have a public design doc for this work; sorry.

[lavajaw]

This is fully implemented in dev-v2, and will be part of 2.12.0. Please give it a try.

Note: Additional commit not ref'd above: c9717f6

Do you have release date for 2.12.0? Or raw estimation?

[ojw28]

We're actively working on burning down pending issues for 2.12. I think late August is probably our best estimate as of now.

More generally, we'd like to avoid having so much work sitting around unreleased for a long time going forward, which has happened with 2.11 compared to the current dev-v2 branch. We're looking at how we can improve our release process going forward (it's not straightforward though, so it remains to be seen how good of a job we'll do :)).

[ojw28]

Closing since this will be released in 2.12 any day now!