Exoplayer: Generalize/enhance persistent caching functionality

[ojw28]

ExoCache is designed specifically for DASH implementations where every request can be in the form of a bounded range request (which in practice means single-segment on-demand streams containing sidx boxes). It doesn't work for anything else (DASH where streams are segmented in the manifest, SmoothStreaming, HLS, ExtractorSampleSource). There are three things to fix:

1. We don't support unbounded range requests. This is because the cache currently has no concept of the length of a piece of data. If we have a file in the cache with byte range [0-1000], and make a request for range [0-*], we currently don't know what to do after we've return the first 1000 bytes from the cache. We don't want to make an unconditional request to the network starting at offset=1000 because it's inefficient, wont work in the offline case, and because the server may return a 416 (unsatisfiable byte-range) in the case that the content really is 1000 bytes long, which would be awkward to handle.
2. It's probably more common for servers that are configured to serve every chunk from a separate URL to not support range requests. We may need the cache to support a "request the whole chunk or nothing from the upstream source" mode for this use case. I'm not sure how we'd decide when to turn it on. Awkward.
3. The cache currently indexes: contentId->Tree[byte-range->file]. For chunks that are requested from different URLs, we need the Tree to be indexed by chunk-index+byte-range, or perhaps just chunk-index depending on the answer to (2).

[nzer]

It also should be able to cache mp3, mp4 files distributed through plain http. In my use case app plays media file from server, but URL and CDN server may change everytime file is requested. I think we should use key based caching.

[nzer]

I'm making my own cache for now, should I modify DataSource interface or create CacheableDataSource instead?

[nzer]

I did some work, please check it. https://github.com/nzer/ExoPlayer/commit/6c3c89073e25b7af97fc7c55e6a7d15d308a9fbc
Idea is when media is being downloaded (MP3, AAC, whatever) its written to disk at same time. If download succeeds to the end, we mark file finished and offer it to datasource to instead of loading it from network.
Key should be designed by app developer. MD5 of artist+title for music files for example.

[Arqu]

One question about the indexes (key), why not just use a hashing function of the DataSpec info, like uri+position+length? Would be unique enough for the caching structure and would also avoid having another segments chunk index overlap with some other.

[ojw28]

Because if you've cached the whole of a media stream under some hash h(uri, 0, length_of_stream), then you'll get a cache miss for every request to for the same uri unless you happen to be requesting from position 0 for the whole stream, even though you have this data cached already.

[Arqu]

Hmmm, right, however I've managed to get the cache working using a simple hash function of uri+offset+length, which works great currently because it is pretty much unique to that stream and to its requests so if a user seeks he immediately gets the stream playing if it is cached. Only a small hick up happens if the video has switched to a higher resolution meanwhile and you seek back to the stream, than it has to reload that part in another variant, but I guess that is more of a feature/expected than a bug. Also in my tests so far, it doesn't impede any streams at all, live works well with it too - it doesn't cache it. I think that it can be easily leveraged to implement additional functionality like live seeking and similar I just need to extend it to support these more or less unbounded requests.

[ojw28]

The point of this bug is to generalize support for all use cases. What you're doing only optimizes for a subset of cases (from your description, I'm assuming either HLS or a particular variant of DASH/SmoothStreaming). It likely also breaks if a read error occurs part way through a request (subsequent read attempts probably get a cache hit to the incomplete data).

[Arqu]

Yes, I'm currently using the existing implementation and only testing from my HLS use cases. I should definitely take a look what happens with incomplete data and how it handles.

[42footPaul]

Question - just to be clear, this enhancement when done will include the capability to save mp4 from http source to the device as it's playing?

our use case is to play the video from http the first time and if the file download completes as it's playing, then the file gets saved to our app cache and next time it can be played from disk instead.

I have sort of a working solution but I'm not sure it's really the correct approach.
I'm using a customized copy of the upstream.HttpDataSource and my own "caching" transfer listener.

Essentially the http data source passes the bytes (readInternal method) to a custom transfer listener which writes the bytes to a file output stream as the video downloads/plays. When the download is complete/ file is done being written, that video is marked as saved in our app.

This seems to be working as expected except when I ask the player to seek during download - it looks like the connection and input stream are closed and re-opened with a new request and the file output gets hosed.

Couple of questions:
Is there a timeframe that you think this use case will be implemented in ExoPlayer?

Is there another approach I might take in the meantime?

Is it currently possible to create my own downloader and play the video from a file input stream as it downloads?

Is it possible to create an internal http proxy server and pass the internal url to the httpdatasource?

[cmdkhh]

Anyone have a dev branch currently working on this? @Arqu @42footPaul @ojw28

[welshk91]

@cmdkhh Agreed. An update on the status of Exo caching would be most appreciated. Has anyone else found any type of success with caching?

I'm currently attempting to cache mp4 files served from a URL with CacheDataSource, but it doesn't seem to be actually cache anything from what I can tell (powering off/on the screen, releasing the player and reinitializing it, still redownloads the mp4 from the network again).

[jpshelley]

If there is a current roadmap of tasks that need to be done to implement caching, or if anyone has an official development branch started, that would be better then starting from scratch.

If any of these are available do let us know. We plan to start implementing caching ourselves but would love to provide support back to Exo.

[dekalo-stanislav]

Would like to share my investigations in this way.

I have spent some time to implement cache for exo player in my app.
Implementation is very custom and will not fit as general solution that
could be reintegrated back.

Requirements for my app:
Show small looped videos(mp4) < 3MB. Cache it on disk.

Issues that I have faced with:

1. There are class DataSpec (Defines a region of media data), it could be
   bounded (end point defined) and unbouned(end point undefined)
   exo's CacheDataSource doesn't cache unbouned requests as it doesn't know
   how.
2. Player requested to provide DataSpec [0-_) - unbounded request.
   But it does not read it fully, after reading first few kilobytes (header as
   I understand) it stops and opened another unbounded request open(DataSpec
   [4012-_]) (Note that end of first request reading are not equal to start of
   next bounded request)

1:
In my implementation I have no seek functionality, so I have
RandomAccessFile and just append network content to it's end.
To detect cache hit, we store file sizes in separate xml file, and cache
hit happens if file sizes on disk and in xml are same.

2:
I general implementation, with seek functionality I think player could ask
DataSource to provide any part of stream, so our cache will contain various
cached ranges with empty(not requested) regions between cached. Here we
have next issue: how to store that separate cachd data ranges in easy to
process way? Plus we need to implement detection if requested DataSpec is
partially cached, and download only missed.
I do not see other solution except local db implementation.

Let me know if I am unclear or missed something.

2015-11-10 20:15 GMT+02:00 John Shelley [email protected]:

If there is a current roadmap of tasks that need to be done to implement
caching, or if anyone has an official development branch started, that
would be better then starting from scratch.

If any of these are available do let us know. We plan to start
implementing caching ourselves but would love to provide support back to
Exo.

—
Reply to this email directly or view it on GitHub
https://github.com/google/ExoPlayer/issues/420#issuecomment-155518944.

[MaTriXy]

I will just add my interaction with the caching issue.
It's a very simple solution :
I've added my own disklrucache implementation together with creating a custom DataSource implementation where i would cache the response according to the url request issued out and saved it to later on fetch from the disklrucache if it's present.

[laurencedawson]

If you setup OkHttp it's pretty straight forward to get files to cache. I can provide details if needed.

[welshk91]

@laurencedawson an example or some direction would be awesome! I'm thoroughly lost after attempting it for a while now.

Even getting something relatively simple like the ExoPlayer Demo to cache with OkHttp would be tremendously helpful. I swapped in OkHttp and added a cache for it, but the demo still seems to re-download when that activity is recreated (rotation for example).

[laminsk]

@laurencedawson I'm using okhttpdatasource and still no luck trying to download the audio stream to disk. Any input you can provide will be greatly appreciated. Thanks

[b95505017]

Check this:
https://github.com/square/okhttp/wiki/Recipes#response-caching

Using CacheControl to set max stale, and OkHttp's Cache will honor them

[laminsk]

Thanks for the prompt reply. I've followed your suggestion and included the
header in makerequest and configured sOkHttpClient to use cache but it's
not writing to disk at all. Perhaps I'm doing it wrong and if you could
update your sample project with caching, that would be most grateful.

On Fri, Jan 29, 2016 at 11:01 AM, Yu-Hsuan Lin [email protected]
wrote:

Check this:
https://github.com/square/okhttp/wiki/Recipes#response-caching

Using CacheControl to set max stale, and OkHttp's Cache will honor them

—
Reply to this email directly or view it on GitHub
https://github.com/google/ExoPlayer/issues/420#issuecomment-176693906.

[b95505017]

@laminsk Did you install Cache to OkHttpClient first?
For example,

okHttpClientBuilder.cache(new Cache(context.getCacheDir(), CACHE_SIZE));

[laminsk]

Cheers for the reply. I did create the cachedir like this.

Cache cache = new Cache(cacheDir, cacheSize);
sOkHttpClient = new OkHttpClient.Builder()
.addNetworkInterceptor(new StethoInterceptor())
.connectTimeout(DefaultHttpDataSource.DEFAULT_CONNECT_TIMEOUT_MILLIS, TimeUnit.MILLISECONDS)
.readTimeout(DefaultHttpDataSource.DEFAULT_READ_TIMEOUT_MILLIS, TimeUnit.MILLISECONDS)
.followSslRedirects(false)
.cache(cache)
.cookieJar(new JavaNetCookieJar(cookieManager))
.build();

[vjames19]

@laminsk You need to make sure your server returns a Cache-Control header, so that OkHttp can cache the response.

If your server doesn't return any Cache-Control header, then you can add the header to the response on an interceptor.

Here is an excerpt from my working code.

    OkHttpClient provideOkHttpClient() {
        Cache cache = new Cache(cacheDir, cacheSize);

        OkHttpClient.Builder builder = new OkHttpClient.Builder()
                .cache(cache)
                .connectTimeout(10, TimeUnit.SECONDS)
                .readTimeout(10, TimeUnit.SECONDS);

        // Makes OkHttp cache the response for 2 days. You should adjust this time for your need.
        CacheControl cacheControl = new CacheControl.Builder()
                .maxStale(2, TimeUnit.DAYS)
                .maxAge(2, TimeUnit.DAYS)
                .build();

        // Overrides the response Cache-Control header with the value specified by cacheControl
        builder.addInterceptor(new OverrideCacheControlHeaderInterceptor(cacheControl));

        return builder.build();
    }

    public static class OverrideCacheControlHeaderInterceptor implements Interceptor {

        private final String cacheControlValue;

        public OverrideCacheControlHeaderInterceptor(CacheControl cacheControl) {
            cacheControlValue = cacheControl.toString();
        }

        @Override
        public Response intercept(Chain chain) throws IOException {
            Response response = chain.proceed(chain.request());

            return response.newBuilder()
                    .header("Cache-Control", cacheControlValue)
                    .build();
        }
    }

[laminsk]

@vjames19 Thanks for the reply. Despite following your advise, cache files are not being created unfortunately. All I want is simply download and play the audio file from the server and store it on the device for future use.

[MaTriXy]

@laminsk if you wish to download it and then play just use simple file download request with okhttp and then play the file.
If you wish to play and save it in the same time - going the okhttp cache path should be quite simple.

[laminsk]

@MaTriXy Thanks for the reply. The latter is what I'm trying to achieve and I'm using this sample project https://github.com/b95505017/ExoPlayer/tree/okhttp_http_data_source. It simply not creating the cache files despite following the suggestions!

[b95505017]

@laminsk I've force update my branch to sync with latest ExoPlayer.
Also I set CacheControl in PlayerActivity and add CacheMonitorInterceptor.
Un-comment these lines to force caching one year:

  //private final CacheControl cacheControl = new CacheControl.Builder()
  //        .maxStale(365, TimeUnit.DAYS)
  //        .build();

CacheMonitorInterceptor will tell you the response comes from cache or network.
e.g.

D/CacheMonitorInterceptor: response: Response{protocol=http/1.1, code=200, message=OK, url=https://devimages.apple.com.edgekey.net/streaming/examples/bipbop_4x3/gear4/prog_index.m3u8}
D/CacheMonitorInterceptor: response cache: Response{protocol=http/1.1, code=200, message=OK, url=https://devimages.apple.com.edgekey.net/streaming/examples/bipbop_4x3/gear4/prog_index.m3u8}
D/CacheMonitorInterceptor: response network: null

[laminsk]

@b95505017 Thanks for helping me out, much appreciated. I've just tested it again as you have suggested but still not working. I've commented out this code in PlayerActivity and remove the comments to force cache as per your advice.

private final CacheControl cacheControl = new CacheControl.Builder().maxStale(365, TimeUnit.DAYS).build();

//private final CacheControl cacheControl = CacheControl.FORCE_NETWORK;

This is the response I get from CacheMonitorInterceptor. Any ideas?

02-05 17:16:23.193 7019-7880/com.google.android.exoplayer.demo D/CacheMonitorInterceptor: response: Response{protocol=http/1.1, code=206, message=Partial Content, url=http://thesixteendigital.com.s3.amazonaws.com/testfiles/Hallelujah.mp3} 02-05 17:16:23.193 7019-7880/com.google.android.exoplayer.demo D/CacheMonitorInterceptor: response cache: null 02-05 17:16:23.193 7019-7880/com.google.android.exoplayer.demo D/CacheMonitorInterceptor: response network: Response{protocol=http/1.1, code=206, message=Partial Content, url=http://thesixteendigital.com.s3.amazonaws.com/testfiles/Hallelujah.mp3}

[b95505017]

If you use official samples provided by ExoPlayer, e.g. HLS, will the cache work?

[laminsk]

Yes, it does. Got lots of cache files under "cache" directory. Thanks

[b95505017]

I'm not familiar with mp3, not sure why it always request two times.
The problem may be audio stream using byte-range partial downloading files, so OkHttp doesn't fully download the whole file.

[vvincirey]

@b95505017 i tried all the above steps for caching gfycat videos, to play in loop, but it is not working.
I used your demo app, removed the comments for cache control. Still not working. Can you help me with that.

[mariuspena]

Hi.
I set up the latest version of Exoplayer (r1.5.5) and I made it work with OkHttpClient.

Cache cache = new Cache(context.getCacheDir(), 1024*1024*100);
        CacheControl cacheControl = new CacheControl.Builder()
                .maxStale(22, TimeUnit.DAYS)
                .maxAge(22, TimeUnit.DAYS)
                .build(); 
OkHttpClient.Builder builder = new OkHttpClient.Builder()
                .cache(cache)
                .connectTimeout(DefaultHttpDataSource.DEFAULT_CONNECT_TIMEOUT_MILLIS, TimeUnit.MILLISECONDS)
                .readTimeout(DefaultHttpDataSource.DEFAULT_READ_TIMEOUT_MILLIS, TimeUnit.MILLISECONDS)
                .followSslRedirects(false);
builder.addInterceptor(new OverrideCacheControlHeaderInterceptor(cacheControl));

        sOkHttpClient = builder.build();

The header interceptor looks like this (although the server sends Cache-Control: max-age=xxxx)
`public static class OverrideCacheControlHeaderInterceptor implements Interceptor {

    private final String cacheControlValue;

    public OverrideCacheControlHeaderInterceptor(CacheControl cacheControl) {
        cacheControlValue = cacheControl.toString();
    }

    @Override
    public Response intercept(Chain chain) throws IOException {
        Response response = chain.proceed(chain.request());

        return response.newBuilder()
                .header("Cache-Control", cacheControlValue)
                .build();
    }
}`

then in ExtractorRendererBuilder:
DataSource dataSource = new DefaultUriDataSource(context, null, new OkHttpDataSource(getClient(context), userAgent, null, null/*, CacheControl.FORCE_CACHE*/));

The url is lets say this: http://html5demos.com/assets/dizzy.mp4 - so it is an mp4 (used in another thread here https://github.com/google/ExoPlayer/issues/441)
It does not cache anything, no files are written to the disk..

does anyone think this is the right path to cache the videos ?

[brian-chu-twc]

@ceetah If you set a breakpoint in your intercept code, then you will know what happened. You may want to use addNetworkInterceptor instead of addInterceptor

[b95505017]

@ceetah Why not set CacheControl via OkHttpDataSource directly instead of Interceptor?

CacheControl cacheControl = new CacheControl.Builder()
                .maxStale(22, TimeUnit.DAYS)
                .maxAge(22, TimeUnit.DAYS)
                .build(); 
DataSource dataSource = new DefaultUriDataSource(context, null, new OkHttpDataSource(getClient(context), userAgent, null, cacheControl));

[mariuspena]

Hey @brian-chu-twc thanks for pointing that out. But I still do not know what I should look for. It still downloads each time

[b95505017]

Checkout what first post mentioned about
https://github.com/google/ExoPlayer/issues/420#issue-71420600

This is because the cache currently has no concept of the length of a piece of data. If we have a file in the cache with byte range [0-1000], and make a request for range [0-*], we currently don't know what to do after we've return the first 1000 bytes from the cache. We don't want to make an unconditional request to the network starting at offset=1000 because it's inefficient, wont work in the offline case, and because the server may return a 416 (unsatisfiable byte-range) in the case that the content really is 1000 bytes long, which would be awkward to handle.

[mariuspena]

@b95505017 thanks. I set cache control in the OkHttpDataSource.
Regarding you last comment, what do you think I should do in my case? where I have an mp4 (so unbounded length right?) What are my options to cache them ?

[b95505017]

No idea right now, I think that's why the issue is still open.

[mariuspena]

hi again.
I managed to get the cache working with OkHttpDataSource and OkHttp library and its cache implementation (too bad Cache is a final class). As far as the 206 partial responses I was getting from the server (this prevented the file to be written to disk by the cache) I am disregarding them with the use of an interceptor. I don't really know if this is the right approach.. but it seems to be working in my case.
Can't wait to see a proper solution for caching mp4 files (even those sent by a server that uses partial responses) !

[oubeichen]

@ceetah could you please post your implementation code here? Thank you.

[kinsleykajiva]

hi ,on caching i Exoplayer i hav just thot of using android Video Cache by danukula a library for caching found this solution on Android Arsenal (https://github.com/danikula/AndroidVideoCache). iam yet to test the code but i hope this helps so i had to override the Renders in my exoplayer like this:

String newuri=uri.toString();
HttpProxyCacheServer proxy = App.getProxy(context);
proxy.registerCacheListener(this, newuri);
// HttpProxyCacheServer proxy = getProxy();
uri = Uri.parse(proxy.getProxyUrl(newuri));
or you can use this file to compare or read fully what i was doing in the file Attachment-
i have just uploaded the one file for you to see my implementation

[dan-dr]

@ceetah how did you disregard it?

[edisonw]

@ojw28 is this still on the roadmap or the decision to make the file system healthy has already made this removed from roadmap.

[dadouf]

@ceetah Having the same kind of problem where the data apparently doesn't get written to the cache. Could you give us indications on how you dealt with the server's response? Cheers.

[gbhall]

@ceetah I'd like to know too. Thanks :)

[giangttpham]

b95505017 commented on Feb 5

If you use official samples provided by ExoPlayer, e.g. HLS, will the cache work

Can someone please point me to where that official samples are? thank you.

[b95505017]

@giangttpham You could checkout my branch, it works for HLS cache but mp3.
https://github.com/b95505017/ExoPlayer/tree/okhttp_http_data_source

[laminsk]

Confirmed. Doesn't work for mp3 files unfortunately.

On Fri, Apr 29, 2016 at 3:47 AM, Yu-Hsuan Lin [email protected]
wrote:

@giangttpham https://github.com/giangttpham You could checkout my
branch, it works for HLS cache but mp3.
https://github.com/b95505017/ExoPlayer/tree/okhttp_http_data_source

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/google/ExoPlayer/issues/420#issuecomment-215614860

[giangttpham]

@b95505017 Hi, Thank you so much for the quick response. I followed your example but for DASH instead of HLS, and it's caching the data. The problem is, when I did a seek backward, instead of using the cached data, it went and downloaded the stream again, that's not what we want right?

[b95505017]

@giangttpham That's what the issue is. You could check whether the byte range of the request is unbounded, if so, then that's the same reason for mp3 not being cached.
Checkout what first post mentioned about
https://github.com/google/ExoPlayer/issues/420#issue-71420600

This is because the cache currently has no concept of the length of a piece of data. If we have a file in the cache with byte range [0-1000], and make a request for range [0-*], we currently don't know what to do after we've return the first 1000 bytes from the cache. We don't want to make an unconditional request to the network starting at offset=1000 because it's inefficient, wont work in the offline case, and because the server may return a 416 (unsatisfiable byte-range) in the case that the content really is 1000 bytes long, which would be awkward to handle.

[giangttpham]

@b95505017 ahhhh, I got it now. Thank you.

[Bhavin20290]

Hello
I am play mp4 video using exoplayer ExtractorRendererBuilder. that works fine.
But, when i am play video offline using okhttp , its not work.
i am stuck for that part for last week.
please any one help me out?
any sample or else?

Thanks :)

[eneim]

Hm, why do you need okhttp to play it offline?

[Bhavin20290]

any other option?

[Bhavin20290]

i play video from server like live streaming .
So, first timw video play online and after that buffer store in cache file
and video play offline from cache

So, for this any other option?

[kinsleykajiva]

@Bhavin20290 hey how are you able to play or stream okhttp ...can you share your source code project please because i have been trying to do that especially with mp4 file streams.

[Bhavin20290]

Ya ofcourse i have shared my code
But the problem is cache didn't work

[Bhavin20290]

@eneim Can u give me solution?

[eneim]

@Bhavin20290 not really, I just want to know why you play offline file by an http client. Turns out you use 'same Renderer' for your file.

[Bhavin20290]

Ya i understand your question, that is my mistake.
I have want to solution for this that how i pay video from cache file using exoplayer ExtractorRendererBuilder?

[Bhavin20290]

@eneim
Sorry for bothering mate , But i was stuck for last week and project was delay.
i have liberate once again , i have created exoplayer to play 160MB mp4 video and play streaming. now client requirement was video streaming one time after that video store from cache file and play from it. that was offline.

So my problem was play video that's fine , but didn't create cache for that and don't play video from that.
exactly i was create cache file and store video from it while streaming video.

Thanks
and please any solution or any other way to solve this , give me steps.

[Bhavin20290]

Hello
I import exoplayer okhttp_http_datasource project and caching video.
But, i didn't create any cache file.
i get D/CacheMonitorInterceptor: response cache: null

Please Help me out

[Bhavin20290]

42footPaul

Hello Mate,
I show your question mate, i have same problem
I streaming video and meanwhile video store in cache
using exoplayer okhtto_htttp_docs not sloved this

If u got your answer , than please share with me.
I am stuck on that part

[yhd4711499]

Hello

Here is my approach: https://github.com/yhd4711499/ExoPlayer/commit/fda3e6f1a52b2e4bace591e967a0b58656e96f31

It uses a ContentLengthCache to store Content-Length fetched by actual DataSource of a given uri.

[qqli007]

@ceetah ,how to disregard 206 partial responses?

[qqli007]

caching seems to work on dev-v2, used like this:

  private DataSource.Factory buildDataSourceFactory(final boolean useBandwidthMeter) {
//    return new DefaultDataSourceFactory(this, useBandwidthMeter ? BANDWIDTH_METER : null,
//        buildHttpDataSourceFactory(useBandwidthMeter));
    return new DataSource.Factory() {
      @Override
      public DataSource createDataSource() {
        LeastRecentlyUsedCacheEvictor evictor = new LeastRecentlyUsedCacheEvictor(100 * 1024 * 1024);
        SimpleCache simpleCache = new SimpleCache(new File(getCacheDir(), "media_cache"), evictor);
        return new CacheDataSource(simpleCache, buildMyDataSourceFactory(useBandwidthMeter).createDataSource(),
                CacheDataSource.FLAG_BLOCK_ON_CACHE | CacheDataSource.FLAG_CACHE_UNBOUNDED_REQUESTS,
                10 * 1024 * 1024);
      }
    };
  }

  private DefaultDataSource.Factory buildMyDataSourceFactory(boolean useBandwidthMeter) {
    return new DefaultDataSourceFactory(PlayerActivity.this, userAgent, useBandwidthMeter ? BANDWIDTH_METER : null);
  }

It works for playing an mp4 file from network even though it is unbounded request.

[zhangvb]

@laminsk Partial Request is used in OkHttpDataSource. It seems like OkHttp do not support partial cache. I remove Range header in request, cache worked.
`private Request makeRequest(DataSpec dataSpec, boolean forceCache) {
long position = dataSpec.position;
long length = dataSpec.length;
boolean allowGzip = (dataSpec.flags & DataSpec.FLAG_ALLOW_GZIP) != 0;

HttpUrl url = HttpUrl.parse(dataSpec.uri.toString());
Request.Builder builder = new Request.Builder().url(url);
if (forceCache){
  builder.cacheControl(CacheControl.FORCE_CACHE);
} else {
  builder.cacheControl(CacheControl.FORCE_NETWORK);
}

synchronized (requestProperties) {
  for (Map.Entry<String, String> property : requestProperties.entrySet()) {
    builder.addHeader(property.getKey(), property.getValue());
  }
}

// if (!(position == 0 && length == C.LENGTH_UNBOUNDED)) {
// String rangeRequest = "bytes=" + position + "-";
// if (length != C.LENGTH_UNBOUNDED) {
// rangeRequest += (position + length - 1);
// }
// Log.d("SOGOU_CAR_MUSIC", "Range " + rangeRequest);
// builder.addHeader("Range", rangeRequest);
// } else {
// Log.d("SOGOU_CAR_MUSIC", "Not a range request");
// }
builder.addHeader("User-Agent", userAgent);
if (!allowGzip) {
builder.addHeader("Accept-Encoding", "identity");
}
if (dataSpec.postBody != null) {
builder.post(RequestBody.create(null, dataSpec.postBody));
}
return builder.build();
}`

Header also can be removed in a Interceptor.
But partical request is very useful for a large online resource. I don't know is there some http tools support cache partical content.

About cache partial content, please refer to https://square.github.io/okhttp/3.x/okhttp/okhttp3/Cache.html

[kinsleykajiva]

@zhangvb the link you gave is dead.

[ojw28]

@kinsleykajiva the link is fine if you copy and paste the URL as text (it's just that it's linked incorrectly).
@qqli007 is correct about dev-v2. We recently pushed support for caching for regular media files, provided you specify FLAG_CACHE_UNBOUNDED_REQUESTS as in his sample code.

[calulelalu]

@qqli007 great example... caching seems to work but now seeking backward or forward is not working any more. I'm working with mp3 files.

[zhangvb]

Appending my code, it worked for me. If play without seeking cache works.

Interceptor interceptor = new Interceptor() {
                @Override
                public Response intercept(Chain chain) throws IOException {

                    Request request = chain.request();
                    request = request.newBuilder()
                            .cacheControl(CacheControl.FORCE_CACHE)
                            .build();

                    Response response = chain.proceed(request);

                    if (!response.isSuccessful()) {
                        Request.Builder requestBuilder = request.newBuilder();
                        String range = request.header("Range");
                        LogUtil.logD("Range " + range);
                        // Parse Range, if Range is "dddd-" format, remove Range to active OkHttp Cache.
                        if (!TextUtils.isEmpty(range) && range.startsWith("bytes=")) {
                            range = range.replace("bytes=", "");
                            String[] ranges = range.split(",");
                            LogUtil.logD("ranges length " + ranges.length);
                            if (ranges.length == 1) {
                                String range0 = ranges[0].trim();
                                if ((!range0.startsWith("-")) && range0.endsWith("-")) {
                                    String startString = range0.substring(0, range0.indexOf("-"));
                                    int start = -1;
                                    try {
                                        start = Integer.valueOf(startString);
                                    } catch (Throwable t) {
                                    }
                                    LogUtil.logD("Range start " + start);

                                    // if start is too large, maybe the user is seeking, keep Range.
                                    if (start > 0 && start < 1024 * 1024) {
                                        requestBuilder.removeHeader("Range");
                                    }
                                }
                            }
                        }

                        request = requestBuilder
                                .cacheControl(CacheControl.FORCE_NETWORK)
                                .build();
                        response = chain.proceed(request);
                    } else {
                        LogUtil.logD("Cache request success.");
                    }

                    CacheControl localCacheControl = new CacheControl.Builder()
                            .maxStale(Integer.MAX_VALUE, TimeUnit.DAYS)
                            .maxAge(Integer.MAX_VALUE, TimeUnit.DAYS)
                            .build();

                    return response
                            .newBuilder()
                            .header("Cache-Control", localCacheControl.toString())
                            .removeHeader("Pragma")
                            .build();
                }
            };

            File cacheFile = new File(cacheParentFile, "OkHttpCache");
            okhttp3.Cache cache = new Cache(cacheFile, 200 * 1024 * 1024);
            OK_HTTP_CLIENT = new OkHttpClient.Builder()
                    .addInterceptor(interceptor)
                    .cache(cache)
                    .build();

[Anshumansharma12]

Hello by using dev-v2 am able to create local cache of stream video. Now its showing multiple files in following format "xxxxx.v2.exo" is there any way by which i can create .mp4 file.
Or any other way by which i can cache it into ".mp4" format. As am streaming mp4 video http url.

[erdemguven]

@Anshumansharma12 it isn't possible but you can always download the file yourself and play it using a FileDataSource.

[danrossi]

Can somebody please explain how the CacheDataSource is supposed to be setup ?

All I can see is code like this but no idea how to integrate back into the demo application.

ie

 LeastRecentlyUsedCacheEvictor evictor = new LeastRecentlyUsedCacheEvictor(100 * 1024 * 1024);
        SimpleCache simpleCache = new SimpleCache(new File(getCacheDir(), "media_cache"), evictor);
        DefaultDataSource upstream = defaultMediaDataSourceFactory.createDataSource();
        return new CacheDataSource(simpleCache, upstream, false, false);

The demo application has code like this, so where does CacheDataSource go here ?

case C.TYPE_DASH:
        return new DashMediaSource(uri, buildDataSourceFactory(false),
            new DefaultDashChunkSource.Factory(mediaDataSourceFactory), mainHandler, eventLogger);

I'm hoping this will work as an offline cache ? Why is it not integrated into it already ? offline playback seems to be a common request.

[danrossi]

I've got this but I can't seem to debug if it's working. There is zero documentation and I've resorted to dev-v2-id3 as it seems the most up to date with all the refactoring.

If I start playing a dash source. Turn off remote connection and try and replay it fails to load the mpd file from the cache.

DataSource.Factory buildDataSourceFactory(boolean cache, final DefaultBandwidthMeter bandwidthMeter, final CacheDataSource.EventListener listener) {

    if (!cache) {
      return new DefaultDataSourceFactory(this, bandwidthMeter,
              buildHttpDataSourceFactory(bandwidthMeter));
    }

    return new DataSource.Factory() {
      @Override
      public DataSource createDataSource() {
        LeastRecentlyUsedCacheEvictor evictor = new LeastRecentlyUsedCacheEvictor(100 * 1024 * 1024);
        SimpleCache simpleCache = new SimpleCache(new File(getCacheDir(), "media_cache"), evictor);


        return new CacheDataSource(simpleCache, buildCachedHttpDataSourceFactory(bandwidthMeter).createDataSource(),
                new FileDataSource(), new CacheDataSink(simpleCache, 10 * 1024 * 1024),
                CacheDataSource.FLAG_BLOCK_ON_CACHE | CacheDataSource.FLAG_IGNORE_CACHE_ON_ERROR, listener);
      }
    };
  }


  private DefaultDataSource.Factory buildCachedHttpDataSourceFactory(DefaultBandwidthMeter bandwidthMeter) {
    return new DefaultDataSourceFactory(this, bandwidthMeter, buildHttpDataSourceFactory(bandwidthMeter));
  }

  HttpDataSource.Factory buildHttpDataSourceFactory(DefaultBandwidthMeter bandwidthMeter) {
    return new DefaultHttpDataSourceFactory(userAgent, bandwidthMeter);
  }

The dash source stays the same

return new DashMediaSource(uri, buildDataSourceFactory(true, false),
            new DefaultDashChunkSource.Factory(mediaDataSourceFactory), mainHandler, eventLogger);

The event logger implements this event method but nothing.

  @Override
  public void onCachedBytesRead(long cacheSizeBytes, long cachedBytesRead) {
    Log.d(TAG, "onCachedBytesRead[cachedBytes:" + cacheSizeBytes + ", cachedBytesRead: " + cachedBytesRead + "]");
  }

[danrossi]

Confirming this to be working. It is causing audio buffer underruns when playing from the cache so dropping frames. Turning off connection the fragments can play back from the cache but the main manifest still wants a connection and is not being stored into the cache. Perhaps this solution can be used for providing an offline downloader without having to re-engineer one. Just run the caching in the background ?

[danrossi]

I have both caching and persistent widevine dash working. However it's not caching the Mpd yet perhaps there is still a setup problem there ?

[danrossi]

The cache gets messed up with adaptive switching. Still can't figure out why its not caching the Mpd file. It is possible not setting the key correctly for the Dash segments.

Need to now figure out how to turn Adaptive off and choose the largest bitrate while trying to build a cached offline file.

There has been zero proper documentation, even code commenting for any of it sadly. As I mentioned FairPlay has a downloader and offline playback feature for AVPlayer out of the box with the current SDK. It would be nice to match it.

[danrossi]

This code is so convoluted. I was able to track down why the mpd is not being cached. It's because of a content length being -1. I can't work out right now where or why it's returning -1. Really hard work.

You have other options than to use the cache directory I believe like these. One will clear content when the app is removed, the other will not.

File storageDir = new File(getExternalFilesDir(Environment.DIRECTORY_MOVIES), cacheDir);

File storageDir = new File(Environment.getExternalStoragePublicDirectory(Environment.DIRECTORY_MOVIES), "media_cache");

[danrossi]

As far as I can see the manifest loader runs through a parser. No content length is given before it's opened. It will fail to store in the cache because no content length is given.

[danrossi]

https://github.com/google/ExoPlayer/blob/dev-v2/library/src/main/java/com/google/android/exoplayer2/upstream/ParsingLoadable.java#L75

This provides no length therefore the issue. the length can't be set after reading bytes. Requires some hacking elsewhere to allow writing the cache without checking for length possibly.

[ojw28]

@erdemguven - Do we cache data if even the resolved length is unknown (this is likely a result of the mpd response being gzip'd) and/or have a solution to this?

@danrossi - I think it would be fairer to call the code necessarily complex ;). There are many small nuances to caching media properly, particularly when considering things like caching of non-completed requests. As an aside, you probably shouldn't be using the caching components as-is to build a proper DASH offlining solution. A proper DASH offlining solution would more than likely involve explicitly downloading only a single representation (and not the manifest, although you might optionally recreate an offline manifest containing only the single representation downloaded). The caching components can be used as part of an implementation for this, but the actual downloading bit should probably not be implemented using a player instance.

[erdemguven]

@ojw28 @danrossi - Unfortunately not. We skip the cache if the resolved length is unknown as the content is likely to be a live stream. I don't know how we can handle gzip'd responses.

[danrossi]

@ojw28 ok. I was hoping to use this then play the locally stored sources ?

I have the custom offline widevine manager working and will load the keys. I could be interested to upload that to github.

The last past of course was hoping to switch to the largest bitrate and turn abr off and just let the internals do the downloading without having to re-engineer anything. I couldn't even find documentation how to do that sadly.

I wouldn't know where to start to. Im assuming the entire DashDataSource needs to be duplicated and modified to use a filedatasource for playback like the cache system does.

[danrossi]

I thought the latest changes were intended to deal with lack of content length issues. It still won't cache the mpd file sadly.

It seems this system is not designed for and not adequate for offline storage purposes, and an offline concurrent downloader feature needs to be built on top of the dash parser feature possibly. Then another system like the cachedatasource to play back those local files. None of this readily available.

[danrossi]

Is there an internal feature like the cachedatasource, to download parsed dash manifests and fragments then use the FileDataSource to play it back ?

It would be easier if it just concurrently downloaded everything into the cache path and CacheDataSource played them back ?

[erdemguven]

@danrossi We're working on full DASH offlining solution but there isn't anything for downloading yet. I don't know when it'll be ready.

[matclayton]

@erdemguven Any idea when this is likely to land? We need this functionality so might end up building it in the mean time, but it would be fantastic to not have to!

[erdemguven]

@matclayton if you mean the DASH downloader, basic functionality should be in github in 2 weeks time.

[matclayton]

Yes I did, fantastic!

[bnussey]

Hey @erdemguven was the DASH downloader released? Can't seem to find it.

[erdemguven]

@bnussey we decided to release it once the downloader service and manager are ready as major api changes might be necessary. It's hard to give an estimate now. You can follow it on #2643.