Exoplayer: Change playback rate (video + audio only use cases)

[viacheslavokolitiy]

Hello guys. Is it possible to change playback rate of video track e.g play faster or play slow ?

[runningcode]

+1 this would be very useful!

[ojw28]

To clarify, do you mean with audio (sped up but corrected to still be at the correct pitch) or without?

[viacheslavokolitiy]

@ojw28 with audio

[ojw28]

This isn't something that's supported currently. It is something you could potentially do yourself, either by extending MediaCodecAudioTrackRenderer, or by creating your own audio renderer to use instead of the one provided by the library.

Basically, you'd need to do some adjustment in your extension to get the audio playing at the desired rate (and reporting time as progressing at that rate). This may involve pitch correction. The video track will follow the speed of the audio track (provided it can decode fast enough).

[lsjwzh]

+1

[shukob]

+1

[vishwa-vyom]

+1

[emakovsky]

+1

[shukob]

I need this functionality for my current project and used libSonic to change audio speed as in Prestissimo's Track.java.

Changing reported time part requires multiplying result of framesToDurationUs and durationUsToFrame, and conditions regarding MIN_PLAYHEAD_OFFSET_SAMPLE_INTERVAL_US and MIN_TIMESTAMP_SAMPLE_INTERVAL_US by current speed on MediaCodecAudioTrackRenderer.

I think I should multiply other Usec numbers in both renderers to work without flaw.
As I had to comment out dropOutputBuffer section in processOutputBuffer on MediaCodecVideoTrackRenderer to correctly refresh video frame.

Also
" // TODO: Don't bother doing this once [redacted] is fixed.
// Work out how many bytes we can write without the risk of blocking."
part in processOutputBuffer may need some edit when actually written byte length differs from bytesToWrite because of changing pitch?

[evanhalley]

@skonb Can you provide more details. I'm attempting to incorporate variable speed playback in MediaCodecAudioTrackRenderer (subclassed it, overriding the processOutputBuffer function) . I've done some of it (passing the byte buffer into and out of sonic), but even as a speed and pitch of 1.0f, it's really static-ky. Thanks!

[shukob]

@emuneee I have uploaded current implementation here. That is based on Sep 8 version of ExoPlayer so may not work with the latest one.
It contains bug that getCurrentPositionUs returns larger/smaller value than expected if speed is up/down.

[kboyarshinov]

+1

[kboyarshinov]

@skonb Have you tried to update your implementation to latest ExoPlayer version? Thanks!

[shukob]

@kboyarshinov No. I am now working on the latest ExoPlayer, but just saw it is almost not working on 5.0. So will check on 4.x and try to port above implementation if it reliably works on them in a few days. Thanks.

[mikumi]

@skonb In my project we use a similar implementation with libsonic based on August 15 version of ExoPlayer. Currently I am trying to port it to the newest ExoPlayer release as the older version has problems with webm/ogg files. But no luck so far. Have you made progress porting yours?

[shukob]

@mikumi
I'm now working on it. It seems that at least on 4.2.2, above implementation based on Sep 8 works, but on 5.0.1, audios do not slows down. I tried to port it within exoplayer.audio.AudioTrack to preserve new AudioTrackRenderer structure without success..., I have not investigated about whether Android or ExoPlayer that has problems on 5.x and if new AudioTrack-AudioTrackRenderer implementation is related to playing webm/ogg files.

Regardless of Exoplayer version, if playback speed changes while playing, getPlaybackPositionUs must be completely rewritten to accumulate played frame counts on above implementation.

[mikumi]

@skonb Thanks for the update. I realized that the webm/ogg issues happened because of another modification we had in our project, it was unrelated to ExoPlayer version.
I have ported our previous modification to the new ExoPlayer version by mainly changing the AudioTrack and it's working, but it does not support changing speed while playing because as you said, we would have to rewrite how to calculate the playing position.
I am posting the link here but don't really recommend to use it like that, as it is modified to just suit our needs:
https://github.com/Invisibi/ExoPlayer/pull/1

[shukob]

@mikumi Thanks for the info! I'll check it : )

[PaulWoitaschek]

Are there any news on this? I tried the last implementation, but it gave really bad sound. I think this is somethig that would also be great to have ExoPlayer implement it natively.

[Ood-Tsen]

We may add a new TrackRenderer as new time source and disable audioTrackRenderer by overwrite the function isTimerSource.

boolean isTimerSource() {
}

And change the play rate by overwrite function "getCurrentPositionUs"

long getCurrentPositionUs(){
}

[PaulWoitaschek]

Can you do that? Is that a big task? Im the developer of Material Audiobook Player and this whole audio-internal stuff goes way beyond my capabilities ;-)

[crossle]

+1

[Ood-Tsen]

@Ph1b This concept is for video playback, video renderer will sync video frame to the clock automatically.

But for audio is not that simple.
We need to resampling audio raw data.

[kkovach]

+1 Would love to see this as an official feature.

[jacoviza]

+1

[danikula]

+1

[Arqu]

+1

[kevindaniels]

+1

[andreimarinescu]

+1. A typical usecase for us is ExoPlayer running on AndroidTV or Kindle FireTV (you can increase the playback rate from the remote control as opposed to just seeking forward a random amount of seconds)

[crossle]

Android M preview new MediaPlayer.setPlaybackParams() method to set the media playback rate for fast or slow motion playback. It also stretches or speeds up the audio playback automatically in conjunction with the video.

[shukob]

@crossle thanks for the info!

[PaulWoitaschek]

Any news on someone being able to implement variable playback speed? Or is it possible to backport the MediaPlayer?

[jpshelley]

+1 @Ood-Tsen any updates? Would love to submit a PR but have no idea where to start.

[wclr]

It's a shame, but currently android native audio components (MediaPlayer, AudioTrack, SoundPool) can hardly support playback rate change (each lacks something). MediaPlayer doesn't have such capabilities at all, AudioTrack doesn't have seekTo, soundPool is for short audio samples.

[KalinKanev]

+1

[finneapps]

+1

[bsysop]

+1

[snard6]

+1

[brianchu]

+1

[ps-tfriedl]

+1

[c3stream]

+1

[MantasMiksys]

+1

[benjaminbarbe]

+1

[wclr]

http://developer.android.com/reference/android/media/MediaPlayer.html
There is setPlaybackRate method in Android SDK docs

Does it work? Has anyone checked/ From wich android version?

[ojw28]

There are a bunch of new APIs in M (API level 23) that provide underlying platform support for changing of playback rate. For ExoPlayer, the relevant ones would be:

http://developer.android.com/reference/android/media/AudioTrack.html#setPlaybackParams(android.media.PlaybackParams)
http://developer.android.com/reference/android/media/PlaybackParams.html

For MediaPlayer, the relevant method is:

http://developer.android.com/reference/android/media/MediaPlayer.html#setPlaybackParams(android.media.PlaybackParams)

I don't think there is a setPlaybackRate method. I think the inclusion of that method in the valid/invalid states table is an issue with the documentation being inaccurate.

[jpshelley]

@ojw28 With the methods you described above do you think adjusting the playback rate is now feasible with API >= 23? What would be the approach to do so?

[eshkoliGilad]

Any news regarding this issue ?

It would have been awesome for this kind of feature.

[jpshelley]

There has been an issue tracked in regards to setSpeed method in PlaybackParams https://code.google.com/p/android/issues/detail?id=191368&q=params&colspec=ID%20Type%20Status%20Owner%20Summary%20Stars
Assuming it is able to work for MediaPlayer it should be able to work for Exo.

[Folyd]

+1
Look forward this feature!
Android MediaPlayer api is unfriendly! :sob:

[richk]

On Android M and above, native media player supports playback params with speed being one of the them. I have been able to successfully speed up audio (with video) by making very small changes to the AudioTrack and extending the MediaCodecAudioTrackRenderer as mentioned above.

http://developer.android.com/reference/android/media/AudioTrack.html#setPlaybackParams(android.media.PlaybackParams)
http://developer.android.com/reference/android/media/PlaybackParams.html

But this involves bumping up the target SDK version to API 23. If thats ok, I can submit a pull request.

[ojw28]

Hi! As it happens, we have a change that does exactly this on our side too. We'll be pushing it to the dev branch this week. The dev branch is already API 23, by the way :). Thanks!

[jpshelley]

Awesome work @ojw28 and @richk ! I tried out the variable speeds with a MediaPlayer and noticed it was only possible to move forwards. Is there any information on reverse support as well? (Little off topic for this thread but didn't think it warranted another github issue)

[richk]

Thats awesome @ojw28 Will be great if it can pushed soon and ready to be used in an RC version.

[richk]

@ojw28 On that note, is there anything planned to support video speedup on versions below M as well?

[ojw28]

• Reverse playback is a different and more complicated problem. There are many implications of reversing the playback direction, such as the need to buffer backward rather than forward. For video, the way that most video is encoded makes reverse playback massively more expensive. So no, we have no plans to support reverse playback in any form.
• We have no plans to support pre-M playback rate changes at this time.

[richk]

@ojw28 Is there a good way to do that on pre-M with exoplayer? "setPlaybackRate" - http://developer.android.com/reference/android/media/AudioTrack.html#setPlaybackRate(int) seems like the only API in pre-M versions to control playback but that does not take care of audio pitch.

[ojw28]

You would have to implement speed/pitch adjustment yourself on the audio data before feeding it to the AudioTrack. Which is doable, but non-trivial.

[richk]

Thanks @ojw28

[richk]

@ojw28 Any dates on when is the next exoplayer RC planned and whether it will have the video speedup on M and above?

[ojw28]

Tweaked in https://github.com/google/ExoPlayer/commit/fb7ddb722c7d377d54817caebb53d014d679b283

[saberpowermo]

+1

[einmalfel]

+1
@ojw28 @andrewlewis Any chances to see this feature on pre-M target API?

[ojw28]

Not as part of ExoPlayer, no. At least, not for the foreseeable future.

[richk]

Will be great to know if 1.5.3 will be releasing this year. Thanks!

[PaulWoitaschek]

I really don't understand all the audio processing, but the project titles says

An extensible media player for Android

Why is it so complicated to integrate a library like sonic if this is focused on being extensible?

[ojw28]

Application developers who are interested in doing so are free to perform such an integration (it can be done in application code; there's no need to modify the core library). It's not the goal of this project to implement all extensions that might be useful ourselves.

[PaulWoitaschek]

At the moment with audio track it is done like this in a custom solution.

Can you point me / us in a direction how to integrate sonic in a similar way with ExoPlayer?

[ojw28]

I've seen a solution that extends MediaCodecAudioTrackRenderer to implement this functionality. The idea was to override processOutputBuffer. In that method a transformation was applied to the audio held in the buffer (this is where a library hook would be called), and the transformed buffer was passed to super.processOutputBuffer to cause the transformed audio to play. It was a little more involved than that in practice, but that was the basic idea.

[richk]

Is there a reason why the solutions implemented or suggested above were not accepted or contributed back into Exoplayer? For pre-M, unless the speedup solution is part of exoplayer, there is always the danger of later updates breaking them.

[ojw28]

It's not the case that if it were part of ExoPlayer it would become zero cost to maintain and keep in line with future updates. Moving it into ExoPlayer shifts the responsibility and maintenance burden from an application developer who wants the feature to us, but someone still has to do it. If someone wants this feature badly enough, it's possible for them to implement it themselves with a non-invasive extension. If someone wants to do that and open source their solution for others, they are free to do that too.

[richk]

Understood. From developers' point of view though, it can get a little frustrating if Exo player releases changes that may break their speedup solution. Is there a way to tag an acceptable solution and be mindful of it in future releases? Also it looks like a lot of application developers do want this feature.

[PaulWoitaschek]

So my plan was to modify the bytes form the ByteBuffer and just pass them to super in processOutputBuffer. Mut I was not able to modify the ByteBuffer. Any ideas?

[pernilla]

Well, the playbackspeed works well in our solution. Thanks for that! But of course it would be interesting to have it pre-M versions since so few android devices out there runs M. And we're also interesting in knowing if 1.5.3 dev will be out before x-mas or new year?

[jedhoffmann]

I got playback speed working using the Sonic Java port along with a MediaCodecAudioTrackRenderer override. As Ph1b mentioned the ByteBuffers are read only, at least > API 21, so you will need to wrap your modified bytes (the output from Sonic) in a ByteBuffer and pass it along to the super class. The other thing to keep in mind is that processOutputBuffer can be called again with the same input buffer if it was not completely consumed by the AudioTrack. You'll need to keep track of the last buffer so you don't run it through Sonic again. Hope that helps.

[PaulWoitaschek]

Can you share your solution? I think you would do many people here a great favor @jedhoffmann

[jedhoffmann]

@Ph1b here is a Gist. Designed for audio only; I don't have any insight into the effect it will have on video. Used with ExoPlayer r1.5.2. One thing I haven't solved is handling time discontinuity. I've tried adjusting BufferInfo.presentationTimeUs but I don't think that approach can work because the discontinuity check is calculated based on the total number of bytes submitted to the AudioTrack which we are modifying with Sonic. It's not actually problematic for audio playback because com.google.android.exoplayer.audio.AudioTrack adjusts for the discontinuity but I'm continually getting an error log.

[PaulWoitaschek]

Thanks! @jedhoffmann May I use it in Material Audiobook Player and can you license it as Apache License 2?

[jedhoffmann]

Unfortunately I can't license it. Please use it only as an idea for how to do your own implementation.

[jedhoffmann]

BTW @pernilla, the new AudioTrack capabilities available in API 23 are using Sonic which has been integrated into the Android source. See the comment here from the author of Sonic.

[PaulWoitaschek]

By the way about the new playback params implementation. Wouldn't a bit more object orientated library fit better to this player?
Stuff like having (String, Object) as method signature and then casting is quite error prone. Maybe just make that an implementation detail and provide a helper method like setVolume(float) and then delegate this to handle message?

Back to topic:
My solution inspired by yours works partly. When seeking a lot in short time it blows up.

E/ExoPlayerImplInternal: Internal track renderer error.
com.google.android.exoplayer.ExoPlaybackException:
    com.google.android.exoplayer.audio.AudioTrack$WriteException: 
        AudioTrack write failed: -2
at com.google.android.exoplayer.MediaCodecAudioTrackRenderer.processOutputBuffer(
    MediaCodecAudioTrackRenderer.java:384)

at

if (bufferIndex == lastBufferIndex) ...

Any ideas on this?
My implemention so far. Uses PlaybackParams on API >= M and regular sonic processing below. That part is explicitly licensed as Apache v2. So feel free to tweak, reuse it and send improvements back ;-)

[eshkoliGilad]

Didn't quite fully understand this, does this feature enabled already in ExoPlayer on the latest version ?
If so, how can I use this to fast-forward a video ?

@ojw28

[hermanbanken]

+1 would really like to have this officially supported by ExoPlayer.

[hermanbanken]

Would audio be more difficult due to pitch correction @ojw28? I only need speed up/down video playback, but I have no clue where to start. Would juggling with time codes in MediaCodecTrackRenderer#processOutputBuffer be the way to go?

Ideally I would like to have full control of the playback speed during the video: at certain timestamps the video should speed up or speed down. How would such a clock renderer work? It sounds promising as well.

[c3stream]

I think that an important function.
Is the education system, such as academic content, and videos are there videos easier to speak slowly.
So the function can adjust the speed and slow playback functions like video playback and audio playback speed is important to users of is.
Once again, Google even discussion.
"iOS" has been implemented.

I think Google gives hopes (force) educational content industry in the future.
Please, consider the implementation.

[uguraykilic]

+1

[Umopepisdn]

+1. Really useful for the educational space

[KirillMakarov]

+1

[Meera94]

+1

[AndriyBas]

+1

[besteed]

+1

[danielto]

+1

[jun-lee]

+1

[tanqu2013]

+1

[Yuhannaa]

+1

[ciromine]

+1. In fact this is a important function that Media Player never supported.

[hezhisu]

@skonb do you have some idea of changing the plackback rate of video track with the latest exoplayer

[yangwuan55]

@all Please see here ,I have a implementation of variable speed for exoplayer,support the preversion of 23.

[heartbeep]

@yangwuan55 What do you mean by preversion of 23? Does your implementation work for API 16 and up?

[yangwuan55]

@0xABC it can work on > api 17 .

[ojw28]

Not "properly" though. It simply isn't as good of a solution as the solution provided for >= 23 in the core library, and would likely be of unacceptable quality for use by the majority of apps. Please don't advertise your solution as being a complete one (until the point at which audio plays smoothly and is pitch corrected).

[yangwuan55]

@ojw28 I think it's not the point.We just need this function.

[kir-medvedev]

Agreed with @ojw28. I've seen some quality solution of the feature, and it is a little bit more complicated in audio rendering part. At least, the patch provided does not take in attention the case when AudioTrack object is also a time source. Timestamp estimation and sanity check code should be updated there too. And that is not mentioning the pitch shift.

[mseroczynski]

Manipulating (audio) playback speed will be one of core features in my app which I'm currently developing. I have no doubt that ExoPlayer 2.x would be the best choice for me if it only had this feature on api 16+. @andrewlewis Would it be possible for you to estimate when such feature (at least audio only) could be potentially released or when will you know if it's coming for sure? My progress is blocked by this issue and I'm considering integrating other solution, but that could be a total waste of time.

Thank you for your time and your awesome input into making media apps stable and reliable.

[c3stream]

I choose this because I want to achieve.

https://github.com/protyposis/MediaPlayer-Extended

[mseroczynski]

@c3stream I think it's not a proper place for such input, most of all we're all interested in specifically Exoplayer's capability of satisfying our business needs. I think we all already know there are alternatives around but perceive Exoplayer as standard.

@ojw28 @andrewlewis If you could answer my question I would be really, really grateful. Deadlines, etc..

[ojw28]

We've scheduled time to work on this in Q4 (i.e. by the end of the year). This is in no way a promise. Priorities may change, etc.

[PaulWoitaschek]

Wow thats really great news @ojw28 I'd really like this project but without speed adjustment I cant use it.

Is Q4 still scheduled?

[roccozanni]

Hi @ojw28 any update on this? Has it been rescheduled or is it still planned for Q4?

Thanks for the great work!

[mseroczynski]

@ojw28 @andrewlewis And one more question - is there any possibility you'd focus on audio sources only first to speed up this feature a bit? I may be wrong but I think this is the main request (despite the first posts question).

[yangwuan55]

I have fix this problem(Including the problem of sound).
Hope useful to you.See here

[PaulWoitaschek]

@yangwuan55
ojw commented your solution with

Not "properly" though. It simply isn't as good of a solution as the solution provided for >= 23 in the core library, and would likely be of unacceptable quality for use by the majority of apps. Please don't advertise your solution as being a complete one (until the point at which audio plays smoothly and is pitch corrected).

Is that fixed?

@ojw28 It would be relly great if you could find the time to update this issue with your schedule.

[anup1245]

Hi @ojw28 @andrewlewis any update on the issue?
Thanks

[CallBackMe]

@ojw28 @andrewlewis any recent updates on this issue ? thx !

[andrewlewis]

Sorry for the delay. This slipped from Q4 but we are working on it and plan to have something ready later this quarter.

Please continue to treat this as a rough estimate rather than a guaranteed deadline and thanks for your patience!

[PaulWoitaschek]

@andrewlewis thanks. Do you focus on audio only (which is the majority of requests) or will this include video?

[andrewlewis]

@PaulWoitaschek The enhancement is to be able to set the playback speed for audio before API 23 (it's already supported on Marshmallow and later, as mentioned in this thread). The audio renderer is used as the media clock, so changing the audio playback speed will have the effect of speeding up or slowing down video as well.

For video-only playback, you might be able implement your own speed adjustment feature by extending the video renderer and making it implement MediaClock. The implementation would be similar to StandaloneMediaClock but using renderer callbacks to start/stop the clock and adding a way to set the clock's speed. I haven't tried this but it should work.

Once the audio work is done we could consider providing a general-purpose 'set playback speed' method for any type of playback (assuming the audio renderer, if any, supports it).

[yangwuan55]

@andrewlewis You can refer to my code.I have already realized.here

[c3stream]

I hope the good solution.

[kdelanerolle]

@yangwuan55 your PR accepted and merged?

[yangwuan55]

@kdelanerolle I've tried,but they don't agree to add this feature.So can only exist in my library.

[ojw28]

Our position on this issue is pretty clear. We're actively working on this but it's not done yet. We had intended to provide support in Q4 2016 but this has now slipped to later in Q1 2017. Please treat this as a rough estimate rather than a guaranteed deadline. If you cannot wait for official support then there are various unofficial approaches referenced in this issue that you can take a look at incorporating into your application code in the meantime. Thanks for your continued patience.

[kdelanerolle]

@ojw28 Appreciate your quick response. It's good to hear that it's not written off totally but unfortunately it's not possible to even plan on it without having a committed timeline.
@yangwuan55 I am looking at your fork and it looks like you are bringing in your SonicMediaCodecAudioTrackRenderer for older (<23) API levels. Can you enlighten me the reason why this is needed?
Also, when I run your demo app, I am seeing exception logs( Ex: "Discontinuity detected [expected 68664458, got 68874666]" )for any speeds other than 1x. Is this something yet to be addressed?

[ojw28]

@kdelanerolle We've indicated it's planned for later in Q1 2017. That is our provisional timeline. It's unclear what "committed" means. Timelines change frequently in many projects, including this one, for example if unforeseen technical issues arise during development or if higher priority issues come to light that need to be assigned resources. We try not to make promises for this reason. If you need maximal certainty then your best option is always to hedge by provisioning your own engineering resources to undertake the work.

[kdelanerolle]

@ojw28 Thanks for the response. I totally get why you are not making promises and timelines can slip. I am really looking forward to seeing this feature on ExoPlayer sooner rather than later. However, I also need to look for any fallback strategy in-case this feature doesn't make it in time (so that my own project doesn't slip it's timeline). So you are right on the money about provisioning my own resources. Having said that, I have to also underline that your efforts are very much appreciated on ExoPlayer and looking forward to using it going forward.

[PaulWoitaschek]

Of cause timelines can slip but it would be great if we can get updates on the schedule rather 2 weeks before than 2 weeks after the due date.

[mseroczynski]

Exactly - to be honest I've never seen such 'wanted' feature on GitHub before. I guess lot of schedules depend on this issue so I'm truly hoping you'd be able to update frequently.

[paulo-coutinho]

+1 for audio and video playback speed

[aliasgarlabs]

Excited for the audio playback speed feature. My project completely depends on it and it's stuck. It's mid Q1 2017 already! Looking forward to it!

[PaulWoitaschek]

It would be great to be notified rather two weeks before than two weeks after the due date about schedule changes.

[henorek]

Please update on this issue - we’re talking about this particular Exoplayer’s feature on our teams daily meetings every few days now, it is really crucial for us even to know if you already see any problems in shipping successfully till the end of March.

[paulo-coutinho]

If anyone need it now for API version 23 or later, it is working on our app and i make a branch using last exoplayer version changing only the speed function based on community code:

https://github.com/prsolucoes/ExoPlayer/tree/release-v2-playback-speed

You can compare to see that nothing was changed instead of player speed:

https://github.com/google/ExoPlayer/compare/release-v2...prsolucoes:release-v2-playback-speed

On older API versions it is working without change playback speed, so you can do now something like us, hide the speed button on API < 23. 60% of our customers have API 23 or later.

Think on it.

[AquilesCanta]

So that you know guys, there are not going to be progress updates for this for at least two weeks. As @ojw28 said above,

If you need maximal certainty then your best option is always to hedge by provisioning your own engineering resources to undertake the work.

This does not mean it's going to be ready in two weeks. It means it's not going to be ready in two weeks.

[mseroczynski]

@prsolucoes Thank you for your suggestion. In my case 70%+ customers would have a serious reason to leave negative feedback on store so it's not acceptable solution.

@AquilesCanta To be honest I don't understand why you mentioned 'two weeks' - I already assumed that if there were any work in progress (that could guarantee release in two weeks) we would already know about it (?). Our intention - as I guess - is to define if you already placed this feature on Q1 roadmap and - just maybe - decided not to announce the fact that it's coming.. You know, some of us may think that You guys are just waiting till <23 API users will be like <30% on Android Dashboard and then just suddenly close this issue. We're just digging for important info here.

[yangwuan55]

@mseroczynski You can see https://github.com/yangwuan55/ExoPlayer/tree/playbackSpeed/demo_speed

[WeiChungChang]

Dear all:

Besides setting speed, I think we should inform the track selection (AdaptiveVideoTrackSelection) also for adaptive streaming case. Otherwise, it may lead to rebuffering by wrongly (over) estimated bit-rate.

You could compare the result below to understand why we should do so.

1. Without the patch, several re-buffering happened:
   https://youtu.be/snDOuQ2yG5E
2. With the patch, NO rebuffering anymore:
   https://youtu.be/xyBz0rHZ5xY

You could check the issue by playbackStatus as:

It is, without informing AdaptiveVideoTrackSelection, we will select the representation set whose bit-rate is the maximum one among all representations having smaller bit-rate than current estimated network bandwidth.

For example, by the 4K test link https://www.youtube.com/watch?v=0wCC3aLXdOw, there are 8 representation sets there. The bit-rates from high to low are:
r1. 22876018
r2. 10577952
r3. 4466109
r4. 2354860
r5. 1164538
r6. 631150
r7. 251705
r8. ...

For my trial here, the network speed is about 4500000 ~ 5500000.
Hence we will select r3 = 4466109.
Also, my HTC X9 device could at most support 4X speedup.
Since the video is played by X4, obviously we will run out of buffering data soon. It is why we could see the rebuffering happened.

Therefore instead of selecting r3, we should select (4500000/4) ~ (5500000/4) = 1125000 ~ 1375000 by taking speed into consideration.
It makes the selection locates from r5~r6.
The test result shows it could avoid the frequent rebuffering issue.

Please check the patch below.
https://github.com/WeiChungChang/ExoPlayer/commit/5f441187588aa9f5b2b16934bd7fce356155c839

Also, I make a test on my fork at:
https://github.com/WeiChungChang/ExoPlayer/tree/trickPlayBack
Please refer to it.

Thanks

[robin2046]

@yangwuan55 I tried your work and it is impressive. However, when I try to play some video clip without audio track, I always get following error:

02-22 15:08:05.833 31708 31747 E ExoPlayerImplInternal: Internal runtime error.
02-22 15:08:05.833 31708 31747 E ExoPlayerImplInternal: java.lang.NullPointerException: Attempt to invoke interface method 'long com.google.android.exoplayer2.util.MediaClock.getPositionUs()' on a null object reference
02-22 15:08:05.833 31708 31747 E ExoPlayerImplInternal: at com.google.android.exoplayer2.ExoPlayerImplInternal.updatePlaybackPositions(ExoPlayerImplInternal.java:437)
02-22 15:08:05.833 31708 31747 E ExoPlayerImplInternal: at com.google.android.exoplayer2.ExoPlayerImplInternal.doSomeWork(ExoPlayerImplInternal.java:473)
02-22 15:08:05.833 31708 31747 E ExoPlayerImplInternal: at com.google.android.exoplayer2.ExoPlayerImplInternal.handleMessage(ExoPlayerImplInternal.java:310)
02-22 15:08:05.833 31708 31747 E ExoPlayerImplInternal: at android.os.Handler.dispatchMessage(Handler.java:98)
02-22 15:08:05.833 31708 31747 E ExoPlayerImplInternal: at android.os.Looper.loop(Looper.java:154)
02-22 15:08:05.833 31708 31747 E ExoPlayerImplInternal: at android.os.HandlerThread.run(HandlerThread.java:61)
02-22 15:08:05.833 31708 31747 E ExoPlayerImplInternal: at com.google.android.exoplayer2.util.PriorityHandlerThread.run(PriorityHandlerThread.java:40)
02-22 15:08:05.835 31708 31708 E EventLogger: playerFailed [11.72]
02-22 15:08:05.835 31708 31708 E EventLogger: com.google.android.exoplayer2.ExoPlaybackException
02-22 15:08:05.835 31708 31708 E EventLogger: at com.google.android.exoplayer2.ExoPlayerImplInternal.handleMessage(ExoPlayerImplInternal.java:360)
02-22 15:08:05.835 31708 31708 E EventLogger: at android.os.Handler.dispatchMessage(Handler.java:98)
02-22 15:08:05.835 31708 31708 E EventLogger: at android.os.Looper.loop(Looper.java:154)
02-22 15:08:05.835 31708 31708 E EventLogger: at android.os.HandlerThread.run(HandlerThread.java:61)
02-22 15:08:05.835 31708 31708 E EventLogger: at com.google.android.exoplayer2.util.PriorityHandlerThread.run(PriorityHandlerThread.java:40)
02-22 15:08:05.835 31708 31708 E EventLogger: Caused by: java.lang.NullPointerException: Attempt to invoke interface method 'long com.google.android.exoplayer2.util.MediaClock.getPositionUs()' on a null object reference
02-22 15:08:05.835 31708 31708 E EventLogger: at com.google.android.exoplayer2.ExoPlayerImplInternal.updatePlaybackPositions(ExoPlayerImplInternal.java:437)
02-22 15:08:05.835 31708 31708 E EventLogger: at com.google.android.exoplayer2.ExoPlayerImplInternal.doSomeWork(ExoPlayerImplInternal.java:473)
02-22 15:08:05.835 31708 31708 E EventLogger: at com.google.android.exoplayer2.ExoPlayerImplInternal.handleMessage(ExoPlayerImplInternal.java:310)
02-22 15:08:05.835 31708 31708 E EventLogger: ... 4 more
02-22 15:08:05.843 31708 31708 D EventLogger: loading [false]
02-22 15:08:05.844 31708 31708 D EventLogger: state [11.73, true, I]
02-22 15:08:06.417 25265 25282 I PlayCommon: [17054] com.google.android.play.a.g.a(1127): Successfully uploaded logs.

Can you fix this issue?
Can you also enable issue track on your forked repository?

[MateusZitelli]

@yangwuan55 I'm using your fork and it worked perfectly in some of my devices. However, I had a problem to play a 2K video on a S6 with 2X playback speed, where the video was freezing. Do you have any idea of the cause or solution for such issue. Thanks!

[yangwuan55]

@robin2046 I didn't play without audio track,I will try it.Is it possible that exoplayer self bug?

[yangwuan55]

@MateusZitelli I don't have so many devices to test.But I've met some low version of the devices didn't work.

[workspace]

@WeiChungChang I tried to run your demo on my old devices(such as Galaxy note 2, Galaxy s4 < API 23 which don't support PlaybackParams), however, any changes were not appeared in speed. Actually, all people commented in this article are interested in changing speed in old devices(not rewind or fast forward things). Do you have any plan for supporting old devices? or these things are unexpected bugs?

[WeiChungChang]

Dear @KiminRyu:

1. What's kind of file you tested upon? The so called 'trick play' depends on the kind of files played much.

2. It should be unrelated to API; at least, for slow speed x2 I demo, it is the same as the work of @yangwuan55 or @prsolucoes.

   _In fact, the most advantage of the scheme to implement trick play by seek is: it does NOT need the_ _support of_ setPlaybackRate _( > API 23)_,

3. There are still some mentions about 'reverse support';
   In fact, I saw the functionality works on MacBook (includes rewind & high speed fast-forward;).
   As a lot of users' prejudice, APPLE provides much better operation convenience than Andorid. To eliminate it, why not give it a try to provide the functionality (or at least, to approach it) if we could?

PS: Per the experience I brought up this functionality previously on some Android smart TVs, when trick playback the user may expect the audio to be mute.

Thanks a lot for your feedback, if actually fails on old devices (such as Galaxy note 2, Galaxy s4 ), I will try to get one of them to see the root cause. Could you provide the device you tested also?

[workspace]

@WeiChungChang Thnak you for response. I just played 'Youtube Dash - Google Play, Google Glass' samples. Rewind seems to work well, however, fast-forward doesn't. Here are some attachments to show my tests. Thank you!

Galaxy S4 (There are some discontinuities)
https://youtu.be/0sN5hn_Hl2k
Nexus 5 (what I expected)
device-2017-02-28-112449.zip

[robin2046]

@WeiChungChang , this should not be ExoPlayer's issue. Those 'muted' video can be played by ExoPlayer without any problem. However, it cannot be rendered by your forked version. You can try to play any video clip downloaded from https://www.reddit.com/r/gifs/. This kind of video clip is very popular this days as a substitution for GIF files. And I believe the ability to for trick playing of "no-audio-track" video clips will also be welcomed by end users.

[MateusZitelli]

@WeiChungChang In fact I had the same issues that @KiminRyu related. Playing a 2k mp4 video in < API 23 results in a freezing playback and sometimes freezes the player completely.

I have a whole branch of my application replacing another player by the ExoPlayer, however I can't release it due to this issue.

[WeiChungChang]

Thanks a lot for your comments in advance~

@KiminRyu:

_As agreed, only when >= API 23 it is able to use setPlaybackRate API.
Please try the patch to use seek when < API 23.
https://github.com/WeiChungChang/ExoPlayer/commit/9b879e73a42c553c14873445ac1175764c3feaed
It is provided by @monkey-who._

Galaxy S4 (There are some discontinuities)
https://youtu.be/0sN5hn_Hl2k
Nexus 5 (what I expected)
device-2017-02-28-112449.zip

Per your attached, do you mean for Nexus 5 it works but S4 fails?
They are both on < API 23, right?
_Do you play it locally or by streaming (the link seems to be of MPEG DASH)?_

_If it is a streaming source, please notice my comments before._

It provides my way to response the call-for-feature of trick playback here (an ‘acceptable’ way).
It includes:

1. Locally reverse playback.
2. Locally high speed playback (more than X2 (or X4)).

The patch could be applied to streaming also.
However, I will improve the streaming case hereafter.

_It means for streaming case we still have a lot of work to be done._

Also, please notice that for adaptive streaming, we SHOULD notify the bandwidth calculator ( AdaptiveVideoTrackSelection) about the change of play speed. Otherwise, it may lead to buffer underrun.

@MateusZitelli

The same as @KiminRyu also;
As I remembered, for 2K _locally_ it may not as smooth as small size movies.
_However it should NOT freeze._
If you played locally, I will check the case here.
_If you played a streaming source, as mentioned, it needs a lot of work later & belongs to TODO._

@robin2046

_Do you mean the patch make the files from https://www.reddit.com/r/gifs/ out of work?_
If so, I will give it a check.

Demo

Finally, the test below shows the patch works on an old _HTC one mini with Android 4.4, API = 19._
https://youtu.be/olyRXZX3-l8
_originally(use setPlaybackRate), it is out-of-work(for X2, X4...)._
_With the patch, it provides a way to X2, X4 & rewind._
(Thanks a lot for @monkey-who help on the test)

[workspace]

@WeiChungChang Nexus 5 (API 23) works well when I try to play with 2x speed, no discontinuities, with sound. I applied your latest patch(WeiChungChang@9b879e7). I think Galaxy S4(API 21) fails again. I expected same result as Nexus 5(without discontinuity, with sound), however, your demo seems to seek media with 2x speed not play with 2x speed. Is it intended result? and streaming / local source don't have any differences. (I played your baseball source also).

[monkey-who]

@KiminRyu the latest patch WeiChungChang@9b879e7 works as following description:

• sdk >=23
  ff mode and the speed < high_speed_threshold, it plays with audio and in right pitch;
  rewind mode or ( ff mode and speed >= high_speed_threshold ), it plays with no audio;
• sdk < 23
  ff and rewind mode are playing with no audio;

And I think ExoPlayer is application level library, if you want ff/rwd with audio, modifications on framework might be inevitable, because it involves rendering/decoding issues.

[workspace]

@monkey-who A year ago, I integrated third-party native library with ExoPlayer-v1.
My goal was to implement speed control without having to modify the low API framework.
I have written to make it compatible with PlaybackParams in API version 23, but there are two limitations. First, V2 is released. I was not able to proceed because it seems to be difficult to maintain because development is underway. Secondly, I was not sure if it works well for all types of media. I was expecting this kind of approach. However, I agree with your statement that the ExoPlayer is an application level library, and that behavior seems to be the best solution.

P.S. Do you have any idea to fix it in low level later?

[monkey-who]

@KiminRyu Well, sure we all seek for a solution that can handle trick play in app level. If 'fix it in low level' means modifying framework, then you could refer aosp's code. My leader used to migrate code from later version into previous ones to solve some problems. And it definitely not works on general devices.

And your concerning of media compatibility, I think you could run CTS to verify the solution.

Other than above, I think ExoPlayer aims to optimize data before it goes into framework, so is it possible to provide modified data that actually realize trick play? Since I'm not familiar with media play, it is just a thought.

[WeiChungChang]

@robin2046
About your case when playing video only movie system will crash.
I have found the root cause and made a patch.

@yangwuan55 I tried your work and it is impressive. However, when I try to play some video clip without audio track, I always get following error:
02-22 15:08:05.833 31708 31747 E ExoPlayerImplInternal: Internal runtime error.
02-22 15:08:05.833 31708 31747 E ExoPlayerImplInternal: java.lang.NullPointerException: Attempt to invoke interface method 'long com.google.android.exoplayer2.util.MediaClock.getPositionUs()' on a null object reference

_It is because we access rendererMediaClock but with a video-only movie, it is NULL;_

I have patched it within my branch as below.
https://github.com/WeiChungChang/ExoPlayer/commit/e0d9b7cbbd90a6008ea70bbdc3c000b8dbe2595d
You could fix it by yourself if you are based upon a different branch such as from Sir @yangwuan55 in a similar way.

Thanks~

[WeiChungChang]

@robin2046

@yangwuan55 I tried your work and it is impressive. However, when I try to play some video clip without audio track, I always get following error:
02-22 15:08:05.833 31708 31747 E ExoPlayerImplInternal: Internal runtime error.
02-22 15:08:05.833 31708 31747 E ExoPlayerImplInternal: java.lang.NullPointerException: Attempt to invoke interface method 'long com.google.android.exoplayer2.util.MediaClock.getPositionUs()' on a null object reference...

You could fix this issue by the patch below if you are at https://github.com/yangwuan55/ExoPlayer.
Thanks.

patch.txt

[mseroczynski]

Sorry for the delay. This slipped from Q4 but we are working on it and plan to have something ready later this quarter.

@andrewlewis Hello once again! Is it now at least confirmed that it's doable? If so, have you guys already started working on this issue (by working I mean writing code) or is it more like Q2/Q3 now :)?

[ojw28]

Note: This work is still in progress. Further changes will be landing soon.

[piotrmadry]

@ojw28 Is it possible to provide some release date for this future ? (Video speed control)

[PaulWoitaschek]

@piotrmadry its Q1 (friday)

[ojw28]

The work should be complete (or nearly complete) in dev-v2 by that point, but it may take a week or two extra before we cut a release that includes it.

[ojw28]

Playback rate and pitch adjustments are seamless again as of https://github.com/google/ExoPlayer/commit/f7fff0d5835b42f6d6db7e46c7b1e546f84f545a. The feature is now at the level of our previous (API 23+ only) support, only now across all API levels. Expect a release in the next week or two as mentioned above.

Subsequent changes will follow that propagate the playback rate to some other components that would benefit from it (e.g. the ABR algorithm, in the case of adaptive playbacks).

[workspace]

I've applied the new playback speed control on the dev-v2 branch, but the sync of video and audio did not fit well. I wonder if you are aware of this problem. (x2)

[andrewlewis]

@KiminRyu I'm not aware of any issues with audio/video synchronization. If you file a new issue including steps to reproduce the problem we will take a look. Thanks.

[ojw28]

Apologies we've not managed to provide a release version containing this functionality yet. The functionality will be in the 2.4 release, but we're having to wait for a few other changes to land before we can publish it. We're now targeting toward the middle of next week. Thanks!

[KirillMakarov]

@ojw28 can you notify us in this issue, when the feature will be released?

[PaulWoitaschek]

@KirillMakarov GitHub has an atom feed you can add to your RSS reader:
https://github.com/google/ExoPlayer/releases.atom

[ojw28]

You can also watch https://bintray.com/google/exoplayer/exoplayer. There are quite a few ways to be notified of new releases without us having to update issues manually ;).

[workspace]

@ojw28
The new playback speed change method seems to be heavily influenced by CPU performance. I just put this code in demo when player is initialized.

player.setPlaybackParameters(new PlaybackParameters(2.0f, 1.0f));

For PlaybackParams added in API 23, the video played back successfully on Nexus 5. However, with the new PlaybackParameter, frame drops occur on devices with low CPU performance such as Nexus 5 and Galaxy S4.

Nexus 5 with API 23
Galaxy S4 with API 21

However, the Nexus 5X, Galaxy Note 5 and Galaxy S8 Plus (Android N devices) played well without problems. I played demo in the API 23 version of the emulator to see if it was a difference in the Android API.
 
Nexus 5x emulator with API 23

So I'm worried that the new playback speed change method will work well on various devices.

Error Log in Android Studio

04-21 16:56:50.368 20130-20130/com.google.android.exoplayer2.demo E/EventLogger: internalError [21.31, audioTrackUnderrun [56448, 320, 27]]
04-21 16:56:50.443 20130-20130/com.google.android.exoplayer2.demo E/EventLogger: internalError [21.38, audioTrackUnderrun [56448, 320, 1]]
04-21 16:56:50.618 20130-20130/com.google.android.exoplayer2.demo E/EventLogger: internalError [21.55, audioTrackUnderrun [56448, 320, 22]]

[andrewlewis]

@KiminRyu Did you see these problems when running a release APK or a debuggable APK?

The new implementation uses the Java version of Sonic, so its performance is affected by what optimizations the Java runtime applies. There is a certain optimization that is turned off on debuggable builds, which causes an inner loop in Sonic to be very slow. If you build a release APK this optimization should apply (on all API versions) and I found that performance was fine on various devices I tested, including a Samsung Galaxy S4 running Android 4.2.2 (API 17).

On API 24 Android builds onwards, performance should be fine on both debuggable and release builds.

[workspace]

I got the latest version of the dev-v2 branch and built it. (so it is a debuggable apk.) So...How can I set the same environment with a release apk?

[andrewlewis]

I used the instructions for Android Studio: Sign your release build.

[workspace]

I built it with the release version, and the playback speed change worked just as you said. As shown in the video attached below, playback of Secure_xx (SD, HD, UHD) of Widevine DASH: MP4, H264 sample was not smooth. On the other hand, clear_xx samples of the same widevine drm sample worked well, and Youtube DASH also worked.

widevine secure hd -> clear hd -> youtube dash

[PaulWoitaschek]

Thanks for the release. It works really great!

[workspace]

@andrewlewis
As I mentioned above, when playing from 2.4.0 version with widevine secure path, the screen stops immediately after 10 seconds. NEXUS 5 (23), Galaxy Note 5 (24), Galaxy S8 Plus(24) and other API 23 and above work normally. On the other hand, older devices such as the Galaxy S4 (21), Galaxy Note 2 (19) and Galaxy Note (16) have a problem in changing the playback speed faster. I have built and tested a new 2.4.0 release apk.

[andrewlewis]

@PaulWoitaschek Glad to hear it's working!

@KiminRyu Please could you file a new issue to track that, including all the details requested in the new issue template? Thanks.

I'm closing this issue now as the core functionality is implemented, but as @ojw28 noted above further changes will be needed to propagate the playback speed to some other player components (like the adaptive track selection logic and LoadControl). In case it's useful: there's a blog post summarizing how to use this feature.

[gemiren]

I am seeing high CPU usage and battery drain since the 2.4.0 version when using variable playback speed. On the Nexus 5x it uses about 8% to play with Android's built-in media player while it uses about 16% CPU when using ExoPlayer with the same playback speed. Looking at the code I can see ExoPlayer now always uses Sonic to adjust the playback speed, even when the Android system supports the variable playback natively on API>=23. That doesn't seem to be an efficient way to handle the variable playback speed to me. What not only use Sonic for API<23 and use the system's built-in support for API>=23? The C version of Sonic on API>=23 is much faster than the Java version of Sonic included in ExoPlayer.

[PaulWoitaschek]

I'm seeing multiple user complaints about high battery drain too.

The C version of Sonic on API>=23 is much faster than the Java version of Sonic included in ExoPlayer.
Did you measure that? According to the author the java and c version are quite equal.

[gemiren]

@PaulWoitaschek, I didn't measure that. Maybe "faster" isn't the proper word to describe this difference in the C version and the Java version. There are could be many other factors involved here. Anyway, the CPU usage is way too high when playing with variable playback speed for API>=23 compared with using the C version Sonic library in the Android system. I just don't understand why do we need to use the Java version of Sonic for API>=23 since the Android system already has that implemented. We don't need to reinvent the wheel for API>=23 to play with variable playback speed. The Android system has that supported and it works nicely and perfectly in the versions before 2.4.0.

Sadly this is a deal break for me. I had to revert back to old ExoPlayer version to fix the battery drain.

[ojw28]

Do you have solid data backing up the claim of increased battery drain? Looking at CPU usage isn't sufficient. Neither is linking ad-hoc user reports to something you know has changed, when other things have also changed. It really needs to be actual battery drain data comparing the two whilst minimizing any other changes.

Note that CPU usage for the playback speed adjustment will likely be attributed to a different process in each case (your process when using the Java version, or the mediaserver process when using the C version). So if you're just looking at the CPU usage of your own process then that's not a correct comparison. As noted above the author of Sonic suggests the performance of the two are give-or-take equal. Note also that you need to be using a release build of your apk, as discussed above. Debug builds may yield vastly inferior performance with variable playback speed enabled.

As for why we're using the Java version everywhere: One reason is simplicity; having a single code path is just a lot simpler. Another reason is that the Android system support isn't perfect. We actually found a bug in Sonic (which is fixed in ExoPlayer's bundled version, but will only be fixed in Android from O onward). If we see solid evidence that power drain is significantly worse then we can revisit the decision.

[gemiren]

@ojw28, unfortunately I don't have the solid data to pin down what's exactly causing the huge increased battery drain. Since I updated my app to use ExoPlayer 2.4.0 I have received numerous user feedback regarding the increased battery drain. So I did some comparison between ExoPlayer and Android Media Player on a Nexus 5x with Android 7.1.2. I did see the battery usage went down significantly if I switched to use the Android Media Player.

I am using the release build compile 'com.google.android.exoplayer:exoplayer:r2.4.1' so it is not an issue of the debug version.

You are right that CPU usage isn't the only factor causing the battery drain. It's just convenient to monitor the CPU usage in the Android Studio to get the rough idea of how the battery performances.

It is true that the CPU usage should include the mediaserver process when using the Android Media player. However, that's not how the battery usage stats works. Now my app is listed at the top of the battery drain list. I can not tell users that the battery usage is probably the same, just because now it includes the battery usage from mediaserver which isn't included before. From a user point of view, when the battery usage goes up almost doubled, any explanation is pale.

[ojw28]

I don't think there's evidence in this thread (yet) of "huge increased battery drain" or that the Java version is significantly less efficient. It seems to me there are two possibilities at this point:

1. Total battery drain is actually about the same as before. Due to the way battery accounting works, battery drain associated with the variable speed adjustment is now being attributed to your app where-as previously it was attributed to mediaserver. This is causing your users to complain.
2. Total battery drain is actually higher (i.e. the user's battery will actually drain faster than before).

If it's the first of these then I don't think we're going to fix it. After all, your app is causing the battery drain that's being associated to it in this scenario. The fact the drain was not accounted to your app previously would effectively be an accounting trick, and something we should probably fix in the platform to ensure correct attribution in future Android releases. If it's the second of the two possibilities then this is definitely something we should look at. If you could help us to determine which of these cases is actually true, for example by running some experiments, that would be very helpful.

Note that it's how you build your apk that affects whether it's a release build or not, not how you depend on ExoPlayer in your build.gradle. See some of the comments from @andrewlewis further up this thread.

[Orbyt]

This is quite a long thread. Can I get a summation of whether or not this functionality has been implemented, and what API levels are supported?

[PaulWoitaschek]

It's supported on all api levels through ExoPlayer.setPlaybackParameters.