Exoplayer: Exoplayer slow seekTo() update

[vxhviet]

Dear ExoPlayer Devs,
I would like to use ExoPlayer to seek through a local video via the seekTo(long position) method. The video is not actually played, but instead being scrubbed through like this.

What I've done so far has basically achieved this function, however the performance is not very great. The returned frames are quite slow to update and basically get stuck after a while. I suspect this is because the touch event ACTION_MOVE has to wait for the video to buffer, so I try to disable it via
mExoPlayer = ExoPlayer.Factory.newInstance(RENDERER_COUNT, 0, 0);

However I can not figure out what to do with BUFFER_SEGMENT_SIZE and BUFFER_SEGMENT_COUNT as from my research and experiment, setting these too low and the video can't be played at all, setting these too high and it just make my problem worse.

I would like to ask is there a way to completely remove the buffer so that exoPlayer can just instantly return the frame at seekTo(long position) and if not, is there any way to improve the performance?

I would like to add these severe stuttering or stuck only happen with videos captured by the Samsung Galaxy s6 running Android 5.1.1. With videos captured by Nexus 5 running 6.0.1 or low resolution videos, the problem is not as severe. (I'm sorry since currently I only has those two devices to test. In case I can get a hold of more devices, I'll update the results.)

Below is my code snippet using ExoPlayer version 1.5.6:

    private static final int RENDERER_COUNT = 1;
    private static final int BUFFER_SEGMENT_SIZE = 64 * 1024;
    private static final int BUFFER_SEGMENT_COUNT = 64; // default 256
.....

        // 1. Instantiate the player. RENDERER_COUNT is the number of renderers to be used.
        // In this case it’s 1 since we will only use a video renderer. If you need audio and video renderers it should be 2.
        mExoPlayer = ExoPlayer.Factory.newInstance(RENDERER_COUNT, 0, 0); //disable the buffer and rebuffer for better performance

        // 2. Construct renderer.
        Uri uri = Uri.parse(mVideoPath);
        Allocator allocator = new DefaultAllocator(BUFFER_SEGMENT_SIZE);
        String userAgent = Util.getUserAgent(this, "Android Demo");
        DataSource dataSource = new DefaultUriDataSource(this, null, userAgent);
        ExtractorSampleSource sampleSource = new ExtractorSampleSource(uri, dataSource, allocator, BUFFER_SEGMENT_COUNT * BUFFER_SEGMENT_SIZE);
        MediaCodecVideoTrackRenderer videoRenderer = new MediaCodecVideoTrackRenderer(
                this, sampleSource, MediaCodecSelector.DEFAULT, MediaCodec.VIDEO_SCALING_MODE_SCALE_TO_FIT);

        // 3. Inject the renderer through prepare.
        mExoPlayer.prepare(videoRenderer);

        // 4. Pass the surface to the video renderer.
        mExoPlayer.sendMessage(videoRenderer, MediaCodecVideoTrackRenderer.MSG_SET_SURFACE, holder.getSurface());

        // 5. Start playback.
        mExoPlayer.seekTo(currentVideoPosition);

Thanks you very much for your time,

[andrewlewis]

Passing zero as the buffer duration as in your code snippet looks fine for what you are trying to do. I think one possible explanation is that decoding the required frame sometimes just takes a long time: if you seek to a frame, the cost of decoding it depends on its distance from the previous key-frame. Key-frames can be decoded quickly but later frames are increasingly expensive to decode because (generally) all earlier frames in the group of pictures (i.e., since the last key-frame) have to be decoded first. So it is generally not feasible to make seekTo give you a frame instantly, I'm afraid.

There may be something else going wrong, so you could test out this hypothesis by checking that the time to decode a frame is low for frames that are just after a key-frame, and high for frames that are just before a key-frame. One way to find out the timestamps of key-frames is to run ffprobe -show_packets -print_format compact -select_streams v:0 filename | grep flags=K. If the speed doesn't depend on the frame position then the problem is different.

Modern compressed video formats are not designed for fast frame-level random access like this. You could try reducing the key-frame interval if you control the encoder, but this can reduce the video quality or increase the file size. Seeking will still be slower on devices with slower decoders. Depending on how smooth you need seeking to be, it may be necessary to store decoded frames as bitmaps so every frame is cheap to access, transcode to a different format or do something more complicated.

(Note: there is a small optimization possible in the case of seeking forwards by a few frames where you 'decode forwards' rather than flushing the decoder state. I don't think we do this at the moment, but it wouldn't help with rewinding anyway.)

[ojw28]

To clarify the final paragraph of @andrewlewis 's reply: We did actually used to implement this optimization in certain specific cases, but it was really difficult to implement properly. Our implementation was subtly broken, and it turned out that it would be _really_ tricky to get it just right. Since the improvement was marginal, only helped with a fairly niche use case, and even then only helped with fast-forward and not with rewind, we ended up removing it. I think this was a good decision.

[vxhviet]

Thanks you so much for your prompt reply @andrewlewis and @ojw28 ,
As per your hypothesis, could you please elaborate how can I check for the frame's decode time? I will test it and reply as soon as I can.
In case seekTo() doesn't work, is there any other method that I can use to achieve this scrubbing function?

Thanks you so much for your time,

[andrewlewis]

@vxhviet You could (for example) log the timestamps of frames as they are decoded, at the start of MediaCodecVideoTrackRenderer.processOutputBuffer. If you pause the player, then tap on the seek bar you should find that the number of frames that have to be decoded depends on where you tap (on or just after a key-frame should require fewer frames to be decoded). To find out how long it's taking, you could use adb logcat -v threadtime to see the how much time elapses between logging a request to seek in your app and the final frame being decoded.

I don't have any other suggestions, except to store the frames in a different format which is more amenable to random access, as per my previous comment.

[saket]

It's really common in iOS to have fast scrubbing support in media players and it's slightly sad to learn that exoplayer does not support this. May I ask what is the limiting factor here. Is it Android as the platform?

[andrewlewis]

@Saketme Could you clarify whether you're looking for support for the optimization @ojw28 mentioned earlier in this thread, or something different?

The time it takes to scrub to a frame partly depends on the performance of the video decoder, which varies between different Android devices. I can't think of any reason Android would be inherently slower or faster than another platform in general.

[saket]

So I was comparing scrubbing this video on my Pixel using ExoPlayer and on an iPad using Narwhal. I don't know what stack is Narwhal using, but scrubbing is fairly smooth and seeking is instantaneous for buffered parts of the video including rewind, but ExoPlayer was taking half a second for the same task.

I was also using this library to cache the video so the Uri supplied to ExoPlayer was a locally cached File.

@andrewlewis does this make sense?

[nitini]

Hi,

I too am curious if ExoPlayer supports smooth / fast scrubbing of video. I'm struggling to describe what I'm looking for but on iOS when I record a video with my camera and then play it back, I can use my finger to swipe through the video fast or slow and the scrubbing is smooth. On Android however (Nexus 5X) I'm finding that this same behavior is very choppy.

Are there video players on Android that can do what I am describing on iOS? Thanks for any info!