Exoplayer: Audio track position is unstable during initialization

[mmbahaa]

Issue description

In MediaCodecAudioTrackRenderer getPositionUs, the audio position starts fine and progresses but after a fraction of a second playing, getCurrentPosition stays for 50 - 300ms not updated because the newCurrentPositionUs is smaller than currentPositionUs.

Can you please explain why this is happening? it seems related to the audio decoder warm up.

Reproduction steps

1. Re-build the demo app with logs in the getPlaybackPositionUs, print newCurrentPositionUs and currentPositionUs.
2. Play for 5 seconds.
   Expected: newCurrentPositionUs continually progresses
   Actual: newCurrentPositionUs at some point decreases which causes the currentPositionUs to not be updated.
   

   Link to test content

Super speed (PlayReady) - demo app

Version of ExoPlayer being used

2.5.4

Device(s) and version(s) of Android being used

Sony Android M

A full bug report captured from the device

exoplayer-playbacktime-bug.txt

[tonihei]

Might be related to the way we access the audio timestamp internally (see #3830). Maybe @ojw28 can add some further explanation?

[mmbahaa]

@tonihei thanks for the update, I'd really appreciate @ojw28 input as this freeze causes failures in some of my unit tests.

[andrewlewis]

Just a quick update on this:

The short pause just after starting playback seems to be caused ExoPlayer's derived audio track position not increasing smoothly. For example, in one playback I saw the video renderer showing a frame for about 100 ms rather than the expected 33 ms because the player position did not increase for 100 ms. This was due to the internal audio track position going backwards (as you noted above, we force the player position increase monotonically).

When rendering audio, we calculate the player position roughly as follows:

• Get a position from the underlying android.media.AudioTrack, either using getTimestamp if available, or getPlaybackHeadPosition if not.
• Offset this value by the elapsed time since the position was sampled.
• Offset this position by the start media time and apply any required speedup. We also ensure the position does not exceed the amount of data written to the audio track.
• Output a position that's the maximum of this position and the previous output position, so the position cannot decrease.

The behavior seems to vary a bit between devices and builds, but in general I've found that positions from the preferred getTimestamp API after 500 ms of playback imply a later playback start position than the positions we derive at the very beginning of playback (which are based on either getPlaybackHeadPosition or getTimestamp). That is, if you look at the timestamp updates provided by the platform after 500 ms and extrapolate backwards, the audio track appears to have started playback later than we thought. It's as if the position jumps backwards.

I initially thought this might be due to audio playback not actually starting at the time we get our first timestamp, but, even compensating for the latency of the audio track, the playback head position starts increasing from zero earlier than expected based on later timestamp updates.

I haven't yet found the root cause for this. One option is to smooth position updates by increasing the position slowly in cases when the underlying audio track timestamp goes backwards, but enough to avoid freezing video playback completely.

[mmbahaa]

Thanks @andrewlewis for the update. I also had the same assumption as you. When I was debugging this issue, this sudden change in time came down to audioTimestamp.nanoTime. Maybe the platform's getTimestamp is unstable during warmup? Smoothing position updates is a good idea, however, I'm wondering if this will be scalable to many devices because - as you said - the behaviour varies.
You probably know this, but one of my first assumptions seemed related to what the platform documented about AudioTrack's getTimestamp:
"during initial warmup or after a routing or mode change, you should request a new timestamp periodically until the reported timestamps show that the frame position is advancing, or until it becomes clear that timestamps are unavailable for this route."

[mmbahaa]

@andrewlewis are there any updates?

[andrewlewis]

We can resolve this issue by polling timestamps frequently when the AudioTrack is starting, then falling back to polling infrequently. That lets us notice the point when the position starts increasing. I have a prototype fix but it needs cleaning up, so hopefully we can push this to the development branch some time over the next few weeks. Thanks!

[mmbahaa]

@andrewlewis can you point me to your changes even if they're not cleaned up yet?

[andrewlewis]

I can't easily share my local changes, but you may be able to implement a quick fix for experimentation by modifying the code that decides when to sample the AudioTimestamp in DefaultAudioSink. There are a few fixes we need to make, but the main one is to poll the timestamp more frequently just after initializing the AudioTrack. It seems like sampling ever 10 ms until the timestamp starts to increase then polling every 10 seconds works well and avoids having the position jump.

[andrewlewis]

The player position now advances more smoothly on various devices we've tested on. The large position jump after about 500 ms should be fixed, so I'll close this issue for now.

There may still be further opportunities to make the position advance more smoothly, especially at the point where we switch from getPlaybackHeadPosition to getTimestamp.