Exoplayer: How can I get the callback when Google Assistant command is triggered?

[aureobeck]

Searched documentation and issues

Official ExoPlayer documentation and source code of MediaControllerCompat, MediaSessionConnector, MediaSession classes.

My issue is similar to #6057 and #6446 but none of the solutions worked as expected (details on question).

Question

I have an Android cordova app which has Exoplayer Media Session enabled, the player was not resuming after seeking on AndroidTV using Google Assistant.

So I tried to save the playWhenReady on the dispatchSetPlayWhenReady(which is called when Google Assistant is opened) in a local property wasPlaying, to check if it should resume after seeking (if it was playing before).

private class MyControlDispatcher implements ControlDispatcher {

    private boolean wasPlaying = false;

    @Override
    public boolean dispatchSetPlayWhenReady(Player player, boolean playWhenReady) {
        this.wasPlaying = player.getPlayWhenReady();
        player.setPlayWhenReady(playWhenReady);
        return true;
    }

    @Override
    public boolean dispatchSeekTo(Player player, int windowIndex, long positionMs) {
        if (this.wasPlaying) {
            player.setPlayWhenReady(true);
        }

        player.seekTo(windowIndex, positionMs);
        return true;
    }
    ...
}

MyControlDispatcher myControlDispatcher = new MyControlDispatcher();
MediaSessionConnector mediaSessionConnector = new MediaSessionConnector(mediaSession);
mediaSessionConnector.setControlDispatcher(myControlDispatcher);

That solved the initial problem, but then I have a problem when user pauses and then seeks, both using Google Assistant. It will save wasPlaying as true when user calls PAUSE command (on the dispatchSetPlayWhenReady when assistant is opened) and resume after seeking when it shouldn't.

So basically I need to somehow get the PAUSE command callback from google assistant so I can differentiate from the dispatchSetPlayWhenReady which is called when assistant is opened. And guarantee that wasPlaying is set to false when user calls PAUSE command and prevent it from resuming after seeking.

Tried so far:

mediaSession.setCallback(new MediaSessionCompat.Callback() {
    public void onPause() {
        wasPlaying = false;
        super.onPause();
    }
    ...
});

And:

player.addListener(new Player.EventListener() {
    @Override
    public void onPlayerStateChanged(boolean playWhenReady, int playbackState) {
        if (playbackState == PlaybackStateCompat.STATE_PAUSED) {
            wasPlaying = false;
        }
    }
});

But none of them are called on PAUSE command of Google Assistant, but only when it's opened.

[AquilesCanta]

@marcbaechinger Marc, mind taking a look?

[marcbaechinger]

Can you give more details on the root problem, why the player was not resuming. Agree that the workaround you are trying is complicated as we can not distinguish from actual intended pausing.

I'd prefer to find the root cause of why the player is not resuming after a seek. That does not sound right to me. If we can remove that, it would be the better solution than a workaround which is dificult to keep away from valid use cases.

Can you put breakpoints in the callback methods of MediaSessionConnector to find out what commands Assistant is sending when it starts?

[aureobeck]

Thanks for the response @marcbaechinger. I've added some breakpoints on MediaSessionConnector and onPause is called just after I open google assistant, then giving seek command calls onSeekTo. So it's not calling onPlay after seek if it was playing before. This is how we're initializing media session and player:

...

MediaSessionCompat mediaSession = new MediaSessionCompat(context, TAG);
MediaSessionConnector mediaSessionConnector = new MediaSessionConnector(mediaSession);

MyControlDispatcher controlDispatcher = new MyControlDispatcher();
controlDispatcher.setNotificator(notificator);

MyTimelineQueueNavigator timelineQueueNavigator = new MyTimelineQueueNavigator(mediaSession);
timelineQueueNavigator.setNotificator(notificator);

mediaSessionConnector.setRewindIncrementMs(skipTimeMs);
mediaSessionConnector.setFastForwardIncrementMs(skipTimeMs);
mediaSessionConnector.setControlDispatcher(controlDispatcher);
mediaSessionConnector.setQueueNavigator(timelineQueueNavigator);

SimpleExoPlayerView simpleExoPlayerView = new SimpleExoPlayerView(context);
SimpleExoPlayer player = ExoPlayerFactory.newSimpleInstance(getContext(), trackSelector, new DefaultLoadControl(),
    drmSessionManager, DefaultRenderersFactory.EXTENSION_RENDERER_MODE_OFF);

ExoPlayer.EventListener exoPlayerEventListener = new MyExoPlayerEventListener();

player.addListener(exoPlayerEventListener);
simpleExoPlayerView.setPlayer(player);    
mediaSessionConnector.setPlayer(player);
mediaSession.setActive(true);

MyTimelineQueueNavigator:

private class MyTimelineQueueNavigator extends TimelineQueueNavigator {
    private WeakReference<Notificator> notificatorReference = new WeakReference<>(null);
    private JSONObject mediaDescriptionAsset;

    private MyTimelineQueueNavigator(MediaSessionCompat mediaSession) {
        super(mediaSession);
    }

    private void setNotificator(Notificator notificator) {
        notificatorReference = new WeakReference<>(notificator);
    }

    private void setMediaDescriptionAsset(JSONObject mediaDescriptionAsset) {
        this.mediaDescriptionAsset = mediaDescriptionAsset;
    }

    @Override
    public long getSupportedQueueNavigatorActions(Player player) {
        return PlaybackStateCompat.ACTION_SKIP_TO_NEXT;
    }

    @Override
    public void onSkipToNext(Player player, ControlDispatcher controlDispatcher) {
        Notificator notificator = notificatorReference.get();
        if (notificator == null) {
            return;
        }

        notificator.success(ResponseType.onSkipToNext);
    }

    @Override
    public MediaDescriptionCompat getMediaDescription(Player player, int windowIndex) {
        final JSONObject asset = (mediaDescriptionAsset == null) ? new JSONObject() : mediaDescriptionAsset;

        MediaDescriptionCompat.Builder builder = new MediaDescriptionCompat.Builder();
        builder.setMediaId(asset.optString(MEDIA_DESCRIPTION_MEDIA_ID, null))
                .setTitle(asset.optString(MEDIA_DESCRIPTION_TITLE, null))
                .setSubtitle(asset.optString(MEDIA_DESCRIPTION_SUBTITLE, null))
                .setDescription(asset.optString(MEDIA_DESCRIPTION_DESCRIPTION, null));

        String imageUrl = asset.optString(MEDIA_DESCRIPTION_IMAGE_URL, null);
        if (imageUrl != null) {
            builder.setIconUri(Uri.parse(imageUrl));
        }
        String mediaUrl = asset.optString(MEDIA_DESCRIPTION_MEDIA_URL, null);
        if (mediaUrl != null) {
            builder.setMediaUri(Uri.parse(mediaUrl));
        }

        return builder.build();
    }
}

MyControlDispatcher:

private class MyControlDispatcher implements ControlDispatcher {

    private WeakReference<Notificator> notificatorReference = new WeakReference<>(null);

    private void setNotificator(Notificator notificator) {
        notificatorReference = new WeakReference<>(notificator);
    }

    @Override
        public boolean dispatchSetPlayWhenReady(Player player, boolean playWhenReady) {
        Notificator notificator = notificatorReference.get();
        if (notificator == null) {
            return false;
        }
        player.setPlayWhenReady(playWhenReady);
        notificator.success(ResponseType.onPlayPause, playWhenReady);
        return true;
    }

    @Override
    public boolean dispatchSetRepeatMode(Player player, int repeatMode) {
        return false;
    }

    @Override
    public boolean dispatchSetShuffleModeEnabled(Player player, boolean shuffleModeEnabled) {
        return false;
    }

    @Override
    public boolean dispatchStop(Player player, boolean reset) {
        Notificator notificator = notificatorReference.get();
        if (notificator == null) {
            return false;
        }

        player.stop();
        notificator.success(ResponseType.onPlayPause, false);
        return true;
    }

    @Override
    public boolean dispatchSeekTo(Player player, int windowIndex, long positionMs) {
        Notificator notificator = notificatorReference.get();
        if (notificator == null) {
            return false;
        }

        if (canSeekTo(player, positionMs)) {
            player.seekTo(windowIndex, positionMs);
            notificator.success(ResponseType.onSeek, positionMs);
        }
        return true;
    }

    private boolean canSeekTo(Player player, long position) {
        return (position >= 0) && (position <= player.getDuration());
    }
}

MyExoPlayerEventListener:

public class MyExoPlayerEventListener implements ExoPlayer.EventListener {
    @Override
    public void onLoadingChanged(boolean isLoading) {
    }

    @Override
    public void onRepeatModeChanged(int repeatMode) {
    }

    @Override
    public void onPlaybackParametersChanged(PlaybackParameters playbackParameters) {
    }

    @Override
    public void onPositionDiscontinuity(int reason) {
    }

    @Override
    public void onPlayerStateChanged(boolean playWhenReady, int playbackState) {
    }

    @Override
    public void onTimelineChanged(Timeline timeline, Object manifest, int reason) {
        if (timeline != null && timeline.getWindowCount() > 0) {
            Timeline.Window timeLineWindow = timeline.getWindow(timeline.getWindowCount() - 1, window);
            isTimelineStatic = !timeLineWindow.isDynamic;

            if (initialWindowStartTimeMs == 0)
                initialWindowStartTimeMs = timeLineWindow.windowStartTimeMs;

            windowOffsetMs = (timeLineWindow.isDynamic && initialWindowStartTimeMs > 0)
                ? timeLineWindow.windowStartTimeMs - initialWindowStartTimeMs
                : 0;
        } else {
            isTimelineStatic = false;
        }
    }

    @Override
    public void onPlayerError(ExoPlaybackException e) {
        playerNeedsSource = true;
    }

    @Override
    public void onTracksChanged(TrackGroupArray ignored, TrackSelectionArray trackSelections) {
        if (player == null) {
            return;
        }

        MappedTrackInfo trackInfo = trackSelector.getCurrentMappedTrackInfo();
        if (trackInfo == null) {
            return;
        }

        for (int rendererIndex = 0; rendererIndex < trackSelections.length; rendererIndex++) {
            TrackGroupArray rendererTrackGroups = trackInfo.getTrackGroups(rendererIndex);
            for (int i = 0; i < rendererTrackGroups.length; i++) {
                ArrayList<Format> formats = new ArrayList<>();
                for (int j = 0; j < rendererTrackGroups.get(i).length; j++) {
                    formats.add(rendererTrackGroups.get(i).getFormat(j));
                }
                trackFormats.put(player.getRendererType(rendererIndex), formats);
            }
        }
    }
}

[marcbaechinger]

Thanks! That looks good actually.

It seems that some ATV devices do exhibit the behaviour you are describing (like pausing when Assistant is activated). It seems that it does not happen on all devices though, which is of course a bit unfortunate, because there is no way to keep these apart.

Can you tell me on what device you are experiencing this behaviour? Would it be possible for you to check on another device as well like on a Nvidia Shield? Or can you test on an emulator to see what the behaviour is there?

[aureobeck]

We've been testing on Nvidia Shield, the problem is always happening there

[marcbaechinger]

Ok. Would be interesting to know how other ATV devices/Assistant versions behave. I can't see how ExoPlayer or the MediaSessionConnector can help much with that. As you describe, Assistant acts like another user who is using the media controller to send commands. If Assistant pauses the player when it starts listening, it should also resume playback after seek. If not that looks like a bug to me.

The ony thing you can do is just resume playback after seek. Obviously this hinders a real user to pause and then seek without resuming, but I can't see another approach fin case of the behaviour exhibited by Assistant.

[aureobeck]

I see, one thing we noticed was that on version 2.9.6 of Exoplayer, if we added a breakpoint or delay on onSeekTo callback from depracated DefaultPlaybackController, the aplication was working exactly as expected, it seemed that this was a racing issue between Google Assistant and Exoplayer. After we updated Exoplayer to 2.11.3 and started using ControlDispatcher interface instead of overriding the methods of DefaultPlaybackController the issue continued happening and now the delay didn't help at all (now on dispatchSeekTo). So I'm guessing there could be some implementation from Exoplayer's side that could help solve this issue.

[marcbaechinger]

Hmmm. I was under the impression that Assistant is sending an onPause which is causing the problem that after the seek you need to resume playback again from app code.

How can delaying the onSeekTo help not pausing the player? I probably understand not properly, sorry. Can you elaborate?

[aureobeck]

OnPause is always being called when google assistant is opened (both 2.9.6 and 2.11.3), but on 2.9.6 a delay on seek makes app resume after seek (if it was playing). The delay doesn't help on 2.11.3 though

[aureobeck]

Any updates on this @marcbaechinger ? We are open to help investigating the issue, I think this could be solved on Exoplayer's side, let me know, thanks!

[marcbaechinger]

Please let me know about your ideas. :)

[aureobeck]

The main problem, as you said @marcbaechinger, is that on the MediaSessionConector the onPause method is being called when Google Assistant is going to foreground, what as far as I understand isn't correct.

More info on device tested:
AndroidTV 9 Nvidia Shield 8.0.2 (32.5.205.105)

[marcbaechinger]

Yes, I agree that onPause is called. This call is triggered by a media session event which comes from Assistant. Assistant is using MediaController.TransportControl to send the pause event.

This onPause event from Assistant is the same like when a user issues a 'Ok Google, pause' voice command. Pausing the app in Android Auto or on a watch running WearOS results in the same command. All these clients are using the same events which end up in the MediaSessionConnector. The connector then delegates to the player only.

Because of this just ignoring the onPause event seems not a solution and there is no way to distinguish these event like for a source of origin. The MediaSession.Callback does not deliver information about the origin of the event.

[aureobeck]

I see, so should I open an issue on Google Assistant's side @marcbaechinger ?

[inv3rse]

The underlying issue appears to be that the MediaSession is in the STATE_BUFFERING after the seek dispatch. According to the documentation:

State indicating this item is currently buffering and will begin playing when enough data has buffered.

So the assistant probably assumes that there is no need dispatch the play event, since it thinks that it is already playing. This is however not true while playWhenReady is not set.
With the changes from https://github.com/google/ExoPlayer/pull/7367 it is working as expected for me. Meaning that after the seek another play event will be dispatched.

[marcbaechinger]

That looks interesting. Thanks for your comment and the PR. I look into this.

I think the problem above is that Assistant is sending a pause event which is not expected, but it's very well possible this is caused by the same reason.

@aureobeck Seems like the semantics described by @inv3rse indicates the problem is in the MediaSessionConnector.

[marcbaechinger]

We merged the pr #7367 of @inv3rse into thee dev2 branch.
@aureobeck Can you please test if this helps?

[marcbaechinger]

I'm closing this issue assuming the pr #7367 fixed the problem. Please re-open if you think this is required.

[aureobeck]

Greetings, @marcbaechinger , sorry for late response. I'm still having trouble to test a local branch of Exoplayer on our Cordova app. Since we have some dependency complexity, but we're working on it. I'll let you know if the problem persists.

[aureobeck]

FYI @marcbaechinger @inv3rse we were able to test this and resuming after seeking seems to be working now on dev2 branch 🚀️🚀️

[aureobeck]

Any ideas when can we expect a new version @marcbaechinger?

[ojw28]

This should be in 2.11.5, which will hopefully go out sometime next week.