Newpipe: Seamless / smooth transition when switching players

[bkqtnte10]

Not sure if this is only for me (Lineage 15.1) or everyone.
When I play a video and then switch it to background it takes a second to resume playback.
It would be nice if the app would continue playback without a gap (like it works when in popup mode and then locking the screen).

[opusforlife2]

2907

playback position is synchronized between players. Easy to switch from one to another

[avently]

@opusforlife2 my PR will not fix the issue because when switching between players quality of the stream changes and it requires to load a new data. So when a user switches from video players to background player (audio only) the player will drop video part and will start loading audio part of the stream. It requires to start loading too (the gap is less then a second but it exists anyway).

[opusforlife2]

@avently I see. I don't know how it works exactly, but if the audio stream is already loaded with the video stream, can it not just continue playing? It's already buffered to an extent. Or are the streams muxed immediately upon buffering?

Also, would the switch between popup and full screen be instantaneous, since it doesn't need to load a new stream?

[avently]

@opusforlife2

if the audio stream is already loaded with the video stream, can it not just continue playing?

Yes, it can. The problem is some Youtube's videos have single track for audio + video parts, and some videos have different tracks for audio and another one for video.

For example, this is a list of tracks provided for one of videos:
360p videoOnly false
720p videoOnly false

1440p videoOnly true
1080p videoOnly true
720p videoOnly true
480p videoOnly true
240p videoOnly true
144p videoOnly true

plus audio-only streams in different tracks.

What does it mean? It means that our player will merge video and audio tracks for all qualities except 360p. This quality contains an audio + video parts inside one track. You may ask why not just merge two tracks for 360p too: it will require additional bandwidth for unneeded audio track and some dirty hacks to disable audio part of video+audio track in Main video player. Another option is to play audio + video track even in background player. It means that you'll load additional megabytes of video while you listen audio in background player. It is unexpected and bad for those who listens via a mobile network.

In other words, it can be implemented but not for all qualities. For some qualities the transition will not be instant.

would the switch between popup and full screen be instantaneous, since it doesn't need to load a new stream?

Nope, popup player and video player have different qualities selected by default (in settings). For example. you don't want to load 1080p video for popup player but you want to load 1080p for main video player. Right? Right. It requires to load different streams for different qualities when you switch from one player to another with different quality. The transition from one player to another could be instant without time gap only when two qualities are the same (but I didn't implemented this use-case too :) ).

[opusforlife2]

@avently Are you saying that the 360p exception makes it impossible to implement at all? Or that the transition can be made seamless for all resolutions except 360p? Because the second option seems fine for most cases.

Hmm. I'm thinking that one possible solution for popup/full screen seamless transition could be to set the same resolution for both popup and full screen as default. If the user sets different resolutions for both, a notice could be shown that it would prevent seamless switching from one to the other as the new resolution would have to be loaded. That leaves the user to make an informed decision.

By the way, I don't know why the Newpipe Extractor prefers these video+audio 360p and 720p streams over the separate video only streams which are available for both resolutions on many videos. This is the case for the downloader. I have no idea about streaming.

[avently]

the 360p exception makes it impossible to implement at all? Or that the transition can be made seamless for all resolutions except 360p?

The transition can be made seamless for all resolutions except some resolutions that have only audio+video tracks inside one stream. But to make it possible someone should write a code for it.

By the way, I don't know why the Newpipe Extractor prefers these video+audio 360p and 720p streams over the separate video only streams which are available for both resolutions on many videos

NewPipe has video format selectable from settings (MP4, WebM, 3GP). Some videos have only MP4 streams with different tracks for audio and video, some video have only WebM streams with different tracks. So this this not only 360p but more qualities can have missing video-only tracks per video format.

I have no idea why the video format can be selectable (who cares about it?).

[opusforlife2]

Some videos have only MP4 streams with different tracks for audio and video, some video have only WebM streams with different tracks. So this this not only 360p but more qualities can have missing video-only tracks per video format.

I'm talking about videos where all the tracks _are_ available, yet Newpipe Extractor chooses to download the larger files with inefficient encoding. If it chose them only when the corresponding video-only streams weren't available, there would be no issue, of course.

[bertin0]

@opusforlife2

Nope, popup player and video player have different qualities selected by default (in settings). For example. you don't want to load 1080p video for popup player but you want to load 1080p for main video player. Right? Right. It requires to load different streams for different qualities when you switch from one player to another with different quality. The transition from one player to another could be instant without time gap only when two qualities are the same (but I didn't implemented this use-case too :) ).

This would be a good compromise IMO. For example, I have enough bandwidth for 1080p all the time so I don't mind loading it in the popup player for the seamless transitions. ATM I sometimes use the popup player "maximised" (hold to fit width) just to avoid that gap, but it's a suboptimal solution as it doesn't have the controls.

Also regarding the miniplayer maximise behaviour, could you make it maximise to the max width only if the height doesn't excede 100% of the available space? For example, if I maximise it in landscape mode, the width becomes 100%, but that makes the height 100% too which is too big as the statusbar pushes it down.

[opusforlife2]

Alright. The way I see it, and I think I'm right based on what avently explained, the one major roadblock to seamless transitions is the combined video+audio streams, which leads me to believe that this issue will stay on the back burner until #2415 is fixed.

Once Newpipe gains the ability to _ignore_ the combined streams in favour of individual video and audio DASH streams (which always exist, to my knowledge, as I have never seen a video where yt-dl only reports combined streams and not their corresponding individual streams), then ways to make the transition seamless can be thought up.

My intuition says that it will be easier to make the transition _to_ background play _from_ the other two, because it involves pausing/discarding the video stream and continuing the audio stream, compared to transitioning _from_ background play _to_ the other two, which involves loading a video stream, probably causing a delay.

A way to make this _seem_ seamless (heh) could be to _delay_ the switch until the video stream has already loaded and is ready to play. So if the video stream takes 2 seconds to load, the audio should continue to play for those 2 seconds until it is ready to switch to popup/main, giving the _appearance_ of a seamless transition. But I'm not sure that users won't perceive it as a bug.

[avently]

@opusforlife2 @MD77MD
This video
https://www.youtube.com/watch?v=MHrsygIxC5k
has the following streams available:

360p format MPEG-4 videoOnly false
720p format MPEG-4 videoOnly false
1080p format MPEG-4 videoOnly true
1080p format WebM videoOnly true
1080p60 format MPEG-4 videoOnly true
1080p60 format WebM videoOnly true
720p format WebM videoOnly true
720p60 format MPEG-4 videoOnly true
720p60 format WebM videoOnly true
480p format MPEG-4 videoOnly true
480p format WebM videoOnly true
240p format MPEG-4 videoOnly true
240p format WebM videoOnly true
144p format MPEG-4 videoOnly true
144p format WebM videoOnly true

YouTubeVanced able to play without loading video in background. So I think YouTube team plays ALWAYS audio-only stream + video stream with muted sound (if it exists). For that video audio-only streams have the following sizes:
10,19 MB (128kbps)
4.11 MB (50kbps)
5.42 MB (70kbps)
10.44 MB (160kbps)

In order to have a seamless transition here NewPipe should load additional audio stream even for videostream with audio merged with it. It will add 4-5-10 MB to the video size. I don't think that this is too bad because the video will add much more MB compared to audio. Anyway, nothing changed since our (@opusforlife2) discussion and to make it possible someone needs to create two separate parallel playbacks using ExoPlayers features and MergingMediaSource and to mute sound for unneeded audio+video stream.

[wb9688]

@avently: YouTube has everything as a video-only stream and an audio-only stream, with the video+audio streams as a fallback for old devices (and someone thought it was a good idea to use those if available in NewPipe). I'm currently reworking the stream stuff in NewPipeExtractor, which will also let NewPipeExtractor return all available streams instead of just a subset, so that you could easily implement this feature. This stuff shouldn't be done until your unified player PR has been merged though.

[opusforlife2]

Yep. As I say here:

Once Newpipe gains the ability to ignore the combined streams in favour of individual video and audio DASH streams (which always exist, to my knowledge, as I have never seen a video where yt-dl only reports combined streams and not their corresponding individual streams), then ways to make the transition seamless can be thought up.

No matter which video URL I check (and I just confirmed it again with the URL @avently shared above), separate video only streams always exist for 360p and 720p.

[avently]

@wb9688 @opusforlife2
The move to DASH streams will require to remove all options regarding of video format. And this is great:)

[opusforlife2]

Why is that? youtube-dl can stream/download chosen formats perfectly well.

[wb9688]

@avently: No, I could split the DASH manifest into one DASH manifest for every itag

[avently]

@opusforlife2 I mean if NewPipe will support support Dash and seamless transition dash shouldn't be optional. It should be the only choice. Multiple choices will probably require more code to support both seamless and non-seamless transition which is not needed at all

[opusforlife2]

@avently That could cause problems. In my own case, I have enough bandwidth to easily stream 1080p60. However, if left up to Youtube to decide, the resolution is almost always downgraded to 360p or even 240p. Later it might upgrade to 480p or even 720p... if I'm lucky.

[avently]

@opusforlife2 I didn't mean to remove quality selector. Only to remove a format selector

[opusforlife2]

@avently Oh. Scared me for a bit, there, hehe.

Regarding format selector, as I say here #3159, it should be changed to a codec selector so that people can still choose between hardware decoded VP9 and (so far) software decoded AV1 for battery/thermal reasons.

[opusforlife2]

What about a toggle that lets the user continue playback in the same resolution regardless of the default ones? With a caveat mentioned below.