Newpipe: No playback of certain videos, infinite loading for HLS/DASH streams

Created on 21 Jun 2019  Â·  66Comments  Â·  Source: TeamNewPipe/NewPipe

I am having issues playing certain videos, they all exhibit the same problem by sitting with the spinning wheel and nothing happening. If i select an external player, there is a pause followed by an error notification, which I have screencapped and says 'the location.......full screen url....cannot be played'. The fault occurs only on some resolutions, and it has been observed that in most cases the qualities 720p or 360p always work (e.g. https://youtu.be/zCGouRiflIw).

For example videos;
https://www.youtube.com/watch?v=2NT-MRkYeSo
All the recent videos on his channel exhibit the same issue.

https://www.youtube.com/watch?v=ZHeqOwV5bow
Some of the video on this channel play, some do not.

bug requires extractor change

Most helpful comment

@TobiGr may I pin this issue, so that people don't keep reporting the same thing?

All 66 comments

Why are there both "&sig=" and "&lsig="? I think the second one makes no sense, but I could be wrong. I'll investigate

Which version are u using?

0.16.2

It doesn't work for me either, but only on the WebM format. Here is the url I got from the extractor (I replaced my IP with X):

https://r16---sn-nx5cvox-hpae7.googlevideo.com/videoplayback?expire=1561147001&ei=GeIMXYzZF9bugAfg15b4Dw&ip=XX.XXX.XXX.X&id=o-AG0ApOUZOJ33f4ajMQAG7YiNYK1JvEH6BYveBSSKPGHM&itag=247&aitags=133%2C134%2C135%2C136%2C160%2C242%2C243%2C244%2C247%2C278&source=yt_otf&requiressl=yes&mm=31%2C26&mn=sn-nx5cvox-hpae7%2Csn-4g5e6nlk&ms=au%2Conr&mv=m&nh=EAE%2C&pcm2cms=yes&pl=19&initcwndbps=770000&mime=video%2Fwebm&otf=1&otfp=1&dur=0.000&lmt=1561056919842256&mt=1561125283&fvip=6&keepalive=yes&c=WEB&sparams=expire%2Cei%2Cip%2Cid%2Caitags%2Csource%2Crequiressl%2Cmime%2Cotf%2Cotfp%2Cdur%2Clmt&lsparams=mm%2Cmn%2Cms%2Cmv%2Cnh%2Cpcm2cms%2Cpl%2Cinitcwndbps&lsig=AHylml4wRQIhAOv1OJtRCSX85JZ-myiK-KrotoD6YcSE-2KeKUqWNJsNAiBLYXkNmxGviHtuyE_YXQbGW4lkbwkkxAwZ1OmZOoFqrg%3D%3D

Decoded:

https://r16---sn-nx5cvox-hpae7.googlevideo.com/videoplayback?expire=1561147001&ei=GeIMXYzZF9bugAfg15b4Dw&ip=XX.XXX.XXX.X&id=o-AG0ApOUZOJ33f4ajMQAG7YiNYK1JvEH6BYveBSSKPGHM&itag=247&aitags=133,134,135,136,160,242,243,244,247,278&source=yt_otf&requiressl=yes&mm=31,26&mn=sn-nx5cvox-hpae7,sn-4g5e6nlk&ms=au,onr&mv=m&nh=EAE,&pcm2cms=yes&pl=19&initcwndbps=770000&mime=video/webm&otf=1&otfp=1&dur=0.000&lmt=1561056919842256&mt=1561125283&fvip=6&keepalive=yes&c=WEB&sparams=expire,ei,ip,id,aitags,source,requiressl,mime,otf,otfp,dur,lmt&lsparams=mm,mn,ms,mv,nh,pcm2cms,pl,initcwndbps&lsig=AHylml4wRQIhAOv1OJtRCSX85JZ-myiK-KrotoD6YcSE-2KeKUqWNJsNAiBLYXkNmxGviHtuyE_YXQbGW4lkbwkkxAwZ1OmZOoFqrg==

Appears jdf76/plugin.video.youtube#624 fixes this by adding r'(?P<name>[a-zA-Z0-9$]+)\(', to the list of function patterns it uses.

I'd recommend changing DECYRYPTION_SIGNATURE_FUNCTION_REGEX to what Invidious uses (/^(?<name>[^=]+)=function\(a\){a=a\.split\(""\)/m) to see if it fixes the issue.

I can't convert that regex to the java dialect... this is the regex I tried:

"/^([^=]+)=function\\(a\\)\\{a=a\\.split\\(\"\"\\)/m"

I removed the ?<name> part since the java parser currently matches the first group (it should give the same resoult, shouldn't it?). Also I put a \ before the {, otherwise it would have given this error: PatternSyntaxException: Error in {min,max} interval near index 25, where 25 pointed to the = in {a=. The current error I get is RegexException: failed to find pattern "/^([^=]+)=function\(a\)\{a=a\.split\(""\)/m (I don't know why " is missing at the end)

Try removing the leading and trailing slashes (/, /m).

cannot resolve with "/^([^=]+)=function(a){a=a.split("")/m and
^([^=]+)=function(a){a=a.split("")

@TobiGr this calls for an emergency release shall i take care of release once it it fixed or will you?

By using only the first part of the regex ^([^=]+)=function\(a\)\{a=a\.split\(\" against playerCode I manually found two matches:

Aq=function(a){a=a.split("&");for(var b={},c=0,d=a.length;c<d;c++){var e=a[c].split("=");if(1==e.length&&e[0]||2==e.length)try{var f=Xc(e[0]||""),k=Xc(e[1]||"");f in b?g.Ka(b[f])?ib(b[f],k):b[f]=[b[f],k]:b[f]=k}catch(m){var l=Error("Error decoding URL component");l.params="key: "+e[0]+", value: "+e[1];"q"==e[0]?g.Vp(l):g.L(l)}}return b};
Dq=function(a){a=a.split(",");return a=a.map(function(b){return g.Cq(b)})};



md5-862d784712a7c60858e5654b045e7eae



```js
g.Wa=function(a,b,c){for(var d=a.length,e=g.Da(a)?a.split(""):a,f=0;f<d;f++)if(f in e&&b.call(c,e[f],f,a))return f;return-1};



md5-84ef471938c89f8d3ab8f710a486f6c0



```js
xi=function(a,b,c,d,e){if(null==a)return"";b=b||"&";c=c||",$";"string"==typeof c&&(c=c.split(""));if(a instanceof Array){if(d=d||0,d<c.length){for(var f=[],k=0;k<a.length;k++)f.push(xi(a[k],b,c,d+1,e));return f.join(c[d])}}else if("object"==typeof a)return e=e||0,2>e?encodeURIComponent(yi(a,b,c,d,e+1)):"...";return encodeURIComponent(String(a))};

I also tried this regex but it only works if I use the part before " ", as above.

@theScrabi That's a good question. I currently have little time due to upcoming exams. So it would be better if you take care of it. Nevertheless, we need a fix before we release something :D

I'll take care of it. @Stypox has a fix already been send to the extractor?

No, I'm sorry, but I wasn't able to get the regex to work (see comment above). Maybe @omarroth can help? :-)
I have playerCode in a plain text file, if needed.

Would you mind testing with omarroth/NewPipeExtractor@098b835 to see if it works as expected? Currently it appears NewPipe has no trouble with the existing regex, so please do provide the saved playerCode to test with.

omarroth/NewPipeExtractor@098b835 seems to have the same problems (Error 404)...
Here is the playerCode: playercode.txt. Strings are in italian language since I am in Italy, but I don't think that matters.
Hope not to bother you :)

I just tried downloading on invidious the same video and the only video format that can be downloaded is 360pMPEG4 (the same as NewPipe) so I don't think that new regex will fix this problem

Taking another look, the video in question is likely a livestream that was uploaded later as a normal video. Playback appears to work as expected, download I expect doesn't work since the file itself is split up into ~90 segments (from looking at the DASH manifest).

I am also having this problem, almost any music-related video stays forever in the spinner for me... (Note: background play doesn't work too.)

Example of video that plays: https://www.youtube.com/watch?v=kpk2tdsPh0A
Example of video that stays in the spinner forever: https://www.youtube.com/watch?v=JGwWNGJdvx8

I've been dealing with this problem for quite some time now. Nearly all music-related videos stay spinning, but no non-music videos do. Any updates?

@omarroth @Stypox have you tried to take a look at youtube-dl? Best guess is that it works there, if not we have ptoblems.

youtube-dl has this issue, too.
By doing youtube-dl --get-url --all-formats https://youtube.com/watch?v=umdZD8zRj4A I got 14 different urls, and I opened them all in Firefox:

  • 3 of them were working audio-only streams
  • 3 of them were working video-only streams
  • the other 8 urls did not work (error 404)

🤔 The 8 urls that do not work in the browser are DASH-videos, and they download fine with youtube-dl. By looking through the dash manifests I found an url that works in the browser. Is the NewPipeExtractor downloading dash manifests and parsing them?

After some hours of reading code I understand what's wrong. The extractor correctly extracts the urls, but those links lead to DASH playbacks. I will call those DASH urls "base url"s.

By adding &sq=0 to the base url we get the DASH "Initialization" url and when a request is sent to it the response is something along these lines (after decoding from UTF7-base64):

ftypdashiso6avc1mp41moovlmvhdXgXg_@(mvex trextraktkhdXgXg@mdia mdhdXgXg_U_hdlrvideISO Media file produced by Google Inc. Created on: 07/20/2019.2minf$dinfdrefurl stblstsdavc1HH0avcCM@gM@
(U@hsttsstscstcostszstssvmhdemsghttp://youtube.com/streaming/otf/durations/112015*Segment-Count: 55
Segment-Durations-Ms: 5339(r=53),2736,

There is some junk (probably UTF7-base64 is not the correct encoding), but some useful DASH information can be extracted anyway: Segment-Count tells us that there are 55 DASH fragments. This means that by adding &sq=N with 1<=N<=55 to the base url we can get the 55 fragments. YAY!

The base url works even after replacing all params separators (i.e. '&', '?', '=') with '/'. In this case the initialization url would be BaseUrl+"/" + "sq/0" and the fragment urls would be BaseUrl+"/" + "sq/N". This new url is required to build a valid XML file that has no unescaped characters like '&' (MPD uses XML).
Using this information I was able to build a valid MPD file (playable by vlc, although not seekable):

<?xml version="1.0" encoding="UTF-8"?>
<MPD xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="urn:mpeg:DASH:schema:MPD:2011" xmlns:yt="http://youtube.com/yt/2012/10/10" xsi:schemaLocation="urn:mpeg:DASH:schema:MPD:2011 DASH-MPD.xsd" minBufferTime="PT1.500S" profiles="urn:mpeg:dash:profile:isoff-main:2011" type="static" mediaPresentationDuration="PT291.131S">
    <Period>
        <AdaptationSet id="2" mimeType="video/mp4" subsegmentAlignment="true">
            <Representation id="135" codecs="avc1.4d4014" width="854" height="480" startWithSAP="1" maxPlayoutRate="1" bandwidth="1155000" frameRate="30">
                <BaseURL>DECODED_AND_ESCAPED_BASE_URL</BaseURL>
                <SegmentList>
                    <Initialization sourceURL="sq/0"/>
                    <SegmentURL media="sq/1"/>
                    <SegmentURL media="sq/2"/>
                    <SegmentURL media="sq/3"/>
                    <SegmentURL media="sq/4"/>
                    <SegmentURL media="sq/5"/>
                    <SegmentURL media="sq/6"/>
                    <SegmentURL media="sq/7"/>
                    <SegmentURL media="sq/8"/>
                    <SegmentURL media="sq/9"/>
                    <SegmentURL media="sq/10"/>
                    <SegmentURL media="sq/11"/>
                    <SegmentURL media="sq/12"/>
                    <SegmentURL media="sq/13"/>
                    <SegmentURL media="sq/14"/>
                    <SegmentURL media="sq/15"/>
                    <SegmentURL media="sq/16"/>
                    <SegmentURL media="sq/17"/>
                    <SegmentURL media="sq/18"/>
                    <SegmentURL media="sq/19"/>
                    <SegmentURL media="sq/20"/>
                    <SegmentURL media="sq/21"/>
                    <SegmentURL media="sq/22"/>
                    <SegmentURL media="sq/23"/>
                    <SegmentURL media="sq/24"/>
                    <SegmentURL media="sq/25"/>
                    <SegmentURL media="sq/26"/>
                    <SegmentURL media="sq/27"/>
                    <SegmentURL media="sq/28"/>
                    <SegmentURL media="sq/29"/>
                    <SegmentURL media="sq/30"/>
                    <SegmentURL media="sq/31"/>
                    <SegmentURL media="sq/32"/>
                    <SegmentURL media="sq/33"/>
                    <SegmentURL media="sq/34"/>
                    <SegmentURL media="sq/35"/>
                    <SegmentURL media="sq/36"/>
                    <SegmentURL media="sq/37"/>
                    <SegmentURL media="sq/38"/>
                    <SegmentURL media="sq/39"/>
                    <SegmentURL media="sq/40"/>
                    <SegmentURL media="sq/41"/>
                    <SegmentURL media="sq/42"/>
                    <SegmentURL media="sq/43"/>
                    <SegmentURL media="sq/44"/>
                    <SegmentURL media="sq/45"/>
                    <SegmentURL media="sq/46"/>
                    <SegmentURL media="sq/47"/>
                    <SegmentURL media="sq/48"/>
                    <SegmentURL media="sq/49"/>
                    <SegmentURL media="sq/50"/>
                    <SegmentURL media="sq/51"/>
                    <SegmentURL media="sq/52"/>
                    <SegmentURL media="sq/53"/>
                    <SegmentURL media="sq/54"/>
                    <SegmentURL media="sq/55"/>
                </SegmentList>
            </Representation>
        </AdaptationSet>
    </Period>
</MPD>

DECODED_AND_ESCAPED_BASE_URL is created by doing this (in pseudocode): baseUrl.decodeUrl().replace('&', '/').replace('?', '/').replace('=', '/').append('/')

Note: youtube-dl does something different with DASH files. It downloads the MPD file provided in ytPlayerConfig["args"]["player_response"]["streamingData"]["dashManifestUrl"], that contains all of the supported formats (even non-DASH ones, it seems like). That file is ready to be played by a DASH player: vlc opened it just fine. Then ytdl extracts the formats along with their segment data: every extracted DASH stream has:

  • the "BaseUrl": same as the base url from above, but in the XML-compatible format
  • the "Initialization" string always equal to sq/0 to append to the base url (same as above, but in the other format).
  • many SegmentURLs: they are sq/N/dur/DURATION. /dur/DURATION seems to be useless and can be removed, but ytdl does not do it. Notice the similarity with &sq=N from above?

When ytdl wants to download a DASH file it just downloads all the fragments by concatenating the base url with the url of the segment (it just ignores initialization).

@TobiGr @theScrabi @omarroth how should I proceed?
The ExoPlayer is able to play DASH MPD files, so we could just do the same procedure as above to generate a MPD file and feed it to the player.

But I don't think this can be implemented with the current NewPipeExtractor infrastructure, since a Stream can only provide an url.

An option would be to add the function getDashFile() to Stream (that returns the MPD file content or null) along with the boolean isDashStream() function.

Another option would be to treat DASH files as a completely different stream type and create a new DashStream class extending Stream, that provides getDashFile(). Would this be feasible? I don't think so, since NewPipe uses all stream instances through the base class.

Another option still would be to ignore DASH streams. Very few videos have this problem with DASH video streams, and as far as I know there is always an alternative to that (e.g. the video from @obfripper has the 360p format that works)

youtube-dl currently has a list of possible protocols and chooses the needed one based on the stream format.

@Stypox can I ask you to make NewPipe ignore them for now, until Dash is fully supported? It would avoid a lot of confusion and would fix backround playing for a lot of people.
Or this could be a option in the settings.

@selurvedu I was thinking about that, too, lately. I think we could do that, for now, yes ;-)
I will open a PR

Guys, this is a _critical_ bug that needs all eyes on it. It makes it so that _all_ resolutions except 2 or 3 don't work. At some point, I think other features/bug fixes should be suspended in favor of making this one the top priority.

@opusforlife2 I agree, I think getting playback and stability improvements need to be top priority. I've had this issue for multiple versions now.

Guys, this is a _critical_ bug that needs all eyes on it. It makes it so that _all_ resolutions except 2 or 3 don't work. At some point, I think other features/bug fixes should be suspended in favor of making this one the top priority.

I agree and I've had this bug since 14.x at the very least. We should sticky this bug!

I've noticed that 1 way to work around this bug is to use the video downloader to see which videos show filesize info, which indicates which videos actually work, but this bug should be our number 1 priority. @Stypox @theScrabi @TobiGr please make this our top priority ASAP.

@Pentaphon That is exactly what I do to check if it's a temporary problem with my internet or if the resolutions other than 360p mp4 actually aren't loading.

I'm very surprised this bug wasn't a release blocker when it was discovered with all the work Stypox did in investigating it.

I'm very surprised this bug wasn't a release blocker when it was discovered with all the work Stypox did in investigating it.

I hope it becomes a release blocker for 0.20.0 at the very least. I see that 19.3 is already stickied so we should sticky this bug when that sticky comes down.

There are already PRs in the 0.19.4 and 0.20.0 milestones. The former seems to be a spillover from 0.19.3, and 0.20.0 has the ginormous unified player UI which will probably take at least a month or two to hammer out.

@TobiGr may I pin this issue, so that people don't keep reporting the same thing?

@TobiGr may I pin this issue, so that people don't keep reporting the same thing?

That would be a great idea as #3497 was opened today about this same issue/closed.

Maybe as an intermediate fix would it be possible to display an error indicating that the video which can't be played doesn't support the user's default resolution?

Currently as you know, there's no hint as to what's going on when the video doesn't load and just the spinner icon is visible due to video not available for default resolution.

@Stypox,
If I understand this PR correctly, it applies to multiple video formats, so maybe add them to the title or remove WebM from the title so it'll be easier for users to know if they are searching for this issue/see the title.

Anyone is getting random error "Could not play this stream" while watching certain video

This is the video i'm watching in 720p https://youtube.com/watch?v=51-AUvNwOwQ

PS. My issue has been referenced here

@Stypox or other devs,
How is the stream type NewPipe uses to stream a video chosen between HLS and DASH? Is it a function of the default video/audio format users pick in NewPipe settings?

Wasn't even familiar with HLS or DASH streams prior to PR title changing, so just trying to learn a bit here.

How is the stream type NewPipe uses to stream a video chosen between HLS and DASH?

I'm not even sure why this issue happens considering that it doesn't seem to happen to youtube-dl

I believe I had issues with one of these videos on youtube-dl's feburary version aswell but didn't investigate further at the time.

Currently HLS streams are being handled as if they were normal streams, so obviously they do not work: when the player makes a request to the stream url a 404 is returned because HLS needs additional arguments to be added to the url. See https://github.com/TeamNewPipe/NewPipe/issues/2415#issuecomment-515543289

We should simply use a HlsMediaSource or DashMediaSource…

Edit: I'm working on at least the extractor-side of things.

Note: youtube-dl does something different with DASH files. It downloads the MPD file provided in ytPlayerConfig["args"]["player_response"]["streamingData"]["dashManifestUrl"], that contains all of the supported formats (even non-DASH ones, it seems like). That file is ready to be played by a DASH player: vlc opened it just fine. Then ytdl extracts the formats along with their segment data: every extracted DASH stream has:

@Stypox: That's not really accurate, as there are more than enough videos with no DASH MPD on YouTube.

Would it potentially be a small change to identify cases where this is the reason load is failing, and show an appropriate error message? Or would that be harder than just fixing the issue entirely?

I don’t know if this is the right place but this video https://youtu.be/zCGouRiflIw cannot be played and cannot be downloaded, only the quality 720p or 360p can be downloaded and played.

@Stypox could you edit the issue to state this fact? I just realised that all our info is from multiple issues and people, but this pinned issue doesn't contain it in one place. The major noticeable symptom, that resolutions other than 720 and 360 don't play, isn't even mentioned in the OP.

You have a point, done. ;-)
Don't you now have the permission to do that?

Nyope.

Uhm. You should. I'll change that later

Been having more than usual issues the past week or two with videos just refusing to play, seems like more videos don't want to play than normal

Funnily, all of the alternative apps I found have the same issue too.

It works fine on youtube-dl on Windows.

TY is migrating all videos to DASH/HLS format which is more data-efficient. starting from newer uploads and going back to older uploads. the classic "one a/v file" is becoming more rare.

edit: ytdl handles HLS fairly well. DASH (of live events) not so much (but with plenty of fallbacks using both native, ffmpeg or aria2c when available). ffmpeg does most (all) of the heavy lifting, both downloading and muxing when possible. NP does not uses (as a directive decision) ffmpeg, nor going for solutions that ends with the need to encode (just mux'ing).

For some reason 1080p is now available on videos with DASH/HLS format whatever, so now we have 1080p, 720p and 360p

Would it not be possible to give an option to use ytdl on our backend?

Would it not be possible to give an option to use ytdl on our backend?

I wish Newpipe would just use yt-dl and let us update it ourselves instead of having to maintain their own extractor

I appreciate the team using their own extractor, it also gives room to improve on it.

@Pentaphon I agree, but I'm not entirely sure if there's a specific reason why they choose not to do that, because it seems like it'd be the easiest.
That being said, I still think it would be a good option if we could have it. At the very least as an option. Even if we like Newpipes extractor, it's clear it's not stable. Maybe adding the option for ytdl if one chooses, or as a fallback.

I appreciate the team using their own extractor, it also gives room to improve on it.

Or everybody with extraction skills can just focus on improving youtube-dl while people with other skills work on Newpipe as a yt-dl frontend only. Every time Google breaks Newpipe, yt-dl fixes it first and then we have to wait for Newpipe to basically adapt the fix to the extractor. It makes way more sense to just download the latest yt-dl to Newpipe instead of rolling out a new version of Newpipe every time something breaks.

I've raised the topic here: https://github.com/TeamNewPipe/NewPipe/issues/4202

Hopefully, enough people can agree that we are just creating more work and the project can turn to that direction.

mpv-android is adding youtube-dl support. You can use that as an external player now.

@opusforlife2 It would be nice however not needing to use an external player.

Wouldn't it be hard to implement yt-dl for Android since they are written in python?

mpv-android is adding youtube-dl support. You can use that as an external player now.

Correction: it's actually been available since _2017_ or so. The apk is available in the PR. It's just that the NanoDroid version of mpv-android was updated to include youtube-dl, which is where I learned of it.

In v0.20.0, we still won't be able to play those streams, but the non-playing streams won't be shown in the app.

Which means you won't have to get irritated, then try the 1080p stream, see if it works or not, then try 360p.

I haven't experienced this issue in quite a while eventhough I regularly watch smaller channels (views in the tens or hundreds) and this used to happen very often.

The videos in this thread that are still available and some other ones I remember not working play just fine now.

Is anyone still able to reproduce?

I've noticed that the streams are now the 1080p h264 versions eventhough vp9 ones are available according to ytdl but, while not optimal, it's a rather minor issue compared to not playing at all.

@Atemu this issue was fixed in 0.20.0, but only as a workaround, i.e. detecting the failing streams and hiding them. A real fix, allowing to play dash streams, needs to be implemented.

I keep saying to close this and open a new issue for implementing DASH playback. <(`^´)>

I keep saying to close this and open a new issue for implementing DASH playback. <(`^´)>

Or just edit the title to say "Implement DASH playback"

Was this page helpful?
0 / 5 - 0 ratings

Related issues

probonopd picture probonopd  Â·  3Comments

Hunter9888x picture Hunter9888x  Â·  3Comments

android1973 picture android1973  Â·  3Comments

cavemandaveman picture cavemandaveman  Â·  3Comments

ghost picture ghost  Â·  3Comments