Newpipe: newpipe can't load any youtube link and show no result for any search

[mhdzli]

Version

-0.19.8

Steps to reproduce the bug

I search for anything in newpipe and there is no result with a sorry some error occurred message.

If I open any youtube link in newpipe it can't load the page and info.

Screenshots/Screen recordings

Logs

Exception

• __User Action:__ searched
• __Request:__ swaywm
• __Content Country:__ US
• __Content Language:__ en-US
• __App Language:__ en_US
• __Service:__ YouTube
• __Version:__ 0.19.8
• __OS:__ Linux samsung/a6pltedx/a6plte:10/QP1A.190711.020/A605GDXU6CTC9:user/release-keys 10 - 29
  
  Crash log

org.schabi.newpipe.extractor.exceptions.ParsingException: Could not get ytInitialData
    at org.schabi.newpipe.extractor.services.youtube.YoutubeParsingHelper.getInitialData(YoutubeParsingHelper.java:203)
    at org.schabi.newpipe.extractor.services.youtube.YoutubeParsingHelper.extractClientVersionAndKey(YoutubeParsingHelper.java:222)
    at org.schabi.newpipe.extractor.services.youtube.YoutubeParsingHelper.getKey(YoutubeParsingHelper.java:298)
    at org.schabi.newpipe.extractor.services.youtube.extractors.YoutubeSearchExtractor.getNewNextPageFrom(YoutubeSearchExtractor.java:233)
    at org.schabi.newpipe.extractor.services.youtube.extractors.YoutubeSearchExtractor.getInitialPage(YoutubeSearchExtractor.java:127)
    at org.schabi.newpipe.extractor.utils.ExtractorHelper.getItemsPageOrLogError(ExtractorHelper.java:19)
    at org.schabi.newpipe.extractor.search.SearchInfo.getInfo(SearchInfo.java:55)
    at org.schabi.newpipe.extractor.search.SearchInfo.getInfo(SearchInfo.java:30)
    at org.schabi.newpipe.util.ExtractorHelper.lambda$searchFor$0(ExtractorHelper.java:81)
    at org.schabi.newpipe.util.-$$Lambda$ExtractorHelper$BBduYDeZ_vXMQYaemaggmTPtqvA.call(Unknown Source:8)
    at io.reactivex.internal.operators.single.SingleFromCallable.subscribeActual(SingleFromCallable.java:44)
    at io.reactivex.Single.subscribe(Single.java:3666)
    at io.reactivex.internal.operators.single.SingleSubscribeOn$SubscribeOnObserver.run(SingleSubscribeOn.java:89)
    at io.reactivex.Scheduler$DisposeTask.run(Scheduler.java:578)
    at io.reactivex.internal.schedulers.ScheduledRunnable.run(ScheduledRunnable.java:66)
    at io.reactivex.internal.schedulers.ScheduledRunnable.call(ScheduledRunnable.java:57)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:301)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
    at java.lang.Thread.run(Thread.java:919)
Caused by: org.schabi.newpipe.extractor.utils.Parser$RegexException: failed to find pattern "window\["ytInitialData"\]\s*=\s*(\{.*?\});
    at org.schabi.newpipe.extractor.utils.Parser.matchGroup(Parser.java:72)
    at org.schabi.newpipe.extractor.utils.Parser.matchGroup(Parser.java:61)
    at org.schabi.newpipe.extractor.utils.Parser.matchGroup1(Parser.java:52)
    at org.schabi.newpipe.extractor.services.youtube.YoutubeParsingHelper.getInitialData(YoutubeParsingHelper.java:200)
    ... 20 more

---

[opusforlife2]

Interesting! I've never seen that error message before.

[sandeeptardigrade]

this is so frustrating. I am also facing the same error from about a month , before that it was working fine. my phone oneplus 5

[mhdzli]

@sandeeptardigrade
I just delete all newpipe data and it works for a while till this happen again ¯_(ツ)_/¯

[wb9688]

If you could still reproduce it, could you install a debug version, connect your phone to your PC through USB, open Chrome, go to chrome://inspect, open that debug version of NewPipe on your phone, select that NewPipe instance in Chrome, go to the Network tab, try to reproduce the issue, and once you've successfully reproduced the issue, right-click in Chrome, select "Save as HAR with Content" and send me that HAR? Then I might be able to fix this issue.

[mhdzli]

@wb9688
Absolutely sure! I will be happy if I can be of any help.

[wb9688]

@mhdzli: You could download a debug apk at https://github.com/TeamNewPipe/NewPipe/files/5083926/app-debug.zip (it will install side-by-side)

You'll probably need to export database (in Settings -> Content) from your normal NewPipe and import it into that debugging NewPipe to be able to reproduce it.

Btw have you at any point got a reCAPTCHA in NewPipe?

[mhdzli]

@wb9688
Yes I got reCAPCHA every now and then. And thank you and all of newpipe team for this great app.

[vomindoraan]

I am having the same issues. Homepage, search and video preview pages all show “Network error” and don't work.

The issues first started occurring about a week ago, then the app went back to normal for a few days, and then it started acting up again yesterday. Both times it started happening around the time I reinstalled my browser app (Firefox), but I doubt that's related.

Edit: Restarting the phone and reinstalling the app did not fix the issue.

Screenshots:

Version info:
NewPipe v0.19.8
Android 7.1.1
SM-J510FN

[opusforlife2]

@vomindoraan Could you try this https://github.com/TeamNewPipe/NewPipe/issues/4201#issuecomment-683668294? Thanks.

[vomindoraan]

@opusforlife2 I will try that later tonight.

[Tybalta]

I have the same issue...

[opusforlife2]

@Tybalta Could you try this https://github.com/TeamNewPipe/NewPipe/issues/4201#issuecomment-683668294? Thanks.

[vomindoraan]

@opusforlife2 I tried reproducing the issue with the debug app several times, like you suggested, but every time the problem seemed to magically go away, just to come back the next day.

In the end I realized that the issue only seemed to occur when I was using NewPipe on mobile data. As soon as I connected to Wi-Fi, the app would work like normal again. Since my phone was always connected to my home Wi-Fi network whenever I tried debugging with Chrome, I couldn't reproduce the issue.

@Tybalta @mhdzli Can you guys confirm if the app works for you on Wi-Fi?

[opusforlife2]

@vomindoraan That's good to know! So can you try this using mobile hotspot?

[vomindoraan]

Here's a screen recording that demonstrates this.
Edit: Apologies, it looks like Imgur limited the recording to 30 sec. I will upload it to a different service.
Edit2: Reuploaded, the link should be good now.

@opusforlife2 I will try it right away.

[vomindoraan]

https://www.dropbox.com/s/9rygt6hqx3ldyhn/newpipe_networkerror_har.zip?dl=1

The above archive contains three HARs:

1. a snapshot after opening the app (while on mobile data), doing a search and getting a network error;
2. a snapshot after doing a second search and getting another network error;
3. a snapshot after opening the preview page for a video and getting a third network error.

[opusforlife2]

Thanks!

[Tybalta]

@Tybalta Could you try this #4201 (comment)? Thanks.
@opusforlife2
I dont know how to do that in chrome? Which chrome? Phone or Pc chrome?

[opusforlife2]

That's okay. We have got HARs above. If none of them helps solve the problem, then we could ask for them again.

Chrome on PC, by the way.

[mhdzli]

@wb9688
I installed the debug version and tried reproducing it several times, but everything works fine and the problem just went away. I did everything came into my mind. If you can think of anything that I should consider give me the instruction and I'll try again.

[vomindoraan]

I am still experiencing this issue on v0.20.0 when not connected to Wi-Fi.

[Stypox]

The above archive contains three HARs:

Unfortunately I couldn't find any useful information in those HARs: they contain only a bunch of successful search suggestion requests and one successful comment request, so for some reason there is nothing that could cause an error

[Stypox]

After thinking about it, this is probably cause by a REAL network error: the fact that no failing requests appear in the HARs led me to thinking that those requests might not have been made in the first place. Are you sure you don't have something on your phone which blocks some requests? For now I'm closing, since even though the issue is there I don't think there is a way to fix it unless there is more technical information

[vomindoraan]

I have had no similar issues with any applications other than NewPipe v0.19.8 and v0.20.0. To the best of my knowledge, there is nothing on my phone that could be blocking requests. Also, the fact that it only breaks on mobile data while working fine on Wi-Fi, and the fact that other users are experiencing similar problems, leads me to believe that the issue is on NewPipe's end.

If the requests in the HARs were successful, how come the application reported network errors having occurred?

[Stypox]

The HARs just do not contain the failing requests: those are probably blocked by something else before they are even sent. This is then probably caused by the outdated OkHttp version we have to use in order to support Android 4.4

[opusforlife2]

I'm more and more convinced that this issue is caused by some error or corruption during database import.

[mrlinnth]

Working fine until few days ago. I didn't do any import or anything.

Below is the log.

Exception

• __User Action:__ searched
• __Request:__ alan watts
• __Content Country:__ US
• __Content Language:__ en-US
• __App Language:__ en_US
• __Service:__ YouTube
• __Version:__ 0.20.0
• __OS:__ Linux Android 10 - 29
  
  Crash log

org.schabi.newpipe.extractor.exceptions.ParsingException: Could not get ytInitialData
    at org.schabi.newpipe.extractor.services.youtube.YoutubeParsingHelper.getInitialData(YoutubeParsingHelper.java:203)
    at org.schabi.newpipe.extractor.services.youtube.YoutubeParsingHelper.extractClientVersionAndKey(YoutubeParsingHelper.java:222)
    at org.schabi.newpipe.extractor.services.youtube.YoutubeParsingHelper.getKey(YoutubeParsingHelper.java:298)
    at org.schabi.newpipe.extractor.services.youtube.extractors.YoutubeSearchExtractor.getNewNextPageFrom(YoutubeSearchExtractor.java:233)
    at org.schabi.newpipe.extractor.services.youtube.extractors.YoutubeSearchExtractor.getInitialPage(YoutubeSearchExtractor.java:127)
    at org.schabi.newpipe.extractor.utils.ExtractorHelper.getItemsPageOrLogError(ExtractorHelper.java:19)
    at org.schabi.newpipe.extractor.search.SearchInfo.getInfo(SearchInfo.java:55)
    at org.schabi.newpipe.extractor.search.SearchInfo.getInfo(SearchInfo.java:30)
    at org.schabi.newpipe.util.ExtractorHelper.lambda$searchFor$0(ExtractorHelper.java:81)
    at org.schabi.newpipe.util.-$$Lambda$ExtractorHelper$BBduYDeZ_vXMQYaemaggmTPtqvA.call(Unknown Source:8)
    at io.reactivex.internal.operators.single.SingleFromCallable.subscribeActual(SingleFromCallable.java:44)
    at io.reactivex.Single.subscribe(Single.java:3666)
    at io.reactivex.internal.operators.single.SingleSubscribeOn$SubscribeOnObserver.run(SingleSubscribeOn.java:89)
    at io.reactivex.Scheduler$DisposeTask.run(Scheduler.java:578)
    at io.reactivex.internal.schedulers.ScheduledRunnable.run(ScheduledRunnable.java:66)
    at io.reactivex.internal.schedulers.ScheduledRunnable.call(ScheduledRunnable.java:57)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:301)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
    at java.lang.Thread.run(Thread.java:919)
Caused by: org.schabi.newpipe.extractor.utils.Parser$RegexException: failed to find pattern "window\["ytInitialData"\]\s*=\s*(\{.*?\});
    at org.schabi.newpipe.extractor.utils.Parser.matchGroup(Parser.java:72)
    at org.schabi.newpipe.extractor.utils.Parser.matchGroup(Parser.java:61)
    at org.schabi.newpipe.extractor.utils.Parser.matchGroup1(Parser.java:52)
    at org.schabi.newpipe.extractor.services.youtube.YoutubeParsingHelper.getInitialData(YoutubeParsingHelper.java:200)
    ... 20 more

---

[opusforlife2]

Oh hey! I just got this error when searching. And on Wi-Fi. Yay!

Only got it the once, though. Same search worked a second time. :/

[ghost]

Just now facing this issue. Using latest 0.20.0 git release.

None of search result query fetched always shows no result. Tried searching multiple times.

Here is the LOGS:
https://del.dog/icyporimyp

[Freepony]

For me the issue started yesterday and this is what I could figure out. First I cleaned up app cache and that didn't work. Then I made data backup (settings - content), cleaned up app data and restored backup WITH restoring app settings. That didn't work either. Deleting metadata, search history and so on in History and Search menu did no good.
So I again cleaned up app data, restored backup WITHOUT restoring app settings and it worked. The search now works and I could save all the subscriptions and search results which is important for me. But still I have no idea what setting causes the error.

[Stypox]

Here is the two network files causing the following crash @wb9688 network.zip

Exception

• __User Action:__ searched
• __Request:__ fsd
• __Content Country:__ US
• __Content Language:__ en-US
• __App Language:__ en_US
• __Service:__ YouTube
• __Version:__ 0.20.0
• __OS:__ Linux Android 10 - 29
  
  Crash log

org.schabi.newpipe.extractor.exceptions.ParsingException: Could not get ytInitialData
    at org.schabi.newpipe.extractor.services.youtube.YoutubeParsingHelper.getInitialData(YoutubeParsingHelper.java:203)
    at org.schabi.newpipe.extractor.services.youtube.YoutubeParsingHelper.extractClientVersionAndKey(YoutubeParsingHelper.java:222)
    at org.schabi.newpipe.extractor.services.youtube.YoutubeParsingHelper.getKey(YoutubeParsingHelper.java:298)
    at org.schabi.newpipe.extractor.services.youtube.extractors.YoutubeSearchExtractor.getNewNextPageFrom(YoutubeSearchExtractor.java:233)
    at org.schabi.newpipe.extractor.services.youtube.extractors.YoutubeSearchExtractor.getInitialPage(YoutubeSearchExtractor.java:127)
    at org.schabi.newpipe.extractor.utils.ExtractorHelper.getItemsPageOrLogError(ExtractorHelper.java:19)
    at org.schabi.newpipe.extractor.search.SearchInfo.getInfo(SearchInfo.java:55)
    at org.schabi.newpipe.extractor.search.SearchInfo.getInfo(SearchInfo.java:30)
    at org.schabi.newpipe.util.ExtractorHelper.lambda$searchFor$0(ExtractorHelper.java:81)
    at org.schabi.newpipe.util.-$$Lambda$ExtractorHelper$qyxpuXgomWa-cbONQns-pd7zxm0.call(Unknown Source:8)
    at io.reactivex.internal.operators.single.SingleFromCallable.subscribeActual(SingleFromCallable.java:44)
    at io.reactivex.Single.subscribe(Single.java:3666)
    at io.reactivex.internal.operators.single.SingleSubscribeOn$SubscribeOnObserver.run(SingleSubscribeOn.java:89)
    at io.reactivex.Scheduler$DisposeTask.run(Scheduler.java:578)
    at io.reactivex.internal.schedulers.ScheduledRunnable.run(ScheduledRunnable.java:66)
    at io.reactivex.internal.schedulers.ScheduledRunnable.call(ScheduledRunnable.java:57)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:301)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
    at java.lang.Thread.run(Thread.java:919)
Caused by: org.schabi.newpipe.extractor.utils.Parser$RegexException: failed to find pattern "window\["ytInitialData"\]\s*=\s*(\{.*?\});
    at org.schabi.newpipe.extractor.utils.Parser.matchGroup(Parser.java:72)
    at org.schabi.newpipe.extractor.utils.Parser.matchGroup(Parser.java:61)
    at org.schabi.newpipe.extractor.utils.Parser.matchGroup1(Parser.java:52)
    at org.schabi.newpipe.extractor.services.youtube.YoutubeParsingHelper.getInitialData(YoutubeParsingHelper.java:200)
    ... 20 more

---

[TobiGr]

Please test the APK and tell us if the problem still persists: app-debug.zip

[wb9688]

Could you guys please test the following APK? app-debug.apk.zip

[mehedihasanziku]

Please test the APK and tell us if the problem still persists: app-debug.zip

Just tested seems working fine.

Step:
Installed > Import Database from release version > Searched Multiple times, working fine

[opusforlife2]

@mehedihasanziku We don't need the screenshots. Your word is enough.

[EliasKomar]

so i had the same issue the past couple of days and I was wondering what I should do now. Will a fix be released or should I use the apk that you provided?

[opusforlife2]

@EliasKomar It's already released.

[EliasKomar]

I'm not sure where to put this question, but i can't install the apk from the website and f-droid warns me that the version is signed with a different key and that I would have to delete the old app (I would lose all the data) in order to install it. Why did the key change and how can I transition without losing my data?

[opusforlife2]

@EliasKomar Export database, uninstall the app, install the new version, import database.

[wb9688]

@EliasKomar: If you've downloaded it from F-Droid, it's possible you downloaded the version that was signed by F-Droid. On GitHub, and on our F-Droid repo we of course sign it with our own keys. Normal F-Droid nowadays has an .apk signed by us and one signed by them.

[EliasKomar]

Thanks, it works now and I am going to use the NewPipe signed version from now on since I trust the project :)

[opusforlife2]

@EliasKomar Since Newpipe is reproducible, F-Droid's server also signs using Newpipe's key, which means your trust is verified as well. ;)

[vomindoraan]

I was able to sidestep this issue by changing my DNS resolver to 1.1.1.1 using the Cloudflare 1.1.1.1 app from the Play Store. Only after doing this has NewPipe v0.20.2 started working over mobile data on my Android 7.1.1 phone.

The issue seems to have occurred because www.youtube.com was inaccessible, whereas m.youtube.com was working fine.

[mehedihasanziku]

I was able to sidestep this issue by changing my DNS resolver to 1.1.1.1 using the Cloudflare 1.1.1.1 app from the Play Store. Only after doing this has NewPipe v0.20.2 started working over mobile data on my Android 7.1.1 phone.

The issue seems to have occurred because www.youtube.com was inaccessible, whereas m.youtube.com was working fine.

It was fixed on last version and it was captcha cookie issue, so your issue seems to me different.