Node-ytdl-core: All videos broken

Created on 11 Sep 2019 · 26Comments · Source: fent/node-ytdl-core

Seems like YouTube has made a breaking change as of a few minutes ago. Every video ID that I've tried is broken in ytdl-core as well as youtube-dl and Lavaplayer. Interestingly, Invidious seems to have survived this change.

youtube-dl issue: https://github.com/ytdl-org/youtube-dl/issues/22367

> ytdl = require("ytdl-core")
...
> ytdl.getInfo("kJQP7kiw5Fk")
Promise { <pending> }
> (node:389040) UnhandledPromiseRejectionWarning: Error: This video is unavailable
     at /tmp/tmp.h8A12K3WoR/node_modules/ytdl-core/lib/info.js:255:16
     at /tmp/tmp.h8A12K3WoR/node_modules/ytdl-core/lib/info.js:373:9
     at /tmp/tmp.h8A12K3WoR/node_modules/ytdl-core/lib/info.js:198:5
     at PassThrough.<anonymous> (/tmp/tmp.h8A12K3WoR/node_modules/miniget/lib/index.js:195:30)
     at PassThrough.emit (events.js:209:13)
     at PassThrough.EventEmitter.emit (domain.js:499:23)
     at endReadableNT (_stream_readable.js:1178:12)
     at processTicksAndRejections (internal/process/task_queues.js:80:21)

bug it fixed itself

Source

cloudrac3r

👍15

Most helpful comment

    format.cipher = parseQueryString(format.cipher);

It should be querystring.parse(format.cipher);

Also formats have bitrate now (number), so we get an error here.

https://github.com/fent/node-ytdl-core/blob/e867cbd23ea91326b3bcd6c6e39bf50696d39465/lib/util.js#L37

And itags are also numbers in formats so chooseFormat does not work with itags.

https://github.com/fent/node-ytdl-core/blob/e867cbd23ea91326b3bcd6c6e39bf50696d39465/lib/util.js#L156

Quick fix:
```
formats = formats.concat(info.player_response.streamingData.formats);
formats = formats.concat(info.player_response.streamingData.adaptiveFormats);
formats.forEach(function(format){
if (format.cipher){
format.cipher = querystring.parse(format.cipher);
Object.assign(format, format.cipher);
}
format.type = format.mimeType;
format.quality_label = format.qualityLabel;
format.itag = format.itag.toString();
delete format.bitrate;
});
info.length_seconds = info.player_response.videoDetails.lengthSeconds;

WaqasIbrahim on 11 Sep 2019

👍8 ❤5 😄1

All 26 comments

A few reports of the same error from a few minutes ago are also in the thread for issue #371.

FD- on 11 Sep 2019

👍1

It seems like fmt_list is not longer included in the server info response.

FD- on 11 Sep 2019

Formats and adaptive formats are also available in palyerResponse. We can use that?

https://github.com/fent/node-ytdl-core/blob/a154822b532be9ef05056ba063d447d97d07008f/lib/info.js#L158

Not sure how reliable this data is, I mean does it always have the formats.

WaqasIbrahim on 11 Sep 2019

I have the same problem..

AuKllM on 11 Sep 2019

According to the test data files, player_response.streamingData used to just contain a property called "expiresInSeconds". Now it contains much more information:

{
    "expiresInSeconds": "21540",
    "formats": [
        {
            "itag": 18,
            "url": "https://r3---sn-h0jeen76.googlevideo.com/videoplayback?expire=1568236766&ei=fhB5XbfhG4is1wLoyoC4BQ&ip=85.127.253.109&id=o-AItoMUWZ4cStSVOiZ7z7_1VpTpDzQeIWergqsoT7RoW6&itag=18&source=youtube&requiressl=yes&mm=31%2C26&mn=sn-h0jeen76%2Csn-4g5e6nze&ms=au%2Conr&mv=m&mvi=2&pl=15&initcwndbps=1862500&mime=video%2Fmp4&gir=yes&clen=22828020&ratebypass=yes&dur=262.199&lmt=1392560250516306&mt=1568214966&fvip=3&c=WEB&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cmime%2Cgir%2Cclen%2Cratebypass%2Cdur%2Clmt&sig=ALgxI2wwRgIhAIBQdl7vu9ANBBYfG-XtBlzC2I815baeEa1GoSpV5LaCAiEAgMYaWZqOd9p9qajYkqAKJgtVhF_IaYwEn0YhzllL-18%3D&lsparams=mm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AHylml4wRQIgAJ_NtKjuGwlGHF1lvSj1seZggfh2B6d3edOP3e0BljwCIQDGIJv5EgtaoEJAbIeeTHpTrqfbnoiEcjdGsJzYG-ZO8Q%3D%3D",
            "mimeType": "video/mp4; codecs=\"avc1.42001E, mp4a.40.2\"",
            "bitrate": 696698,
            "width": 480,
            "height": 360,
            "lastModified": "1392560250516306",
            "contentLength": "22828020",
            "quality": "medium",
            "qualityLabel": "360p",
            "projectionType": "RECTANGULAR",
            "averageBitrate": 696509,
            "audioQuality": "AUDIO_QUALITY_LOW",
            "approxDurationMs": "262199",
            "audioSampleRate": "44100",
            "audioChannels": 1
        }
    ],
    "adaptiveFormats": [
        {
            "itag": 134,
            "url": "https://r3---sn-h0jeen76.googlevideo.com/videoplayback?expire=1568236766&ei=fhB5XbfhG4is1wLoyoC4BQ&ip=85.127.253.109&id=o-AItoMUWZ4cStSVOiZ7z7_1VpTpDzQeIWergqsoT7RoW6&itag=134&aitags=133%2C134%2C160&source=youtube&requiressl=yes&mm=31%2C26&mn=sn-h0jeen76%2Csn-4g5e6nze&ms=au%2Conr&mv=m&mvi=2&pl=15&initcwndbps=1862500&mime=video%2Fmp4&gir=yes&clen=15664974&dur=262.096&lmt=1392560231114616&mt=1568214966&fvip=3&keepalive=yes&c=WEB&sparams=expire%2Cei%2Cip%2Cid%2Caitags%2Csource%2Crequiressl%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&sig=ALgxI2wwRAIgAlKav-dUE-CZBsB0SLAkNZu8ml_ZP3lWHOg_s0iTVHMCIAPAC8W1Tpm9-NiR5EppFhtoDfrLduqtYDmSsvmc9zNi&lsparams=mm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AHylml4wRQIgAJ_NtKjuGwlGHF1lvSj1seZggfh2B6d3edOP3e0BljwCIQDGIJv5EgtaoEJAbIeeTHpTrqfbnoiEcjdGsJzYG-ZO8Q%3D%3D",
            "mimeType": "video/mp4; codecs=\"avc1.4d401e\"",
            "bitrate": 606491,
            "width": 480,
            "height": 360,
            "initRange": {
                "start": "0",
                "end": "708"
            },
            "indexRange": {
                "start": "709",
                "end": "1376"
            },
            "lastModified": "1392560231114616",
            "contentLength": "15664974",
            "quality": "medium",
            "fps": 30,
            "qualityLabel": "360p",
            "projectionType": "RECTANGULAR",
            "averageBitrate": 478144,
            "highReplication": true,
            "approxDurationMs": "262096"
        },
        {
            "itag": 133,
            "url": "https://r3---sn-h0jeen76.googlevideo.com/videoplayback?expire=1568236766&ei=fhB5XbfhG4is1wLoyoC4BQ&ip=85.127.253.109&id=o-AItoMUWZ4cStSVOiZ7z7_1VpTpDzQeIWergqsoT7RoW6&itag=133&aitags=133%2C134%2C160&source=youtube&requiressl=yes&mm=31%2C26&mn=sn-h0jeen76%2Csn-4g5e6nze&ms=au%2Conr&mv=m&mvi=2&pl=15&initcwndbps=1862500&mime=video%2Fmp4&gir=yes&clen=8000185&dur=262.096&lmt=1392560291567194&mt=1568214966&fvip=3&keepalive=yes&c=WEB&sparams=expire%2Cei%2Cip%2Cid%2Caitags%2Csource%2Crequiressl%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&sig=ALgxI2wwRAIgH07C9HGn7PDiq_g4c8qK48Rcm2jhanCmKo0KlTOSPM0CIFjFiY95ED3elszkagrnNtwXRmiNQ-Via6XbOgqiO1JR&lsparams=mm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AHylml4wRQIgAJ_NtKjuGwlGHF1lvSj1seZggfh2B6d3edOP3e0BljwCIQDGIJv5EgtaoEJAbIeeTHpTrqfbnoiEcjdGsJzYG-ZO8Q%3D%3D",
            "mimeType": "video/mp4; codecs=\"avc1.4d400d\"",
            "bitrate": 251222,
            "width": 320,
            "height": 240,
            "initRange": {
                "start": "0",
                "end": "671"
            },
            "indexRange": {
                "start": "672",
                "end": "1339"
            },
            "lastModified": "1392560291567194",
            "contentLength": "8000185",
            "quality": "small",
            "fps": 30,
            "qualityLabel": "240p",
            "projectionType": "RECTANGULAR",
            "averageBitrate": 244190,
            "approxDurationMs": "262096"
        },
        {
            "itag": 160,
            "url": "https://r3---sn-h0jeen76.googlevideo.com/videoplayback?expire=1568236766&ei=fhB5XbfhG4is1wLoyoC4BQ&ip=85.127.253.109&id=o-AItoMUWZ4cStSVOiZ7z7_1VpTpDzQeIWergqsoT7RoW6&itag=160&aitags=133%2C134%2C160&source=youtube&requiressl=yes&mm=31%2C26&mn=sn-h0jeen76%2Csn-4g5e6nze&ms=au%2Conr&mv=m&mvi=2&pl=15&initcwndbps=1862500&mime=video%2Fmp4&gir=yes&clen=3698377&dur=262.096&lmt=1392560205510693&mt=1568214966&fvip=3&keepalive=yes&c=WEB&sparams=expire%2Cei%2Cip%2Cid%2Caitags%2Csource%2Crequiressl%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&sig=ALgxI2wwRQIgftzDulcXjVq-xAn0j26hUY4Gmz0eojnNg08YvcXpcpoCIQCsTAt4QT_B2MTbotgdaxQw-xj-vd1zVwJhiIFR41Rt_w%3D%3D&lsparams=mm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AHylml4wRQIgAJ_NtKjuGwlGHF1lvSj1seZggfh2B6d3edOP3e0BljwCIQDGIJv5EgtaoEJAbIeeTHpTrqfbnoiEcjdGsJzYG-ZO8Q%3D%3D",
            "mimeType": "video/mp4; codecs=\"avc1.4d400c\"",
            "bitrate": 118901,
            "width": 192,
            "height": 144,
            "initRange": {
                "start": "0",
                "end": "672"
            },
            "indexRange": {
                "start": "673",
                "end": "1340"
            },
            "lastModified": "1392560205510693",
            "contentLength": "3698377",
            "quality": "tiny",
            "fps": 15,
            "qualityLabel": "144p",
            "projectionType": "RECTANGULAR",
            "averageBitrate": 112886,
            "approxDurationMs": "262096"
        },
        {
            "itag": 140,
            "url": "https://r3---sn-h0jeen76.googlevideo.com/videoplayback?expire=1568236766&ei=fhB5XbfhG4is1wLoyoC4BQ&ip=85.127.253.109&id=o-AItoMUWZ4cStSVOiZ7z7_1VpTpDzQeIWergqsoT7RoW6&itag=140&source=youtube&requiressl=yes&mm=31%2C26&mn=sn-h0jeen76%2Csn-4g5e6nze&ms=au%2Conr&mv=m&mvi=2&pl=15&initcwndbps=1862500&mime=audio%2Fmp4&gir=yes&clen=3116224&dur=262.199&lmt=1392560200450967&mt=1568214966&fvip=3&keepalive=yes&c=WEB&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&sig=ALgxI2wwRQIgDxUMLNu4aRY6MxWAlB8APnhWQdL0Ib3-bEKfv9s1z3UCIQD1x9awmo6QoqyBNQduRCyWI1IhjP_1wOvV8tGQiIBcLA%3D%3D&lsparams=mm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AHylml4wRQIgAJ_NtKjuGwlGHF1lvSj1seZggfh2B6d3edOP3e0BljwCIQDGIJv5EgtaoEJAbIeeTHpTrqfbnoiEcjdGsJzYG-ZO8Q%3D%3D",
            "mimeType": "audio/mp4; codecs=\"mp4a.40.2\"",
            "bitrate": 95952,
            "initRange": {
                "start": "0",
                "end": "591"
            },
            "indexRange": {
                "start": "592",
                "end": "947"
            },
            "lastModified": "1392560200450967",
            "contentLength": "3116224",
            "quality": "tiny",
            "projectionType": "RECTANGULAR",
            "averageBitrate": 95079,
            "highReplication": true,
            "audioQuality": "AUDIO_QUALITY_MEDIUM",
            "approxDurationMs": "262199",
            "audioSampleRate": "44100",
            "audioChannels": 1
        },
        {
            "itag": 251,
            "url": "https://r3---sn-h0jeen76.googlevideo.com/videoplayback?expire=1568236766&ei=fhB5XbfhG4is1wLoyoC4BQ&ip=85.127.253.109&id=o-AItoMUWZ4cStSVOiZ7z7_1VpTpDzQeIWergqsoT7RoW6&itag=251&source=youtube&requiressl=yes&mm=31%2C26&mn=sn-h0jeen76%2Csn-4g5e6nze&ms=au%2Conr&mv=m&mvi=2&pl=15&initcwndbps=1862500&mime=audio%2Fwebm&gir=yes&clen=3303678&dur=262.221&lmt=1565114931492150&mt=1568214966&fvip=3&keepalive=yes&c=WEB&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&sig=ALgxI2wwRQIgIOMc191XuYazkHIn90mmKg40ZuNm4t9PTX18ctLZv0cCIQD3wncti_OfYsKEnCxJVZitu4isQC4BBG15vntQOGJ7uA%3D%3D&lsparams=mm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AHylml4wRQIgAJ_NtKjuGwlGHF1lvSj1seZggfh2B6d3edOP3e0BljwCIQDGIJv5EgtaoEJAbIeeTHpTrqfbnoiEcjdGsJzYG-ZO8Q%3D%3D",
            "mimeType": "audio/webm; codecs=\"opus\"",
            "bitrate": 115903,
            "initRange": {
                "start": "0",
                "end": "258"
            },
            "indexRange": {
                "start": "259",
                "end": "714"
            },
            "lastModified": "1565114931492150",
            "contentLength": "3303678",
            "quality": "tiny",
            "projectionType": "RECTANGULAR",
            "averageBitrate": 100790,
            "audioQuality": "AUDIO_QUALITY_MEDIUM",
            "approxDurationMs": "262221",
            "audioSampleRate": "48000",
            "audioChannels": 1
        }
    ]
}

FD- on 11 Sep 2019

At least for the _HSylqgVYQI regular test case, player_response.streamingData.formats contains the same formats that were previously in fmt_list.

FD- on 11 Sep 2019

Yep, should be possible to pull all necessary info from streamingData.formats and streamingData.adaptiveFormats. For reference.

omarroth on 11 Sep 2019

The signature stuff is now included in each player_response.streamingData.adaptiveFormat's cipher field. I hope the s param is the same format as before.

FD- on 11 Sep 2019

@FD- Thanks, I was just about to ask. I can't find a video with signature data. Can you share a link?

WaqasIbrahim on 11 Sep 2019

@WaqasIbrahim It's from the vevo test case: https://www.youtube.com/watch?v=pJk0p-98Xzc

FD- on 11 Sep 2019

temp fix like this
ytdl-org/youtube-dl@bf1317d

a26364177 on 11 Sep 2019

@FD- Thanks again. Looks like cipher is a query string that has the url, sp and s. And signature deciphering is working fine with my testing.

WaqasIbrahim on 11 Sep 2019

Working fine for me as well. Actually, a quick fix is implemented in just a few lines in parseFormats (but I'm using a modified version of the library, so might be different for unmodified). All in all, a few fields were moved to different places and some were renamed to camel casing.

FD- on 11 Sep 2019

👍1

What I still don't know though:

Where are dash manifests now?
~~Are subtitles still in info.player_response.captions?~~ Yes, just as before.
~~Is the request to get_info still necessary now that we already get the streams in the config~~? Likely no longer needed

FD- on 11 Sep 2019

Indeed seems like the request to get_info can be removed (sorry, had a mistake in an update to my comment above). The only remaining reason seemed to be to check whether requires_purchase is set, but that property seems no longer working either. Instead, videos that require a purchase can be detected via player_response.playabilityStatus:

{"status":"UNPLAYABLE","reason":"This video requires payment to watch.","errorScreen":...}

FD- on 11 Sep 2019

A quick and dirty fix is to add this to the end of the parseFormats function:

formats = formats.concat(info.player_response.streamingData.formats);
formats = formats.concat(info.player_response.streamingData.adaptiveFormats);
formats.forEach(function(format){
    if (format.cipher){
        format.cipher = parseQueryString(format.cipher);
        Object.assign(format, format.cipher);
    }
    format.type = format.mimeType;
    format.quality_label = format.qualityLabel;
});
info.length_seconds = info.player_response.videoDetails.lengthSeconds;

Please note that this is in no way a permanent solution. The permanent fix likely requires modifications to quite a few places within the library.

FD- on 11 Sep 2019

👍9

    format.cipher = parseQueryString(format.cipher);

It should be querystring.parse(format.cipher);

Also formats have bitrate now (number), so we get an error here.

https://github.com/fent/node-ytdl-core/blob/e867cbd23ea91326b3bcd6c6e39bf50696d39465/lib/util.js#L37

And itags are also numbers in formats so chooseFormat does not work with itags.

https://github.com/fent/node-ytdl-core/blob/e867cbd23ea91326b3bcd6c6e39bf50696d39465/lib/util.js#L156

WaqasIbrahim on 11 Sep 2019

👍8 ❤5 😄1

@WaqasIbrahim this is the one that worked for me (for a Discord bot). Thank you!

JosefBud on 11 Sep 2019

@WaqasIbrahim Thanks for the improvements! As mentioned, I‘m using a modified version of node-ytdl-core, and didn’t test with the public version.

FD- on 11 Sep 2019

👍1

@FD- wdym modified version?

Roki100 on 11 Sep 2019

is this still an issue? just tested this on a few random videos and not getting any errors

fent on 12 Sep 2019

Indeed seems like the request to get_info can be removed (sorry, had a mistake in an update to my comment above). The only remaining reason seemed to be to check whether requires_purchase is set, but that property seems no longer working either. Instead, videos that require a purchase can be detected via player_response.playabilityStatus:

yeah I think it's safe to remove, most of what it was used for can be found in player_response.videoDetails

fent on 12 Sep 2019

@FD- wdym modified version?

I'm using it in a different custom runtime (not node), so had to remove the dependencies and rewrite some parts.

is this still an issue? just tested this on a few random videos and not getting any errors

Wow, adaptive_fmts and fmt_list are back again! Seems like player_response.streamingData still contains formats and adaptiveFormats though. No idea whether they are in any way better to use than adaptive_fmts and fmt_list.
Also seems like info.length_seconds is still gone.

FD- on 12 Sep 2019

@FD- wdym modified version?

I'm using it in a different custom runtime (not node), so had to remove the dependencies and rewrite some parts.

is this still an issue? just tested this on a few random videos and not getting any errors

Wow, adaptive_fmts and fmt_list are back again! Seems like player_response.streamingData still contains formats and adaptiveFormats though. No idea whether they are in any way better to use than adaptive_fmts and fmt_list.
Also seems like info.length_seconds is still gone.

streamingData is already parsed and contains content length and other useful info.

WaqasIbrahim on 12 Sep 2019

I'm no longer having problems with ytdl

yesterday yes I was in trouble

but not today

SergioGlorias on 12 Sep 2019

👍1

var videoPageContent = client.GetStringAsync(videoInfoUrl).Result;
var videoParameters = HttpUtility.ParseQueryString(videoPageContent);
var encodedStreamsDelimited1 = WebUtility.HtmlDecode(videoParameters["player_response"]);
JObject jObject = JObject.Parse(encodedStreamsDelimited1);
string url = (string) jObject["streamingData"]["formats"][0]["url"];
return VideoSource.FromUri(url);