Node-ytdl-core: Error: input stream: Status code: 429

[Harmiox]

Everything was working fine earlier today and yesterday until all of a sudden I started getting this error multiple time in my log file. I've never had a 429 response with this lib before. (Edit: This issue only happens on my VPS, it doesn't happen on my local machine.)

_Running [email protected]_

import ytdl from 'ytdl-core';
const ytdlOptions: {} = { filter: 'audioonly', quality: 'highestaudio' };
ytdl('https://youtube.com/watch?v=O6RyKbcpBfw', ytdlOptions);

Error: input stream: Status code: 429
    at ClientRequest.httpLib.get (/root/discord/music/poco/node_modules/miniget/lib/index.js:125:19)
    at Object.onceWrapper (events.js:286:20)
    at ClientRequest.emit (events.js:198:13)
    at HTTPParser.parserOnIncomingClient [as onIncoming] (_http_client.js:556:21)
    at HTTPParser.parserOnHeadersComplete (_http_common.js:109:17)
    at TLSSocket.socketOnData (_http_client.js:442:20)
    at TLSSocket.emit (events.js:198:13)
    at addChunk (_stream_readable.js:288:12)
    at readableAddChunk (_stream_readable.js:269:11)
    at TLSSocket.Readable.push (_stream_readable.js:224:10)

[encloinc]

Same thing is happening to me!

[encloinc]

What seems to happen is, like you im using a vps, it stops working after like a few requests.

[fent]

this status means that too many requests are being made too quickly

https://httpstatuses.com/429

I recently updated miniget, the http library that this library uses. the reconnect logic was refactored and moved around. so at first i thought that might be the issue. but in #445, @encloinc is using [email protected], which uses the old version of miniget.

@encloinc, can you run npm ls miniget and see what version it prints?

could also be that youtube changed their website, adding more rate limiting.

[encloinc]

-- [email protected] +-- [email protected] | `-- [email protected] deduped `-- [email protected]

[encloinc]

I dont know if its a rate limiting issue though, because when I restart the vps it works again for a few requests. Its quite weird. (And I wouldnt be like surpassing the limit too much, my vps makes no more than a request every 3 minutes)

[fent]

if youtube is rate limiting requests more, one solution can be to

-- [email protected]
+-- [email protected]
| -- [email protected] deduped -- [email protected]```

ok thanks. that means this issue is originating from a change from youtube, rather than the recent update.

I dont know if its a rate limiting issue though, because when I restart the vps it works again for a few requests.

it's working again because there was some time when your vps didn't make any requests, not because it was restarted.

And I wouldnt be like surpassing the limit too much, my vps makes no more than a request every 3 minutes

is this for a bot that's shared in a server? does it get a lot of requests?

[encloinc]

if youtube is rate limiting requests more, one solution can be to

-- [email protected]
+-- [email protected]
| -- [email protected] deduped-- [email protected]```

ok thanks. that means this issue is originating from a change from youtube, rather than the recent update.

I dont know if its a rate limiting issue though, because when I restart the vps it works again for a few requests.

it's working again because there was some time when your vps didn't make any requests, not because it was restarted.

And I wouldnt be like surpassing the limit too much, my vps makes no more than a request every 3 minutes

is this for a bot that's shared in a server? does it get a lot of requests?

It is a youtube-mp3 converter site, and it doesnt really. Also when I restart the vps it restarts in like 5 minutes. All I do is restart the script.

[HcgRandon]

Same here, And I'm on an older version. Def a YouTube change. Hopefully there is a workaround

[encloinc]

Yeah.

[edunad]

Just started happening as well for me, i use this for a personal player and i'm the only user

[encloinc]

Yeah it definitley isnt related to actual rate limits I think.

[mh4ck]

Same here, we're fucked up.

[HcgRandon]

I have friends seeing some issues but much less then me, however they use Lavaplayer. Additionally they also have US servers, mine are EU, might just mean it's not deployed globally yet. Some very basic testing proves they are cutting off requests pretty quickly. I'm wondering if they are detecting ytdl-core specifically somehow. Or maybe the user agent etc. I'll be getting some sleep then testing this pretty extensively I'll keep you updated to my findings

[Perflyst]

Google/YouTube has blocked my whole /64 IPv6 subnet due to one IPv6 address which I deployed recently to my invidious instance. The IPv4 address worked since a few months and is still working. Only IPv6 is blocked.

What provider do you use? I use Hetzner. Maybe it is something provider specific?

[odedhb]

It looks like just a new, more strict, throttling.
I’m trying exponential backoff.

[mh4ck]

I added some request caching (espacially for the video info) to my code - now 60% of servers are up again.
Maybe interesting for somebody.

[Harmiox]

I'm still receiving this issue, and my program is barely even using any requests (maybe 50 In total today). Does anyone know if this is provider specific? I'm not getting this issue on my local machine at all, just on my VPS.

[encloinc]

I fixed it by deploying different instances of the actual part of the code that downloads the youtube mp3's and having the main server pipe into one of these instances (at random). I made 15 of them, this seems its the only way to get past this atm.

[mh4ck]

I fixed it by deploying different instances of the actual part of the code that downloads the youtube mp3's and having the main server pipe into one of these instances (at random). I made 15 of them, this seems its the only way to get past this atm.

You did it from 1 IP?

[encloinc]

I dont know if the service uses 1 ip, they do it with "deployments". Either
way it seemed to fix it.

On Thu, Jul 11, 2019 at 9:23 AM m.h4ck [e-mail] wrote:

I fixed it by deploying different instances of the actual part of the code
that downloads the youtube mp3's and having the main server pipe into one
of these instances (at random). I made 15 of them, this seems its the only
way to get past this atm.

You did it from 1 IP?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/fent/node-ytdl-core/issues/444?email_source=notifications&email_token=ADI2LIVATHASWBIGI6FSZFLP64665A5CNFSM4H7KBMAKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZW3S2Y#issuecomment-510507371,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADI2LIWPFUC7V5QXG7D5LJTP64665ANCNFSM4H7KBMAA
.

[mh4ck]

Anybody else had figured out smth. ?

Interesting questions are:
How much requests could be done before they ban the IP?
How long will it take till the IP's are unblocked?

Haven't figured it out now.

[WaqasIbrahim]

@mh4ck I used 2 proxies (not rotating, static ip) last night for two different instances that were both blocked . But they seemed to work fine and did not get blocked even with high parallel traffic and still seem to be working.

[mh4ck]

@mh4ck I used 2 proxies (not rotating, static ip) last night for two different instances that were both blocked . But they seemed to work fine and did not get blocked even with high parallel traffic and still seem to be working.

Where you got the proxies from? I bought today a lot of new ip's but all got blocked now.
Each IP had sent ~1 request in 10-30s...

[WaqasIbrahim]

@mh4ck It's a local service, so can't reveal that. But I have added more just to be safe and so far it has resolved the issue. What VPS service are you using? I tried adding another instacne and that was blocked instantly. Maybe it's targeting spefic vendors?

[omarroth]

See also ytdl-org/youtube-dl#21729.

Similar to what @Perflyst mentioned I believe this is only an issue when connecting to YouTube over IPv6.

If possible, you may try disabling it or force connecting over IPv4. Unfortunately I'm not sure how easy that is only using this library, but it may be possible to disable it depending on cloud provider or some other configuration.

[mh4ck]

See also ytdl-org/youtube-dl#21729.

Similar to what @Perflyst mentioned I believe this is only an issue when connecting to YouTube over IPv6.

If possible, you may try disabling it or force connecting over IPv4. Unfortunately I'm not sure how easy that is only using this library, but it may be possible to disable it depending on cloud provider or some other configuration.

I'm only using IPv4 and still got this error after a few requests.

[ShivamJoker]

Guys guys guys please fix this issue as fast as possible. I have worked so hard on my music app from months if this thing happens my all work will be wasted @fent
Getting this today just requested few songs and error 429 coming
using AWS EC2 Mumbai

[mh4ck]

Guys guys guys please fix this issue as fast as possible. I have worked so hard on my music app from months if this thing happens my all work will be wasted @fent
Getting this today just requested few songs and error 429 coming
using AWS EC2 Mumbai

I guess there is nothing programmatically to do.
YouTube just ban IP's that requests videos to fast.

[ShivamJoker]

@mh4ck so we are fucked up now. 😭
and how do i unban it

[fent]

whoever is getting this, can you log the url where it happens? not the video url but the urls passed to miniget here. I'm curious if it's one of the urls used in getInfo() or if it's from the video download urls

[ShivamJoker]

@fent how to log it ???

[WaqasIbrahim]

@fent It's for the getInfo(), I use proxies for getInfo() and then I use those urls for further processing on VPS and they work fine.

[ShivamJoker]

@waqaslbrahim how do you use proxy can you please tell me also

[WaqasIbrahim]

@ShivamJoker Just pass an agent to requestOptions. I am using https-proxy-agent.

[encloinc]

@waqaslbrahim how do you use proxy can you please tell me also

Dude relax itll be ok, I also have a site that Ive been working on for over a year that stopped working because of this. When this sort of stuff happens it is best you look for a short term solution by yourself while this gets fixed. What seemed to help me was deploying the part of my code that made a lot of youtube requests into different smaller deployments in what I think is the same ip, maybe thisll work for you.

[DELUUXE]

Any update on this?
Since i guess that using a proxy will get the proxy banned/rate limited pretty quick as well, and getting new IP's/proxies all the time isn't really a long term solution either.

[WaqasIbrahim]

@DELUUXE you can try rotating proxies, the IP changes on every request or you can set an interval.

[ShivamJoker]

Do you get proxy servers for free ?
Or you buy them from somewhere

[DELUUXE]

@WaqasIbrahim thanks for the suggestion, however as that option is rather expensive, i don't think that this is relevant to me either. 🤔
Is this issue a thing that this library/package should fix (seeing the cause of the issue and resent bump in 924's) or rather something we (people who use ytdl) should fix (or rather work around) our self's?

[ShivamJoker]

guys there should be some way we can do this thing on client side (in browser generate link)
like youtube does and our problem solve 😆
client's ip will ban if they request many songs at a time and most of us have dynamic ip at home
That's the solution

[DELUUXE]

@ShivamJoker keep in mind that not all "apps" have a front-end in the browser or aren't able to execute code on the client side.

[WaqasIbrahim]

@ShivamJoker Even if you have a front-end, you will not be able to make requests to YouTube due to CORS blocking them.

[ShivamJoker]

@WaqasIbrahim so how does the youtube fetches songs and videos ?? can't we implement something like that and put client's ip instead of our server's

[edunad]

@ShivamJoker i've actually modified the lib to allow me doing that, tough you have to get the "get_info" json object on the client, then send that to the server for processing

[danielboos1]

i need a solution asap, i changed my ip about 11 times, what if we use windows? cause google tracks your cookies!

[ShivamJoker]

We all are waiting for solutions only 😒😒 you will have to change your sever for now

[danielboos1]

so wait if i use windows to host my bot , windows server will it work?

[ShivamJoker]

Just change your vps it should work make sure not to request many songs in less time

[danielboos1]

well the thing is, it is working on my current linux server but my bot is growing and too many people using the bot at the same time causing error 429. also how does this not happen to bigger bots?

[danielboos1]

lets all have a talk on discord, add me FighterMan0#0162. lets all create a group or a server and then we talk their for a solution.

[Harmiox]

Does anyone know if switching over to a NorthAmerican VPS would help with this issue? I currently use a German VPS.

[gomikestrat]

Does anyone know if switching over to a NorthAmerican VPS would help with this issue? I currently use a German VPS.

i'm using a north american vps and i'm facing the same issue

[stephane-r]

Hello guys,

Same problem on Hetzner server. I'm not sure I understood the cause of the problem ? Is it fixed ?

Is Youtube limit ? https://tools.ietf.org/html/rfc6585#page-3

[adamwoo2444]

same issue guys.Please fix this.

[fent]

for those experiencing this issue, try adding the option range: { start: '0' }. so like

ytdl(videoUrl, { range: { start: '0' });

make sure it's '0', the string, and not 0, the number. let me know if this fixes your issue. if so, I can add it as a default option.

[adamwoo2444]

what this code will do? can you please tell?

[adamwoo2444]

@fent no, it did not.Its still the same issue.

[Harmiox]

for those experiencing this issue, try adding the option range: { start: '0' }. so like

ytdl(videoUrl, { range: { start: '0' });

make sure it's '0', the string, and not 0, the number. let me know if this fixes your issue. if so, I can add it as a default option.

Unfortunately, this does not work. Every day my application hits the rate limit.

[BlookyL]

This is happening to me too, It's pretty annoying for me and other people.
Anyone found a workaround yet?

[Harmiox]

I tried using another lib (LavaLink) but I still ran into the 429 issue.

[WaqasIbrahim]

@fent Can you give a brief explaination how https://github.com/fent/node-miniget/commit/1ed846783ad5eb757ee02f35bcae2f79ca2563ed solves this issue? Does youtube send retry-after header? And how slow would it be when it's rate limited? Thanks

[fent]

I don't know if youtube does, because I was never able to replicate it. but even if they don't, miniget will still retry the request.

[WaqasIbrahim]

@fent Thanks.

[Atruo]

@fent so basically if I change this file it will work? fent/node-miniget@1ed8467

[mahhov]

This still occurs, and retrying doens't help since all requests begin returning 429. I'm not sure this should be closed.
For my use case, I sent 1 request per 3-5 minutes, for about 7 hours (so about 140 requests total).
I've sent 100 requests / hour for previously (months ago) and not had a similar issue.

edit, nevermind, I see this issue is still open here https://github.com/fent/node-ytdl-core/issues/635

[fent]

have you tried using cookies?

edit, nevermind, I see this issue is still open here #635

I've kept that opened to remind myself to add throttling

[mahhov]

Yes, cookies worked, thanks.
I'm not sure throttling alone will avoid the issue since my use case was already a pretty slow rate of requests (1 per 3-5 minutes).

[mh4ck]

How long the cookies work, when they expire? Or they don't? Did anyone got experience with that?

[mahhov]

I would guess they would work for a few months; i.e. however long your youtube login remains valid.
Maybe there's a way to retrieve a new cookie with oauth?

[mh4ck]

I just tried cookies again but can't get it to work. Do you use the same IP for the cookie wich you used to create the cookie?

I generated it on my local pc and used it on my server - now i get the following error:
**Error parsing info: Cookie header used in request, but unable to retrieve video metadata**

I'm pretty sure to use the right Cookie and identity-token... made xss attacks 10 years ago and normaly know what i'm doing haha...

let cookie = worker.getCookie();
        let idToken = worker.getIdentityToken();
        if(cookie && idToken) {
          console.log("[Info]: Using cookie for info request.");
          if(typeof opts.requestOptions.headers == "undefined") {
            opts.requestOptions.headers = {
              cookie: worker.getCookie(),
              'x-youtube-identity-token': worker.getIdentityToken()
            }
          } else {
            opts.requestOptions.headers.cookie = worker.getCookie();
            opts.requestOptions.headers['x-youtube-identity-token'] = worker.getIdentityToken();
          }
        }
ytdl.getInfo(videoId, opts)
        .then(handleResolve)
        .catch(handleReject);

[mahhov]

idk about your specific use case and what your worker is?
But for a simple example, just do:

ytdl(<id>, {
            ...
            requestOptions: {
                headers: {
                    'Cookie': '<paste cookie as string>'
                }
            }
        })

Where your cookie is a string (not an object) that includes SID, HSID, and SSID; e.g.
'SID=<~70 characters>.;HSID=<~15 chars>; SSID=<~15 chars>;' Note, unlike JSON, it's semicolon ; deliminated and uses = instead of :.

[mahhov]

I've also gotten it to work with oauth, which imo is more versatile since it would be awkward to ask users to copy & paste cookies from their browser (unless there's an automated way of retrieving cookies I don't know about?)

ytdl(<id>, {
            ...
            requestOptions: {
                headers: {
                   Authorization: "Bearer ...",
                   Content-Type: "application/json"
                }
            }
        })

[fent]

what's all you need for oauth? i'd be interested in adding another example to the repo

[mahhov]

I use oauth for 2 things in my app.
1) (unrelated to ytdl), I use oauth to allow the user to add & remove videos to their playlists, and
2) to avoid the 429 error when downloading videos with ytdl in lieu of asking the user to copy paste a cookies from their browser.

[notadevps]

I am also getting same error any solutions?

[cchen028]

I'm getting 429 error as well. I hosted on google cloud function to get video info from Youtube using ytdl.GetBasicInfo, it seems like it happens every couple days, the last time it happened it made total of 9 requests before throwing the error. Though when I look at the log it seems like for every request it actually makes 2 request (one returning 204 preflight and one returning 200), not sure if this would be an issue? Anyone else facing similar issues?

[mahhov]

Did you guys (@notadevps & @cchen028) try using cookies or oauth token?

[notadevps]

yea i tried using cookies but it didn't worked

[mahhov]

Did your cookie look like: 'SID=<~70 characters>.;HSID=<~15 chars>; SSID=<~15 chars>;'? Do you have a minimal reproducable example?

[fent]

if you got your cookies from document.cookies, try getting them from the header list in a request in the network tab. I've been told document.cookie omits some cookies

[cchen028]

Thanks for the prompt responses. Yes I have added the cookies and so far it is still working (Will give it a couple more days).

The reason I'm asking is because it works fine on my compute engine which runs 24/7 but not on the cloud functions. The 3 failures (err 429) I've encountered on cloud functions follow a similar pattern which is after the functions have been idled for a while and on the initial request it would fail until I re-deploy the function again. Just figured if anyone with a deeper understanding would know the potential cause behind this?

I'm also trying to keep the functions alive to see if this will solve it

[TehPigYT]

I'm getting the 429 status code too, sadly.
This is what I get in console

Error: input stream: Status code: 429
at ClientRequest.<anonymous> (/home/container/node_modules/miniget/dist/index.js:150:31)
at Object.onceWrapper (events.js:417:26)
at ClientRequest.emit (events.js:310:20)
at HTTPParser.parserOnIncomingClient [as onIncoming] (_http_client.js:603:27)
at HTTPParser.parserOnHeadersComplete (_http_common.js:119:17)
at TLSSocket.socketOnData (_http_client.js:476:22)
at TLSSocket.emit (events.js:310:20)
at addChunk (_stream_readable.js:286:12)
at readableAddChunk (_stream_readable.js:268:9)
at TLSSocket.Readable.push (_stream_readable.js:209:10)

Are there any fixes for this?