Instagram-php-scraper: rate-limit / too many requests

[th4deu]

Sup!

Guys, how are you doing to bypass the rate limit?
I'm trying to create different accounts and select them radomly to query the Instagram's comments and media, but it reaches its limit too fast.

Is there any workaround about this?

[luanrox]

350 proxies with 350 threads

[lerni]

Because of #325 and also for speed reasons I use authenticated requests an caching them for an hour. Can you make an approximation at how many request in x-time rate-limit kicks in?

[raiym]

I read somewhere that it is 200 per hour

On Tue, Apr 10, 2018, 11:54 Lukas [e-mail] wrote:

Because of #325
https://github.com/postaddictme/instagram-php-scraper/issues/325 and
also for speed reasons I use authenticated requests an caching them for an
hour. Can you make an approximation at how many request in x-time
rate-limit kicks in?

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/postaddictme/instagram-php-scraper/issues/330#issuecomment-379982494,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFb0GRYZC2CcVisPD6oJ1PEATVKg0WWRks5tnEkMgaJpZM4TNMDq
.

[350d]

@luanrox Any recommendations for affordable and fast proxy list service with api?

[luanrox]

@350d

https://buy.fineproxy.org/

350 proxy package - only $20

That’s what i’m using.

[th4deu]

@luanrox, I was testing last weekend and even using different IP's (my ISP IP and the server's one) I got the rate limit message. Isn't the rate limit applied to the account?

[lerni]

Are you fetching data from Instagram with every request and if so why? To me this just sounds like asking for trouble ;) It'll be slow, even slower with proxies and you have to pay for a questionable solution that could be handled easily with a cache or am I missing something, your use case implies that you need so many Instagram-queries?

[th4deu]

@lerni , my website gets all the post comments 'til the moment the user asks for it.
Since Instagram queries from the last to the first comment, I can't save a pointer to resume from the last query because the user rarelly uses it more than once per post.

[andrewyoo]

Starting today, i've been getting severely rate limited. I can't even do the stated 200 req / hour! Anyone else having trouble doing 200 calls per hour?

[adiv-phpian]

I found a simple way to avoid rate limits. We are missing something to send something because of that we are getting rate limited. Before that, within 10 requests I will get rate limited. but now I can able to get 100K likes without any problem.

[mahmod2000]

@muthu-kc could you explain about that, what did you send to API to avoid rate limits?

[BB-000]

I don't get blocked from my localhost but cannot get it to work on live server. I got a 429 now a 403 yesterday, tried today and still 403, should it not have refreshed by now? Do I need to change the proxy?
Getting 24 posts and 50 comments for each (but there are usually only ~10 comments) is this too many requests?

[adiv-phpian]

You have generate mobile api cookies and use it with web api endpoint. In
that way you can avoid rate limitations. Even though make sure you use
random intervals between requests.

On Tue, 26 Feb 2019, 14:49 Barnaby, [e-mail] wrote:

I don't get blocked from my localhost but cannot get it to work on live
server. I got a 429 now a 403 yesterday, tried today and still 403, should
it not have refreshed by now? Do I need to change the proxy?
Getting 24 posts and 100 comments for each (but there are usually only ~10
comments) is this too many requests?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/postaddictme/instagram-php-scraper/issues/330#issuecomment-467361173,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AHjnGHoWuLvZNWe5yBU06OLx-wY7-xXsks5vRPw9gaJpZM4TNMDq
.

[tawfek]

You have generate mobile api cookies and use it with web api endpoint. In that way you can avoid rate limitations. Even though make sure you use random intervals between requests.
…
On Tue, 26 Feb 2019, 14:49 Barnaby, @.*> wrote: I don't get blocked from my localhost but cannot get it to work on live server. I got a 429 now a 403 yesterday, tried today and still 403, should it not have refreshed by now? Do I need to change the proxy? Getting 24 posts and 100 comments for each (but there are usually only ~10 comments) is this too many requests? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#330 (comment)>, or mute the thread https://github.com/notifications/unsubscribe-auth/AHjnGHoWuLvZNWe5yBU06OLx-wY7-xXsks5vRPw9gaJpZM4TNMDq .

Hi
how can i generate mobile api cookies and use it with web api endpoint
do u mean some thing like this or what ?
$instagram->setUserAgent("Mozilla/5.0 (Mobile; Windows Phone 8.1; Android 4.0; ARM; Trident/7.0; Touch; rv:11.0; IEMobile/11.0; NOKIA; 909) like iPhone OS 7_0_3 Mac OS X AppleWebKit/537 (KHTML, like Gecko) Mobile Safari/537");

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[akhibing]

@muthu-kc could you explain how do you generate mobile api cookies and use it with web api endpoint.?
and also could you explain more about random intervals between requests? because I heard about 200 request per hour and other limits, I want to know do you consider these limits also or you just use random intervals between requests? these random intervals are are around some seconds or some minutes or some hours?

[restyler]

First question that you should ask before spending $20-50 on proxies per month is - do you really need authenticated requests to get comments? For the cases when the post is public, this is not neccessary. Another factor is that even private proxies from popular sellers might be banned by Instagram, so retry logic is useful for failed requests - and it gets complicated quickly especially if we talk about PHP&Curl. I've hit the same issue and had to spend fair amount of time on it for my own project.
Here is the code I came up with: https://github.com/restyler/instagram-php-scraper - it uses Rapid API ( https://rapidapi.com/restyler/api/instagram40 ) to bypass ip restrictions.

[JoeGarcia2]

@restyler I'm trying to use your API to get recent posts by hashtag i.e. https://www.instagram.com/explore/tags/wedding/ and it is no longer working. I imagine it has something to do with Instagram's note about trying to stop misinformation about the US election. Is there any other way to get recent posts by hashtag?

[restyler]

@restyler I'm trying to use your API to get recent posts by hashtag i.e. https://www.instagram.com/explore/tags/wedding/ and it is no longer working. I imagine it has something to do with Instagram's note about trying to stop misinformation about the US election. Is there any other way to get recent posts by hashtag?

Yes, this API works fine for your use case, and any other use cases when the instagram page is available for anonymous user.

[JoeGarcia2]

Ok. I'm trying to test the endpoint on RapidAPI but it's no longer working as it used to. I don't get all of the page with the json. Is the RapidAPI version still working?

[restyler]

Ok. I'm trying to test the endpoint on RapidAPI but it's no longer working as it used to. I don't get all of the page with the json. Is the RapidAPI version still working?

Yep, it is working, and I think it is working even on your screenshot, it's just rapidapi web preview which trims some part of the returned html.

[restyler]

Just to clarify: this instagram url is intended to return HTML, not JSON. If you want to parse it - you need to get the json part via regex. E.g. see https://github.com/restyler/instagram-php-scraper/blob/proxified/src/InstagramScraper/Instagram.php#L510

[JoeGarcia2]

Got it. Yeah I realize I have to parse it. The error must be in by parsing code. I just saw that the Instagram home page wasn't showing recent posts anymore and thought it might be connected. I'll keep checking on my end. Sorry to hijack this thread.

[350d]

@JoeGarcia2 Try to use non US proxy. Instagram just disabled recent hash tag feed for US due to elections.