Instapy: Hangs after random number of follows, CPU and Memory usage creeping up

[spacewaffle]

Expected Behavior

I just pulled the latest changes after having InstaPy working previously. I've set it to follow a bunch of people. The same quickstart script was working earlier with no changes now exhibits this behavior.

Current Behavior

It will follow anywhere from 30-50 people, then hang. CPU and Memory resource usage will creep up uncontrolled until the process is killed.

^CERROR - follow loop error 
^C^CTraceback (most recent call last):
  File "quickstart.py", line 15, in <module>
    session.follow_user_followers(['_shokunin_', 'bjjscout'], amount=100, sleep_delay=30, randomize=True, interact=True)
  File "/Users/spacewaffle/InstaPy/instapy/instapy.py", line 1181, in follow_user_followers
    self.follow_times)
  File "/Users/spacewaffle/InstaPy/instapy/unfollow_util.py", line 731, in follow_given_user_followers
    callbacks=[])
KeyboardInterrupt

InstaPy configuration

from instapy import InstaPy

session = InstaPy(username='namehere', password='pwhere', nogui=True, headless_browser=True)

session.login()

session.set_upper_follower_count(limit = 5000)
session.set_lower_follower_count(limit = 20)
session.set_do_follow(enabled=True, percentage=80, times=1)
#session.like_by_tags(['#bjj', '#bjjlifestyle', '#jiujitsulifestyle', '#jiujitsu'], amount=100)


session.set_user_interact(amount=1, randomize=True, percentage=100)
session.set_do_like(enabled=True, percentage=70)
session.follow_user_followers(['_shokunin_', 'bjjscout'], amount=100, sleep_delay=30, randomize=True, interact=True)

session.unfollow_users(amount=200, onlyNotFollowMe=True, onlyInstapyMethod = 'FIFO', sleep_delay=30 )

session.end()

[uluQulu]

Hi, a few questions. How many CPU cores and how much total memory do you have?
Are those resources very busy while running script?
Generally, since InstaPy is loading working files into temporary folder, it gonna fill up there in each start until disk is full. You should schedule an auto %tmp% cleaning activity to remove all temporary files in last modification date order (e.g.: delete files if they are older than 2 days, removing fresh temp files can cause issues). This will improve performance of you computer.
I am wroking on a feature which will consume some more resources, that's why i am interested in this issue, to know how to prevent hangs.

[spacewaffle]

I've got 4 gigs of memory and 2 cores. Can I just manually remove stuff in the temp folder for now? Where is it?

This issue wasn't happening until I pulled the latest commits. I could run the script overnight without problems before.

[spacewaffle]

For more info, I updated from commit 1b00df19a25097c4a6e234053a9b405f8f403d26 which was working without these issues.

[spacewaffle]

I also have a hunch it was this 7920f71a9dc27ff1bfab6c20590e170d5c896440 commit that caused it since that commit seems to have changed the way follows work.

Beforehand, my max cpu load was maybe 16%. Memory was about 3.5gb out of 4. But when I use the latest code, my cpu load is at like 98% and my memory usage will go to like 20gb or something crazy.

[rediv]

Hi,
I have InstaPy running on a small GCE instance (1 vCPU, 1.7 GB RAM) and I'm also experiencing this issue. Everything was fine a couple days ago.

Now the script starts following accounts normally, but when it gets 26~29 accounts followed (about 13 minutes) it takes the whole CPU, memory and swap for it self and outputs: ERROR - follow loop error

The whole instance stop responding and I can't even continue to use the shell.

I could notice that, even with this error, the py script does not stop and get back to work after some moment.

I tried to use _nice_ to start the script with low priority, but I got the same issue. I guess the VM stops responding because of the lack of free memory, not CPU. If I use cgroup to enforce low memory usage, the far I can get is the process being killed.

The problem is that I have other important services running in this instance (webserver) and cannot dedicate all resources to InstaPy. I could deploy InstaPy on another instance (wasting resources) or use dockers, nevertheless there is a serious problem with memory consumption going on here.

Please tell me if I can do anything to help finding a solution. I'm not a Python developer, but I know my way around Linux...

[rediv]

I was trying to analyze this issue further and the problem seems to be with function follow_user_followers.
I can run the script with follow_by_tags or unfollow_users for hours without any issue.

By the way, the problem is indeed with InstaPy script, because the only process eating the whole server memory and being killed by cgroup policy is 'python3 myrules.py', and not the associated processes (like chromedriver).

I'm currently using Python 3.6.3 on Ubuntu 17.10. @spacewaffle which version are you running InstaPy on?
It's odd that just me and @spacewaffle are having this problem so far...

I will try to rollback the changes made in the last commits and try to find what piece of code is responsible...

[converge]

I had this issue before, all I had to do was:

1. kill every chrome process
2. kill every Xvfb (or xvfb) process
3. use the this quickstart.py approach (try/catch)

Never had this issue again.

[spacewaffle]

@rediv I'm running python 2.7.14 on osx. My script was also failing during follow_user_followers so I think you're onto something. It could be that not everyone is on the latest version.

[rediv]

@spacewaffle The problem seems to be with followRestriction.json file under InstaPy/logs folder
Try to replace yours with a fresh new from the git repository to see it the problem remains.

I will try to run my script a couple more times to see if this fixed the problem for good.

[spacewaffle]

I'll try this when I get a sec. Is there a new format? If so, what does it look like?

[rediv]

@spacewaffle I have no idea. In the process of finding the problem I simply realized that a fresh installation was running without problems, and then I started to replace existing files with new one.
The culprit, as I realized, was followRestriction.json but you may want to replace the whole logs folder if this does not help.
It didn't even open this file to debug it, but it should be a simple json file. Maybe the InstaPy script outputted something to this file out of the json format...

[spacewaffle]

Replacing followRestriction.json worked for me, but now it seems like followRestriction.json isn't getting populated when the script is run. Does that mean followRestriction isn't working properly?
That seems like an issue.

[rediv]

@spacewaffle review the new file permission. I'm not having this issue.

Now we know the problem is definitely related to followRestriction.json.

If we replace this file, the InstaPy runs smoothly. The problem is that after InstaPy runs a couple more times, the problem reappears. I'll try to analyze this file further, but I don't know if I can get far not being a python developer...

[spacewaffle]

Permissions are the same between new and old files. I think this is no longer an issue with me, but it seems like I'm running into a different problem which is as followRestriction.json grows when you use the script, something with the scroll behavior causes it to max out after a certain point. I checked out this thread:
https://github.com/timgrossmann/InstaPy/issues/1182

@uluQulu suggested increasing the range_int variable and that seems to have worked. I'm not sure i fully understand what's going on but it seems like range_int is being outpaced by the growing number of users in followRestriction.json

I will close out for now. Feel free to open up if you're still having this issue.

[spacewaffle]

Nevermind, I ran into this problem again after using the fix stated above. I hit about 84 follows before my computer hung up and python started eating up my memory.

[rediv]

I'm still having this issue.

The problem happens, as I realized, when a restriction is found (InstaPy should not follow a certain user according to followRestrinction.json and for some reason starts eating up the whole memory).

So, cleaning this file will help for a day or two, and you will have problems with restricting follow activity.
If you set follow_times to 2 or 3, InstaPy will run smoothy for some time and then it will find a restriction again later on and crash the machine again.

A new method of dealing with this restriction is necessary, but unfortunately my hands are tied. :(

[spacewaffle]

I couldn't figure this out so I stopped trying to use follow users followers and now I just use like_by_tags and follow from that. That seems to have no resource creep problems.

[igisho]

Any news on this issue?

[igisho]

@timgrossmann is there any way ho i can debug memory leak? I´d like to help but i´m not python developer. There is problem with follow_users_followers, but i cant figure out how to debug it.

[GitNees]

No real hang here, but I have 24GB RAM and an SSD, but it is "destroying" my SSD, since I noticed once serious disk activity, then saw python.exe using 20GB RAM, so swapfile was going full-retard, and you know what they say, never go full retard. Then checking my disks written amount, it had gone up at least 1TB in a few days, which is more then ever since I got the drive, This isn't viable till fixed.

[GitNees]

@uluQulu @timgrossmann anyone any idea? check my issue on memory leak, looks to be the same. Willing to test/debug/help.

[igisho]

@GitNees do you have any progress on this one?

[uluQulu]

Hi @GitNees
I have an idea, to be sure, please provide a few details:
How is the content of that json file when you encounter that excessive resource usage (how many users it holds and how about those users' follow_times values)

try to test more than one times, and post the results accordingly as mentioned above, this will help to put correct diagnosis

[igisho]

@uluQulu aproximatelly 3500 users and mostly follow_times values are mostly 1 and few are 2 - 6

[uluQulu]

@igisho
_How many_ times you have tested the content in the json file? _Are they_ average values? (_these are important_)

Having considered that you have posted the values _averagely_ above, try this:
_open up_ unfollow_util.py file, _in_ follow_through_dialog() function, _find_ the _last_ else _statement_ and add a new sleep(15) line as in below:

else:
    if randomize:
        repickedNum = -1
        while repickedNum not in sample and repickedNum != -1:
            repickedNum = random.randint(0, len(btnPerson))
        sample.append(repickedNum)
        finalBtnPerson.append(btnPerson[repickedNum])
    sleep(15)   #YOU will add this NEW line
    continue

As little change as it is, it will hopefully save you from resource _hung_. Just make sure you have got a _huge_ (_content of over 3500 users_) json file to test out this new modification so that we _quickly_ know if it works _effectively_.

[igisho]

@uluQulu this helped in not crashing machine but actually this time script fails silently there are some follow loop errors preceding and after that the script just hungs and do nothing. No exeption thrown in shell. Script looks like it is runing but there is nothing it is doing.

[igisho]

@uluQulu im using multiple accounts and all start to consume memory if follow_user_followers is used and the json file contains more than 3500 users

[uluQulu]

@igisho
The change we did above was for memory hung. But the follow loop error is another story. Post the full console output of that error to see what is the problem.
Also, how frequently do you get follow_loop_error?

[uluQulu]

Also, there is a problem with the follow_through_dialog function's scroller which also is used for follow_user_followers feature that returns less links than desired
Recently, I've shared a way to make it work smoothly in #1548

[MrBeardedGuy]

@uluQulu yeah I have the same issue using follow_user_followers it happens once every 4-5 runs. Does your fix https://github.com/timgrossmann/InstaPy/issues/1548#issuecomment-372803884 will work just for unfollow_users or also for the function follow_user_followers?

[uluQulu]

sleep(15) line, in above post, should solve your problems with resource creep IF it does not, provide the details of the error (_such as you might get follow_loop_error as @igisho said_)

@MrBeardedGuy if you read that post I said that "_Also, this solution is fixing other features which are also using follow_through_dialog function and is a general approach_"
Moreover it was written exactly for follow_user_followers feature in request of @cormo1990

How did you think of it be related to unfollow_users? Cos of it is inside unfollow_util.py? - Well, it is understandable cos all of the follow & unfollow engines are inside unfollow_util.py..
And yes unfollow_users is antoher function and is not affected by that change, do you have problems with it, too?

[GitNees]

hey @uluQulu Been a while. Is the fix above related to this? https://github.com/timgrossmann/InstaPy/issues/1630

Turned it on yesterday without updating files and forgot to monitor it and fell asleep. Woke up during it running and it ate again 500GB!!!! of SSD wear with swipfile RAM data. :( I'll update now and see what happens. Wish I could contain ram usage and make it crash when the reserved ram is used. Instead of going full-retard on my swapfile (and I have 24GB of ram) And you know what they say: Never go full retard :)

[ekumar96]

I'm experiencing similar issues. I started off using InstaPy in the background while I was doing other things on my computer. I only used follow_user_followers. It worked like a charm for the first few days, but then I noticed I began getting the following follow loop error

WARNING [date time] [my_username] users_followers_to_follow -> Less users to follow than requested.
ERROR [date time] [my_username] follow loop error Sample larger than population or is negative

However, if I had terminal active/open/in the foreground on my Desktop (if I wasn't doing other stuff on Chrome while letting terminal do its thing in the background) then it worked fine.

Unfortunately, recently, like @spacewaffle and @rediv, it just stopped working after about 30 follows and ate up my memory, and gave me the same follow loop error as them when killed. After reading this thread, I just emptied the followRestrinction.json file (replaced the usernames with {}) and it's back to working (but only when terminal is in the front).

My first question; would it be safe to simply empty the followRestriction.json file like this after every run? Would it still eat up my memory? Second, what does the "Sample larger than population or is negative " error mean? Is there any easy way to fix this? As you can probably tell, I'm not very python literate so any assistance would be immensely appreciated.

[uluQulu]

hey @ekumar96,

1. Important - have you tried the _suggestions I have posted_ in comments above? (_read my comments with suggestions and write down what did you do and what you got: 1st comment, 2nd comment & 2nd comment (_helpful_)_)
2. It is weird that actions are tied up to terminal window to be in focus. If you did minimize (_or opened other tabs other than auto-opened webpage(s)_) Chrome (_which was started by selenium_) browser it would be a problem, but terminal window being inactive (out of focus or minimized) should not add problems. Tell about what is your OS and terminal (_xterm, gnome-terminal, command prompt, powershell, tty console, or sth, else?_) emulator
3. That JSON file is innocent! All it does is _passing data to python dictionary_ once InstaPy starts (_session = InstaPy()_) and retrive end data once end() method is called (_session.end()_), so if there is a problem, it is with using _kind of_ huge dict (that dictionary is self.follow_restrict) frequently which could consume memory. that's why I have told in above comment to use that missing sleep(15) line.

But, if you insist, you can remove that mechanism (_follow restriction based on followRestriction.json file_) but then your follow_times value will make no sense.

@ekumar96 @GitNees @MrBeardedGuy try the suggestions I have stated then post the details of the situation, we will solve it only with good feedback! (_for those having the same excessive memory consumption, too_)

@ekumar96, Sample larger than population or is negative is a typical ValueError execption raised by _the python_ interpreter meaning you are trying to _sample_ more elements than you have and is possibly occured in this line of unfollow_util.py,

sample = random.sample(range(0, len(follow_buttons)), real_amount)

_which literally means, follow_buttons list does not have enough elements as requested by real_amount integer..._

[ekumar96]

@uluQulu thank you for your prompt reply! I did all the things you suggested, and it seems to be working as expected, except for the weird #2 issue. I didn't minimize any of the windows. I'm on macOS Sierra, 10.12.6. I run the commands from terminal. It will return these errors.

INFO [time] [my_username] Failed to load desired amount of users.
WARNING [time] [my_username] account1 -> Less users to follow than requested.
ERROR [time] [my_username] follow loop error Sample larger than population or is negative
INFO [time] [my_username] Failed to load desired amount of users.
WARNING [time] [my_username] account2 -> Less users to follow than requested.
ERROR [time] [my_username] follow loop error Sample larger than population or is negative

Even when it does work (I am not using my laptop and just letting it run), it will frequently tell me that there are "Less users to follow than requested" though some of these accounts have millions of followers.

[uluQulu]

It's crystal clear now @ekumar96 but have you really applied the modificatons from #1548 comment
Cos if you did, you would get correct amount of links and not get Less users to follow than requested message.
Please, attach your unfollow_util.py file, I will see the changes done

[ekumar96]

@uluQulu thanks, I have attached it here (unfollow_util.py.zip). Also, not sure if this will help elucidate the problem, but as long as terminal is open for the first follow of an account's followers
, it will follow the rest. For example, if it is supposed to follow 20 followers of an account, if terminal is up for the first follow, it will complete the rest regardless of if terminal is in the front or not. If, while it is moving to the next account on the list, the terminal is in the background, it will show the errors described above and will not follow any of account 2's followers

[uluQulu]

@ekumar96, you have misapplied 1st step & did not apply 2nd step in that fix of #1548. I've reapplied the fix and corrected lines:
unfollow_util.zip

Now, after this, try again and post all of the troubleshoot information in details

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this problem still occurs, please open a new issue