Instapy: Another XPath Error

[awesome205]

Hi everyone,

I have come across another XPath error and do not know how to solve it.

` File "/root/InstaPy/instapy/instapy.py", line 1925, in follow_user_followers
self.logfolder)
File "/root/InstaPy/instapy/unfollow_util.py", line 766, in get_given_user_followers
logger, logfolder)
File "/root/InstaPy/instapy/unfollow_util.py", line 540, in get_users_through_dialog
"//div[text()='Followers' or text()='Following']/following-sibling::div")
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 293, in find_element_by_xpath
return self.find_element(by=By.XPATH, value=xpath)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 752, in find_element
'value': value})['value']
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 236, in execute
self.error_handler.check_response(response)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/errorhandler.py", line 192, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//div[text()='Followers' or text()='Following']/following-sibling::div"}
(Session info: headless chrome=66.0.3359.181)
(Driver info: chromedriver=2.38.552522 (437e6fbedfa8762dec75e2c5b3ddb86763dc9dcb),platform=Linux 4.13.0-43-generic x86_64)
``

Thanks.

[Marow]

Same here

[craigricker]

@Marow and @awesome205 could you give additional details on what your script looks like?

`try:
session = InstaPy(username=insta_username,
password=insta_password,
headless_browser=True,
multi_logs=True)
session.login()

# settings
session.unfollow_users(amount=100, sleep_delay=60 )`

I am able to unfollow users successfully after having following/commenting/liking but if I attempt to run the above script, it crashes. For this reason, I think there is a chance this might be some special case, where you login, and immediately attempt to unfollow users. For some reason, the expected path is not present. My

[mehmetdilber]

I think its instagram error not instapy

[JodyHicks]

@mehmetdilber I believe you're right. After looking at the error I logged into Instagram on my PC and navigated to my profile. I tried clicking the "Following" button, but the dialog box does not appear. I watched the URL bar as I clicked the "Following" button. I was briefly taken to the "instagram.com/$USER_NAME/following" sub-domain but was almost instantly redirected back to my profile. Instagram has made changes to how users view their profiles on the PC. To illustrate this, navigate to your profile in your browser of choice (mine being Chrome) and inspect the web page. In the top left corner of the Inspect Element dialog box, there will be a button that has a phone icon on it. Click it then try to click the "Following" button. Your following list will now open. Adjusting the InstaPy scripts to view the web pages as if it were displayed on a mobile device may be a quick and dirty solution to this issue. Forgive me if I am wrong, I just learned Python a week ago and I don't know much about it. Maybe someone more experienced will have better insight and skill to fix this issue.

[torbenberger]

@JodyHicks thanks for the hint, that actually did the trick. When i set

mobile_emulation = {"deviceName": "Nexus 5"}
chrome_options.add_experimental_option("mobileEmulation", mobile_emulation)

in instapy/instapy.py:262 -> set_selenium_local_session the follow function is running as usual

is is just a very quick fix, hopefully IG fixes the Broken Desktop Version soon

[univers3]

same here!

[SammyHam]

I've had the same error since last night, impossible to run the script either for following or unfollowing !

[ArdySan2012]

@torenberger - would you mind sharing where did you set that? :)

[torbenberger]

@ArdySan2012 i attached a screenshot, but i still have some crashes sometimes

[philipok]

@torbenberger your instapy file differs from what I have, can you please attach the file itself?

[torbenberger]

@philipok not at my mac right now, but the file only differs in the lines i marked in the screenshot, everything else is like in master. anyway: there have to be made a lot more adjustments, elements differ in the mobile version and the following wont work once you come to a point where normally the bot would scroll the followers list to load new users

[najbot]

@SammyHam Until it works at instagram. Write your own bot to get new followers. For example I use selenium with pyautogui which allows me to go to the page that I want to target then thanks to pyautogui I put myself in mobile mode with the key f12 then click on mobile, click followers from here remain more than follow

[torbenberger]

the problem with using the mobile version is that the current scrolling machanism tries to scroll the dialog window which is no longer present. but i cant get the page to scroll right now.. seems like IG did some trick to prevent scrolling by code :-/

[ArdySan2012]

I think this workaround is for mac only :)

[Mehran]

@torbenberger your trick worked for me thanks .

[torbenberger]

@Mehran but is it really working well? for me the bot crashes once all followers visible in the viewport are followed and the bot tries to scroll the page to load the next users

[torbenberger]

@ArdySan2012 i dont think so as mobile simulation is supported by chrome webdriver even in headless mode on my raspbian linux

[Diddy512]

@torbenberger your workaround is working for me, at least so far

[ArdySan2012]

Sorry, and where do you put in
instapy/instapy.py:262 -> set_selenium_local_session ?

[Mehran]

@torbenberger Yes , im using nogui and work without any problem .

[Mehran]

@ArdySan2012 look at image attach . do it exactly same as image .

[jeremycjang]

@torbenberger do you know if adjustments can be made to your fix to make it work for FireFox/Pi users? I don’t think anyone’s been able to successfully launch the program in Chrome on a Pi

[torbenberger]

i made a few other changes to unfollow_util.py util.py and instapy.py i am currently testing the fixes on my account and the follow functionality seems to be running again. other functions like unfollow etc, not tested yet so i am not willing to provide a real pull request

anyway i attached the files with my changes to this post, if you use it and encounter any issues please let me know.

@jeremycjang i am running chrome on my headless raspberrypi so in fact it is working. i dont know if there is a solution for firefox because i dont know the webdriver for firefox. for informations on how to run the chrome version on a Pi checkout the main Readme of this project, there is a link on how to set it up. working like a charm for a couple of weeks now for me

If you use the attached files you have to add another option to your InstaPy Config in your quickstart.py file. set use_mobile=True to use the mobile version. thats all. everything should work now. if you are using gui mode you will see the page is loaded in mobile chrome version

Archiv.zip

[najbot]

I have tested solutions that are quite complex at my level. the only thing for me as i told you is to do it with the module pyautogui merge with selenium.

Selenium to open the browser, competitor account page.
Then in the same script at the next line call the functions pyautogui to do "f12" (open mode developer "), function to click on the moving image, click on the image followers, and do a loop for example:

for x in range (100):
if pyautogui.locateCenterOnScreen ('C: // Users // my_account // Desktop // captured image path // image.png'):
pyautogui.click (pyautogui.locateCenterOnScreen ('C: // Users // my_account // Desktop // captured image path // image.png'))
time.sleep (60)
continue
else:
for x in range(9):

   driver.find_element_by_tag_name("html").send_keys(Keys.END)
   time.sleep(5)`

[Marow]

@torbenberger thanks!

That mobile trick is fixed that. Instagram changed that on browser

[TechsCode]

I just changed it like @torbenberger suggested and I still encounter the same error. Is it maybe because im running the headless version of it? Or do I need to recompile something?

[zsvendo]

When it started it is giving the error below, I execute it by cmd, I already actuaizei the chrome drive, but the error persists what to do?

INFO [2018-06-12 10:52:18] [] Logged in successfully!
INFO [2018-06-12 10:52:22] [] User '**' [1/1]
Traceback (most recent call last):
File "C:\InstaPy\1.py", line 8, in
session.interact_user_followers([''], amount=1000, randomize=False)
File "C:\InstaPy\instapyinstapy.py", line 1744, in interact_user_followers
self.logfolder)
File "C:\InstaPy\instapyunfollow_util.py", line 766, in get_given_user_followers
logger, logfolder)
File "C:\InstaPy\instapyunfollow_util.py", line 540, in get_users_through_dialog
"//div[text()='Followers' or text()='Following']/following-sibling::div")
File "C:\Users\Avell\AppData\Local\Programs\Python\Python37\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 385, in find_element_by_xpath
return self.find_element(by=By.XPATH, value=xpath)
File "C:\Users\Avell\AppData\Local\Programs\Python\Python37\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 955, in find_element
'value': value})['value']
File "C:\Users\Avell\AppData\Local\Programs\Python\Python37\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 312, in execute
self.error_handler.check_response(response)
File "C:\Users\Avell\AppData\Local\Programs\Python\Python37\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//div[text()='Followers' or text()='Following']/following-sibling::div"}
(Session info: headless chrome=67.0.3396.79)
(Driver info: chromedriver=2.40.565498 (ea082db3280dd6843ebfb08a625e3eb905c4f5ab),platform=Windows NT 10.0.17134 x86_64)

[uluQulu]

Hi guys
Thank @JodyHicks for the super quick catch AND thanks @torbenberger for super quick fix.

It's sad to see this update and the thing is a design decision and from now on, desktop web view will be unable to load those data at that position.
I talk so sure cos it is about the way it has been implemented:
It's not a bug or even a missing support. Rather the access link now owns an event listener which triggers an event dispatcher to do redirection.

It could be 2 other possibilities, too:

• Since this update changed data load style, it still is not supported well (_or has problems_) by Desktop web view that's why redirection dispatcher will hold the tie during this time.
• With so less possibility, knowing that that decision is intentional, it can be reverted back in case of so many users wanting that capability and it could be re-considered.

To the solution

Considered that this change is permanent.

@najbot it's a good idea and pyautogui module is amazing but it has drawbacks. Since it can do image checks and in case of headless mode and images disabled mode (_beta_) it will misbehave.
There are other modules which can do interact by the process ID of the running applications but there is no any cross-platform module to do that. And even those modules are prone to misbehave in certain scenarios and is difficult to maintain.

@torbenberger
I have seen your archive, thanks for sharing it, and there the scope variable is amazing.
So I will tell you about the efficient solution since you know the situation so well:
_there is no need to do conditional statements, the scroll_bottom definition is not good to use there
there must be added that new scroll line which works for all conditions and yes it needs only one time scroll so it will be fine_

In future scroll_bottom can be improved to handle all the scrolls, but now it is not so good

What I propose is:

Having mobile emulation in entire session is not good. Cos current files rely on Desktop web view elements and in case of a mobile emulated session, tons of elements will retire.

I have prepared a solution and tested a few times, what it does is it requests mobile instance as desired for the features require it, and the change is seamless, the program will operate as is before.

The implementation took several parts edited so after a few tests I will share it in a PR.

_Also, with my latest work on relationship data which made data scraping easier will open the possibilities of getting those data through graphql which is 5 times faster than this kind of data scraping._

Well, I also thought to retire this functionality to only use that new approach I mentioned but having more options is good, so I think this functionality also should remain and work properly.

Cheers.

[torbenberger]

Looks like you didn’t add the use_mobile=True to your config or forget to exchange one of the files..

[uluQulu]

@torbenberger
I think you have told to me 😂
Yes I did not add it, I just saw your modifications to see how it works
The problem with it as I outlined in my previous comment is, it starts the session entirely in mobile view.
Please have a read my comment above to find out the consequences coming with it.

Regards

[torbenberger]

@uluQulu sorry i wrote my last post before yours was visible to me. It was an answer to the post above your comment. Your solution sounds more reliable and I am looking forward to your PR. Actually i have like 2hrs of Python experience, that’s why i called my solution dirty in above comments and did not provide a PR.

[zsvendo]

I use windows because of the cmd how to do because it is giving error? @uluQulu

[uluQulu]

Thanks again, @torbenberger
@zsvendo I think you should use @torbenberger's solution (_get his archived zip shared above_) until I do a new PR.
@torbenberger it's amazing to hear, you have dedicated skills that I wonder in which language you write programs 💪🏼

[zsvendo]

ok but the error continues, I use the followers @uluQulu.

INFO [2018-06-12 15:00:18] [] Logged in successfully!
INFO [2018-06-12 15:00:31] [] User '*' [1/1]
ERROR [2018-06-12 15:02:56] [] Could not determine if gremio has followers
Traceback (most recent call last):
File "C:\InstaPy\1.py", line 8, in
session.interact_user_followers(['*'], amount=1000, randomize=False)
File "C:\InstaPy\instapyinstapy.py", line 1753, in interact_user_followers
self.logfolder)
ValueError: not enough values to unpack (expected 2, got 0)

[zsvendo]

file 1.py
which is mentioned above is:

from instapy import InstaPy

session = InstaPy (username = '$$$$$', password = '$$$$$', headless_browser = True)
session.login ()
session.set_relationship_bounds (enabled = False, potency_ratio = None, delimit_by_numbers = False)
session.set_user_interact (amount = 1, randomize = False, percentage = 100, media = None)
session.set_do_like (enabled = True, percentage = 100)
session.interact_user_followers (['$$$$$'], amount = 1000, randomize = False)

session.end ()

[uluQulu]

@zsvendo
It's another issue, and it was fixed.
Get latest files to solve it.

[najbot]

@uluQulu I use instapy just to follow not to follow. So what I have left is to use an android emulator like genymotion or bluestacks and automate with pyautogui.

I can also send automatic messages to my new followers or those who liked my publication. For example to send a private message I make sure that this person is not received one of my private messages before and thanks to the image capture I test if the message field is empty if it's true I send the message if it's false it goes back and goes back to the next person.

In any case it's for my personal use I do not have any more because write a script to follow unfollow but I always use selenium to extract people who liked my publications to save them in a csv file and work on it android emulator.

I really hope that the possibility of seeing the list of jaimes will remain visible by instagram

[torbenberger]

@zsvendo @uluQulu actually i dont think it's another issue, i just ran into the same problem and now its working again without any changes. I also get a lot of problems within the instagram ios app. Connection timeouts and general "there was a problem" warnings. it seems instagram is currently changing some things on there platform so expect some other issues coming up

[najbot]

@torbenberger Yes I was going to talk about it lol it can hurt a lol

[uluQulu]

It's impressing @najbot
I also like Genymotion a lot but I do wonder how you use IG on BlueStacks cos it couldn't support it last time I checked.

But those programs both are so resource hungry (_even the lightest weight ones like MEmu_) and it is not so efficient to use in general, but considering the safety of the automation, it is amazingly great with some smarter algorithms.

[najbot]

I just install the latest version bluestacks and install since bluestacks instagram and it works. I have the choice between 3 emulator 1- bluestacks, 2- genymotion, 3- igdm which allows him just to send dm on instagram.

Otherwise with pyautogui I am satisfied but ceela does not consume a lot of resource the only drawback and have another pc because pyautogui uses the coordinates of the screen. So either have a pc running with the bot or a vps

[najbot]

There is also selendroid (selenium for android) but I do not have time to see that but I think I understood that we can target by xpath

[uluQulu]

Yes @torbenberger
I also get 5XX Server error in website Desktop view but it works now and then after refreshes, IP changes, etc.
Do you think that that change can be reverted back?
Cos my inspection is, since they added a dispatcher and changed the design of previous dialog box to inner page element, it sounds more likely to be a permanent change.
But those server access errors says something. Either there really is a problem with new design or it's a poorly considered beta test.

[najbot]

@uluQulu http://downdetector.co.uk/problems/instagram

[uluQulu]

Cool @najbot
You've gone deep into those styles, it's impressing.
But I got a slow laptop and I close every other big application before boots a Genymotion Galaxy S8 device.

[najbot]

@uluQulu yes it takes a computer with at least 8gb of ram.

I also think I can use it on a virtual machine to release my main pc I would see if it works with pyautogui

[uluQulu]

Wow @najbot
The outage area looks huge.
Now I think I should wait a bit to introduce my general solution which will change some bits for current situation.
@torbenberger
What do you guys think?

[uluQulu]

@najbot
I got 6G of RAM but my CPU is slow 1.67Ghz AMD and the HDD is at 19% of health so these things sum up makes it slow.
Although Genymotion already runs on VirtualBox, having another Virutal Machine will consume much more resources, and VMware itself is more resource hungry than VirtualBox if you try.

[najbot]

or a vps not the choice

[sionking]

Hi, followers/following lists are not working also from regular chrome, do you think they just don't want us to see it ? or IG has issues ?

[zsvendo]

to run python on mobile?

[torbenberger]

@uluQulu i think we should still wait a bit more. Even if the change on the desktop view was intended i don’t think it is correct as it’s implemented now. Because a link that just redirects to the same page again is not a good user experience and I think they will just remove those links On desktop IF it’s intended to be removed.

but why would you remove it on desktop and not on mobile?

[ArdySan2012]

My guess - not many people run bots on mobiles :P

[uluQulu]

@torbenberger
Yes it's not the way to do it but it is also not new in IG's style.
E.g. try to send an image to the input element directly in a mobile emulated mode and it will first navigate to .com/create/style and in a few milliseconds will redirect to .com/.
There are lots of event listeners under the hood and they reside in the very core auto-generated scripts which reads activity flow from several entry points (_e.g. mouse clicks_) and calls specific dispatchers.

Once I tried to hack it but it needs decent JS knowledge to do it.

Also it's good to know that only so less people realize that there is a redirection so user experience is not affected in general.
There is a reason to do it specifically on Desktop views and those reasons do not smell so good..

Let's see how far it goes.

[ArdySan2012]

@torbenberger got to my PC. Tried your fix, - now it loads as mobile, and Following is clickable - but:

[sionking]

anyway mobile emulation in web view works well.
Graphql still working...
My opinion IG developers are the money they worth.

[torbenberger]

@ArdySan2012 seems like you did not change all files I provided. Or you don’t have current version from repo?

@uluQulu i also think that this is somehow a first step of new bot prevention of instagram.. hopefully they won’t take it too far

[zsvendo]

@uluQulu is the same error as mine, even though I have changed all the files
@ArdySan2012
see below

[uluQulu]

@zsvendo
Your problem is different than that of @ArdySan2012 posted above in a screenshot.
Your problem should be fixed after you properly apply @torbenberger's solution.

But @ArdySan2012's problem is
in unfollow feature if you give onlyNotFollowMe=False and onlyInstapyFollowed=False, it will try to unfollow all of the users you are following from your Following where it will crash cos of the latest update. (_I have not checked, maybe @torbenberger archived zip also contains the fix for it_)

[Essbuisness]

@sionking Please do you know how i can run the script on a server
(windows)

[LoneWolfsMoon]

So... Can you upload Edited files ? It's really hard to catch up to 200 comments..... @uluQulu @torbenberger

Thank you so much, anyway. <3

[uluQulu]

Hi @LoneWolfsMoon
@torbenberger already shared the edited files in an archived zip in a comment above

My approach is a bit different and it does edit several files in different places that's why I will share it in a PR soon. But the idea is the same it's using mobile emulation as @torbenberger did.

[uluQulu]

Heads up guys!
https://github.com/timgrossmann/InstaPy/issues/2220#issuecomment-396798961
It has been fixed and operates properly as before.

Although, I have not checked page elements, I think they should also remain the same.

@torbenberger good that I did not PR that soon, for some reasons the revert happened, but I will take a backup of those lines in case of a similar situation in the future.

[jeremycjang]

@uluQulu With all this happening starting yesterday (though it does seem like it was more likely due to a buggy update with the service crash) , do you think we should continue using a desktop interface design for the program, or indeed look into the mobile emulation capabilities the community put together so quickly today?

[uluQulu]

Yes @jeremycjang, in these hot summer days bugs are looking for a cool code to reside in 😂

Using the normal (_Desktop web view_) is the way to go cos page elements are being picked for it. And in case of the situation like in above, the only way would be do again all stuff in Desktop web view and only request Mobile web view only for certain tasks WHICH would cost some bits from performance and computer resources.
Else than that, I like bigger Desktop interface cos sometimes I also watch the activity ☕

What is best for you to do if you have such a concern is:

• Take a backup of your current file which has been modified with the changes in this thread
• Feel good to request to it anytime

[sionking]

Mobile emulation on desktop is highly detectable! Not a good idea!

[torbenberger]

@uluQulu great to hear that everything is back to normal now. let's hope we wont see such a change to the website anytime soon :-)

[najbot]

So it was just a bug, that's what I said yesterday too much connection concern and I was persuaded that the followers page had suffered the same problem and not because instagram decided to change access to this page.

Anyway it will take a day to find a solution in case it will be decided to remove this page.

[Essbuisness]

how can i run the script on a server for windows os
please

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this problem still occurs, please open a new issue

[converge]

hey guys, I had this issue today, took me a lot of time to understand what's going on here.

I had 2 issues:

1 - language selection

I was on a french server and it was loading the page into portuguese, reseting locales to portuguese and deleting Firefox profile wasn't enough, so I deleted Firefox and reinstalled the Debian version (version 60).
Now that it was loading the page in english, I turned the language switch of on InstaPy setup (quickstart.py):
session.set_switch_language(option=False)

2 - unable to reach Log In button

for clicking on Log In button I added a try/catch:

    if login_elem is not None:
        try:
            (ActionChains(browser)
                .move_to_element(login_elem)
                .click()
                .perform())
        except Exception:
            login_elem.click()

And now it's clicking using the second option login_elem.click()

The main point of it all is:

move_to_element only perform the action if the button (login_elem) is visible on screen, that's why login_elem.click() works and move_to_element doesn't.

Can someone help testing it ? so we can propose a PR.