Instapy: [feature request ] if no response on request in x time, go to next request or retry offer 30$

[fgisslen]

I have problem with timeout on some request. Think maybe it is because the proxy I use seldom but sometimes fails on the request. In that cases the program crashes.

I would like to be able to let InstaPy retry the request if it hasn’t got a response in say 5 minutes.

Is it possible?

[fgisslen]

I will give 30$ to the one who fixes this for me

[uluQulu]

@fgisslen
Post the full error information you get from the console.
And provide a solid explanation about what you know about the problem. e.g.,

• Do you face (_and want to solve_) _timeout_ exception only?
• OR also get _stale element reference_ exceptions?
• sth else

_I think it is a special need but I will help, hopefully._

---

Cheers 😁

[fgisslen]

@uluQulu
Perfect, I will write to you again tonight with an error that will occur today 😂

Do you think it’s bets to reload the request after a timeout?

[uluQulu]

@fgisslen
After evaluating the problem in-depth I will say what is best, for now, I can't be sure.
_Make sure to include all of the details of problem to be solved._

But most of time I do ask again. Let's see your dedication and awareness skills 🙂

[fgisslen]

Here you have a typical error:
It not only happens with like_by_feed, it can happen with all kinds of requests.

I think probably it happens when the request to the proxy server fails and it just waits until timeout.
It is timeout that is the problem.

INFO [2018-08-16 09:41:35] [xxx] Post [2/5]
INFO [2018-08-16 09:41:35] [xxx] https://www.instagram.com/p/BmiKSNyHieG/?taken-by=joshi_ji_1111
WARNING [2018-08-16 10:01:35] [xxx] Could not locate Chrome: Message: timeout
(Session info: headless chrome=68.0.3440.106)
(Driver info: chromedriver=2.41.578700 (2f1ed5f9343c13f73144538f15c00b370eda6706),platform=Linux 4.15.0-32-generic x86_64)

OOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO
INFO [2018-08-16 10:01:35] [xxx] Session ended - 2018-08-16 10:01:35
oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo

Traceback (most recent call last):
File "/home/fredrik/InstaPy/5b62a628a5231c0014b19e48.py", line 25, in
session.like_by_feed(amount=60, randomize=True, unfollow=False, interact=True)
File "/home/fredrik/InstaPy/instapy/instapy.py", line 2327, in like_by_feed
for i in self.like_by_feed_generator(**kwargs):
File "/home/fredrik/InstaPy/instapy/instapy.py", line 2456, in like_by_feed_generator
self.user_interact_media)
File "/home/fredrik/InstaPy/instapy/instapy.py", line 1462, in like_by_users
self.logger)
File "/home/fredrik/InstaPy/instapy/like_util.py", line 405, in check_link
web_adress_navigator(browser, post_link)
File "/home/fredrik/InstaPy/instapy/util.py", line 596, in web_adress_navigator
browser.get(link)
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 332, in get
self.execute(Command.GET, {'url': url})
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 320, in execute
self.error_handler.check_response(response)
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message: timeout
(Session info: headless chrome=68.0.3440.106)
(Driver info: chromedriver=2.41.578700 (2f1ed5f9343c13f73144538f15c00b370eda6706),platform=Linux 4.15.0-32-generic x86_64)

[uluQulu]

It's clear, @fgisslen.
To solve it, open up util.py file and,
1-) add this _line_ near other imports (_at top of the file_)

from selenium.common.exceptions import TimeoutException

2-) replace your web_adress_navigator _definition_ with this,

def web_adress_navigator(browser, link):
    """Checks and compares current URL of web page and the URL to be navigated and if it is different, it does navigate"""
    total_timeouts = 0

    try:
        current_url = browser.current_url
    except WebDriverException:
        try:
            current_url = browser.execute_script("return window.location.href")
        except WebDriverException:
            current_url = None

    if current_url is None or current_url != link:
        # handle famous timeout exceptions during `GET` method
        while True:
            try:
                browser.get(link)
                break

            except TimeoutException as exc:
                if total_timeouts >= 7:
                    raise TimeoutException("Retried {} times to GET '{}' webpage "
                       "but failed out of a timeout!\n\t{}".format(total_timeouts,
                             str(link).encode("utf-8"), str(exc).encode("utf-8")))
                total_timeouts += 1
                sleep(2)

        # update server calls
        update_activity()
        sleep(2)

• OR get it from ready gist
• OR apply the extra lines manually.

It will solve your problem, hopefully 🔑

---

Cheers 😁

[fgisslen]

@uluQulu perfect, I will try it tomorrow. Email me on [email protected] and I will arrange a money deposit to you :)

[fgisslen]

@uluQulu do you know why this happens?

Traceback (most recent call last):
File "/home/fredrik/InstaPy/5b6b2c94cecd67001461cda9.py", line 27, in
session.interact_user_followers(['borninstockholm','mollyrustas','jannid'], amount=30, randomize=True)
File "/home/fredrik/InstaPy/instapy/instapy.py", line 1820, in interact_user_followers
self.logfolder)
File "/home/fredrik/InstaPy/instapy/unfollow_util.py", line 869, in get_given_user_followers
channel, logger, logfolder)
File "/home/fredrik/InstaPy/instapy/unfollow_util.py", line 679, in get_users_through_dialog
if quick_username[0] not in simulated_list:
IndexError: list index out of range

[uluQulu]

@fgisslen,
I really appreciate you donation request but I can't accept it, thanks!

For the problem above, have a read https://github.com/timgrossmann/InstaPy/issues/2555#issuecomment-410475845 to get solution.

---

Cheers 😁

[fgisslen]

@uluQulu you are the best

[fgisslen]

@uluQulu I got this error now:

INFO [2018-08-17 09:14:13] [xxx] Unable to get Following data:
b'Message: timeout\n (Session info: headless chrome=68.0.3440.106)\n (Driver info: chromedriver=2.41.578700 (2f1ed5f9343c13f73144538f15c00b370eda6706),platform=Linux 4.15.0-32-generic x86_64)\n'

INFO [2018-08-17 09:14:13] [xxx] Grabbed 4519 usernames from Following in 4.82 minutes
INFO [2018-08-17 09:14:13] [xxx] Stored Following data at /home/fredrik/InstaPy/logs/xxx//relationship_data/xxx/following/17-08-2018~full~4519.json local file

INFO [2018-08-17 09:14:13] [xxx] Zz :[ time to take a good nap ~sleeping 3.0 minutes
INFO [2018-08-17 09:17:30] [xxx] Yawn :] let's go!

INFO [2018-08-17 09:17:30] [xxx] There are 4429 Nonfollowers of xxx ~the users xxx is following WHO do not follow back

INFO [2018-08-17 09:17:31] [xxx] Stored Nonfollowers data at /home/fredrik/InstaPy/logs/xxx//relationship_data/xxx/nonfollowers/17-08-2018~[349-4519]~4429.json local file

INFO [2018-08-17 09:17:31] [xxx] Total 4429 users available to unfollow ~not found in 'followedPool.csv': 170 | didn't pass unfollow_after: 0

INFO [2018-08-17 09:17:31] [xxx] You have requested more amount: 5000 than 4429 of users available to unfollow ~using available amount
ERROR [2018-08-17 09:22:31] [xxx] Unfollow loop error: Message: timeout
(Session info: headless chrome=68.0.3440.106)
(Driver info: chromedriver=2.41.578700 (2f1ed5f9343c13f73144538f15c00b370eda6706),platform=Linux 4.15.0-32-generic x86_64)

INFO [2018-08-17 09:22:31] [xxx] --> Total people unfollowed : 0

WARNING [2018-08-17 09:27:31] [xxx] Could not locate Chrome: Message: timeout
(Session info: headless chrome=68.0.3440.106)
(Driver info: chromedriver=2.41.578700 (2f1ed5f9343c13f73144538f15c00b370eda6706),platform=Linux 4.15.0-32-generic x86_64)

[fgisslen]

and this @uluQulu:

Traceback (most recent call last):
File "/home/fredrik/InstaPy/instapy/util.py", line 602, in web_adress_navigator
browser.get(link)
File "/home/fredrik/.local/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 332, in get
self.execute(Command.GET, {'url': url})
File "/home/fredrik/.local/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 320, in execute
self.error_handler.check_response(response)
File "/home/fredrik/.local/lib/python3.6/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message: timeout
(Session info: headless chrome=68.0.3440.106)
(Driver info: chromedriver=2.41.578700 (2f1ed5f9343c13f73144538f15c00b370eda6706),platform=Linux 4.15.0-32-generic x86_64)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/fredrik/InstaPy/5b71e1c3cbbe5b0014f6062b.py", line 22, in
session.like_by_tags(['ootd','dressingdiary','styleinfluencer','styleblogger','scandistyle','dailyoutfit','styleblog','streetstyle','womeninmedicine','solidstreetstyle'], amount=100)
File "/home/fredrik/InstaPy/instapy/instapy.py", line 1223, in like_by_tags
self.logger)
File "/home/fredrik/InstaPy/instapy/like_util.py", line 405, in check_link
web_adress_navigator(browser, post_link)
File "/home/fredrik/InstaPy/instapy/util.py", line 605, in web_adress_navigator
except TimeoutException as exc:
NameError: name 'TimeoutException' is not defined

[uluQulu]

@fgisslen
I forgot to tell you import TimeoutException 😂

Open up util.py and add this _line_ near other imports (_at top of the file_)

from selenium.common.exceptions import TimeoutException

---

Cheers 😁😁

[fgisslen]

@uluQulu I still get this error:

WARNING [2018-08-22 13:10:49] [xxx] Could not locate Chrome: Message: timeout
(Session info: headless chrome=68.0.3440.106)
(Driver info: chromedriver=2.41.578700 (2f1ed5f9343c13f73144538f15c00b370eda6706),platform=Linux 4.15.0-32-generic x86_64)

[uluQulu]

@fgisslen
The code I have shared handles all of the errors while selenium executes GET method.
browser.get()
Why just it?

Cos after you have raise this issue I checked my own general.log file and all of the timeouts were coming from that method and I thought just handling it would be sufficient.

If you think you have applied the changes correctly and only after you see another timeout,
do this change in instapy.py file's end _method_ to troubleshoot it,

except WebDriverException as exc:
    self.logger.warning('Could not locate Chrome: {}'.format(exc))
    raise   ## ADD THIS LINE

Post all of the traceback from the exception occurred.

Note: In general there are many places timeout exception can occur.
And the ultimate solution without having learnt about the problems would be adding a new thread to watch timeout exceptions but that is totally inefficient solution. That's why I still recommend to find all of the cases that could occur and produce an efficient solution for it.

---

Cheers 😁

[santhoshraje]

Hi @uluQulu, I have been having the same issue and have been following this thread for awhile. I have made both the modifications suggested above (changed web_adress_navigator and end methods as shown above) but still get the same error.

I was performing unfollow action when the error occurred, but it happens for following too. I thought the problem was with chrome version 68, so I downgraded to chrome version 67, but the error remains. The error is as follows:

INFO [2018-08-23 18:54:31] [XXX] --> Ongoing Unfollow 49/1596, now unfollowing: b'XXX'

ERROR [2018-08-23 18:54:42] [XXX] Sorry, an error occured: 'WebDriverException' object has no attribute 'encode'
WARNING [2018-08-23 18:54:42] [XXX] Could not locate Chrome: Message: chrome not reachable
(Session info: headless chrome=67.0.3396.79)
(Driver info: chromedriver=2.41.578700 (2f1ed5f9343c13f73144538f15c00b370eda6706),platform=Linux 4.15.0-32-generic x86_64)

OOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO
ooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo

......................................................................
''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''

---

"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
Traceback (most recent call last):
File "InstaPy/quickstart.py", line 57, in
session.end()
File "/home/user/InstaPy/instapy/instapy.py", line 2742, in end
self.browser.delete_all_cookies()
File "/usr/local/lib/python3.6/dist-packages/selenium-2.53.6-py3.6.egg/selenium/webdriver/remote/webdriver.py", line 655, in delete_all_cookies
self.execute(Command.DELETE_ALL_COOKIES)
File "/usr/local/lib/python3.6/dist-packages/selenium-2.53.6-py3.6.egg/selenium/webdriver/remote/webdriver.py", line 236, in execute
self.error_handler.check_response(response)
File "/usr/local/lib/python3.6/dist-packages/selenium-2.53.6-py3.6.egg/selenium/webdriver/remote/errorhandler.py", line 192, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: chrome not reachable
(Session info: headless chrome=67.0.3396.79)
(Driver info: chromedriver=2.41.578700 (2f1ed5f9343c13f73144538f15c00b370eda6706),platform=Linux 4.15.0-32-generic x86_64)

Exception ignored in: >
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/selenium-2.53.6-py3.6.egg/selenium/webdriver/common/service.py", line 163, in __del__
File "/usr/local/lib/python3.6/dist-packages/selenium-2.53.6-py3.6.egg/selenium/webdriver/common/service.py", line 139, in stop
File "/usr/local/lib/python3.6/dist-packages/selenium-2.53.6-py3.6.egg/selenium/webdriver/common/service.py", line 110, in send_remote_shutdown_command
ImportError: sys.meta_path is None, Python is likely shutting down

[uluQulu]

Hey @santhoshraje
You are getting
'WebDriverException' object has no attribute 'encode'
which is already solved, pull latest file.

Then again if you still get another error or any other misbehavior, post its details to get help.

Also the second error message you get is
Message: chrome not reachable

Just note that the topic of this thread is opened for Timeout exceptions and you have no any such problem with it. If you have, please post its details.

For Message: chrome not reachable error message,
It is a typical error happened while trying to delete cookies while executing the end method (session.end()).
I can guess your quickstart finishes after that unfollow feature where it executes the end method.

---

Cheers 😁