Instapy: error when enabling nogui=True in InstaPy constructor (Heroku)

[chrismaltais]

Hey, I have my quickstart script deployed on Heroku, however when I try to run it via Heroku Scheduler I get the error:

Traceback (most recent call last):
  File "/app/.heroku/python/lib/python3.6/site-packages/easyprocess/__init__.py", line 225, in start
    env=self.env,
  File "/app/.heroku/python/lib/python3.6/subprocess.py", line 707, in __init__
    restore_signals, start_new_session)
  File "/app/.heroku/python/lib/python3.6/subprocess.py", line 1333, in _execute_child
    raise child_exception_type(errno_num, err_msg)
FileNotFoundError: [Errno 2] No such file or directory: 'Xvfb'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/.heroku/python/lib/python3.6/site-packages/easyprocess/__init__.py", line 178, in check_installed
    self.call()
  File "/app/.heroku/python/lib/python3.6/site-packages/easyprocess/__init__.py", line 194, in call
    self.start().wait(timeout=timeout)
  File "/app/.heroku/python/lib/python3.6/site-packages/easyprocess/__init__.py", line 230, in start
    raise EasyProcessError(self, 'start error')
easyprocess.EasyProcessError: start error <EasyProcess cmd_param=['Xvfb', '-help'] cmd=['Xvfb', '-help'] oserror=[Errno 2] No such file or directory: 'Xvfb' return_code=None stdout="None" stderr="None" timeout_happened=False>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "quickstart_heroku_scheduler.py", line 7, in <module>
    session = InstaPy(username=insta_username, password=insta_password, nogui=True)
  File "/app/instapy/instapy.py", line 56, in __init__
    self.display = Display(visible=0, size=(800, 600))
  File "/app/.heroku/python/lib/python3.6/site-packages/pyvirtualdisplay/display.py", line 34, in __init__
    self._obj = self.display_class(
  File "/app/.heroku/python/lib/python3.6/site-packages/pyvirtualdisplay/display.py", line 52, in display_class
    cls.check_installed()
  File "/app/.heroku/python/lib/python3.6/site-packages/pyvirtualdisplay/xvfb.py", line 38, in check_installed
    ubuntu_package=PACKAGE).check_installed()
  File "/app/.heroku/python/lib/python3.6/site-packages/easyprocess/__init__.py", line 180, in check_installed
    raise EasyProcessCheckInstalledError(self)
easyprocess.EasyProcessCheckInstalledError: cmd=['Xvfb', '-help']
OSError=[Errno 2] No such file or directory: 'Xvfb'
Program install error! 

It runs perfectly fine on my local without the nogui=True setting, I'm just not sure what to do here.
In addition, I've also tried pip install xvfbwrapper and placed xvfbwrapper in the requirements.txt file, and still no luck. Any help is appreciated!
quickstart.py for reference:

from instapy import InstaPy

insta_username = "removed"
insta_password = "removed"
#if you want to run this script on a server,
#simply add nogui=True to the InstaPy() constructor

session = InstaPy(username=insta_username, password=insta_password, nogui=True)
session.login()
session.set_lower_follower_count(limit = 100)
session.set_do_comment(True, percentage=20)
session.set_comments(['Cool! :)', 'This is awesome!', 'Love this! :)'])
session.set_unfollow_active_users(enabled=False, posts=5)
session.like_by_locations(['212946345/mountain-view-california/'], amount=100)
session.like_by_locations(['214459798/menlo-park-california/'], amount=100)
session.like_by_locations(['209661169/palo-alto-california/'], amount=100)
session.like_by_locations(['213941548/seattle-washington/'], amount=100)
session.like_by_tags(['algorithms', 'javascript', 'engineering'], amount=100)
session.unfollow_users(amount=10, onlyInstapyFollowed = True, onlyInstapyMethod = 'FIFO', sleep_delay=60)
session.follow_user_followers(['techcrunch', 'memes', 'googlestudents'], amount=20, random=False, sleep_delay=60)
session.interact_by_users(['techcrunch', 'google', 'facebook', 'kalesalad', 'amazon'], amount=2, random=True, media='Photo')
session.like_by_feed(amount=50, randomize=True, unfollow=False, interact=True)
session.end()

[chrismaltais]

This issue has now been resolved using chrome headless

1) Download chromedriver with _a version later_ than 2.9 (pref 2.33) and place in assets folder
2) In my quickstart.py remove nogui=True from the constructor
3) In instapy.py go to line 154-156, where chrome options are declared
4) Add chrome_options.add_argument('--headless') to enable chrome headless

Outcome: quickstart.py will now be able to run on a server without GUI

[timgrossmann]

@chrismaltais Did you encounter any quick bans or something similar from Instagram by using the headless mode?

[chrismaltais]

@timgrossmann not yet, but I've encountered quick bans from other sites using the headless mode.

I _have_ figured out a way around the bot detection (CAPTCHA, mostly) for some of the sites, I don't know how well it'll work with Instagram.

Some detection software tries to check if the browser's User Agent states its headless, and unfortunately on Headless Chrome states explicitly that we're using Headless Chrome. (See below)

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/60.0.3112.50 Safari/537.36

In python, we can explicitly change this to trick the bot detection software into thinking we're a normal user by replacing HeadlessChrome with Chrome. As far as I've tested it can actually be anything other than HeadlessChrome, it doesn't have to be anything specific.

The code below for instapy.py is one way to allow Headless Chrome to bypass any CAPTCHA screens that may appear from using the browser:

options = webdriver.ChromeOptions()
options.add_argument('--headless')
# change the user agent
user_agent = "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.50 Safari/537.36"
options.add_argument(f'user-agent={user_agent}')
self.browser = webdriver.Chrome(chromedriver_location, chrome_options=chrome_options)

[timgrossmann]

@chrismaltais This is awesome, could you create a PR for that? :wink:
Thank you very much for figuring this out!

[chrismaltais]

@timgrossmann in the middle of exams currently haha, give me a couple of days and it'll be done! :)

[timgrossmann]

@chrismaltais Sounds perfect, thank you very much!

[squigglesMcW]

Chris, thanks for sharing! Timely as I'm currently trying to migrate my code to an AWS server.

[carmenhchung]

Hi guys, sorry to bring this up again. I'm getting the same error (also when I deploy using Heroku Scheduler), but unlike the original poster, I do not have any reference to nogui - so I have no idea what I need to fix! 😞

The error I'm getting is this:

2019-01-12T11:39:35.358848+00:00 app[scheduler.7566]: easyprocess.EasyProcessCheckInstalledError: cmd=['Xvfb', '-help']
2019-01-12T11:39:35.358851+00:00 app[scheduler.7566]: OSError=[Errno 2] No such file or directory: 'Xvfb': 'Xvfb'
2019-01-12T11:39:35.358886+00:00 app[scheduler.7566]: Program install error!

Steps I have taken:

1. Set headless_browser=True in the quickstart.py script.
2. Added the Heroku chromedriver buildpack to my app.
3. Added the Heroku Google chrome buildpack to my app.
4. Added the following environment variables to my Heroku app (so that Chromedriver knows where Chrome is installed): GOOGLE_CHROME_BIN and GOOGLE_CHROME_SHIM, both with the values of /app/assets/chromedriver.

If anyone is able to point me in the right direction for getting this running via Heroku Scheduler, that would be massively appreciated! 🙃

[Chukstart]

Hi guys, sorry to bring this up again. I'm getting the same error (also when I deploy using Heroku Scheduler), but unlike the original poster, I do not have any reference to nogui - so I have no idea what I need to fix!

The error I'm getting is this:

2019-01-12T11:39:35.358848+00:00 app[scheduler.7566]: easyprocess.EasyProcessCheckInstalledError: cmd=['Xvfb', '-help']
2019-01-12T11:39:35.358851+00:00 app[scheduler.7566]: OSError=[Errno 2] No such file or directory: 'Xvfb': 'Xvfb'
2019-01-12T11:39:35.358886+00:00 app[scheduler.7566]: Program install error!

Steps I have taken:

1. Set headless_browser=True in the quickstart.py script.
2. Added the Heroku chromedriver buildpack to my app.
3. Added the Heroku Google chrome buildpack to my app.
4. Added the following environment variables to my Heroku app (so that Chromedriver knows where Chrome is installed): GOOGLE_CHROME_BIN and GOOGLE_CHROME_SHIM, both with the values of /app/assets/chromedriver.

If anyone is able to point me in the right direction for getting this running via Heroku Scheduler, that would be massively appreciated!

Does it work without using Heroku Scheduler