Netbox: gunicorn doesn't start on Ubuntu 16.04

Created on 30 Jun 2016  ·  19Comments  ·  Source: netbox-community/netbox

I followed the docs and installed everything to the default directories. After installing gunicorn and supervisor, I get '502 Bad Gateway' when I browse to the site.

When looking at 'service gunicorn status', I see:

● gunicorn.service
   Loaded: loaded (/etc/init.d/gunicorn; bad; vendor preset: enabled)
   Active: active (exited) since Wed 2016-06-29 20:41:33 MDT; 11min ago
     Docs: man:systemd-sysv-generator(8)
  Process: 1770 ExecStop=/etc/init.d/gunicorn stop (code=exited, status=0/SUC
  Process: 1778 ExecStart=/etc/init.d/gunicorn start (code=exited, status=0/S

Jun 29 20:41:33 host systemd[1]: Starting gunicorn.service...
Jun 29 20:41:33 host gunicorn[1778]:  * Starting Gunicorn workers
Jun 29 20:41:33 host gunicorn[1778]:  *
Jun 29 20:41:33 host systemd[1]: Started gunicorn.service.

When I try to run the gunicorn -c /opt/netbox/gunicorn_config.py netbox.wsgi command in the /etc/supervisor/conf.d/netbox.conf config file, from the /opt/netbox/netbox/ directory, I get the following:

[2016-06-29 20:51:34 +0000] [1890] [INFO] Starting gunicorn 19.4.5
[2016-06-29 20:51:34 +0000] [1890] [INFO] Listening at: http://127.0.0.1:8001 (1890)
[2016-06-29 20:51:34 +0000] [1890] [INFO] Using worker: sync
[2016-06-29 20:51:34 +0000] [1890] [INFO] Unhandled exception in main loop:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 180, in run
    self.manage_workers()
  File "/usr/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 482, in manage_workers
    self.spawn_workers()
  File "/usr/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 548, in spawn_workers
    self.spawn_worker()
  File "/usr/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 502, in spawn_worker
    self.cfg, self.log)
  File "/usr/lib/python2.7/dist-packages/gunicorn/workers/base.py", line 55, in __init__
    self.tmp = WorkerTmp(cfg)
  File "/usr/lib/python2.7/dist-packages/gunicorn/workers/workertmp.py", line 26, in __init__
    util.chown(name, cfg.uid, cfg.gid)
  File "/usr/lib/python2.7/dist-packages/gunicorn/util.py", line 172, in chown
    os.chown(path, uid, gid)
OSError: [Errno 1] Operation not permitted: '/tmp/wgunicorn-V9j_ep'

if I run as sudo, I get this:

[2016-06-29 20:52:26 +0000] [1929] [INFO] Starting gunicorn 19.4.5
[2016-06-29 20:52:26 +0000] [1929] [INFO] Listening at: http://127.0.0.1:8001 (1929)
[2016-06-29 20:52:26 +0000] [1929] [INFO] Using worker: sync
[2016-06-29 20:52:26 +0000] [1934] [INFO] Booting worker with pid: 1934
[2016-06-29 20:52:26 +0000] [1934] [ERROR] Exception in worker process:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 515, in spawn_worker
    worker.init_process()
  File "/usr/lib/python2.7/dist-packages/gunicorn/workers/base.py", line 122, in init_process
    self.load_wsgi()
  File "/usr/lib/python2.7/dist-packages/gunicorn/workers/base.py", line 130, in load_wsgi
    self.wsgi = self.app.wsgi()
  File "/usr/lib/python2.7/dist-packages/gunicorn/app/base.py", line 67, in wsgi
    self.callable = self.load()
  File "/usr/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 65, in load
    return self.load_wsgiapp()
  File "/usr/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 52, in load_wsgiapp
    return util.import_app(self.app_uri)
  File "/usr/lib/python2.7/dist-packages/gunicorn/util.py", line 366, in import_app
    __import__(module)
  File "/opt/netbox/netbox/netbox/wsgi.py", line 12, in <module>
    from django.core.wsgi import get_wsgi_application
ImportError: No module named django.core.wsgi
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 515, in spawn_worker
    worker.init_process()
  File "/usr/lib/python2.7/dist-packages/gunicorn/workers/base.py", line 122, in init_process
    self.load_wsgi()
  File "/usr/lib/python2.7/dist-packages/gunicorn/workers/base.py", line 130, in load_wsgi
    self.wsgi = self.app.wsgi()
  File "/usr/lib/python2.7/dist-packages/gunicorn/app/base.py", line 67, in wsgi
    self.callable = self.load()
  File "/usr/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 65, in load
    return self.load_wsgiapp()
  File "/usr/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 52, in load_wsgiapp
    return util.import_app(self.app_uri)
  File "/usr/lib/python2.7/dist-packages/gunicorn/util.py", line 366, in import_app
    __import__(module)
  File "/opt/netbox/netbox/netbox/wsgi.py", line 12, in <module>
    from django.core.wsgi import get_wsgi_application
ImportError: No module named django.core.wsgi
[2016-06-29 20:52:26 +0000] [1934] [INFO] Worker exiting (pid: 1934)
[2016-06-29 20:52:26 +0000] [1935] [INFO] Booting worker with pid: 1935
[2016-06-29 20:52:26 +0000] [1935] [ERROR] Exception in worker process:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 515, in spawn_worker
    worker.init_process()
  File "/usr/lib/python2.7/dist-packages/gunicorn/workers/base.py", line 122, in init_process
    self.load_wsgi()
  File "/usr/lib/python2.7/dist-packages/gunicorn/workers/base.py", line 130, in load_wsgi
    self.wsgi = self.app.wsgi()
  File "/usr/lib/python2.7/dist-packages/gunicorn/app/base.py", line 67, in wsgi
    self.callable = self.load()
  File "/usr/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 65, in load
    return self.load_wsgiapp()
  File "/usr/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 52, in load_wsgiapp
    return util.import_app(self.app_uri)
  File "/usr/lib/python2.7/dist-packages/gunicorn/util.py", line 366, in import_app
    __import__(module)
  File "/opt/netbox/netbox/netbox/wsgi.py", line 12, in <module>
    from django.core.wsgi import get_wsgi_application
ImportError: No module named django.core.wsgi
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 515, in spawn_worker
    worker.init_process()
  File "/usr/lib/python2.7/dist-packages/gunicorn/workers/base.py", line 122, in init_process
    self.load_wsgi()
  File "/usr/lib/python2.7/dist-packages/gunicorn/workers/base.py", line 130, in load_wsgi
    self.wsgi = self.app.wsgi()
  File "/usr/lib/python2.7/dist-packages/gunicorn/app/base.py", line 67, in wsgi
    self.callable = self.load()
  File "/usr/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 65, in load
    return self.load_wsgiapp()
  File "/usr/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 52, in load_wsgiapp
    return util.import_app(self.app_uri)
  File "/usr/lib/python2.7/dist-packages/gunicorn/util.py", line 366, in import_app
    __import__(module)
  File "/opt/netbox/netbox/netbox/wsgi.py", line 12, in <module>
    from django.core.wsgi import get_wsgi_application
ImportError: No module named django.core.wsgi
[2016-06-29 20:52:26 +0000] [1935] [INFO] Worker exiting (pid: 1935)
[2016-06-29 20:52:26 +0000] [1929] [INFO] Shutting down: Master
[2016-06-29 20:52:26 +0000] [1929] [INFO] Reason: Worker failed to boot.
bug
Source
picture mike-mosher

Most helpful comment

I fixed this with the link from @therealjoshuad and running ALL of my "pip" commands as "sudo -H"

picture cobbst-lcs on 8 Nov 2016
👍4

All 19 comments

Did you run pip install -r requirements.txt during the installation of Netbox?
This should resolve your missing modules.

troxil picture troxil on 30 Jun 2016

I'm going to close this issue. I worked with stretch on irc all night, and he got it working running Ubuntu 16.04 (which is what I'm running, freshly installed). Must be something with my specific server.

mike-mosher picture mike-mosher on 30 Jun 2016

I rebuild a new VM with Ubuntu 16.04 and ran through the documentation again. I am having the same exact issue. Anyone else seeing this?

mike-mosher picture mike-mosher on 30 Jun 2016
👍3

Does pip install django.core.wsgi help?
According to your error dump before - it's not installed/available.

troxil picture troxil on 30 Jun 2016

@troxil:

~$ pip install django.core.wsgi
Collecting django.core.wsgi
  Could not find a version that satisfies the requirement django.core.wsgi (from versions: )
No matching distribution found for django.core.wsgi
mike-mosher picture mike-mosher on 30 Jun 2016

I rebuilt with Ubuntu 14.04 and everything is working great. Does anyone else have issues with 16.04?

mike-mosher picture mike-mosher on 30 Jun 2016

@mike-mosher I'm going to close this issue and we can start a new one for Ubuntu 16.04.

mdlayher picture mdlayher on 30 Jun 2016

I am having the exact same issue on Debian Jessie

$ sudo gunicorn -c /opt/netbox/gunicorn_config.py netbox.wsgi
Traceback (most recent call last):
  File "/usr/bin/gunicorn", line 9, in <module>
    load_entry_point('gunicorn==19.0.0', 'console_scripts', 'gunicorn')()
  File "/usr/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 74, in run
    WSGIApplication("%(prog)s [OPTIONS] [APP_MODULE]").run()
  File "/usr/lib/python2.7/dist-packages/gunicorn/app/base.py", line 166, in run
    super(Application, self).run()
  File "/usr/lib/python2.7/dist-packages/gunicorn/app/base.py", line 71, in run
    Arbiter(self).run()
  File "/usr/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 169, in run
    self.manage_workers()
  File "/usr/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 477, in manage_workers
    self.spawn_workers()
  File "/usr/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 537, in spawn_workers
    time.sleep(0.1 * random.random())
  File "/usr/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 209, in handle_chld
    self.reap_workers()
  File "/usr/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 459, in reap_workers
    raise HaltServer(reason, self.WORKER_BOOT_ERROR)
gunicorn.errors.HaltServer: <HaltServer 'Worker failed to boot.' 3>

2016-08-01 16:09:09,305 INFO exited: netbox (exit status 127; not expected)
2016-08-01 16:09:12,313 INFO spawned: 'netbox' with pid 8617
2016-08-01 16:09:12,327 INFO exited: netbox (exit status 127; not expected)
2016-08-01 16:09:13,329 INFO gave up: netbox entered FATAL state, too many start retries too quickly

supervisor: couldn't chdir to /opt/netbox/netbox/: EACCES
supervisor: child process was not spawned
supervisor: couldn't chdir to /opt/netbox/netbox/: EACCES
supervisor: child process was not spawned

Earlier the error was probably caused by permission. I set www-data to own /opt/netbox/netbox, but gunicorn is still failing

jkldgoefgkljefogeg picture jkldgoefgkljefogeg on 1 Aug 2016

I am running Debian 8.6 and I had the same permission issue but what I did was a chown www-data:www-data -R /opt/netbox and after that I could start gunicorn manually:

root@netbox:/opt/netbox# sudo gunicorn -c /opt/netbox/gunicorn_config.py netbox.wsgi
[2016-10-04 08:13:21 +0000] [14124] [INFO] Starting gunicorn 19.6.0
[2016-10-04 08:13:21 +0000] [14124] [INFO] Listening at: http://127.0.0.1:8001 (14124)
[2016-10-04 08:13:21 +0000] [14124] [INFO] Using worker: sync
[2016-10-04 08:13:21 +0000] [14129] [INFO] Booting worker with pid: 14129
[2016-10-04 08:13:21 +0000] [14132] [INFO] Booting worker with pid: 14132
[2016-10-04 08:13:21 +0000] [14135] [INFO] Booting worker with pid: 14135

I have another issue though: I try to connect to netbox, locally, using elinks http://127.0.0.1:8001 and I get a 400 eror (Bad request). I thought at first that it has to do with nginx config so I switched to apache. Same error applies.

My guess is that something is wrong with gunicorn but I can't figure it out. If I start nebox manually as in the guide (/manage.py runserver 0.0.0.0:8000 --insecure) it works just fine.

Any help would be appreciated!

bogdanstoica35 picture bogdanstoica35 on 4 Oct 2016

I'm still new to Linux, so this may not be your issue, but this helped me:
http://unix.stackexchange.com/questions/281774/ubuntu-server-16-04-cannot-get-supervisor-to-start-automatically

therealjoshuad picture therealjoshuad on 4 Oct 2016

@therealjoshuad I tried that and my error went from a 502 to 400. So progress at least.

Edit: The 400 was an unrelated "ALLOWED_HOSTS" issue. That fix seemed to work for me.

Fyzzle picture Fyzzle on 5 Oct 2016

I fixed this with the link from @therealjoshuad and running ALL of my "pip" commands as "sudo -H"

cobbst-lcs picture cobbst-lcs on 8 Nov 2016
👍4

I ran into this issue as well from a fresh Ubuntu 16.04.2 install and using python 3. Gunicorn is trying to run from python2.7 

File "/usr/lib/python2.7/dist-packages/gunicorn/util.py", line 366, in import_app
    __import__(module)
  File "/opt/netbox/netbox/netbox/wsgi.py", line 12, in <module>
    from django.core.wsgi import get_wsgi_application

Installing gunicorn (and uwsgi, maybe not required) for python3 fixed the issue for me.
Run:sudo -H pip3 install gunicorn uwsgi

LociNetworks picture LociNetworks on 21 Jul 2017
👍2

@LociNetworks, I have the same issue. I am using python3.6 on Linux Ubuntu but when I run this command gunicorn --bind 0.0.0.0:8000 myproject.wsgi it looks into Python2.7 directories. I tried the command what you mentioned Run:sudo -H pip3 install gunicorn uwsgi, but its not working for me. Any idea?

waqas80 picture waqas80 on 2 Oct 2017

I use the following command
gunicorn --bind 0.0.0.0:8000 django_project.wsgi:application

and below is the error detail,
[2017-10-02 18:50:57 +0000] [15059] [INFO] Starting gunicorn 19.4.5
[2017-10-02 18:50:57 +0000] [15059] [INFO] Listening at: http://0.0.0.0:8000 (15059)
[2017-10-02 18:50:57 +0000] [15059] [INFO] Using worker: sync
[2017-10-02 18:50:57 +0000] [15064] [INFO] Booting worker with pid: 15064
[2017-10-02 18:50:57 +0000] [15064] [ERROR] Exception in worker process:
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 515, in spawn_worker
worker.init_process()
File "/usr/lib/python2.7/dist-packages/gunicorn/workers/base.py", line 122, in init_process
self.load_wsgi()
File "/usr/lib/python2.7/dist-packages/gunicorn/workers/base.py", line 130, in load_wsgi
self.wsgi = self.app.wsgi()
File "/usr/lib/python2.7/dist-packages/gunicorn/app/base.py", line 67, in wsgi
self.callable = self.load()
File "/usr/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 65, in load
return self.load_wsgiapp()
File "/usr/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 52, in load_wsgiapp
return util.import_app(self.app_uri)
File "/usr/lib/python2.7/dist-packages/gunicorn/util.py", line 366, in import_app
__import__(module)
ImportError: No module named django_project.wsgi
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 515, in spawn_worker
worker.init_process()
File "/usr/lib/python2.7/dist-packages/gunicorn/workers/base.py", line 122, in init_process
self.load_wsgi()
File "/usr/lib/python2.7/dist-packages/gunicorn/workers/base.py", line 130, in load_wsgi
self.wsgi = self.app.wsgi()
File "/usr/lib/python2.7/dist-packages/gunicorn/app/base.py", line 67, in wsgi
self.callable = self.load()
File "/usr/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 65, in load
return self.load_wsgiapp()
File "/usr/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 52, in load_wsgiapp
return util.import_app(self.app_uri)
File "/usr/lib/python2.7/dist-packages/gunicorn/util.py", line 366, in import_app
__import__(module)
ImportError: No module named django_project.wsgi
[2017-10-02 18:50:57 +0000] [15064] [INFO] Worker exiting (pid: 15064)
[2017-10-02 18:50:58 +0000] [15059] [INFO] Shutting down: Master
[2017-10-02 18:50:58 +0000] [15059] [INFO] Reason: Worker failed to boot.

I am not sure why its checking into python2.7. I am using python 3.6.2 on Linux Ubuntu.

waqas80 picture waqas80 on 2 Oct 2017

@waqas80 Can you provide the output of 'pip3 list'

LociNetworks picture LociNetworks on 2 Oct 2017

@LociNetworks thanks I found the mistake. I install it package at OS level. I remove the package and reinstall while activating the virtual environment I created. Now its fine.

But now I am not able to find the following file at the following path
sudo nano /etc/systemd/system/gunicorn.service

setting up django on Linux I am following the steps from the below location,
https://gist.github.com/ganmedia/afbada8ff6ff1733e27f7fd818eb34fd

waqas80 picture waqas80 on 3 Oct 2017
👍1

@LociNetworks sudo -H pip3 install gunicorn uwsgi fixed the problem for me, thanks!

jeremyjenkins5 picture jeremyjenkins5 on 23 Mar 2018

pip3 uninstall gunicorn at the OS & env level, then pip3 install gunicorn at the env level, fixed the issue for me.

rstmsn picture rstmsn on 18 Jul 2019
Was this page helpful?
0 / 5 - 0 ratings

Related issues

shugotek picture shugotek  ·  3Comments
Grokzen picture Grokzen  ·  3Comments
cloos picture cloos  ·  3Comments
markve-sa picture markve-sa  ·  4Comments
bellwood picture bellwood  ·  4Comments