Gunicorn: WorkerTmp unlinking of temporary file causes issues under Cygwin

Created on 12 Sep 2012 · 13Comments · Source: benoitc/gunicorn

The "os.unlink(name)" call in WorkerTmp seems to cause Gunicorn to be totally non-functional (crash on run) on my windows machine under cygwin. Commenting it out fixes the issue. I'm not sure if the leaking of temporary files that a comment above refers to is occuring though. This could be some posix corner case, as the code is creating a file, getting a file descriptor for it, deleting it by name (not descriptor), then opening it with the file descriptor from before it was deleted.

Source

leftshoe

Most helpful comment

When using vagrant synced folders with virtualbox (vboxsf) this problem still occurs.

This problem can be circumvented by using the --worker-tmp-dir parameter and setting it to a local (non synced) filesystem.

WoLpH on 13 Jul 2015

❤7

All 13 comments

which error does it return?

benoitc on 21 Sep 2012

Some sort of file not found error. The file it is attempting to open using the file descriptor doesn't exist.

leftshoe on 21 Sep 2012

can you paste the exact trace?

benoitc on 27 Sep 2012

I've hit this error as well. The error output is below -- let me know if you need other diags.

$ gunicorn --log-level=debug beerulatrix.wsgi:application
2012-09-30 12:57:47 [5420] [INFO] Starting gunicorn 0.14.6
2012-09-30 12:57:48 [5420] [DEBUG] Arbiter booted
2012-09-30 12:57:48 [5420] [INFO] Listening at: http://127.0.0.1:8000 (5420)
2012-09-30 12:57:48 [5420] [INFO] Using worker: sync
2012-09-30 12:57:48 [4316] [INFO] Booting worker with pid: 4316
2012-09-30 12:57:48 [4316] [DEBUG] Exception in worker process:
Traceback (most recent call last):
  File "/cygdrive/f/dev/beerulatrix_project/venv/lib/python2.6/site-packages/gunicorn/arbiter.py", line 459, in spawn_worker
    worker.init_process()
  File "/cygdrive/f/dev/beerulatrix_project/venv/lib/python2.6/site-packages/gunicorn/workers/base.py", line 103, in init_process
    self.run()
  File "/cygdrive/f/dev/beerulatrix_project/venv/lib/python2.6/site-packages/gunicorn/workers/sync.py", line 26, in run
    self.notify()
  File "/cygdrive/f/dev/beerulatrix_project/venv/lib/python2.6/site-packages/gunicorn/workers/base.py", line 64, in notify
    self.tmp.notify()
  File "/cygdrive/f/dev/beerulatrix_project/venv/lib/python2.6/site-packages/gunicorn/workers/workertmp.py", line 34, in notify
    os.fchmod(self._tmp.fileno(), self.spinner)
OSError: [Errno 2] No such file or directory

Symmetric on 30 Sep 2012

👍1

fixed in 33feccafffeed20fb1d9ce049fe32ae5c6220d06

benoitc on 21 Dec 2012

@benoitc :+1: thanks for the fix!

pombredanne on 23 Dec 2012

though ... it does not fix the issue... let me investigate and come with some hints
Do you mind reopening the issue, or shall I create a new one?

pombredanne on 23 Dec 2012

reopened. That's odd after the fix I couldn't reproduce it. Also do you mind to upgrade to python 2.7. If you can let me know about the result.

benoitc on 23 Dec 2012

@pombredanne any more details?

benoitc on 5 Jan 2013

Here is the output when running the latest master HEAD test app:
$ gunicorn --workers=2 test:app 2013-01-05 23:10:15 [4408] [INFO] Starting gunicorn 0.17.1 2013-01-05 23:10:15 [4408] [INFO] Listening at: http://127.0.0.1:8000 (4408) 2013-01-05 23:10:15 [4408] [INFO] Using worker: sync 2013-01-05 23:10:15 [5060] [INFO] Booting worker with pid: 5060 2013-01-05 23:10:15 [1668] [INFO] Booting worker with pid: 1668 2013-01-05 23:10:15 [5060] [ERROR] Exception in worker process: Traceback (most recent call last): File "/home/pombredanne/gunicorn/src/gunicorn/gunicorn/arbiter.py", line 485, in spawn_worker worker.init_process() File "/home/pombredanne/gunicorn/src/gunicorn/gunicorn/workers/base.py", line 104, in init_process self.run() File "/home/pombredanne/gunicorn/src/gunicorn/gunicorn/workers/sync.py", line 30, in run self.notify() File "/home/pombredanne/gunicorn/src/gunicorn/gunicorn/workers/base.py", line 64, in notify self.tmp.notify() File "/home/pombredanne/gunicorn/src/gunicorn/gunicorn/workers/workertmp.py", line 35, in notify os.fchmod(self._tmp.fileno(), self.spinner) OSError: [Errno 2] No such file or directory Traceback (most recent call last): File "/home/pombredanne/gunicorn/src/gunicorn/gunicorn/arbiter.py", line 485, in spawn_worker worker.init_process() File "/home/pombredanne/gunicorn/src/gunicorn/gunicorn/workers/base.py", line 104, in init_process self.run() File "/home/pombredanne/gunicorn/src/gunicorn/gunicorn/workers/sync.py", line 30, in run self.notify() File "/home/pombredanne/gunicorn/src/gunicorn/gunicorn/workers/base.py", line 64, in notify self.tmp.notify() File "/home/pombredanne/gunicorn/src/gunicorn/gunicorn/workers/workertmp.py", line 35, in notify os.fchmod(self._tmp.fileno(), self.spinner) OSError: [Errno 2] No such file or directory 2013-01-05 23:10:15 [5060] [INFO] Worker exiting (pid: 5060) 2013-01-05 23:10:15 [1668] [ERROR] Exception in worker process: Traceback (most recent call last): File "/home/pombredanne/gunicorn/src/gunicorn/gunicorn/arbiter.py", line 485, in spawn_worker worker.init_process() File "/home/pombredanne/gunicorn/src/gunicorn/gunicorn/workers/base.py", line 104, in init_process self.run() File "/home/pombredanne/gunicorn/src/gunicorn/gunicorn/workers/sync.py", line 30, in run self.notify() File "/home/pombredanne/gunicorn/src/gunicorn/gunicorn/workers/base.py", line 64, in notify self.tmp.notify() File "/home/pombredanne/gunicorn/src/gunicorn/gunicorn/workers/workertmp.py", line 35, in notify os.fchmod(self._tmp.fileno(), self.spinner) OSError: [Errno 2] No such file or directory Traceback (most recent call last): File "/home/pombredanne/gunicorn/src/gunicorn/gunicorn/arbiter.py", line 485, in spawn_worker worker.init_process() File "/home/pombredanne/gunicorn/src/gunicorn/gunicorn/workers/base.py", line 104, in init_process self.run() File "/home/pombredanne/gunicorn/src/gunicorn/gunicorn/workers/sync.py", line 30, in run self.notify() File "/home/pombredanne/gunicorn/src/gunicorn/gunicorn/workers/base.py", line 64, in notify self.tmp.notify() File "/home/pombredanne/gunicorn/src/gunicorn/gunicorn/workers/workertmp.py", line 35, in notify os.fchmod(self._tmp.fileno(), self.spinner) OSError: [Errno 2] No such file or directory [... and so on in a loop ]

pombredanne on 5 Jan 2013

verified as fixed :

$ git pull
remote: Counting objects: 27, done.
remote: Compressing objects: 100% (11/11), done.
remote: Total 19 (delta 15), reused 12 (delta 8)
Unpacking objects: 100% (19/19), done.
From https://github.com/benoitc/gunicorn
   008161d..eb73681  master     -> origin/master
Updating 008161d..eb73681
Fast-forward
 gunicorn/http/wsgi.py         |    3 ---
 gunicorn/util.py              |   24 +++++++++++++++++++-----
 gunicorn/workers/gtornado.py  |    9 +++++++--
 gunicorn/workers/workertmp.py |    9 ++++++++-
 4 files changed, 34 insertions(+), 11 deletions(-)
(gunicorn)
pombredanne@computer ~/gunicorn/gunicorn
$ cd examples/
(gunicorn)
pombredanne@computer ~/gunicorn/gunicorn/examples
$ gunicorn --workers=2 test:app
2013-01-06 14:19:05 [3080] [INFO] Starting gunicorn 0.17.1
2013-01-06 14:19:05 [3080] [INFO] Listening at: http://127.0.0.1:8000 (3080)
2013-01-06 14:19:05 [3080] [INFO] Using worker: sync
2013-01-06 14:19:05 [3148] [INFO] Booting worker with pid: 3148
2013-01-06 14:19:05 [3844] [INFO] Booting worker with pid: 3844

and we have liftoff!
Thanks ++

pombredanne on 6 Jan 2013

When using vagrant synced folders with virtualbox (vboxsf) this problem still occurs.

This problem can be circumvented by using the --worker-tmp-dir parameter and setting it to a local (non synced) filesystem.

WoLpH on 13 Jul 2015

❤7

Thank you @WoLpH , telling gunicorn to use a local temp dir like that fixed this problem for us.