Gunicorn: os.chown not working in snap packaging

[yakkle]

WorkerTmp create temp file and chown to allow write to file.

When I want to packaging snap our app with gunicorn, it show error.

Traceback (most recent call last):
  File "/snap/loopchain/6/lib/python3.6/site-packages/gunicorn/arbiter.py", line 203, in run
    self.manage_workers()
  File "/snap/loopchain/6/lib/python3.6/site-packages/gunicorn/arbiter.py", line 545, in manage_workers
    self.spawn_workers()
  File "/snap/loopchain/6/lib/python3.6/site-packages/gunicorn/arbiter.py", line 616, in spawn_workers
    self.spawn_worker()
  File "/snap/loopchain/6/lib/python3.6/site-packages/gunicorn/arbiter.py", line 565, in spawn_worker
    self.cfg, self.log)
  File "/snap/loopchain/6/lib/python3.6/site-packages/sanic/worker.py", line 30, in __init__
    super().__init__(*args, **kw)
  File "/snap/loopchain/6/lib/python3.6/site-packages/gunicorn/workers/base.py", line 58, in __init__
    self.tmp = WorkerTmp(cfg)
  File "/snap/loopchain/6/lib/python3.6/site-packages/gunicorn/workers/workertmp.py", line 26, in __init__
    util.chown(name, cfg.uid, cfg.gid)
  File "/snap/loopchain/6/lib/python3.6/site-packages/gunicorn/util.py", line 173, in chown
    os.chown(path, uid, gid)
PermissionError: [Errno 1] Operation not permitted: '/tmp/wgunicorn-7phm2h5f'

chown is not working that running via snap strict confinement.

So, I workaround with monkey_patch to do not chown when `SNAP=1' env option with provided.

https://forum.snapcraft.io/t/problems-packaging-app-that-uses-gunicorn/11749/12

I think gunicorn need alternative method to running via snap, for example new option for snap.

[tilgovi]

That's very interesting. Are you providing the --user or --group options to Gunicorn?

[tilgovi]

The chown system call may be prohibited by Snap. Right now Gunicorn calls it always, but it should only be needed when Gunicorn runs workers as a different user or group than the master process.

This patch to have Gunicorn only call chown if it needs to might be a good fix:

diff --git a/gunicorn/workers/workertmp.py b/gunicorn/workers/workertmp.py
index 22aaef3..d33cf88 100644
--- a/gunicorn/workers/workertmp.py
+++ b/gunicorn/workers/workertmp.py
@@ -21,11 +21,12 @@ class WorkerTmp(object):
         if fdir and not os.path.isdir(fdir):
             raise RuntimeError("%s doesn't exist. Can't create workertmp." % fdir)
         fd, name = tempfile.mkstemp(prefix="wgunicorn-", dir=fdir)
-
-        # allows the process to write to the file
-        util.chown(name, cfg.uid, cfg.gid)
         os.umask(old_umask)

+        # ensure the worker process will be able to write to the file
+        if cfg.uid != os.geteuid() or cfg.gid != os.getegid():
+            util.chown(name, cfg.uid, cfg.gid)
+
         # unlink the file so we don't leak tempory files
         try:
             if not IS_CYGWIN:

[yakkle]

That's very interesting. Are you providing the --user or --group options to Gunicorn?

No, I didn't provide user or group option. Our app using gunicorn as custom application.
Maybe worker process user and group as same as gunicorn process one.

Your suggestion patch would be better than using env option. 👍
I will try it.

thanks.

[benoitc]

did you try setting to set the temp directory in a storagr where your snap can write also ? You can do it with `--worker-tmp-dir DIRˋ .

[tilgovi]

My recommendation is based on a quick search about snaps where it seemed that snaps typically _can_ write to the temporary directory but chown may be a restricted system call.

[benoitc]

@tilgovi hrm OK that's interesting, i'm wondering about their reasons, ddid you found anything about it? Is chown completely forbidden in /tmp with snap? In this case setting a user and group will return an error isn't it? In such case shouldn't we rather warn the user about it?

I propose also to investigate if there are some flags or tools provided by snap already that can allow us to work around. Thoughts?

[tilgovi]

I did not look further because I realized that we have a potential patch that seems safe and effective for the case where Gunicorn is not changing users/groups.

[tilgovi]

We could implement the patch I have, above, and it would work for some cases but raise an error for the case where Gunicorn has --user or --group but fails to set it.

[benoitc]

@tilgovi indeed :) Let's apply it then. In the mean time we should open a ticket to see if we can find a complete solution for the user. I can take care care of it if it's oK for you.

[tilgovi]

Works for me. I'll push that change right now, since I have it in my workdir.

[tilgovi]

I don't think it will fix this issue, though, because we still need to call fchmod later on the file.

[tilgovi]

In that case, let's just leave this open and look for a better solution, since I now realize it won't actually fix the problem I'm describing.

[benoitc]

alright :-)

IIn the mean time @yakkle i think setting the worker temporary directory outside the snap on in a zone where snap can write may work. Let us know.

[yakkle]

did you try setting to set the temp directory in a storagr where your snap can write also ? You can do it with `--worker-tmp-dir DIRˋ .

@benoitc
I had tried to using--worker-tmp-dir on $SNAP_USER_DATA(rw), but chown didn't work too.

snap using squashfs and it is running via isolated environment.
check this document _Snap file format_ section and this.

I don't think it will fix this issue, though, because we still need to call fchmod later on the file.

@tilgovi
fchmod is working on /tmp.
you can try to test with snap in ubuntu.

$ sudo snap install --edge loopchain
$ snap run --shell loopchain
inside_snap/shell $ python3
Python 3.6.7 (default, Oct 22 2018, 11:32:17)
[GCC 8.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> import tempfile
>>> fd, name = tempfile.mkstemp(prefix='wgunicorn-', dir='/tmp')
>>> fd, name
(3, '/tmp/wgunicorn-f0kg66op')
>>> os.fstat(fd)
os.stat_result(st_mode=33152, st_ino=258220, st_dev=2049, st_nlink=1, st_uid=1000, st_gid=1000, st_size=0, st_atime=1560750423, st_mtime=1560750423, st_ctime=1560750423)
>>> os.fchmod(fd, 0)
>>> os.fstat(fd)
os.stat_result(st_mode=32768, st_ino=258220, st_dev=2049, st_nlink=1, st_uid=1000, st_gid=1000, st_size=0, st_atime=1560750423, st_mtime=1560750423, st_ctime=1560750638)
>>>

[benoitc]

that's odd, why would fchmod work and not chown... . Can you try to replace the chown line by the following:

util.fchown(fd, cfg.uid, cfg.gid)

Let me know.

[yakkle]

@benoitc

Could you check this link first.
https://forum.snapcraft.io/t/permission-denied-error-when-running-a-daemon/10299/2

os.fchown is not working too.

Python 3.6.7 (default, Oct 22 2018, 11:32:17)
[GCC 8.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> import tempfile
>>> fd, name = tempfile.mkstemp(prefix='wgunicorn-', dir='/tmp')
>>> os.fchown(fd, 1000, 1000)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
PermissionError: [Errno 1] Operation not permitted
>>>

[benoitc]

@yakkle thanks for the link and the test. The suggestion of using XDG_RUNTIME_DIR is good . Gunicorn can be patched to use it IMO.

[tilgovi]

If fchmod works and fchown does not, then I think we can commit my patch for now, too.

[tilgovi]

2060

[tilgovi]

I merged #2060 and I'm curious if this is still a problem.

[yakkle]

@tilgovi

I have tested tag/19.9.0 and master.
using this snapcraft.yaml : https://gist.github.com/yakkle/6a2291beea23e26eea81e60d1fb689b0

tag/19.9.0 :

multipass@ubuntu:~/snap/gunicorn/common$ snap list gunicorn
Name      Version  Rev  Tracking  Publisher  Notes
gunicorn  19.9.0   x1   -         -          -
multipass@ubuntu:~/snap/gunicorn/common$ gunicorn unicorn:app
[2019-08-22 07:52:30 +0900] [16097] [INFO] Starting gunicorn 19.9.0
[2019-08-22 07:52:30 +0900] [16097] [INFO] Listening at: http://127.0.0.1:8000 (16097)
[2019-08-22 07:52:30 +0900] [16097] [INFO] Using worker: sync
[2019-08-22 07:52:30 +0900] [16097] [INFO] Unhandled exception in main loop
Traceback (most recent call last):
  File "/snap/gunicorn/x1/lib/python3.6/site-packages/gunicorn/arbiter.py", line 203, in run
    self.manage_workers()
  File "/snap/gunicorn/x1/lib/python3.6/site-packages/gunicorn/arbiter.py", line 545, in manage_workers
    self.spawn_workers()
  File "/snap/gunicorn/x1/lib/python3.6/site-packages/gunicorn/arbiter.py", line 616, in spawn_workers
    self.spawn_worker()
  File "/snap/gunicorn/x1/lib/python3.6/site-packages/gunicorn/arbiter.py", line 565, in spawn_worker
    self.cfg, self.log)
  File "/snap/gunicorn/x1/lib/python3.6/site-packages/gunicorn/workers/base.py", line 58, in __init__
    self.tmp = WorkerTmp(cfg)
  File "/snap/gunicorn/x1/lib/python3.6/site-packages/gunicorn/workers/workertmp.py", line 26, in __init__
    util.chown(name, cfg.uid, cfg.gid)
  File "/snap/gunicorn/x1/lib/python3.6/site-packages/gunicorn/util.py", line 173, in chown
    os.chown(path, uid, gid)
PermissionError: [Errno 1] Operation not permitted: '/tmp/wgunicorn-siadxldp'

master :

multipass@ubuntu:~/snap/gunicorn/common$ snap list gunicorn
Name      Version  Rev  Tracking  Publisher  Notes
gunicorn  master   x2   -         -          -
multipass@ubuntu:~/snap/gunicorn/common$ gunicorn unicorn:app
[2019-08-22 07:56:37 +0900] [16272] [INFO] Starting gunicorn 19.9.0
[2019-08-22 07:56:37 +0900] [16272] [INFO] Listening at: http://127.0.0.1:8000 (16272)
[2019-08-22 07:56:37 +0900] [16272] [INFO] Using worker: sync
[2019-08-22 07:56:37 +0900] [16298] [INFO] Booting worker with pid: 16298
^C[2019-08-22 07:56:39 +0900] [16272] [INFO] Handling signal: int
[2019-08-22 07:56:39 +0900] [16272] [INFO] Shutting down: Master

It works now on master branch.
thank you!

[tilgovi]

Perfect. Thank you for testing!