Gunicorn: `--reload` option not working

[ahmedsaoudi]

Upon using Gunicorn for my Django app, I tried to enable the --reload option to make my work easier, avoiding manual restarting for every change.

Even though I tried the option on the command line (gunicron --reload --bind ...), I normally run Gunicorn as a systemd service. Here the config file:

[Unit]
Description='gunicorn daemon'
After=network.target

[Service]
User=me
Group=www-data
WorkingDirectory=path/to/djangoapp
ExecStart=/path/to/Envs/venv/bin/gunicorn --access-logfile - --error-logfile /path/to/gunicorn-error.log --workers 3 --reload --bind unix:/path/to/djangoapp/appname.sock appname.wsgi:application
ExecReload=/bin/kill -s HUP $MAINPID
ExecStop=/bin/kill -s TERM $MAINPID

[Install]
WantedBy=multi-user.target

As you can see, I'm using Gunicorn from a Virtualenv, as the rest of the app; but I don't know if that's related.

And apparently, I'm not alone; somebody else has the same issue: https://unix.stackexchange.com/questions/375292/how-to-restart-systemd-service-after-files-update-under-gunicorn-server

[benoitc]

is there any log that can be useful?

[Code0x58]

Does it just never work? The way the reloading works might not be quite as simple as you expect, see the code:

• if inotify isn't installed (or you use --reload-engine=poll) then gunicorn will only look for changes in your loaded python modules (including those added after each worker starts), and files added using --reload-extra-file
  
  
  
  • if you don't use a .py file extension and your file name ends with pyo/pyc, then it won't be watched
  
  

• otherwise the directories of python modules which were loaded when the watcher started, and of the files provided with --reload-extra-file will be watched for changes using inotify

  • files added to directories which were created after the watcher was created won't be watched
  • subdirectories of those watched are not automatically watched
  • if your editor uses memory mapping to modify the file, then it the changes won't be detected

  Modules that only imported in functions or inside something like if __name__ == '__main__':, unless they are loaded as a result of your gunicorn configuration file, or WSGI entry point.
  
  You could use inotifywait to see if the issue is to do with patchy watching, or inotify limitations.

Both implementations will have issues with your templates and static content also won't be monitored for changes.

As you are using Django, it may be reasonable for you to just use manage.py runserver, although that may also not be as good as using a hand configured inotifywait to manage reloads.

[tilgovi]

It might not work if you don't load your application code during configuration phase. I addressed that in my #1565 change. I can port that change to master.
https://github.com/benoitc/gunicorn/pull/1565/files#diff-454219f8d55400a662c02ab2a83186ae

[tilgovi]

A few separate issues:

• The .py file that corresponds to a .pyc file or .pyo file is watched. It's true that if you were to change the .pyc file it would not trigger a reload, but in the common case a .py file is changed and only if it's reloaded will a new .pyc be written. I suppose if you are vendoring .pyc files and you want to update them, or shipping pre-bytecode-compiled packages and wanting a reload when they're updated, this doesn't serve your use right now. We can open an issue specifically for that. I'm not immediately sure the best way to address it. What you say about watching directories might make sense.

• If your editor uses memory mapping to modify the file, it's not detected? Why not? When the editor saves, shouldn't the file be changed?

• I think we need to start the reloader after loading the wsgi entry point, or we might miss directories. Again, I think your recursive directory approach sounds okay, too, though.

[Code0x58]

• the regex that converts the name will change sampyo to sampy as well (module file names don't require a .py suffix, and the regex doesn't check for \.). Admittedly very unlikely, especially with .pyo generated for imported modules. There would also be the issue with pre-compiled byte code that you mentioned too
• see the limitations of inotify under notes on the man page. I think this may also be brought up in the python module's documentation about editors
• I agree. I am not sure if changing the order will lead to issues with user/gunicorn code being executed in a different order and causing issues for some users upgrading

[Code0x58]

I just thought of another possible reason which would explain it not working as expected: the reload works by killing the worker processes and then letting the arbiter fork new workers, so any modules loaded pre-fork won't be reloaded by the python interpreter.

If this is happening then you'll see in the logs that workers are firing up for requests after a change, but the app itself hasn't changed/reloaded.

[tilgovi]

The docs for reload say that it's incompatible with preloading.
http://docs.gunicorn.org/en/latest/settings.html#reload

I think we do not cache any modules imported while loading a Python config file, so that should not break reloading either.

[tilgovi]

Maybe if the user imports application code for the arbiter hooks, that could break reloading.

[javabrett]

Running gunicorn 19.7.1, I have an application which imports a Flask app from webapp.py:

from webapp import app

...

class StandaloneApplication(gunicorn.app.base.BaseApplication):

...

    def __init__(self, app, options=None):
        self.options = options or {}
        self.application = app
        super(StandaloneApplication, self).__init__()

    def load_config(self):
        config = dict([(key, value) for key, value in iteritems(self.options)
                       if key in self.cfg.settings and value is not None])

        for key, value in iteritems(config):
            self.cfg.set(key.lower(), value)

    def load(self):
        return self.application

if __name__ == '__main__':
    options = {
        'bind': ...,
        'workers': 8,
        'accesslog': '/....',
        'errorlog': '/....',
        'capture_output': True,
        'loglevel': 'debug',
        'reload': True,
    }
    # make Flask propagate exceptions in middleware back to us for logging
    app.config['PROPAGATE_EXCEPTIONS'] = True
    StandaloneApplication(app, options).run()

I don't believe I have inotify installed and am probably polling with reload-mode auto.

When I modify source-code, I see this in the logs:

[2018-02-14 03:01:22 +0000] [534] [INFO] Worker reloading: /...../health_check.py modified
[2018-02-14 03:01:22 +0000] [531] [INFO] Worker reloading: /...../health_check.py modified
...
[2018-02-14 03:01:26 +0000] [529] [INFO] Worker exiting (pid: 529)
[2018-02-14 03:01:26 +0000] [527] [INFO] Worker exiting (pid: 527)
[2018-02-14 03:01:27 +0000] [899] [INFO] Booting worker with pid: 899
[2018-02-14 03:01:27 +0000] [29546] [DEBUG] 8 workers
[2018-02-14 03:01:27 +0000] [901] [INFO] Booting worker with pid: 901

This was my second attempt to get the health-check code to return a different dict - I removed a variable and replaced it with a string-constant. A GET serviced by health_check.py still returns the old values.

Any idea why Gunicorn is able to detect the source-change, but nothing changes in the response (other than a timestamp i.e. the response is not cached).

[tilgovi]

You would need to delay the import until the load() call.

def load(self):
    from wsgiapp import app
    return app

With the way you have it, the webapp code is imported by the Gunicorn arbiter at startup. Even when workers are killed and the arbiter forks new workers the old code is already imported and in the Python module cache. If you defer the import until load() it will be loaded by each of the workers after they fork.

[berkerpeksag]

Perhaps we could improve the example at http://docs.gunicorn.org/en/stable/custom.html or add a note about how to import the WSGI app?

[javabrett]

@tilgovi thanks, indeed with the late import reloading works perfectly.

I wonder if there is any way to detect this sort of error when reload is true? Otherwise yes @berkerpeksag updated example code would also be useful.

Anyway, this works for me now.

[billzhong]

I have the exactly same issue @javabrett mentioned. Thanks @tilgovi for the solution.
An updated document or example will be great as @berkerpeksag suggested.

[qcho]

Hi,

I think I'm having the same issue. The difference I'm not using any "Custom App" not I have gunicorn dependency on my proyect. My idea is to provide a WSGI app and then in production mounit it over gunicorn.

Also the idea is to have similar dev environment thats why i'm using --reload option setting it up in my gunicorn config file. In docker the only difference between prod and dev are some debugging params ^_^

Is there any way to define the "load" function in the GUNICORN Conf file so it tells future workers to perform the reload later?

It seems the process /usr/local/bin/python /usr/local/bin/gunicorn is caching the code.

I too receive the notifications properly:

app_1  | [2019-04-07 18:33:09 +0000] [18] [INFO] Worker reloading: /app/apis/auth.py modified
app_1  | [2019-04-07 18:33:09 +0000] [20] [INFO] Worker reloading: /app/apis/auth.py modified
app_1  | [2019-04-07 18:33:09 +0000] [26] [INFO] Worker reloading: /app/apis/auth.py modified
app_1  | [2019-04-07 18:33:09 +0000] [30] [INFO] Worker reloading: /app/apis/auth.py modified
app_1  | [2019-04-07 18:33:09 +0000] [10] [INFO] Worker reloading: /app/apis/auth.py modified
app_1  | [2019-04-07 18:33:09 +0000] [12] [INFO] Worker reloading: /app/apis/auth.py modified
app_1  | [2019-04-07 18:33:09 +0000] [14] [INFO] Worker reloading: /app/apis/auth.py modified
app_1  | [2019-04-07 18:33:10 +0000] [16] [INFO] Worker reloading: /app/apis/auth.py modified

This is what i have running:

root         7  0.2  1.1 104588 23148 ?        S    18:32   0:00 /usr/local/bin/python /usr/local/bin/gunicorn -k egg:meinheld#gunicorn_worker -c /app/gunicorn_conf.py main:app
root        10  0.5  1.8 367552 37436 ?        Sl   18:32   0:00  \_ /usr/local/bin/python /usr/local/bin/gunicorn -k egg:meinheld#gunicorn_worker -c /app/gunicorn_conf.py main:app
root        12  0.5  1.8 367296 37408 ?        Sl   18:32   0:00  \_ /usr/local/bin/python /usr/local/bin/gunicorn -k egg:meinheld#gunicorn_worker -c /app/gunicorn_conf.py main:app
root        14  0.5  1.8 367296 37412 ?        Sl   18:32   0:00  \_ /usr/local/bin/python /usr/local/bin/gunicorn -k egg:meinheld#gunicorn_worker -c /app/gunicorn_conf.py main:app
root        16  0.6  1.8 367204 37416 ?        Sl   18:32   0:00  \_ /usr/local/bin/python /usr/local/bin/gunicorn -k egg:meinheld#gunicorn_worker -c /app/gunicorn_conf.py main:app
root        18  0.5  1.8 367200 37200 ?        Sl   18:32   0:00  \_ /usr/local/bin/python /usr/local/bin/gunicorn -k egg:meinheld#gunicorn_worker -c /app/gunicorn_conf.py main:app
root        20  0.5  1.8 367296 37204 ?        Sl   18:32   0:00  \_ /usr/local/bin/python /usr/local/bin/gunicorn -k egg:meinheld#gunicorn_worker -c /app/gunicorn_conf.py main:app
root        26  0.5  1.8 367296 37204 ?        Sl   18:32   0:00  \_ /usr/local/bin/python /usr/local/bin/gunicorn -k egg:meinheld#gunicorn_worker -c /app/gunicorn_conf.py main:app
root        30  0.5  1.8 367208 37324 ?        Sl   18:32   0:00  \_ /usr/local/bin/python /usr/local/bin/gunicorn -k egg:meinheld#gunicorn_worker -c /app/gunicorn_conf.py main:app

RUN INFO:

 ❯ docker-compose up                                                                                                                                             [15:32:36]
Recreating nerd_app_1 ... done
Attaching to nerd_app_1
app_1  | Checking for script in /app/prestart.sh
app_1  | There is no script /app/prestart.sh
app_1  | [2019-04-07 18:32:40 +0000] [7] [DEBUG] Current configuration:
app_1  |   config: /app/gunicorn_conf.py
app_1  |   bind: ['0.0.0.0:80']
app_1  |   backlog: 2048
app_1  |   workers: 8
app_1  |   worker_class: egg:meinheld#gunicorn_worker
app_1  |   threads: 1
app_1  |   worker_connections: 1000
app_1  |   max_requests: 0
app_1  |   max_requests_jitter: 0
app_1  |   timeout: 30
app_1  |   graceful_timeout: 30
app_1  |   keepalive: 120
app_1  |   limit_request_line: 4094
app_1  |   limit_request_fields: 100
app_1  |   limit_request_field_size: 8190
app_1  |   reload: True
app_1  |   reload_engine: auto
app_1  |   reload_extra_files: []
app_1  |   spew: False
app_1  |   check_config: False
app_1  |   preload_app: False
app_1  |   sendfile: None
app_1  |   reuse_port: False
app_1  |   chdir: /app
app_1  |   daemon: False
app_1  |   raw_env: []
app_1  |   pidfile: None
app_1  |   worker_tmp_dir: None
app_1  |   user: 0
app_1  |   group: 0
app_1  |   umask: 0
app_1  |   initgroups: False
app_1  |   tmp_upload_dir: None
app_1  |   secure_scheme_headers: {'X-FORWARDED-PROTOCOL': 'ssl', 'X-FORWARDED-PROTO': 'https', 'X-FORWARDED-SSL': 'on'}
app_1  |   forwarded_allow_ips: ['127.0.0.1']
app_1  |   accesslog: None
app_1  |   disable_redirect_access_to_syslog: False
app_1  |   access_log_format: %(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s "%(f)s" "%(a)s"
app_1  |   errorlog: -
app_1  |   loglevel: debug
app_1  |   capture_output: False
app_1  |   logger_class: gunicorn.glogging.Logger
app_1  |   logconfig: None
app_1  |   logconfig_dict: {}
app_1  |   syslog_addr: udp://localhost:514
app_1  |   syslog: False
app_1  |   syslog_prefix: None
app_1  |   syslog_facility: user
app_1  |   enable_stdio_inheritance: False
app_1  |   statsd_host: None
app_1  |   statsd_prefix:
app_1  |   proc_name: None
app_1  |   default_proc_name: main:app
app_1  |   pythonpath: None
app_1  |   paste: None
app_1  |   on_starting: <function OnStarting.on_starting at 0x7f01a395d268>
app_1  |   on_reload: <function OnReload.on_reload at 0x7f01a395d378>
app_1  |   when_ready: <function WhenReady.when_ready at 0x7f01a395d488>
app_1  |   pre_fork: <function Prefork.pre_fork at 0x7f01a395d598>
app_1  |   post_fork: <function Postfork.post_fork at 0x7f01a395d6a8>
app_1  |   post_worker_init: <function PostWorkerInit.post_worker_init at 0x7f01a395d7b8>
app_1  |   worker_int: <function worker_int at 0x7f01a3971c80>
app_1  |   worker_abort: <function WorkerAbort.worker_abort at 0x7f01a395d9d8>
app_1  |   pre_exec: <function PreExec.pre_exec at 0x7f01a395dae8>
app_1  |   pre_request: <function PreRequest.pre_request at 0x7f01a395dbf8>
app_1  |   post_request: <function PostRequest.post_request at 0x7f01a395dc80>
app_1  |   child_exit: <function ChildExit.child_exit at 0x7f01a395dd90>
app_1  |   worker_exit: <function WorkerExit.worker_exit at 0x7f01a395dea0>
app_1  |   nworkers_changed: <function NumWorkersChanged.nworkers_changed at 0x7f01a3971048>
app_1  |   on_exit: <function OnExit.on_exit at 0x7f01a3971158>
app_1  |   proxy_protocol: False
app_1  |   proxy_allow_ips: ['127.0.0.1']
app_1  |   keyfile: None
app_1  |   certfile: None
app_1  |   ssl_version: 2
app_1  |   cert_reqs: 0
app_1  |   ca_certs: None
app_1  |   suppress_ragged_eofs: True
app_1  |   do_handshake_on_connect: False
app_1  |   ciphers: TLSv1
app_1  |   raw_paste_global_conf: []
app_1  | [2019-04-07 18:32:40 +0000] [7] [INFO] Starting gunicorn 19.9.0
app_1  | [2019-04-07 18:32:40 +0000] [7] [DEBUG] Arbiter booted
app_1  | [2019-04-07 18:32:40 +0000] [7] [INFO] Listening at: http://0.0.0.0:80 (7)
app_1  | [2019-04-07 18:32:40 +0000] [7] [INFO] Using worker: egg:meinheld#gunicorn_worker
app_1  | [2019-04-07 18:32:40 +0000] [10] [INFO] Booting worker with pid: 10
app_1  | [2019-04-07 18:32:40 +0000] [12] [INFO] Booting worker with pid: 12
app_1  | [2019-04-07 18:32:40 +0000] [14] [INFO] Booting worker with pid: 14
app_1  | [2019-04-07 18:32:40 +0000] [16] [INFO] Booting worker with pid: 16
app_1  | [2019-04-07 18:32:40 +0000] [18] [INFO] Booting worker with pid: 18
app_1  | [2019-04-07 18:32:40 +0000] [20] [INFO] Booting worker with pid: 20
app_1  | [2019-04-07 18:32:40 +0000] [26] [INFO] Booting worker with pid: 26
app_1  | [2019-04-07 18:32:40 +0000] [30] [INFO] Booting worker with pid: 30

[qcho]

BTW I'd seen #1565 pull request, seems it would fix my issue.

But it went into a frozen state the last update is from Feb2018 and it suggest replacing the entire pr with hupper

[tilgovi]

@qcho are you importing any of your application code from gunicorn_conf.py? Normally, the application will not be imported in the master process. I see that you do not have preload setting turned on, so the master process should not have a copy of the module cached, unless it's imported before the worker fork.

[qcho]

@tilgovi

Don't think I am. This is my gunicorn_conf.py

import json
import multiprocessing
import os
import sys

workers_per_core_str = os.getenv("WORKERS_PER_CORE", "2")
web_concurrency_str = os.getenv("WEB_CONCURRENCY", None)
host = os.getenv("HOST", "0.0.0.0")
port = os.getenv("PORT", "80")
bind_env = os.getenv("BIND", None)
use_loglevel = os.getenv("LOG_LEVEL", "info")
if bind_env:
    use_bind = bind_env
else:
    use_bind = f"{host}:{port}"

cores = multiprocessing.cpu_count()
workers_per_core = float(workers_per_core_str)
default_web_concurrency = workers_per_core * cores
if web_concurrency_str:
    web_concurrency = int(web_concurrency_str)
    assert web_concurrency > 0
else:
    web_concurrency = int(default_web_concurrency)

# Gunicorn config variables
loglevel = use_loglevel
workers = web_concurrency
bind = use_bind
keepalive = 120
errorlog = "-"

# For debugging and testing
log_data = {
    "loglevel": loglevel,
    "workers": workers,
    "bind": bind,
    # Additional, non-gunicorn variables
    "workers_per_core": workers_per_core,
    "host": host,
    "port": port,
}
reload = os.getenv("RELOAD", False) == 'True'

print(json.dumps(log_data))

[philgyford]

I think I'm finding a similar problem, and apologies if it isn't...

I have a Flask app, running in a Docker container for development, and reload works fine without inotify installed. But once I've installed inotify (and set --reload-engine=inotify for good measure) then nothing's reloaded when I make changes to my code.

I'm doing this to start gunicorn:

gunicorn --reload --reload-engine=inotify -b 0.0.0.0:5006 wsgi:app --workers=4 --timeout=600

And wsgi.py is like this:

import os
from myproject import create_app

app = create_app(os.getenv("FLASK_CONFIG"))

The create_app() method returns an instance of Flask(__name__), plus sets up various configs, registers blueprints, etc.

I've checked the directories that gunicorn's reloader.py is watching with inotify, and none of them are the directory containing my code.

Is there something else I should be doing?

[tilgovi]

@philgyford are you on a Mac? It looks like there might be filesystem notification event limitations there: https://github.com/docker/for-mac/issues/2375

[philgyford]

Ah, I am! That sounds like the culprit, thanks...

[benoitc]

closing the issue since thre have no action since awhile. Also It worth to upgrade to latest master or 19.x since the reload feature have been improved

[kiruh]

What can I do if gunicorn doesn't auto reload?

If I modify a file called consumers.py I'll get a message

[2020-11-07 18:18:43 +0000] [624362] [INFO] Worker reloading: consumers.py.1ad5166f6fb31946184d3f9ddd0fea86.tmp modified

But the server does not actually respond to changes in files.

Those are my requirements

channels==3.0.1
gunicorn==20.0.4
httptools==0.1.1
uvloop==0.14.0
uvicorn==0.12.2
websockets==8.1

I run gunicorn with these flags

gunicorn pd.asgi:application --log-file /tmp/gunicorn-pd.log --reload -b localhost:9001 -k uvicorn.workers.UvicornWorker

Thanks

[sknaumov]

I've just tried to use --reload with gunicorn 20.0.4 and uvicorn.workers.UvicornWorker.
The server runs in docker container and application code is mapped there as a volume.

Reload functionality doesn't work, even in inotify mode. When I use manage.py runserver or uvicorn --reload as an entrypoint for container instead of gunicorn, code reload works.

In container logs I see the following when deliberately introduced an error into the code:

  File "/app/terminal/consumers.py", line 47, in receive
    self.ssh.sssendall(bytes_data.decode()) # timeout is 1 second
AttributeError: 'Channel' object has no attribute 'sssendall'

and then this exception when corrected it without restarting the container:

  File "/app/terminal/consumers.py", line 47, in receive
    self.ssh.sendall(bytes_data.decode()) # timeout is 1 second
AttributeError: 'Channel' object has no attribute 'sssendall'

As you can see, stack trace in the latter case shows code without an error, but exception itself still says that attribute name is incorrect.