Sanic: Simple example doesn't work anymore with latest master with auto_reload

[garyo]

I'm using 5ff4819 on Mac, Python 3.6.4. I have a simple example structured like this:

.
`-- app
    |-- __init__.py (empty)
    |-- extensions.py (defines a single function, foo())
    `-- server.py

server.py looks like this:

from sanic import Sanic

# Extensions
from app.extensions import foo

def create_app(**kwargs):
    app = Sanic()
    return app

if __name__ == "__main__":
    sanic_app: Sanic = create_app()

    sanic_app.run(debug=True)

But it fails when run in debug mode; can't load module app.extensions after it's running:

python -m app.server   
[2018-06-18 10:13:02 -0400] [65764] [DEBUG] 
[logo]
[2018-06-18 10:14:57 -0400] [65851] [INFO] Goin' Fast @ http://0.0.0.0:8000
Traceback (most recent call last):
  File "/Users/garyo/dss/consulting/shorelight/Director/sanic-test/app/server.py", line 14, in <module>
    from app.extensions import foo
ModuleNotFoundError: No module named 'app'

This seems to be related to auto_reload. In fact if I run it with debug=False, auto_reload=True I get the same results.

[garyo]

The simple reason is when you run python -m <module>, python helpfully rewrites sys.argv to contain the full path of the module rather than -m <module>. So the reloader can't get the original args properly, and ends up trying to run python <modulename>.py which loads it as __main__ so package imports don't work. Have I mentioned how much I hate the python import system?

[garyo]

Note that even if I fix (or work-around) that by appending the proper dir to sys.path in server.py, the reloader fails in a different way when it tries to reload on some change:

[2018-06-18 11:12:09 -0400] [66721] [ERROR] Unable to start server
Traceback (most recent call last):
  File "uvloop/loop.pyx", line 1083, in uvloop.loop.Loop._create_server
  File "uvloop/handles/streamserver.pyx", line 51, in uvloop.loop.UVStreamServer.listen
  File "uvloop/handles/streamserver.pyx", line 89, in uvloop.loop.UVStreamServer._fatal_error
OSError: [Errno 48] Address already in use

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/garyo/.local/share/virtualenvs/shorelight-director/lib/python3.6/site-packages/sanic/server.py", line 614, in serve
    http_server = loop.run_until_complete(server_coroutine)
  File "uvloop/loop.pyx", line 1422, in uvloop.loop.Loop.run_until_complete
  File "uvloop/loop.pyx", line 1599, in create_server
  File "uvloop/loop.pyx", line 1087, in uvloop.loop.Loop._create_server
OSError: [Errno 48] error while attempting to bind on address ('0.0.0.0', 8000): address already in use
[2018-06-18 11:12:09 -0400] [66721] [INFO] Server Stopped

Perhaps the reloader isn't ready for use on Mac?

[senciucserban]

Same problem on Ubuntu 18.04 with python 3.7.0.

[garyo]

I'm currently working around this by starting the server from an external run-server.py above the app dir:

import sys
from app.server import main

sys.exit(main())

and that works (with my prev fix in #1249).

[MyNameIsCosmo]

I've had a similar issue with the auto_reloader not supporting relative imports, so using the following in my entry point failed:
from . import __description__ as description, __title__ as title, log

Setting auto_reload=False in the app.Run() fixed my issue.

You may be seeing a similar issue with auto_reload when running python3 -m <package>
However, after the package is built and installed, you will not have these issues.

This is a little unfortunate as auto_reload would ideally be used in development.

EDIT:
To clarify: debug=True automatically enables auto_reload=True

[geyang]

python import system is simply trash. I feel your pain.

[vltr]

There's been a discussion regarding the auto_reload feature in the community forums. It does not only seeks to be a solution for Windows developers but also to avoid these kind of errors, while taking some of the logic out of the main Sanic repository.

[cthwaite]

Just run into this problem myself.

The simple reason is when you run python -m <module>, python helpfully rewrites sys.argv to contain the full path of the module rather than -m <module>. So the reloader can't get the original args properly, and ends up trying to run python <modulename>.py which loads it as __main__ so package imports don't work. Have I mentioned how much I hate the python import system?

This is true, but surely the reloader _can_ get the original args? You'd just have to use _get_args_for_reloading to amend the contents of sys.argv using __spec__:

def _get_args_for_reloading():
    """Returns the executable."""
    rv = [sys.executable]
    # at this point sys.argv is e.g. ['/tmp/my_module/__main__.py']
    spec = getattr(sys.modules['__main__'], '__spec__', None)
    if spec and spec[:2] != ['-m', spec.name]:
        # now sys.argv is e.g. ['-m my_module.__main__']
        sys.argv = ['-m', spec.name] + sys.argv[1:]
    rv.extend(sys.argv)
    return rv

[ohahlev]

Why need to have this auto_reload when we can use entr?

My folder structure:

---app
-----| app.py
---run.sh

run.sh

#!/usr/bin/env bash
find app/ -name \*.py -o -name \*.html | entr -r python -m app.app

1. Escalate this feature to the external tool will save everyone's time
2. We can tell shell script to only reload the whole project if some file content in any folder change; and how do we do that in Sanic? I guess there might be some additional works need to be done if we want this auto_reload to be fully grown.
3. The above line of script is no where attached to Sanic; so it is very portable. We can just run it in any python project supposing the same folder structure is there.

[Tronic]

You just added a UNIX subsystem and entr (doesn't seem to be available on my package manager) to the requirements, to implement a reloader that is worse than Sanic's. Sanic detects changes and reloads even if modules outside current directory are updated. The dozens of other autoreloaders (such as gunicorn) weren't unified enough, so let's add one more to the pile.

There are clear advantages of having this internal to Sanic. First of all, no extra deps beyond what pip install gets for Sanic, so it is trivial to use it. Secondly, livereload.js (reloads the page in browser when files change on server) could easily be supported within a web framework (this is already used in aiohttp-devtools, which does autoreloading and also livereload).

Granted, there could be room for a shared python-asyncio autoreloader. I wonder how difficult it would be to add Sanic support into aiohttp-devtools or livereload Python packages.