Uvicorn: Use optional package installs.

[tomchristie]

Instead of the platform detection I’d like uvicorn to use optional installs.

• pip install uvicorn - Just the package itself.
• pip install uvicorn[standard] - uvloop/httptools/websockets
• pip install uvicorn[pure] - asyncio/h11/wsproto
• pip install uvicorn[full] - Everything

[almarklein]

I like this, since this would solve an issue I have with a Docker container on which the installation of uvloop fails (not enough RAM).

However, does this mean that just doing pip install uvicorn will not give me a working uvicorn?

[almarklein]

What about:

• pip install uvicorn - deps click/h11/wsproto
• pip install uvicorn[fast] - deps h11/wsproto/uvloop/httptools/websocket

In this case a plain install should give a fully working server, all in pure Python. The user can then separately install uvloop/httptools/websockets, or use the [fast] option.

[tomchristie]

I'd be happy with that yup. Tho I'd probably prefer uvicorn and uvicorn[full].

[tomchristie]

Implementation in #224 resolves this as requested in the opening of this...

uvicorn/uvicorn[pure]/uvicorn[standard]/uvicorn[full]

We could easily make sure that Uvicorn raises loud errors if the cli is used and click isn't installed. Or we could go with @almarklein's suggestion and make the base case be the pure install.

[almarklein]

cc @carlwgeorge

[carlwgeorge]

Should I go ahead and rework the PR to match the layout in https://github.com/encode/uvicorn/issues/219#issuecomment-430178997? Also, should uvicorn[fast] also install click?

[tomchristie]

@carltongibson @almarklein Depends which of the two we think makes more sense.

(And yes, we'd include click whatever.)

The alternative tack would be to keep everything as per #219, except using this:

    install_requires=[
        'click',
        'h11',
        'wsproto', 
   ],
    extras_require={
        'full': [
            'uvloop',
            'httptools',
            'websockets>=6.0'
        ]
    },

[carltongibson]

(Did you mean me?)

[tomchristie]

@carltongibson Nope. I meant @carlwgeorge

(hits autocomplete a few more times)

[carlwgeorge]

I agree that pip install uvicorn should give you a functional base install. I modified the PR to always install click, h11, and wsproto, with a full option to also install uvloop, httptools, and websockets.

[tomchristie]

A lightweight option here would be...

pip install uvicorn  # Just click, h11.
pip install uvicorn[full]  # +websockets +watchdog +uvloop/httptools (conditional on env markers)

[tomchristie]

That’s probably a good balance of:

1. Keeping it simple.
2. Allowing for minimal installs if wanted.
3. Ensuring that the base install is operational.

Right?

[apollo13]

@tomchristie Jupp, that would really be helpful. I currently have to use h11 because httptools ships no wheels and I'd need manylinux wheels to be able to use it. What's needed to move this forward?

[apollo13]

This just made me realize that https://www.uvicorn.org/settings/#implementation seems to be wrong nowadays. httptools requires compilation on every system since it just ships a .tar.gz :)

[tomchristie]

@apollo13 - I'm not sure what you mean?

$ python -m venv venv
$ venv/bin/pip install httptools
Collecting httptools
Installing collected packages: httptools
Successfully installed httptools-0.0.13

[apollo13]

@tomchristie Looking at PyPI it is just a tar.gz https://pypi.org/project/httptools/#files and the setup.py says: https://github.com/MagicStack/httptools/blob/master/setup.py#L27-L32 so I'd argue it needs a compiler. Your system seems to have one so it just work (tm)

[apollo13]

In a docker container:

root@08fb3e67185f:/# rm /usr/bin/gcc*
root@08fb3e67185f:/# pip install httptools
Collecting httptools
  Downloading https://files.pythonhosted.org/packages/1b/03/215969db11abe8741e9c266a4cbe803a372bd86dd35fa0084c4df6d4bd00/httptools-0.0.13.tar.gz (104kB)
     |████████████████████████████████| 112kB 1.2MB/s 
Building wheels for collected packages: httptools
  Building wheel for httptools (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /usr/local/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-1flxic8y/httptools/setup.py'"'"'; __file__='"'"'/tmp/pip-install-1flxic8y/httptools/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-_7dxx7x1 --python-tag cp37
       cwd: /tmp/pip-install-1flxic8y/httptools/
  Complete output (28 lines):
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-3.7
  creating build/lib.linux-x86_64-3.7/httptools
  copying httptools/__init__.py -> build/lib.linux-x86_64-3.7/httptools
  creating build/lib.linux-x86_64-3.7/httptools/parser
  copying httptools/parser/__init__.py -> build/lib.linux-x86_64-3.7/httptools/parser
  copying httptools/parser/errors.py -> build/lib.linux-x86_64-3.7/httptools/parser
  running egg_info
  writing httptools.egg-info/PKG-INFO
  writing dependency_links to httptools.egg-info/dependency_links.txt
  writing top-level names to httptools.egg-info/top_level.txt
  reading manifest file 'httptools.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  writing manifest file 'httptools.egg-info/SOURCES.txt'
  copying httptools/parser/parser.c -> build/lib.linux-x86_64-3.7/httptools/parser
  running build_ext
  building 'httptools.parser.parser' extension
  creating build/temp.linux-x86_64-3.7
  creating build/temp.linux-x86_64-3.7/httptools
  creating build/temp.linux-x86_64-3.7/httptools/parser
  creating build/temp.linux-x86_64-3.7/vendor
  creating build/temp.linux-x86_64-3.7/vendor/http-parser
  gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/usr/local/include/python3.7m -c httptools/parser/parser.c -o build/temp.linux-x86_64-3.7/httptools/parser/parser.o -O2
  unable to execute 'gcc': No such file or directory
  error: command 'gcc' failed with exit status 1
  ----------------------------------------
  ERROR: Failed building wheel for httptools
  Running setup.py clean for httptools
Failed to build httptools
Installing collected packages: httptools
    Running setup.py install for httptools ... error
    ERROR: Command errored out with exit status 1:
     command: /usr/local/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-1flxic8y/httptools/setup.py'"'"'; __file__='"'"'/tmp/pip-install-1flxic8y/httptools/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-sgcnentz/install-record.txt --single-version-externally-managed --compile
         cwd: /tmp/pip-install-1flxic8y/httptools/
    Complete output (28 lines):
    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-3.7
    creating build/lib.linux-x86_64-3.7/httptools
    copying httptools/__init__.py -> build/lib.linux-x86_64-3.7/httptools
    creating build/lib.linux-x86_64-3.7/httptools/parser
    copying httptools/parser/__init__.py -> build/lib.linux-x86_64-3.7/httptools/parser
    copying httptools/parser/errors.py -> build/lib.linux-x86_64-3.7/httptools/parser
    running egg_info
    writing httptools.egg-info/PKG-INFO
    writing dependency_links to httptools.egg-info/dependency_links.txt
    writing top-level names to httptools.egg-info/top_level.txt
    reading manifest file 'httptools.egg-info/SOURCES.txt'
    reading manifest template 'MANIFEST.in'
    writing manifest file 'httptools.egg-info/SOURCES.txt'
    copying httptools/parser/parser.c -> build/lib.linux-x86_64-3.7/httptools/parser
    running build_ext
    building 'httptools.parser.parser' extension
    creating build/temp.linux-x86_64-3.7
    creating build/temp.linux-x86_64-3.7/httptools
    creating build/temp.linux-x86_64-3.7/httptools/parser
    creating build/temp.linux-x86_64-3.7/vendor
    creating build/temp.linux-x86_64-3.7/vendor/http-parser
    gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/usr/local/include/python3.7m -c httptools/parser/parser.c -o build/temp.linux-x86_64-3.7/httptools/parser/parser.o -O2
    unable to execute 'gcc': No such file or directory
    error: command 'gcc' failed with exit status 1
    ----------------------------------------
ERROR: Command errored out with exit status 1: /usr/local/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-1flxic8y/httptools/setup.py'"'"'; __file__='"'"'/tmp/pip-install-1flxic8y/httptools/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-sgcnentz/install-record.txt --single-version-externally-managed --compile Check the logs for full command output.

[tomchristie]

So is pip install uvicorn failing for you right now then?

[almarklein]

My initial interest for this issue was also because in a docker container it would compile the dependencies from source. The problem for me (at the time) was that the build of uvloop requires more RAM than I had available on my cheap VM.

All that aside, for services that have low traffic, I prefer the pure Python libs, because it makes the docker container much lighter, so I can host more services on a VM without needing to upgrade my hardware.

BTW: #298 is ready to go from my end :)

[tomchristie]

BTW: #298 is ready to go from my end :)

@almarklein - Great yup! It's number one on my list. Just waiting for a proper moment so I can review if there's anything else we'd want in a "Uvicorn 0.11" release.

[apollo13]

So is pip install uvicorn failing for you right now then?

Yes, and it would fail for you to if you wouldn't have a compiler installed (pip nicely hides that fact).

[almarklein]

if you wouldn't have a compiler installed

Not necessarily, because binary wheels, right? I'd think the binary wheels are compatible with most Linux Docker images, but somehow Pip does not think so. (Not my area of expertise tho, maybe someone else knows better.)

[apollo13]

@almarklein httptools does not ship wheels though: https://pypi.org/project/httptools/#files

I'd think the binary wheels are compatible with most Linux Docker images, but somehow Pip does not think so.

depends, most of them are just not compatible with alpine linux afaiks.

[almarklein]

Oh: "Alpine uses a different C library, musl, instead of the more common glibc." (source)

[tomchristie]

I'm slightly surprised we're not seeing more folks chiming in with "installing uvicorn fails on my platform".

Do either of "websockets" (Includes a small C extention but I think has an optional pure python fallback) or "uvloop" also fail to install for you.

I'm wondering if there's an environment marker or some other workaround we could use to make sure that they only installed optionally.

(Tho we're probably okay once we move to pip install uvicorn and pip install uvicorn[standard], and can just clearly document any failure cases)

[apollo13]

I'm slightly surprised we're not seeing more folks chiming in with "installing uvicorn fails on my platform".

Because most people can compile stuff; I also can but I try to avoid it because it should run everywhere.

Do either of "websockets" (Includes a small C extention but I think has an optional pure python fallback) or "uvloop" also fail to install for you.

If you look at websockets (https://pypi.org/project/websockets/#files) they include binary wheels, no compilation needed during install, same for https://pypi.org/project/uvloop/#files

I'm wondering if there's an environment marker or some other workaround we could use to make sure that they only installed optionally.

Sadly no, there is nothing that says "I can compile something"

[apollo13]

Just to clarify: Binary packages are fine as long as PyPI has .whl files (at least if they are maintained properly). If there are no .whl files but only .tar.gz on PyPI it means that you need a compiler, which generally means the wheel file pip locally builds and caches will work on your platform and your platform only.

[apollo13]

Oh: "Alpine uses a different C library, musl, instead of the more common glibc." (source)

Yes, this is one of the reasons why I generally try to use debian-based python docker containers because installing in them is way faster (as you can use the prebuilt wheels). As a result you can also use python:slimin the containers which don't ship GCC. The default python containers ship gcc so you can compile missing stuff.

[apollo13]

@tomchristie is there anything I can do to help move this forward?