Uvicorn: Deploying inside a process manager: must account for inherited socket family and type

[jxrossel]

Hi,

I get an error when trying to run uvicorn with Circus, but only if using a socket with file descriptor (which I'd like to do to increase the number of processes).

circus.ini:

[socket:web]
host = 0.0.0.0
port = 80
[watcher:uvicorn]
cmd = uvicorn
# args = --host 0.0.0.0 --port 80 satemo.asgi:application 
# --> works ok
args = --fd $(circus.sockets.web) satemo.asgi:application 
# --> Fails
use_sockets = True
numprocesses = 1

Error:

File "c:\nss\venv\channels2\lib\site-packages\uvicorn\main.py", line 247, in startup
    sock = socket.fromfd(config.fd, socket.AF_UNIX, socket.SOCK_STREAM)
UnboundLocalError: local variable 'socket' referenced before assignment

Environment:

• Windows 10
• Python 3.6.5
• uvicorn 0.4.1

[tomchristie]

Resolved and released as 0.4.2

[jxrossel]

That was fast...! but I still get the same error:

File "c:\nss\venv\channels2\lib\site-packages\uvicorn\main.py", line 248, in startup
    sock = socket.fromfd(config.fd, socket.AF_UNIX, socket.SOCK_STREAM)
UnboundLocalError: local variable 'socket' referenced before assignment

In main.py (line 242), the loop variable socket prevents the use of the socket module (name clash). If I change that variable name to something different, I get another error:

File "c:\nss\venv\channels2\lib\site-packages\uvicorn\main.py", line 248, in startup
    sock = socket.fromfd(config.fd, socket.AF_UNIX, socket.SOCK_STREAM)
AttributeError: module 'socket' has no attribute 'AF_UNIX'

Because AF_UNIX doesn't exist on Windows. It might be replaced by AF_INET for windows, but I don't know the implications of such a change.

[jxrossel]

I've digged a bit deeper in that issue, although I must admit my lack of expertise in the domain.

In main.py, the current inherited socket implementation is:

sock = socket.fromfd(config.fd, socket.AF_UNIX, socket.SOCK_STREAM)

This will work on Linux if the inherited socket is of family AF_UNIX. This cannot work in general, although it could also work on Linux with AF_INET sockets because the family and type arguments are probably only used to set the Python representation of the socket not the actual OS resources (this is the behaviour followed in socket.socket( fileno=... )).

The issue here is that when using socket.fromfd, one must know the inherited socket family and type. I see 2 possibles solutions here:

Solution 1

• Use command-line arguments to specify the inherited socket parameters. By default AF_INET and SOCK_STREAM would be used. The user is then responsible to give a consistent configuration for the socket and the server in the process manager configuration file

With a simple fix, I got it working on Windows (with 3 processes behind a circus-managed socket):

In main.py:

# Header
...
import socket
...
# Lines 297 and 299, replace `socket` by `this_socket`
for this_socket in config.sockets:
    server = await loop.create_server(  create_protocol, sock=this_socket, ssl=config.ssl ) 
...
# Line 305, replace `AF_UNIX` by `AF_INET`. That makes more sense if the process supervisor manages
# the socket bound to the web host and works both for Windows and Linux
sock = socket.fromfd(config.fd, socket.AF_INET, socket.SOCK_STREAM)

Solution 2

• Use socket.socket( fileno=config.fd) to rely on the inherited socket parameter auto-detection by the socket module
• Warning: only for Python >= 3.7 (so that https://bugs.python.org/issue28134 is solved, see https://docs.python.org/3/whatsnew/3.7.html)
• Warning: must take care not to close the inherited socket (even during garbage-collection), since the result of socket.socket() doesn't return a duplicate of the inherited socket

[tomchristie]

I'd certainly by up for solution 2, and either make it strictly 3.7+ only, or else issue a warning that it may not work reliably at earlier versions. (Probably the warning option, if it does work in some subset of cases, but it'd be good to clearly document which cases those are.)

[jxrossel]

With solution 2, if I understand https://bugs.python.org/issue28134 correctly, for Python < 3.7, the default family (AF_INET) and type (SOCK_STREAM) will be used for all the inherited sockets, although that only impacts the Python description of the socket (not the real OS resource). Therefore, whenever the inherited socket has these properties, the default values will be fine (I'd say in most cases, except AF_UNIX).

So:
Below 3.7: only AF_INET and SOCK_STREAM sockets can be inherited reliably
After 3.7: no limitation (but I haven't tested it)

There is still the bit about not closing the inherited socket that I am not sure how to handle. Do nothing? Or get sock = socket.socket( fileno=fd ), then duplicate (with .dup ? or socket.fromfd (with socket parameters from sock)? ) and do a sock.detach()?

[tomchristie]

There is still the bit about not closing the inherited socket that I am not sure how to handle. Do nothing? Or get sock = socket.socket( fileno=fd ), then duplicate (with .dup ? or socket.fromfd (with socket parameters from sock)? ) and do a sock.detach()?

Not sure. One other thing that'd really help progress this would be to see if there's a similar issue filed against Daphne or Hypercorn, or indeed against Sanic or aiohttp.

[jxrossel]

I looked at Daphne:

• No related issue found in the issue tracker
• Implementation: a painful trail to follow from Daphne to Twisted to Daphne and back to Twisted... The fd command-line parameter is converted to a Twisted endpoint string with no further argument. It is then interpreted in a Twisted endpoint plugin shipped with Daphne where the socket.AF_INET family is used by default (and cannot be changed) and where the socket.SOCK_STREAM type is imposed by the call to the endpoints method AdoptedStreamServerEndpoint. socket.fromfd(...) is then used.

The Daphne --endpoint argument can be used to specify a raw endpoint string to twisted. One could then give something like --endpoint fd:FILE_DESCRIPTOR:domain=SOCKET_FAMILY but I don't think it's implemented.

So basically, it's solution 1 with default values and no user choice.

I'll see if I have time to track the other servers implementation.

[tomchristie]

I'll see if I have time to track the other servers implementation.

Sure thing. Hypercorn is a close match so that's probably the one to look at next if you do find any time. Thanks for having a bit of a look into it.

[jxrossel]

Hypercorn is a very close match indeed:

...
elif bind.startswith("fd://"):
                sock = socket.fromfd(int(bind[5:]), socket.AF_UNIX, socket.SOCK_STREAM)
...

[jxrossel]

PS: I haven't found an equivalent of socket inheritance through file descriptor for sanic and aiohttp

[jxrossel]

After re-reading this thread https://bugs.python.org/issue28134, I am not sure how much "solution 2" would be compatible with Windows, which means it should be thoroughly tested on Windows first. Personally, I would go with a flavour of solution 1, which looks more reliable and less constraining in terms of Python version. The command-line argument could be like:

--fd FD:FAMILY[:TYPE[:PROTO]]

The parameter string would be parsed to find the arguments for socket.fromfd(...). TYPE and PROTO would have default values (SOCKET_STREAM and 0, respectively). For backward compatibility, FAMILY would default to AF_UNIX with a deprecation warning. In the long term (when socket.socket(fileno=...) is clearly reliable across OS), solution 2 could be implemented without backward-compatibility problems (just ignoring the extra arguments and returning a deprecation warning)

[tomchristie]

@jxrossel That sounds like a great option. Could you expand on what valid values would be for each of :FAMILY, :TYPE, :PROTO?

[jxrossel]

According to the socket module documentation:

The address family should be AF_INET (the default), AF_INET6, AF_UNIX, AF_CAN, AF_PACKET, or AF_RDS. The socket type should be SOCK_STREAM (the default), SOCK_DGRAM, SOCK_RAW or perhaps one of the other SOCK_ constants. The protocol number is usually zero and may be omitted or in the case where the address family is AF_CAN the protocol should be one of CAN_RAW, CAN_BCM or CAN_ISOTP.

I'd suggest to require full strings (e.g. 'AF_INET') and then get the constant values with getattr( socket, user_given_input ). To avoid following up with the versions of Python, I would not bother checking the arguments, but just return the exceptions given by socket, either at the getattr or socket.fromfd() levels.

[ldanielburr]

Hi, just bumping this issue to note that I get this error on macOS, running uvicorn under supervisord.

Interestingly, I tried adding "import socket" at the top of the file, as was done in https://github.com/encode/uvicorn/pull/296, but that did not solve the issue. What did work was adding "import socket" right after line 352:

elif config.fd is not None:
    # Use an existing socket, from a file descriptor.
    import socket
    sock = socket.fromfd(config.fd, socket.AF_UNIX, socket.SOCK_STREAM)

[ZuSe]

I can confirm that @ldanielburr solution works.
I ran into the same error with uvicorn 0.8.1 and python 3.7.3 (on Ubuntu).
The import statement fixed it.

@tomchristie Maybe we can add this for now to main.py?

[tomchristie]

Released as 0.8.2 - thanks folks!

[jxrossel]

It's nice that this bug has been corrected, but the socket inheritance still won't work on Windows (see my comments above): https://github.com/encode/uvicorn/issues/295#issuecomment-468291584, https://github.com/encode/uvicorn/issues/295#issuecomment-469182571 and https://github.com/encode/uvicorn/issues/295#issuecomment-469190962

[tomchristie]

I'm going to close this issue off, as it's too unclear to me what state it's in.
If there's still an existing issue at this point let's get a new issue filed, which should narrowed down to any remaining problems.