Httpx: Reconsider UNSET?

[florimondmanca]

Prompted by #1372

Right now several of our public API functions or methods have a reference to an UnsetType instantiated as UNSET in their signature. For example:

https://github.com/encode/httpx/blob/589c6e0f2f4f3821095bfc247417aa98a5dad838/httpx/_client.py#L690-L705

Here's the full list of public API objects and methods that refer to UnsetType:

• Client.request
• Client.stream
• Client.send
• Client.get, Client.post, Client.put, Client.patch, Client.delete, Client.options, Client.head
• Equivalent methods on AsyncClient
• Timeout(...)

This style was introduced in preference to eg auth=None. The goal was to be able to distinguish between for example "auth was not passed" (indicating "use the default auth on the client"), and "auth was passed explicitly with None" (indicating that the user _really does not want auth to be used_).

But the problem is that now this UNSET instance and UnsetType are part of the public API: if people want to override methods on Client and keep the default behavior, then they need to import UNSET and default to it in the method.

import httpx
from httpx._config import UNSET  # Oops

class MyClient(httpx.Client):
    def post(self, *args, auth=UNSET, **kwargs):
        ...  # Perhaps pre-process `auth`

It's fair to ask whether those use cases are valid, and if people can't always find a way to _not_ have to refer to UNSET.

But there's also a case to be made for us dropping that "sentinel value" idea entirely. For example, we could:

• Accept a **kwargs and pull auth from there.
  
  
  
  • Pros: no need for a special sentinel value.
  
  
  • Cons: we lose type hinting on auth and timeout (at least until typed **kwargs are a thing, for example: python/mypy#4441) and autocompletion.
  
  
• Use a builtin instead, for example auth="__UNSET__".
  
  
  
  • Pros: builtin type (string) means it removes the "I need to import UNSET" problem.
  
  
  • Cons:
  
  
  • Still a sentinel value which users need to be aware of.
  
  
  • To type this properly we'd need Literal["__UNSET__"], and Literal is Py3.8+ (or we need to add typing_extensions as a dependency).
  
  

I'm not sure what option is better, or if there are other options to consider, but I'm flagging this for 1.0 because this touches the public API.

[lepture]

I’m wondering if it is ok to use typing.Undefined

[florimondmanca]

@lepture Where does typing.Undefined exist? https://github.com/python/mypy/issues/675 mentions it as an "old convention", and import typing; typing.Undefined is an AttributeError.

However, _this_ makes mypy happy enough…

import builtins

class Client:
    def request(
        self,
        # ...
        auth: typing.Union[AuthTypes, builtins.ellipsis] = ...,
        allow_redirects: bool = True,
        timeout: typing.Union[TimeoutTypes, builtins.ellipsis] = ...,
    ) -> Response:

In otherwords, ... (Ellipsis, aka builtins.ellipsis) could be treated as a shared UNSET sentinel value.

I've seen FastAPI do this in places. In general FastAPI uses it to mean "this API parameter is _required_", whereas here we're giving it a different meaning). But it still means means it's not totally unprecedented in large typing-heavy projects.

And for users who want to override things then it looks reasonable enough…

import httpx

class MyClient(httpx.Client):
    def post(self, *args, auth=..., **kwargs):
        ...  # Perhaps pre-process `auth`

[jcugat]

I've used ... (ellipsis) as a shortcut for undefined a few times at $WORK before, but to be honest it seemed a bit of "magic" (reviewers were confused initially, it's not an usual pattern) and it was never part of any public API.

What's wrong with having the current sentinel UNSET as part of the public API? It has no issues with mypy, and I see the only usages would be for libs that want to subclass httpx so it shouldn't be that widespread.

[florimondmanca]

Yeah, I've just closed #1385 because builtins.ellipsis is actually invalid at runtime (raises AttributeError), it's only available to type checkers, and so most likely for their private use only.

[cdeler]

@florimondmanca

We can specify

UnsetType = type(Ellipsis)

in _types.py

Then mypy would be able to accept Ellipsis as a sentinel value

[tomchristie]

Some thoughts here...

I think we could perfectly well drop it on Timeout(...) in favour of kwargs.
That seems to just make sense there to me somehow.

The remaining cases are auth and timeout arguments on client methods, where we have it because we necessarily need a value that means "Use the client defaults" which is distinct from None, meaning "no auth" or "no timeout". Perhaps we could expose our sentinel as public API for those cases, but renaming it slightly to "USE_CLIENT_DEFAULT", which is much more clear?