Typing: How to annotate empty containers?

[matthiaskramm]

Suppose I have a function that returns an empty list:

def make_list():
  return []

How to best annotate the return type?

def make_list() -> List: ...  # presumably equivalent to List[Any], hence too general
def make_list() -> List[]: ...  # invalid syntax

[matthiaskramm]

I believe the standard type system approach is to do something like

def make_list() -> List[Nothing]: ...

(with "Nothing" defined as ⊥)

Would adding "Nothing" to typing.py be an option?

[gvanrossum]

What's the use case of having a function that returns an empty list? What is the caller going to do with that list next?

[matthiaskramm]

This problem arose from trying to annotate itertools.chain. An "empty-aware" annotation might look like this:

@overload
def chain() -> Iterator[Nothing]: ...
@overload
def chain(*iterables: Iterable[T]) -> Iterator[T]: ...

With that information, a type inferencer can figure out that in the code below, l is a list of integers, not a list of Any.

a = []
l = list(itertools.chain(*a)) + [3]

Another example would be __builtins__.zip().

(Both these examples use *args, so another solution might be to just assume that since there's no "T" in the function arguments, the "T" in the return shouldn't exist. That's a bit too magical for my taste, though.)

[gvanrossum]

Still looks very theoretical to me. Why would anyone call chain(*a) knowing that a is an empty list? It also smells like Haskell-style pattern matching to me, which is just not a direction I want Python to take.

[o11c]

A Nothing class would also be very useful for a generic-aware type([]) at runtime.

[JukkaL]

The type system defined by typing can't type check a ton of possible things Python code can do, and that's totally fine. Somehow I doubt that this is one of the most important things that isn't supported. The philosophy as I understand it is that for uncommon things it's better to fall back to Any instead of making the type system very complicated (that's why Any is defined in a very special way). So I'd argue that this wouldn't be worthwhile, unless somebody can show that this would actually bring considerable real-world benefits.

[ghost]

I think this issue is relevant. A common use case would be to express: "this is either an empty list or a list of xyz", e.g. "a list of tuples containing two strings". This occurs when dealing with tags: The html.parser.HTMLParser gives the attributes of a tag either as an empty list (whenever there are no attributes) or as a list of tuples containing two strings (namely name and value of the attribute). In a class derived from this Parser, there may be a method like:

def transform_attrs(
        tag:str,
        attrs: [] or [(str, str)],
        startend:bool,
        ) -> 'Tuple[str, [] or [(str, str)], bool]':
    new_attrs = []
    for attr in attrs:
        key, value = attr
        if not key.startswith('data-'):
            # process value here
            new_attrs.append((key, value))
    return tag, new_attrs, startend

So the question is how to express "[] or [(str, str)]" here. Something similar to the case 'Optional[list]' for 'Union[list, None]' would be nice. (Indeed, I had more use cases e.g. for "[] or [str]" than for 'Optional[List[str]]'.) In analogy to 'Optional', I would suggest 'Vacant'. E.g.

def transform_attrs(
        tag:str,
        attrs: 'Vacant[List[Tuple[str, str]]]',
        startend:bool,
        ) -> 'Tuple[str, Vacant[List[Tuple[str, str]]], bool]':
    ... # see above

[gvanrossum]

I don't get what you're trying to say here. The type List[Tuple[str, str]] (in PEP 484 notation) covers lists of any size including the empty list. So saying that it could be empty is redundant.

[ghost]

Thank you for your answer (and, besides, for the work on typing and python in general, which is useful for me every day). So the rules probably are:

List[str] - may be empty
Tuple[str, ...] - may be empty
Tuple[str, str] - must not be empty(?)
Dict[str, str] - may be empty(?)
AbstractSet[str] - may be empty
FrozenSet[str] - may be empty(?)

Sorry, I couldn't infer this for certain from reading https://www.python.org/dev/peps/pep-0484/ and https://docs.python.org/3.5/library/typing.html. I admit, the example x = [] # type: List[Employee] (from the PEP) indicates it for List. On the other hand, e.g. this function from https://www.python.org/dev/peps/pep-0484/#id34:

from typing import List, cast

def find_first_str(a: List[object]) -> str:
    index = next(i for i, x in enumerate(a) if isinstance(x, str))
    # We only get here if there's at least one string in a
    return cast(str, a[index])

If I call this function with an empty list, a StopIteration happens. Depending on the internal implementation of the function, it might also be an IndexError instead. I admit, in this example, the StopIteration also occurs if I call the function with a list that is non-empty but doesn't contain a str.

So I thought it practical to specify (explicitly or implicitly) whether a container can be empty or not in order to prevent IndexErrors or unpacking errors. If List[Tuple[str, str]] includes the empty list, as you made clear to me, then this covers the majority of cases and I can surely avoid writing code which blindly expects a non-empty list.

[gvanrossum]

The rule is just that _all_ containers with a uniform type can be empty.
The trick is to realize that _only_ Tuple[] without an ellipsis is not in
this category.

While there may be some code that specifically requires a non-empty
container of uniform type, it's hard to justify adding a special case to
the type syntax or type checker behavior for this in this early iteration
of the design.

A much more important case is distinguishing nullable types from
non-nullable types; this is a real cause of run-time errors in many
real-world cases (Dropbox's logs are full of them, for example). The type
syntax supports this (Optional[T] vs just T) but mypy doesn't yet.

[AIGeneratedUsername]

What's the use case of having a function that returns an empty list? What is the caller going to do with that list next?

Sample API response JSON:

{
    "foo": 1,
    "currency": "EUR",
    "eggs": "some_string",
    "all_bars": []    # this is a deprecated value and it is always an empty list
}

The JSON response has an always empty list.

auto-generated Pydantic model with the datamodel-code-generator:

class Response(BaseModel):
    foo: int
    currency: str
    eggs: str
    all_bars: List   # <-- here is the type issue 

So, what is the best practice to declare an always empty list in a model? What about Literal[[]] ?

UPD: Sorry, I did not checked: Literal[[]] is a wrong type hint. 'Literal' may be parameterized with literal ints, byte and unicode strings, bools, Enum values, None, other literal types, or type aliases to other literal types

UPD_2:
An ugly workaround:

EmptyList = Literal[""]
EmptyList.__args__ = []  # type: ignore[attr-defined]

class SomeClass:
     all_bars: EmptyList

---

UPD_3:

MY CURRENT SOLUTION (for Pydantic):

class EmptyList:
    @classmethod
    def __get_validators__(cls):
        yield cls.validate

    @classmethod
    def validate(cls, v):
        if not isinstance(v, list):
            raise TypeError('list required')
        if len(v) > 0:
            raise ValueError('list must be empty')
        return []


class Model(BaseModel):
    foo: int
    spams: EmptyList

[JelleZijlstra]

@AIGeneratedUsername List[object] seems fine in this case.