Mypy: Unpacking tuples of variable length

[mihneagiurgea]

Given the following code:

from typing import Any, Dict, Tuple, Union

VarTuple = Union[Tuple[int, int], Tuple[int, int, int]]

def make_tuple():
    # type: () -> VarTuple
    return None

x = make_tuple()
a = b = c = 0
if len(x) == 3:
    a, b, c = x
else:
    a, b = x
    c = 0

mypy complains about the following error:

sandbox.py:12: error: 'Union[Tuple[builtins.int, builtins.int], Tuple[builtins.int, builtins.int, builtins.int]]' object is not iterable
sandbox.py:14: error: 'Union[Tuple[builtins.int, builtins.int], Tuple[builtins.int, builtins.int, builtins.int]]' object is not iterable

[JukkaL]

The error message is incorrect. However, mypy may not be able to type check this properly since it doesn't recognize the len(x) == 3 check. So there seem to be two related issues:

• Union types aren't recognized as iterables when type checking assignment statements.
• len(x) == n checks can't be used to narrow down tuple types or unions. These could work similarly to isinstance checks.

The recommended way to work around the issue is to use casts:

if len(x) == 3:
    a, b, c = cast(Tuple[int, int, int], x)
else:
    a, b = cast(Tuple[int, int], x)
    c = 0

[JukkaL]

Also, mypy could recognize len(x) when x is a variable-length tuple. Example:

def f(x: Tuple[int, ...]) -> None:
    assert len(x) == 3 
    # Type of x should be Tuple[int, int, int] now
    ...

[neiljp]

I just ran into this issue and was going to post the following test case, but didn't realise that len(x) wasn't yet a way to distinguish among unions:

def f_4(a: Tuple[int, int, int, int]) -> None:
    pass

def f_2_4(a: Union[Tuple[int, int], Tuple[int, int, int, int]]) -> None:
    if len(a) == 2:
        aa = a + (0,0)
    else:
        aa = a
    f_4(aa)

[pgunn]

I'm getting a different error, but I'm guessing this is the same issue:

#!/usr/bin/env python

from typing import Tuple, Union

####################
# test

def patient_caller():
    a,b,c = annoying_returner(calculate_more=True)
    print(f"I got {a}, {b}, {c}")

def annoying_returner(calculate_more:bool=False) -> Union[Tuple[int,int], Tuple[int,int,int]]:
    if calculate_more:
        return 1,2,3
    else:
        return 4,5

def main():
    patient_caller()

##########

main()

scclin009:~/src/caiman_dev-clean$ mypy mypytest.py mypytest.py:9: error: Need more than 2 values to unpack (3 expected)

(I have a mypy.ini specifying check_untyped_defs)

[ilevkivskyi]

@pgunn In your particular case you can use @overload on Literal[True]/Literal[False] for the flag.

[pgunn]

@ilevkivskyi Will look into that; unfortunate that it's actually necessary, but it's great that it's possible. Thanks.

[pgunn]

@ilevkivskyi I'm trying to follow the docs here:
https://mypy.readthedocs.io/en/latest/literal_types.html
And I'm getting a

caiman/components_evaluation.py:539: error: Overloaded function signatures 1 and 2 overlap with incompatible return types

This is also getting very verbose. Not sure if it's that these are keyword rather than positional arguments that's giving me the problem.

# The below tells the type-checker that if return_all is a literal True, it returns 5 values
@overload
def estimate_components_quality(traces, Y, A, C, b, f, final_frate=30, Npeaks=10, r_values_min=.95,
                                fitness_min=-100, fitness_delta_min=-100, return_all:Literal[True]=True, N=5,
                                remove_baseline=True, dview=None, robust_std=False, Athresh=0.1, thresh_C=0.3, num_traces_per_group=20) -> Tuple[np.ndarray, np.ndarray, np.ndarray, np.ndarray, np.array]: ...
# and if it's a literal False, it returns two
@overload
def estimate_components_quality(traces, Y, A, C, b, f, final_frate=30, Npeaks=10, r_values_min=.95,
                                fitness_min=-100, fitness_delta_min=-100, return_all:Literal[False]=False, N=5,
                                remove_baseline=True, dview=None, robust_std=False, Athresh=0.1, thresh_C=0.3, num_traces_per_group=20) -> Tuple[np.ndarray, np.ndarray]: ...
# Fallback overload
def estimate_components_quality(traces, Y, A, C, b, f, final_frate=30, Npeaks=10, r_values_min=.95,
                                fitness_min=-100, fitness_delta_min=-100, return_all:bool=False, N=5,
                                remove_baseline=True, dview=None, robust_std=False, Athresh=0.1, thresh_C=0.3, num_traces_per_group=20) -> Union[Tuple[np.ndarray, np.ndarray, np.ndarray, np.ndarray, np.array], Tuple[np.ndarray, np.ndarray]]: ...

def estimate_components_quality(traces, Y, A, C, b, f, final_frate=30, Npeaks=10, r_values_min=.95,
                                fitness_min=-100, fitness_delta_min=-100, return_all:bool=False, N=5,
                                remove_baseline=True, dview=None, robust_std=False, Athresh=0.1, thresh_C=0.3, num_traces_per_group=20) -> Union[Tuple[np.ndarray, np.ndarray, np.ndarray, np.ndarray, np.array], Tuple[np.ndarray, np.ndarray]]:
     # Function body

This may be wandering away from the topic of the issue though; if it's wandered too far I'm fine with not getting an answer; don't want to clutter things for devs.

[ilevkivskyi]

Yes, this is because they are keyword args, see https://github.com/python/mypy/issues/6580

Anyway, in your case I would just return a less precise type like Tuple[np.ndarray, ...] (if this works).

[pgunn]

@ilevkivskyi It does. Thanks! I have some other code not well-covered by this tack, but perhaps by the time we're ready to switch to requiring mypy to pass in CI before merges, this kind of thing will have a more graceful solution.

[goodmami]

There seems to be an issue even with fixed-length tuples:

# file: ex.py
from typing import Tuple
points = (1, 2)  # type: Tuple[int, int]
x, y, z = *points, 0

This results in incorrect inference about the tuple size:

$ mypy ex.py 
ex.py:4: error: Need more than 2 values to unpack (3 expected)
Found 1 error in 1 file (checked 1 source file)

A workaround is to use cast(), as @JukkaL suggested earlier in this thread:

x, y, z = cast(Tuple[int, int, int], (*points, 3))

[ethanhs]

@goodmami could you open a new issue for that? It seems like a different (enough) problem.

[ilevkivskyi]

I think this is a missing feature rather than a bug (although this does cause a false positive).