Mypy: Internal Error on complex lambda expression

[walles]

~/s/px (python|●2✚1) $ mypy --show-traceback --check-untyped-defs --ignore-missing-imports *.py */*.py
px/px_terminal.py:23: error: Incompatible types in assignment (expression has type "int", variable has type "str")
px/px_terminal.py:24: error: Unsupported operand types for > ("int" and "str")
px/px_terminal.py:29: error: Incompatible types in assignment (expression has type "int", variable has type "str")
px/px_terminal.py:30: error: Unsupported operand types for > ("int" and "str")
px/px_ipc_map.py:45: error: INTERNAL ERROR -- please report a bug at https://github.com/python/mypy/issues version: 0.511
Traceback (most recent call last):
  File "/usr/local/bin/mypy", line 6, in <module>
    main(__file__)
  File "/usr/local/lib/python3.6/site-packages/mypy/main.py", line 46, in main
    res = type_check_only(sources, bin_dir, options)
  File "/usr/local/lib/python3.6/site-packages/mypy/main.py", line 93, in type_check_only
    options=options)
  File "/usr/local/lib/python3.6/site-packages/mypy/build.py", line 188, in build
    graph = dispatch(sources, manager)
  File "/usr/local/lib/python3.6/site-packages/mypy/build.py", line 1595, in dispatch
    process_graph(graph, manager)
  File "/usr/local/lib/python3.6/site-packages/mypy/build.py", line 1838, in process_graph
    process_stale_scc(graph, scc, manager)
  File "/usr/local/lib/python3.6/site-packages/mypy/build.py", line 1942, in process_stale_scc
    if graph[id].type_check_second_pass():
  File "/usr/local/lib/python3.6/site-packages/mypy/build.py", line 1516, in type_check_second_pass
    return self.type_checker.check_second_pass()
  File "/usr/local/lib/python3.6/site-packages/mypy/checker.py", line 219, in check_second_pass
    self.check_partial(node)
  File "/usr/local/lib/python3.6/site-packages/mypy/checker.py", line 226, in check_partial
    self.expr_checker.accept(node)
  File "/usr/local/lib/python3.6/site-packages/mypy/checkexpr.py", line 2058, in accept
    typ = node.accept(self)
  File "/usr/local/lib/python3.6/site-packages/mypy/nodes.py", line 1548, in accept
    return visitor.visit_lambda_expr(self)
  File "/usr/local/lib/python3.6/site-packages/mypy/checkexpr.py", line 1792, in visit_lambda_expr
    ret_type = self.accept(e.expr(), allow_none_return=True)
  File "/usr/local/lib/python3.6/site-packages/mypy/checkexpr.py", line 2058, in accept
    typ = node.accept(self)
  File "/usr/local/lib/python3.6/site-packages/mypy/nodes.py", line 1451, in accept
    return visitor.visit_op_expr(self)
  File "/usr/local/lib/python3.6/site-packages/mypy/checkexpr.py", line 1137, in visit_op_expr
    return self.check_boolean_op(e, e)
  File "/usr/local/lib/python3.6/site-packages/mypy/checkexpr.py", line 1370, in check_boolean_op
    right_type = self.analyze_cond_branch(right_map, e.right, left_type)
  File "/usr/local/lib/python3.6/site-packages/mypy/checkexpr.py", line 2025, in analyze_cond_branch
    with self.chk.binder.frame_context(can_skip=True, fall_through=0):
  File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/contextlib.py", line 82, in __enter__
    return next(self.gen)
  File "/usr/local/lib/python3.6/site-packages/mypy/binder.py", line 312, in frame_context
    assert len(self.frames) > 1
AssertionError:
px/px_ipc_map.py:45: note: use --pdb to drop into pdb
~/s/px (python|●2✚1) $

[gvanrossum]

Does it still occur with master? Can you show the source code?

[ethanhs]

Hm, I am unable to reproduce this. I cloned https://github.com/walles/px, and did the following:
With mypy=0.511:

$mypy --show-traceback --check-untyped-defs --ignore-missing-imports px
px\px_terminal.py:23: error: Incompatible types in assignment (expression has type "int", variable has type "str")
px\px_terminal.py:24: error: Unsupported operand types for > ("int" and "str")
px\px_terminal.py:29: error: Incompatible types in assignment (expression has type "int", variable has type "str")
px\px_terminal.py:30: error: Unsupported operand types for > ("int" and "str")
$cd px
$mypy --show-traceback --check-untyped-defs --ignore-missing-imports *.py */*.py
mypy: can't read file '*.py': Invalid argument
$mypy --show-traceback --check-untyped-defs --ignore-missing-imports px.py px_commandline.py px_cpuinfo.py px_file.py px_install.py px_ipc_map.py px_load.py px_load_bar.py px_loginhistory.py px_process.py px_processinfo.py px_terminal.py px_top.py __init__.py
Traceback (most recent call last):
  File "C:\Python36\Scripts\mypy", line 6, in <module>
    main(__file__)
  File "C:\Python36\lib\site-packages\mypy\main.py", line 46, in main
    res = type_check_only(sources, bin_dir, options)
  File "C:\Python36\lib\site-packages\mypy\main.py", line 93, in type_check_only
    options=options)
  File "C:\Python36\lib\site-packages\mypy\build.py", line 188, in build
    graph = dispatch(sources, manager)
  File "C:\Python36\lib\site-packages\mypy\build.py", line 1587, in dispatch
    graph = load_graph(sources, manager)
  File "C:\Python36\lib\site-packages\mypy\build.py", line 1696, in load_graph
    ancestor_for=st)
  File "C:\Python36\lib\site-packages\mypy\build.py", line 1166, in __init__
    assert id or path or source is not None, "Neither id, path nor source given"
AssertionError: Neither id, path nor source given

I also ran mypy --show-traceback --check-untyped-defs --ignore-missing-imports px with master and it was exactly the same as 0.511.

[gvanrossum]

Hm, you shoudn't be running mypy from inside a package. Maybe the OP's
exception is because he's passing the files in a different order?

[walles]

Tar file of what I have right now:
https://drive.google.com/file/d/0Byagq3owDalGVXh1UlBkOHVIcHM/view?usp=sharing

Then cd into there and do:

mypy --show-traceback --check-untyped-defs --ignore-missing-imports *.py */*.py

[walles]

I'm using the fish shell, here's the file order:

~/s/px (python|●2✚1) $ echo *.py */*.py
setup.py px/__init__.py px/px.py px/px_commandline.py px/px_cpuinfo.py px/px_file.py px/px_install.py px/px_ipc_map.py px/px_load.py px/px_load_bar.py px/px_loginhistory.py px/px_process.py px/px_processinfo.py px/px_terminal.py px/px_top.py tests/benchmark_ipcmap.py tests/px_commandline_test.py tests/px_cpuinfo_test.py tests/px_file_test.py tests/px_ipc_map_test.py tests/px_load_bar_test.py tests/px_load_test.py tests/px_loginhistory_test.py tests/px_process_test.py tests/px_processinfo_test.py tests/px_terminal_test.py tests/px_top_test.py tests/testutils.py
~/s/px (python|●2✚1) $

[ethanhs]

Hm, I still am unable to repro it. This is the same as the tip of the python branch, yes?

$python3 -m mypy --show-traceback --check-untyped-defs --ignore-missing-imports setup.py px/__init__.py px/px.py px/px_commandline.py px/px_cpuinfo.py px/px_file.py px/px_install.py px/px_ipc_map.py px/px_load.py px/px_load_bar.py px/px_loginhistory.py px/px_process.py px/px_processinfo.py px/px_terminal.py px/px_top.py tests/benchmark_ipcmap.py tests/px_commandline_test.py tests/px_cpuinfo_test.py tests/px_file_test.py tests/px_ipc_map_test.py tests/px_load_bar_test.py tests/px_load_test.py tests/px_loginhistory_test.py tests/
px_process_test.py tests/px_processinfo_test.py tests/px_terminal_test.py tests/px_top_test.py tests/testutils.py
px\px_terminal.py:23: error: Incompatible types in assignment (expression has type "int", variable has type "str")
px\px_terminal.py:24: error: Unsupported operand types for > ("int" and "str")
px\px_terminal.py:29: error: Incompatible types in assignment (expression has type "int", variable has type "str")
px\px_terminal.py:30: error: Unsupported operand types for > ("int" and "str")

EDIT: I also installed and ran from within fish, and _still_ couldn't repro it.

[walles]

Nope, I have some changes, this is not the same as the tip.

I also have a .mypy_cache in the tar file, which could be involved I guess.

[walles]

Here's the diff between HEAD of the python branch and what's in my working dir:

~/s/px (python|●2✚1) $ git diff HEAD|cat
diff --git a/px/px_ipc_map.py b/px/px_ipc_map.py
index 4e73f6c..34b6e77 100644
--- a/px/px_ipc_map.py
+++ b/px/px_ipc_map.py
@@ -40,8 +40,9 @@ class IpcMap(object):
         # process. Putting the files in a set gives us each file only once.
         files = set(files)

-        # Only deal with IPC related files
-        self.files = list(filter(lambda f: f.type in FILE_TYPES, files))
+        # Only deal with IPC related files and files for our own process
+        self.files = list(filter(
+            lambda f: (f.type in FILE_TYPES) or (f.pid == self.process.pid), files))

         self.process = process
         self.processes = processes
@@ -50,6 +51,10 @@ class IpcMap(object):
         self._map = {}  # type: MutableMapping[px_process.PxProcess, Set[px_file.PxFile]]
         self._create_mapping()

+        self.stdin = None  # type: str
+        self.stdout = None  # type: str
+        self.stderr = None  # type: str
+
     def _create_mapping(self):
         # type: () -> None
         self._create_indices()
diff --git a/tests/px_ipc_map_test.py b/tests/px_ipc_map_test.py
index 8a25a83..a8cd873 100644
--- a/tests/px_ipc_map_test.py
+++ b/tests/px_ipc_map_test.py
@@ -169,3 +169,15 @@ def test_get_ipc_map_2():

     peer0 = keys[0]
     assert len(ipc_map[peer0]) == 4
+
+
+def test_stdfd_ipc():
+    files = None
+    my_dir = os.path.dirname(__file__)
+    with open(os.path.join(my_dir, "lsof-test-output-linux-1.txt"), "r") as lsof_output:
+        files = px_file.lsof_to_files(lsof_output.read())
+
+    ipc_map = testutils.create_ipc_map(287, files)
+    assert ipc_map.stdin == 'FIXME: A reference to the other process'
+    assert ipc_map.stdout == '/dev/null'
+    assert ipc_map.stderr == 'FIXME: A reference to the other process'

[ethanhs]

Repro'd. It is (as I suspected) the lambda statement (making _just_ that change triggers the crash). I will look at the cause. Thanks for reporting!

[ethanhs]

from typing import List

FILE_TYPES  # type: List[str]

class PxProcess:
    pid: int

class File:
    type: str
    pid: int

class IpcMap(object):

    def __init__(self, files: List[File], process: PxProcess) -> None:
        self.files = list(filter(
            lambda f: (f.type in FILE_TYPES) or (f.pid == self.process.pid), files))
        self.process = process

This seems to be an issue where Mypy fails to check things correctly for self.process.pid. @walles you should probably put this statement _after_ assigning process to self.process, so there is a programmer error here, but also a bug in MyPy of course.
If you move self.process = process before the assignment to self.files things work.

This bug _is_ present in master.

[gvanrossum]

(Removing the 'bug' label because the 'crash' label implies it.)

[gvanrossum]

The crash reported by @ethanhs ("AssertionError: Neither id, path nor source given") is a dupe of #2974. But the crash reported by the OP (and later also reproduced by Ethan) is not, so don't close this.

[daboross]

I'm also getting this with what I presume is a complex lambda expression. I haven't been able to make a smaller reproduction, but here's the error from running on my codebase:

$ mypy src/ --verbose --show-traceback 

...

Traceback (most recent call last):
  File "/home/daboross/Projects/Python/Screeps/env/bin/mypy", line 11, in <module>
    sys.exit(console_entry())
  File "/home/daboross/Projects/Python/Screeps/env/lib/python3.5/site-packages/mypy/__main__.py", line 7, in console_entry
    main(None)
  File "/home/daboross/Projects/Python/Screeps/env/lib/python3.5/site-packages/mypy/main.py", line 50, in main
    res = type_check_only(sources, bin_dir, options)
  File "/home/daboross/Projects/Python/Screeps/env/lib/python3.5/site-packages/mypy/main.py", line 97, in type_check_only
    options=options)
  File "/home/daboross/Projects/Python/Screeps/env/lib/python3.5/site-packages/mypy/build.py", line 196, in build
    graph = dispatch(sources, manager)
  File "/home/daboross/Projects/Python/Screeps/env/lib/python3.5/site-packages/mypy/build.py", line 1801, in dispatch
    process_graph(graph, manager)
  File "/home/daboross/Projects/Python/Screeps/env/lib/python3.5/site-packages/mypy/build.py", line 2044, in process_graph
    process_stale_scc(graph, scc, manager)
  File "/home/daboross/Projects/Python/Screeps/env/lib/python3.5/site-packages/mypy/build.py", line 2152, in process_stale_scc
    if graph[id].type_check_second_pass():
  File "/home/daboross/Projects/Python/Screeps/env/lib/python3.5/site-packages/mypy/build.py", line 1722, in type_check_second_pass
    return self.type_checker.check_second_pass()
  File "/home/daboross/Projects/Python/Screeps/env/lib/python3.5/site-packages/mypy/checker.py", line 228, in check_second_pass
    self.check_partial(node)
  File "/home/daboross/Projects/Python/Screeps/env/lib/python3.5/site-packages/mypy/checker.py", line 235, in check_partial
    self.expr_checker.accept(node)
  File "/home/daboross/Projects/Python/Screeps/env/lib/python3.5/site-packages/mypy/checkexpr.py", line 2213, in accept
    typ = node.accept(self)
  File "/home/daboross/Projects/Python/Screeps/env/lib/python3.5/site-packages/mypy/nodes.py", line 1556, in accept
    return visitor.visit_lambda_expr(self)
  File "/home/daboross/Projects/Python/Screeps/env/lib/python3.5/site-packages/mypy/checkexpr.py", line 1941, in visit_lambda_expr
    ret_type = self.accept(e.expr(), allow_none_return=True)
  File "/home/daboross/Projects/Python/Screeps/env/lib/python3.5/site-packages/mypy/checkexpr.py", line 2213, in accept
    typ = node.accept(self)
  File "/home/daboross/Projects/Python/Screeps/env/lib/python3.5/site-packages/mypy/nodes.py", line 1459, in accept
    return visitor.visit_op_expr(self)
  File "/home/daboross/Projects/Python/Screeps/env/lib/python3.5/site-packages/mypy/checkexpr.py", line 1276, in visit_op_expr
    return self.check_boolean_op(e, e)
  File "/home/daboross/Projects/Python/Screeps/env/lib/python3.5/site-packages/mypy/checkexpr.py", line 1522, in check_boolean_op
    right_type = self.analyze_cond_branch(right_map, e.right, left_type)
  File "/home/daboross/Projects/Python/Screeps/env/lib/python3.5/site-packages/mypy/checkexpr.py", line 2179, in analyze_cond_branch
    with self.chk.binder.frame_context(can_skip=True, fall_through=0):
  File "/usr/lib64/python3.5/contextlib.py", line 59, in __enter__
    return next(self.gen)
  File "/home/daboross/Projects/Python/Screeps/env/lib/python3.5/site-packages/mypy/binder.py", line 326, in frame_context
    assert len(self.frames) > 1
AssertionError: 

Without --show-stacktrace, the error is:

src/rooms/minerals.py:687: error: INTERNAL ERROR -- please report a bug at https://github.com/python/mypy/issues version: 0.521
src/rooms/minerals.py:687: note: please use --show-traceback to print a traceback when reporting a bug

related lines being:

686                mineral_chosen = _.find(Object.keys(self.terminal.store),
687                                        lambda r: r != RESOURCE_ENERGY and self.terminal.store[r] >= 100)

[gvanrossum]

Could you reveal the signature of that find() method?

[gvanrossum]

(Update: I don't think we actually need more info, since we already have a small repro here.)

[ethanhs]

@daboross I am guessing that self.terminal.store or self.terminal is being defined after that find/lambda expression? Otherwise this may be more complicated than I thought.

[daboross]

@ethanhs It should all be defined before, but it might be a bit convoluted - I'm still fairly new to getting everything in typing right.

self.terminal is defined in __init__, copied from a terminal property of other object, which is typed as Optional[StructureTerminal]. StructureTerminal.store is a Dict[str, int]

other things:

# Object.keys:
_K = TypeVar('_K')
_V = TypeVar('_V')
    @classmethod
    def keys(cls, obj: Dict[_K, _V]) -> List[_K]:
        pass

# _.find
_L1 = TypeVar('_L1')
_LodashPredicate = Union[Dict[str, Any], Callable[[_L1], bool], None]
_Collection = Union[List[_L1], Dict[str, _L1]]
    @staticmethod
    def find(collection: _Collection, predicate: _LodashPredicate = None, thisArg: Any = None) -> _L1:
        pass

I haven't gotten a successful reproduction with any more minimal cases than my entire codebase though - not sure how helpful this is.

[ethanhs]

Ok. I think for now I can work on a patch based on my repro, and you can test it once I have made the patch.

[JukkaL]

@ethanhs Are you still working on this? It looks like a user got hit by this at Dropbox.

[ethanhs]

@JukkaL I was planning on working on it when I had time this weekend. If you want the fix to make it into 0.530, feel free to go ahead and fix it, I don't think I will have time today.

[gvanrossum]

This is still happening a lot for our internal codebase.

[ethanhs]

I'm currently working on my PEP implementation and spec at the moment, so perhaps someone else should take this one over?

[gvanrossum]

No worries.

[JukkaL]

I can take this.

[JukkaL]

Simplified example that still causes a crash:

from typing import Callable

class C:
    def f(self) -> None:
        g: Callable[[], int] = lambda: 1 or self.x
        self.x = int()