Yapf: Tests fail on python 3.7

[eli-schwartz]

Arch Linux is currently trying to update to python 3.7.0, and while rebuilding all packages in our repositories, yapf failed the testsuite with the following log:

FAIL: testAsyncAsNonKeyword (yapftests.reformatter_basic_test.BasicReformatterTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/build/yapf/src/yapf-0.22.0/yapftests/reformatter_basic_test.py", line 2489, in testAsyncAsNonKeyword
    self.assertCodeEqual(code, reformatter.Reformat(uwlines, verify=False))
  File "/build/yapf/src/yapf-0.22.0/yapftests/yapf_test_helper.py", line 57, in assertCodeEqual
    self.fail('\n'.join(msg))
AssertionError: Code format mismatch:
Expected:
 > from util import async
 > 
 > 
 > class A(object):
 >     def foo(self):
 >         async.run()
 > 
 >     def bar(self):
 >         pass
Actual:
 > from util import async
 > 
 > 
 > class A(object):
 >     def foo(self):
 >         async .run()
 > 
 >     def bar(self):
 >         pass
Diff:
--- actual
+++ expected
@@ -3,7 +3,7 @@

 class A(object):
     def foo(self):
-        async .run()
+        async.run()

     def bar(self):
         pass

======================================================================
FAIL: testListComprehensionPreferThreeLinesForLineWrap (yapftests.reformatter_basic_test.BasicReformatterTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/build/yapf/src/yapf-0.22.0/yapftests/reformatter_basic_test.py", line 526, in testListComprehensionPreferThreeLinesForLineWrap
    self.assertCodeEqual(expected_formatted_code, reformatter.Reformat(uwlines))
  File "/build/yapf/src/yapf-0.22.0/yapftests/yapf_test_helper.py", line 57, in assertCodeEqual
    self.fail('\n'.join(msg))
AssertionError: Code format mismatch:
Expected:
 > def given(y):
 >   long_variable_name = [
 >       long_var_name + 1
 >       for long_var_name, number_two in ()
 >       if long_var_name == 2 and number_two == 3
 >   ]
Actual:
 > def given(y):
 >   long_variable_name = [
 >       long_var_name + 1 for long_var_name, number_two in ()
 >       if long_var_name == 2 and number_two == 3
 >   ]
Diff:
--- actual
+++ expected
@@ -1,5 +1,6 @@
 def given(y):
   long_variable_name = [
-      long_var_name + 1 for long_var_name, number_two in ()
+      long_var_name + 1
+      for long_var_name, number_two in ()
       if long_var_name == 2 and number_two == 3
   ]

======================================================================
FAIL: testStronglyConnected (yapftests.split_penalty_test.SplitPenaltyTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/build/yapf/src/yapf-0.22.0/yapftests/split_penalty_test.py", line 205, in testStronglyConnected
    (']', None),
  File "/build/yapf/src/yapf-0.22.0/yapftests/split_penalty_test.py", line 72, in _CheckPenalties
    self.assertEqual(list_of_expected, FlattenRec(tree))
AssertionError: Lists differ: [('['[25 chars]or', 0), ('a', 3000), ('in', 3000), ('foo', 30[93 chars]one)] != [('['[25 chars]or', None), ('a', None), ('in', None), ('foo',[99 chars]one)]

First differing element 2:
('for', 0)
('for', None)

  [('[', None),
   ('a', None),
-  ('for', 0),
?          ^

+  ('for', None),
?          ^^^^

-  ('a', 3000),
-  ('in', 3000),
-  ('foo', 3000),
+  ('a', None),
+  ('in', None),
+  ('foo', None),
-  ('if', 0),
?         ^

+  ('if', None),
?         ^^^^

-  ('a', 3000),
+  ('a', None),
   ('.', 1000000),
   ('x', 4000),
   ('==', 3000),
   ('37', 3000),
   (']', None)]

----------------------------------------------------------------------
Ran 492 tests in 7.784s

FAILED (SKIP=4, failures=3)

[kamahen]

"async" is a keyword in Python3.7

$ python3.7
Python 3.7.0 (default, Jun 28 2018, 02:32:19)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.

from util import async
from util import async
File "", line 1
from util import async
^
SyntaxError: invalid syntax

$ python3.6
Python 3.6.6 (default, Jun 28 2018, 04:42:43)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.

from util import async
from util import async
Traceback (most recent call last):
File "", line 1, in
ModuleNotFoundError: No module named 'util'

[irl]

Also an issue in Debian, https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=903533

[doko42]

"async" is a keyword in Python3.7

yes.

[kamahen]

The Python 3.7 @dataclass notation also fails because this is new syntax:

@dataclass
class KeyValue:
    key: str
    value: int

(There are a few other small changes to the grammar; it took me half a day or so to modify some code that used lib2to3.pytree ... in particular all references to == lib2to3.pygram.python_symbols.comp_for needed to be changed to in (comp_for, _old_comp_for), which was slightly over-general but not important for the kind of analysis I was doing.)

[kamahen]

It appears that the variable annotation still doesn't work (see issue #500) ... perhaps add a test case for that?