Yapf: Support for 3.6 f-string syntax

[dan-passaro]

Apologies for the brief report, but here's a simple bug reproduction session in the shell:

$ mktmpenv -p python3.6  # this command is from virtualenvwrapper
(tmp-38ce2dcf0268df9) $ python --version
Python 3.6.0b2
(tmp-38ce2dcf0268df9) $ cat test.py 
target = "world"
print(f"Hello, {target}!")
(tmp-38ce2dcf0268df9) $ python test.py 
Hello, world!
(tmp-38ce2dcf0268df9) $ pip install yapf==0.14.0
(tmp-38ce2dcf0268df9) $ yapf -i test.py
Traceback (most recent call last):
  File "/home/dpassaro/.virtualenvs/tmp-38ce2dcf0268df9/lib/python3.6/site-packages/yapf/yapflib/pytree_utils.py", line 102, in ParseCodeToTree
    tree = parser_driver.parse_string(code, debug=False)
  File "/home/dpassaro/.pyenv/versions/3.6.0b2/lib/python3.6/lib2to3/pgen2/driver.py", line 106, in parse_string
    return self.parse_tokens(tokens, debug)
  File "/home/dpassaro/.pyenv/versions/3.6.0b2/lib/python3.6/lib2to3/pgen2/driver.py", line 71, in parse_tokens
    if p.addtoken(type, value, (prefix, start)):
  File "/home/dpassaro/.pyenv/versions/3.6.0b2/lib/python3.6/lib2to3/pgen2/parse.py", line 159, in addtoken
    raise ParseError("bad input", type, value, context)
lib2to3.pgen2.parse.ParseError: bad input: type=3, value='"Hello, {target}!"', context=('', (2, 7))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/dpassaro/.virtualenvs/tmp-38ce2dcf0268df9/bin/yapf", line 11, in <module>
    sys.exit(run_main())
  File "/home/dpassaro/.virtualenvs/tmp-38ce2dcf0268df9/lib/python3.6/site-packages/yapf/__init__.py", line 296, in run_main
    sys.exit(main(sys.argv))
  File "/home/dpassaro/.virtualenvs/tmp-38ce2dcf0268df9/lib/python3.6/site-packages/yapf/__init__.py", line 188, in main
    parallel=args.parallel)
  File "/home/dpassaro/.virtualenvs/tmp-38ce2dcf0268df9/lib/python3.6/site-packages/yapf/__init__.py", line 236, in FormatFiles
    in_place, print_diff, verify)
  File "/home/dpassaro/.virtualenvs/tmp-38ce2dcf0268df9/lib/python3.6/site-packages/yapf/__init__.py", line 259, in _FormatFile
    logger=logging.warning)
  File "/home/dpassaro/.virtualenvs/tmp-38ce2dcf0268df9/lib/python3.6/site-packages/yapf/yapflib/yapf_api.py", line 90, in FormatFile
    verify=verify)
  File "/home/dpassaro/.virtualenvs/tmp-38ce2dcf0268df9/lib/python3.6/site-packages/yapf/yapflib/yapf_api.py", line 126, in FormatCode
    tree = pytree_utils.ParseCodeToTree(unformatted_source)
  File "/home/dpassaro/.virtualenvs/tmp-38ce2dcf0268df9/lib/python3.6/site-packages/yapf/yapflib/pytree_utils.py", line 108, in ParseCodeToTree
    tree = parser_driver.parse_string(code, debug=False)
  File "/home/dpassaro/.pyenv/versions/3.6.0b2/lib/python3.6/lib2to3/pgen2/driver.py", line 106, in parse_string
    return self.parse_tokens(tokens, debug)
  File "/home/dpassaro/.pyenv/versions/3.6.0b2/lib/python3.6/lib2to3/pgen2/driver.py", line 71, in parse_tokens
    if p.addtoken(type, value, (prefix, start)):
  File "/home/dpassaro/.pyenv/versions/3.6.0b2/lib/python3.6/lib2to3/pgen2/parse.py", line 159, in addtoken
    raise ParseError("bad input", type, value, context)
lib2to3.pgen2.parse.ParseError: bad input: type=3, value='"Hello, {target}!"', context=('', (2, 7))

[gwelymernans]

This might be because of the f in the string. I'm a bit surprised that lib2to3's parser can't handle it...

[Zopieux]

The (terrible) patch below solves this issue. It monkey-patches internal objects in lib2to3.pgen2.tokenize so that it accepts all kinds of f-strings.

Sample transformation

print(  F"omg"
Rf'''tripple strings
are cool
'''
  f"such yapf"
)

```python
print(F"omg" Rf'''tripple strings
are cool
''' f"such yapf")

### The patch (brace yourself)
```diff
diff --git a/yapf/yapflib/pytree_utils.py b/yapf/yapflib/pytree_utils.py
index 1bebcb8..2384a04 100644
--- a/yapf/yapflib/pytree_utils.py
+++ b/yapf/yapflib/pytree_utils.py
@@ -41,6 +41,58 @@ OPENING_BRACKETS = frozenset({'(', '[', '{'})
 CLOSING_BRACKETS = frozenset({')', ']', '}'})


+from lib2to3.pgen2 import tokenize
+import re
+tokenize.Triple = tokenize.group("[ubUB]?[rR]?'''",
+                                 '[ubUB]?[rR]?"""',
+                                 "[rR]?[fF]?'''",
+                                 '[fF]?[rR]?"""')
+tokenize.String = tokenize.group(r"[uU]?[rR]?'[^\n'\\]*(?:\\.[^\n'\\]*)*'",
+                                 r'[uU]?[rR]?"[^\n"\\]*(?:\\.[^\n"\\]*)*"',
+                                 r"[fF]?[rR]?'[^\n'\\]*(?:\\.[^\n'\\]*)*'",
+                                 r"[rR]?[fF]?'[^\n'\\]*(?:\\.[^\n'\\]*)*'",
+                                 r'[fF]?[rR]?"[^\n"\\]*(?:\\.[^\n"\\]*)*"',
+                                 r'[rR]?[fF]?"[^\n"\\]*(?:\\.[^\n"\\]*)*"')
+tokenize.ContStr = tokenize.group(r"[uUbB]?[rR]?'[^\n'\\]*(?:\\.[^\n'\\]*)*" +
+                                  tokenize.group("'", r'\\\r?\n'),
+                                  r'[uUbB]?[rR]?"[^\n"\\]*(?:\\.[^\n"\\]*)*' +
+                                  tokenize.group('"', r'\\\r?\n'),
+                                  r"[fF]?[rR]?'[^\n'\\]*(?:\\.[^\n'\\]*)*" +
+                                  tokenize.group("'", r'\\\r?\n'),
+                                  r"[rR]?[fF]?'[^\n'\\]*(?:\\.[^\n'\\]*)*" +
+                                  tokenize.group("'", r'\\\r?\n'),
+                                  r'[fF]?[rR]?"[^\n"\\]*(?:\\.[^\n"\\]*)*' +
+                                  tokenize.group('"', r'\\\r?\n'),
+                                  r'[rR]?[fF]?"[^\n"\\]*(?:\\.[^\n"\\]*)*' +
+                                  tokenize.group('"', r'\\\r?\n'))
+tokenize.PlainToken = tokenize.group(tokenize.Number, tokenize.Funny, tokenize.String, tokenize.Name)
+tokenize.Token = tokenize.Ignore + tokenize.PlainToken
+tokenize.PseudoExtras = tokenize.group(r'\\\r?\n', tokenize.Comment, tokenize.Triple)
+tokenize.PseudoToken = tokenize.Whitespace + tokenize.group(
+    tokenize.PseudoExtras, tokenize.Number, tokenize.Funny, tokenize.ContStr,
+    tokenize.Name)
+
+(tokenize.tokenprog, tokenize.pseudoprog, tokenize.single3prog,
+ tokenize.double3prog) = list(
+    map(re.compile, (tokenize.Token, tokenize.PseudoToken, tokenize.Single3,
+                     tokenize.Double3)))
+
+tokenize.endprogs.update({'f': None, 'F': None})
+
+for t in ('f', 'F', 'fr', 'rf', 'Fr', 'rF', 'fR', 'Rf', 'FR', 'RF'):
+    tt = f'{t}"""'
+    tokenize.endprogs[tt] = tokenize.double3prog
+    tokenize.triple_quoted[tt] = tt
+    tt = f"{t}'''"
+    tokenize.endprogs[tt] = tokenize.single3prog
+    tokenize.triple_quoted[tt] = tt
+    tt = f'{t}"'
+    tokenize.single_quoted[tt] = tt
+    tt = f"{t}'"
+    tokenize.single_quoted[tt] = tt
+
+
+
 class Annotation(object):
   """Annotation names associated with pytrees."""
   CHILD_INDENT = 'child_indent'

[gwelymernans]

Whoa! That's certainly an interesting patch. :-) I know nothing about lib2to3's code, so I'll have to take your word for it that it's good. If you want to create a pull request, I can see about merging it. You probably should make it a Python 3-specific patch, though.

[Zopieux]

It's a terrible solution, nothing sustainable long-term. I couldn't possibly submit a PR for that and be able to look in the mirror.

The actual issue is that yapf uses the lib2to3 parser for both Python 2 and 3, even though it was only made for Python 2. The syntaxes are quite different and the gap will continue to widen on each Python 3 release. yapf needs a proper Python 3 parser!

[gwelymernans]

I totally agree (with all of the sentiments :-) ). I was mostly hoping to get past this hurdle, which seems to be affecting a lot of people. Anyway, you're right that I'd love a proper Python 3 parser, but there doesn't appear to be one around that gives the same AST as lib2to3.

[spott]

So, what is the status of this? As 3.6 gets more adoption, it seems this is going to become a bigger issue.

Is there a way to have YAPF just skip fstrings when doing formating as a workaround for this?

[Zopieux]

Let me stress the fact yapf uses a Python 2 only lexer/parser. As keywords and syntactic sugars get added to Python 3, yapf will become more and more unable to format anything without layers afters layers of terrible monkey patches such as the one I wrote.

[spott]

Is there a plan to add a Python 3 lexer/parser? Or is that just way too large of a project and there isn't enough need by the maintainers?

[bolinfest]

Can we accept the really hacky fix then? Once you start using f-strings, there's no going back to %s.

[beaugunderson]

I'm also in favor of the hacky fix... I want to use yapf on our team but we're also using f-strings now and there's no going back. :)

If the hacky fix isn't merged to master could we get it in a python3 branch? That way we'd still have a central place to install from via pip...

[bolinfest]

@Zopieux It looks like earlier on this thread, @gwelymernans asked if you could turn your patch into a proper pull request. Would you be willing to do that? I'd love to see this move forward!

[Zopieux]

@bolinfest I thought I made my point clear enough by now: this is not something that can be turned into a "proper" PR. This is a hack. There are potentially many other subtle syntax changes introduced by Python 3.x releases waiting to explode in our faces. In my opinion, the current yapf for Python 3 works by coincidence and I shall not contribute to this monstrosity.

I would gladly accept contributing to some linter (yapf or other) that would use a proper Python 3 lexer/parser. But at the moment, yapf maintainers – as most Google open source projects – do not seem to be willing to progress on Python 3 stuff, let alone cutting-edge versions.

[bolinfest]

@Zopieux By "proper" PR, I don't mean "proper" in terms of code quality, but "proper" in the sense that it is a pull request that the maintainers could potentially formally accept and ingest rather than just some code pasted in a bug report.

I get that the underpinnings of yapf make all Python 3 work a hack, but it seems like myself and others would rather have a giant hack rather than nothing, which seems to be our alternative.

[gwelymernans]

I'm not unwilling to progress on Python 3 stuff. It's quite simply that I have no choice in the matter. lib2to3 for better or for worse, is the parser we used. There were reasons why we did that. Changing from it is difficult, to say the least. I'm very open to ideas on how to address this. But to be honest the best way would be to write a Python parser that retains all of the syntax clues and comments that the ast module loses.

[gwelymernans]

Notice that this is something they plan on fixing in lib2to3

https://bugs.python.org/issue23894

[leth]

Looks like the python issue has now been marked as 'resolved'. 🎉

[Zopieux]

Link to actual commit that is more or less equivalent to my patch: https://github.com/python/cpython/commit/e8412e684ed741df246e8430f4911b31b0d8be1f

[audiolion]

You should have had more faith in your solution @Zopieux ! Nice work coming up with that

[RenatoUtsch]

Now that this has been fixed upstream, what is the procedure to get this into yapf so that I can start using f-strings?

[greut]

@RenatoUtsch waiting for 3.6.2?

[RenatoUtsch]

Makes sense. Thanks.

[gwelymernans]

HUZZAH!!! Our long national nightmare will soon be over. :-D

[ipfans]

I already using 3.6.2rc1, it works fine with yapf.

[jlowin]

FYI: Python 3.6.2 was just released

[dmytrokyrychuk]

I had the same issue and I can confirm that upgrading Python from 3.6.1 to 3.6.2 resolved the issue for me.

[alok]

I think this issue should be closed.

[gyli]

f-string still crashes yapf on my end, with yapf 0.22.0, Python 3.7.0

from yapf.yapflib.yapf_api import FormatCode
FormatCode('f"test"')

returns the same traceback message as the original post:
raise ParseError("bad input", type, value, context)
lib2to3.pgen2.parse.ParseError: bad input: type=3, value='"test"', context=('', (1, 1))

[dmytrokyrychuk]

@ligyxy I can't reproduce on 3.7.0:

$ docker run -it python:3.7.0 /bin/bash
root@7bcee8f355e1:/# pip install yapf==0.22.0
Collecting yapf==0.22.0
  Downloading https://files.pythonhosted.org/packages/d5/ae/9d2e8f43f2ce467991c8310e361bbf4f1e1bf32afc6441b4e3416685b7ef/yapf-0.22.0-py2.py3-none-any.whl (166kB)
    100% |████████████████████████████████| 174kB 1.3MB/s 
Installing collected packages: yapf
Successfully installed yapf-0.22.0
root@7bcee8f355e1:/# python
Python 3.7.0 (default, Aug  4 2018, 02:33:39) 
[GCC 6.3.0 20170516] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from yapf.yapflib.yapf_api import FormatCode
>>> FormatCode('f"test"')
('f"test"\n', False)

[gyli]

Ah, it's my bad. I was using Python 3.6.0 wrongly.

[tvykruta]

fstrings are not working in python 3.74 yapf 0.28.0 from commandline:

(venv) Tomass-MacBook-Pro:data-pipeline tv$ yapf --version
yapf 0.28.0
(venv) Tomass-MacBook-Pro:data-pipeline tv$ python --version
Python 3.7.4
(venv) Tomass-MacBook-Pro:data-pipeline tv$ yapf fstrings_test.py 
Traceback (most recent call last):
  File "/usr/local/bin/yapf", line 10, in <module>
    sys.exit(run_main())
  File "/usr/local/lib/python2.7/site-packages/yapf/__init__.py", line 335, in run_main
    sys.exit(main(sys.argv))
  File "/usr/local/lib/python2.7/site-packages/yapf/__init__.py", line 220, in main
    verbose=args.verbose)
  File "/usr/local/lib/python2.7/site-packages/yapf/__init__.py", line 270, in FormatFiles
    in_place, print_diff, verify, verbose)
  File "/usr/local/lib/python2.7/site-packages/yapf/__init__.py", line 296, in _FormatFile
    logger=logging.warning)
  File "/usr/local/lib/python2.7/site-packages/yapf/yapflib/yapf_api.py", line 91, in FormatFile
    verify=verify)
  File "/usr/local/lib/python2.7/site-packages/yapf/yapflib/yapf_api.py", line 129, in FormatCode
    tree = pytree_utils.ParseCodeToTree(unformatted_source)
  File "/usr/local/lib/python2.7/site-packages/yapf/yapflib/pytree_utils.py", line 127, in ParseCodeToTree
    raise e
  File "fstrings_test.py", line 3
    print(f"He said his name is {name}.")
                                       ^
SyntaxError: invalid syntax

Test file:

(venv) Tomass-MacBook-Pro:data-pipeline tv$ more fstrings_test.py 
name      = "Fred"
print("He said his name is {}" % name)
print(f"He said his name is {name}.")

Running FormatCode directly works:

(venv) Tomass-MacBook-Pro:data-pipeline tv$ python fstring_test2.py 
('f"test"\n', False)
(venv) Tomass-MacBook-Pro:data-pipeline tv$ cat fstring_test2.py 
from yapf.yapflib.yapf_api import FormatCode
p = FormatCode('f"test"')
print(p)
(venv) Tomass-MacBook-Pro:data-pipeline tv$ 

[greut]

@tvykruta this issue has been closed a while ago. By the way, when calling yapf you're using Python 2.7 as shown by the stacktrace.