Black: Black autodetection of 3.6 makes broken PyPy3 code

[dstufft]

Operating system: macOs Mojave
Python version: 3.7.0
Black version: 18.9.0
Does also happen on master: Didn't check, but a scan of the code appears so.

It looks like Black is attempting to auto detect the version of Python that a specific file supports by looking for features that only exist in a certain version of Python. The doc string suggests that it is specifically looking for f strings, numeric underscores, and trailing commas as *args and **kwargs.

This detection currently generates code that is broken for PyPy3 (which implements 3.5) because PyPy3 has backported f-string support (but not numeric underscores or trailing commas).

I'm not sure what the right answer here is. Certainly I understand the rationale behind auto detecting Python 3.6+ code, however in doing that there are now edge cases where broken code is being generated. Perhaps changing --py36 to be a boolean instead of a flag, and add a --no-py36 flag that projects that support PyPy3 as well would work?

[rooterkyberian]

This seems somewhat relevant: https://github.com/ambv/black/issues/419#issuecomment-422090963

AFAIK introduction of option flags into black, especially because of some edge cases, seem to be in direct conflict to what this project is about - which is "uncompromising formatting". You would use this to make your code run on PyPy3, other user would use it to disable some things (s)he doesn't like.

Alternative solutions:

• do a feature detection using target interpreter instead of custom options
  
  
  
  • I believe this would reduce complexity of feature detection
  
  
  • increase complexity of debugging user reported bugs
  
  
• very issue specific solution: wait for PyPy3 to implement trailing comma
  
  
  
  • it seems as one of the easiest things to support and should require much less effort than f-string implementation
  
  
  • in the mean time users can just forgo using f-strings and other Python 3.6 (or higher) features if they interpreter does not implement full set
  
  

@dstufft Regardless what happens in black, as a PyPy user it would be good to report it to them as well and that you have a need for this and/or that mixed Python version compatiblity can lead to auxiliary support tools incompatibilities. https://bitbucket.org/pypy/pypy/issues/2827/trailing-commas-are-occassionaly seems like a good place to start.

[dstufft]

AFAIK introduction of option flags into black, especially because of some edge cases, seem to be in direct conflict to what this project is about - which is "uncompromising formatting". You would use this to make your code run on PyPy3, other user would use it to disable some things (s)he doesn't like.

So there's no such thing as "no configuration" as long as there is something which influences the behavior of the output. There's just "explicit configuration" and "implicit configuration".

For instance, I could make this work by just avoiding the f-string language feature, which would mean the auto detection wouldn't detect a Python 3.6 file, and would then format things correctly. That can be considered a form of "implicit" configuration, since I'm essentially telling black to format the file in a hypothetical --no-py36 mode, I just have to do it in a very round about, and error prone way (since whether it work "work" or not would depend on the file in question).

do a feature detection using target interpreter instead of custom options

I don't think this is actually workable? Black only runs on Python 3.6+, and if you're talking about some sort of python flag that points it to the target Python, then PyPy3 would still trigger a Python 3.6 supported file (since it supports fstrings)... unless you'd do something like test all of the features instead of any of the features. That doesn't seem like a better option than just allowing people to disable auto detection (or alternatively, instead of a boolean --py36 flag, add a --min-python flag to control the minimum Python version this code supports.

very issue specific solution: wait for PyPy3 to implement trailing comma

It's not just trailing comma, it's also numeric literals (although that itself has an optional flag to disable it, so that can be worked around by just disabling it. They're unlikely going to also backport those features to PyPy 3.5, and instead are going to wait until the PyPy3 that supports 3.6 is ready and released.

[JelleZijlstra]

I think it would make sense to have some sort of flag like --target-pythons=cpy36,pypy35 that would tell Black to use only features available in these Python versions.

[flying-sheep]

Why is the default for py36 not None? Then we could do:

[tool.black]
py36 = false  # deactivate 3.6 syntax

and

$ black . --no-py36

This issue btw doesn’t only apply to pypy, but also to CPython 3.5 with future-fstrings.

[zsol]

I'm going to close this because #618 has landed and is a reasonable workaround IMO.