Black: Package up `blib2to3`?

[asottile]

The improvements made in black's fork of lib2to3 seem like something that would be useful for the wider community. I'd like to be able to leverage this to write my own code rewriting tools. Currently lib2to3 isn't really an option due to its numerous bugs and shortcomings (as well as being sequestered to the stdlib where it can't get the love it deserves). Would you be interested in packaging this?

[ambv]

blib2to3 is a partial vendored version of lib2to3. It doesn't include the test suite nor the refactoring bits. It's not a viable full replacement.

It also has a small but crucial set of incompatibilities with lib2to3. The biggest ones are: it caches grammar pickles in Black's cache directory, it never puts comments on INDENT nodes and only puts the right comments on DEDENT nodes.

You can depend on Black and that way get blib2to3 if it fits your use case. Keep in mind that while unlikely, it's possible there will be future incompatibilities if I need them.

I'm not interested maintaining a public fork. For that, see @davidhalter's Parso.

Also, it's a bit unfair to say lib2to3 had numerous bugs. It has some and I fixed almost all of them upstream, too. Python 3.7 will be in a better position than 3.6 and older were.

[davidhalter]

@asottile I really believe that parso is what you're looking for. There are currently no open bugs there, so feel free to report those :)

[asottile]

I'll try it out! thanks!

[asottile]

yikes

[sten0]

blib2to3 is a partial vendored version of lib2to3. It doesn't include the test suite nor the refactoring bits. It's not a viable full replacement.

It also has a small but crucial set of incompatibilities with lib2to3. The biggest ones are: it caches grammar pickles in Black's cache directory, it never puts comments on INDENT nodes and only puts the right comments on DEDENT nodes. [1]

You can depend on Black and that way get blib2to3 if it fits your use case. Keep in mind that while unlikely, it's possible there will be future incompatibilities if I need them.

I'm not interested maintaining a public fork. For that, see @davidhalter's Parso.

Also, it's a bit unfair to say lib2to3 had numerous bugs. It has some and I fixed almost all of them upstream, too. Python 3.7 will be in a better position than 3.6 and older were.

Would it be possible for Black to switch from blib2to3 and to use fissix instead, if it accommodated [1] (see above)? Also, would it be possible for Black and Bowler to share grammar pickles via fissix? @jdufresne, what do you think? Can those requirements be met in your fork, or can you think of a solution?

Other than requiring a lot of work, I'm guessing Parso is out, for the reasons similar to the ones discussed here: https://github.com/facebookincubator/Bowler/issues/9 IMHO that's a shame, given that it's allegedly faster than alternatives https://medium.com/@boxed/a-quick-performance-comparison-of-python-parsers-eb86497ac733 I wonder if users who use both Bowler and Black could see collateral speedups if Black used fissix.

Regards,
Nicholas

[davidhalter]

Just as a head's up: I'm probably going to rewrite parso (or big parts of it) in Rust. But it will take a while. I will probably also implement some kind of partial file caching (which speeds parsing up even more). Parso has that as well, but it uses mutable state, which is problematic. I will probably implement something similar to red/green trees: https://stackoverflow.com/questions/40266011/why-does-roslyn-have-two-versions-of-syntax-per-language.

But at the same time I have yet to look at Guido's latest PEG parser efforts.