Black: Cython grammar support

[ambv]

We'd like to be able to format .pyx, .pxd, and .pxi files, too.

[ambv]

Hi, @pablogsal.

[grigoriosgiann]

Hello Łukasz!

I have been working on extending black for Cython.
Here is my approach:
Use a different grammar file, for the Cython grammar.
Start with the Python grammar and add Cython grammar rules to it one by one.
I have added most "simple" rules to the grammar so far, and I'm adding formatting rules for the blindingly obvious cases.

Here are some issues I faced in my endeavor:

• As a result of using a different grammar file, there is a symbol mismatch (the same symbol may be represented by a different integer in the Python grammar and Cython grammar). This requires functions and top level variables using type_repr and syms to be parameterized with regards to the grammar's symbols. This does not affect tokens. (this may require encapsulating some of the global state)
• Because of the way pgen works right now, string literals in the grammar cannot match NAME in the grammar. This is not that big of an issue in some cases and can be solved by disabling the keywords, however in other cases it's much harder as it conflicts with other rules. (like with the new expression: new = 5 vs new vector[int]()) I plan on deferring this issue for later.

Also I've expanded the tokenizer to accept integer literals ending in an arbitrary amount of (l, L, u, U), which affects affects python parsing as well. (single (l/L)-suffixed integer literals were already tokenized, even though they are invalid in Python3.)

Once I gather a sizable amount of rules, I'll let you know so you can inform me of your preferred formatting rules
I'd love to hear any suggestions/recommendations you have with regards to anything I've implemented so far, or any preference on how to proceed on the mentioned issues.

[pablogsal]

[Just as a clarification, @GrigoriosGiann is working with us (Python team in Bloomberg) in the Cython support as part of his internship]

[jakirkham]

@pablogsal @GrigoriosGiann, do you have something that you would like people to try/provide feedback on?

[Synss]

Are there any news on this? Even if you only have C syntax (no C++) or only .pxd or something, it would be really interesting to publish...

[pablogsal]

Are there any news on this?

Yeah, we will soon do a PR to iterate over :). I was caught in many other responsibilities and could not finish some required steps until now.

[RyanHope]

error: cannot format filename.pyx: Cannot parse: 11:13: from cpython cimport array
error: cannot format filename.pyx: Cannot parse: 145:6:     cdef int ml = len(mtype)

im getting this on latest dev version

[jlucier]

Is there any further update on this? I'm running into issues still with formatting Cython. Any tips @pablogsal?

[ArseniiDunaevQC]

Hi all!

I have a small update on the issue since I was working on it a bit in the last time. Basically I have rebased changes made by @pablogsal (https://github.com/pablogsal/black/tree/cython - hope you don't mind me taking it) on top of the current master branch and tried to add absent Cython features/rules, and to fix some existing bugs (you can check out my WIP-branch at https://github.com/ArseniiDunaevQC/black/tree/cython-dev).
First of all, it could be already (limitedly) used for some of the basic .pyx files. Here I would for sure be happy to hear any suggestions/recommendations as well as preferred formatting rules.

There are still some other issues that hinder further development:

• As @grigoriosgiann mentioned above, there are difficulties with matching grammar string literals with NAMEs. The hardest things for the moment are Python enumerations (Enum class) and using new as variable name.
• It would be great to have support for Cython compile-time constants and conditional compilation using DEF, IF etc. keywords (see docs) and include.

[pablogsal]

Hi all!

I have a small update on the issue since I was working on it a bit in the last time. Basically I have rebased changes made by @pablogsal (https://github.com/pablogsal/black/tree/cython - hope you don't mind me taking it)

The actual work in progress that we had does not live there but in https://github.com/bloomberg/black (branch cython_support). If you are going to base your work on top of this, please talk with @grigoriosgiann first.