Jedi: Releasing the parser as a separate project

[davidhalter]

Based on some discussions in #630 and having my own ideas as well, I've been planning to release Jedi's parser to the public for a while now (as a separate pypi package). There is now a prototype of how that could look like. If you're interested, please test it and tell me what you think.

I want to know how the API could be different (better!), what you're missing and what you would remove.

The current prototype is in the dev branch under jedi/parser/. You probably only need the Python part, because the parser generator behind the whole thing will be able to understand different types of grammars.

Most people will want to use the from jedi.parser.python import parse function. It's pretty powerful.

Unfortunately there's no good documentation, yet. But if you're willing to PR a few changes, I'm thankful for all the help.

[davidhalter]

@alexbw Please give me a list of things that would need to be stabilized in the API. Some things are probably pretty easy to fix (especially in the parser tree), but I'm not sure which things you currently need. A release of this parser is probably at least months away. There's a high chance that many things will stay the same (apart from a few function renames).

[alexbw]

Maybe best to take discussion offline to reduce noise. Sending an email to the email on your github account.

[Nurdok]

@alexbw actually it would be helpful to have the discussion here so it will be public.

---

From playing around with the internal Jedi parser in PyCQA/pydocstyle#240 I found two issues that were lacking:

1. I want to get the _raw_ docstring of functions and classes, before it is evaled.
2. The parser currently ignores comments, but they can be important (specifically for noqa statements, but other use cases can apply, such as type hinting).

[davidhalter]

For (2) I can only say that this is semi true. The comments are in the prefix attribute in each leaf. I know this is probably not optimal for you (and we can figure out a better API to get comments). However they are there.

It's pretty much impossible to use a parser generator and include comments as in the parser tree. I just don't see how you would do that. With a custom parser and a lot of magic it's obviously possible, but it wouldn't be as generic.

[davidhalter]

And to give you actual answers:

1. I totally agree. Not sure yet how such an API would look like. Do you want the full literal (including r""") or the content?
2. Building on my answer above, let's add a better API to read comments. Do you have any suggestions. Where would you want them to be? What information would you want? Read-only or read-write?

[alexbw]

Read-write is important for my use case. I'm generating Python code that I hope to be legible. Part of that strategy is informative comments prepended above each block of generated code lines.

[gotgenes]

Please give me a list of things that would need to be stabilized in the API.

I would refer back to the "community wishlist" I compiled on #630

[davidhalter]

Pretty much all the features are implemented. There's a few that are not, but in a lot of cases it's either something that could also be done in third-party applications or eventually arrive (since it just adds to the API, it doesn't modify).

I'm pretty much finished with the API now. It's nothing extremely fancy. The syntax tree will lose some functions as well as gain some. But the differences will be marginal I think.

[davidhalter]

After another round of new features, it's finally done:
https://parso.readthedocs.io/en/latest/