Rubocop: Ruby 2.6 AST

[rrosenblum]

It looks like Ruby is testing out having an AST parser built into the language itself. I'm having trouble finding any useful documentation at the moment. We may want to investigate using it in RuboCop.

https://www.ruby-lang.org/en/news/2018/05/31/ruby-2-6-0-preview2-released/

[bbatsov]

Yeah, they've mentioned this yesterday on RubyKaigi and I was wondering what this was. In their typical style the Ruby Core Team created something without consulting with their actual potential users, which is disappointing, but maybe we can still give them some feedback before 2.6 is finalized.

I don't foresee us moving away from our parser any time soon, though. At the conference I also learned from @ptarjan that GitHub have created a fork of the original @whitequark parser written in C++ that's pretty fast, while retaining perfect API compatibility. That seemed like something more interesting to me.

[whitequark]

Doesn't the new Ruby AST lack column-precision spans?

[bbatsov]

No idea. When it was announced they mentioned something like "We think it's not documented yet.". I've got no idea what exactly have they built and what problems are they trying to solve with it.

[bbatsov]

I think at this point everything "official" about it is in the release notes that @rrosenblum shared.

RubyVM::AST::Node class is also introduced you can get location information and children nodes from Node objects. This feature is experimental. Compatibility of the structure of AST nodes are not guaranteed.

[ptarjan]

@charliesome is the original author of the C++ port of whitequark's parser. Charlie, are you still thinking about open sourcing the parser component? If not, we are happy to help out if you want.

Our current implementation has something like this:

$ cat /tmp/a.rb
def foo
    1 + 2
end
$ sorbet -p parse-tree-json /tmp/a.rb
{
  "type" : "DefMethod",
  "name" : "foo",
  "args" : null,
  "body" : {
    "type" : "Send",
    "receiver" : {
      "type" : "Integer",
      "val" : "1"
    },
    "method" : "+",
    "args" : [
      {
        "type" : "Integer",
        "val" : "2"
      }
    ]
  }
}

So that would be easy to package up in a gem if you like that interface. It should directly match the shape whitequark is giving you today. If you'd like a different interface I'm happy to discuss.

We also have a desugared AST which translates from the 96 nodes of whitequark's parser to 38 (things like translating a AndAsgn into some Sends) but that would be a bigger transition for you.

[charliesome]

@ptarjan Yep, give me a couple of days to get the approval and I'll throw the repo open! :)

[rrosenblum]

In their typical style the Ruby Core Team created something without consulting with their actual potential users, which is disappointing, but maybe we can still give them some feedback before 2.6 is finalized.

The feature is still marked as Experimental so we should be able to influence the direction. I agree, it would have been nice for the Core Team to reach out to some of the existing projects using AST for some direction. It seems like we're going to have to plan around with the functionality and open feature and bug report accordingly.

Regardless, it is cool to see support for AST being built into core Ruby.

[bbatsov]

RubyVM::AST::Node

Odd choice for a module name, though. It implies this is something VM-specific, which it should not be. That's my first feedback for it. 😆

[whitequark]

It is VM-specific because it's just the MRI internal AST structures exposed to the user code.

[bbatsov]

Well, it seems we won't be using it then.

[charliesome]

@ptarjan It's open source! https://github.com/typedruby/typedruby/tree/master/parser

[ptarjan]

Exciting! Thank you @charliesome. Are you planning on bundling up your parser into an easy to consume thing (make a binary, and a gem, and a package, etc.)? If not, would you be opposed to use doing it at some point?