Flow: Feature Request: annotate `flow-parser` estree output with `type-at-pos` type info

[juicetin]

Proposal

Add the information that flow type-at-pos <filename> <line> <column> would give us, to each node in the estree output from flow-parser, with an option that can be passed to flow-parser, such as annotate-types: <boolean>.

Use case

At the moment, to create custom rules within flow in a codebase, I first generate the tree using flow-parser and traverse it using a library such as estraverse, but to get the typing information, I have to use methods from child_process to be able to execute flow type-at-pos, which blows up the time required to run and severely limits the ability to run it on a large codebase as part of a regular process, such as builds.

Being able to generate all this info within flow-parser without the overhead of forking millions of child processes to run flow type-at-pos across a codebase would cut down time required to run the logic for such a use case by I expect orders of magnitude (at least _an_ order of magnitude, anyway).

[goodmind]

Related: https://github.com/facebook/flow/issues/248

[jbrown215]

@panagosg7: How hard is this to do given the Typed AST and Type Normalizer?

[panagosg7]

One way to speed up the use case is to use flow dump-types and match against the locations of the nodes that you want to query. I believe most location ranges that you'd be interested in are answered in that type dump.

A couple of issues you might find:

1. The produced types are not always parseable. You'd see things like class A which is not something we parse as an annotation. We've addressed this in other contexts, so this should be easy to port.
2. Not all inferred types can be serialized to a parseable form. Some types are "internal" (associated with implementation specifics) and are hard to serialize to an equivalent parseable form. The type normalizer tries to bridge this gap but it only achieves this to an extent. On the bright side, in most cases you should be able to get something reasonable.
3. Types often refer to type definitions that are remote or hidden in nested scopes. Flow can figure out where these are mostly reliably, but it's not clear how a tool like dump types can benefit from that. Any solution you might come up for this will have to be spec'ed and encoded in a meaningful way.
4. Types are large and repetitive (despite the normalizer trying to be clever about it).

How hard is this to do given the Typed AST and Type Normalizer?

This depends on how tolerant you are with failures (ie. types that you might not be able to work with). (1) is easily fixable, (2), (3) are more involved but you can make progress. (4) is hard. With that in mind, one could probably put something together pretty quickly, but there are many blockers as they try to increase the quality of the output.

with an option that can be passed to flow-parser

I'm not sure how you'd extend the flow-parser to include Flow's inferred types. AFAIK flow-parser does not depend on Flow (the typing part). So this would probably be a standalone tool.

[goodmind]

@panagosg7

I'm not sure how you'd extend the flow-parser

add "symbol" field to all ast nodes with type information?

can dump-types output json, not string? It is string even with --json

I have something working with type-at-pos already but it is slow as hell
https://github.com/goodmind/babel-flow/blob/master/src/ast.js

[panagosg7]

add "symbol" field to all ast nodes with type information?

My concern is that the flow-parser project does not invoke type-checking at all, and it doesn't look like a dependency we want to add.

can dump-types output json, not string? It is string even with --json

No, currently is uses ty_printer.ml to convert directly to string. There is however ty_serializer.ml that converts Ty.t (the output of the type normalizer) to Ast.Type.t. From that format one can convert to JSON using the same code as with any AST node.

[goodmind]

@panagosg7 I think there already was --raw option but it was removed, why?
http://github.com/flowtype/flow-inspect this was using it

[panagosg7]

This was probably before we rewrote the normalizer. We don't output the internal types anymore. Instead we have a simpler type language now (see ty.ml).

There is a JSON printer for this language (in ty_debug.ml), but hasn't been exercised much and it's not called by any command right now. The above approach, that serializes to Ast.Type.t first, feels better to me since it would reuse code that has been more widely used.