Typescript: Proposal: Replace emitter with syntax tree transformations and a simplified node emitter.

[rbuckton]

In an effort to better support down-level transformations for complex emit like generators and Async Functions, and to be able to add new down-level transformations for the evolving ECMA-262 specification, we propose to replace the current emitter logic for the TypeScript compiler with an implementation that relies on transformations of the source tree.

Our current emitter is not built to handle the requirements for some of the more complex rewriting needed to support features such as generators (ES6/ES2015) and Async Functions (ES2016). In addition, as time goes on we will need to add more and more branching logic to handle down-level emit of new language features due to the yearly cadence that TC39 is adopting for the ECMAScript specification.

Syntax tree transformations will give us the ability to transform our source tree in an iterative fashion, allowing us to inject new transformations at the head of a transformation chain so that few (if any) changes need to be made to existing transformations in later iterations. As a result, there would be minimal maintenance for the complex transformations needed for down-level generators, as long as new features are already transformed into a syntactically valid ES6 tree.

Generally this will also help to reduce the branching logic of our current emitter, and keep syntactic transformations isolated to an individual file for a language version or feature. As a result, the emitter itself can be simplified drastically and focus specifically on emitting the given syntax tree with almost no branching logic.

Requirements:

• No transformation should directly modify the original source tree.
• Transformations should be isolated to a specific language version or, when necessary, a language feature (i.e. TypeScript to ES6, ES6 to ES5, AMD module transformations, etc.).
• Transformations should reuse existing syntax tree nodes when possible.
• Transformations should employ a mechanism to quickly identify whether a node or subtree requires transformation to avoid a full walk of a source tree for each iteration.
• Transformations should preserve source locations for use with source maps.
• Transformations should not significantly impact the performance of the compiler, and ideally should improve compile time.

Goals:

• Transform a TypeScript syntax tree into a compatible JavaScript syntax tree based on compiler settings.
• Source Map emit should behave as it does in the current compiler.
• Comment preservation should behave as it does in the current compiler.

Non-Goals:

• We will not be replacing our declaration emitter with transformations at this time, though we may consider this in a future proposal.
• We will not support end-user extensibility of these transformations at this time, though we may consider this in a future proposal.

Remaining Tasks:

• [x] Update to support block-scoped declarations captured in loops.
• [x] Verify comment preservation
• [x] Investigate comment preservation performance issues
• [x] Verify/adjust source map output
• [x] Fix failing tests
• [x] Update transformations to match recent changes to the emitter.
• [x] Investigate/improve performance of tree-transforming emitter.

[falsandtru]

+1 TypeScript should has more modularbility for testability and maintainability for implementations of new ECMA-262 specification. If possible, I want the plugable design and hook points for transforms like the babel. TypeScript team will be able to get feedbacks from user implemented plugins.

[tobich]

+1

[jjrv]

It would be very nice to allow extending the emitter to output JSDoc comments with types, so transpiled ES5 code could then be fed into Closure Compiler with advanced optimizations turned on.

[ctaggart]

Will you be able to create the Syntax Tree from scratch?
Will be be to write the transformed Syntax Tree to a file (a .ts file)?

[mhegazy]

that is the plan :)

[D1no]

It would be great if this would be compatible to the Babel-plugin style. Great way to open the door to implement typescript "native" runtime type checking

[ctaggart]

I see this was bumped back to TypeScript 2.1. :-(

[dasa]

https://github.com/Microsoft/TypeScript/wiki/Roadmap hasn't been updated to match.

[whitecolor]

Hope to see it soon! another big step to make babel obsolete.

[rbuckton]

Completed as of 4685646.

[whitecolor]

@rbuckton where it is possible to read how to use committed changes?

[phiresky]

@whitecolor see https://github.com/Microsoft/TypeScript/issues/1564#issuecomment-245148134 (use typescript@next)

[ctaggart]

Would it be possible to provide an example that creates a syntax tree from scratch and emits the TypeScript? I've been wanting to be able to do this for a couple of years now. See How do you do a straight TS emit?

[MastroLindus]

One thing I would like to do is to manipulate the emitted values for string literals. For example I would love to be able to specify strings to be translated with some kind of special syntax in string literals, like "@foo", and have the emitter translate all strings literals that match the given syntax to a specific function call.

In the non-goal there is : "We will not support end-user extensibility of these transformations at this time, though we may consider this in a future proposal."

So should I assume that doing something like that is not currently possible?

[mhegazy]

So should I assume that doing something like that is not currently possible?

yes. but we intend to expose all the transformation and factory APIs so it should be possible in the near future.

[MastroLindus]

Thanks for the answer, looking forward to that

[ToddThomson]

In my Typescript packaging compiler I create a single module bundle (via import dependency analysis) and then directly minify the bundle. To minify the ts bundle I must first perform identifier shortening, compile the bundle and then perform whitespace removal on the js output ( a multistep process as I currently don't have access to the emitter ).

With the new transformation and factory APIs I am hoping to do this all in 1 step.

My question is how to provide directives to a transform within the ts source file. Since I have total control of how I package my bundle I can essentially make the whole package internal and thus shorten the majority of identifiers. However, when using Angular 2 components, mangling identifiers which are used within component html templates is a problem!. If I had a mechanism to provide a directive to the minification transform I could then selectively bypass identifier shortening for those used within the template.

As an alternative to transform directives would using /* @transformDirective */, like your use of /* @internal */ be the way forward for now?

Thanks for any help!

[mischkl]

Out of curiosity, might this change be of benefit to transforming ES6 Maps and Sets to ES5 (#4031, #11209)?

[trusktr]

@mhegazy

yes. but we intend to expose all the transformation and factory APIs so it should be possible in the near future.

How near is this future?

[trusktr]

@mischkl

Out of curiosity, might this change be of benefit to transforming ES6 Maps and Sets to ES5 (#4031, #11209)?

I'm not entirely sure, but I don't think it is a goal of TypeScript to transform non-language features.Map and Set are just references to constructor functions that are implemented by the JS engine, they are not language features really (i.e. they are not syntax features of the language, they are just runtime tools provided to us).

Is there any plan to compile such things in TypeScript?

[RyanCavanaugh]

Is there any plan to compile such things in TypeScript?

No