Typescript: Plugin Support for Custom Transformers

[MichaReiser]

Since #13764 has landed, it's easy to write custom transformers. However, if I understand the API correctly, it is needed to duplicate the whole tsc command line just to add a single transformer. This leads to incompatibilities since these, on typescript depending, applications do not support the same features as the tsc command line tool.

It, therefore, would be favored to have a plugin system that allows loading Custom Transformers from third party node modules as this is the case for the language service proxies #12231. These transformers then easily integrate into existing build tools / build workflows.

If someone experienced has inputs on how to implement the changes, I'm willing to create a PR as it would simplify my project tremendously.

[mhegazy]

We do not plan on exposing compiler plugin system in the short term. The transforms are exposed as part of the public API as you noted, and we are in the process of writing documentation and samples for these.

[MichaReiser]

I don't want to have an exposed plugin API. Instead, I would prefere way to register my custom transforms by just specifying them in the tsconfig.json instead of having to use the TS-Compiler API. Because using the TS-Compiler API implies that I can no longer use the tsc command line tool which further can imply that it does no longer integrate nicely into existing build tools and workflows.

[WanderWang]

anything update about documentation and samples ? @mhegazy

[longlho]

@MichaReiser you can write a simple compiler based on the API (sample here). We actually use TS very heavily within Yahoo Finance and the transformer public API have been awesome.

Those are some of the transformers we wrote after it became public:
https://github.com/longlho/ts-transform-system-import
https://github.com/longlho/ts-transform-img
https://github.com/longlho/ts-transform-react-intl
https://github.com/longlho/ts-transform-css-modules-transform

@mhegazy lmk if you need help documenting/gathering samples

[MichaReiser]

@longlho
Thanks for your sample.

That is what I'm currently doing. However, it makes it impossible to use other build tools created for typescript, e.g. web pack loaders, jest loaders. Besides, how can I use one of your plugins together with one of mine when each of us uses a different frontend?

Therefore, I do believe that the current approach is a good start but not sufficient for a plugin ecosystem. Because using a plugin is too much effort for the user and requires more than just writing the transformation code for the author.

I believe a plugin system like the one of Babel is essential for typescript if the goal is to encourage the community to create and use custom transformers.

[longlho]

@MichaReiser yup. I'm not saying it's sufficient for the ecosystem, just good enough for the 1st step towards it.

[ivogabe]

I'd like to add support for transforms in gulp-typescript, but I want to prevent that all TypeScript plugins (for gulp, webpack etc) propose a different API. And TypeScript might add even a different way for configurating this later on. So do you currently have plans to add this in the near or far future?

[RubaXa]

Hi all, I try to write a preprocessor, but nothing comes out :[

In fact, I have two questions:

• How to create AST-fragment from string?
• How to add import?

// 1. Input
class Foo {
    templateString = 'some value';
}

// 2. After transformation
import __LIB__ from '@external/lib';

class Foo {
    templateString = (function compiledTemplate(deps) {
        // ...
        return result;
    })({lib: __LIB__});
}

// 3. Expected result
var lib_1 = require("@external/lib");
var Foo = (function () {
    function Foo() {
        this.templateString = (function compiledTemplate(deps) {
            // ...
            return result;
        })({ lib: lib_1 }); 
    }
    return Foo;
}());

A Simplified Example: https://github.com/RubaXa/typescript-api-questions/tree/master/import-add

[RubaXa]

⬆️ ⬆️ ⬆️
@longlho, @mhegazy Can you give any hint?

[rbuckton]

@RubaXa Since you added the import declaration in a transform, it isn't bound or type checked. Since it wasn't bound or type checked, we cannot resolve __LIB__ in your expression to the __LIB__ in the declaration.

One option is to use a namepace import instead of a default import, so that your emit is something like:

import * as __LIB__ from "@external/lib"

As there is no aliasing that can occur.

The other hold onto a generated identifier for the import declaration, as per the attached zip

[rbuckton]

@RubaXa if your goal is to pass in the entire module object, you probably want to use the import * as __LIB__ from "@external/lib" syntax (which does not require aliasing) and not import __LIB__ from "@external/lib" as the latter imports the default export rather than the entire module object.

[longlho]

yeah we're doing import * as foo in our CSS modules transformer as well to prevent aliasing. Although it'd be nice to tap into that to inline certain named exports

[RubaXa]

@rbuckton O, thanks a lot!
Still interested in how to create an arbitrary AST-fragment from string?

function createFragmentFromString(code: string) {
  // ????
}

function visitPropertyDeclaration(node) {
    if (ts.isIdentifier(node.name) && node.name.text === "templateString") {
        // ...
        return ts.updateProperty(
            node,
            ts.visitNodes(node.decorators, visitor),
            ts.visitNodes(node.modifiers, visitor),
            ts.visitNode(node.name, visitor),
            ts.visitNode(node.type, visitor),
            createFragmentFromString('(function compiledTemplate() { /* ... */ })()')
        );
    }
    return node;
}

[rightisleft]

This is what i was hoping the transformer support and workflow would behave like in TypeScript.

https://www.dartlang.org/tools/pub/transformers

[chandu0101]

I am also very much interested in this feature.

as @MichaReiser mentioned
If someone experienced has inputs on how to implement the changes, I'm willing to create a PR as it would simplify my project tremendously.

love to see inputs from experts/typescript compiler DEV .

[MeirionHughes]

Looks like someone has wrapped typescript to expose this functionality. https://github.com/cevek/ttypescript

Also looks like ts-loader has it: https://www.npmjs.com/package/ts-loader#getcustomtransformers-----before-transformerfactory--after-transformerfactory----

[jcimoch]

I think this is something that could land in typescript 2.8 release or at least in roadmap at some point, @ahejlsberg @mhegazy @DanielRosenwasser what do you think? In this way custom transformers might be more popular and therefore more powerfull. Having option to plugin in transfromer from tsconfig.json perspective would simplify life a lot.

[mhegazy]

We have no plans to expose plugins in the short term, as noted earlier.

[raveclassic]

@mhegazy Is it a considered decision or is it out of scope just because of low community interest?

[mhegazy]

I did not say that. we would like to maintain a small public API/maintainability cost as well as flexibility in changing how the build is driven moving forward, and these both are affected by a plugin model.

[zerkalica]

Please, stop transformer plugin api fragmentation. Stabilize plugin api on top of transfomer api and expose them in tsconfig.json.

ts-loader, parcel-plugin-typescript, rollup-plugin-typescript2, ts-transformer-keys, ttypescript and others.

Each of them provides custom plugin registration method and custom plugin entry point format. It's a way to ts plugin hell.

[zerkalica]

Now cevec/ttypescript supports all common transformer plugin formats. It's a drop-in wrapper on top of all typescript modules and tsc, tsserver commands (just small runtime ts.createProgram patch). Webpack ts-loader and rollup-plugin-typescript2 can be easily configured with ttypescript.

TTypescript suggest universal transformer plugin format, but existing transformers supported too.

{
    "compilerOptions": {
        "plugins": [
            { "transform": "ts-transform-graphql-tag" },
            { "transform": "ts-transform-css-modules", "type": "config" },
            { "transform": "ts-nameof", "type": "raw", "after": true}
            { "transform": "./transformers/my-transformer.ts", "someOption1": 123, "someOption2": 321  }
        ]
    },
    "exclude": ["node_modules", "transformers/**/*"]
}

[therealmikz]

Any news on this one? Seems to me like TypeScript authors somehow don't want to allow custom transformers on vanilla TypeScript - that's a shame, it would be an awesome feature.

[MeirionHughes]

they don't want another api/surface to maintain for fear of leading to difficulty breaking things in the internals. Ironically, all these 3rd-party plugin solutions are, because there is no standard by the ts-team, all somewhat different and will break eventually.

[pcan]

they don't want another api/surface to maintain for fear of leading to difficulty breaking things in the internals.

After 3 years working with typescript and watching its evolution, I have come to a simple conclusion. Microsoft has open-sourced Typescript in the sense of the _code_, but not in the sense of the _process_. In the majority of the cases, open-source projects are more or less community-driven both in the decisions and in the coding, and this should be the natural way to evolve. Do you remember the IO.js fork of node.js? The community realized that the company interests were not so aligned with the common open source governance model, and they simply forked. I hope this won't happen for TypeScript, but Microsoft should take into account this possibility.
To be clear, I'm not blaming the devs, anyway, they do a great job, really.
Just my 2c, sorry for the OT.

[kitsonk]

Microsoft has open-sourced Typescript in the sense of the code, but not in the sense of the process.

I wholeheartedly disagree with this. I have never worked for Microsoft, but I have had direct influence on several features of TypeScript over the past 4 years. I was representing an open source library built in TypeScript, and we arranged to have some direct conversations, but all of the "debate" of any features was done open, in public, here. The core team publishes their design meeting notes. The only "insider" I got, representing an open source library, was an opportunity to argue my case for some features in person and get to meet the core team. It made me realise that the team acts as a team. One feature we talked about, Anders basically said the problem was too hard and that it would take a lot of time to chip away at it and solve the "real" problems people faced. That was two years ago and in TypeScript 3.0 we finally get it. But the community has a voice in the design process, but like any community, we have factured and various voices, and no team anywhere could please everyone, and if they did, TypeScript would be a _really_ bad tool.

You also want to label the core team "Microsoft". The core team was the least Microsoft team I have ever encountered. They fought battles to get total control of their release cadence, after they were founded on full control of the their release content. They also fought for openness and transparency. The fought to move to GitHub from CodePlex. Admittedly they aren't that unique anymore in Microsoft as several other teams have adopted their model, but they were, as far as I am aware, the ones who started it all, with the VSCode team following quickly after.

So just because the core team pick and choose their battles, stick to their Design Principles doesn't mean that the community doesn't have a voice, but as a voice we suffer from psychotic multiple personality disorder. We are a difficult customer to serve.

[mrmckeb]

My understanding is that this is the same as #16607, or would this achieve a different goal? As a very recent plugin-author myself, I'd also like to see this exposed - including when working with TypeScript using the Babel plugin.

[MeirionHughes]

@mrmckeb this issue is for standardizing and exposing typescript AST transformation, which already exists, while the linked issue seems to be discussing whole file transformation in a very broad (undefined) scope.

[FredyC]

I want to add my 2 cents here. Some of you may have heard about the brilliant project babel-plugin-macros. Basically, instead of having thousands of different Babel plugins, there can be the only one which is then powered by a user code to do a transformation. An awesomely transparent system without a need to mess with configuration all the time. This is especially great for projects like CRA which supports macros but forbids additional configuration.

Today I've opened the discussion on how to possibly get this to TypeScript world. If you have some insight, please do come and share.

Ultimately, I hope that if we somehow succeed, there won't be a need for this as only one transform will be necessary.

[dsherret]

@FredyC that's a cool project, but we'd need this issue to be resolved in order to easily inject something like that into the compiler when only using tsc. For example, with babel-plugin-macros it still needs to be specified in .babelrc and it would be nice to specify something like that in typescript in tsconfig.json.

Actually, are you suggesting that it would be nice to have babel-plugin-macro-like behaviour built into TypeScript and no configuration would be necessary? That might be nice, but personally I would prefer a configurable solution that allows specifying custom transformers in tsconfig.json and then a version of babel-plugin-macros that supports typescript ast nodes could be specified.

[FredyC]

@dsherret Yea, I don't expect it would be a part of TypeScript itself. However, I can imagine that once the transform is ready, it could be easy to prepare a tsc-like tiny wrapper that would be injecting only that transform for macros and rest could be done from a user code.

Edit: Actually, the tiny wrapper is already there, with above mentioned https://github.com/cevek/ttypescript it would be easy to instruct to use that along with the universal macros plugin and it's a win-win.

I just need to find people who are well versed in Typescript transforms to figure some basic prototype. I can discuss and talk, but implementing it is above my current skillset.

[pietrovismara]

they don't want another api/surface to maintain for fear of leading to difficulty breaking things in the internals.

After 3 years working with typescript and watching its evolution, I have come to a simple conclusion. Microsoft has open-sourced Typescript in the sense of the _code_, but not in the sense of the _process_. In the majority of the cases, open-source projects are more or less community-driven both in the _decisions_ and in the _coding_, and this should be the natural way to evolve. Do you remember the IO.js fork of node.js? The community realized that the company interests were not so aligned with the common open source governance model, and they simply forked. I hope this won't happen for TypeScript, but Microsoft should take into account this possibility.
To be clear, I'm not blaming the devs, anyway, they do a great job, really.
Just my 2c, sorry for the OT.

I agree. It is clear that despite strong interest from the community, devs are simply rejecting the idea of custom transformers/macros without actually providing a reason for the rejection except something on the lines of "People will misuse it", which also sounds offending towards the community as it implies it is a community of low quality devs.

I would accept a rejection if a reasonable justification other than "We won't simply do it" could be provided, but it seems despite all the requests from the community nobody has yet took the effort to provide that.

[mrmckeb]

I think this is unfair criticism @pietrovismara. I agree that this is a must-have feature, but this isn't "Microsoft" blocking this feature. The team, as I understand it, are saying that this is not easy to support and that's why they haven't added such a feature yet.

Again, I want this feature as much as anyone else - but we need to discuss this issue using facts. The facts are that the team aren't ready to support it yet. So let's keep voicing interest, but also keep the conversation on the right track.

[pietrovismara]

@mrmckeb If that's the case I can understand it. I must have missed such a declaration in the big stream of comments contained in these issues then. Didn't mean to offend anyone, just trying to provoke a clear answer from mantainers in order to, as you said, keep the conversation on the right track.

[mrmckeb]

It's OK, it's frustrating that the support for this doesn't exist - as I said, I feel it, and we're asked for some kind of solution for CSS modules all the time at CRA now that we have TypeScript support. I hope it'll come soon, but I also understand that it opens up a huge area of risk for the team.

[SalvatorePreviti]

... still waiting :(

[danielpza]

yep, still waiting ..., but waiting for what, is something been done ?

[Kirill89]

Is there any better solution than ttypescript now?
Are TypeScript developers eventually planing to support plugins?

[tanguyantoine]

You can create you own compiler ;-)

[inad9300]

Any news on the topic? It seems to me that the community has made use of this feature extensively enough to sit, collect feedback, and finally add support for transformers to the config file. There are even plenty of implementations to use as reference.

[samhuk]

It has been over 2 years (wow!)*. Are people open to a rethink of the pros and cons to opening up plugins for transformers?

I feel like the initial concerns that existed all that time ago, such as a lack of documentation about the API, and having to support the meshing of usercode <-> transformers <-> API, are not as relevant anymore. Open plugin support has worked wonders for packages like babel: progression through collaboration. I can personally attest to Babel specifically, having used it extensively.

Easier "just works" integration of transformers like typescript-is _would_ be nice. This package in particular I see as having quiet the revolutionary influence on typescript as a whole. True strong contracts! 😃

I think a conversation about the pros and cons would help dispel any dejection or differences on this. I get a feeling that this zombie-issue will keep on going until such happens - to be honest.

* And in the meantime someone creating the feature for TS instead (ttypescript). I've used this to success.

[Janpot]

@DanielRosenwasser I saw that there was "Investigate TypeScript plugin APIs" on the 3.5 iteration plan (and that work has already started on it).
Does that point handle about what's discussed in this issue? The lack of a link to an issue makes me assume it's too much WIP to share anything about it yet?

[myuseringithub]

I think work should be put on adding full support for Babel parser, that includes adding necessary features for Babel to support Typescript static analysis (e.g. multiple file transforms or dependency on multiple files during transformation). Rather than having to duplicate syntax/transform Babel plugins to Typescript equivalents.

[zpdDG4gta8XKpMCd]

in case you have any trouble with the following instruction, let me know, i will be glad to help

given the sheer negligence of the design team, here is a workaround

we are going to take advantage of tslint and its code fixing capabilities

in the following example we are going to do a type-to-code transformation, in its most basic form

we are going to use decorators as markers of places in code that need to be rewritten

in our case, a decorator is a fake function whose only goal is

• to mark a place of the transformation
• and to capture the type that is going to be the source of information for the generator

for the demonstration purposes, we are going to dump the names and types of properties of an interface as plain strings, into an underlying class

the following steps are for windows users only, if you are not one of them, may god help you:

1. start your VSCode
2. install the following extension: https://github.com/Microsoft/vscode-typescript-tslint-plugin
3. close your VSCode
4. go to a folder of your choice
5. run git clone https://github.com/zpdDG4gta8XKpMCd/code-gen.git
6. go to the folder code-gen: cd ./code-gen
7. run
   
   
   
   • 1-install.bat
   
   
   • 2-build.bat
   
   
   • 3-install-some-more.bat
   
   
   • 4-try.bat
   
   
8. observe an instance of VSCode opened
9. go to test.ts (Ctrl + P -> test.ts)
10. take a note of the following interface, it's going to be the source for the code generator

interface Data {
    one: string;
    another: number;
}

1. take a note of the following function, which comprises a phantom decorator which we are going to use as a marker

function gen<_T>() {
    return function (meh: any) {};
}

1. take a note of the site of rewriting, where we want to shove some auto-generate code based on the interface and the marker decorator

@gen<Data>()
export class Gen {
}

1. observe a squiggly line under @gen<Data>()
   

2. pick Quick fix... -> Needs a rewrite, wanna fix?
   

3. observe the auto-generated code:
   

---

for the reference here is the source code of the generator that you can find in codeGenRule.ts

import * as Lint from 'tslint';
import * as ts from 'typescript';

export class Rule extends Lint.Rules.TypedRule {
    public applyWithProgram(sourceFile: ts.SourceFile, program: ts.Program): Lint.RuleFailure[] {
        return this.applyWithWalker(new Walker(sourceFile, this.getOptions(), program.getTypeChecker()));
    }
}

class Walker extends Lint.RuleWalker {
    constructor(sourceFile: ts.SourceFile, options: Lint.IOptions, private checker: ts.TypeChecker) {
        super(sourceFile, options);
    }
    public visitNode(node: ts.Node) {
        if (ts.isDecorator(node)) {
            const checked = check(node, this.checker);
            if (checked !== undefined) {
                const [node, message, replacement] = checked;
                this.addFailureAtNode(node, message, replacement);
            }
        }
        super.visitNode(node);
    }
}

function check(node: ts.Decorator, checker: ts.TypeChecker) {
    const { expression, parent } = node;
    if (!ts.isClassDeclaration(parent)) return;
    if (!ts.isCallExpression(expression)) return;
    const { expression: identifier, typeArguments: typeArgs } = expression;
    if (!ts.isIdentifier(identifier)) return;
    const { text: name } = identifier;
    if (name !== 'gen') return;
    if (typeArgs === undefined) return;
    if (typeArgs.length > 1) return;
    if (typeArgs.length < 1) return;
    const [only] = typeArgs;
    const type = checker.getTypeFromTypeNode(only);

    // if you got to here, you are at the right place and you have a type

    // working on a fix
    const properties = checker.getPropertiesOfType(type);
    const allNameTypes = properties.map(p => {
        const { name } = p
        const type = checker.getTypeOfSymbolAtLocation(p, node);
        return name + ': \'' + checker.typeToString(type) + '\';'
    })
    const { newLine } = ts.sys;
    const body = newLine + allNameTypes.join(newLine);
    const { pos: start, end } = parent.members;
    return [
        node,
        'Needs a rewrite, wanna fix?',
        Lint.Replacement.replaceFromTo(start, end, body)
    ] as const;
}

[mrmckeb]

@zpdDG4gta8XKpMCd, thanks for sharing your concept - but please be civil.

given the sheer negligence of the design team, here is a workaround

This is not the appropriate way to address the TypeScript for not implementing a _feature_ that you (and many others, including myself) would like to have. This isn't a major defect, and is not reflective of "negligence". Please be respectful.

[zpdDG4gta8XKpMCd]

oh please do not take it personal, this is just the way i always start my complaints, i have a very sour personality (medical condition) and this is the best i can squeeze out of myself, i am very sorry

[zpdDG4gta8XKpMCd]

you people are not going to stop surprising me

• what you see is a piece of code that seems capable of solving a major today's problem with comfort (no tricks, no hacking)
• this workaround is a wake up call to the design team, and as we saw it happening before when the community was about to take a turn in a wrong direction, the design team was there to promptly addressed it: #4212,
• so if they care, they can do it right rather sooner until it's too late, or if they don't, we are free to dig in this direction and it won't be a problem for them down the road
• so the tone of my feedback was something inappropriate, but all the appropriate tone of yours has got you nothing since March 2, 2017 (over 2 years), and yes, you can wait another 2 years

[jhpratt]

@zpdDG4gta8XKpMCd What have you contributed towards an actual implementation? TypeScript is open source; I have no doubt they would seriously consider a pull request if you created one (even partially).

Blaming your rudeness on a medical condition is unacceptable; they are words that you typed and voluntarily chose to click "comment". No medical condition makes you do that. Speak, maybe, not type. You also have had the chance to edit your comment to remove it, yet you have not done so.

If you need custom transformers _now_, you can use Babel. It can remove type annotations from almost all TypeScript, while allowing syntax transformations as readily as you please. As a bonus, it's also faster.

I suggest you stop commenting before you get kicked from Microsoft's repository permanently. I know if it was my repo, you'd be on your last straw.

[zpdDG4gta8XKpMCd]

ok, you kicked me out, then what
can i ever come back? can i create a new account?
do you think i care about this account too much?

anyway, yes, i tried pull requests, unfortunately it's not feasible for the following reasons:

1. there is a certain category of problems you are welcome to solve, that are rather trivial and of less interest, anything serious like this - and your are not

2. since you cannot solve it on your own, a problem like this (even having 224 thumbs up) can wait for years if ever considered at all

3. luckily you can do something today using whatever means you have and start making a difference that can be seen, hence the suggestion (don't blame me for doing nothing)

[DanielRosenwasser]

@zpdDG4gta8XKpMCd as much as you know I've been a fan of the cynical takes over the years, I've gotta agree with others here - we need to keep the conversation civil and respectful. It's always good reminder that everyone tracking this issue wants to make the project better.

---

@Janpot Yup, it's very much a WIP, though when @rbuckton gets back, he may be able to give some more details on status.

As we investigate plugin points (like I alluded to in the 3.5 iteration plan), I think exposing easier hooks for custom transformers is a major scenario we want to consider. The reason we wouldn't necessarily just do this outright is that we might have broader work that subsumes this. In other words, maybe there's something broader than just pre- and post- transformers.

[zpdDG4gta8XKpMCd]

@DanielRosenwasser there is not that much cynical in it, i only said 2 words and then i said i am deeply sorry about that

anyway, point is i think we all agree that there is a good part of healthy frustration over how things go, and we do not blame it on you, we understand what drives this project

but if we look at it from our angle there must be somebody to blame, because letting issues like this sit over years just collecting thumbs up, is what stops from being more productive

i am sure there are reasons for putting it off and focusing on some other things first, so i think it would be better if you let your audience know why certain highly wanted features cannot be fulfilled, it could lower the degree of frustration, say:

• we cannot implement this feature because of A, B C and those need D, E, and F done first

so i guess i am referring to this conversation: https://github.com/Microsoft/TypeScript/issues/30696#issuecomment-478799258

[RyanCavanaugh]

Having thumbs ups doesn't create new resources out of thin air, make complex and difficult decisions simple and easy, change existing business priorities, stop in-flight projects, deprioritize other work, cause overly-ambitious proposals to become well-scoped, or alter a suggestion with wide-reaching and permanent impacts to become focused and maintainable.

I have no apologies for us doing (or allowing PRs for, which is what you were complaining about in #30696) upfront well-understood good and simple work ahead of extremely complex and high-maintenance work like this. It should not be hard to guess why we're willing to let someone come in and fix a leaky faucet while the plans to remodel the basement and add another story on top of the house are still being worked out.

[FredyC]

@zpdDG4gta8XKpMCd Why would you need to blame somebody? Why don't you blame yourself in the first place that you haven't made a PR to fix this problem? Or at least some constructive write up on how to approach it from an implementation point of view?

Oh man, even after that transparent write up you just linked you have decided to poke the bear again after only 2 weeks? Try to imagine to be in that position having 1500+ tasks at hand and you have to pick a couple of those. On our team, we are moving around 100 tasks and struggling fairly enough. Cannot imagine it would 15 times more 😮

[zpdDG4gta8XKpMCd]

@FredyC for god sake, look at my workaround before saying i am not doing anything

yes, i am not doing formal pull requests because:

1. i am not up to the level for making a good one

2. for simple ones, there is a long line of approval, and it takes tremendous work to keep a pull request created 2 years ago up to date with the current codebase, i don't have that much time

3. certain problems will not be even considered, given the grand design considerations of which only few people like anders hejlsberg aware

you cannot take away the frustration part, and i am not alone here

we know there are these factors at play:

• grand design of the language
• budget / resources / politics inside MS
• wishes of the community

i would be happier if the decision making with the mix of these ingredients was a bit clearer

i am done, it's gone too far, sorry to everyone whose feelings got hurt, i won't say a word

[zpdDG4gta8XKpMCd]

continuation of https://github.com/Microsoft/TypeScript/issues/14419#issuecomment-483920640

another option is to use function declarations themselves as markers, what's below is an implementation of a code generator for functions whose names start with toRandom...

the information source for the generator is the result type of the function

this is how it works:

1. notice the squiggly line under a function that starts with toRandom...
   
2. explore your options:
   
3. do a quick fix
   
4. observe the results:
   

here is the code of codeGenRule.ts that does it

as a bonus this implementation will only raise a linting issue if the current code of the function doesn't match the code that is supposed to be generated

import * as Lint from 'tslint';
import * as ts from 'typescript';

export class Rule extends Lint.Rules.TypedRule {
    public applyWithProgram(sourceFile: ts.SourceFile, program: ts.Program): Lint.RuleFailure[] {
        return this.applyWithWalker(new Walker(sourceFile, this.getOptions(), program.getTypeChecker()));
    }
}

class Walker extends Lint.RuleWalker {
    constructor(sourceFile: ts.SourceFile, options: Lint.IOptions, private checker: ts.TypeChecker) {
        super(sourceFile, options);
    }
    public visitFunctionDeclaration(node: ts.FunctionDeclaration) {
        const checked = check(node, this.checker);
        if (checked !== undefined) {
            const [node, message, fix] = checked;
            this.addFailureAtNode(node, message, fix);
        }
        super.visitFunctionDeclaration(node);
    }
}

function check(node: ts.FunctionDeclaration, checker: ts.TypeChecker) {
    const { name: identifier, type: result, body } = node;
    if (body === undefined) return;
    if (identifier === undefined) return;
    const { text: name } = identifier;
    if (!name.startsWith('toRandom')) return;
    if (result === undefined) return;
    const type = checker.getTypeFromTypeNode(result);

    // if you got to here, you are at the right place and you have a type

    // working on a fix
    const properties = checker.getPropertiesOfType(type);
    const newerBody =
        `{
    return {${properties.map(prop => {
            const { name } = prop;
            const type = checker.getTypeOfSymbolAtLocation(prop, node);
            const typeName = capitalize(checker.typeToString(type));
            return `
        ${name}: toRandom${typeName}(),`;
        }).join('')}
    };
}`;
    const olderBody = body.getFullText();
    if (areEqual(olderBody, newerBody)) return;
    const start = body.getFullStart();
    const end = start + body.getFullWidth();
    return [
        node,
        'Needs a rewrite, wanna fix?',
        Lint.Replacement.replaceFromTo(start, end, newerBody),
    ] as const;
}

function areEqual(one: string, another: string) {
    // AB: we cannot make any assumption what line endings are,
    // this is why we compare the text of code without them
    return one.replace(/\r\n|\n/g, ' ') === another.replace(/\r\n|\n/g, ' ');
}

export function capitalize(value: string): string {
    const length = value.length;
    if (length > 1) {
        return value.substr(0, 1).toUpperCase() + value.substr(1);
    } else if (length > 0) {
        return value.substr(0, 1).toUpperCase();
    } else {
        return value;
    }
}

[dsherret]

@zpdDG4gta8XKpMCd I believe this issue was more about having an easier way of plugging in custom transformers while emitting? Ex. specifying transforms in tsconfig.json so they can be used with tsc rather than using the api. Are you wanting custom code changes/fixes in the editor?

I haven't used them at all, but I think what you're talking about might already be possible with a language service plugin (I think by overriding getCodeFixesAtPosition on language service and inserting your own code actions... not sure though) https://github.com/Microsoft/TypeScript/wiki/Writing-a-Language-Service-Plugin

Or maybe I'm misunderstanding?

[zpdDG4gta8XKpMCd]

right. that's exactly what i used to think

i thought that a way to specify my custom transforms via tsconfig.json would be a dream come true

but then i realized that i like my workaround even better

here is my line of thinking:

• i needed a pluggable configurable code generator
• typescript doesn't have anything like that handy
• on the other hand i know that tslint (although being depreciated by the end of 2019) is a decent plugin platform to typescript that has a way to plugin and configure your code with great freedom
• turns out it has all i need:
  
  
  
  • pluggable and configurable
  
  
  • with all necessary UI
  
  
  • gives me an API to typescript
  
  
  • has a few little extras
  
  

given all that i cannot see myself writing something that uses the typescript language service, because:

• my laptop can only hold that many typescript language services, and some of them already are:
  
  
  
  • one of vscode
  
  
  • one of tslint
  
  

and i don't want to add another one on top of that

i wish i had a better way of plugging in my code transformers, but we have what we have

---

and this is how we do macros: #4892

[dsherret]

• my laptop can only hold that many typescript language services, and some of them already are:

  • one by of vscode
  • one by tslint

@zpdDG4gta8XKpMCd I believe everything should be using the same language service. So you'll have a single language service and then the tslint plugin along with your plugin will proxy that.

Btw, what you're trying to do here should definitely be possible with a plain language service plugin because that's what done here.

Anyway, let's try to keep the conversation in here on topic and only about custom transformers as part of the build with tsc rather than editor code changes/language service stuff. Most people are looking for a solution to do a transform across an entire project and doing that within the editor is not a viable solution (especially since doing a transform at that point defeats the purpose of using them). Ideally what should be implemented here should have nothing to do with the editor/tsserver or language service.

[zpdDG4gta8XKpMCd]

thank you for valuable input, i will consider it

i disagree with you on keeping the conversation only about custom transformers as part of the build with tsc, since

• the original request as written is not specific in which form transforms to be delivered, it only states they should not require having to
  > duplicate the whole tsc command line just to add a single transformer
• over 2 years this discussion has drawn quite a bit of attention from many people who don't seem to care that much about how transformers are implemented, provided there is a solution, it works and it's easy to use
• also, tslint team did a good job building the infrastructure for plugins, not using it is a waste of effort, provided it's de-facto the only officially recognized platform outside of typescript that deals with typescript
• also, we all agree that even if transformers ever find their way to the language it's not going to happen any time soon, but people here (including myself) needed them like yesterday
• lastly, since there are not that many better alternatives, why won't we consider at least what we have?

my suggestion is clearly labeled as a workaround (while the design team is taking their time to think harder on a proper solution), so why not?

if you insist we should do it elsewhere, please speak up

[zpdDG4gta8XKpMCd]

but to address your very concern of doing a

transform across an entire project

tslint does exactly that with its --fix argument for cli

[dsherret]

@zpdDG4gta8XKpMCd --fix will update TypeScript code in place. This issue is about doing transforms during emit (so the changes only end up in the final JavaScript files). See the referenced issue's PR (#13940):

Since #13764 has landed, it's easy to write custom transformers. However, if I understand the API correctly, it is needed to duplicate the whole tsc command line just to add a single transformer. This leads to incompatibilities since these, on typescript depending, applications do not support the same features as the tsc command line tool.

In other words, that issue brought CustomTransformers (click that link and see where it's used in the API), but in order to use them tsc needs to be duplicated somehow or wrapped, which is what ttypescript does. The second paragraph then talks about how this would be good because it would integrate nicely in existing build tools (since getting the custom transformers to use would be handled by tsc rather than different build tools doing it in different ways).

I think it would be more productive to open a new issue with your concerns with code fixes or wanting an easier way to generate TypeScript code within TypeScript code. I'm not sure exactly what those concerns are as I think a lot of it is already possible today, but what you've been discussing is how to do code fixes/actions, which seems like an unrelated subject to me.

I think the main questions to answer in this thread are what Daniel discussed—"maybe there's something broader than just pre- and post- transformers"—and what the solution to making this part of the build would look like.

[zpdDG4gta8XKpMCd]

you are vague

i am a person who understands and appreciates simple things

my problem that i am dealing with a some 3500+ files project, of which 15% doesn't need to be written and maintained by hands and there are about 5% of similar stuff on top of it to come

at the same time i know there is an API that can do it for me, but it's so half-baked that i can never justify my time dealing with it to get it fully baked

so when i came here i thought this is the right place to talk about it, and turns out there might be a quick and simple solution

bad luck some people who came here are to discuss some much finer matters

ok, outstanding!

and no, i am not going to open a new issue just to show people once again how tslint can solve one of the major problems, i did my job, i am done

[vsviridov]

Having configurable transforms out of the box would be nice. I'm trying to tackle a problem of in-code string l10n and i18n. I can leverage tools like tslint with a custom rule for extraction, but my options of transforming the code are limited, unless I use something like ttypescript, but that is problematic because the app I'm working on is an un-ejected angular app. It just forces me to add layer upon layer of build tools. With the pace of development the whole stack becomes really precarious.

[NickSeagull]

We are currently developing a framework and we are currently using ttypescript in order to access type-level information at runtime, it'd be nice to have the plugins option in the official compiler eventually 😄

[longlho]

I’m on vacation right now but will look next week when I’m back

On Sun, Jul 21, 2019 at 5:54 PM Giorgio Boa [e-mail]
wrote:

@longlho https://github.com/longlho You created a lot of great ast
transformers, I need your help please 👍
I really would like modify decorator metadata before Angular copilation.
My goal is modify this node
Component({ selector: 'standard' ... }) into Component({ selector: 'custom'
... })
I made a github repo for test it
https://github.com/gioboa/ng-ts-transformer . I think it's possibile but
without documentation I need some help. 🙏 Thanks.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/microsoft/TypeScript/issues/14419?email_source=notifications&email_token=AABQM335QEEDNVT4NUICJQDQASBDXA5CNFSM4DCFER5KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2OGIWQ#issuecomment-513565786,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AABQM33S5IYXB5HOZTRVVX3QASBDXANCNFSM4DCFER5A
.

[uittorio]

I've contributed to create a library to create real mocks from interfaces without proxy . I'm recommending to use ttypescript for now but it would be nice to have an integrate way with typescript

[SephReed]

I would really love to see this feature happen. I'm pretty sure this is the only thing standing between Typescript and some of the greatest innovations it may lead to.

[langpavel]

Typescript is huge contribution to frontend development, but it also has a brake on further innovation…
Do not break the semantics. No plugin ATM cannot break semantics; and no plugins are allowed at all :man_shrugging: Why?
There are really good plugins, but. Every plugin must respect (key)word tokenization and AST instation)… , really hard way to do something inside.
It's no so hard. It is no easy, but babel project show that it's possible.
In fact it is almost trivial for babel to introduce new semantics or even syntactics phrases.

I do not expect from typescript to allow semantics. (also I'm not expecting syntax changes of course! )
What I wish is from compiler is shift in some cases, like (may be only type) preeval.
Like if I have a took function, like route or message or something, compiler can preeval and (oh, please, at phase one) check correctness.

If you reading my previous paragraph weird. What is tsx and all React goodness?

No React, no flow, no typescript at all, almost not at glance.

Please. I'm well understanding careful look-standing at all this; but as we learn from IE5.5+; blocking is no way forward.

If something needs improvement, it's still layout engine. D.Knuth was guenie who designed all the missing parts 50 years ago. Why we have not this today?? :-)
Please, tell the people to stop CSS in JS evilness.

Please. Open TS for semantics intelligence; like function with static string input, providing type hint for the rest; analyse actual const input string…
A Little Example:

• GraphQL call return shape
• Intl input shape
• …much more_
  Please. No semantics changes needed. Everything will work as desired. Something more may be checked at build time. That's all folks! :rabbit:

[Somebi]

bump

[zloirock]

I'm experimenting with writing TS transforms for optimization core-js import - automatic polyfilling for target environments, like @babel/preset-env and @babel/runtime, but without babel. It could seriously help a big part of TS developers. But without support custom transforms from the box without additional tools like ttypescript, the possibility of using those transforms will be seriously limited, so I'm not sure that it makes sense.

[webschik]

I've been experimenting with such an idea - https://github.com/webschik/typescript-polyfills-generator.
I think that Transformers API will open a possibility to drop babel from the development chain. I think a lot of developers do not want to use them both, but they still need plugins such @babel/preset-env.

[lucas-rudd]

We currently have rolled our own custom compiler script (using the TypeScript compiler API) due to this limitation. We briefly looked at ttypescript, but the appetite to move away from the vanilla TypeScript compiler to an extension of the TypeScript compiler is simply not there.

It would be fantastic if custom transformers were supported in tsconfig.json. It's cumbersome to write a script which simply parses tsconfig, declares transformers, then runs the compiler; or, worse, to actually try and attempt to re-implement foundational features of the compiler, like watching files, in a custom script.

Other people have been echoing this sentiment, but allowing for the ease of integrating custom transformers into the compiler would take a significant load off of the TypeScript development team and allow for innovation and additional forward momentum with this language from the community.

An example of a very successful plugin ecosystem is the Serverless Framework's plugin community. The development team cannot keep up with the demand for feature development, and plugins allow for a great way to allow for the integration of new (possibly experimental) features without the need to open PRs or water down the core product with features which may or may not provide value to the broader user base.

[waterplea]

I'm building a library and I have decorators in it that work great for me and that I depend on. I realized some time ago that decorators make my library not tree-shakable. After looking into it I came to conclusion that transformers could help me turn those decorators into tree-shakable code upon compilation. It is unfortunate that I cannot define them in my pipeline. I hope you will get to this issue soon, just wanted to drop my 2 cents on a use case where it would be very helpful.

[nonara]

In the interest of making TypeScript support custom transformers without wrapper programs and complicated instructions, I've written a tool called ts-patch (npm github)

Simply install ts-patch (globally or locally) and run ts-patch install to patch typescript. You can also specify the typescript installation directory and/or enable persistence, which maintains the patch if TS is updated or reinstalled. (details: ts-patch /?)

The patch logic is mostly based on ttypescript. (Big thanks to cevek's great work!) It's written in a way where it can easily be added to npm package install process. It's also easy to unpatch.

To be clear, this directly patches relevant files in typescript itself and is not a wrapper. Simply run the patch after dependency installation, and typescript will be transformer ready.

Hope this helps some people!

[madou]

In the interest of making TypeScript support custom transformers without wrapper programs and complicated instructions, I've written a tool called ts-patch (npm github)

Simply install ts-patch (globally or locally) and run ts-patch install to patch typescript. You can also specify the typescript installation directory and/or enable persistence, which maintains the patch if TS is updated or reinstalled. (details: ts-patch /?)

The patch logic is mostly based on ttypescript. (Big thanks to cevek's great work!) It's written in a way where it can easily be added to npm package install process. It's also easy to unpatch.

To be clear, this directly patches relevant files in typescript itself and is not a wrapper. Simply run the patch after dependency installation, and typescript will be transformer ready.

Hope this helps some people!

could this be raised as a PR to add/discuss proper support? :)

[SephReed]

Hey. For what it's worth, there's a bundler called fuse-box which makes it very easy to add custom transformers. Its 4.0 version is coming up soon... so it's in an in-between spot. But overall, I really like the bundler. Maybe it can help you too.

https://github.com/fuse-box/fuse-box/blob/master/docs/plugins/pluginCustomTransform.md

[nonara]

could this be raised as a PR to add/discuss proper support? :)

The TS team has expressed that they do not want to provide this as a native feature. Their reasons for the decision make sense!

The good news is that because TS is open source and is already compiled in a modular fashion, ts-patch behaves in a way very much like it would if it were built in. So it's not like a buggy reverse engineered bytecode patch.

If by support, you're concerned over it being maintained, I plan to keep it current and working for all future versions of TS! A lot of my commercial infrastructure already depends on it.

As a side note, I will be releasing a new version soon which allows for packaging multiple plugins with different entry points into a single package as well as some other optimizations to improve it and make it easier to use.

[MeirionHughes]

After writing a plugin I can appreciate why the ts team are hesitant - the current transformer is a post-compile step. You run into nasty problems if you do anything complicated and start adding new content that wasn't originally binded. As it stands, ts-node --compiler ttypescript foo.ts works perfectly fine. I'd rather the ts team deal with transformer plugins at different compilation steps... or allow for a rebinding step.

[nonara]

The current transformer is a post-compile step. You run into nasty problems if you do anything complicated and start adding new content that wasn't originally binded.

Transformers can run either before or after TSC compilation. It sounds like what you may be referring to is that the checker doesn't update after modifying nodes. This can actually be circumvented, the functionality just wasn't built out for it. When you invoke ts.transform(), it doesn't create a full Program instance for you to work with, so if you want to work with the checker, you need to create one. Using CompilerHost, you can reload it with the modified files after altering the AST.

I haven't gotten too far into it yet, but it looks like it also may be possible to use 'watch' functionality to avoid having to rebuild the entire Program instance each time. Put simply, it turns out it's not actually _too_ difficult to have a more complete Plugin system, but transform simply lacks the support in its provided context.

[MeirionHughes]

@nonara I've used transforms with ttypescript and ts-loader`, both have full program entry points. My problem is that if you add a new import statement, its missing "something" and those new statements are stripped out of the final output emit. @weswigham says its because transform is a post-binding step. The solution apparently is to make an entirely new Program for the file - but that just seems like a headache when you're dealing with things like ts-loader (especially when its in transform-only mode and completely hides the webpack/bundler file-resolver. In the end I dropped the import and just inlined the code I needed.

[nonara]

@MeirionHughes Ah, ok. In that case, yes, you do have a Program instance. ttypescript works by hooking createProgram to patch the resulting program's emit method to include the transformers.

• ts.emitFiles calls ts.transformNodes during the process.
• ts.transformNodes can be called with or without a Program instance. Directly calling ts.transform() does it without one.

If I'm understanding it correctly, it sounds like your issue is also tied up in the fact that the types & symbols have already been walked, and they do not get updated after AST transformation.

[madou]

Hey everyone I've been writing up a Transformer Handbook to colocate all the information about transformers as well as to make it easier to get started.

https://github.com/madou/ts-transformer-handbook/blob/master/translations/en/transformer-handbook.md

Would love some eyes on it if you can spare a sec 👍

[longlho]

@madou I also wrote a buncha posts on TS transformers https://levelup.gitconnected.com/writing-typescript-custom-ast-transformer-part-1-7585d6916819

[alshdavid]

What's the status on this?

[nth-commit]

Love typescript a lot, but it feels especially clunky when you start to interop between static types and runtime JS. There are numerous community-written plugins to make this better, e.g. https://github.com/dsherret/ts-nameof, https://github.com/kimamula/ts-transformer-keys - but without first-class plugin support it feels like a risky technical decision to include them in a project. Enabling plugins natively would allow the community to play around with potential future typescript features, before they need to be included in the core library.

[canonic-epicure]

I think TypeScript team should re-consider their argumentation. The plugin eco-system is a proven way of moving the language forward, enabling (and encouraging) experiments (see Haskell with its pragmas).

Some of those experiments will be wild and crazy, and some will eventually stabilize and become common. Its always good to provide the ability to implement language features in the user space.

In general, the position of TypeScript team looks unnecessary rigid.

[jantimon]

There are also transforms which don't change the language but provide additional information for easier debugging.

For example the babel-plugin-transform-react-jsx-source is part of the default react transpiling preset (@babel/preset-react).
It transforms

<sometag />

Into

<sometag __source={ { fileName: 'this/file.js', lineNumber: 10 } } />

This information allows react to provide better error stack traces during development.

For typescript there is no standard way to achieve the same although there is an open source transform: https://github.com/dropbox/ts-transform-react-jsx-source

[nojvek]

I understand @mhegazy mentioned twice that TS team has no plan of including “plugins” as part of tsconfig. What is the reasoning behind this?

Would you be open to a PR? The way ttypescript handles the transformers via tsconfig is pretty nice.

Usecases:

1) generating interface types to schemas so they can be runtime checked.
2) importing json with strings as literals so typescript doesn’t barf when assigned to a type with discriminated unions.
3) importing css/yaml and other object like syntax
4) making graphql queries that dynamically create the right types
5) compiling jsx to html strings so they can be innerHTML injected
5) rewriting absolute imports to relative imports based on tsconfig paths.

The list goes on and on. The plugin api is nice. Thanks for landing it. Having “plugins” part of “tsconfig” makes tsc really powerful. Is there something big we’re missing ?

[madou]

you can also do a lot of performance optimizations. i've been writing https://github.com/atlassian-labs/compiled-css-in-js with typescript transformers 🤙 would be cool if consumers didn't need to jump through hoops to use it though.

[pldilley]

Dear Typescript team, please stop dragging your heels on this <_<

[imrickywong]

What's the status on this?

[NSExceptional]

Status?

[ilya-buligin]

Friends what news?

[jantimon]

The typescript team has decided against this idea - if you need flexibility or better dev tooling using typescript is not an option without hacks - go with babel instead and use typescript only for typechecking

[ilya-buligin]

The typescript team has decided against this idea - if you need flexibility or better dev tooling using typescript is not an option without hacks - go with babel instead and use typescript only for typechecking

But what if I want some hacks for type-checking? I write a little transformer that makes some magic:

type Human = {
    name: string,
    age: number
}

const isValid = check<Human>({ name: 'Carl', age: 16 }) // => true
const isValid = check<Human>({ name: 'Carl' }) // => false

Function check transformed into specific type-guard function! How I can make it with Babel? Currently, I should use ttypescript...

[nonara]

The typescript team has decided against this idea - if you need flexibility or better dev tooling using typescript is not an option without hacks - go with babel instead and use typescript only for typechecking

There is actually some talk of this becoming supported in the future. That said, it really is already supported. The TypeScript compiler API supports transformers, and it's _really_ simple to write a custom compiler that uses transformers in only a few lines of code.

But in order to make it easier, you can use ttypescript or ts-patch.

These libraries aren't really a "hacks" in the sense that they are not augmenting the compiler to add transformer ability. Instead, they simply expose the existing functionality of the compiler API to tsc, making it usable via tsconfig.

[just-boris]

@ilya-buligin if I understand your use-case correctly, you can declare an ambient constant

declare const check: <T>(value: unknown) => value is T;

const isValid = check<Human>({ name: 'Carl', age: 16 }) // => true

Typescript will compile the code relying on check type definition and then you will replace it with a generated code using Babel.

[ilya-buligin]

@just-boris how I can get access to generic type <T> in Babel?

[MartinJohns]

This thread is getting fairly repetitive and off-topic. @RyanCavanaugh Perhaps this thread should be locked, until you have any updates on the subject.