Typescript: Skip typechecking; only emit (support `--transpileOnly` in `tsc`, re-open of #4176)

[dtinth]

Search Terms

isolatedModules, incremental build slow, allowJs, transpileOnly. #4176

Suggestion

Support a compilation mode where files are only transpiled without typechecking. This can greatly improve compilation speed. Similar to the transpileOnly flag in ts-node and ts-loader.

Use Cases

At @taskworld, we are trying to migrate our project to TypeScript. We have 1400 source files.

As we try to get our .js files to be processed and transpiled by tsc, setting "allowJs": true makes tsc take a painfully long time (40 seconds) to complete, even in --watch mode. tsc --diagnostics shows that lots of time is spent in typechecking phase.

I have checked these issues:

• https://github.com/Microsoft/TypeScript/issues/7808
  No resolution.

• https://github.com/Microsoft/TypeScript/issues/10018
  Irrelevant — compiler speed has since been improved but that did not solve our problem.

• https://github.com/Microsoft/TypeScript/issues/21221
  I used tsc --listFiles to check the files included in the compilation unit. Looks normal. In that issue, they solved the problem by renaming all .js to .ts, which we can’t do yet without causing errors all over the place, due to our JS files are still CommonJS modules, which is only recognized when using JS files. I try to migrate with minimal code changes, so I avoid mass-converting CommonJS to ES modules for now.

• https://github.com/Microsoft/TypeScript/issues/13538
  People at Google solved this problem by separating their project into libraries and creates their own build system (based on Bazel). Since I tried to add the tooling with minimal code change, we don’t want to “split our code base into multiple units” right now.

• https://github.com/Microsoft/TypeScript/issues/10157
  OP solved this problem by using transpileOnly mode in ts-loader, which uses the “single-module transpilation mode” (the ts.transpileModule API). However, we aren’t using webpack and we’re directly using tsc to compile .ts files to .js files.

• https://github.com/Microsoft/TypeScript/issues/22953
  No resolution.

I tried to profile the tsc process, and found that a lot of time is spent in resolveCallSignature.

If we can skip the type-checking process, this compilation phase should be faster.

This seems to be supported in both ts-node and ts-loader, since TypeScript provides the “single-module transpilation mode” (the ts.transpileModule API). So, I looked for a way to do it using tsc. Turns out, it is not available, and we have to somehow use the ts.transpileModule API directly.

https://github.com/Microsoft/TypeScript/issues/4176#issuecomment-128505179

A fancier solution would be to use the compiler's transpile API directly.

https://github.com/Microsoft/TypeScript/issues/13538#issuecomment-273695261

If you are willing to get your entire project compiling under the isolatedModules switch, then you can safely wire up your build system to do a simple emit of only changed files, which should be practically instant, followed by a re-typecheck.

Examples

All evidence so far suggests that we have to build our own tooling which behaves like babel -d build-dir source-dir (e.g. compiles each file separately) but for TypeScript. And so we implemented our own workaround:

// tsc-fast.js
const args = require('yargs')
  .options({
    force: {
      alias: 'f',
      description: 'Recompiles even if output file is newer.',
      type: 'boolean',
    },
    watch: {
      alias: 'w',
      description: 'Watches for file changes.',
      type: 'boolean',
    },
  })
  .strict()
  .help()
  .parse()

const watch = require('gulp-watch')
const ts = require('gulp-typescript')
const newer = require('gulp-newer')
const tsProject = ts.createProject('tsconfig.json', {
  isolatedModules: true,
})
const vfs = require('vinyl-fs')
const debug = require('gulp-debug')
const sourcemaps = require('gulp-sourcemaps')

function main() {
  let compiling = false
  let pending = false

  function compile() {
    if (compiling) {
      pending = true
      return
    }
    compiling = true
    const rawInput = tsProject.src()
    const input = args.force
      ? rawInput
      : rawInput.pipe(
          newer({
            dest: 'dist',
            map: f => f.replace(/\.ts$/, '.js'),
          })
        )
    input
      .pipe(sourcemaps.init())
      .pipe(tsProject())
      .pipe(sourcemaps.write('.'))
      .on('error', () => {
        /* Ignore compiler errors */
      })
      .pipe(debug({ title: 'tsc:' }))
      .pipe(vfs.dest('dist'))
      .on('end', () => {
        compiling = false
        if (pending) {
          pending = false
          compile()
        }
      })
  }

  compile()
  if (args.watch) {
    watch(['app/**/*.js', 'app/**/*.ts', '!app/vpc-admin/front/**/*'], compile)
  }
}

main()

To typecheck in separate step, we simply run tsc --noEmit in a separate CI job. Also, VS Code takes care of typechecking in the editor, so we already get instant feedback for type errors.

Checklist

My suggestion meets these guidelines:

• [x] This wouldn't be a breaking change in existing TypeScript/JavaScript code
• [x] This wouldn't change the runtime behavior of existing JavaScript code
• [x] This could be implemented without emitting different JS based on the types of the expressions
• [x] This isn't a runtime feature (e.g. library functionality, non-ECMAScript syntax with JavaScript output, etc.)
• [x] This feature would agree with the rest of TypeScript's Design Goals.

[dtinth]

I noticed the questions in #30117 and would like to give some answers for context:

Is this just piping errors to /dev/null ?

If this has to do with .on('error', () => { /* Ignore compiler errors */ }), this only prevents the script from crashing Node.js when a compiler error occurs. gulp-typescript outputs error messages to console already.

Why is this user's project so slow?

I realize that I haven’t put the diagnostic information in the issue.

This is the diagnostic info for incremental compilation.

Files:           2099
Lines:         967290
Nodes:        2998862
Identifiers:   983959
Symbols:      1266945
Types:         211713
Memory used: 1473430K
I/O read:       0.01s
I/O write:      0.98s
Parse time:     0.57s
Bind time:      0.02s
Check time:    16.95s
Emit time:      7.99s
Total time:    25.54s

Here’s the corresponding CPU profile:

As you can see, about 80% of “Check time” is spent in checkCallExpression function. So maybe there might be some slow stuff going on there — I may have to do a more thorough investigation.

Is this a global project or a module project?

This is a module project, but allowJs is on and most of JS files are CommonJS modules.

(cc: @RyanCavanaugh)

[RyanCavanaugh]

Discussed. We felt like we had a good handle on this and then totally deadlocked on the question of whether this mode would imply --noResolve:

• If the idea is simply to transpile ASAP, then not doing module resolution is a huge time savings
• ... but this means you have no straightforward path to migrate an existing project to transpileOnly
• Some healthy mix of users wants to a) transpile everything under some folder vs b) do a fast no-check build to get a faster build output

Separately, the problem that this mode only "works" if --isolatedModules is on was a point against it.

Ultimately a tool that just calls ts.transpileModule on some defined set of files is quite trivial to write, and would have clearer semantics than what we might provide out of the box.

We were curious if things got better for you with --incremental.

[biels]

I would add another reason c) treat all ts errors as warnings and still emit the results.
This is available on browser projects when using webpack with ts-loader using transpileOnly: true.

The advantage of this workflow is that it enables faster prototyping when combined with hot reload / run on save. You can see type errors alongside runtime errors and get the best of both worlds. You still care about ts errors as they show up in your IDE and build logs but they do not necessarily break your build.

This may not be desired on all projects or environments but that's where the flag comes in. I would suggest to add a transpileOnly boolean flag to the compliler so that it supports this behavior natively and can be used for projects that do not target the browser.

[ayroblu]

tsc can emit even if there are errors (how we quickly fix bugs by trying things)
Looking at this cause --incremental doesn't re emit files that haven't changed, nor does it remove files that have been deleted (which is why I want this)
ts-node already has a fast flag, so it'd be nice to have this here

[ayroblu]

Discovered typescript-transpile-only guess this solves my problems:
https://github.com/cspotcode/typescript-transpile-only

[biels]

This should be included in tsc as --transpileOnly flag

On Mon, Jun 24, 2019, 07:50 Ben Lu [e-mail] wrote:

Discovered typescript-transpile-only guess this solves my problems:
https://github.com/cspotcode/typescript-transpile-only

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/microsoft/TypeScript/issues/29651?email_source=notifications&email_token=ABHLQMG5MAGNM2PWKEJD7PDP4BOBJA5CNFSM4GTLYFDKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYL2RAA#issuecomment-504866944,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABHLQMC5IOHUUBHISY3BKBLP4BOBJANCNFSM4GTLYFDA
.

[canvural]

Another reason this would be great is compiling in the production environment. In production we don't install dev dependencies, so running tsc on production results in an error because it can't find type definitions. But our code is checked before it reaches to production, so in production, we know its fine type wise.

[sheerun]

My use case is quickly transiling typescript snippet so I can use it in non-typescript project. Currently I need to use online playground for this...

[kitsonk]

There is feedback for Deno that having a transpile only option might be a viable way to speed up startup of some workloads (denoland/deno#3321) While we cache the output of a compile, if you are loading some known good TypeScript for the first time, just parsing and emitting, and not type checking, might be a good option.

[weswigham]

@kitsonk if you have code you assume is "known good", why not add some more into that definition of "known good", rather than just "was type checked", and support shipping and loading a bytecode? Kinda like .pyc files for python.

[kitsonk]

@weswigham that is something @ry and I talked about, V8 does have a byte code type format that we could cache instead of even JavaScript. But let's say you were consuming a source module that your instance of Deno hasn't cached yet. You would go to a well maintained source, that you know us valid and injest it, and all you want out is the "bytecode" as fast as you can. Personally I wouldn't want to do it that way, I would always want to spend a few cycles checking the code, but I guess I can see an "just erase the types" emit only as supporting some types of workloads.

[weswigham]

🤷‍♂ Deno uses the typescript API, right? It just needs to use ts.transpile in that scenario, then.

[kitsonk]

@weswigham 😊 um... yeah... 🤦‍♂ sorry for the noise.

[beenotung]

@canvural my workaround is to compile the project locally than rsync the dist to the server

[nicojs]

I've got 2 more use cases. One has to do with performance in a CI/CD pipeline, the other has to do with mutation testing.

CI/CD

Let's say your CI/CD pipeline looks like this:

                               +--->+  test  |
+-------------+    +--------+  |    +--------+
| npm install +--->+ tsc -b +--+
+-------------+    +--------+  |    +--------+
                               +--->+  lint  |
                                    +--------+

When tsc -b here takes minutes it might make sense to split type checking off into a separate task like so:

                                                +-->+  test           |
                                                |   +-----------------+
                                                |
+-------------+   +------------------------+    |   +-----------------+
| npm install +-->+ tsc -b --transpileOnly +------->+  lint           |
+-------------+   +------------------------+    |   +-----------------+
                                                |
                                                |   +-----------------+
                                                +-->+ tsc -b --noEmit |
                                                    +-----------------+

Depending on the duration caching between build jobs (gitlab ci does a good job at this) this saves minutes per pipeline.

Mutation testing

Let's say you're building a mutation testing framework that supports typescript (something like Stryker mutator, which I am maintainer of) and you're planning to implement mutation switching in order to improve performance (which we're planning to do). You would want to split of type checking and transpiling into separate steps.

Let's say you have this statement:

const foo = 'bar ' + (40 + 2);

The mutation testing framework might want to create 2 mutants here:

// mutant 1
const foo = 'bar ' - (40 + 2);
// mutant 2
const foo = 'bar ' + (40 - 2);

_Note that mutant 1 results in a compile error, since in TypeScript - isn't defined for strings._

When using mutation switching, the mutation testing framework will make 1 version of the code that contains both mutants. Like so:

let foo;
switch(global.activeMutant) { 
  case 0:
    // mutant 1
    foo = 'bar ' - (40 + 2);
    break;
  case 1:
    // mutant 2
    foo = 'bar ' + (40 - 2);
    break;
}

The mutation testing framework will now test both mutants. First it will type check (using the type checker api) and if valid, run tests for that mutant.

The model of transpiling once and in the background type checking all mutants individually saves a lot of time over compiling each mutant indivually, since the test runner process will not have to load in new files at all.

Ultimately a tool that just calls ts.transpileModule on some defined set of files is quite trivial to write, and would have clearer semantics than what we might provide out of the box.

Writing such a tool might not take that much time, but writing and maintaining a _drop in replacement_ will take a lot of time. Just think about the CLI changes between releases. A daunting task if you ask me.

[askirmas]

It was my first time of writing babel config. So
babel.config.json:

{
  "presets": [
    ["@babel/preset-env",
      {
        "targets": {
          "node": 10
        }
      }
    ]
  ],
  "plugins": [
    "@babel/plugin-transform-typescript"
  ]
}

execution:

babel-node --extensions .ts -- index.ts

[askirmas]

I didn't check yet ts-node --transpile-only

[nicojs]

I found a workaround that might work for some use cases.

It works by prefixing every file content with // @ts-nocheck. It only works from TS3.7, since that version introduced // @ts-nocheck. It uses the createSolutionBuilder api, so works with project references 👍

// my-little-builder.js
const compiler = ts.createSolutionBuilder(ts.createSolutionBuilderHost(
  {
    ...ts.sys,
    readFile(fileName, encoding = 'utf8') {
      if (fs.existsSync(fileName)) {
        let content = fs.readFileSync(fileName, encoding);
        if (!fileName.includes('node_modules') && fileName.endsWith('.ts')) {
          content = '// @ts-nocheck\n' + content;
        }
        return content;
      }
    }
  },
  ts.createEmitAndSemanticDiagnosticsBuilderProgram,
  (d) => console.log(ts.formatDiagnosticsWithColorAndContext([d], {
    getCanonicalFileName: fileName => fileName,
    getCurrentDirectory: process.cwd,
    getNewLine: () => os.EOL
  })),
  (status) => console.log(`status: ${status.messageText}`),
  (summary) => console.log(`build summary: ${summary}`)
), ['tsconfig.json'], {});
const exitStatus = compiler.build();
console.log(`Exit status: ${exitStatus}`);

@RyanCavanaugh What do you think of allowing users to set a global @ts-nocheck option? Maybe by adding --no-check? People would be allowed to override it per file. This would be inline with the way type checking for js files works. It would probably need a warning in the console, since type checking would be off.

_Note_: This does not allow for notable performance improvement:

$ time npx tsc -b

real    0m55.219s
user    0m0.106s
sys     0m0.246s

$ npm run clean
$ time node my-little-builder.js
build summary: 0
Exit status: 0

real    0m54.140s
user    0m0.015s
sys     0m0.031s

[dsagal]

Since this is "awaiting more feedback". In watch mode, I ideally want to have my output ASAP and my errors whenever they are ready. We used to use ts-loader with fork-ts-checker-webpack-plugin, which together provided exactly that. We switched to tsc --build, and now lost that significant speed-up.

If --transpileOnly option were present, we could in theory run in parallel one --watch build with --transpileOnly, and another with --noEmit (to just report type errors) to get the best of both worlds. But even better would be a --watch-fast option which does both together (produce the transpiled output quickly, then run the type-checking to report any errors).