Typescript: tsc randomly produces no output at all and then on next invocation builds fine

[henhal]

TypeScript Version: 3.9.7

Search Terms:

tsc no output
tsc does nothing
tsc randomly skips all input

Code

// A *self-contained* demonstration of the problem follows...
// Test this by running `tsc` on the command-line, rather than through another build tool such as Gulp, Webpack, etc.

{
  "compilerOptions": {
    /* Basic Options */
    "incremental": true,                      /* Enable incremental compilation */
    "target": "es2019",                       /* Specify ECMAScript target version: 'ES3' (default), 'ES5', 'ES2015', 'ES2016', 'ES2017', 'ES2018', 'ES2019', 'ES2020', or 'ESNEXT'. */
    "module": "commonjs",                     /* Specify module code generation: 'none', 'commonjs', 'amd', 'system', 'umd', 'es2015', 'es2020', or 'ESNext'. */
    "lib": ["es2019", "es2020.bigint", "es2020.string", "es2020.symbol.wellknown"], /* Specify library files to be included in the compilation. */
    "allowJs": true,                          /* Allow javascript files to be compiled. */
    // "checkJs": true,                       /* Report errors in .js files. */
    // "jsx": "preserve",                     /* Specify JSX code generation: 'preserve', 'react-native', or 'react'. */
    // "declaration": true,                   /* Generates corresponding '.d.ts' file. */
    // "declarationMap": true,                /* Generates a sourcemap for each corresponding '.d.ts' file. */
    "sourceMap": true,                        /* Generates corresponding '.map' file. */
    // "outFile": "./",                       /* Concatenate and emit output to single file. */
    "outDir": "build",                        /* Redirect output structure to the directory. */
    "rootDir": "./src",                           /* Specify the root directory of input files. Use to control the output directory structure with --outDir. */
    // "composite": true,                     /* Enable project compilation */
    // "tsBuildInfoFile": "./",               /* Specify file to store incremental compilation information */
    // "removeComments": true,                /* Do not emit comments to output. */
    // "noEmit": true,                        /* Do not emit outputs. */
    // "importHelpers": true,                 /* Import emit helpers from 'tslib'. */
    // "downlevelIteration": true,            /* Provide full support for iterables in 'for-of', spread, and destructuring when targeting 'ES5' or 'ES3'. */
    // "isolatedModules": true,               /* Transpile each file as a separate module (similar to 'ts.transpileModule'). */

    /* Strict Type-Checking Options */
    "strict": true,                           /* Enable all strict type-checking options. */
    "noImplicitAny": true,                    /* Raise error on expressions and declarations with an implied 'any' type. */
    // "strictNullChecks": true,              /* Enable strict null checks. */
    // "strictFunctionTypes": true,           /* Enable strict checking of function types. */
    // "strictBindCallApply": true,           /* Enable strict 'bind', 'call', and 'apply' methods on functions. */
    // "strictPropertyInitialization": true,  /* Enable strict checking of property initialization in classes. */
    // "noImplicitThis": true,                /* Raise error on 'this' expressions with an implied 'any' type. */
    "alwaysStrict": true,                  /* Parse in strict mode and emit "use strict" for each source file. */

    /* Additional Checks */
    // "noUnusedLocals": true,                /* Report errors on unused locals. */
    // "noUnusedParameters": true,            /* Report errors on unused parameters. */
    // "noImplicitReturns": true,             /* Report error when not all code paths in function return a value. */
    // "noFallthroughCasesInSwitch": true,    /* Report errors for fallthrough cases in switch statement. */

    /* Module Resolution Options */
    // "moduleResolution": "node",            /* Specify module resolution strategy: 'node' (Node.js) or 'classic' (TypeScript pre-1.6). */
    "baseUrl": "./",                          /* Base directory to resolve non-absolute module names. */
    "paths": {
      "*": [
        "*",
        "../../layers/foo-layer/nodejs/node_modules/*"
      ]
    },                                        /* A series of entries which re-map imports to lookup locations relative to the 'baseUrl'. */
    // "rootDirs": [],                        /* List of root folders whose combined content represents the structure of the project at runtime. */
    // "typeRoots": [],                       /* List of folders to include type definitions from. */
    // "types": [],                           /* Type declaration files to be included in compilation. */
    // "allowSyntheticDefaultImports": true,  /* Allow default imports from modules with no default export. This does not affect code emit, just typechecking. */
    "esModuleInterop": true,                  /* Enables emit interoperability between CommonJS and ES Modules via creation of namespace objects for all imports. Implies 'allowSyntheticDefaultImports'. */
    // "preserveSymlinks": true,              /* Do not resolve the real path of symlinks. */
    // "allowUmdGlobalAccess": true,          /* Allow accessing UMD globals from modules. */

    /* Source Map Options */
    // "sourceRoot": "",                      /* Specify the location where debugger should locate TypeScript files instead of source locations. */
    // "mapRoot": "",                         /* Specify the location where debugger should locate map files instead of generated locations. */
    // "inlineSourceMap": true,               /* Emit a single file with source maps instead of having a separate file. */
    // "inlineSources": true,                 /* Emit the source alongside the sourcemaps within a single file; requires '--inlineSourceMap' or '--sourceMap' to be set. */

    /* Experimental Options */
    // "experimentalDecorators": true,        /* Enables experimental support for ES7 decorators. */
    // "emitDecoratorMetadata": true,         /* Enables experimental support for emitting type metadata for decorators. */

    /* Advanced Options */
    "resolveJsonModule": true,                /* Include modules imported with '.json' extension */
    "forceConsistentCasingInFileNames": true  /* Disallow inconsistently-cased references to the same file. */
  },
  "include": [
    "./src"
  ],
  "exclude": [
    "./test/**",
    "./tools/**"
  ]
}

Expected behavior:

tsc compiles all *.ts files under src and places them under build

Actual behavior:

tsc build successfully sometimes, and randomly (half the time-ish) does nothing, doesn't even create the build directory.

This has been driving me mad since about April. I today updated to latest 3.9.5 and the problem remains.
It seems to only ever happen when using outDir and rootDir, but I cannot for my life find anything else in common.

I can run

$ rm -rf build && tsc --extendedDiagnostics && find build
Files:                         694
Lines:                      751494
Nodes:                     2237525
Identifiers:                825626
Symbols:                    478276
Types:                          76
Instantiations:                  0
Memory used:               633460K
Assignability cache size:        0
Identity cache size:             0
Subtype cache size:              0
Strict subtype cache size:       0
I/O Read time:               0.13s
Parse time:                  4.37s
ResolveTypeReference time:   0.03s
ResolveModule time:          0.58s
Program time:                5.43s
Bind time:                   1.77s
Total time:                  7.20s
find: ‘build’: No such file or directory

$ rm -rf build && tsc --extendedDiagnostics && find build
Files:                         694
Lines:                      751494
Nodes:                     2237525
Identifiers:                825626
Symbols:                    478276
Types:                          76
Instantiations:                  0
Memory used:               632660K
Assignability cache size:        0
Identity cache size:             0
Subtype cache size:              0
Strict subtype cache size:       0
I/O Read time:               0.12s
Parse time:                  4.32s
ResolveTypeReference time:   0.03s
ResolveModule time:          0.55s
Program time:                5.36s
Bind time:                   1.76s
Total time:                  7.13s
find: ‘build’: No such file or directory

# now make some change to tsconfig.json such as change src to src/foo.ts, run tsc 
# (compiles the single file fine), 
# then change back to src again, and NOW suddenly what failed repeatedly before works:

$ rm -rf build && tsc --extendedDiagnostics && find build
Files:                         694
Lines:                      751494
Nodes:                     2237525
Identifiers:                825626
Symbols:                    555209
Types:                      178638
Instantiations:              57946
Memory used:               809918K
Assignability cache size:     9078
Identity cache size:            75
Subtype cache size:             64
Strict subtype cache size:       3
I/O Read time:               0.14s
Parse time:                  4.47s
ResolveTypeReference time:   0.03s
ResolveModule time:          0.58s
Program time:                5.52s
Bind time:                   1.76s
Check time:                  6.17s
transformTime time:          0.12s
commentTime time:            0.03s
printTime time:              0.38s
Emit time:                   0.39s
Source Map time:             0.02s
I/O Write time:              0.00s
Total time:                 13.84s
build
build/index.js.map
build/utils
build/utils/jose.js
build/utils/dates.js.map
build/utils/jose.js.map
build/utils/dates.js
build/index.js

Usually when it's failed once, it's kind of stuck until I make some irrelevant change to tsconfig or something (like fiddling with the exclude patterns, building once, changing back to how it was when it failed a second ago, now succeeds, etc etc).

I realize this is not a self-contained example since I refer to e.g. an external node_modules which is part of my AWS lambda layer, but that also seems irrelevant. My project is organized with all source files under src, tests under test and build being the output directory.

If this is due to my machine getting out of memory or something without any noticeable output, I don't know.
I'm basically looking for some direction or help how to debug this further. As you can see, when it succeeds, the first few lines of the diagnostics are identical, but then the symbol count goes up and types go up _drastically_ when it works.

Playground Link:

Related Issues:

[RyanCavanaugh]

Never heard of anything like this! I would try running node --inspect-brk path_to_tsc.js --build instead of tsc --build and seeing if you see any unhandled exceptions.

[henhal]

@RyanCavanaugh Thanks. Tried that, but due to an exception on every failed attempt to find a module in one of many locations (handleErrorFromBinding), I had to step past _hundreds_ of ENOENT:s. This made it a bit cumbersome, but I could not see any other exception except one instance of this:

Could not resolve JS module './compiler-debug' starting at '[... ] /node_modules/typescript/lib

Unfortunately, when I try to dumb this down to a simple reproducible example it stops happening.

[denghongcai]

I ran into the same situation.

OS：Darwin Kernel Version 19.6.0
Node: v12.16.1

tsc run with exit code 0.

and run fswatch get no file create/modify operation at all.

there is no unhandled exception using inspector.

[denghongcai]

@henhal and when it not work, i get exact the same types count with you(76).

Files:                         376
Lines:                       85202
Nodes:                      365091
Identifiers:                133358
Symbols:                     77244
Types:                          76
Instantiations:                  0
Memory used:               117796K
Assignability cache size:        0
Identity cache size:             0
Subtype cache size:              0
Strict subtype cache size:       0
I/O Read time:               0.14s
Parse time:                  0.66s
ResolveTypeReference time:   0.02s
ResolveModule time:          0.30s
Program time:                1.22s
Bind time:                   0.53s
Total time:                  1.74s

[henhal]

@denghongcai This is very interesting. Unfortunately I've not been able to get any new leads on when and why this happens, but something is fishy...

[denghongcai]

I found it's related to incremental. when i remove tsconfig.tsbuildinfo, it works!
though dist folder was removed, tsc will think it hit incremental compile?

[henhal]

@denghongcai Ah, this seems feasible. Thanks for finding this!

@RyanCavanaugh Care to comment on whether usign incremental compile and removing the output folder might behave like this? If so, is it indended? For example, how is incremental compile supposed to behave if no source file is changed but for some reason one output file is modified - would it also skip overwriting the modified file since the tsbuildinfo would indicate that nothing is changed? That obviously seems like a more far-fetched situation (and like a more reasonable behaviour from the compiler), but actually removing the output directory seems like something a npm run clean command would do? If it then must also remove tsconfig.tsbuildinfo, so be it, but I'm wondering if you can confirm? And if so, whether even an incremental build could at least stat the output directory to force a full compile in case it doesn't even exist?

[RyanCavanaugh]

We don't do the extra work that would be necessary to detect a "modified" output file that was modified after it was built; it's assumed that you did this on purpose and don't want to pay the extra performance cost of accounting for it. Any 'clean' script should delete the tsbuildinfo file to maintain a correct build.

I guess we could check if the output folder is simply missing entirely, but this is just a very small subset of the ways your build could be sideways if your clean script isn't set up properly. Seems weird to special-case it and make it look like we can detect modified output directories when in fact we don't.

[denghongcai]

I think we did this not on purpose to enable 'incremental' feature, composite: true will force it to enable.

'clean' script should delete tsbuildinfo or tsc detect output folder whether missing depends on your choice, just clarify it on document should be ok

[henhal]

Agree with your conclusions @RyanCavanaugh. This makes sense. As for clean scripts etc, I guess one good option would be to put the .buildinfo file in my build directory.

Thanks. From my perspective this can be closed.