Typescript: "Build mode" for TypeScript

[RyanCavanaugh]

This is a follow-up to #3469. Most of this work is already done at https://github.com/RyanCavanaugh/tsbuild and just needs to be ported into the TypeScript code base.

"Build Mode" for TypeScript

TypeScript currently (soon...) supports project references. However, if an upstream project is unbuilt, the developer will see an error telling them to build that project first.

There is a clear need for a build-scheduling mode for TypeScript that intelligently builds upstream projects.

In build mode (tsc -b), tsc will determine which other builds need to happen, order them correctly, and perform that build.

One way to think of it:

"tsc -p" : "tsc -b" :: "csc" : "msbuild"
 or
"tsc -p" : "tsc -b" :: "gcc" : "make" 

Terminology

• Project: A project is a single compilation unit, i.e. result of TypeScript's internal createProgram call, originating from a tsconfig file, with a defined set of inputs and outputs
• Upstream/downstream: A project depends on upstream projects; projects which depend on it are downstream

Commandline Options

The following options are supported under -b:

• --clean: Remove all output files from output folders and exit
• --force: Force all projects to be rebuilt, even if they appear to be up-to-date
• --dry: Do nothing, but display which projects would be built
  
  
  
  • Special case: --dry --clean displays which files would be deleted
  
  
• --verbose: Include verbose logging
• (What else do we need?)

The following regular tsc options are available when -b is specified.
Using any other non--b-specific flag is an error.

• --help
• --watch

Input Specification

The -b switch takes any number of arguments:

tsc -b [projectOrGlob1] [projectOrGlob2] [projectOrGlobN...]

Each argument can be:

• A filename pointing to a tsconfig.json file (though it may have a different name, e.g. mybuildcfg.json)
• A directory name, which is assumed to contain a tsconfig.json file
• A glob that matches filenames

What happens next?

• If tsc -b is invoked with zero arguments, then
  
  
  
  • If a tsconfig.json file exists in the current directory, it is the only input
  
  
  • Otherwise, the current directory is recursively enumerated (excluding node_modules) and any tsconfig.json files found are added to the input
  
  
• Otherwise, the input is the specified list of configs or globs

After determining the set of inputs from the commandline, each input is opened and its references are added to the input list (if it wasn't already present). This repeats until no new inputs are found.

If any input cannot be found, an error is issued and nothing happens.

Inputs are then topologically sorted according to their dependencies and built in that order (upstream projects first).

Circular dependencies

Circular dependencies are allowed as long as at least one dependency in the circularity-causing cycle is marked "circular": true.

For the purposes of the topological sorting of inputs, edges which would cause circularity are ignored. Note that this makes the final build order dependent on the initial ordering of inputs, which may be in turn dependent on file system enumeration order. Developers using circular dependencies should observe their build behavior carefully to ensure it's sufficiently deterministic.

Errors

When a project has a configuration, syntactic, or semantic error, no files are emitted.

Sidebar: Why? If an upstream project has errors but emits anyway, a downstream build will see the project as "up to date" and the errors will be hidden until a clean build or edit to the upstream project occurs. This is potentially fixable; we could write out the errors to disk and re-issue them on subsequent builds. It's not clear if this is desirable or not.

Why have two modes (-p and -b) ?

Users running tsc -p may sometimes see errors that basically amount to "You should run tsc -b instead". So why not just make -p and -b the same?

The reason is to prevent an O(n^2) increase in build I/O for developers writing large solutions using msbuild or similar tools. If msbuild invokes tsc -p on each project in a deep graph, then the files of the every project will get checked once for each downstream project.

For example, in a linear graph of 3 projects with 10 files each, tsc -p on each project (in the correct order) will do 10 + 10 + 10 file checks, whereas tsc -b would check 10 + 20 + 30 files.

Faster Rebuilds

During a build, tsc -b will detect if it's writing an output that is (bytewise) identical to an existing disk artifact. This is an unchanged output.

When a downstream project would need to be rebuilt, but only depends on older or unchanged outputs, then it can be fast rebuilt.

A fast rebuild of a module compilation simply increments the timestamps of all the output files to the current time.

A fast rebuild of an outFile compilation with upstream-changed prepends needs to break apart the .js file on disk, use the bundleinfo file to identify the correct offset, and re-output the JS file.

TODO: Consider cutting this feature? If all endpoints have small additions, this isn't a big perf hit

---

More to come...

[krryan]

Does the final section imply that tsc -p really should only be used by other software tools that are themselves managing dependencies/build orders, and that otherwise users should expect tsc -b to be how they should always use the compiler?

[RyanCavanaugh]

and that otherwise users should expect tsc -b to be how they should always use the compiler?

If they're using project references, yes.

An open question is which behavior should be invoked if running tsc with no commandline arguments at all.

[RyanCavanaugh]

Analyzed feedback from @chriseppstein who took an intrepid early shot at this https://github.com/linkedin/opticss/blob/ts2.9_use_project_references/project_references_feedback.md

Caveats & Setup Info & Unix Compatability Issues - legit pain points that will thankfully be moot once this functionality is integrated into tsc

*.d.ts files ambiently parsed? / Incompatibility with forceConsistentCasingInFileNames option - big bug in forceConsistentCasingInFileNames as we're incorrectly comparing the unmapped reference path to the mapped one

Path problems from transitive dependencies? - will need to clone this one to better understand it

tsconfig.json - need to add references to the schema

Monorepo semantics are not supported - What I'd like to have is a way to specify that a reference to a module is publicly addressable as an abstract identifier at runtime. - this is key. There should be a way to name modules via their nonrelative root name and have that be a mapping to its outDir.

[DanielRosenwasser]

Notes from today's disussion

tsb scenarios (building composite projects)

• Originally thinking running
• tsc [-b|--build] [--?] [...]
• Desirable to add tsc -b **/tsconfig.json
  
  
  
  • People've wanted this for a while for files in general.
  
  
  • Can build in command-line globbing support.
  
  
  • It's a separate convo though.
  
  
• What's tsc -b ** do?
  
  
  
  • Probably equivalent to **/tsconfig.json
  
  
• But if you let people do globs, then you have no way of reasoning about the complete context.
  
  
  
  • Cut globs, use an explicit file list.
  
  
• tsc -b with 0 arguments?
  
  
  
  • same as tsc -b .
  
  
• Does -b need to be the first argument?
  
  
  
  • Well, you can't do tsc -b false, so it's truly special.
  
  
  • So yes, it has to come first.
  
  
• Flags
  
  
  
  • --dry
    
    
    
    
    
    • Build without output (e.g. "show me what you would do")
    
    
    *
    
    
  
  
  • --force
  
  
  • --verbose
  
  
  • --clean?
    
    
    
    
    
    • How can you do this if your input files change in between a build?
    
    
    
    • I guess you can't!
    
    
    
    • That's fine though.
    
    
    
  
  
  • What about the SVG?
  
  
  • --watch
  
  
• Combinations of flags?
  
  
  
  • dry/clean: here's the file I would delete
  
  
  • dry/force?
    
    
    
    
    
    • Probably makes sense.
    
    
    
  
  
  • clean/watch
  
  
  • dry probably doesn't make sense with watch
  
  
• Batching/debouncing behavior
  
  
  
  • When a file changes, should you just build everything eagerly?
    
    
    
    
    
    • Yes
    
    
    
  
  
  • Wait, eagerly as opposed to what?
    
    
    
    
    
    • Well, things might've changed in the time it took to compile.
    
    
    
  
  
  • We debounce file changes.
  
  
  • But you really want to be able to say "once I've built this, double-check whether things are up-to-date?"
  
  
  • You really want something more granular
    
    
    
    
    
    • We have a builder API which has file-by-file granularity for checking.
    
    
    
  
  
  • So we'll do that next iteration :D
  
  
• If A and B are unrelated, A has an error, should we build B?
  
  
  
  • Yes, it's arbitrary; just build all projects that aren't affected.
  
  
• What timestamp do we use to update output files?
  
  
  
  • Just use Date.now().
  
  
• Why doesn't tsc always imply -b in a project?
  
  
  
  • MSBuild needs this behavior.
  
  

[mihailik]

Why doesn't tsc always imply -b in a project?
MSBuild needs this behavior.

How about shifting MSBuild to use another msbuild-specific option like --project-no-ref or something?

It sounds like -b behaviour should be the common sense default for everything except intrusive complex automation tools.