Typescript: Support "medium-sized" projects

[jonathandturner]

Listening to @nycdotnet has me fired up to tackle this one. Thanks, Steve. (btw, you can check out his good interview here: http://www.dotnetrocks.com/default.aspx?showNum=1149)

The proposal here was first started in times prehistoric (even before #11), when dinosaurs walked the scorched earth. While nothing in this proposal is novel, per se, I believe it's high time we tackled the issue. Steve's own proposal is #3394.

The Problem

Currently, in TypeScript, it's rather easy to start and get going, and we're making it easier with each day (with the help of things like #2338 and the work on System.js). This is wonderful. But there is a bit of a hurdle as project size grows. We currently have a mental model that goes something like this:

• Small-sized projects: use tsconfig.json, keep most of your source in the current directory
• Large-sized projects: use custom builds, put source where you need it

For small-sized projects, tsconfig.json gives you an easy-to-setup way of getting going with any of the editors in a cross platform way. For large-scale projects, you will likely end up switching to build systems because of the varied requirements of large-scale projects, and the end result will be something that works for your scenarios but is difficult to tool because it's far too difficult to tool the variety of build systems and options.

Steve, in his interview, points out that this isn't quite the right model of the world, and I tend to agree with him. Instead, there are three sizes of project:

• Small-sized projects: use tsconfig.json, keep most of your source in the current directory
• Medium-sized projects: those with standard builds and shared components
• Large-sized projects: use custom builds, put source where you need it

As you scale in size of project, Steve argues, you need to be able to scale through the medium step, or tool support falls off too quickly.

Proposal

To solve this, I propose we support "medium-sized" projects. These projects have standard build steps that could be described in tsconfig.json today, with the exception that the project is built from multiple components. The hypothesis here is that there are quite a number of projects at this level that could be well-served by this support.

Goals

Provide an easy-to-use experience for developers creating "medium-sized" projects for both command-line compilation and when working with these projects in an IDE.

Non-goals

This proposal does _not_ include optional compilation, or any steps outside of what the compiler handles today. This proposal also does not cover bundling or packaging, which will be handled in a separate proposal. In short, as mentioned in the name, this proposal covers only the 'medium-sized' projects and not the needs of those at large scales.

Design

To support medium-sized projects, we focus on the use case of one tsconfig.json referencing another.

Example tsconfig.json of today:

{
    "compilerOptions": {
        "module": "commonjs",
        "noImplicitAny": true,
        "sourceMap": true
    },
    "files": [
        "core.ts",
        "sys.ts"
    ]
}

Proposed tsconfig.json 'dependencies' section:

{
    "compilerOptions": {
        "module": "commonjs",
        "noImplicitAny": true,
        "sourceMap": true
    },
    "dependencies": [
        "../common", 
        "../util"
    ],
    "files": [
        "core.ts",
        "sys.ts"
    ]
}

Dependencies point to either:

• A directory, where a tsconfig.json can be found
• A tsconfig.json directly

Dependencies are hierarchical. To edit the full project, you need to open the correct directory that contains the root tsconfig.json. This implies that dependencies can't be cyclic. While it may be possible to handle cyclic dependencies in some cases, other cases, namely those with types that have circular dependencies, it may not be possible to do a full resolution.

How it works

Dependencies are built first, in the order they are listed in the 'dependencies' section. If a dependency fails to build, the compiler will exit with an error and not continue to build the rest of the project.

As each dependency completes, a '.d.ts' file representing the outputs is made available to the current build. Once all dependencies complete, the current project is built.

If the user specifies a subdirectory as a dependency, and also implies its compilation by not providing a 'files' section, the dependency is compiled during dependency compilation and is also removed from the compilation of the current project.

The language service can see into each dependency. Because each dependency will be driven off its own tsconfig.json, this may mean that multiple language service instances would need to be created. The end result would be a coordinated language service that was capable of refactoring, code navigation, find all references, etc across dependencies.

Limitations

Adding a directory as a dependency that has no tsconfig.json is considered an error.

Outputs of dependencies are assumed to be self-contained and separate from the current project. This implies that you can't concatenate the output .js of a dependency with the current project via tsconfig.json. External tools, of course, can provide this functionality.

As mentioned earlier, circular dependencies are considered an error. In the simple case:

A - B
\ C

A is the 'current project' and depends on two dependencies: B and C. If B and C do not themselves have dependencies, this case is trivial. If C depends on B, B is made available to C. This is not considered to be circular. If, however, B depends on A, this is considered circular and would be an error.

If, in the example, B depends on C and C is self-contained, this would not be considered a cycle. In this case, the compilation order would be C, B, A, which follows the logic we have for ///ref.

Optional optimizations/improvements

If a dependency does not to be rebuilt, then its build step is skipped and the '.d.ts' representation from the previous build is reused. This could be extended to handle if the compilation of a dependencies has built dependencies that will show up later in the 'dependencies' list of the current project (as happened in the example given in the Limitations section).

Rather than treating directories passed as dependencies that do not have a tsconfig.json as error cases, we could optionally apply default 'files' and the settings of the current project to that dependency.

[nycdotnet]

Oh my goodness!

[unional]

:+1:

[kungfusheep]

Yep! This makes perfect sense for the use-cases you've provided. Providing the tools we use with a better view on how our code is intended to be consumed is definitely the right thing to do. Every time I F12 into a .d.ts file by mistake I feel like strangling a kitten!

[nycdotnet]

Jonathan,

Thank you so much for the kind feedback and for deciding to take this on. TypeScript is such a great tool and this functionality will help many people who want to componentize their medium-sized codebases that can't justify the inefficiency mandated by enormous projects that rely on strict division of concerns (for example the Azure portals or project Monacos of the world with > 100kloc and many independent teams). In other words, this will really help "regular people". Also, others have already proposed stuff for this, for example @NoelAbrahams (#2180), and others so I can't claim originality here. This is just something I've been needing for a while.

I think that your proposal is excellent. The only shortcoming that I see versus my proposal (#3394), which I have now closed, is the lack of a fallback mechanism for references.

Consider the following real-world scenario which I detailed here: https://github.com/Microsoft/TypeScript/issues/3394#issuecomment-109359701

I have a TypeScript project grunt-ts that depends on a different project csproj2ts. Hardly anyone who works on grunt-ts will also want to work on csproj2ts as it has a very limited scope of functionality. However, for someone like me - It'd be great to be able to work on both projects simultaneously and do refactoring/go to definition/find all references across them.

When I made my proposal, I suggested that the dependencies object be an object literal with named fallbacks. A version more in keeping with your proposal would be:

"dependencies": {
   "csproj2ts": ["../csproj2ts","node_modules/csproj2ts/csproj2ts.d.ts"],
   "SomeRequiredLibrary": "../SomeRequiredLibraryWithNoFallback"
}

To simplify it to still be an array, I suggest the following alternate implementation of a future hypothetical dependencies section of the grunt-ts tsconfig.json file:

"dependencies": [
   ["../csproj2ts","node_modules/csproj2ts/csproj2ts.d.ts"],
   "../SomeRequiredLibraryWithNoFallback"
]

The resolution rule for each array-type item in dependencies would be: the _first_ item that is found in each is the one that is used, and the rest are ignored. String-type items are handled just as Jonathan's proposal states.

This is a very slightly more complicated-to-implement solution, however it gives the developer (and library authors) _far greater_ flexibility. For developers who do not need to develop on csproj2ts (and therefore do not have a ../csproj2ts/tsconfig.json file), the dependency will just be a definition file that gets added to the compilation context. For developers who _do_ have a ../csproj2ts/tsconfig.json file, the proposal will work exactly as you've described above.

In the above example, the "../SomeRequiredLibraryWithNoFallback" would be required to be there just like in your existing proposal, and its absence would be a compiler error.

Thank you so much for considering this.

[mhegazy]

There are two probelms here that we need to break apart, there is build, and there is language service support.

For Build, I do not think tsconfig is the right place for this. this is clearly a build system issue. with this proposal the typescript compiler needs to be in the businesses of:

• figuring out dependency
• up-to-date checks fro asserts
• configuration management (release vs. debug)

These are all clearly the responsibility of build systems; these are hard problems and there are already tools that do that, e.g. MSBuild, grunt, gulp, etc..
Once tsconfig and tsc become the build driver you would want it to use all CPUs to build unrelated subtrees, or have post and pre build commands for each project, and possibly build other projects as well. again, i think there are build tools out there that are good at what they do, and no need for us to recreate that.

For Language Service,
I think it is fine for tooling to know about multiple tsconfig files and can infer from that and help you more, but that should not affect your build. i would consider something like:

"files" : [
    "file1.ts",
    {
        "path": "../projectB/out/projectB.d.ts",
         "sourceProject": "../projectB/"
     }
]

Where tsc will only look at "path" but tools can look at other information and try to be as helpful as they can.

I acknowledge the existence of the problem but do not think lumping build and tooling is the correct solution. tsconfig.json should remain a configuration bag (i.e. a json alternative to response files) an not become a build system. one tsconfig.json represents a single tsc invocation. tsc should remain as a single project compiler.

MSBuild projects in VS are an example of using a build system to build IDE features and now ppl are not happy with it because it is too big.

[nycdotnet]

Thanks for your reply, Mohamed. Let me restate to see if I understand:

• You think that the task of coordinating multi-project builds should remain the domain of dedicated build tools.
• You think that there may be something to this suggestion for the TypeScript language service.
• You think that running tsc --project on this tsconfig.json would be the same as running tsc file1.ts ../project/out/project.d.ts. However, opening such a project in VS or another editor with the TypeScript language service would allow "go to definition" to bring the developer to the _actual TypeScript file_ where the feature was defined (rather to the definition in projectB.d.ts)

Do I have that right?

If so, I think this is very fair. In my original proposal (https://github.com/Microsoft/TypeScript/issues/3394), I did say that my idea was incomplete because it didn't include the step of copying the emitted results from where they'd be output in the referenced library to where the referencing library would expect them at runtime. I think you're saying "why go halfway down a road for building when what is really needed is the language service support".

To change the data a bit in your example, would you be willing to support something like this?

"files" : [
    "file1.ts",
    {
        "path": "externalLibraries/projectB.d.ts",
         "sourceProject": "../projectB/"
     }
]

The assumption is that the current project would ship with a definition for projectB that would be used by default, but if the actual source for projectB is available, the actual source would be used instead.

[mhegazy]

@nycdotnet you summed it up right; i would like to create a system that is loosely coupled and allows mixing and matching diffrent build tools with diffrent IDEs but still get a great design time experience.

[nycdotnet]

Sounds great!

[frankwallis]

I agree with @mhegazy and actually I think it is important for TypeScript to stop thinking of itself as a 'compiler' and start thinking of itself as a 'type-checker' and 'transpiler'. Now there is single-file transpilation support I don't see any reason for compiled JavaScript files to be created except for at runtime/bundling. I also don't see why it is necessary to generate the external reference definitions during type-checking when the actual typescript source is available.

File and package resolution is the responsibility of the build system you are using (browserify, systemjs, webpack etc), so for the tooling to work the language service needs to be able to resolve files in the same way as whatever build system/platform you are using. This either means implementing a custom LanguageServicesHost for each build system or providing a tool for each one which generates the correct mapping entries in tsconfig.json. Either of these is acceptable.

@nycdotnet I think your use case for multiple fallback paths would be better handled using npm link ../csproj2ts?

[dbaeumer]

I agree with @mhegazy that we should keep build separate from the compiler/transpiler. I do like to include the tsconfig -> tsconfig dependency in the files section assuming that if the section doesn't list any *.ts files it still scans for them. E.g.

"files" : [
{
"path": "externalLibraries/projectB.d.ts",
"sourceProject": "../projectB/"
}
]

Will still include all ts files in the directory and subdirectory containing the tsconfig.json file.

[mhegazy]

@dbaeumer you then want to have it in a different property, correct? currently, if files is defined, it is always used and we ignore the include *.ts part.

[dbaeumer]

@mhegazy not necessarily a different section although it would make things clearer at the end. All I want to avoid is to be forced to list all files if I use a tsconfig -> tsconfig dependencies. In the above example I would still like to not list any *.ts files to feed them into the compiler.

[billti]

I think this is much needed. I don't think we can avoid the build question however. That doesn't mean we need to implement a build system along with this proposal, but we should have thought through some good guidance that works in a configuration such as that proposed. It will be non-trivial to get right (and we do need to solve it for Visual Studio).

In the above proposal (where both the .d.ts and the source is referenced), does it detect if the .d.ts is out of date (i.e. needs to be rebuilt)? Do operations like Refactor/Rename work across projects (i.e. updates the name in the referenced project source, not just its .d.ts file which will get overwritten next build)? Does GoToDef take me to the original code in the referenced project (not the middle of a giant .d.ts file for the whole project)? These would seem it imply needing to resolve, and in some cases analyze, the source of the referenced projects, in which case is the .d.ts that useful?

[mhegazy]

The general solution, which we have today, you have a .d.ts as a build output of one project, and then referenced as input in the other project. This works fine for build, so no need to change that.

The problem is the editing scenario. you do not want to go through a generated file while editing. My proposed solution is to provide a "hint" of where the generated .d.ts came from. The language service will then not load the .d.ts, and instead load a "project" from the hint path. this way goto def will take you to the implementation file instead of the .d.ts and similarly errors would work without the need of a compilation.

operations like rename, will "propagate" from one project to another, similarly find references, will do.

[jbrantly]

Today it is completely up to the host (the IDE, whatever) to find the tsconfig.json file, although TS then provides APIs to read and parse it. How would you envision this working if there were multiple tsconfig.json located in a hierarchical fashion? Would the host still be responsible for resolving the initial file but not the others or would the host be responsible for resolving all of the tsconfigs?

Seems like there is a tradeoff there between convenience/convention and flexibility.

[spion]

Wouldn't this start with the ability to build d.ts files as described in #2568 (or at least have relative imports)?

[mhegazy]

@spion i am not sure i see the dependency here. you can have multiple outputs of a project, it does not have yo be a single delectation file. the build system should be able to know that and wire them as inputs to dependent projects.

[spion]

@mhegazy Oops, sorry. Looking at the issue again, it appears that this is more related to the language service. I read the following

• Medium-sized projects: those with standard builds and shared components

and automatically assumed its related to better support for npm/browserify (or webpack) build worfklows where parts of the project are external modules.

AFAIK there is no way to generate .d.ts file(s) for external modules yet? If so, the only way a language service could link projects that import external modules would be to have something like this in tsconfig.json :

{ 
  "provides": "external-module-name"
}

which would inform the LS when the projects is referenced in another tsconfig.json

[mhegazy]

AFAIK there is no way to generate .d.ts file(s) for external modules yet?

I do not think this is true. calling tsc --m --d will generate a declaration file that is an external module itself. resolution logic will try to find a .ts and if not then a .d.ts with the same name,

[nycdotnet]

@spion TypeScript can generate d.ts files for external modules as @mhegazy said, but this results in a 1:1 ratio of definitions to source files which is different from how a library's TypeScript definitions are typically consumed. One way to work around this is this TypeStrong library: https://github.com/TypeStrong/dts-bundle

[spion]

@mhegazy sorry, I meant for "ambient external modules" i.e. if I write external-module-name in TypeScript and import one of its classes from another module:

import {MyClass} from 'external-module-name'

there is no way to make tsc generate the appropriate .d.ts file that declares 'external-module-name'

@nycdotnet I'm aware of dts-bundle and dts-generator but still if the language service is to know about the source of my other project, it should also know what module name it provides to be able to track the imports correctly

[llgcode]

What's the status of this feature? it seems this is an important option for medium size projects. How do you configure a project that have source in different folders with a specific "requirejs" config?

[mhegazy]

@llgcode please take a look at https://github.com/Microsoft/TypeScript/issues/5039, this should be available in typescript@next.

[felixfbecker]

I don't really get what's so hard about writing a gulpfile with tasks that compile the sub-projects just how you need it for medium-sized projects. I even do this in small-sized projects. The only reason I use tsconfig.json at all is for VS Code

[llgcode]

First I don't use gulp. Second this can be a big project where you don't want to recompile all every time. But if you have a good solution with gulp let me know how to do this.

[felixfbecker]

@llgcode Well, gulp is a task runner. You write a gulpfile.js where you define as many tasks as you like with gulp.task(). Inside your task, you can take a stream of input files with gulp.src() and then .pipe() them through a pipeline of transformations, like compilation, concatenation, minification, sourcemaps, copying assets ... You can do anything that is possible with Node and NPM modules.
If you must compile multiple projects, just define a task that does this. If you want to use multiple tsconfig.json, gulp-typescript has support for that, or you could just read the json files. Incremental builds are possible as well. I don't know how your project is structured, if you have them in different repos and use submodules, or whatever. But gulp is 100% flexible.

[llgcode]

Ok thanks seems to be a great tool. if I have require with mapping like require("mylibs/lib") and my files are for example in a folder project/src/lib.js then completion will not work in atom and I don't know how typescript or gulp will solve the mapping/config done with "mylibs" and a local path. so I think this new option paths in #5039 is a good solution for this problem.

[felixfbecker]

@llgcode Well with gulp you can get all the files (including .d.ts files) with globs, see https://www.npmjs.com/package/gulp-typescript#resolving-files.

I just think TypeScript is trying to do to much here, it is a transpiler and not a build tool. Managing "dependencies" like mentioned in the proposal is really the task of package managers or version control systems and wiring them together is the task of build tools.

[llgcode]

@felixfbecker I not agree. Every compiler (with type checking) that I know has an option like this. for example:
gcc -> include files
java -> classpath & sourcepath
go -> GOPATH
python -> PYTHONPATH
the compiler / transpiler need to know what source files that need to be transpiled what source files is just include/lib files.
Build tool like gulp is needed to know what to do when a file change.

[unional]

Agree wtih @llgcode . Also, beside exposing as a compiler, TypeScript also expose as a language service, which provide syntax highlight (detection actually) and completion functionality to IDE. And THAT also need to walk the dependency tree.

[felixfbecker]

@llgcode @unional Valid points. One thing that may also help is to make the files property in tsconfig.json accept globs so you can define all the files in all the folders you want to include. But I see where you are coming from and why one may want multiple tsconfig.json for larger projects.

[spion]

AFAIK for CommonJS this is already supported via node_modules and npm link ../path/to/other-project

[EricABC]

npm link doesn't work as soon as you start reusing libraries across projects. If you use a common library between two separate projects of your own, (taking rxjs as an example) typescript will tell you that 'Observable is not assignable to Observable'. That's because the include paths are following symlink folders to two different node_modules folders and despite being the same library. Workarounds result in building gulp tasks or local/private npm repos, basically back to the large project option.

[spion]

@EricABC thats probably because they are using ambient external module declarations, in which case they should also include definitions for the newly supported node_modules based .d.ts files. Other than that, there shouldn't be a problem as TS types are only structurally checked, so it doesn't matter if they come from a different modules or have a different names as long as structures match.

[EricABC]

Thanks @spion, just assumed it was file-based, looks like your going to save me from some self-afflicted pain.

[unional]

One thing that may also help is to make the files property in tsconfig.json accept globs...

There is a include property in discussion

[tkrotoff]

Questions and remarks:

• dependencies should allow for full tsconfig.json path because tsc allows it
• Why introduce a new keyword (dependencies) when files already exist and is fine?
  Example:

{
    "compilerOptions": {
        // ...
    },
    "files": [
        "../common/tsconfig.json", // <== takes the `files` part of the tsconfig.json
        "../common/tsconfig.util.json", // <==
        "core.ts",
        "sys.ts"
    ]
}

• What happens if the dependency tsconfig.json also specifies compilerOptions? Makes sense to only use the files part from the dependency

Let's go further/wild :-) and possibly allow (in the future) compilerOptions, exclude... to reference another tsconfig.json:

// File app/tsconfig.json
{
    "compilerOptions": "../common/tsconfig.compilerOptions.json",
    "files": [
        "../common/tsconfig.json",
        "../common/tsconfig.util.json",
        "core.ts",
        "sys.ts"
    ],
    "exclude": "../common/exclude.json"
}

// File ../common/tsconfig.compilerOptions.json
{
    "compilerOptions": {
        "module": "commonjs",
        "noImplicitAny": true,
        "sourceMap": true
    }
}

// File ../common/exclude.json
{
    "exclude": [
        "node_modules",
        "wwwroot"
    ]
}

// File ../common/tsconfig.util.json
{
    "files": [
        "foo.ts",
        "bar.ts"
    ]
}

You got the logic: files, compilerOptions, exclude... can reference other tsconfig.json files and it will only "take" the matching keyword part from the other tsconfig.json file => easy and scalable. You can thus split a tsconfig.json in multiple files if you want and re-use them.

[t-beckmann]

Reading part of your discussion, it seams most relevant to get this "language service"/goto definition thing right. JavaScript debuggers use sourceMaps. Now, if tsc generated sourceMap data not just in .js but also in .d.ts files...

Beyond that I really don't see much benefit in triggering builds of child projects from within the tsconfig.json file. If you need this basic sort of build time dependencies a simple shell script would do the job. If you need intelligent incremental building, on the other hand, the approach proposed seems too simple. In many scenarios tsc ends up as being just one build step among others. How odd writing dependencies for tsc in tsconfig.json but some other file for the rest of it? Again, for simple things with tsc being the only build step a shell script would do.

Anyways, how about generating source mapping in .d.ts files just as in .js files?

[wizzard0]

we simply use node modules + npm link, the only thing that does not work is that the moduleResolution: node is not compatible with ES6 modules, which enable inlining / tree-shaking optimizations (see also #11103 )

[joelday]

Not to go off topic, but in some ways, this does seem parallel to the challenges of working with multiple Node package projects locally. I don't think it's as simple as just using "npm link." They might have different build scripts, you'd need to run all of them in the right order, it's harder to do so incrementally, it's harder to use watch mode, it's harder to debug things and it's harder to resolve source maps. Depending on your editor of choice, this might be even more challenging.

Generally I just give up, put it all in one giant project, then split it out into separate packages once they've stabilized, but that only helps because the challenge is being faced less frequently. I find the whole experience really, really annoying. Am I missing something?

So, whatever this winds up being, I just hope I can finally have an elegant solution for this entire development experience.

[wardds]

Just chiming in that this would be great for our day-to-day work!

Due to the nature of web development, we have multiple ts projects with each project containing multiple ts files compiled in to a single js file (--outFile). These are either app-like projects (they do a specific thing or add a specific feature) or lib-like projects (reusable code for the apps). Often we work on multiple of these ts-projects at the same time, enhancing the libraries to facilitate the development on the apps. But I don't think any of my fellow developers have all of our ts-projects on their local environments at any time either.

To improve our workflow, our current options are

• Throw everything into 1 ts project
  
  
  
  • We can reference .ts files directly, which is much nicer than using the d.ts files
  
  
  • But we need an external tool to map and concatenate sets of files, since we need to limit the js files requested over the interwebs while maintaining modularity of the apps (which are feature- or page specific).
  
  
  • As mentioned, not all of these ts-projects are required at all times for everyone so this would add a lot of bloat to the filesystem. Someone working on project X might need projects A, B and D but someone working on Y might need A and C.
  
  
• Keep projects separated (our current situation)
  
  
  
  • We need to reference compiled d.ts files from the other ts-projects as including a .ts file directly would add it to the output. It would be a lot faster if we could just f12 straight into our own source code.
  
  
  • And we need to add tools/scripts to compile multiple of these projects at the same time. Currently we either start tsc -d -w commands from multiple terminals, or fire up a script that does that for all subdirectories where a tsconfig is found.
  
  
  • I understand other tools can help with this (e.g. gulp) or maybe we can figure something out with file globs in the tsconfigs, however that wouldn't help with the first issue of the d.ts files.
  
  

Our project(s) is simply not large enough to keep the libraries' development strictly separated from the apps, but not small enough that we can simply throw everything together. We find ourselves lacking graceful options with typescript.

If dependencies can be that best-of-both-worlds option, that would be amazing; the functionality of tags to all of a dependency's ts files, but compile the output according to that dependency's own tsconfig.

[llgcode]

Is there's any update on this topic. How can we have multiple typescript projects that compile independently each other? How can we describe such a dependency in tsconfig.json?

[gulshan]

This is now included in the roadmap in the future section with the title "Support for project references". So, I guess a .tsconfig file will be able to link another .tsconfig file as a dependency.

[mhegazy]

Is there's any update on this topic. How can we have multiple typescript projects that compile independently each other? How can we describe such a dependency in tsconfig.json?

The build dependency should be encoded in your build system. build systems like gulp, grunt, broccoli, msbuild, basal, etc.. are built to handle such cases.
For type information, the output of one project should include a .d.ts and that should be passed as input to the other.

[azz]

@mhegazy Our project works like this. We have a number of packages in a lerna monorepo, each package has their own dependencies in package.json, and those same dependencies in the "types" property in their tsconfig.json. Each project is compiled with --outFile (it's an older project that hasn't moved to ES modules yet), and the "typings" package.json key points to the bundled .d.ts file.

We use gulp for the bundling/watching.

It works for the most part, but there are a few issues:

• Due to issues like #15488 and #15487, we need to have an explicit link in order for references to work correctly.
• Go-to-definition will take you to a bundled .d.ts file. Ideally this would take you to the source in another project.
• The fastest way to do a full build is lerna run build --sort (effectively tsc in each directory), which has additional overhead because it will spawn one TypeScript compiler process for each package, performing a lot of repeated work.

[lorenzodallavecchia]

I'm keeping a close eye on this issue since we also are in the same situation that others described.
Multiple "projects", each with its tsconfig.json file.

We have the build process working like @mhegazy pointed out: each project emits a .d.ts file and that is used as input for the depending projects.

The real problem is IDE support: when looking for references, they are found only within the scope of a single tsconfig.json. Even worse, the cascading effects of a changed file do not propagate across projects because depending files outside of the tsconfig.json scope are not recompiled. This is very bad for maintenance of our projects and sometimes causes build errors that could have been caught in the IDE.

[RyanCavanaugh]

[nycdotnet]

OH MY GOODNESS

[nycdotnet]

An updated scenario in which I'd love to have this involves React components. We have a components repo that contains JSX modules (atoms, molecules, and organisms) which render UI components appropriate across all applications at our company. This components repo is used by all front-end developers as they work on their individual applications. It would be SO NICE if I could have a TypeScript language service experience that would allow me to be editing my specific-application's UI and "Go to definition" into the common UI components repo. Today we have to bundle these components individually and copy them over. This is the "make your own project plumbing" problem that I would love to see fixed (for which there is a very nice story in the .NET world with projects under a solution).

[RyanCavanaugh]

Project References: Built-in Scalability for TypeScript

Introduction

TypeScript has scaled up to projects of hundreds of thousands of lines, but doesn't natively support this kind of scale as a built-in behavior. Teams have developed workarounds of varying effectiveness and there isn't a standardized way to represent large projects. While we've greatly improved the performance of the typechecker over time, there are still hard limits in terms of how fast TS can reasonably get,
and constraints like 32-bit address space which prevent the language service from scaling up "infinitely" in interactive scenarios.

Intuitively, changing one line of JSX in a front-end component should not require re-typechecking the entire core business logic component of a 500,000 LOC project. The goal of project references is to give developers tools to partition their code into smaller blocks. By enabling tools to operate on smaller chunks of work at a time, we can improve responsiveness and tighten the core development loop.

We believe our current architecture has very few remaining "free lunches" in terms of drastic improvmenets in performance or memory consumption. Instead, this partitioning is an explicit trade-off that increases speed at the expense of some upfront work. Developers will have to spend some time reasoning about the dependency graph of their system, and certain interactive features (e.g. cross-project renames) may be unavailable until we further enhance tooling.

We'll identify key constraints imposed by this system and establish guidelines for project sizing, directory structure, and build patterns.

Scenarios

There are three main scenarios to consider.

Relative Modules

Some projects extensively use relative imports. These imports unambiguously resolve to another file on disk. Paths like ../../core/utils/otherMod would be common to find, though flatter directory structures are usually preferred in these repos.

Example

Here's an example from Khan Academy's perseus project:

Adapted from https://github.com/Khan/perseus/blob/master/src/components/graph.jsx

const Util = require("../util.js");
const GraphUtils = require("../util/graph-utils.js");
const {interactiveSizes} = require("../styles/constants.js");
const SvgImage = require("../components/svg-image.jsx");

Observations

While the directory structure implies a project stucture, it's not necessarily definitive. In the Khan Academy sample above, will can infer that util, styles, and components would probably be their own project. But it's also possible that these directories are quite small and would actually be grouped into one build unit.

Mono-repo

A mono-repo consists of a number of modules that are imported via non-relative paths. Imports from sub-modules (e.g. import * as C from 'core/thing) may be common. Usually, but not always, each root module is actually published on NPM.

Example

Adapted from https://github.com/angular/angular/blob/master/packages/forms/src/validators.ts

import {InjectionToken, ɵisObservable as isObservable, ɵisPromise as isPromise} from '@angular/core';
import {forkJoin} from 'rxjs/observable/forkJoin';
import {map} from 'rxjs/operator/map';
import {AbstractControl, FormControl} from './model';

Observations

The unit of division is not necessarily the leading part of the module name. rxjs, for example, actually compiles its subparts (observable, operator) separately, as does any scoped package (e.g. @angular/core).

Outfile

TypeScript can concatenate its input files into a single output JavaScript file. Reference directives, or file ordering in tsconfig.json, create a deterministic output order for the resulting file. This is rarely used for new projects, but is still prevelant among older codebases (including TypeScript itself).

Example

https://github.com/Microsoft/TypeScript/blob/master/src/compiler/tsc.ts

/// <reference path="program.ts"/>
/// <reference path="watch.ts"/>
/// <reference path="commandLineParser.ts"/>

https://github.com/Microsoft/TypeScript/blob/master/src/harness/unittests/customTransforms.ts

/// <reference path="..\..\compiler\emitter.ts" />
/// <reference path="..\harness.ts" />

Observations

Some solutions using this configuration will be loading each outFile via a separate script tag (or equivalent), but others (e.g. TypeScript itself) require concatenation of the prior files because they're building monolithic outputs.

Project References: A new unit of isolation

Some critical observations from interacting with real projects:

• TypeScript is usually "fast" (< 5-10s) when checking projects under 50,000 LOC of implementation (non-.d.ts) code
• .d.ts files, especially under skipLibCheck, are almost "free" in terms of their typechecking and memory cost
• Almost all software can be subdivided into components smaller than 50,000 LOC
• Almost all large projects already impose some structuring of their files by directory in a way that produces moderately-sized subcomponents
• Most edits occur in leaf-node or near-leaf-node components that do not require re-checking or re-emitting of the entire solution

Putting these together, if it were possible to only ever be typechecking one 50,000 LOC chunk of implementation code at once, there would be almost no "slow" interactions in an interactive scenario, and we'd almost never run out of memory.

We introduce a new concept, a project reference, that declares a new kind of dependency between two TypeScript compilation units where the dependent unit's implementation code is not checked; instead we simply load its .d.ts output from a deterministic location.

Syntax

A new references option (TODO: Bikeshed!) is added to tsconfig.json:

{
  "extends": "../tsproject.json",
  "compilerOptions": {
    "outDir": "../bin",
    "references": [
      { "path": "../otherProject" }
    ]
  }
}

The references array specifies a set of other projects to reference from this project.
Each references object's path points to a tsconfig.json file or a folder containing a tsconfig.json file.
Other options may be added to this object as we discover their needs.

Semantics

Project references change the following behavior:

• When module resolution would resolve to a .ts file in a subdirectory of a project's rootDir, it instead resolves to a .d.ts file in that project's outDir
  
  
  
  • If this resolution fails, we can probably detect it and issue a smarter error, e.g. Referenced project "../otherProject" is not built rather than a simple "file not found"
  
  
• Nothing else (TODO: so far?)

Restrictions for Performance of Referenced Projects

To meaningfully improve build performance, we need to be sure to restrict the behavior of TypeScript when it sees a project reference.

Specifically, the following things should be true:

• Never read or parse the input .ts files of a referenced project
• Only the tsconfig.json of a referenced project should be read from disk
• Up-to-date checking should not require violating the above constraints

To keep these promises, we need to impose some restrictions on projects you reference.

• declaration is automatically set to true. It is an error to try to override this setting
• rootDir defaults to "." (the directory containing the tsconfig.json file), rather than being inferred from the set of input files
• If a files array is provided, it must provide the names of all input files
  
  
  
  • Exception: Files included as part of type references (e.g. those in node_modules/@types) do not need to be specified
  
  
• Any project that is referenced must itself have a references array (which may be empty).

Why "declaration": true ?

Project references improve build speed by using declaration files (.d.ts) in place of their implementation files (.ts).
So, naturally, any referenced project must have the declaration setting on.
This is implied by "project": true

Why alter rootDir ?

The rootDir controls how input files map to output file names. TypeScript's default behavior is to compute the common source directory of the complete graph of input files. For example, the set of input files ["src/a.ts", "src/b.ts"] will produce the output files ["a.js", "b.js"],
but the set of input files ["src/a.ts", "b.ts"] will produce the output files ["src/a.js", "b.js"].

Computing the set of input files requires parsing every root file and all of its references recursively,
which is expensive in a large project. But we can't change this behavior today without breaking existing projects in a bad way, so this change only occurs when the references array is provided.

No Circularity

Naturally, projects may not form a graph with any circularity. (TODO: What problems does this actually cause, other than build orchestration nightmares?) If this occurs, you'll see an error message that indicates the circular path that was formed:

TS6187: Project references may not form a circular graph. Cycle detected:
    C:/github/project-references-demo/core/tsconfig.json ->
    C:/github/project-references-demo/zoo/tsconfig.json ->
    C:/github/project-references-demo/animals/tsconfig.json ->
    C:/github/project-references-demo/core/tsconfig.json

tsbuild

This proposal is intentionally vague on how it would be used in a "real" build system. Very few projects scale past the 50,000 LOC "fast" boundary without introducing something other than tsc for compiling .ts code.

The user scenario of "You can't build foo because bar isn't built yet" is an obvious "Go fetch that" sort of task that a computer should be taking care of, rather than a mental burden on developers.

We expect that tools like gulp, webpack, etc, (or their respective TS plugins) will build in understanding of project references and correctly handle these build dependencies, including up-to-date checking.

To ensure that this is possible, we'll provide a reference implementation for a TypeScript build orchestration tool that demonstrates the following behaviors:

• Fast up-to-date checking
• Project graph ordering
• Build parallelization
• (TODO: others?)

This tool should use only public APIs, and be well-documented to help build tool authors understand the correct way to implement project references.

TODO

Sections to fill in to fully complete this proposal

• How you'd transition an existing project
  
  
  
  • Basically just drop tsconfig.json files in and then add references needed to fix build errors
  
  
• Impact on baseUrl
  
  
  
  • Makes implementation difficult but effectively no end-user impact
  
  
• Brief discussion of nesting projects (TL;DR it needs to be allowed)
• Outline scenario of subdividing a projects that has grown "too large" into smaller projects without breaking consumers
• Figure out the Lerna scenario
  
  
  
  • Available data point (N = 1) says they wouldn't need this because their build is already effectively structured this way
  
  
  • Find more examples or counterexamples to better understand how people do this
  
  
• Do we need a dtsEmitOnly setting for people who are piping their JS through e.g. webpack/babel/rollup?
  
  
  
  • Maybe references + noEmit implies this
  
  

[azz]

Fantastic!

Figure out the Lerna scenario

• Available data point (N = 1) says they wouldn't need this because their build is already effectively structured this way

Does "this" refer to the proposal or the reference build implemention? While you could use lerna to do the build (my team does), it's gnarly and would be much more efficient if TS (or a tool built from this proposal) takes care of itself.

[RyanCavanaugh]

The TODO section is TODOs for the entire proposal

[poelstra]

Nice!

Any project that is referenced must itself have a references array (which may be empty).

Is this really necessary? Wouldn't it be sufficient if such a package has .d.ts files?
(In that case, it may not even be necessary for there being a tsconfig.json, too?)

My use-case: consider a (e.g. third-party) project that does not use outDir, so .ts, .js and .d.ts will be next to each other, and TS will currently try to compile the .ts instead of using the .d.ts.

The reason for not using outDir for me is to more easily allow import "package/subthing"-style imports, which would otherwise have to be e.g. import "package/dist/subthing" with outDir: "dist".
And to be able to use either the NPM package, or its source repository directly (e.g. with npm link).

(It would be helpful if package.json allowed specifying a directory in main, but alas...)

[azz]

Do we need a dtsEmitOnly setting for people who are piping their JS through e.g. webpack/babel/rollup?

Absolutely! This is a big missing piece at the moment. Currently you can get a single d.ts file when using outFile, but when you switch to modules and use a bundler, you lose this. Being able to emit a single d.ts file for the entry point of a module (with export as namespace MyLib) would be amazing. I know external tools can do this but it really would be great if it integrated into the emitter and language services.

[RyanCavanaugh]

Is this really necessary? Wouldn't it be sufficient if such a package has .d.ts files?

We need something in the target tsconfig that tells us where to expect the output files to be. A previous version of this proposal had "You must specify an explicit rootDir" which was rather cumbersome (you had to write "rootDir": "." in every tsconfig). Since we want to flip a variety of behaviors in this world, it made more sense to just say that you get "project" behavior if you have a references array and have that be the thing that's keyed off of, rather than specifying a bunch of flags you'd have to explicitly state.

[michaelaird]

This proposal would line up closely with how we've already structured our TypeScript projects. We've subdivided into smaller units that each have a tsconfig.json and get built independently through gulp. Projects reference each other by referencing the d.ts files.

In an ideal world, the referenced project wouldn't need to be pre-built. ie. TypeScript does a "build" of the referenced project and maintains the "d.ts" equivalent in memory in the language service. this would allow changes made in the "source" project to show up in the "dependent" project without needing a rebuild.

[poelstra]

We need something in the target tsconfig that tells us where to expect the output files to be.

That is only true when outDir is used, isn't it?

As in: if I have a tsconfig that:

• does NOT use outDir (but does have declaration: true, of course), then we don't need rootDir, nor references
• does have outDir, then you would need references and/or rootDir (and declaration: true) to be set

Reason for asking is that I could then enable 'project mode' for any TS package by just referencing it, i.e. it is in my control.

In that case, it would also be good if it also works as soon as it finds the .d.ts file it is looking for (i.e. won't complain if there are no .ts files, or tsconfig files). Because that will enable another case of 'replacing' a NPM version (which may only have .d.ts files) by its source version when necessary.

For example, consider NPM packages MyApp and SomeLib.
SomeLib could have tsconfig: declaration: true.

Repository like:

package.json
tsconfig.json
index.ts
sub.ts

Compiled, this becomes:

package.json
tsconfig.json
index.ts
index.d.ts
index.js
sub.ts
sub.d.ts
sub.js

This structure enables e.g.

// somewhere in MyApp
import something from "SomeLib/sub";

In the published NPM package, I currently always have to strip the .ts files, otherwise all sources will be recompiled by TS if MyApp uses SomeLib:

So, on NPM, this becomes:

package.json
index.d.ts
index.js
sub.d.ts
sub.js

Now, if I put references: ["SomeLib"] in MyApp's tsconfig, it would be good if it works 'as is' for both the NPM version and the source version of SomeLib, i.e. that it won't complain about e.g. missing tsconfig, as long as it does find sub.d.ts in the right place.

---

Related, but different question:

I do now realize, that IF the author of SomeLib puts references in his tsconfig, this would allow to publish NPM packages WITH the .ts files, in the future. But then, I suppose TS would still always recompile these when any dependent package does not explicitly put references: ["SomeLib"] in their tsconfig.

Or is the intention also that references in MyLib will automatically also introduce a 'project boundary' when just import'ing it (i.e. not references'ing it)?

IIRC, one of the initial ideas was that if a module was e.g. located through node_modules, that then .d.ts files would be preferred over .ts files, but this was later changed back, because the heuristic ("through node_modules") was too problematic in general. In could be that having an explicit 'project boundary' would solve this (e.g. a projectRoot: true, instead of or in addition to having references)?

[spion]

For the lerna case, I was hoping for a simpler solution.

1. In the individual package directories, the packages should not know of anything regarding the monorepo structure. The individual tsconfig json files should not contain any references.
   
   
   
   • this allows you to split individual packages into separate repositories and simply have the main repo tooling clone them e.g. ProseMirror: https://github.com/ProseMirror/prosemirror
   
   
2. In the root "workspace" repo (which can contain all the code, but also could simply clone other repos), have the references in its tsconfig.json
   
   
   
   • this is done just so tooling can recognise and follow references all the way to the source instead of going into .d.ts files.
   
   

My whole concern is that once you add a reference using a relative descending "../xx" path to the individual project config files, they are no longer usable as stand-alone modules - they have to be in a specific workspace structure.

Adding the new concept of a "workspace" tsconfig.json solves this problem. That way if you e.g. "git clone" the individual package, installing its dependencies the normal way (e.g. using npm or yarn) should let you work on it separately, since the compiled dependencies would bring in their definition files. If you clone the entire workspace and run the command to bring in all packages, the workspace config will let you navigate through all the sources.

[spion]

Note that a workspace tsconfig.json also aligns perfectly with Yarn's workspace package.json https://yarnpkg.com/lang/en/docs/workspaces/

[spion]

I did a little proof of concept here

https://github.com/spion/typescript-workspace-plugin

Simply add the plugin to all your tsconfig.json files of the individual repos

{
  "plugins": [{"name": "typescript-workspace-plugin"}]
}

Then at the toplevel package.json alongside yarn's "workspaces" entry, add a "workspace-sources" entry:

{
  "workspaces": ["packages/*"],
  "workspace-sources": {
    "*": ["packages/*/src"]
  }
}

The field works exatly like the "paths" field in tsconfig.json but it only affects the language service of the individual projects, pointing them to the package sources. Restores proper "go to definition / type" functionality and similar.

[RyanCavanaugh]

That is only true when outDir is used, isn't it?

Correct. We had hypothesized that almost everyone with a large project is using outDir. I'd be interested to hear about projects that don't

Now, if I put references: ["SomeLib"] in MyApp's tsconfig, it would be good if it works 'as is' for both the NPM version and the source version of SomeLib

Big fan, I like this idea a lot. I need to think about if it's truly required or not.

One caveat here is that I think package authors need to either a) publish both .ts and tsconfig files together in a place where TS finds them, or b) publish neither and only have .d.ts files reachable. In the (a) case we'd follow the project references recursively and the right thing would happen and in (b) we wouldn't spider out to the wrong places.

Or is the intention also that references in MyLib will automatically also introduce a 'project boundary' when just import'ing it (i.e. not references'ing it)?

Talked with @mhegazy and we think there's actually a simple model for how project references behavior: any project with a references array never "sees" a .ts file outside of the project folder - this includes files under excluded directories. This change alone makes the lerna scenario work ("work" meaning "module references always resolve to .d.ts") out of the box, as well as others.

I need to look at the "workspace" model more.

[HolgerJeromin]

That is only true when outDir is used, isn't it?

Correct. We had hypothesized that almost everyone with a large project is using outDir. I'd be interested to hear about projects that don't

We have 67 TS Projects in the visual studio solution which are compiled without outdir and postbuild grunttasks for creating the output directory structure (and uglify and other postprocessing).

Most projects have such a tsconfig.json

 "include": [
    "../baseProj/Lib/jquery.d.ts",
    "../baseProj/baseProj.d.ts"
  ]

[spion]

I took some time to read through the references proposal, and correct - AFAICT lerna and yarn workspace users don't need any of the workspace functionality proposed here:

1. We already have a dependency graph based on package.json so we know the order in which to run the build. In fact, lerna has a generic run command that can run this in order, and its not difficult to write a tool that also adds parallelism where applicable. skipLibCheck should make the performance impact negligible, but I haven't checked it.
2. Lerna and Yarn already create symlinks to the other modules in the appropriate node_modules location. As a result, all dependants are able to follow to the other module's package.json, read the types/typings field and find the referenced module.d.ts type definition file.

What we don't have, and what the plugin I wrote provides, is a way to load all the sources at the same time. When I need to make changes to two or more modules at the same time, I don't want "go to definition" and "go to type definition" to send me to the .d.ts file. I want it to send me to the original source code location, so that perhaps I might edit it. Otherwise, I would just load the individual project directory and the node_modules symlinks created by lerna/yarn would just work.

[octogonz]

Same thing for us. Instead of Lerna, we use Rush to calculate our dependency graph, but the effect is the same. When we build projects, tsc is just one of many tasks that need to be run. Our compiler options are calculated by a larger build system, and we're moving to a model where tsconfig.json is not an input file, but rather a generated output (mainly for the benefit of VS Code).

What we don't have, and what the plugin I wrote provides, is a way to load all the sources at the same time. When I need to make changes to two or more modules at the same time, I don't want "go to definition" and "go to type definition" to send me to the .d.ts file. I want it to send me to the original source code location, so that perhaps I might edit it.

+1 this would be awesome.

If we're dreaming about better multi-project support, my first request would be a compiler service, something like how VS Code IntelliSense works. Our builds would be significantly faster if Rush could invoke tsc 100 times without having to spin up the compiler engine 100 times. The compiler is one of our most expensive build steps. In a monorepo, build times are really important.

@iclanton @nickpape-msft @qz2017

[SalvatorePreviti]

Yes, please!

[michaelaird]

I think one of the most useful outcomes of the project system would be if
‘go to definition’ went to the source file instead of the d.ts file and
‘find all references’ searched down through the project reference tree.

Presumably this would also unlock ‘global rename’ type refactorings.

On Thu, Nov 9, 2017 at 9:30 PM Salvatore Previti [e-mail]
wrote:

Yes, please!

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/Microsoft/TypeScript/issues/3469#issuecomment-343356868,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AANX6d19Zz7TCd_GsP7Kzb-9XJAisG6Hks5s07VXgaJpZM4E-oPT
.

[poelstra]

Talked with @mhegazy and we think there's actually a simple model for how project references behavior: any project with a references array never "sees" a .ts file outside of the project folder - this includes files under excluded directories.

Nice, and in that case, why would one have to specify any specific references at all?
It appears that it would be enough to have a flag (like projectRoot: true).
For example, what would the difference then be between references: ["foo"] and just references: []?
Because if I import "foo" or import "bar", both will then ignore any .ts files.

So, in that case, the proposal becomes:

Given this tsconfig.json (TODO bikeshed on projectRoot):

{
  "extends": "../tsproject.json",
  "compilerOptions": {
    "projectRoot": true
   }
}

When tsc then needs to resolve something outside of the project folder (including files under excluded directories), it will only look at .d.ts files (alternatively, it may just _prefer_ .d.ts files, and fall back to the tsconfig and/or .ts if it only sees that).

This makes resolution during compilation fast and simple.
It works for monorepo references (i.e. import "../foo") and package-based references (i.e. import "foo").
It works for NPM packages and their source code representation.
And it removes the need for resolving tsconfig.json machinery during compile, although the error message if it can't find the .d.ts will be less helpful.

It sounds a bit too good to be true, if it's indeed this simple, so I'm probably overlooking something important :)

---

As others also point out, it is still very important that 'IntelliSense' does keep working with the .ts files.

So, if a request for 'go to definition', 'find references' etc is then made, it should use some reverse mapping to locate the corresponding .ts file from the .d.ts file it used so far.

This mapping could be done using e.g.:

• using an embedded comment string like //# sourceURL = ../src/foo.ts in the .d.ts
  
  
  
  • a more elaborate sourcemap could be used to map from a 'rolled up' .d.ts back to the originating .ts
  
  
• resolving the .js file, and using its sourcemap to locate the `.ts

This does introduce the topic of rebuilding the .d.ts when that .ts is changed, but I'm not sure it should be solved by this proposal. For example, it already is the case today that there needs to be some process to rebuild the .js files, even from the root project itself. So I suppose it would be safe to assume that if that is present, there will also be something to rebuild the dependencies. Could be a bunch of parallel tsc's for each package, could be tsbuild, could be a smart IDE with compileOnSave-like behavior, etc.

[spion]

@pgonzal @michaelaird https://github.com/spion/typescript-workspace-plugin does only that - it restores go to definition and find all references for a multi-tsconfig yarn/lerna/etc workspace project.

[octogonz]

Interesting... I wonder if we could make this work with Rush. We'll take a look.

[spion]

FYI another simple tool we built as part of this effort was wsrun

Similar to lerna run, it runs a command for all the packages in the workspace.

Since typescript dependencies need to be compiled in-order, wsrun is capable of running a package command in a topological ordering based on their package.json dependencies. It supports parallelism during the build. Its not optimally parallel, but that can be improved later.

[cherrydev]

Just a note, oao is another monorepo tool for yarn. It recently added support for 'topological' ordering of commands as well.

[borekb]

I'd just like to post the word "refactoring" here as it's an important goal for us, although probably tangential to the current proposal (https://github.com/Microsoft/TypeScript/issues/3469#issuecomment-341317069) and not mentioned in this issue very often.

Our use case is a monorepo with several TS projects, it can basically be reduced to a common-library plus a couple of apps: app1 and app2. When working in the IDE, those should be treated as a single project, e.g., rename refactoring should work across all three modules, but app1 and app2 are also two separate build targets. While I agree that build is generally a separate concern, the truth is that our apps are quite small and doing something like cd app1 && tsc would be perfectly fine for us.

If TypeScript comes up with a good way to support this, it would be awesome.

[asvetliakov]

For monorepo cross-project refactoring/references i found this setup working for me if you're working in vscode:

root tsconfig:

"compilerOptions": {
   "baseUrl": ".",
   // global types are different per project
   "types": [],
   "paths": {
      "lib": ["packages/lib/src"],
      "xyz1": ["packages/xyz1/src"],
      "xyz2": ["packages/xyz2/src"],
   }
},
"include": ["./stub.ts"], // empty file with export {} to stop vscode complaining about no input files
"exclude": ["node_modules"]

packages/xyz1/tsconfig.json

{
  "extends": "../../tsconfig",
   "compilerOptions": {
      "types": ["node"],
   },
   "include": ["src/**/*"]
}

packages/xyz2/tsconfig.json

{
  "extends": "../../tsconfig",
   "compilerOptions": {
      "types": ["webpack-env"]
   },
  "include": ["src/**/*"]
} 

packages/lib/tsconfig.json

{
   "extends": "../../tsconfig",
    "compilerOptions": { ... },
    "include": ["src/**/*"],
    // special file to load referenced projects when inside in lib package, without it they won't be 
    // visible until you open some file in these projects 
    "files": ["./references.ts"],
}

packages/lib/tsconfig-build.json

{
   "extends": "./tsconfig",
    // exclude referenced projects when building
   "files": []
}

packages/lib/references.ts

import "xyz1";
import "xyz2";

export {};

You need correct main property in package's package.json and may be types even for no-lib packages, for example:

  "main": "src/main.tsx",
  "types": "src/main.tsx",

This way refactoring/renaming something in lib will also refactor references in xyz1 and xyz2. Also projects could have different globals/target libs by this way.

[SalathielGenese]

At Gradle, they simply call it - composite build.

By the way, can one point me were to start if I want to contribute to TypeScript compiler? I've cloned the repo and it is not a small deal (I use to read source : angular, ionic, express when I can't get it from the docs or am away from internet...) I really need help from one to point me the path to go, please.

Thanks!
Bright future for TypeScript.

[RyanCavanaugh]

I have a prototype I'd like people to try if they're using relative module paths. There's a sample repo up at https://github.com/RyanCavanaugh/project-references-demo that outlines the basic scenario and shows how it works.

For trying locally:

git clone https://github.com/RyanCavanaugh/TypeScript
git checkout pr-lkg
npm install
npm run build
npm link

Then in your other folder:

npm link typescript

I'll keep the pr-lkg tag pointing at the latest working commit as things change. Next up on my todos:

• [x] Add better error messages when dependent projects aren't built
• [ ] Apply the redirecting behavior to nonrelative module paths
• [ ] Expose an API for build tools to consume

[borekb]

@RyanCavanaugh I cannot really build it due to errors like missing del module or Error: Cannot find module 'C:\github\TypeScript\built\local\tsc.js' (maybe your local path?) but overall, the structure looks great.

Is this tsc-only or will it also account for project-wide refactorings in VSCode with the language server?

[KostyaTretyak]

...also pr-lkg tag is missing.

[billti]

The tag is there (it's not a branch). See https://github.com/RyanCavanaugh/TypeScript/tree/pr-lkg

[azz]

The references in tsconfig.json is opt-in per-dependency, wouldn't it make more sense for it to apply to everything that resolves outside of rootDir?

I'm imagining something like a sandbox property that could look like:

# tsconfig.json
{
  "compilerOptions": {
    "outDir": "lib",
    "sandbox": "."
  },
  "include": ["src/index.ts"]
}

Sandbox would also set rootDir to the same value. Instead of explicitly providing paths to directories that contain tsconfig.json, normal module resolution would apply, and you could search up the FS tree to find the tsconfig.json automatically.

# package.json
{
  "name": "animals",
  "module": "src",
  "typings": "lib",
  "dependencies": {
    "core": "*"
  }
}

This way:

• Don't have to maintain two synced lists (references and dependencies).
• Package has no knowledge of the file system outside of its own.
• Use node module resolution strategy instead of custom paths.

[KostyaTretyak]

@RyanCavanaugh, as far as I understand, I can only work locally with these changes and I will not be able to send such projects for testing, for example, on travis-ci.org. Right?

[RyanCavanaugh]

Notes from a meeting today with @billti / @mhegazy

---

• Find References / Rename has to work, at least for scenarios where the necessary compilation context to determine the closure of projects fits in memory
• Solution-wide rename will walk to find a "solution" file, work backwards to find referencing projects, and then implementation-load the sections of the graph from which the symbol is visible. This may be an iterative process in order to find the true originating declaration
• tsbuild needs to handle a -w mode
• Solutions may not have subprojects originating outside the solution folder
• Need clear documentation for e.g. gulp, webpack

Sidebar: This rename doesn't work today

function f() {
  if (Math.random() > 0.5) {
    return { foo: 10 };
  } else {
    return { foo: 20 };
}
// rename foo here doesn't rename *both* instances in the function body
f().foo;

[nycdotnet]

Thanks Ryan, Mohamed, and Bill. Getting the Find References/Rename scenario working across projects was one of the core use cases I had in mind when I made the original complaint about TypeScript not supporting medium-sized projects. Medium-sized projects are modular but not large. The proposal and work I've seen here so far feels more like a scalability play. This is super important for the long-term health of TypeScript, but it's mostly of benefit for large projects, not medium. The things I hear in this comment from Ryan sound more along the lines of what is needed to improve the ergonomics of developing medium-sized projects in TypeScript.

As always, thank you so much to the whole TypeScript team for your efforts! You're doing awesome work.

[Bnaya]

There's a trick with lerna/yarn workspace that will make you life much easier.
Point the main and types entries in your package.json of the sub projects to your src/index.ts file, and Find References/Rename scenario will simply work.
And you will be able to compile your entire project with a single tsc running.
You may do that for some of your packages, you may for all. your call.

There are some drawback and pitfalls(If you have augmentation in one package, or import of any global symbols, it will pollute your entire program), but in general it work very well.
When you want to publish to NPM you just setting the main & types to the appropriate values (as part of your build, or so)

With the above setup, i got more or less all of the expected features

[lorenzodallavecchia]

here's a trick with lerna/yarn workspace that will make you life much easier.
Point the main and types entries in your package.json of the sub projects to your src/index.ts file, and Find References/Rename scenario will simply work.

In my experience, the problem with that setup is that TypeScript will start treating the ts files of _external_ packages as if they were _sources_ of the package that requires them, not as external libraries. This causes a number of issues.

• External packages are compiled multiple times, each time using the tsconfig of the _requiring_ package. If the requiring packages have different tsconfigs (e.g. different libs) this may cause false compilation errors to appear on the required package until it is compiled again.

• Requiring packages also compile slower because they include more files than necessary.

• The rootDir of all packages becomes the top-level directory, potentially allowing direct inclusion of any TS file from any package, instead of just including from index. If developers are not careful, they may bypass the required package API. Also, if the build process is not robust, requiring packages may end up containing duplicated code from the required package that was meant to be external.

In our projects we have ruled out depending on TS files because of the drawbacks. All inter-package dependencies are on the index.d.ts files, so the compiler treats them as external and all is good.

Of course depending on .d.ts has the problem of requiring a multi-step build (not possible out-of-the-box with tools like webpack) and the problem of poor IDE experience (renames, references not crossing package boundaries).

[mikehaas763]

I agree with some of the other sentiments - typescript is a compiler, not a build system. We need our tooling to support better multi project builds. I know there are some options out in the community starting to do this. As an example, C# has a compiler named Roslyn and a build tool named MSBuild.

[RyanCavanaugh]

Discussion today with @mhegazy about how to make rename work with as little pain as practical.

The non-degenerate worst case for rename looks like this:

// alpha.ts
const v = { a: 1 };
export function f() { return v; }
export function g() { return v; }

// alpha.d.ts (generated)
export function f(): { a: number };
export function g(): { a: number };

// beta.ts (in another project)
import { f } from '../etc/alpha';
f().a;

// gamma.ts (in yet another project)
import { g } from '../etc/alpha';
g().a;

The key observation is that it is impossible to know that renaming f().a should rename g().a unless you can see alpha.ts to correlate the two.

A rough sketch of the implementation plan:

• Have "rich" and "lean" in-memory SourceFile representations of .d.ts files. "Lean" files get read from disk; "rich" ones are produced as a result of in-memory .d.ts generation
• "Rich" .d.ts SourceFiles have references from their identifier nodes to the originating name[s] in the original source file
• During rename, we first go-to-def on the symbol in question, as usual. If this originates in a project-referenced .d.ts, we implementation-load it and create its "Rich" .d.ts file
  
  
  
  • Note: this process is iterative, i.e. it may require tracing back several levels until it lands in a real implementation file (one that is not a .d.ts redirect due to a project reference)
  
  
• Now use the "rich" pointers to figure out which other identifiers in the project's .d.ts originate from the same source file identifier
• In each downstream project, text-scan for the renamed identifier and see if its "go-to-def" result is any of the "started from the same symbol" locations in the .d.ts
• Perform the rename in the implementation file

[Bnaya]

@RyanCavanaugh will go to definition/find all references will work with this model?

[RyanCavanaugh]

Notes from an earlier discussion with Anders and Mohamed around a large number of open questions

• Does prepend also apply to .d.ts ? Yes
• What do we do with @internal in the dogfood branch? We need to keep the internal declarations in the local .d.ts files but don't want them to appear in the output verions
  
  
  
  • Remove --stripInternal
  
  
  • But don't deprecate it (yet...?)
  
  
  • Ryan to write remove-internal tool (doneish)
  
  
  • built -> LKG process removes the @internal declarations
  
  
• What happens if you change a .d.ts file from e.g. @types ?
  
  
  
  • Need to manually force a rebuild if you want to see possible new errors ☹️
  
  
  • Could eventually write a "files I looked at.txt" file to the output folder if this becomes really problematic
  
  
• Is noEmitOnError mandatory? Yes.
  
  
  
  • Otherwise rebuilds would hide errors!
  
  
• referenceTarget -> composable ✨ 🚲 🏡 ✨
• What about leaf-node projects that don't want declaration emit on, but do want fast rebuild?
  
  
  
  • tsbuild or equivalent can check to see their compliance to the non-upstream-relevant requirements of composable
  
  
• Circular references, (how) do they work?
  
  
  
  • By default, no
  
  
  • You can specify e.g. { path: "../blah", circular: true } if you want to do this
  
  
  • If you do, it becomes up to you to ensure your build is deterministic and always reaches a fixed point (possibly it may not?!)
  
  
  • The remapping of a circular:true import is an optional (but prioritized) remap
  
  

Miscellany

• @weswigham has an alternate idea for rename that we need to discuss with @mhegazy

[weswigham]

I've already lost. I mostly just wanted to keep the interpretation of sourcemaps out of the compiler (seperate responsibilities for seperate tools) but while you were away I worked on adding it anyway (because apparently seamless go-to def is desirable).

[asvetliakov]

@RyanCavanaugh
Should rename/find all references work across referenced projects after merging #23944 ? Also should we use composite: true and projectReferences: [] in case if only language services (but not tsbuild) are needed?

[mhegazy]

Should rename/find all references work across referenced projects after merging #23944 ?

not yet. but we are working on this next.

Also should we use composite: true and projectReferences: [] in case if only language services (but not tsbuild) are needed?

not sure i understand the question.. what do you mean "language service" and not "build"?

[asvetliakov]

not sure i understand the question.. what do you mean "language service" and not "build"?

I'm only interested in editor support (rename/find all references/etc...) across multiple projects in monorepo, not in the new build tool (aka build mode) (#22997) since i'm using babel for my compilation.

[mhegazy]

That should just work. build is an opt-in feature, you are not required to use it if you do not want to.. similar to how tsc is not required for your language service experience in VSCode for instance.

[weswigham]

You will likely need to build with declarations and declaration maps on, though, to produce the metadata required for cross project refs to function.

[spion]

I'm not sure if I understand all aspects of the proposal correctly, but would it be possible to not have individual projects reference others by path, but by name instead? The workspace project should have a way to specify every project path, similar to yarn workspaces via globs, or maybe by listing every individual project name:

Basically, instead of:

"dependencies": [
    "../common", 
    "../util"
],

Can we please have

"dependencies": [
    "common", 
    "util"
],

and have a workspace tsconfig.json

"workspace": {
  "common": "packages/common",
  "util": "packages/util"
}

Or better yet, paths syntax:

"workspace": {
  "*":"packages/*"
}

Benefits:

• ability to specify different lookup rules depending on module system
• ability to e.g. fallback to node_modules when package downloaded outside of workspace
• ability to freely aggregate multiple workspaces to work on by cloning multiple repos and having a different configuration for the paths, so that you can work on even more packages in parallel.

[spion]

Or at the very least, can we have the non-path names (those that don't start with './' or '../') reserved for future use...

[borekb]

I'm not sure how much this is related but Yarn 1.7 introduced a concept of "focused workspaces" recently, see this blog post.

Is anyone here familiar enough with both workspaces and the work @RyanCavanaugh is doing around TypeScript project references / build mode to maybe drop a comment explaining if they relate at all? My gut feeling is that _somewhere_ between Yarn workspaces (npm will get them too this year) and future TypeScript versions lies a good support for monorepos with multiple projects and shared libraries. (We feel the pain, currently.)

[EisenbergEffect]

I'd really love to get an update on the progress of this feature. We're planning to move Aurelia vNext into a monorepo in the next month or so. Our new version is 100% TypeScript and we'd love to use an official TS project system rather than Lerna, if we can. We're also happy to be early-adopter/testers of the new features :)

[mhegazy]

Core support and goto def using source map support were added last release (TS 2.9). tsc --b for build support is already in and is bound for TS 3.0. The typescript code base has moved to use it. We are currently testing this support using the typescript repo.

What is still needed at this point to be done: 1. get find all references/rename to work on multi project scenarios. 2. addressing updating .d.ts files in the background in the editor, and 3. --watch support for multi-project scenarios. also lots and lots of testing.

[claudeduguay]

This ticket has been on he books for 3 years. Still 3/6 outstanding items?

[borekb]

@claudeduguay This is a fundamental change to what projects TypeScript supports, time to celebrate don't you think? I'm extremely happy for this!

[EisenbergEffect]

@mhegazy This is great news. I'm very happy to hear that the TS team is dogfooding it on their own project as well. Looking forward to the last few things getting finished up and having this as an option for Aurelia :) As soon as there's some documentation on setting it up, we'll move our vNext to the new project system. Can't wait!

[EisenbergEffect]

@mhegazy Can you shed some light on how this all would work with package.json files and projects based on ES2015 modules? For example, in Aurelia vNext, we have packages like @aurelia/kernel, @aurelia/runtime, @aurelia/jit, etc. Those are the module names that will be used in import statements throughout the various projects. How will the TS compiler understand that these module names map to the various referenced folders? Will it pick up package.json files placed in each referenced folder? How will this differ from Lerna or Yarn Workspaces? My initial look into the links above makes me think I'd need to use TS projects (build) in combination with Lerna (dep linking and publishing) to get a working solution, but I'm not seeing how TS is going to properly build if it can't integrate with package.json and node_modules. The TS repo source is quite different than your average Lerna project (nothing like it really), so I'm starting to wonder if this is going to be able to meet our needs. Any more information you can provide, and esp. a working demo solution setup similar to what I've described here, would be very helpful. Thanks!

[yortus]

I share the same questions as @EisenbergEffect. In particular I'm also hoping this will work nicely with a lerna-managed monorepo.

[RyanCavanaugh]

Two monorepo scenarios to consider

Let's start with a setup where you've symlinked everything with lerna:

/packages
  /a
    /node_modules
      /b -> symlink to b with package.json "types" pointing to dist/index.d.ts
  /b
    /dist
      /index.d.ts -> built output of entry point declaration file

The key thing we want to happen here is to rebuild b we built a iff a is out of date. So we'd add "references": [ { "path": "../b" } ] to a's tsconfig.json and run tsc --build in a to get correct upstream builds of b. In this world, project references simply serve as a representation of the build graph and allow for smarter increment rebuilds. Ideally lerna and TS could cooperate here and mirror the package.json dependencies into tsconfig.json where appropriate.

Another scenario (probably less common) would be if you weren't symlinking but still wanted to "act as if" you were working on a live set of packages. You can do this today with some fairly tedious path mapping, and some people do. Project references here would similarly help with build ordering, but it'd be very desirable to have support for a property in the referent tsconfig file to automatically create a path mapping whenever it was referenced; e.g. if you had

{
  "compilerOptions": { "outDir": "bin" },
  "package": "@RyanCavanaugh/coolstuff"
}

then adding "references": [{ "path": "../cool" }] would automatically add a path mapping from @RyanCavanaugh/coolstuff -> ../cool/bin/. We haven't added this yet but can look into it if it turns out to be a more common scenario.

[weswigham]

Ideally lerna and TS could cooperate here and mirror the package.json dependencies into tsconfig.json where appropriate

Rather than relying on external tooling, we could opt to read your package.json (provided it is alongside your tsconfig) as potential references if composite: true is set (check to see if each resolved package has a tsconfig.json, if it has one, consider it a buildable project node and recur). Since everything is symlinked into place, we shouldn't even need to alter resolution (much? any?) to handle the workspace. Lerna doesn't actually set up any ts-specific (or build-specifc) stuff as is afaik, it just symlinks everything into place and manages versioning. This would be an optimization over what we do today, which is load the ts files (since we prefer those over declarations) and recompile everything regardless of out-of-date-ness.

[octogonz]

@RyanCavanaugh This sounds pretty exciting. Any idea if it will work with Rush's symlinking strategy? In a nutshell Rush creates a synthetic package common/temp/package.json that contains the superset of all dependencies for all packages in the repo. Then we use pnpm to perform a single install operation for this synthetic package. (PNPM uses symlinks to create a directed-acyclic-graph instead of NPM's tree structure, which eliminates duplicated library instances). Then Rush creates a node_modules folder for each project in the repo, made of symlinks pointing into the appropriate folders under common/temp. The result is fully compatible with TypeScript and the standard NodeJS resolution algorithm. It's very fast because there's one shrinkwrap file and one versioning equation for the entire repo, while still allowing each package to specify its own dependencies.

We don't put anything special in tsconfig.json for this model. If some special TypeScript configuration is required for the goto-definition feature, we would ideally want to autogenerate it during install rather than having it stored in Git.

[EisenbergEffect]

@pgonzal Thanks for the link to Rush! I hadn't seen that yet. I'll check it out tonight.

@RyanCavanaugh Thanks for the explanation. Your first scenario with lerna is closest to what we'd have. Here's our UX repo with TS and lerna as an example of something we'd want to use the new project support on https://github.com/aurelia/ux

@weswigham Sounds like what you are describing would fit our scenario as well. Example repo above.

[spion]

Just a note that in the case of yarn workspaces, modules are not symlinked in each individual package's directory, instead they're symlinked in the toplevel workspace node_modules.

Which is incidentally why I think that references that don't start with a dot ('./' or '../') should be reserved for the future. Hopefully those will end up being "named references", handled via the active module resolution strategy instead of treated as relative paths.

[RyanCavanaugh]

@spion we'll just use a property name other than path for that if needed (e.g. "references": [ { "module": "@foo/baz" } ]); I don't want to cause confusion wherein "bar" and "./bar" mean the same thing in files but a different thing in references

[RyanCavanaugh]

Docs/blogpost work in progress below (will edit this based on feedback)

I'd encourage anyone following this thread to give it a try. I'm working on the monorepo scenario now to iron out any last bugs/features there and should have some guidance on it soon

---

Project References

Project references are a new feature in TypeScript 3.0 that allow you to structure your TypeScript programs into smaller pieces.

By doing this, you can greatly improve build times, enforce logical separation between components, and organize your code in new and better ways.

We're also introducing a new mode for tsc, the --build flag, that works hand in hand with project references to enable faster TypeScript builds.

An Example Project

Let's look at a fairly normal program and see how project references can help us better organize it.
Imagine you have a project with two modules, converter and units, and a corresponding test file for each:

/src/converter.ts
/src/units.ts
/test/converter-tests.ts
/test/units-tests.ts
/tsconfig.json

The test files import the implementation files and do some testing:

// converter-tests.ts
import * as converter from "../converter";

assert.areEqual(converter.celsiusToFahrenheit(0), 32);

Previously, this structure was rather awkward to work with if you used a single tsconfig file:

• It was possible for the implementation files to import the test files
• It wasn't possible to build test and src at the same time without having src appear in the output folder name, which you probably don't want
• Changing just the internals in the implementation files required typechecking the tests again, even though this wouldn't ever cause new errors
• Changing just the tests required typechecking the implementation again, even if nothing changed

You could use multiple tsconfig files to solve some of those problems, but new ones would appear:

• There's no built-in up-to-date checking, so you end up always running tsc twice
• Invoking tsc twice incurs more startup time overhead
• tsc -w can't run on multiple config files at once

Project references can solve all of these problems and more.

What is a Project Reference?

tsconfig.json files have a new top-level property, references. It's an array of objects that specifies projects to reference:

{
    "compilerOptions": {
        // The usual
    },
    "references": [
        { "path": "../src" }
    ]
}

The path property of each reference can point to a directory containing a tsconfig.json file, or to the config file itself (which may have any name).

When you reference a project, new things happen:

• Importing modules from a referenced project will instead load its output declaration file (.d.ts)
• If the referenced project produces an outFile, the output file .d.ts file's declarations will be visible in this project
• Build mode (see below) will automatically build the referenced project if needed

By separating into multiple projects, you can greatly improve the speed of typechecking and compiling, reduce memory usage when using an editor, and improve enforcement of the logical groupings of your program.

composite

Referenced projects must have the new composite setting enabled.
This setting is needed to ensure TypeScript can quickly determine where to find the outputs of the referenced project.
Enabling the composite flag changes a few things:

• The rootDir setting, if not explicitly set, defaults to the directory containing the tsconfig file
• All implementation files must be matched by an include pattern or listed in the files array. If this constraint is violated, tsc will inform you which files weren't specified
• declaration must be turned on

declarationMaps

We've also added support for declaration source maps.
If you enable --declarationMap, you'll be able to use editor features like "Go to Definition" and Rename to transparently navigate and edit code across project boundaries in supported editors.

prepend with outFile

You can also enable prepending the output of a dependency using the prepend option in a reference:

   "references": [
       { "path": "../utils", "prepend": true }
   ]

Prepending a project will include the project's output above the output of the current project.
This works for both .js files and .d.ts files, and source map files will also be emitted correctly.

tsc will only ever use existing files on disk to do this process, so it's possible to create a project where a correct output file can't be generated because some project's output would be present more than once in the resulting file.
For example:

  ^ ^ 
 /   \
B     C
 ^   ^
  \ /
   D

It's important in this situation to not prepend at each reference, because you'll end up with two copies of A in the output of D - this can lead to unexpected results.

Caveats for Project References

Project references have a few trade-offs you should be aware of.

Because dependent projects make use of .d.ts files that are built from their dependencies, you'll either have to check in certain build outputs or build a project after cloning it before you can navigate the project in an editor without seeing spurious errors.
We're working on a behind-the-scenes .d.ts generation process that should be able to mitigate this, but for now we recommend informing developers that they should build after cloning.

Additionally, to preserve compatability with existing build workflows, tsc will not automatically build dependencies unless invoked with the --build switch.
Let's learn more about --build.

Build Mode for TypeScript

A long-awaited feature is smart incremental builds for TypeScript projects.
In 3.0 you can use the --build flag with tsc.
This is effectively a new entry point for tsc that behaves more like a build orchestrator than a simple compiler.

Running tsc --build (tsc -b for short) will do the following:

• Find all referenced projects
• Detect if they are up-to-date
• Build out-of-date projects in the correct order

You can provide tsc -b with multiple config file paths (e.g. tsc -b src test).
Just like tsc -p, specifying the config file name itself is unnecessary if it's named tsconfig.json.

tsc -b Commandline

You can specify any number of config files:

 > tsc -b                                # Build the tsconfig.json in the current directory
 > tsc -b src                            # Build src/tsconfig.json
 > tsc -b foo/release.tsconfig.json bar  # Build foo/release.tsconfig.json and bar/tsconfig.json

Don't worry about ordering the files you pass on the commandline - tsc will re-order them if needed so that dependencies are always built first.

There are also some flags specific to tsc -b:

• --verbose: Prints out verbose logging to explain what's going on (may be combined with any other flag)
• --dry: Shows what would be done but doesn't actually build anything
• --clean: Deletes the outputs of the specified projects (may be combined with --dry)
• --force: Act as if all projects are out of date
• --watch: Watch mode (may not be combined with any flag except --verbose)

Caveats

Normally, tsc will produce outputs (.js and .d.ts) in the presence of syntax or type errors, unless noEmitOnError is on.
Doing this in an incremental build system would be very bad - if one of your out-of-date dependencies had a new error, you'd only see it once because a subsequent build would skip building the now up-to-date project.
For this reason, tsc -b effectively acts as if noEmitOnError is enabled for all all projects.

If you check in any build outputs (.js, .d.ts, .d.ts.map, etc.), you may need to run a --force build after certain source control operations depending on whether your source control tool preserves timestmaps between the local copy and the remote copy.

msbuild

If you have an msbuild project, you can turn enable build mode by adding

    <TypeScriptBuildMode>true</TypeScriptBuildMode>

to your proj file. This will enable automatic incremental build as well as cleaning.

Note that as with tsconfig.json / -p, existing TypeScript project properties will not be respected - all settings should be managed using your tsconfig file.

Some teams have set up msbuild-based workflows wherein tsconfig files have the same implicit graph ordering as the managed projects they are paired with.
If your solution is like this, you can continue to use msbuild with tsc -p along with project references; these are fully interoperable.

Guidance

Overall Structure

With more tsconfig.json files, you'll usually want to use Configuration file inheritance to centralize your common compiler options.
This way you can change a setting in one file rather than having to edit multiple files.

Another good practice is to have a "solution" tsconfig.json file that simply has references to all of your leaf-node projects.
This presents a simple entry point; e.g. in the TypeScript repo we simply run tsc -b src to build all endpoints because we list all the subprojects in src/tsconfig.json
Note that starting with 3.0, it is no longer an error to have an empty files array if you have at least one reference in a tsconfig.json file.

You can see these pattern in the TypeScript repo - see src/tsconfig_base.json, src/tsconfig.json, and src/tsc/tsconfig.json as key examples.

Structuring for relative modules

In general, not much is needed to transition a repo using relative modules.
Simply place a tsconfig.json file in each subdirectory of a given parent folder, and add references to these config files to match the intended layering of the program.
You will need to either set the outDir to an explicit subfolder of the output folder, or set the rootDir to the common root of all project folders.

Structuring for outFiles

Layout for compilations using outFile is more flexible because relative paths don't matter as much.
One thing to keep in mind is that you'll generally want to not use prepend until the "last" project - this will improve build times and reduce the amount of I/O needed in any given build.
The TypeScript repo itself is a good reference here - we have some "library" projects and some "endpoint" projects; "endpoint" projects are kept as small as possible and pull in only the libraries they need.

Structuring for monorepos

TODO: Experiment more and figure this out. Rush and Lerna seem to have different models that imply different things on our end

[RyanCavanaugh]

Also looking for feedback on #25164

[vvs]

@RyanCavanaugh Very nice write-up, and the excellent feature, would be really nice to try it out, esp. after I spent days organizing our big project into sub-projects with config file references.

I have a couple of notes:

1. What's a monorepo? Would be nice to describe this use-case a bit more.
2. In most cases (esp. true for big projects), there are many additional artifacts generated during the build. In our case these are CSS, HTML, Image files, etc, via gulp. I wonder how is usage of such build tools would adopt to this new way of doing things. Say, I'd like to run watch not only on *.ts files, but on all our other files (styles, markup etc). How to do that? Need to run, say gulp watch and tsc -b -w in parallel?

[RyanCavanaugh]

@vvs a monorepo is a collection of NPM packages usually managed by a tool like Rush or Lerna

If you're using gulp, you'd want to use a loader that understood project references natively to get the best experience. @rbuckton has done some work here as we do have some developers using a gulpfile internally; maybe he can weigh in on what good patterns look like there

[EisenbergEffect]

@RyanCavanaugh This is looking good. I'm very interested in the Lerna guidance :)

[yortus]

@RyanCavanaugh this looks great, I'm currently working on trying it out with our lerna monorepo.

The only unclear thing to me in your write up was the prepend option. I didn't quite get what problem it is addressing, in which situation you would want to use it, and what happens if you don't use it.

[johnnyreilly]

This is awesome! I work on ts-loader and related projects. Are changes likely to be necessary to support this in projects that use TypeScript's LanguageServiceHost / WatchHost?

(See https://github.com/TypeStrong/ts-loader/blob/master/src/servicesHost.ts for an example of what I mean.)

If so, all guidance / PRs are gratefully received! In fact if you wanted this to be tested out in the webpack world I'd be happy to help in pushing out a version of ts-loader that supports this.

Of course if it "just works" that's even better :smile:

Great work!

[RyanCavanaugh]

@yortus @EisenbergEffect I've set up a sample lerna repo at https://github.com/RyanCavanaugh/learn-a with a README outlining the steps I took to get it working.

[spion]

If I'm understanding correctly, tsc -b X will do nothing if everything (X and all its dependencies and transitive dependencies) is up to date? Wondering if thats something we could get even without the -b flag for individual projects without references? (minus dependencies in that case, of course)

[claudeduguay]

This is pretty cool. I tend toward using a Lerna a configuration like this (to separate the mono repo by function). I presume that would work just as well.

{
"lerna": "2.11.0",
"packages": [
"packages/components/",
"packages/libraries/",
"packages/frameworks/",
"packages/applications/",
"packages/tools/*"
],
"version": "0.0.0"
}

[timfish]

So this is available on typescript@next?

I'll test this out with our yarn workspace repo. We have to use nohoist for a few modules which don't yet support workspaces so it'll be nice to see how it handles it.

[EisenbergEffect]

@RyanCavanaugh I took the repo for a test run tonight. I opened an issue on the repo to report some issues I had. Thanks again for putting this together. I'm looking forward to using it soon.

[guidsdo]

Really interesting! Currently at my company, we use my own tool called mtsc to support watch mode of multiple projects at the same time. We have around 5 projects that needs to be compiled and watched in the same repo.

Projects have different configs like; ECMA targeting (es5, es6), types (node, jest, DOM etc), emit (some use webpack and some compile to js themself). They all share one thing, and that is the tslint plugin, rest can be all different. My tool also runs tslint after a project compilation (per project and stopped if a project recompiles before tslint is done).

My main concern with the current proposal is that you can't say which projects share which resources. We have a server and a client project, that both use a special utility folder, but that we don't want to see the compile errors of twice. But that can be fixed with a filter, so that's no biggy :)

[yortus]

I've tried out the new --build mode with our lerna monorepo, which currently consists of 17 interdependent packages. It took a while to get everything working, but now it all works, and being able to build incrementally is a great improvement for us.

I did encounter a few issues which I describe below. I hope this is useful feedback for the TS team and might help others to get --build mode working for their projects.

Feedback for tsc --build mode

1. Spurious "\ is out of date because output file '2.map' does not exist"

I got this message for every package on every build, so every build became a full re-build even when nothing was changed. I noticed @RyanCavanaugh has already fixed this in #25281, so it's no longer a problem if you update to the 20180628 or later nightly. The following issues assume you have updated to at least the 20180628nightly.

2. "\ is up to date with .d.ts files from its dependencies" when it isn't

EDIT: reported at #25337.

To reproduce this problem, set up @RyanCavanaugh's learn-a sample repo as per his instructions. Run tsc -b packages --verbose to see everything gets built the first time. Now change line 1 in pkg1/src/index.ts to import {} from "./foo"; and save it. Run tsc -b packages --verbose again. The build for pkg2 is skipped, even though pkg1 was changed in a way that breaks pkg2's source. You can now see a red squiggle in pkg2/src/index.ts. Build again with tsc -b packages --force and the build error is shown. The following issues assume building with --force to work around this.

3. Generated .d.ts files causing 'Duplicate identifier' build errors in downstream packages

EDIT: reported at #25338.

To reproduce this problem, set up @RyanCavanaugh's learn-a sample repo as per his instructions. Now run lerna add @types/node to add Node.js typings to all three packages. Run tsc -b packages --force to confirm it still builds fine. Now add the following code to pkg1/src/index.ts:

// CASE1 - no build errors in pkg1, but 'duplicate identifier' build errors in pkg2
// import {parse} from 'url';
// export const bar = () => parse('bar');

// CASE2 - no build errors in pkg1 or in downstream packages
// import {parse, UrlWithStringQuery} from 'url';
// export const bar = (): UrlWithStringQuery => parse('bar');

// CASE3 - no build errors in pkg1 or in downstream packages
// export declare const bar: () => import("url").UrlWithStringQuery;

// CASE4 - no build errors in pkg1, but 'duplicate identifier' build errors in pkg2
// import {parse} from 'url';
// type UrlWithStringQuery = import("url").UrlWithStringQuery;
// export const bar = (): UrlWithStringQuery => parse('bar');

Uncomment one case at a time and run tsc -b packages --force. Cases 1 and 4 cause a deluge of build errors in pkg2. With cases 2 and 3, there are no build errors. The important difference with cases 1 and 4 seems to be the first line in the generated pkg1/lib/index.d.ts:

/// <reference path="../node_modules/@types/node/index.d.ts" />

Cases 2 and 3 don't generate this line. When pkg2 is built in case 1 and 4, it includes two identical copies of @types/node declarations at different paths, and that causes the 'duplicate identifier' errors.

Perhaps this is by design since cases 2 and 3 work. However it seems pretty confusing. There are no build errors or warnings in pkg1 for any of these 4 cases, but the downstream build behaviour is very sensitive to the exact style of the exported declarations. I think either (a) pkg1 should error for cases 1 & 4, or (b) all four cases should have the same downstream build behaviour, or (c) there should be some clear guidance from the TS team on how to write declarations to avoid this problem.

4. Problems with import types in generated .d.ts files when using yarn workspaces

When trying to get build mode working with our 17 package monorepo, I worked through a number of build errors caused by the relative paths in import types in generated .d.ts files. I finally worked out that the problem related to module hoisting. That is, when using yarn workspaces, all installed modules are hoisted to the monorepo-root node_modules directory, including the symlinks for all the packages in the monorepo. I changed the monorepo over to use the packages property in lerna.json, which causes lerna to use its own non-hoisting bootstrapping algorithm, and that solved the problem. It's a better/safer approach anyway, although slower.

I'm not sure if TS intends to support module-hoisted setups, so I haven't developed a repro for the problems I encountered, but I could try to make one if there is interest. I think the problem may be that some builds are getting the same type via both the top-level packages directory (as per tsconfig settings) and the top-level node_modules directory (as per import types in generated .d.ts files). This sometimes works due to structural typing, but fails for things like exported unique symbols.

5. Repetitive configuration

Setting up a monorepo to use lerna basically just requires putting something like "packages": ["packages/*"] in lerna.json. Lerna works out the exact package list by expanding globstars, and then works out the exact dependency graph by looking at the dependencies declared in each package's package.json. You can add and remove packages and dependencies at will, and lerna keeps up without any need to touch its config.

TypeScript --build mode involves a bit more ceremony. Glob patterns are not recognised, so all packages must be explicitly listed and maintained (eg in packages/tsconfig.json ) in @RyanCavanaugh's learn-a sample repo. Build mode doesn't look at package.json dependencies, so every package must maintain the list of other packages it depends on in both its package.json file (under "dependencies") as well as it's tsconfig.json file (under "references").

This is a minor inconvenience, but I include it here since I found the rigmarole noticeable compared to lerna's approach.

6. tsc crash with global module augmentations

EDIT: reported at #25339.

To reproduce this problem, set up @RyanCavanaugh's learn-a sample repo as per his instructions. Now run lerna add @types/multer to add multer typings to all three packages. Run tsc -b packages --force to confirm it still builds fine. Now add the following line to pkg1/src/index.ts:

export {Options} from 'multer';

Run tsc -b packages --force again. The compiler crashes due to a violated assertion. I looked briefly at the stack trace and assertion, and it seems to be something to do with the global augmentation of the Express namespace.

[mhegazy]

thanks @yortus for the feedback. rely appreciate it. for 3, i think it is https://github.com/Microsoft/TypeScript/issues/25278.

For 4, I am not familiar with module hoisting as a concept. can you elaborate, and/or share a repro?

[rosskevin]

@mhegazy many who use lerna and yarn use workspaces (including myself). More info here: https://yarnpkg.com/lang/en/docs/workspaces/

I'm currently using yarn workspaces, lerna, extended tsconfigs where the base tsconfig declares paths shared for all packages with hoisted module found under root/node_modules. When I hear yarn and monorepo, I think workspaces because that is the very intent of the feature - to ease use and reduce duplication. I was expecting this change would simple remove my lengthy/painful paths declared in my base tsconfig.

Here is a sample of our root monorepo tsconfig (if it is of any help):

{
  "extends": "./packages/build/tsconfig.base.json",
  "compilerOptions": {
    "baseUrl": "./packages",
    "paths": {
      "@alienfast/build/*": ["./build/src/*"],
      "@alienfast/common-node/*": ["./common-node/src/*"],
      "@alienfast/common/*": ["./common/src/*"],
      "@alienfast/concepts/*": ["./concepts/src/*"],
      "@alienfast/faas/*": ["./faas/src/*"],
      "@alienfast/math/*": ["./math/src/*"],
      "@alienfast/notifications/*": ["./notifications/src/*"],
      "@alienfast/ui/*": ["./ui/src/*"],
      "@alienfast/build": ["./build/src"],
      "@alienfast/common-node": ["./common-node/src"],
      "@alienfast/common": ["./common/src"],
      "@alienfast/concepts": ["./concepts/src"],
      "@alienfast/faas": ["./faas/src"],
      "@alienfast/math": ["./math/src"],
      "@alienfast/notifications": ["./notifications/src"],
      "@alienfast/ui": ["./ui/src"],
    }
  },
  "include": ["./typings/**/*", "./packages/*/src/**/*"],
  "exclude": ["node_modules", "./packages/*/node_modules"]
}

I'll take a shot at forking for a sample:
https://github.com/RyanCavanaugh/learn-a

[rosskevin]

Here is a not-for-merging PR to @RyanCavanaugh's repo with yarn workspaces:
https://github.com/RyanCavanaugh/learn-a/pull/3/files

[saulshanabrook]

We also used module hoisting in Jupyterlab, with lerna and yarn. It allows us to basically share our installed dependencies between all our packages, so they only exist once in the filesystem, at the root project.

I understand workspaces as being a little cleaner than having to use the link command between all of the packages so that they can access each other (or at least access their dependencies).

[timfish]

As above, module hoisting moves all dependencies to a root node_modules directory. This takes advantage of the fact that node module resolution will always traverse up the directory tree and search through all the node_modules directories until it finds the required module. The individual modules in your monorepo are then symlinked in this root node_modules and everything just works. The yarn blog post probably explains it better than I can.

Hoisting is not guaranteed to occur. If you have mismatched versions of the same package they will not be hoisted. Also, many existing tools don't support hoisting because they make assumptions about where node_modules will be or they do not correctly follow node module resolution. Because of this there is a nohoist setting which can disable hoisting for specific modules or dependencies.

[yortus]

I added a sixth item to my previous feedback. tsc crashes in the scenario described there.

@mhegazy I'm not sure item 3 is related to #25278. #25278 describes invalid declaration emit. My generated declaration files were syntactically and semantically valid, but caused downstream projects to be built with two copies of node typings, resulting in 'duplicate identifier' errors.

[octogonz]

As above, module hoisting moves all dependencies to a root node_modules directory. This takes advantage of the fact that node module resolution will always traverse up the directory tree and search through all the node_modules directories until it finds the required module.

Btw there is a downside to this model, that it leads to "phantom dependencies" where a project can import a dependency that was not explicitly declared in its package.json file. When you publish your library, this can cause trouble such as (1) a different version of the dependency getting installed than what was tested/expected, or (2) the dependency missing completely because it was hoisted from an unrelated project that is not installed in this context. PNPM and Rush both have architectural choices intended to protect against phantom dependencies.

[octogonz]

I have a general question about tsc --build: Is the TypeScript compiler seeking to take over the role of orchestrating the build for projects in a monorepo? Normally the toolchain is going to have a whole pipeline of tasks, stuff like:

• preprocessing
• generating localized strings
• converting assets to JavaScript (css, images, etc)
• compiling (type checking / transpiling)
• rolling up the .js files (e.g. webpack)
• rolling up the .d.ts files (e.g. API Extractor)
• post processing including unit tests and documentation generation

Normally a system such as Gulp or Webpack manages this pipeline, and the compiler is just one step in the middle of the chain. Sometimes a secondary tool also runs the build in another way, e.g. Jest+ts-jest for jest --watch.

Is tsc aiming to manage these things itself? And if not, is there a way for a conventional build orchestrator to solve the dependency graph itself, and e.g. repeatedly invoke tsc in each project folder in the right order (after the preprocessing has been updated)?

Or, if the design is to process an entire monorepo in a single pass (whereas today we build each project in a separate NodeJs process), I'm also curious how the other build tasks will participate: For example, will we run webpack on all projects at once? (In the past that led to out-of-memory issues.) Will we lose the ability to exploit multi-process concurrency?

These aren't criticisms BTW. I'm just trying to understand the big picture and intended usage.

[yortus]

@pgonzal right, there are many non-tsc parts to building a real world monorepo. For our lerna monorepo I took the following approach:

• each package in the monorepo optionally defines a prebuild script and/or a postbuild script in its package.json. These contain the non-tsc aspects of the build.
• in the monorepo's package.json, there are these scripts:
  "prebuild": "lerna run prebuild", "build": "tsc --build monorepo.tsconfig.json --verbose", "postbuild": "lerna run postbuild",
• That's it. Running yarn build at the monorepo level runs the prebuild scripts for each package that defines them, then it runs the tsc --build step, then it runs all the postbuild scripts. (By convention in both npm and yarn, executing npm run foo is roughly the same as npm run prefoo && npm run foo && npm run postfoo.)

[octogonz]

How do you handle jest --watch or webpack-dev-server? For example when a source file is modified, do the prebuild/postbuild steps run again?

[borekb]

Does this have any implications on ts-node and related workflows? Some of our helper apps run "directly" from TypeScript, like "start": "ts-node ./src/app.ts" or "start:debug": "node -r ts-node/register --inspect-brk ./src/app.ts".

[yortus]

Reported another issue with build mode at #25355.

[RyanCavanaugh]

Thanks for all the great feedback and investigations so far. I really appreciate everyone who took time to try things out and kick the tires.

@yortus re https://github.com/Microsoft/TypeScript/issues/3469#issuecomment-400439520

Great write-up, thanks again for providing this. Your issues in order -

1. Fixed
2. PR up at #25370
3. Discussing at the logged issue - not immediately apparent what the right fix is, but we'll do something
4. Investigating (below)
5. Logged #25376
6. Technically unrelated to --build AFAICT. This is a new assert we added recently; Nathan's investigating

@rosskevin 🎉 for the PR on the learn-a repo! I'm going to merge that into a branch so we can compare and contrast better.

@pgonzal re https://github.com/Microsoft/TypeScript/issues/3469#issuecomment-401577442

I have a general question about tsc --build: Is the TypeScript compiler seeking to take over the role of orchestrating the build for projects in a monorepo?

Great question; I want to answer this one very clearly: definitely not.

If you're happy today using tsc to build your project, we want you to be happy tomorrow using tsc -b to build your multi-part project. If you're happy today using gulp to build your project, we want you to be happy tomorrow using gulp to build your multi-part project. We have control over the first scenario, but need tool and plugin authors to help us with the second, which is why even tsc -b is just a thin wrapper over exposed APIs that tool authors can use to help project references play nicely in their build models.

The broader context is that there was fairly robust internal debate over whether tsc -b should even exist, or instead be a separate tool / entry point - building a general-purpose build orchestrator is an enormous task and not one we're signing up for. For our own repo we used tsc with a light task runner framework and now use tsc -b with the same task runner, and I'd expect anyone else migrating to also keep their existing build chain in place with only small tweaks.

@borekb re https://github.com/Microsoft/TypeScript/issues/3469#issuecomment-401593804

Does this have any implications on ts-node and related workflows? Some of our helper apps run "directly" from TypeScript

For single-file scripts, which implicitly cannot have project references, there is zero impact.

@EisenbergEffect had some questions in the learn-a repo about cross-project rename and other language service features. The big open question here is just if we'll be able to get this feature in a usable state for 3.0 or not. If so, then cross-project rename will "just work", subject to the caveat that it's obviously impossible for us to conclusively find all downstream projects and update them - this will be a "best effort" based on some heuristics for looking for other projects.

If we don't think cross-project rename is acceptably stable+complete for 3.0, we'll likely block rename operations only when the renamed symbol is in the .d.ts output file of another project - allowing you to do this would be very confusing because the .d.ts file would get updated on a subsequent build of the upstream project after the upstream project had been modified, which means it could easily be days between when you make a local rename and when you realize that the declaring code hadn't actually been updated.

For features like Go to Definition, these are working today in VS Code and will work out-of-the-box in future versions of Visual Studio. These features all require .d.ts.map files to be enabled (turn on declarationMap). There's some per-feature work to light this up, so if you see something not working as expected, do log a bug as we may have missed some cases.

Open problems I'm tracking at this point:

• Cross-project rename - @andy-ms is implementing
• Need to analyze hoisted module setups and understand their implications - on me
• There should be a version of the sample learn-a repo that uses yarn, and another that uses pnpm, and another that uses one of those in hoisted mode

Open questions

• Should we implement logic to read package.jsons to infer project references? Logged #25376
• Should .d.ts.map emit be implicitly on for composite projects?

[weswigham]

@RyanCavanaugh to add to

Open problems I'm tracking at this point

We've also mentioned having an incremental output cache, separate from the real project output location, to handle things like updating declarations in the background in the LS (today, changes don't propagate across project boundaries in the editor until you build), stripInternal, and mutating build processes (where our build outputs are mutated in place and so not suitable for LS operations).

[zpdDG4gta8XKpMCd]

sorry for a dumb question, since it's checked in the roadmap, how do i get this feature enabled?

[timfish]

@aleksey-bykov you can use it in typescript@next.

I just tried this out on our yarn workspace powered monorepo and it works well.

One thing I did notice was that tsc --build --watch reports errors but then doesn't output anything to say that the build is now fixed. The standard tsc watch mode in 2.9 has started giving an error count and its nice to see a zero there so you know that building has completed.

[zpdDG4gta8XKpMCd]

i have a folder full of *.d.ts and nothing else what am i supposed to do about it:

• make it a project and reference it (tried, didn't work)
• use "include" for it

[RyanCavanaugh]

@timfish that feedback matches some other I've heard; logged #25562

@aleksey-bykov https://github.com/Microsoft/TypeScript/issues/3469#issuecomment-400439520 should help explain some concepts

[zpdDG4gta8XKpMCd]

@RyanCavanaugh it looks like project referecing only works for commonjs and node module resolution, doesn't it?

in you example:

import * as p1 from "@ryancavanaugh/pkg1";
import * as p2 from "@ryancavanaugh/pkg2";

p1.fn();
p2.fn4();

1. what is @ryancavanaugh module, does it has anything to do with how TS resolves modules?
2. is this example supposed to work with AMD (classic module resolution)?
3. is outFile required for definitions to be found?
4. where d.ts files are supposed to be in order for referencing project to find them (can i still use outDir? will TS find them there?)

i have 2 simple projects essentials and common and things in common cannot resolve stuff compiled in essentials:

[RyanCavanaugh]

@aleksey-bykov

1. It's just a scoped module name, resolved under the usual node module resolution algorithm
2. You can use project references with any module system including classic, but the example names (scoped modules) aren't very friendly to use outside of node
3. No
4. TypeScript looks for the .d.ts files to be in the place where the referenced project builds them to

If you have a sample repo or something I can diagnose why you're getting that error

[zpdDG4gta8XKpMCd]

@RyanCavanaugh please do
example.zip

[timfish]

@RyanCavanaugh, it also looks like tsc --build --watch doesn't initially output any files until its sees a modification of a source file.

[RyanCavanaugh]

Thread too long (in both time and space); let's pick up the discussion at lucky issue number 100 * 2^8 : #25600