Typescript: Library include directives

[RyanCavanaugh]

Problems

/// <reference path="..." /> comments (hereafter "reference directives") currently serve
multiple purposes:

• Including another definition file containing global declarations
• Including another definition file containing ambient module declarations
• Including another implementation file

The current mechanism has some common shortcomings:

• Relative paths to a common typings folder can be come unwieldy
• There's no mechanism to "override" a reference path
• Reference directives only ever look in one place to find a file.

This has resulted in multiple issue reports of people trying to address this:

• #5893 Multiple source roots - or something like classpath, like python_path, like include_path
• #4615 Support environment variables in reference paths
• #6482 Support a 'declarationPath' option

New Use Case in Bundled Type Acquisition

An important scenario we need to address is the case where multiple typings
files need to refer to the same _global_ set of types or namespaces. If these
libraries refer to multiple versions of the same global types, we have no
existing mechanism for resolving the conflict

Solution: Library include directive

Syntax

The proposed syntax (:bike: :house: of course) would look like this

/// <reference library="jquery/jquery.d.ts" />

Behavior

This syntax means that the compiler should include the _first_ file found
when searching the following paths, where relative paths are resolved relative
to the project root (either the location of tsconfig.json, or the common
root of all input files):

• ./typings/jquery/jquery.d.ts
• ./node_modules/jquery/jquery.d.ts
• ./node_modules/@types/jquery/jquery.d.ts

This search order provides the following desirable behavior:

• When needed, the user can resolve a version conflict by placing a definitive
  global version in the local typings folder
• A bundled .d.ts file will be found automatically
• A types package can fill in for libraries which don't have bundled definitions

We could conceivably allow for configuration of this search order in tsconfig.json if
needed for complex scenarios.

UMD

Additionally, when a UMD module definition is found, its global export declaration (if present)
is added to the global scope.

All-up World

Along with the rest of the typings acquisition story, we can now have a very coherent way to
explain how file references should be managed in TypeScript programs:

• Modules which are part of your program are always imported using import
• Modules which have definition files are always imported using import, and those imports should resolve to proper modules
• Global definitions should always be located via /// <reference library = ... directives
• File ordering for concatenated output is managed with /// <reference path = .... directives

[RyanCavanaugh]

Reposting notes from other thread courtesy @mhegazy from Monday

---

• Two folders called “library” only one has a .d.ts, use that
• What if we found two different files for the same declaration file (e.g. node v2),
  
  
  
  • We should show an error if both have dependency on the same library but the files are different
  
  
• It should be an error when importing a package with ///?
• Modules should not use /// <reference path=“..” /> or ///<reference library=”..” />
• Add the compiler library folder to the search path, e.g. ///<reference library=”es6.promise.d.ts” />
• Should I specify the path, or just the name of the library?
  
  
  
  • The library name seems much better
  
  
• Guidance for package authors:
  
  
  
  • What about typing dependencies for a package
  
  
  • Ship package without types, and ship a @typingspackage separately
  
  
  • Ship types with package, and add “dependencies” on all your typings dependencies
  
  

[RyanCavanaugh]

Something we need to figure out -- do we rename all files to index.d.ts to make this happen? Right now we have no special logic that says reference library='jquery' can mean jquery/jquery.d.ts rather than jquery/index.d.ts

[billti]

We discussed using the package.json file for this (either the typings field or main field) to indicate the entry point - just as Node does.

[RyanCavanaugh]

Here's a write-up of how this works

---

Library Directives

Syntax

A _library directive_ is a new simpler way to include definition files for external code which introduces things into the global scope.

They look like this:

/// <reference library="jquery" />

Note that this is similar to the familiar /// <reference path="filename.d.ts" />, but instead path is replaced with library and we don't specify the extension (.d.ts).

Library directives are designed to integrate seamlessly with the type acquisition process, as well as bundled type definitions from NPM.

Behavior

Library definitions can be found in either _primary_ or _secondary_ locations.

The _primary_ library definition locations are:

• The project root folder
  
  
  
  • This is either the root property from tsconfig.json, the folder containing tsconfig.json, or the common directory of all source files specified on the commandline
  
  
• The ./typings folder (relative to the project root folder, described above)
• The ./node_modules folder
• The ./node_modules/@types folder

When resolving a library reference, the compiler looks for a folder of the name of the library in each primary definition location.
Inside that folder should either be a file named index.d.ts, or a package.json whose typings property points to a different filename.
The first such file found this way is the one loaded.

If all _primary_ lookups fail, we defer to the _secondary_ library locations.
The _secondary_ library locations are those determined by the normal node module resolution algorithm (briefly: start at the referencing file's folder, and walk "up" the directory structure looking for node_modules/libraryName).

Because the _secondary_ lookup locations are relative to the referencing file, it is possible that multiple files referencing the _same library name_ might end up loading _different files on disk_.
If this occurs, these different files must have identical content or an error is issued.
This is known as a _global code conflict_.

Conflicts in global code.

This "conflict" situation occurs when two different modules claim to have dependencies on different versions of some global code.
In reality, only one copy of any global script might be loaded, and the developer must simply hope that the versions are "close enough" to be compatible.
To resolve the conflict, the user must copy one of the library definitions to a _primary_ lookup location (see above), indicating which version of the library will actually be present at runtime.

Once this happens, the definition file for one of the referencing libraries may still have errors.
At this point, there is almost certainly an actual incompatibility between the two referencing libraries, and the developer must investigate further.

Use with npm install @types/

TODO: Write this up once our NPM script is live!

[mhegazy]

Implemented in https://github.com/Microsoft/TypeScript/pull/7775

[unional]

In Conflicts in global code:

To resolve the conflict, the user must copy one of the library definitions to a primary lookup location (see above), indicating which version of the library will actually be present at runtime.

IMO, this in highly impractical. The consumer would not know which version(s) of typings to use for a library that is 2+ levels deep. For direct dependencies, yes. For nested dependencies, no.

In reply to: https://github.com/Microsoft/TypeScript/issues/7125#issuecomment-209670094

if you have two files found by the same "name", it's an error if those files aren't identical

What defines a "name"? In different version of a package, they may:

1. Release using a different file name and path (e.g. index.js, dist/index.js, dist/jquery.js)
2. Bundle typings in different name, dictated by package.json/typings

For the error part, as mentioned about it won't be practical for end consumer to manage those versions.

typings handle these problems by inlining the typings of the dependencies. But at the same time, with the current implementation of UMD, we don't have a way to reference the typings without leading into error. (i.e. cannot use /// <reference types=... in typings/main.d.ts).

cc @RyanCavanaugh @blakeembrey @basarat

[RyanCavanaugh]

IMO, this in highly impractical.

It is necessarily practical by virtue of the scenario -- the user _has_ a conflict and will be resolving it by some means. Library 1 says use jQuery 1.2, library 2 says use jQuery 1.3; the user isn't going to put script tags for _both_ in their HTML page (or if they do, they should know it's not going to work).

Inlining isn't really a practical solution for a variety of reasons.

[mhegazy]

What defines a "name"? In different version of a package, they may:

1. Release using a different file name and path (e.g. index.js, dist/index.js, dist/jquery.js)
2. Bundle typings in different name, dictated by package.json/typings

The whole design relies on not doing this. the design relies on the "identity" of a library typing from the file name/ package name. bundelling, inlining, renaming, etc.. destroys that

[unional]

Library 1 says use jQuery 1.2, library 2 says use jQuery 1.3; the user isn't going to put script tags for both in their HTML page

Yes, true. And as long as user not using /// <reference types=..., they won't run into conflict as you mentioned in https://github.com/Microsoft/TypeScript/issues/7125#issuecomment-209534613

So do you see the npm @types/* is the only way (or THE way) to extend / bridge typings for packages?
Because from what you describes, it seems like TSD, Typings, or directly using typings in DefinitelyTyped (JetBrains) would not work correctly.

This is partially align with your "Problems" section in https://github.com/Microsoft/TypeScript/issues/7156#issue-134923063

[unional]

bundelling, inlining, renaming, etc.. destroys that

Do you mean package.json/typings will be deprecated?

the design relies on the "identity" of a library

I discuss with @blakeembrey on this at typings. What is your vision? It is hard to define the identify of a library. The package manager, package name, repo name, source hosting provider can all change during the course of the life of the package.

[RyanCavanaugh]

Today, it's just a singular string name. We expect to use DefinitelyTyped as the arbiter for who "gets" a particular name (for purposes of publishing to @types/that-name), with deference being to whatever maintains a 1:1 mapping between NPM and the folder names on DefinitelyTyped. We also want to support some "redirects" on DT so that someone can have a particular name and get their definition published under @types/, but manage the definitions in a separate repo as we recognize they may want to iterate more quickly than DT can support today.

For people who either don't want to publish their typings (directly or indirectly), tools can choose to write to the local types/ folder and the compiler will treat those as being authoritative.

[unional]

Through working on typings, I learn that there are quite a lot of problems in DT, especially on versioning. Do you have idea on what you can do with DT and fix those problem? I think that's partially why typings/registry is created. @blakeembrey is the authoritative voice in that regards. :smile:

We also want to support some "redirects" on DT so that someone can have a particular name and get their definition published under @types/

I think that is exactly what typings/registry is doing.

[unional]

To clarify: /// <reference types=... will load the global context. What about /// <reference path=...? Deprecated or it will only load module context.?

[unional]

Inlining isn't really a practical solution for a variety of reasons.

Can you educate me why it is not practical? How do you manage to resolve versioning of nested dependencies in the module context? Thanks. :rose:

[mhegazy]

Can you educate me why it is not practical? How do you manage to resolve versioning of nested dependencies in the module context? Thanks.

• private and protected members in a class makes that class compares nominally, i.e. only instances of the exact same class. if you bundle, the compiler has no way of knowing that they are the same, and thus they are duplicate definition errors.
• error reporting is out of the window. file names change, definition scopes change, and even module names change.
• significant increase in the size of projects on the long run, as dependencies are bundled and re-bundled. there is no way for the compiler to unbundle, and thus leverage the value of reusing source files and types. so more node, more types, and more time compiling projects

[unional]

Thanks for your explanation. :rose:
Maybe the term I used ("inlining") causes some confusion. I think we might be talking about slightly different things. Some of the general idea still applies though.

private and protected members in a class makes that class compares nominally

I understand the arguments in https://github.com/Microsoft/TypeScript/issues/7755. But like Blake, I'm not fully convinced either. :stuck_out_tongue: My argument is this: I was expecting the type system of TypeScript is duck typing, and I definitely don't want or need to compare the innards of two ducks to determine if they are the same. :rose:

I see it as a design choice, but it does feel artificial and surprise users.

if you bundle, the compiler has no way

Agree on not bundling, that just doesn't make sense. What I mean by "inlining" is how typings works today. Wrapping dependencies in declare module '~<a>~<b>' {. e.g. for chai:

// generated definitions/chai/index.d.ts (simplified for illustration purpose)
declare module '~chai~assertion-error' {
  // assertion-error typings
}
declare module '~chai' {
  // chai typings
  import * as AE from '~chai~assertion-error';
  ...
}
declare module 'chai' {
  import main = require('~chai');
  export = main;
}

Typings for typings are written in top-level module declaration (just like tsc -d without bundling) and "compiled" at consumption. Because of that, there is no duplication errors (unlike typings in DefinitelyTyped, as those are all scripts). "compile" means converting the module file(s) to a script file.

Edit: rephrase

error reporting is out of the window. file names change, definition scopes change, and even module names change

Not really getting this.....sorry~ :cry:

significant increase in the size of projects on the long run, as dependencies are bundled and re-bundled. there is no way for the compiler to unbundle, and thus leverage the value of reusing source files and types.

Yes. Agree. However, we are not talking about bundling source code. We are talking about "inlining"/"wrapping" of typings. There is a lesser impact, but impact nonetheless.

:tulip:

[unional]

This conversation also related to name conflict and versioning for module augmentation:
"Which version of the module you are actually trying to augment?"

But that can be a separate topic.

[unional]

To complete the story, the current implementation of typings does introduce module name pollution.

There is a plan that can eliminate this (https://github.com/typings/core/issues/1), but currently cannot be implemented because of some limitation.

@blakeembrey said "there'll always be pollution until things are native in TypeScript":
https://github.com/typings/core/issues/1#issuecomment-192802609

Blake, is it relevant to discuss that here? I don't know would that be too much detail related to typings.

[blakeembrey]

Blake, is it relevant to discuss that here?

I'd just leave it for now, it's possible I can refactor the implementation to use the new library directive features anyway. My primary concern here, however, is mentioned in the other thread (since these got kind of messy together). Will the library feature work in sub-dependencies or do those act on the whims of the top-level dependency?

Ideally, I could use (or abuse) this feature and create what I want to see anyway - properly isolated module definitions down the entire dependency tree. With library directives (or types, or whatever) I could point it to the typings/modules directory then just publish my module with the typings/ directory onto NPM. We can even make typings/modules external module definitions (because I fail to see how the UMD feature will work if the types feature is ambient module declarations).

[born2net]

looks like its coming soon: https://blogs.msdn.microsoft.com/typescript/2016/06/15/the-future-of-declaration-files/

[jonaskello]

I tried do use primary acquisition from a local typings folder as described in this issue in but it does not seem to work. Is what is described in this issue actually included in typescript 2.0.3?

[jonaskello]

In addition if I try to use /// <reference library="my-lib" />in typescript 2.0.3 it gives the error error TS1084: Invalid 'reference' directive syntax.. What I am trying to do is to find a way to have a non-ambient module declaration that is not located under node_modules. I really like what this issue describes but I am having a hard time getting it to work or finding any docs about it.

[jonaskello]

Ok after digging around in other issues I found that the syntax is now:

/// <reference types="my-lib" />

Or you can do the same in the tsconfig file:

{
  "compilerOptions": {
    "types": ["my-lib"]
  }
}

So now I can get tsc to pick this up :-). But it still does not resolve, and if I check the output of tsc --traceResolutionI get:

 $ tsc --traceResolution
======== Resolving type reference directive 'my-lib', containing file 'XXXXX/__inferred type names__.ts', root directory not set. ========
Root directory cannot be determined, skipping primary search paths.
Looking up in 'node_modules' folder, initial location 'XXXXX'

So it seems the root directory is not being set. I would guess it should be automatically set to the location of the tsconfig.json being used by tsc but maybe I am missing something. How do I set the root directory?

[jonaskello]

And digging into the source code of the compiler I found that you must set the typeRoots compiler options like this:

{
  "compilerOptions": {
    "types": ["my-lib"],
    "typeRoots": ["./types"]
  }
}

Now it works :-). Sorry for polluting this issue with a lot of comments but I hope it can be helpful for anyone else landing here from google like I did. Also if these options are documented somewhere I would appreciate if someone could point me there.