Typescript: Building composite project with resolveJsonModule fails

[strax]

TypeScript Version: 3.0.0-rc

Search Terms:
resolveJsonModule, composite project, TS6307

Code

Repro at https://github.com/strax/ts-3-json-bug-repro

Expected behavior:

With composite: true and resolveJsonModule: true set in tsconfig.json, JSON modules are resolved in the compilation like they are when the composite compiler option is not set.

Actual behavior:

Running tsc gives the following error:

error TS6307: File '/src/hello.json' is not in project file list. Projects must list all files or use an 'include' pattern.

In addition, trying to include src/hello.json in the files property fails with

error TS6054: File '/src/hello.json' has unsupported extension. The only supported extensions are '.ts', '.tsx', '.d.ts'.

Playground Link:

Related Issues:

[mhegazy]

we should allow .json in files list if --resolveJsonModules is enabled.

[sheetalkamat]

@mhegazy Is assumption that we shouldnt include them through include pattern of config file though?

[mhegazy]

yeah. i think that would be a bigger breaking change.

[cockscomb]

@strax @sheetalkamat @mhegazy Is this problem fixed in TypeScript 3.1.1?

I have tried to use TypeScript 3.1.1 in the repro project, TypeScript gives the following error:

error TS6307: File '/src/hello.json' is not in project file list. Projects must list all files or use an 'include' pattern.

It seems to this problem is still exist.

[sheetalkamat]

@cockscomb you need to have files clause to have the json file("files": ["src/hello.json"] in the tsconfig file). The issue here was it reported unsupported extension (which is not what you are seeing)
In your
cc: @RyanCavanaugh

[raymondfeng]

@sheetalkamat We ran into a similar issue as @cockscomb. tsc complains about json files not in project file list no matter how we tweak tsconfig.json:

{
  "$schema": "http://json.schemastore.org/tsconfig",
  "compilerOptions": {
    "rootDir": "src",
    "composite": true,
    "target": "es2017",
    "outDir": "dist"
  },
  "references": [
    ...
  ],
  "include": [
    "**/*"
  ]
}

None of the following works:

• "include": ["**/*"]
• "include": ["**/*.json", "**/*.ts"]

Do we have to use files to include json files?

[bajtos]

Do we have to use files to include json files?

If I am reading the tests added as part of https://github.com/Microsoft/TypeScript/commit/49ac60f286972f0349a1f39365bcc6403b3772e5 correctly, then yes, JSON files must be explicitly listed in files section of tsconfig.

@sheetalkamat what is the reasoning behind this requirement/limitation? Would you accept a pull request modifying the behavior of include to cover JSON files too? (I.e. change the test "with resolveJsonModule and include only" to expect a successful compilation.)

https://github.com/Microsoft/TypeScript/blob/49ac60f286972f0349a1f39365bcc6403b3772e5/src/testRunner/unittests/tsbuild.ts#L256-L258

[sheetalkamat]

We do not want to add json files in compilation accidently and want users to explicitly specify them instead of using the include/exclude pattern.

[bajtos]

@sheetalkamat Thank you for the comment, I see your point.

Let me explain my use case. In LoopBack, we have a concept of a data source - a class providing access to a database, web service, etc. An application (a LoopBack/TypeScript project) can have multiple datasources. Each datasource is implemented in two files: a JSON file with the configuration (to make it easy to edit programmatically) and a TypeScript file exporting the datasource class.

The TypeScript code in src/datasources/db.datasource.ts (full version):

import * as config from './db.datasource.json';

export class DbDataSource extends juggler.DataSource {
  constructor() {
    super(dsConfig);
  }
}

JSON config in src/datasources/db.datasource.json (full version):

{
  "name": "db",
  "connector": "memory"
}

Relevant parts of tsconfig (we are not using project references yet):

{
  "$schema": "http://json.schemastore.org/tsconfig",
  "compilerOptions": {
    "rootDir": "."
    "resolveJsonModule": true,

    "lib": ["es2018", "dom", "esnext.asynciterable"],
    "module": "commonjs",
    "moduleResolution": "node"
  },
  "include": ["index.ts", "src", "test"]
}

Now back to the topic of this issue:

1. When the project is _not_ composite and compiler is running in the "classic" mode (without -b), it correctly understands that db.datasource.ts is importing db.datasource.json and therefore the JSON file needs to be included in the compilation too.

   I find it confusing that composite/build mode changes this behavior.

2. When the project is switched to composite mode (the only change being adding composite: true to compilerConfig), the compilation starts failing with error TS6307. Despite this error, the actual JSON file is correctly copied to the output directory.

   I find it confusing that the compiler understands that a JSON file is imported and thus needs to be copied to the output directory, yet it complains that it was not explicitly listed in tsconfig's files section.

3. I find the requirement to explicitly add all JSON files one by one as a poor user experience. For applications (projects) containing many JSON files, the tsconfig content can become large and difficult to maintain.

If my first two arguments above are not convincing enough and/or there are other reasons explaining why JSON files must be treated differently in composite vs. non-composite projects: Can we at least allow JSON files to be specified via an explicit pattern in include, e.g. src/datasources/*.datasource.json? By explicit I mean that the pattern must match JSON files only (i.e. the pattern string must end with .json).

|pattern|file|match?|
|:--:|:--:|:--:|
**/* | src/foo.ts | ✔️
**/* | src/data.json | ✖️
src/*.json | src/foo.ts | ✖️
src/*.json | src/data.json | ✔️

[sheetalkamat]

I find it confusing that the compiler understands that a JSON file is imported and thus needs to be copied to the output directory, yet it complains that it was not explicitly listed in tsconfig's files section.

The issue isn't about including json in compiling but its about not specifying the json file in the list of inputs. Composite projects need explicitly include all files in the compilation as referencing projects need to be able to identify its all output.

Adding @DanielRosenwasser for opinion. The requirement you are suggesting is quite a work and need to be handled by multiple hosts as well (since it needs to integrate with our readDirectory interface as well). Apart from that right now if you have js and ts file with same name in location, the jsFile is ignored(since its lower in priority than ts file and that means that behavior needs change for json modules as well)

[sheetalkamat]

Ok I spoke too soon. I think we are able to achieve what you want and I will put up PR for this. Also discussed with @DanielRosenwasser that including json file even if there is ts file with same name is ok. (since its not output of ts file anyways)

[bajtos]

Thank you @sheetalkamat for finding a solution and landing it via #27844 🙇

I think it would be great to describe the interaction between composite: true and resolveJsonModule in the handbook too, e.g. https://github.com/Microsoft/TypeScript-Handbook/blob/master/pages/Project%20References.md, and explain users how to configure their project to avoid "Projects must list all files or use an 'include' pattern." error. I will try to find some time to contribute those changes unless somebody else can contribute them quicker.

[EnixCoda]

This works for me:

{
  "include": ["./src/**/*", "./src/**/*.json"],
}

[DylanVann]

There is still the issue of JSON files in node_modules not working.

[StrongerSY]

Listing JSON files in files section of tsconfig works for me ~ Thx @bajtos

[xtianus79]

This is still an issue

[bsunderhus]

I'm experiencing the same issue my project is setup as followed:

.
├── @types
├── package.json
├── src
│   ├── consts.json
│   ├── main
│   │   ├── index.ts
│   │   └── tsconfig.json
│   ├── tsconfig.json
│   ├── utils
│   │   ├── html-template.ts
│   │   └── tsconfig.json
│   └── worker
│       ├── state.ts
│       └── tsconfig.json
└── tsconfig.base.json

My main entry configuration file is src/tsconfig.json which only have references to other sub projects localized in src/main, src/worker and src/utils and in the main project I'm trying to access the src/consts.json file and I'm getting the error:

File '/workspace/src/consts.json' is not listed within the file list of project '/workspace/src/main/tsconfig.json'. Projects must list all files or use an 'include' pattern.

Although I've listed it explicitly through files in src/main/tsconfig

{
  "extends": "../../tsconfig.base.json",
  "files": [
    "./index.ts",
    "../consts.json",
  ],
  "references": [
    {"path": "../worker"},
    {"path": "../utils"}
  ]
}

[trusktr]

I was spending time fiddling around with this until I realized it was a TS issue.

It is confusing and not intuitive to have to write something like

{
  "include": ["./src/**/*", "./src/**/*.json"],
}

[paulrberg]

@trusktr's solution worked, but I had to restart my IDE.

Why is this issue closed?

[asm2025]

The addition of the new composite attribute sounds reasonable and justified to allow nested config but seriously, guys, I think you are breaking a lot of projects right now!!