Typescript: Incremental --build, then delete generated js file, then another incremental --build does not recreate js file

[mmorearty]

TypeScript Version: 3.4.0-dev.20190326

Search Terms:

incremental

Code

tsconfig.json:

{
  "compilerOptions": {
    "incremental": true
  }
}

x.ts:

console.log("hello");

Expected behavior:

• $ tsc --build
• x.js has been generated
• $ rm x.js
• $ tsc --build

Expected: x.js has been regenerated

Actual behavior:

x.js still does not exist

Related Issues:

Did not find any.

[sheetalkamat]

Hmm the root cause of this is that in general our incremental build (BuilderProgram) isn't equipped to deal with missing output file as indication to build but changed file as more like it. Need to think about what we can do here. Till then the workaround is to run --clean and then build or add comment or something into the file whose output needs to be generated.
Note that this didn't originate with just --incremental, this would be same behavior when you followed same pattern but were running tsc --build --watch from first step (even without --incremental in compiler options)

FYI @RyanCavanaugh @DanielRosenwasser

[irakliy81]

A separate, but possibly related issue: deleting source .ts files does not remove the related .js files. Right now, to address this, I clean the output directory before every compilation - but this makes it impossible for me to use the --incremental flag.

[sheetalkamat]

@irakliy81 that's totally out of scope and does not work even without --incremental. Build cannot delete any output file unless you specify clean (which will also delete output files of config specified at that moment)

[sdwvit]

👍
Also experienced this bug:
We had a piece of code which imported a recently deleted file error in one of the dev-branches, and since we use caching of buildinfo in CI, parallel branch builds started to experience this missing file too. However, the error was different than just 'module not found', it was another type error from patient-0 branch.

[irakliy81]

@sheetalkamat - yes, I wouldn't expect it to work without --incremental flag since in that case there is no info stored about the previous build. But with the --incremental flag, it should be possible to remove output files if the source files have been removed.

A separate question: how do you specify clean? It doesn't seem to be a valid compiler option.

[milesj]

@irakliy81 --clean

[irakliy81]

@milesj - thanks! But it doesn't seem to be listed here: https://www.typescriptlang.org/docs/handbook/compiler-options.html and if I try to set it in VS Code I get "unknown compiler option" error. Am I missing something?

[milesj]

It's part of project references. Not sure if it's available outside of that, nor as a compiler option. https://www.typescriptlang.org/docs/handbook/project-references.html

[sdwvit]

any plans on fixing this? cc @sheetalkamat

it kills the point of caching build info in CI

[RyanCavanaugh]

For clarity: What/who is deleting the output files and why?

[sheetalkamat]

There are multiple workarounds for this issue.

1. Run tsc --b --clean
2. Delete .tsbuildinfo file as well

Its not clear why only .js file is deleted

[haggholm]

@RyanCavanaugh

For clarity: What/who is deleting the output files and why?

I’ve had this happen when an unrelated clean script deleted output directories including, but not necessarily limited to, Typescript output. I mean, normally I assume that if I delete output and re-run a build tool, the output will be recreated; that’s true of all build tools I’m used to, for web development or otherwise, except, in this case, Typescript.

[RyanCavanaugh]

Your expectations are fine, we're just trying to figure out if the scenario is common enough that e.g. people who have 1,200 build outputs should be paying the cost on every single operation for TS to check if all those outputs still exist. Can the "unrelated clean script" just delete the .tsbuildinfo as well?

[sdwvit]

Well yes, we should check files existence on our side in CI. Which is fine, but as already mentioned, it's against expectations. Also, if only 1 file is deleted accidentally, deleting whole buildinfo file is a bit overkill

[b4dnewz]

I've noticed also tsc -b --clean only cleans the output files without rebuilding them, is this the desired behavior? Since the command is tsc -b --clean I assumed it will build also.

[matthiasg]

tsbuildinfo also does not get invalidated when tsconfig.json changes. Thus code generated e.g with target: es2017 does not get rebuilt when the target gets changed.

[glen-84]

@matthiasg See #31118.

[simonbuchan]

Your expectations are fine, we're just trying to figure out if the scenario is common enough that e.g. people who have 1,200 build outputs should be paying the cost _on every single operation_ for TS to check if all those outputs still exist. Can the "unrelated clean script" just delete the .tsbuildinfo as well?

This is reasonable, but I think the 90% case is worth covering here: something like if outDir doesn't exist at all (or maybe the first output file?) does not exist when starting a new tsc --incremental, assume it's been externally deleted and ignore .tsbuildinfo?

Another workaround / fix for people who need these external tools is to simply place the .tsbuildinfo in the outDir so it gets cleaned too.

[sdwvit]

@simonbuchan

so it gets cleaned too.

killing the whole purpose here

[molant]

The --incremental feature is really new and I haven't found a lot of information on how it works internally.

Is it fair to assume that the TS compiler will/could generate the output faster when it has a .tsbuildinfo file even if there isn't any output files?

If the build time is going to be the same as a fully clean one with no .tsbuildinfo, then storing them in git does not make any sense.
On the other side, if the build time should be indeed faster then I think the scenario is compelling enough to investigate how much of cost it will have and if it is worth fixing this. Any project that runs CI or a developer doing a fresh checkout will benefit from this and save precious time.

[sdwvit]

Here is an usecase when incremental build breaks developer experience:

We have a file that reads all A/B experiment configuration files (*.ts) from folder using glob.
These files are compiled using tsc (experiments/*.ts in ts-config).

When developer deletes experiment file.ts they expect compiled file to be removed too, otherwise A/B test stays active. Only manually checking and deleting build info helps.

Hope it’s clear.

[RyanCavanaugh]

Two completely different things are being discussed here and it'd be good to keep things separate. The OP refers to the first issue and it's what we're tracking with this item.

Should --incremental assume that some files may have been deleted from disk?

There's a real perf cost to doing this and it's unclear how you would often get in this state in a way where you couldn't automate yourself back into a good state (e.g., if a script is deleting some random subset of JS files, that script should also delete .tsbuildinfo and force a rebuild).

Should a subsequent build delete files that were generated by a prior build but no longer have a source?

This was discussed offline for a bit and we think the answer is a hard no. As far as we know this behavior isn't precendented by other incremental compiler tools (open to corrections on this point), and the stakes of deleting arbitrary files on disk are simply higher than we'd like to engage with. Scenarios where the enumerated set of files substantially matters are somewhat rare (IOW most of the time the import graph is what matters), as is deleting a source file in the first place. Incurring the cost of a clean or manual delete doesn't seem like a very high burden compared to the amount of work and risk on our side to figure out what a "correct" delete is. We could maybe think about some very clearly-named flag like deleteJavaScriptFilesInOutputFolderThatLackCorrespondingInputs but it's just not something we're keen to do.

[simonbuchan]

Thanks for the clarification, @RyanCavanaugh. Here's my thoughts:

Should --incremental assume that some files may have been deleted from disk?

In the general case, I'm fine with this, even if it breaks the cache transparency, I think. But it's pretty common to delete the entire output directory, and it's an unnecessary small cut to have to remember to delete the build cache too compared to other tools. Perhaps checking for just the output dir leads to false confidence?

Should a subsequent build delete files that were generated by a prior build but no longer have a source?

Scenarios where the enumerated set of files substantially matters

Substantially, perhaps, but the standard use case is the pack up everything in the out directory. For someone that's not paying attention, this means they are bloating their package size.

as is deleting a source file in the first place.

Eh, depends on your style. moving source files is fairly common. The repo I'm in at the moment has 43 renames or deletes in the last 100 commits.

Incurring the cost of a clean or manual delete doesn't seem like a very high burden compared to the amount of work and risk on our side to figure out what a "correct" delete is.

Not sure why there are scare quotes here: what's wrong with lastBuildOutput.filter(f => !newBuildOutput.includes(f))?

[regevbr]

Any news on a fix?

[cyrilchapon]

I so much agree with @simonbuchan

and it's an unnecessary small cut to have to remember to delete the build cache too compared to other tools

I just lost 1 hour with that.

Adopted new "project" strategy (for a src + test simple project); built; cleaned the whole dist; wondered half an hour why subsequent tsc -b didn't produce any output at all; spent more half an hour tried to digg and remove incremental tag (which I hate in general, stateless in mind); got "incremental option may not be disabled for composite projects"; got stuck.

[mkalam-alami]

I would have assumed that --incremental stores and checks the signatures of both the definitions and the generated sources.

In the current state there's basically a secret contract saying that nobody but tsc can touch the build output. It's not what I would assume from something simply advertised as an incremental build. And apart from subtle phrasing in the 3.4 release notes I don't think it's clear from the docs either that only types definitions are checked.

I too would be for making --incremental check the generated sources by default ; if this comes to an unreasonable cost to certain projects, having an additional, clearly labelled setting (eg. --incrementalTypeChecking) to restore current behavior would probably be enough to make everyone happy.

[wojpawlik]

I enabled composite to use references, and I lost an hour tracking down this heisenbug.