Typescript: Slow compilation without outFile

[markboyall]

The output from tsc --diagnostics:

Files: 1417
Lines: 132281
Nodes: 722439
Identifiers: 238709
Symbols: 373699
Types: 150887
I/O read: 0.09s
I/O write: 10.03s
Parse time: 1.34s
Bind time: 0.47s
Check time: 3.48s
Emit time: 12.61s
Total time: 17.90s

As you can see, the runtime of the compiler is dominated by i/o write. Changing modules to AMD (from CJS) and using --outFile produces a dramatic speedup.

Files: 1417
Lines: 132281
Nodes: 722439
Identifiers: 238709
Symbols: 373699
Types: 150887
I/O read: 0.10s
I/O write: 0.23s
Parse time: 1.37s
Bind time: 0.43s
Check time: 3.44s
Emit time: 2.70s
Total time: 7.93s

These timings are produced on Typescript 2.1.4. This is using an i7 6700k and a Samsung 850 Evo SSD.

The check, emit, and parse times are somewhat to be expected considering the size of the project. We accept that we'll need to break it up further. But the I/O write times look like there's a lot of room for optimization.

The problem is that we are using Webpack and it can't to handle the outFile option, so our builds are becoming mind-numbingly slow.

[mhegazy]

do you have an antivirus running on this machine? (e.g. windows defender if this is a Windows box)

[markboyall]

I am running Windows Defender (Windows 10). I turned off real-time protection, but this does not seem to have made any difference to the results.

[mhegazy]

Can you add an exclusion to your build/out directory? also an exclusion for .ts files.

[markboyall]

I tried adding an exclusion for our project, but did not have any luck. I can try adding an exclusion for .ts files next time I'm at work (30th)

[mhegazy]

if you can share the project i can give it a try. would be happy to sign an NDA if needed.

[markboyall]

Exclusion did not have any effect. I can discuss NDA-related things with my boss when he is back in a week or so but I suspect that he will not go for it.

[mhegazy]

Can you try node tsc.js and tsc.exe and let us know if you see any difference? can you try on other non-windows machines?

the difference in time here is mostly IO, the compiler does not do any thing special, so i would like to think that this is caused by some other environment factors.

[markboyall]

We don't have any non-Windows machines lying around. The Node timings are even worse.

Files: 1426
Lines: 135185
Nodes: 755940
Identifiers: 254296
Symbols: 429012
Types: 138819
Memory used: 319485K
I/O read: 5.92s
I/O write: 14.86s
Parse time: 9.25s
Bind time: 0.47s
Check time: 5.67s
Emit time: 16.35s
Total time: 31.73s

Note that this is TS 2.0.6 as we had to revert the upgrade; but I ran one with tsc.exe and got basically the same timings as 2.1.4.

Files: 1426
Lines: 135185
Nodes: 755940
Identifiers: 254296
Symbols: 429012
Types: 138819
I/O read: 0.11s
I/O write: 13.63s
Parse time: 1.64s
Bind time: 0.30s
Check time: 3.48s
Emit time: 15.47s
Total time: 20.89s

[markboyall]

The main thing I noticed in the source is that the file writing functions are actually synchronous, not asynchronous, so it seems logical to me they'd be slow(er). Not sure if I'd expect this much slower though.

[aligneddev]

Excluding the folder from Windows Defender helped for me, but tsc still seems like it is taking longer than it should. Our project uses RequireJs and AMD. I can't say for sure, but after upgrading from Windows7 to Windows 10 last week (both were using TS 2.1.4) on this same computer, compile time seemed to double our triple.

Before Exclusion:
Files: 687
Lines: 113402
Nodes: 528279
Identifiers: 159401
Symbols: 137418
Types: 49271
Memory used: 364541K
I/O read: 1.15s
I/O write: 24.65s
Parse time: 19.98s
Bind time: 1.95s
Check time: 11.70s
Emit time: 31.05s
Total time: 64.68s

After Exclusion:
Files: 687
Lines: 113402
Nodes: 528279
Identifiers: 159401
Symbols: 137418
Types: 49271
Memory used: 349808K
I/O read: 0.41s
I/O write: 2.44s
Parse time: 18.02s
Bind time: 3.09s
Check time: 11.94s
Emit time: 8.91s
Total time: 41.95s

[markboyall]

You have poor performance, but not for the same reason as us. Your parse and check time is very high, I/O time very low since removing Windows Defender. For us it's the other way around- we have about 15% more code than you (and considerably more symbols/types) but our check and parse times are just fine, it's the I/O time that's dominant.

[aligneddev]

Thanks, something is going on with my system. A co-worker with the same hardware and setup gets:
Files: 678
Lines: 111503
Nodes: 518879
Identifiers: 157389
Symbols: 135448
Types: 48680
Memory used: 185100K
I/O read: 0.08s
I/O write: 0.71s
Parse time: 3.97s
Bind time: 0.64s
Check time: 3.12s
Emit time: 2.26s
Total time: 10.00s

[mhegazy]

I will need access to a project that shows this behavior to be able to diagnose more.

[andidavies92]

@mhegazy, I've created a project that reproduces the issue:
https://github.com/BluewireTechnologies/slow-tsc-outfile-reproduction

[mhegazy]

On my windows machine, here is what i see:

Without --outFile, Windows defender on:

c:\test\13053\slow-tsc-outfile-reproduction>tsc --diagnostics GeneratedTypeScript\Main.ts --module amd
Files:          2552
Lines:         28621
Nodes:        134622
Identifiers:   37572
Symbols:       31718
Types:         12253
Memory used: 107844K
I/O read:      0.43s
I/O write:     9.95s
Parse time:    1.46s
Bind time:     0.15s
Check time:    0.66s
Emit time:    10.84s
Total time:   13.10s

Without --outFile, Windows defender off:

c:\test\13053\slow-tsc-outfile-reproduction>tsc --diagnostics GeneratedTypeScript\Main.ts --module amd
Files:          2552
Lines:         28621
Nodes:        134622
Identifiers:   37572
Symbols:       31718
Types:         12253
Memory used: 108610K
I/O read:      0.36s
I/O write:     1.68s
Parse time:    1.21s
Bind time:     0.19s
Check time:    0.71s
Emit time:     2.73s
Total time:    4.83s

With --outFile and Windows Defender on

c:\test\13053\slow-tsc-outfile-reproduction>tsc --diagnostics GeneratedTypeScript\Main.ts --module amd --out file.js
Files:          2552
Lines:         28621
Nodes:        134622
Identifiers:   37572
Symbols:       31718
Types:         12253
Memory used: 114439K
I/O read:      0.47s
I/O write:     0.15s
Parse time:    1.68s
Bind time:     0.18s
Check time:    0.69s
Emit time:     0.78s
Total time:    3.33s

With --outFile and Windows Defender off

c:\test\13053\slow-tsc-outfile-reproduction>tsc --diagnostics GeneratedTypeScript\Main.ts --module amd --out file.js
Files:          2552
Lines:         28621
Nodes:        134622
Identifiers:   37572
Symbols:       31718
Types:         12253
Memory used: 116143K
I/O read:      0.36s
I/O write:     0.01s
Parse time:    1.18s
Bind time:     0.20s
Check time:    0.94s
Emit time:     0.68s
Total time:    2.99s

[mhegazy]

The results, with Windows Defender off, seem reasonable to me.

[markboyall]

We use CJS modules rather than AMD modules; does this make a meaningful difference?

[mhegazy]

If you have a sample, I would love to try it out. My experience is higher io type is usually caused by anti virus scans. The compiler does not do anything intestine when writing to disk.

[markboyall]

I was able to reproduce this difference on my home machine, but not on my work machine. I think that it is time for us to re-examine our environment.

[markboyall]

Turns out that we were just using the wrong time for the build performance- the time we used was only the Webpack time and never included the Typescript time. So when our performance was improved as expected, we simply didn't see the result.

Thanks for your patience- your advice was spot on.

[mhegazy]

thanks for reporting back. closing.