Definitelytyped: [@types/lodash] Version 4.14.99 causes slow compilation

[DnJealt]

• [x] I tried using the @types/xxxx package and had problems.
• [x] I tried using the latest stable version of tsc. https://www.npmjs.com/package/typescript
• [x] I have a question that is inappropriate for StackOverflow. (Please ask any appropriate questions there).
• [x] [Mention](https://github.com/blog/821-mention-somebody-they-re-notified) the authors (see Definitions by: in index.d.ts) so they can respond.
  
  
  
  • Authors: @bczengel, @chrootsu, @stepancar, @ericanderson, @Ailrun, @e-cloud, @thorn0, @jtmthf, @DomiR
  
  

Since the new version, released on 31-1-2018, my webpack build time has increased from 6 seconds to about 35. I came across this problem while updating from TypeScript version 2.6.2 to 2.7.1.
TypeScript v2.7.1 will raise a syntax error while using types/lodash version 4.14.74. Updating to the latest v4.14.99 fixes this error, but causes the slow loading times.
Right now my options are either:

• Roll back to TypeScript v2.6.2 and types/lodash v4.14.74, or
• Uninstall types/lodash completely.

I hope you are able to look into this. If you require any more information, feel free to ask :)

[aj-r]

@andy-ms could this be related to splitting lodash into multiple files?

[aj-r]

Actually for me, running tsc with @types/[email protected] and [email protected], the compile time is a fraction of a second. Can you give some more details about your setup?

[DnJealt]

Thanks for replying.
At the moment we are developing a quite large project with TypeScript. It uses webpack with the ts-loader plugin to compile all those files into a single bundle.js.

I have tried rolling back to [email protected] with @types/[email protected] and it still compiles very slow. Compiling with version 4.14.100 doesn't seem to make it any better either.

[andy-ms]

Could you try with 4.14.93 which was the last release before it was split into multiple files? Also please test the TypeScript and lodash version updates separately in case it's the new TypeScript version that's slowing you down.

[DnJealt]

Okay, I have tried to compile this with the following versions.

With TypeScript 2.7.1

• @types/[email protected] - OK. 6 seconds
• @types/[email protected] - OK. 6 seconds
• @types/[email protected] - OK. 6 seconds
• @types/[email protected] - ERROR. 32 seconds and console giving me an ocean full of errors
• @types/[email protected] - SLOW. 31 seconds
• @types/[email protected] - SLOW. 31 seconds
• @types/[email protected] - SLOW. 35 seconds
• @types/[email protected] - SLOW. 41 seconds

I've also done a small test and rolling back to TypeScript 2.6.2:

• @types/[email protected] - OK. 6 seconds
• @types/[email protected] - ERROR. 31 seconds and ocean of errors.
• @types/[email protected] - SLOW. 30 seconds

I'm thinking that the fix that was introduced in @types/[email protected] has caused this slow compilation to happen.
I hope I'm leading you folks in the right direction here :)

[DomiR]

Just a guess, but this seems to be a webpack configuration error:
@types/[email protected] - splitting up files (using namespace) (8b292ec78a8260ec9f76c9b49a1c179a0f66e12a)
@types/[email protected] - use module augmentation (0f4014dbf9fc47a372e3471964d12f374772bcfd)
@types/[email protected] - using classical style imports (745420a23e9de39bfa0a924d7867a7f0bd26da1d)

Can you look into the webpack output file and see if lodash is imported a bunch of times?
Presumably you are using: awesome-typescript-loader did you make sure to add exclude: /node_modules/, Reference? Or maybe just exclude: /node_modules/@types when building for the browser.

[DnJealt]

@DomiR nope, we're using ts-loader and not awesome-typescript-loader. I did add exclude: /node_modules/ to the module.rules section of the webpack config.

As for the webpack output, I'm only seeing lodash being mentioned once.

[wuyiw]

i got the same problem and tested a lot to find out that switching between @4.14.81 and @4.14.82 makes a great difference in code autocompletion (yep, both vscode and webpack are inffected), almost instantly vs 3 secs, in vue SFC with only pick of lodash and skeleton code (reload vscode after switching).

and as i update my dependencies (also typescript, ts-loader, etc.) to fix lodash type error (now @4.14.101) and other bugs, my webpack prod compilation time has increased from ~80s to ~220s, finally i had to set `transpileOnly' in prod build conf and make it done in ~30s, but developing still suffers...

[wuyiw]

i created a reproduction repo with simplest code, wish it could help:
https://github.com/wuyiw/ts-lodash

[andy-ms]

@aj-r @DomiR I don't think we have to go back to a single file, but maybe the way we are now goes too for -- for example, prototype.value.d.ts and prototype.valueOf.d.ts don't really deserve separate files -- they are just two separate methods on the same interface. Having many declarations of the same interface may be making it slow.

[aj-r]

@andy-ms before, we had the same number of declarations of each interface as we do now - each function re-declared the LoDashStatic interface.

Could the performance hit be due to loading more files than before? Or possibly having many declarations of the same "../index" module?

By any chance could we improve performance by inverting how the types are defined? i.e. instead of declaring a module in index.d.ts and augmenting it, could we have each function's file declare its own interface, and have index.d.ts import all of those files and merge them? Something like this:

// some.d.ts
import common = require("../common");

declare namespace Lodash {
    interface Some {
        <T>(predicate: common.ListIterateeCustom<T, boolean>, collection: _.List<T> | null | undefined): boolean;
       // ... and other overloads
    }
}

declare const some: Lodash.Some;
export = some;

// index.d.ts
import some = require("./some");
// ... and other imports

export = _;
export as namespace _;

declare const _: _.LoDashStatic;
declare namespace _ {
    interface LoDashStatic {
        some: typeof some;
        // ...etc
    }
}

I suspect this would at least improve performance of importing a single function (e.g. "lodash/some"), since we wouldn't need to load the types for all functions just to get one. I'm not sure if it will improve importing the monolith, though. Also not sure if this would break things for some people.

[andy-ms]

@aj-r Not sure if that would improve performance. You actually wouldn't need interface Some in that case, and could just declare function some. Though it might be nice to keep the LoDashImplicitWrapper and LoDashExplicitWrapper versions of some in that same file, and those would still have to be augmentations.

[aj-r]

I think I'm reproducing the same issue just by running npm run lint lodash in the DefinitelyTyped repo. I've added lodash-fp-tests.ts to test the lodash/fp types I'm working on. Here's the interesting part:

• If I modify tsconfig.json so it excludes lodash-fp-tests.ts, the task takes a normal amount of time (~1 minute).
• If I modify tsconfig.json so it includes lodash-fp-tests.ts, but excludes lodash-tests.ts, the task takes about 10 seconds.
• If tsconfig.json contains both test files, the task runs for an extremely long time (~30 minutes) before finally succeeding
• If tsconfig.json contains both test files BUT none of the .d.ts files (index.d.ts, add.d.ts, etc.), the task runs for ~1 minute.

I'll do some testing and see if I can find any way to cut the time down.

Here's the code I used to reproduce this: https://github.com/aj-r/DefinitelyTyped/tree/lodash-fp

[andy-ms]

@DnJealt Could you get a sample project using ts-loader that has slow load times? Want to be able to see if changes I make would have any effect

[andy-ms]

@wuyiw In 4.14.81 _({} as any).pick(['foo', 'bar', 'baz']).value() was any. Now it returns a Pick type. It makes sense that this would slow things down as any basically turns off type checking. You might still file a bug on typescript if you can find a simple case that's particularly slow.

[thorn0]

@DnJealt (and others too) please try 4.14.102. I fear that it might have become even slower.

[DnJealt]

@thorn0 Can confirm, 41 seconds now.

@andy-ms When I have the time I will update this post with a github link.

[aj-r]

@DnJealt try v4.14.105. We have greatly reduced the number of files which should hopefully improve build time.

[DnJealt]

Sorry for my late reply and not providing a github link even though I said I would do so. My working group had been shifted to another project.

That being said... Hooray@!. This did indeed do the trick! Build times are down to 7000milliseconds again!

Thanks a lot :)

[aj-r]

@DnJealt glad to hear it :). Want to close this issue then?

[DnJealt]

Yes please! I'm not seeing the issues coming back now. Thanks for the effort people!
Have a beer :) :beers:

[aj-r]

@DnJealt you have the power to close it not me :)

[kelly-tock]

i'm seeing a similary issue to this for @types/[email protected]. anyone else?

[kelly-tock]

this commit is pretty large, could it be it? it was pretty close to the time when my projects build started slowing wayyy down.

e4f9fd18561349b00945d50b0dde1c34444fa39f

[kelly-tock]

@sandersn any ideas on if this would cause any slowdown in build times?

[kelly-tock]

looks like since 4.14.125 something has been slowing down for TS 3.4.x

[sandersn]

@kelly-tock lodash was often unusably slow on TS3.4. e4f9fd1 was intended to fix that. I expected compile times to be much faster on 3.4 and somewhat faster on 3.3 and 3.5.

Can you open a separate issue and ping me on it there? The next step is to do a 2D comparison of @types/lodash versions vs typescript versions, and that will take enough space to describe that it should be on a new bug.

It's also possible that one of the new definitions is actually slower for some usage pattern. Do you use _.chain (or other chaining methods) heavily? Those are the only ones that I changed.

[DnJealt]

I could re-open this issue and rephrase the title if needed

[sandersn]

@DnJealt It turns out @kelly-tock already did. I just didn't see it until after posting this comment. https://github.com/DefinitelyTyped/DefinitelyTyped/issues/35750