Highcharts: TypeScript: Excessive memory usage in JetBrains products like WebStorm

[bre1470]

Expected behaviour

20 MB memory increase

Actual behaviour

Much higher memory increase with potential crash

Live demo with steps to reproduce

There are three known reports by customers.

Product version

all v7.x

Affected editor(s)

Visual Studio Professional 2017 with ReSharper extension
WebStorm

[bre1470]

I can reproduce it in WebStorm (32 bit). It seams like doclet-less declarations are solving the problem in WebStorm.

You can download them here and place them in your ./node_modules/highcharts.

Therefor I guess, some Markdown/JSDoc in a doclet is triggering this problem. I will now search for it.

[bre1470]

If you throw enough memory to WebStorm it will actually not crash. Instead it looks like the number of doclets is too high. Probably it is prerendering all.

[bre1470]

So the problem is indeed the prerendering of the complete doclet tree by the editor. Restructuring with more common interfaces will not help in this case.

Removing the doclets of the ten properties on the Series interface had already reduced the memory usage by 220 MB (of 650 MB).

[FrozenDroid]

Having this problem as well. The memory usage of WebStorm can get ridiculously high, at about 10gb.

[bre1470]

Thank you for your feedback!

There is a ticket at JetBrains about this issue: https://youtrack.jetbrains.com/issue/WEB-36866

In the meantime we try to reduce the problem by splitting things over several files. Unfortunately this probably will not make it into the next update, because of the tight schedule.

You can try the following declarations without comments as potential workaround: highcharts-declarations-issue10004.zip and place them into ./node_modules/highcharts/.

Please let me know if this would solve your problem. Thank you.

[FrozenDroid]

Yes, I created that ticket.
I've linked this issue as well so that the JetBrains guys could look into this as well.
It does partly work, but I also need typings for Highstock etc. Is it possible for you to create the rest of the typings without docs, as well?

[bre1470]

Oh, I see. Well, you can just use the highstock module with the replacements as well. The declarations of the other products are at the moment simply pointing to the big highcharts declaration.

[FrozenDroid]

Still doesn't really seem to be fixed. I am getting very high CPU usage and WebStorm has 4.8gb memory usage... I'm not sure how to debug this.
I'm going to have to go back to the old typings and old Highcharts, unfortunately...

[FrozenDroid]

@bre1470 can you get rid of the "Can't reproduce" label?

[bre1470]

I can only reproduce it in WebStorm. For now it seams to be a bug in JetBrains software.

[bre1470]

I will modify the issue scope and remove the tag.

[Abbo44]

I reproduce it in IntelliJ IDEA 2018.3.4

[bre1470]

@Abbo44
Thank you for your feedback! Please let us know if a JetBrains update will solve the problem for you.

I am still working on some attempts to solve the issue from our side.

The only longterm solution is a fix by JetBrains. I become aware that the problem also exists with other larger frameworks like Angular 2+, and frameworks in other programming languages.

[bre1470]

I attached a summary of the possible attempts to reduce the memory usage. Every variant has its drawbacks and might not work in the specific JetBrains setup.

Attachment: highcharts-declarations-issue10004.zip

Please let us know if one of the variants helps you by adding a thumb to the three following comments:

[bre1470]

./product-split: Options splitted by product files. The product modules "gantt", "map" and "stock" do not work because of TypeScript limitations.

[bre1470]

./without-doclets: Declarations without comment doclets. Reduces the file size, but requires looking up details on https://api.highcharts.com/

[bre1470]

./without-doclets-and-complexity: Declarations without comment doclets and series-specific interfaces. This provides less autocomplete and requires looking up details on https://api.highcharts.com/

[FrozenDroid]

The only longterm solution is a fix by JetBrains. I become aware that the problem also exists with other larger frameworks like Angular 2+, and frameworks in other programming languages.

That's not really true. I am using Angular 6 and it works great with WebStorm. Highcharts is the only thing that made this issue show up. The fault is probably still at JetBrains.

[firehist]

Same problem here, too large definition file which cause memory problems.

[FrozenDroid]

According to a JetBrains employee, the issue has been fixed and will be released in the 2019.1 EAP.
https://youtrack.jetbrains.com/issue/WEB-36866

[bre1470]

Thank you for the feedback. This sounds good.

[JusuVh]

I'm testing the 2019.1 beta release and although it's better, it still uses a huge amount of memory.
While using the declaration files without doclets, the usage is much better.

I don't know what the best long term solution is, but 15MB declaration files seem like a bit much. Is there no way to optimize the doclets? Maybe remove all the links?

Thanks anyway.

[bre1470]

@JusuVh Thank you for your suggestion. We will remove the links and see if this helps.

[bre1470]

A quick follow-up. We have removed the documentation links with v7.1.0. The main declaration file has now a size of around 10 MB. Please let us know, if it improves your situation.

[marcusjwhelan]

Currently on version 2019.1.1(IntelliJ), The memory will grow till the application passes. If I change Typescript to Bundled and deselect TypeScript Language Service then it seems to slow down the memory clime completely.

[bre1470]

@marcusjwhelan
Thank you for your feedback and workaround suggestion! Without the TypeScript Language Service the memory usage in the WebStorm IDE is improving as well. The drawback though is the lack of inline error reports. Unfortunately a look into the Highcharts declarations is still increasing the memory usage by about 600 MB and the chunk gets not deallocated after closing the file.

[JusuVh]

With IntelliJ 2019.1.3 and Highcharts 7.1.2 this issue is a little better but I think something could be done to make it much better.

I think the main issue is that all the definitions are bundled into one huge file, namely highcharts.d.ts. Currently, if I need to lookup a definition, I need to load the 260 000 lines of the highcharts.d.ts file. And even on a relatively good computer, opening the file in IntelliJ takes at least 15 seconds.

I believe IntelliJ pretty much needs to do the same when indexing for the language service. If I'm only using, say SeriesLineOptions, it still needs to link up all the other types that I'm not using in my application, every time I use a type.

Looking at what is done elsewhere, I think it would be good to package the definitions into several layers of files, organized in categories and sub-categories (see, for example, how angular/material is organized). I believe the best would even be to split each type in its own file.

I'm sure it would even be easier for you guys when dealing with bugs and/or updates, not to have to open a 10MB file each time.

Anyway, let me know if you need any help.

[bre1470]

@JusuVh
Thank you for your suggestion.

We are talking about a 10 MB file (automatic generated), that results in a memory usage of more than one gigabyte. I already tested splitting the declarations, but it has no effect. This is clearly not about the file size, but about the handling of logical relations and the dependency tree. Highcharts is a complex product.

Until IntelliJ stops the way of prebuilding all type data in advance for each import statement, the situation can not be solved without losing types. We do not have problems with opening the declarations by the way, as we do not use products by JetBrain.

[JusuVh]

I understand that Highcharts is a complex product and I also understand that a 10MB file should not result in such a big memory usage. Maybe splitting the definitions file does not solve this issue but I just believe that it's way easier to use multiple well-organised files rather than one huge file, whatever the use case or software you use.

With Highcharts 6.2.0 and types/highcharts npm package, the memory usage is normal.

Maybe Highcharts is too complex for its own sake and maybe (I'm only suggesting here) some functions could use more specific interfaces, for instance multiple interfaces that extend addSeries (e.g. addLineSeries, addPlotBandSeries, ... ). I'm not sure most users need to be able to use all series types in a single application. We, for instance, only use about 4.

I understand the legacy Javascript approach where the addSeries function accepts any kind of series because there is no typing. In Typescript however, this is not really ideal as the addSeries now accepts (as of 7.1.2) 88 different object types for it's first parameter. This might cause some of the problems when prebuilding the type data for the Typescript Language Service.

Anyway. Highcharts is a great product. I'm just trying to find solutions to make it better in all use cases. In the mean time, we might need to revert back to 6.2.0.

[bre1470]

@JusuVh
Thank you for your help and input. It is appreciated much. We always like to hear from all the different use cases for our products. We are aware of legacy limitations and the declarations were a first step to make ground for a shift in how we organize series and modules programmatically.

The second step now is to migrate our internal code base to TypeScript to get everything better in sync. The declarations though will still be generated out of our API documentation in this phase, which is one reason for the beta state.

When we have everything migrated, we will start with changes to how declarations and documentation are generated. Then it is also possible, that the situation here might improve from our side, because we will simply remove stuff and make everything more modular. Said that I still believe, that there is some design issue in IntelliJ, that results in this huge memory usage.

[JusuVh]

Great news. I look forward to those updates.

[bre1470]

I closed the issue for now as there is no further progress. If there is something new, it will be reopened.