Compodoc: [BUG] Documentation Generation is Slow with Large Projects

[jongunter]

Overview of the issue

I have an Angular 2.4 CLI project with 10ish modules and nearly 90 components (and growing!). When I run compodoc, it takes about 2-3 minutes to complete. This is a great piece of software, and I love it, but 3 minutes to build our documentation library is rough, especially coming from AngularJS, where a single pass of gulp-ngdoc on an application twice as big takes less than a second.

Operating System, Node.js, npm, compodoc version(s)

This seems to happen regardless of the environment...
Windows 8 & Mac OS X
compodoc 0.0.40
npm 4.0.5
node 7.4.0

Compodoc installed globally or locally ?

locally

[vogloblinsky]

Yes, i know, i have some enhancements on scanning to do.
But i rely on TypeScript AST parsing, for each statement of a file, a little more complicated than simple JS parsing like ngdocs.
I have a watch flag feature in development and coming in next major release.

[vogloblinsky]

@jongunter Last version 1.0.0-beta.10 contains a little refactoring and a huge boost on documentation generation. +75%
Could you do please a test with your big project ?
Watch flag is also available.

[vogloblinsky]

@jongunter any news ?

[RobertBroersma]

I'm experiencing the same with 1.0.0-beta.10 in a project with around 100 components.

Compodoc has been running for over 2 hours, still "Including".

[vogloblinsky]

@RobertBroersma
Is it possible to have your terminal logs ?
Did these lines appears ?

```
[12:09:47] Searching package.json file
[12:09:47] package.json file found
[12:09:47] Searching README.md file
[12:09:47] README.md file found
[12:09:47] Get dependencies data
````

[vogloblinsky]

@jongunter @RobertBroersma
Just tried to generate a randomly project with angular-cli.

#!/bin/bash
rm -rf ./src/app/module*
rm -rf ./src/app/component*
for i in {1..150}
do
   echo "Generating $i"
   ng generate module "module$i" --spec false --module "app.module"
   ng generate component "component$i" --module "module$i" --spec false
done

Compodoc works fine after.

Did you know an open-source project with big architecture like yours ? It can helps me a lot reproducing this issue.

[RobertBroersma]

@vogloblinsky No, not getting those lines, however not sure if it would happen at the end, since I never had the patience to go there. I can try that sometime soon maybe. For now I get these:

$ compodoc -p tsconfig.json -s













1.0.0-beta.10

Node.js version : v6.9.1

Operating system : Windows 10

[18:23:38] Using tsconfig : D:\Normero\repos\ticombo\web\src\tsconfig.json
[18:24:32] Including      : D:\Normero\repos\ticombo\web\src\app\accounts\components\alerts\alert-item\alert-item.component.ts

And a lot more Including ... following

[vogloblinsky]

@RobertBroersma
Ok, last version is very quicker than before.

When you say "Compodoc has been running for over 2 hours, still 'Including'.", is he including only source files in your D:\Normero\repos\ticombo\web\src folder ?

Is it blocking on one file during these 2 hours ?

[csentis]

@RobertBroersma you might want to check whether what I have experienced about excludes (!) in tsconfig.json also applies to your case - mind the time stamps in terminal log: #174

[vogloblinsky]

@jongunter any news ?

[jongunter]

I'm sorry...I haven't been doing much Angular work recently, so I haven't had a chance to try it. I will keep you posted. Thanks!

[RobertBroersma]

@vogloblinsky @csentis Thanks for the tips.

It is running at about 1 file every minute, and only in the src folder yes. The files it runs through are correct. It's just at a dreading pace.

I will run through @csentis experiment later! Thanks.

[vogloblinsky]

@RobertBroersma @jongunter
I have found the source of this issue, it was about files initial scanning, and glob exclusion in exclude attribute in tsconfig.json.
The scanning part has been rewritten, and successfully tested on Primeng project, which has around 228 modules and 206 components.
The fix will come in next release 1.0.0-beta.14
If you want to test before, install locally the development version of compodoc :

npm i --save-dev https://github.com/compodoc/compodoc#develop
node_modules/bin/compodoc -p src/tsconfig.json

[csentis]

Wow. Just tried compodoc#develop and got down from 2874 sec in standard 1.0.0-beta.13 to 21 sec.

I am not sure whether I miss the excludes from tsconfig.json in terminal log (they are not listed anymore), but I am absolutely convinced by the speed your have managed to come up with, @vogloblinsky. Plus I like the project statistics you brought into the terminal log. Congrats!

Might not be related to the dev version: I do not manage to get the watch mode working properly (I use yarn run compodoc -- --tsconfig tsconfig.json --toggleMenuItems all --output doc --serve --watch) with Node v8.3.1 und Win10. Tells me in terminal it is watching src files but no re-run is triggered. Could be a yarn topic, though.

[vogloblinsky]

@csentis
Thanks for the develop feedbacks about huge project speed improvements.
Did find any issue with watch flag, what are the statistics of your app ?

[csentis]

I have been playing around a bit and could not get watch to trigger a rebuild under Windows 10 Powershell.

Statistic are small:

-------------------
Project statistics
- module     : 20
- component  : 31
- directive  : 2
- injectable : 18
- class      : 23
- interface  : 25
-------------------

Besides: if I clean yarn's cache and mention "@compodoc/compodoc": "https://github.com/compodoc/compodoc#develop", in package.json, I get:

Error: ENOENT: no such file or directory, open 'C:\Users\cse\Documents\Firmen\midyll\Repos\midyll-front\node_modules\@compodoc\compodoc\src\resources\images\banner'

The whole resources folder is missing. If I first mention "@compodoc/compodoc": "1.0.0-beta.13", do a yarn install and only afterwards mention "@compodoc/compodoc": "https://github.com/compodoc/compodoc#develop", in package.json (followed by a yarn install), it plays out nicely of course.

[csentis]

Downgraded to beta.12 just for fun, with which watchmode works as expected (switches used in package.json scripts section remained unchanged). Do you remember any changes you made in the watch (change detection) section?

[vogloblinsky]

@csentis

Fixed, little issue with chokidar and root markdowns files support. Ok in development branch.
Release 1.0.0-beta.14 very soon this week.

[csentis]

@vogloblinsky: simply great. Using latest compodoc#develop I can confirm that watch mode is working properly.

Just as a reminder to everyone eager to try this out: you need to manually copy over directory node_modules\@compodoc\compodoc\src\resources\images when using the compodoc#develop version.

[vogloblinsky]

@csentis
What do you mean by "you need to manually copy ..."

[csentis]

@vogloblinsky I had to download compodoc master's zip, unzip locally and copy over the "resources" folder as it does not get installed if you use github as a source for compodoc in your package.json. If I do not do this, I get the ENOENT error mentioned above in this thread.

[vogloblinsky]

@csentis
Didn't manage to reproduce your issue with "resources" folder.

• installed git, node.js 8.1.4, yarn 0.27.5 inside a fresh Windows VM
• cloned the example repository : https://github.com/compodoc/compodoc-demo-todomvc-angular
• yarn add https://github.com/compodoc/compodoc#develop
• ./node_modules/.bin/compodoc -p src/tsconfig.json -s

[vogloblinsky]

I close this issue, 1.0.0-beta.14 works better with huge projects.

[csentis]

For the sake of completeness in case someone else using yarn should meet the same weird situation that compodoc complains about Error: ENOENT: no such file or directory, open 'node_modules\@compodoc\compodoc\src\resources\images\banner':

this is likely due to the fact that you have a .yarnclean file in your root directory. Read an explanation of of this file is intended to do to some of your modules here.

Baseline is: certain (redundant, non-functional) directories will be removed as a last step after yarn install; this step can be omitted by not placing a .yarnclean. The file is created when using the command yarn clean which I mistook because I had a yarn (package.json) lifecycle script named clean.

@vogloblinsky sorry I misused this issue for a totally different issue

[lock[bot]]

This issue has been automatically locked due to inactivity. Please file a new issue if you are encountering a similar or related problem. Why locking ? Having issues with the most up-to-date context.