Dependencycheck: Initial Database cleanup performance

[mikehalmamoj]

Describe the bug
The time taken to build the CVE database from scratch has increased dramatically.

Version of dependency-check used
5.3.2, gradle plugin.

Log file
This is an example of when it was fast: https://gist.github.com/mikehalmamoj/92726c89e5c1f0f4640ff069aa4fc5be
And an example of slow: https://gist.github.com/mikehalmamoj/067cf29c629dc0b1b084f46e9424281a

Note that the key difference is End database maintenance (52108 ms) compared to End database maintenance (1624822 ms)

To Reproduce
Run the gradle task dependencyCheckUpdate in a clean environment.

Additional context
Having watched the builds, the step that takes forever is Removed the CPE ecosystem on 5515 NVD records.

I traced that back to the UPDATE_ECOSYSTEM2 SQL statement: https://github.com/jeremylong/DependencyCheck/blob/master/core/src/main/resources/data/dbStatements.properties#L47

I wonder if some recent CVE additions have exposed a flaw in the SQL?

[aravindparappil46]

I am facing the same issue. Taking too long (13+ hours) to build the database for the first time (in fact, it's still going on as I write this 😄)

Version of dependency-check used
CLI 5.3.2

I am trying to setup a centralized MySQL database, initialized using the corresponding script (on MacOS 10.15, if that helps). The initialization script worked perfectly & threw no errors.

Here is the exact command I ran to update the database
dependency-check.sh --updateonly --connectionString jdbc:mysql://MY_HOST:3306/dependencycheck --dbDriverName com.mysql.jdbc.Driver --dbPassword MY_DB_PWD --dbUser MY_DB_USER

Let me know if I can provide additional info to help figure this out!

[mikehalmamoj]

Sorry to hear about your woes @aravindparappil46 but I don't think we have the same issue. My problem is only with the database cleanup, not the building of the database which hasn't slowed down for me.

For example I see Processing Complete for NVD CVE - 2017 (109127 ms) whereas you see Processing Complete for NVD CVE - 2017 (47856652 ms). That's more than 400 times slower than me.

[jeremylong]

@aravindparappil46 - yes building the centralized database does currently take a long time. No one has had time to investigate this problem.

@mikehalmamoj this issue has already been reported and we have already adjusted the UPDATE_ECOSYSTEM2 statement to be more performant. See #2610

[mikehalmamoj]

Thanks Jeremy - any timescales on the next release?