Dependencycheck: Ant-Task check fails to remove temp.dir/dctemp<...> folders

[cstsw]

When running the check Ant-task as of version 3.1.1 for each build there remains a dctemp<...>/-folder containing:

• the complete dc.h2.db (of the same size as the potentially updated dc.h2.db in datadirectory)
• GKA.exe
• GKA.exe.config

The folders remain even when the database is considered valid (i.e. the time elapsed since the last update is lower than cveValidForHours).

The result is that temp.dir gets cluttered with a lot of dependency-check-databases of currently about 374MB and its partition potentially runs out of memory after some time (if not purged via e.g. cron).

[putz612]

Going to expand on this one a bit as I think its a bit more then Ant-task. I am running on Windows 2016 using a Jenkins job and every run I get a new temp\dctemp.....\folder with a dc.h2.db file that's about 400MB. Does not take much of those to fill up the filesystem.

[jeremylong]

Thanks for the report. I'll have to run some additional tests - but I honestly don't use the Jenkins or ant task myself very often (outside of testing for development) . @stevespringett have you seen this issue with Jenkins?

[thelateperseus]

This issue also affects the Gradle task, both when I run it on my Windows PC and when we run it on our QuickBuild build server (also Windows).

I ran gradlew dependencyCheckAnalyze --debug and found these lines in the output:

09:17:09.755 [DEBUG] [org.owasp.dependencycheck.utils.FileUtils] Temporary directory is `C:\Users\me\AppData\Local\Temp\dctemp344dafe9-f9ab-4eba-993b-5a373e65a27e`
09:17:09.914 [DEBUG] [org.owasp.dependencycheck.Engine] copying complete 'C:\Users\me\AppData\Local\Temp\dctemp344dafe9-f9ab-4eba-993b-5a373e65a27e'
09:17:09.914 [DEBUG] [org.owasp.dependencycheck.utils.Settings] Setting: data.directory='C:\Users\me\AppData\Local\Temp\dctemp344dafe9-f9ab-4eba-993b-5a373e65a27e'
09:17:09.914 [DEBUG] [org.owasp.dependencycheck.utils.Settings] Setting: data.connection_string='jdbc:h2:file:%s;MV_STORE=FALSE;AUTOCOMMIT=ON;LOG=0;CACHE_SIZE=65536;ACCESS_MODE_DATA=r'
09:17:09.914 [DEBUG] [org.owasp.dependencycheck.data.nvdcve.ConnectionFactory] Loading driver 'org.h2.Driver'
09:17:09.914 [DEBUG] [org.owasp.dependencycheck.utils.Settings] Settings.getDataFile() - file: 'C:\Users\me\AppData\Local\Temp\dctemp344dafe9-f9ab-4eba-993b-5a373e65a27e'
09:17:09.915 [DEBUG] [org.owasp.dependencycheck.utils.Settings] Data directory: C:\Users\me\AppData\Local\Temp\dctemp344dafe9-f9ab-4eba-993b-5a373e65a27e
09:17:09.915 [DEBUG] [org.owasp.dependencycheck.utils.Settings] Connection String: 'jdbc:h2:file:C:\Users\me\AppData\Local\Temp\dctemp344dafe9-f9ab-4eba-993b-5a373e65a27e\dc;MV_STORE=FALSE;AUTOCOMMIT=ON;LOG=0;CACHE_SIZE=65536;ACCESS_MODE_DATA=r'
09:17:09.915 [DEBUG] [org.owasp.dependencycheck.utils.Settings] Settings.getDataFile() - file: 'C:\Users\me\AppData\Local\Temp\dctemp344dafe9-f9ab-4eba-993b-5a373e65a27e'
09:17:09.915 [DEBUG] [org.owasp.dependencycheck.data.nvdcve.ConnectionFactory] Need to create DB Structure: false
09:17:09.915 [DEBUG] [org.owasp.dependencycheck.data.nvdcve.ConnectionFactory] Loading database connection
09:17:09.915 [DEBUG] [org.owasp.dependencycheck.data.nvdcve.ConnectionFactory] Connection String: jdbc:h2:file:C:\Users\me\AppData\Local\Temp\dctemp344dafe9-f9ab-4eba-993b-5a373e65a27e\dc;MV_STORE=FALSE;AUTOCOMMIT=ON;LOG=0;CACHE_SIZE=65536;ACCESS_MODE_DATA=r
09:17:09.915 [DEBUG] [org.owasp.dependencycheck.data.nvdcve.ConnectionFactory] Database User: dcuser
09:17:09.928 [DEBUG] [org.owasp.dependencycheck.data.nvdcve.CveDB] Database product: h2
09:17:09.931 [DEBUG] [org.owasp.dependencycheck.utils.H2DBLock] Lock released (Task worker for ':' Thread 4) 5de80affb30d1c2622395ac8336d9992 @ 2018-04-13 09:17:09.931
...
09:17:42.947 [DEBUG] [org.owasp.dependencycheck.utils.Settings] Deleting ALL temporary files from `C:\Users\me\AppData\Local\Temp\dctemp344dafe9-f9ab-4eba-993b-5a373e65a27e`
09:17:42.947 [DEBUG] [org.owasp.dependencycheck.utils.FileUtils] Failed to delete file: C:\Users\me\AppData\Local\Temp\dctemp344dafe9-f9ab-4eba-993b-5a373e65a27e; attempting to delete on exit.

Is it possible that the H2 database isn't being closed before the deletion of the file is attempted?

[jeremylong]

That is a possibility.. For the gradle side - can you test disabling the daemon?

[thelateperseus]

@jeremylong I just ran the same gradle command with the --no-daemon command line argument and got the same "Failed to delete file" log line.

[thelateperseus]

As a temporary workaround, I created a daily scheduled task on our build server that deletes the dctemp* files:

@echo off
for /d %%G in ("C:\Users\User\AppData\Local\Temp\dctemp*") do rd /s /q "%%~G"

[jeremylong]

Just realized the issue for Ant - a cleanup step was missing in the Ant task. Sorry about that. It will be included in the next release.

@thelateperseus please see https://github.com/jeremylong/dependency-check-gradle/issues/80. The fix for gradle will be in the next release.

[cstsw]

Thanks, Jeremy, for the confirmation and the upcoming fix!

[oosterholt]

This issue is still in v3.1.2. When will this fix be released?

[jeremylong]

This will be included in the 3.1.3 release. We are going through the final checks and cleanup now - so this should be released this weekend.

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.