Dependencycheck: NPE on 5.0.0-M2

Created on 22 Mar 2019 · 6Comments · Source: jeremylong/DependencyCheck

Describe the bug
Running on a jenkins server, get this exception:
/scratch/projects/devtasks/build.xml:407: java.lang.NullPointerException

[2019-03-22T14:08:22.385Z] at org.owasp.dependencycheck.data.update.NvdCveUpdater.performUpdate(NvdCveUpdater.java:292)

[2019-03-22T14:08:22.385Z] at org.owasp.dependencycheck.data.update.NvdCveUpdater.update(NvdCveUpdater.java:122)

[2019-03-22T14:08:22.385Z] at org.owasp.dependencycheck.Engine.doUpdates(Engine.java:916)

[2019-03-22T14:08:22.386Z] at org.owasp.dependencycheck.Engine.initializeAndUpdateDatabase(Engine.java:718)

[2019-03-22T14:08:22.386Z] at org.owasp.dependencycheck.Engine.analyzeDependencies(Engine.java:646)

[2019-03-22T14:08:22.386Z] at org.owasp.dependencycheck.taskdefs.Check.execute(Check.java:1265)

[2019-03-22T14:08:22.386Z] at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:292)

[2019-03-22T14:08:22.386Z] at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)

[2019-03-22T14:08:22.386Z] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

[2019-03-22T14:08:22.386Z] at java.lang.reflect.Method.invoke(Method.java:498)

[2019-03-22T14:08:22.386Z] at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)

[2019-03-22T14:08:22.386Z] at org.apache.tools.ant.Task.perform(Task.java:348)

[2019-03-22T14:08:22.386Z] at org.apache.tools.ant.Target.execute(Target.java:435)

[2019-03-22T14:08:22.386Z] at org.apache.tools.ant.Target.performTasks(Target.java:456)

[2019-03-22T14:08:22.386Z] at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1393)

[2019-03-22T14:08:22.386Z] at org.apache.tools.ant.Project.executeTarget(Project.java:1364)

[2019-03-22T14:08:22.386Z] at org.apache.tools.ant.helper.DefaultExecutor.executeTargets(DefaultExecutor.java:41)

[2019-03-22T14:08:22.386Z] at org.apache.tools.ant.Project.executeTargets(Project.java:1248)

[2019-03-22T14:08:22.386Z] at org.apache.tools.ant.Main.runBuild(Main.java:851)

[2019-03-22T14:08:22.386Z] at org.apache.tools.ant.Main.startAnt(Main.java:235)

[2019-03-22T14:08:22.386Z] at org.apache.tools.ant.launch.Launcher.run(Launcher.java:280)

[2019-03-22T14:08:22.386Z] at org.apache.tools.ant.launch.Launcher.main(Launcher.java:109)

* Version of dependency-check used *
ant 5.0.0-M2

* Log file *
jenkins deleted it, i'll try to get it next time it happens

To Reproduce
running ant task

Expected behavior
no NPE

Additional context
jenkins

bug

Source

mebigfatguy

👍1

All 6 comments

I'm seeing the same in the Maven plugin:

[main] [ERROR] Failed to execute goal org.owasp:dependency-check-maven:5.0.0-M2:aggregate (default-cli) on project foo: Execution default-cli of goal org.owasp:dependency-check-maven:5.0.0-M2:aggregate failed. NullPointerException -> [Help 1]

org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.owasp:dependency-check-maven:5.0.0-M2:aggregate (default-cli) on project auth-service-parent: Execution default-cli of goal org.owasp:dependency-check-maven:5.0.0-M2:aggregate failed.

    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:213)

    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:154)

    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:146)

    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)

    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)

    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)

    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)

    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)

    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)

    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)

    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:954)

    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:288)

    at org.apache.maven.cli.MavenCli.main (MavenCli.java:192)

    at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)

    at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)

    at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)

    at java.lang.reflect.Method.invoke (Method.java:498)

    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:289)

    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:229)

    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:415)

    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:356)

Caused by: org.apache.maven.plugin.PluginExecutionException: Execution default-cli of goal org.owasp:dependency-check-maven:5.0.0-M2:aggregate failed.

    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:148)

    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:208)

    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:154)

    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:146)

    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)

    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)

    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)

    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)

    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)

    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)

    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)

    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:954)

    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:288)

    at org.apache.maven.cli.MavenCli.main (MavenCli.java:192)

    at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)

    at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)

    at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)

    at java.lang.reflect.Method.invoke (Method.java:498)

    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:289)

    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:229)

    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:415)

    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:356)

Caused by: java.lang.NullPointerException

    at org.owasp.dependencycheck.data.update.NvdCveUpdater.performUpdate (NvdCveUpdater.java:292)

    at org.owasp.dependencycheck.data.update.NvdCveUpdater.update (NvdCveUpdater.java:122)

    at org.owasp.dependencycheck.Engine.doUpdates (Engine.java:916)

    at org.owasp.dependencycheck.Engine.initializeAndUpdateDatabase (Engine.java:718)

    at org.owasp.dependencycheck.Engine.analyzeDependencies (Engine.java:646)

    at org.owasp.dependencycheck.maven.BaseDependencyCheckMojo.runCheck (BaseDependencyCheckMojo.java:1284)

    at org.owasp.dependencycheck.maven.BaseDependencyCheckMojo.execute (BaseDependencyCheckMojo.java:697)

    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)

    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:208)

    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:154)

    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:146)

    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)

    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)

    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)

    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)

    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)

    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)

    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)

    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:954)

    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:288)

    at org.apache.maven.cli.MavenCli.main (MavenCli.java:192)

    at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)

    at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)

    at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)

    at java.lang.reflect.Method.invoke (Method.java:498)

    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:289)

    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:229)

    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:415)

    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:356)

Log:

[main] [INFO] --- dependency-check-maven:5.0.0-M2:aggregate (default-cli) @ foo ---

[main] [INFO] Central analyzer disabled

[main] [INFO] Found snapshot reactor project in aggregate for com.acme:foo-service:1.12.0-SNAPSHOT - creating a virtual dependency as the snapshot found in the repository may contain outdated dependencies.

[main] [INFO] Checking for updates

[main] [INFO] NVD CVE requires several updates; this could take a couple of minutes.

[pool-5-thread-6] [INFO] Download Started for NVD CVE - 2002

[pool-5-thread-4] [INFO] Download Started for NVD CVE - 2003

[pool-5-thread-2] [INFO] Download Started for NVD CVE - 2004

[pool-5-thread-1] [INFO] Download Started for NVD CVE - 2005

[pool-5-thread-3] [INFO] Download Started for NVD CVE - 2006

[pool-5-thread-5] [INFO] Download Started for NVD CVE - 2007

[pool-5-thread-4] [INFO] Download Complete for NVD CVE - 2003  (1408 ms)

[pool-5-thread-4] [INFO] Download Started for NVD CVE - 2008

[pool-4-thread-1] [INFO] Processing Started for NVD CVE - 2003

[pool-5-thread-2] [INFO] Download Complete for NVD CVE - 2004  (1586 ms)

[pool-4-thread-2] [INFO] Processing Started for NVD CVE - 2004

[pool-5-thread-2] [INFO] Download Started for NVD CVE - 2009

[pool-5-thread-6] [INFO] Download Complete for NVD CVE - 2002  (2515 ms)

[pool-5-thread-3] [INFO] Download Complete for NVD CVE - 2006  (2525 ms)

[pool-5-thread-6] [INFO] Download Started for NVD CVE - 2010

[pool-5-thread-1] [INFO] Download Complete for NVD CVE - 2005  (2813 ms)

[pool-4-thread-3] [INFO] Processing Started for NVD CVE - 2002

[pool-5-thread-5] [INFO] Download Complete for NVD CVE - 2007  (2851 ms)

[pool-5-thread-2] [INFO] Download Complete for NVD CVE - 2009  (1275 ms)

[pool-5-thread-3] [INFO] Download Started for NVD CVE - 2011

[pool-4-thread-4] [INFO] Processing Started for NVD CVE - 2006

[pool-5-thread-2] [INFO] Download Started for NVD CVE - 2012

[pool-5-thread-5] [INFO] Download Started for NVD CVE - 2013

[pool-5-thread-1] [INFO] Download Started for NVD CVE - 2014

[pool-5-thread-6] [INFO] Download Complete for NVD CVE - 2010  (1222 ms)

[pool-5-thread-6] [INFO] Download Started for NVD CVE - 2015

[pool-5-thread-5] [INFO] Download Complete for NVD CVE - 2013  (992 ms)

[pool-5-thread-5] [INFO] Download Started for NVD CVE - 2016

[pool-5-thread-4] [INFO] Download Complete for NVD CVE - 2008  (3470 ms)

[pool-5-thread-4] [INFO] Download Started for NVD CVE - 2017

[pool-5-thread-2] [INFO] Download Complete for NVD CVE - 2012  (1946 ms)

[pool-5-thread-2] [INFO] Download Started for NVD CVE - 2018

[pool-5-thread-4] [WARNING] Download Failed for NVD CVE - 2017

Some CVEs may not be reported.

[pool-5-thread-4] [INFO] If you are behind a proxy you may need to configure dependency-check to use the proxy.

[pool-5-thread-4] [INFO] Download Started for NVD CVE - 2019

[pool-5-thread-6] [INFO] Download Complete for NVD CVE - 2015  (1459 ms)

[pool-5-thread-2] [WARNING] Download Failed for NVD CVE - 2018

Some CVEs may not be reported.

[pool-5-thread-2] [INFO] If you are behind a proxy you may need to configure dependency-check to use the proxy.

[pool-5-thread-4] [WARNING] Download Failed for NVD CVE - 2019

Some CVEs may not be reported.

[pool-5-thread-4] [INFO] If you are behind a proxy you may need to configure dependency-check to use the proxy.

[pool-5-thread-1] [INFO] Download Complete for NVD CVE - 2014  (2447 ms)

[pool-5-thread-3] [INFO] Download Complete for NVD CVE - 2011  (2652 ms)

[pool-5-thread-5] [INFO] Download Complete for NVD CVE - 2016  (3289 ms)

Based on that trace and the log messages. I think we are seeing null given from DownloadTask https://github.com/jeremylong/DependencyCheck/blob/ef39d0cd6dbe885ef865b0a3d43964ad97d39092/core/src/main/java/org/owasp/dependencycheck/data/update/nvd/DownloadTask.java#L147-L148
Then it looks like a null will need to be accounted for here.
https://github.com/jeremylong/DependencyCheck/blob/ef39d0cd6dbe885ef865b0a3d43964ad97d39092/core/src/main/java/org/owasp/dependencycheck/data/update/NvdCveUpdater.java#L279

reardonm on 25 Mar 2019

👍1

I was able to reproduce this problem with debug logging enabled. It seems that our root cause is an intermittent 403 status back from NIST from some URLs. Am I being rate limited?

[pool-5-thread-5] [WARNING] Download Failed for NVD CVE - 2019

Some CVEs may not be reported.

[pool-5-thread-5] [INFO] If you are behind a proxy you may need to configure dependency-check to use the proxy.

[pool-5-thread-5] [DEBUG] 

org.owasp.dependencycheck.utils.DownloadFailedException: Download failed, unable to copy 'https://nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2019.json.gz' to '/tmp/dctempa5bbb58e-6bf7-4959-911c-8e7ae26b9273/cve2019_3053141337773532689.json.gz'

    at org.owasp.dependencycheck.utils.Downloader.fetchFile (Downloader.java:80)

    at org.owasp.dependencycheck.utils.Downloader.fetchFile (Downloader.java:60)

    at org.owasp.dependencycheck.data.update.nvd.DownloadTask.call (DownloadTask.java:128)

    at org.owasp.dependencycheck.data.update.nvd.DownloadTask.call (DownloadTask.java:41)

    at java.util.concurrent.FutureTask.run (FutureTask.java:266)

    at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1149)

    at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:624)

    at java.lang.Thread.run (Thread.java:748)

Caused by: org.owasp.dependencycheck.utils.DownloadFailedException: Error downloading file https://nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2019.json.gz; unable to connect.

    at org.owasp.dependencycheck.utils.HttpResourceConnection.obtainConnection (HttpResourceConnection.java:217)

    at org.owasp.dependencycheck.utils.HttpResourceConnection.fetch (HttpResourceConnection.java:134)

    at org.owasp.dependencycheck.utils.Downloader.fetchFile (Downloader.java:76)

    at org.owasp.dependencycheck.utils.Downloader.fetchFile (Downloader.java:60)

    at org.owasp.dependencycheck.data.update.nvd.DownloadTask.call (DownloadTask.java:128)

    at org.owasp.dependencycheck.data.update.nvd.DownloadTask.call (DownloadTask.java:41)

    at java.util.concurrent.FutureTask.run (FutureTask.java:266)

    at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1149)

    at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:624)

    at java.lang.Thread.run (Thread.java:748)

Caused by: org.owasp.dependencycheck.utils.DownloadFailedException: Error retrieving https://nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2019.json.gz; received response code 403.

    at org.owasp.dependencycheck.utils.HttpResourceConnection.obtainConnection (HttpResourceConnection.java:199)

    at org.owasp.dependencycheck.utils.HttpResourceConnection.fetch (HttpResourceConnection.java:134)

    at org.owasp.dependencycheck.utils.Downloader.fetchFile (Downloader.java:76)

    at org.owasp.dependencycheck.utils.Downloader.fetchFile (Downloader.java:60)

    at org.owasp.dependencycheck.data.update.nvd.DownloadTask.call (DownloadTask.java:128)

    at org.owasp.dependencycheck.data.update.nvd.DownloadTask.call (DownloadTask.java:41)

    at java.util.concurrent.FutureTask.run (FutureTask.java:266)

    at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1149)

    at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:624)

    at java.lang.Thread.run (Thread.java:748)

[pool-5-thread-4] [INFO] Download Complete for NVD CVE - 2015  (2773 ms)

[pool-5-thread-6] [INFO] Download Complete for NVD CVE - 2014  (2905 ms)

[pool-5-thread-1] [INFO] Download Complete for NVD CVE - 2017  (1270 ms)

[pool-5-thread-3] [INFO] Download Complete for NVD CVE - 2011  (4761 ms)

[pool-5-thread-2] [INFO] Download Complete for NVD CVE - 2018  (1840 ms)

reardonm on 26 Mar 2019

On the rate limiting thought, Between Oct 19th and Oct 25th last year NIST changed the feed site to mention a recommend rate limit:

https://nvd.nist.gov/vuln/data-feeds
How to keep up-to-date with the NVD data
The main vulnerability feeds provide CVE® data organized by the first four digits of a CVE® identifier except for the 2002 feeds which include vulnerabilities prior to and including "CVE-2002-". Each feed is updated only if the content of that feed has changed. For example the 2004 feeds will be updated only if there is an addition or modification to any vulnerability with a starting CVE® identifier of "CVE-2004-". In addition, the "recent" feeds are a list of recently published vulnerabilities and the "modified" feeds are a list of recently published and modified vulnerabilities where "recently" and "modified" are defined as the previous eight days. These feeds are updated approximately every two hours.

If you are locally mirroring the NVD data, the data feeds should be used to stay synchronized. After performing a one-time import of the complete data set using the compressed XML/JSON vulnerability feeds, the "modified" feeds should be used to keep up-to-date. The META file should be used to determine if a given feed has been updated since your last import. This helps prevent unnecessary downloads of the .zip or .gz files and should result in a reasonable use of less than 200 requests per day.

https://web.archive.org/web/20181019002940/https://nvd.nist.gov/vuln/data-feeds
https://web.archive.org/web/20181025232009/https://nvd.nist.gov/vuln/data-feeds

I wonder if this just went beyond being recommended and started being enforced. I need to check across our builds to see.

Update: We contacted NVD, using IP address and time of day of a failure we provided they confirmed we are encountering rate limits. Implemented a central cache a few days ago and everything has been great.

freylingb on 26 Mar 2019

👍1

IMO, all dependency-check users should be using a mirror they keep up-to-date daily. The NVD will restrict traffic when there is capacity issues, they will go down occasionally which can break build pipelines, and the NVD does not have unlimited capacity and a small staff.

Just last week they had certificate issues for part of their infrastructure which failed cert validation when attempting to download the feeds.

https://jeremylong.github.io/DependencyCheck/data/mirrornvd.html

stevespringett on 27 Mar 2019

👍1

Same issue here. Can we have a release please?

jjYBdx4IL on 16 Apr 2019

We are updating ODC (see https://github.com/jeremylong/DependencyCheck/pull/1863) to use the meta data files. Additionally, to support the change - we updated the nist-data-mirror project to mirror the meta data files. Combined, these updates will reduce the number of requests to the NVD (used independently or in combination).