Dependencycheck: New year bug

Created on 1 Jan 2020  路  37Comments  路  Source: jeremylong/DependencyCheck

Describe the bug
dependency-check cli is failing without the nvd metadata file for the new year

[ERROR] Unable to download meta file: https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2020.meta; received 404 -- resource not found
org.owasp.dependencycheck.data.update.exception.UpdateException: Unable to download meta file: https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2020.meta; received 404 -- resource not found
    at org.owasp.dependencycheck.data.update.NvdCveUpdater.getMetaFile(NvdCveUpdater.java:351)
    at org.owasp.dependencycheck.data.update.NvdCveUpdater.getUpdatesNeeded(NvdCveUpdater.java:398)
    at org.owasp.dependencycheck.data.update.NvdCveUpdater.update(NvdCveUpdater.java:122)
    at org.owasp.dependencycheck.Engine.doUpdates(Engine.java:922)
    at org.owasp.dependencycheck.Engine.initializeAndUpdateDatabase(Engine.java:723)
    at org.owasp.dependencycheck.Engine.analyzeDependencies(Engine.java:653)
    at org.owasp.dependencycheck.App.runScan(App.java:252)
    at org.owasp.dependencycheck.App.run(App.java:184)
    at org.owasp.dependencycheck.App.main(App.java:80)
Caused by: org.owasp.dependencycheck.utils.ResourceNotFoundException: Requested resource does not exists - received a 404
    at org.owasp.dependencycheck.utils.HttpResourceConnection.obtainConnection(HttpResourceConnection.java:204)
    at org.owasp.dependencycheck.utils.HttpResourceConnection.fetch(HttpResourceConnection.java:138)
    at org.owasp.dependencycheck.utils.Downloader.fetchContent(Downloader.java:110)
    at org.owasp.dependencycheck.data.update.NvdCveUpdater.getMetaFile(NvdCveUpdater.java:340)

* Version of dependency-check used *
The problem occurs using version 5.2.4 of the the cli (cli, gradle plugin, maven plugin, etc.)

* Log file *
When reporting errors, 99% of the time log file output is required. Please post the log file as a gist and provide a link in the new issue.

To Reproduce
Steps to reproduce the behavior:
Run a dependency check scan on Jan 1 of a new year.

Expected behavior
Tool should continue without the metadata and cve feed for the new year.

Additional context
Add any other context about the problem here.

bug
Source
picture prabhu
👍69

Most helpful comment

The files for 2020 have just been released: https://nvd.nist.gov/vuln/data-feeds#JSON_FEED
That said: Everything should be working again for you. :sunglasses:

picture GFriedrich on 2 Jan 2020
🎉11 👍7

All 37 comments

https://nvd.nist.gov/vuln/data-feeds#JSON_FEED

^^ This does say meta file is unavailable for CVE 2020 feed.

tfarhan56 picture tfarhan56 on 1 Jan 2020
👍3

Seems it would be a more robust solution to attempt year - 1 on failure.

image

dbwiddis picture dbwiddis on 1 Jan 2020
👍2

Fix was implemented - not sure if this will get published before the NVD has a 2020 data file though.

https://github.com/jeremylong/DependencyCheck/blob/e5218ea3594018284875acb7fb0f257e54c7d969/core/src/main/java/org/owasp/dependencycheck/data/update/NvdCveUpdater.java#L407-L417

jeremylong picture jeremylong on 2 Jan 2020
🎉8 👍1

@jeremylong Seems like a reasonable hack although a bit of configurability would be great (e.g. if we could set the grace period for NVD publishing something each year in case next year they take longer than 10 days).

Can you give us any ETA on this ending up in Central?

matthewlowry picture matthewlowry on 2 Jan 2020

@jeremylong P.S. thanks for getting onto this so quickly.

matthewlowry picture matthewlowry on 2 Jan 2020

Happy new year!

happy new year

dylanwatsonsoftware picture dylanwatsonsoftware on 2 Jan 2020
😄13

Happy new year!

bienkma picture bienkma on 2 Jan 2020

This bug really amused me :p.
So, What is temporary solution for this bug? Change configuration file to 2019?

cnlonelyrain picture cnlonelyrain on 2 Jan 2020

The easiest temporary solution is to do nothing and wait for the updated code to publish to maven and/or NMD to update their data file. It requires no action, and a little bit of patience.

The second easiest temporary solution if you need a one-time action is to temporarily change your computer's clock time to a date in 2019.

The third easiest temporary solution if you need automated CI actions is disabling this plugin until it's updated or NVD updates the data feed.

The fourth easiest solution isn't easy but involves using the nist-data-mirror plugin. I'm going to think about this one for a few days and expect the first solution to work first. ;)

dbwiddis picture dbwiddis on 2 Jan 2020
👍6

Yeah, changing the system clock to some date in 2019 works.

junmuz picture junmuz on 2 Jan 2020
👍1

Fix was implemented - not sure if this will get published before the NVD has a 2020 data file though.

https://github.com/jeremylong/DependencyCheck/blob/e5218ea3594018284875acb7fb0f257e54c7d969/core/src/main/java/org/owasp/dependencycheck/data/update/NvdCveUpdater.java#L407-L417

Hi Jeremylong,

Any timelines when the fix will be released ?

Also, meanwhile is there a way to tweak the configuration or maven command to solve this ?

Regards,
Akeeb K Y

Aaqib041 picture Aaqib041 on 2 Jan 2020

If you use mirrored data, you could temporarily duplicate your 2019 files with 2020 names.

nhenneaux picture nhenneaux on 2 Jan 2020
👍3

@Aaqib041 there are no config options that can bypass this, other than 'skip' and friends. Workarounds are changing your system clock, or establishing a mirror and duplicating 2019 files. Or, wait... I expect this will all be resolved within a day or less.

dbwiddis picture dbwiddis on 2 Jan 2020

There is also a question on stackoverflow on this. I am hoping to keep the discussion here.

My suggestion is to (for in the future) create a separate jobs for the update of the database and the checking. The update job should use the data parameter and updateonly parameter to ensure it retrieves the data and the other jobs should use the data and noupdate parameter. When the update fails you still have a database (inside the data folder) you can check against.

vdotjansen picture vdotjansen on 2 Jan 2020

facing similar issue, waiting for its fix

ansha1 picture ansha1 on 2 Jan 2020
👎5

waiting for updates

chaimaerakhim picture chaimaerakhim on 2 Jan 2020
👎5

facing similar issue, hope it will get fix as soon as possible.

SaralSaxena picture SaralSaxena on 2 Jan 2020
👎5

hoping this will be fixed soon

chhunge picture chhunge on 2 Jan 2020
👎6

Any ETA on when this will be released?

joez-f picture joez-f on 2 Jan 2020
👎4

FWIW, I've managed to get this working by duplicating and renaming the files and updating the sha in the meta files like so:

mv nvdcve-1.0-2019.json.gz nvdcve-1.0-2020.json.gz
mv nvdcve-1.0-2019.meta nvdcve-1.0-2020.meta
gunzip -d nvdcve-1.1-2020.json.gz 
sed 's/2019/2020/g' nvdcve-1.1-2020.json > nvdcve-1.1-2020.json_new
shasum -a 256 nvdcve-1.1-2020.json_new | tr '[:lower:]' '[:upper:]'
mv nvdcve-1.1-2020.json_new nvdcve-1.1-2020.json
gzip nvdcve-1.1-2020.json

Then updating the sha in the meta file with the one produced above and renaming the _new file to the original filename, and uploading to my mirror (Artifactory in my case).

Note, the sed is greedy and could bork some things in the files, but I've not seen any issues so far.

chris-gillatt picture chris-gillatt on 2 Jan 2020

waiting too

jnduongtung picture jnduongtung on 2 Jan 2020
👎6

This is open source people, stop demanding a release, a fix has been commited, a release will be made when Jeremy is ready for it. You can compile it on your own if you need it now. If I was Jeremy I would not rush to release as more changes have been made then just this small fix.
If you want to have a 5.2.4 with just this patch you can take the source code from my fork (or preferably make your own) https://github.com/vdotjansen/DependencyCheck/tree/5.2.4-patched

vdotjansen picture vdotjansen on 2 Jan 2020
👍15

Upvote for a patch release

jasimmk picture jasimmk on 2 Jan 2020
👎4

My apologies for asking a redundant question but in reading the thread I could not find any mention about the Ant version of dependency check - we use both Ant and Maven.

Will the Ant plugin also be updated?

Thanks in advance,
max

BTW: I also notice that in my stack trace I have an SSL handshake exception is it related?:

02-Jan-2020 01:01:14 Caused by: org.owasp.dependencycheck.utils.DownloadFailedException: Error downloading file https://nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-modified.meta; unable to connect. 02-Jan-2020 01:01:14 at org.owasp.dependencycheck.utils.HttpResourceConnection.obtainConnection(HttpResourceConnection.java:238) 02-Jan-2020 01:01:14 at org.owasp.dependencycheck.utils.HttpResourceConnection.fetch(HttpResourceConnection.java:138) 02-Jan-2020 01:01:14 at org.owasp.dependencycheck.utils.Downloader.fetchContent(Downloader.java:110) 02-Jan-2020 01:01:14 ... 24 more 02-Jan-2020 01:01:14 Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target 02-Jan-2020 01:01:14 at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) 02-Jan-2020 01:01:14 at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1946) 02-Jan-2020 01:01:14 at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:316) 02-Jan-2020 01:01:14 at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:310) 02-Jan-2020 01:01:14 at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1639) 02-Jan-2020 01:01:14 at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:223) 02-Jan-2020 01:01:14 at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037) 02-Jan-2020 01:01:14 at sun.security.ssl.Handshaker.process_record(Handshaker.java:965) 02-Jan-2020 01:01:14 at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1064) 02-Jan-2020 01:01:14 at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367) 02-Jan-2020 01:01:14 at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1395) 02-Jan-2020 01:01:14 at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1379) 02-Jan-2020 01:01:14 at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559) 02-Jan-2020 01:01:14 at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185) 02-Jan-2020 01:01:14 at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:162) 02-Jan-2020 01:01:14 at org.owasp.dependencycheck.utils.HttpResourceConnection.obtainConnection(HttpResourceConnection.java:178) 02-Jan-2020 01:01:14 ... 26 more

MMirabito picture MMirabito on 2 Jan 2020

@Aaqib041 there are no config options that can bypass this, other than 'skip' and friends. Workarounds are changing your system clock, or establishing a mirror and duplicating 2019 files. Or, wait... I expect this will all be resolved within a day or less.

@dbwiddis Thank You

Aaqib041 picture Aaqib041 on 2 Jan 2020
👎1

The easiest, temporary fix seems to be setting the "--cveUrlBase" parameter (https://jeremylong.github.io/DependencyCheck/dependency-check-cli/arguments.html) to a hardcoded year until the fix/update has happened.

Mercious picture Mercious on 2 Jan 2020
👍6

@Mercious Thanks for that hint. Worked locally and with Travis CI and JitCi, although GitHub action Java CI had a problem inserting records in the internal database:

Caused by: org.h2.jdbc.JdbcSQLException: Unique index or primary key violation: "CONSTRAINT_INDEX_7 ON PUBLIC.VULNERABILITY(CVE) VALUES ('CVE-2019-5877', 7171)"; SQL statement:
INSERT INTO vulnerability (cve, description, cvssV2Score, cvssV2AccessVector, cvssV2AccessComplexity, cvssV2Authentication, cvssV2ConfidentialityImpact, cvssV2IntegrityImpact, cvssV2AvailabilityImpact, cvssV2Severity, cvssV3AttackVector, cvssV3AttackComplexity, cvssV3PrivilegesRequired, cvssV3UserInteraction, cvssV3Scope, cvssV3ConfidentialityImpact, cvssV3IntegrityImpact, cvssV3AvailabilityImpact, cvssV3BaseScore, cvssV3BaseSeverity) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) [23505-196]
    at org.h2.message.DbException.getJdbcSQLException(DbException.java:345)
    at org.h2.message.DbException.get(DbException.java:179)
    at org.h2.message.DbException.get(DbException.java:155)
    at org.h2.index.BaseIndex.getDuplicateKeyException(BaseIndex.java:103)
    at org.h2.mvstore.db.MVSecondaryIndex.checkUnique(MVSecondaryIndex.java:231)
    at org.h2.mvstore.db.MVSecondaryIndex.add(MVSecondaryIndex.java:190)
    at org.h2.mvstore.db.MVTable.addRow(MVTable.java:705)
    at org.h2.command.dml.Insert.insertRows(Insert.java:156)
    at org.h2.command.dml.Insert.update(Insert.java:114)
    at org.h2.command.CommandContainer.update(CommandContainer.java:101)
    at org.h2.command.Command.executeUpdate(Command.java:260)
    at org.h2.jdbc.JdbcPreparedStatement.execute(JdbcPreparedStatement.java:207)
    at org.owasp.dependencycheck.data.nvdcve.CveDB.updateVulnerabilityInsertVulnerability(CveDB.java:1081)
    ... 10 more
There was an error attempting to close the CveDB, see the log for more details.
Unable to update 1 or more Cached Web DataSource, using local data instead. Results may not include recent vulnerabilities.

Just out of curiosity: I cannot find a comparable new year's problem in the closed issues. Has this never occurred? Did NIST always provide empty files for the next year early on and did not this time?

marco-schmidt picture marco-schmidt on 2 Jan 2020
😄1

The easiest, temporary fix seems to be setting the "--cveUrlBase" parameter (https://jeremylong.github.io/DependencyCheck/dependency-check-cli/arguments.html) to a hardcoded year until the fix/update has happened.

I tried with that parameter, and still getting the same error. I am not sure if I have given it right. To existing command I had added. Please let me know if i have given in wrong format. Is it working for others?
-c--cveUrlBase 2019

vaishravin picture vaishravin on 2 Jan 2020

I tried with that parameter, and still getting the same error. I am not sure if I have given it right. To existing command I had added. Please let me know if i have given in wrong format. Is it working for others?
-c--cveUrlBase 2019

The value of the parameter needs to be the entire link, replacing the "%d" placeholder with the actual year: https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2019.json.gz

Mercious picture Mercious on 2 Jan 2020

Just out of curiosity: I cannot find a comparable new year's problem in the closed issues. Has this never occurred? Did NIST always provide empty files for the next year early on and did not this time?

Because last year the resolution of the CVE DB was switched from XML to JSON feed which in turn required some rewrite of the download code. Thats why this issue popped up these days the first time. :wink:

GFriedrich picture GFriedrich on 2 Jan 2020
👍2 😄1

The easiest, temporary fix seems to be setting the "--cveUrlBase" parameter (https://jeremylong.github.io/DependencyCheck/dependency-check-cli/arguments.html) to a hardcoded year until the fix/update has happened.
I tried with that parameter, and still getting the same error. I am not sure if I have given it right. To existing command I had added. Please let me know if i have given in wrong format. Is it working for others?
-c--cveUrlBase 2019

I tried a slightly different approach since our builds are failing I told maven not to fail the build if the depednecy-check fails by using <failOnError>false</failOnError>

However, i am not having much luck when I apply it to the Ant version failOnError = "false"

Just a temporary solution for us.

Quick Update For anyone else that might find useful while waiting for a new release version. I was able to get the ant version to run by simply setting autoupdate="false" same goes for the maven version <autoUpdate>true</autoUpdate>

I get the following exception:

[dependency-check] SLF4J: Found binding in [jar:file:/D:/MyPrograms/dependency-check-ant/dependency-check-ant.jar!/org/slf4j/impl/StaticLoggerBinder.class] [dependency-check] SLF4J: Found binding in [jar:file:/D:/MyPrograms/dependency-check-ant/lib/dependency-check-ant-5.2.4.jar!/org/slf4j/impl/StaticLoggerBinder.class] ...... BUILD FAILED D:\Git\XOXOXO\build-dependency-cve.xml:68: java.lang.NullPointerException at org.owasp.dependencycheck.Engine.writeReports(Engine.java:1247) at org.owasp.dependencycheck.Engine.writeReports(Engine.java:1201) at org.owasp.dependencycheck.taskdefs.Check.execute(Check.java:1543) at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:292) at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:99) at org.apache.tools.ant.Task.perform(Task.java:350) at org.apache.tools.ant.Target.execute(Target.java:449) at org.apache.tools.ant.Target.performTasks(Target.java:470) at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1391) at org.apache.tools.ant.Project.executeTarget(Project.java:1364) at org.apache.tools.ant.helper.DefaultExecutor.executeTargets(DefaultExecutor.java:41) at org.apache.tools.ant.Project.executeTargets(Project.java:1254) at org.apache.tools.ant.Main.runBuild(Main.java:830) at org.apache.tools.ant.Main.startAnt(Main.java:223) at org.apache.tools.ant.launch.Launcher.run(Launcher.java:284) at org.apache.tools.ant.launch.Launcher.main(Launcher.java:101)

MMirabito picture MMirabito on 2 Jan 2020

Below configuration in pom.xml was working till some time for me

image

Again now ended up with below error

Caused by: org.h2.jdbc.JdbcSQLException: Unique index or primary key violation: "CONSTRAINT_INDEX_7 ON PUBLIC.VULNERABILITY(CVE) VALUES ('CVE-2019-5877', 7171)"; SQL statement:
INSERT INTO vulnerability (cve, description, cvssV2Score, cvssV2AccessVector, cvssV2AccessComplexity, cvssV2Authentication, cvssV2ConfidentialityImpact, cvssV2IntegrityImpact, cvssV2AvailabilityImpact, cvssV2Severity, cvssV3AttackVector, cvssV3AttackComplexity, cvssV3PrivilegesRequired, cvssV3UserInteraction, cvssV3Scope, cvssV3ConfidentialityImpact, cvssV3IntegrityImpact, cvssV3AvailabilityImpact, cvssV3BaseScore, cvssV3BaseSeverity) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) [23505-196]
at org.h2.message.DbException.getJdbcSQLException(DbException.java:345)

between CLI works fine with below arguments

dependency-check core --project "$ProjectName" --scan "$CPROJECT_DIR" --cveUrlModified "https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-modified.json.gz" --cveUrlBase "https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2019.json.gz"

gomathiraj86 picture gomathiraj86 on 2 Jan 2020

Thanks for the <failOnError>false</failOnError> hint - we will use that until the first data is compiled or a new version > 5.2.4 is released.

focbenz picture focbenz on 2 Jan 2020

Do you have a "rough" estimate of when a new release will be available? I have no problem waiting for the next release (just wondering "roughly" how long that would be). Thanks again for jumping on this so quickly.

blueleader07 picture blueleader07 on 2 Jan 2020

The files for 2020 have just been released: https://nvd.nist.gov/vuln/data-feeds#JSON_FEED
That said: Everything should be working again for you. :sunglasses:

GFriedrich picture GFriedrich on 2 Jan 2020
🎉11 👍7

The files for 2020 have just been released: https://nvd.nist.gov/vuln/data-feeds#JSON_FEED
That said: Everything should be working again for you. 馃槑

It's working from me now many thanks
max

MMirabito picture MMirabito on 2 Jan 2020
❤3

As I expected - the NVD 2020 files would be published before we could complete testing on the next release. There is one more important update - so we are going to hold off publishing 5.3.0 until the CPE-Parser PR#30 is merged and released.

jeremylong picture jeremylong on 2 Jan 2020
👍14
Was this page helpful?
0 / 5 - 0 ratings

Related issues

ammy1999 picture ammy1999  路  17Comments
baderbuddy picture baderbuddy  路  58Comments
emartynov picture emartynov  路  24Comments
alexanderkjall picture alexanderkjall  路  21Comments
binkley picture binkley  路  21Comments