Dependencycheck: central analyzer doesn't use our mirror, and maven central is often overloaded

[alexanderkjall]

We have configured the OWASP dependency check to run as part of our build process with maven, but it often fails the build due to not being able to contact maven central.

It seems to contact maven central even if we have our own nexus server configured in a block in the pom file.

The relevant part of the log is:

[ERROR] Could not connect to Central search. Analysis failed.
java.io.IOException: Finally failed connecting to Central search. Giving up after 5 tries.
    at org.owasp.dependencycheck.analyzer.CentralAnalyzer.fetchMavenArtifacts (CentralAnalyzer.java:288)
    at org.owasp.dependencycheck.analyzer.CentralAnalyzer.analyzeDependency (CentralAnalyzer.java:198)
    at org.owasp.dependencycheck.analyzer.AbstractAnalyzer.analyze (AbstractAnalyzer.java:136)
    at org.owasp.dependencycheck.AnalysisTask.call (AnalysisTask.java:88)
    at org.owasp.dependencycheck.AnalysisTask.call (AnalysisTask.java:37)
    at java.util.concurrent.FutureTask.run (FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:624)
    at java.lang.Thread.run (Thread.java:748)
Caused by: java.io.IOException: Could not connect to MavenCentral (503): Service Unavailable: Back-end server is at capacity
    at org.owasp.dependencycheck.data.central.CentralSearch.searchSha1 (CentralSearch.java:194)
    at org.owasp.dependencycheck.analyzer.CentralAnalyzer.fetchMavenArtifacts (CentralAnalyzer.java:266)
    at org.owasp.dependencycheck.analyzer.CentralAnalyzer.analyzeDependency (CentralAnalyzer.java:198)
    at org.owasp.dependencycheck.analyzer.AbstractAnalyzer.analyze (AbstractAnalyzer.java:136)
    at org.owasp.dependencycheck.AnalysisTask.call (AnalysisTask.java:88)
    at org.owasp.dependencycheck.AnalysisTask.call (AnalysisTask.java:37)
    at java.util.concurrent.FutureTask.run (FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:624)
    at java.lang.Thread.run (Thread.java:748)

[focbenz]

Same problem using dependency-check-maven 3.1.2 since two days.

[pwhittlesea]

Same issue here. Maven central appears to be quite flaky the last few days.

It appears that the URL that it looks at is configurable via the analyzer.central.url application property.

I use the Maven plugin and this property isn't exposed via the Maven Mojo configuration options.
But if I run:
sh mvn clean install -Danalyzer.central.url=http://i.dont.exist.com
then I get the failure Caused by: java.net.UnknownHostException: i.dont.exist.com which shows its using the property when running the central analyzer.

Hopefully this helps you point it at your mirror.

[aikebah]

For use with a Nexus mirror there is the NexusAnalyzer, which disables the Central analyzer:
https://jeremylong.github.io/DependencyCheck/analyzers/nexus-analyzer.html
its usage with maven is documented on the maven plugin configuration page:
https://jeremylong.github.io/DependencyCheck/dependency-check-maven/configuration.html

in my experience setting nexusUrl to http(s)://<nexus-host>:<nexus-port>/<nexus-context-path>/service/local even works for Nexus OSS v2 (not only for Nexus Pro as documented in DependencyCheck's configuration).

If nexusUrl is NOT defined the NexusAnalyzer will disable itself. on the other hand if nexusUrl is defined NexusAnalyzer will run and will disable the Central Analyzer (as they are essentially performing the same function: try to resolve the artifact in the Maven repository by searching for a sha1 hash match).

[jeremylong]

If you are using Maven or Gradle the benefits of the Central analyzer are negligible. I would suggest disabling it if using Maven. I am considering making the default for the maven and gradle plugins that the Central Analyzer is disabled.

[jeremylong]

@alexanderkjall can you provide your configuration?

[papidal]

And if we can not use the maven plugins and we get this error, what solution do you propose?

We can't disable it because we would get different and incomplete results

[louisburton]

I can see https://github.com/jeremylong/DependencyCheck/pull/1212 was merged, thx for doing this so quickly - so am guessing we're just waiting for a new release?

[alexanderkjall]

@jeremylong Sorry for the late reply, here is my configuration:

            <plugin>
                <groupId>org.owasp</groupId>
                <artifactId>dependency-check-maven</artifactId>
                <version>3.1.2</version>
                <configuration>
                    <suppressionFiles>
                        <suppressionFile>suppression.xml</suppressionFile>
                    </suppressionFiles>
                    <format>XML</format>
                </configuration>
                <executions>
                    <execution>
                        <goals>
                            <goal>check</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>

[jeremylong]

@alexanderkjall I misread your original question. Dependency-check will always use the Central APIs regardless of setting up an internal repository within the pom.xml. Not all repositories have the same APIs available. If you are using Nexus 2.0 or Nexus 3.0 Pro you can likely switch to using the NexusAnalyzer (see configuration here).

That being said, in the next release of dependency-check the CentralAnalyzer is going to be disabled by default for both the Maven and Gradle plugins. Very little is gained by having this enabled for Maven and Gradle projects.

[nhenneaux]

Very little is gained by having this enabled for Maven and Gradle projects

@jeremylong Could you elaborate a bit about that? What is gained and what is lost by disabling the central analyzer?
Do you know if it is planned to implement an Artifactory analyzer?

[davidkarlsen]

@jeremylong I don't think it will work with nexus 3 as there is no /service/local api any longer: https://issues.sonatype.org/browse/NEXUS-11894

[jeremylong]

@jeremylong It was my understanding the Nexus Pro 3 still had the API.

@nhenneaux the GAV is collected as product/vendor/version evidence from the dependency management framework instead of from Maven Central. Also, if the dependency is found (via hash) to be contained in Central the HTML report will have a green check indicating it is a published version. With regards to Artifactory - it has been a long standing request. However, if you are using Maven or Gradle very little would be gained.

[nhenneaux]

@jeremylong I understand the way the plugin is working but what do you mean by "very little would be gained"? What is the gain? As far as I understood the hash is not checked by Maven (or Gradle) after the download of an artifact. The dependency check plugin then checks there was no one in the middle to change the artifact. Am I missing something?

[jeremylong]

The hash check to central could be used to check for MiTM attacks (it shows up as a green check in the HTML report); but in reality this is more a side affect of the original intention of the Central Analyzer. More importantly, if the GAV information is not available (i.e. you were using the CLI) and the pom.xml was not available - the CentralAnalyzer would add the GAV information. Thus, very little is gained as the Maven plugin already has the GAV information.

[papidal]

@jeremylong And if we not use the maven plugin and we get this error, what solution do you propose?

We can't disable it because we would get different and incomplete results. If I remove from \META-INF\services\org.owasp.dependencycheck.analyzer.Analyzer: org.owasp.dependencycheck.analyzer.CentralAnalyzer I get differents results

[jeremylong]

Consider setting up a Squid proxy to cache the maven results longer term... I've been toying around with a docker image to do this - but I haven't made a ton of headway due to other priorities.

[jeremylong]

I've tried looking for other services that can take a hash of a JAR and return the coordinates (GAV, etc.) - but I haven't found anything better yet.

[papidal]

Thank you, it would be very interesting to integrate with artifactory (we use it as internal repository)
If you see another service similar to maven central, please, let me know

[aikebah]

@jeremylong Have you recently looked into Artifactory's useability from DepCheck? If not I'm offering to take a look. Their issue-tracker indicates that hash-search for artifacts is implemented in their REST API since end of 2010 (https://www.jfrog.com/jira/browse/RTFACT-3676).

[jeremylong]

@aikebah sure - PRs are always welcome. See #60 - a little thought has gone into it. However, no code has been written. There might be something useful in the bintray branch - specifically around trying to convert the path into actual coordinates...

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.