Dependencycheck: Documentation on configuring for use in CI

[binkley]

Thank you for this project!

I have been struggling to work out how best to configure a CI for using DependencyCheck.

For example, I'd like to use the caching feature of GitHub actions with DependencyCheck to avoid redownloading all the CVEs on each build.

1) Is there a shared cache I can refer to?
2) If not; what's the "right" way to set up a shared cache for my project?

Ideally, this would be in the documentation. I've been reading through the source code to do this myself, but have not found the right knobs to tweak.

[mprins]

here's an example: https://github.com/mprins/geoserver/blob/owasp-dep-scan/.github/workflows/owasp-scan.yml

[binkley]

@mprins Thank you! After I opened this issue, I worked out the CVEs and data are stored under ~/.m2/repository, and I'm glad to see that confirmed in your action.

It would be helpful for the documentation to:

1. Note where data is cached by DependencyCheck, or note more prominently
2. Provide some instructions or tips on setting up CI (an open-ended thing, since there are many CI systems out there)

[jeremylong]

@binkley we accept PRs. If you willing to add to the documentation based on your struggles and successes it would benefit the community.

[binkley]

@jeremylong Excellent suggestion, will look at this during the week or this weekend.

I pulled down sources, and see it's a Maven project. Is there any interest in a separate PR to add Maven wrapper? I generally recommend this to clients and teams so the version of Maven used is fixed by the project's configuration, and not reliant on whichever version is installed locally.

https://github.com/takari/maven-wrapper

[jeremylong]

I don't see an issue using a maven-wrapper. On the gradle side we are already using gradelw. This project was started back in 2011 and we just haven't switched over to using the wrapper for maven yet. so sure send a PR.

[binkley]

@jeremylong A punt. Would you prefer this documentation go to?

• README.md, added in toto
• BUILD.md with a link from the Readme, including refactoring (eg, pulling out "Development Usage" section)
• CI.md, separate from other Readme instructions, but with a link from Readme
• the GitHub Wiki for this project

As I thought about the instructions over the week and weekend, I realized the style of presentation depends on the project. I'm looking to make a PR which might be accepted on first pass, modulo typos, etc.

[gmariotti]

Is there any update on this issue? I'm also interested in understanding what I need to cache in Gradle-based projects to speed up the dependencyCheck task

[jeremylong]

I missed the question from @binkley earlier. Honestly, if you are still willing to write up what you've done - I think a CI.md might be best. We can adjust as needed.

[binkley]

@jeremylong I've picked this up again. After forking the project and cloning (in preparation) for a PR, a local build complains:

[INFO] Running org.owasp.dependencycheck.utils.DownloaderIT
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.558 s <<< FAILURE! - in org.owasp.dependencycheck.utils.DownloaderIT
[ERROR] testFetchFile(org.owasp.dependencycheck.utils.DownloaderIT)  Time elapsed: 0.512 s  <<< ERROR!
java.net.MalformedURLException: no protocol: 
        at org.owasp.dependencycheck.utils.DownloaderIT.testFetchFile(DownloaderIT.java:50)

[INFO] Running org.owasp.dependencycheck.utils.URLConnectionFactoryIT
[INFO] 46071 started on port: 46071
[INFO] 46071 resetting all expectations and request logs
Loading JavaScript to validate ECMA262 regular expression in JsonSchema because java.util.regex package in Java does not match ECMA262
Warning: Nashorn engine is planned to be removed from a future JDK release
[INFO] 46071 creating expectation:

Followed by a lot of JSON.

I am setup to use latest JDK 11 and Maven 3.6.3. Am I doing something wrong?

[jeremylong]

I've been building this with java 8 - as we need to support 8+. As you are solely working on documentation I would suggest running only mvn site or possibly mvn package site -DskipTests=true.

[binkley]

@jeremylong My issue is solved using GitHub actions and caching for the Maven local copy of repositories, ie:

      - name: Restore cache
        uses: actions/cache@master
        env:
          cache-name: cache-dependencies
        with:
          path: ~/.gradle
          key: ${{ runner.os }}-build-${{ env.cache-name }}-${{hashFiles('**/build.gradle.kts') }}
          restore-keys: |
            ${{ runner.os }}-build-${{ env.cache-name }}-
            ${{ runner.os }}-build-
            ${{ runner.os }}-

Apologies for the slow reply.

[gmariotti]

Is there any more fine-grained approach other than storing the entire .gradle? For example, https://docs.gradle.org/current/userguide/dependency_resolution.html#sub:cache_copy suggests to only cache three specific folders. Any chance we could do the same with the plugin?

[binkley]

@gmariotti Can you sketch out what that would look like?

I pasted in the recommended pattern for caching Gradle, but it has issues. For example, sharing the daemon or workers directory leads me to sometimes need ./gradlew --stop before I launch the local CI container. Your point is clear to me!

[gmariotti]

Sure, let me share what my team does with GitLab CI/CD

build:
  stage: build
  image: gradle:jre11-slim
  script:
    - ./gradlew check dependencyCheckAggregate
  cache:
    key: $CI_JOB_STAGE
    paths:
      - $GRADLE_USER_HOME/caches/modules-2/files-2.1
      - $GRADLE_USER_HOME/caches/modules-2/metadata-2.*
      - $GRADLE_USER_HOME/caches/modules-2/resources-2.1
      - $GRADLE_USER_HOME/dependency-check-data
  variables:
    GRADLE_USER_HOME: ${CI_PROJECT_DIR}/.gradle
    GRADLE_OPTS: "-Dorg.gradle.daemon=false"

We make sure that, inside the CI, we run without the Gradle daemon and that the Gradle user home is in the same directory of the project. And then, make sure to only cache those folders. I saw locally that DependencyCheck creates a .gradle/dependency-check-data folder, and it seems to speed up the CI a bit by avoiding to download the CSV every time, but I'm not sure it is enough.

[binkley]

@gmariotti Thank you for the cache list! I'll update my own config, and see if my niggles with daemons goes away.

I presently run CI with ./gradlew --no-daemon verify, using GitHub actions for "real" CI, and Batect (https://batect.dev/) for local CI simulation.

[gmariotti]

Hi @jeremylong, just to follow up on this, is there anything more that can be done to speed up the job in the CI? Unfortunately, I don't have data to support it, but I have the feeling that the folders we currently cache are not enough for DependencyCheck

[binkley]

@gmariotti, Howdy! Are you using Gradle or Maven for your build?

I use these settings for GitHub caches which seem to perform OK (I can provide timings if that helps):

      - name: Restore Gradle caches
        uses: actions/cache@master
        env:
          cache-name: gradle-cache
        with:
          path: ~/.gradle
          key: ${{ runner.os }}-build-${{ env.cache-name }}-${{hashFiles('**/build.gradle') }}
          # TODO: See discussion: https://github.com/jeremylong/DependencyCheck/issues/2560#issuecomment-702098748
          restore-keys: |
            ${{ runner.os }}-build-${{ env.cache-name }}-
            ${{ runner.os }}-build-
            ${{ runner.os }}-

      - name: Restore Maven caches
        uses: actions/cache@master
        env:
          cache-name: maven-cache
        with:
          path: ~/.m2/repository
          key: ${{ runner.os }}-build-${{ env.cache-name }}-${{ hashFiles('**/pom.xml') }}
          restore-keys: |
            ${{ runner.os }}-build-${{ env.cache-name }}-
            ${{ runner.os }}-build-
            ${{ runner.os }}-

[gmariotti]

I'm using Gradle. The problem with caching the entire .gradle folder is that there is a lot of "garbage" coming from Gradle that you take around between builds. In the past, I had issues with it in the Gitlab CI because of permissions and how big the folder became, that's why I'd like to avoid it. I was hoping that, apart for dependency-check-data/, there was something else worth caching that could speed up the process.

[binkley]

@gmariotti Do you have a recommended YAML block in the GH action for Gradle?

[gmariotti]

Sorry @binkley, unfortunately, I never used GH actions, so I don't have any recommendation for it, but I think that the GitLab example can probably be rewritten for GH actions too

[binkley]

Answered in the issue chat. I'll issue a new issue if something unrelated turns up. Love the project :)