Trivy: Trivy doesn't cache db in "image" mode

[pznamensky]

Description
It looks like trivy doesn't cache its database that's why after several checks github blocks db downloading.

[2020-10-05T03:51:11.958Z] 2020-10-05T06:51:10.602+0300 INFO    Need to update DB
[2020-10-05T03:51:11.958Z] 2020-10-05T06:51:10.602+0300 INFO    Downloading DB...
[2020-10-05T03:51:11.958Z] 2020-10-05T06:51:10.659+0300 FATAL   failed to download vulnerability DB: failed to download vulnerability DB: failed to list releases: GET https://api.github.com/repos/aquasecurity/trivy-db/releases: 403 API rate limit exceeded for 93.190.136.169. (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.) [rate reset in 51m11s]

I know that it is possible to use api key as a workaround for this issue. But it would be very helpful to use cache instead.

What did you expect to happen?
Trivy uses cached db until it expires.

What happened instead?
Trivy downloads its db every check.

Output of the first run (without .cache/trivy dir) with -debug:

user@tasks1 ~ $ trivy -d image --exit-code 1 --no-progress --ignore-unfixed --severity CRITICAL internal-registry/some/image:master
2020-10-05T12:04:51.285+0300    DEBUG   Severities: CRITICAL
2020-10-05T12:04:51.295+0300    DEBUG   cache dir:  /home/user/.cache/trivy
2020-10-05T12:04:51.295+0300    DEBUG   There is no valid metadata file: unable to open a file: open /home/user/.cache/trivy/db/metadata.json: no such file or directory
2020-10-05T12:04:51.295+0300    INFO    Need to update DB
2020-10-05T12:04:51.296+0300    INFO    Downloading DB...
2020-10-05T12:04:51.296+0300    DEBUG   no metadata file
2020-10-05T12:04:51.371+0300    DEBUG   release name: v1-2020100100
2020-10-05T12:04:51.371+0300    DEBUG   asset name: trivy-light-offline.db.tgz
2020-10-05T12:04:51.371+0300    DEBUG   file name doesn't match
2020-10-05T12:04:51.371+0300    DEBUG   asset name: trivy-light.db.gz
2020-10-05T12:04:51.371+0300    DEBUG   file name doesn't match
2020-10-05T12:04:51.371+0300    DEBUG   asset name: trivy-offline.db.tgz
2020-10-05T12:04:51.371+0300    DEBUG   file name doesn't match
2020-10-05T12:04:51.371+0300    DEBUG   asset name: trivy.db.gz
2020-10-05T12:04:51.383+0300    DEBUG   asset URL: https://github-production-release-asset-2e65be.s3.amazonaws.com/216830441/8100ad80-037e-11eb-95aa-339b1a4f7d29?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20201005%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20201005T090414Z&X-Amz-Expires=300&X-Amz-Signature=f4876887af3574b19a06064e804907ddcf54f03b3dd1f54bb72fc93d786ca49d&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=216830441&response-content-disposition=attachment%3B%20filename%3Dtrivy.db.gz&response-content-type=application%2Foctet-stream
2020-10-05T12:04:53.486+0300    DEBUG   Updating database metadata...
2020-10-05T12:04:53.488+0300    DEBUG   DB Schema: 1, Type: 1, UpdatedAt: 2020-10-01 00:36:30.496570788 +0000 UTC, NextUpdate: 2020-10-01 12:36:30.496570388 +0000 UTC
2020-10-05T12:04:59.646+0300    DEBUG   Vulnerability type:  [os library]
2020-10-05T12:05:07.010+0300    DEBUG   Artifact ID: sha256:4f0e8b99ebf2e9c287a66bfd07ad476c643483f8fde6b9d104332f1136d431a0
2020-10-05T12:05:07.012+0300    DEBUG   Blob IDs: [sha256:613be09ab3c0860a5216936f412f09927947012f86bfa89b263dfa087a725f81 sha256:6e9e69112e314be909a2eb2ec84fcec50b75e60c7d49683f447172f3d4b74179 sha256:40b9d0cc8557e959846bbe79ba76853117bee1c15f8c8f1413e3e199ae6d473b sha256:2161476cf924c8fa4e2b73dca10b888496c7af92f41a00b44aad895fd85f709b sha256:40679b59771f6205e900311d81fb83453e2971ac4f6eca61d1bf6e4d14015fe7 sha256:012b8b7394a33be8d7c973ff6a9fcf15f9d3baa6c3054ceaaeb4039c27d58b0d sha256:9c6ca2e2ef7e63651c92672496201052cc3d7441776618489868e916c50b269b sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef sha256:178bd6f70eca8d7d18ed898b586b7e96d85f4b59671a64fa0bd4540fbc0e8b7a sha256:6adfc6bed4316444e52573f8dc1789c57e8e1e7ee7dc096a5d554a838d4e9b61 sha256:b01ecf879b6bdaa2d302ad9bbd556cfb67ac3478d8e1861767634e523c504a78]
2020-10-05T12:05:07.020+0300    INFO    Detecting RHEL/CentOS vulnerabilities...
2020-10-05T12:05:07.020+0300    DEBUG   redhat: os version: 7
2020-10-05T12:05:07.021+0300    DEBUG   redhat: the number of packages: 216
2020-10-05T12:05:07.051+0300    DEBUG   Detecting library vulnerabilities, path: srv/my-app/Gemfile.lock
2020-10-05T12:05:07.051+0300    INFO    Detecting ruby vulnerabilities...

internal-registry/some/image:master (centos 7.8.2003)
=============================================================================================================
Total: 0 (CRITICAL: 0)


srv/my-app/Gemfile.lock
=============================================
Total: 0 (CRITICAL: 0)

Output of the second and the following runs with -debug:

user@tasks1 ~ $ trivy -d image --exit-code 1 --no-progress --ignore-unfixed --severity CRITICAL internal-registry/some/image:master
2020-10-05T12:05:09.769+0300    DEBUG   Severities: CRITICAL
2020-10-05T12:05:09.789+0300    DEBUG   cache dir:  /home/user/.cache/trivy
2020-10-05T12:05:09.789+0300    INFO    Need to update DB
2020-10-05T12:05:09.789+0300    INFO    Downloading DB...
2020-10-05T12:05:09.876+0300    DEBUG   release name: v1-2020100100
2020-10-05T12:05:09.876+0300    DEBUG   asset name: trivy-light-offline.db.tgz
2020-10-05T12:05:09.877+0300    DEBUG   file name doesn't match
2020-10-05T12:05:09.878+0300    DEBUG   asset name: trivy-light.db.gz
2020-10-05T12:05:09.879+0300    DEBUG   file name doesn't match
2020-10-05T12:05:09.879+0300    DEBUG   asset name: trivy-offline.db.tgz
2020-10-05T12:05:09.879+0300    DEBUG   file name doesn't match
2020-10-05T12:05:09.879+0300    DEBUG   asset name: trivy.db.gz
2020-10-05T12:05:09.891+0300    DEBUG   asset URL: https://github-production-release-asset-2e65be.s3.amazonaws.com/216830441/8100ad80-037e-11eb-95aa-339b1a4f7d29?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20201005%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20201005T090414Z&X-Amz-Expires=300&X-Amz-Signature=f4876887af3574b19a06064e804907ddcf54f03b3dd1f54bb72fc93d786ca49d&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=216830441&response-content-disposition=attachment%3B%20filename%3Dtrivy.db.gz&response-content-type=application%2Foctet-stream
2020-10-05T12:05:12.849+0300    DEBUG   Updating database metadata...
2020-10-05T12:05:12.850+0300    DEBUG   DB Schema: 1, Type: 1, UpdatedAt: 2020-10-01 00:36:30.496570788 +0000 UTC, NextUpdate: 2020-10-01 12:36:30.496570388 +0000 UTC
2020-10-05T12:05:12.926+0300    DEBUG   Vulnerability type:  [os library]
2020-10-05T12:05:12.953+0300    DEBUG   Artifact ID: sha256:4f0e8b99ebf2e9c287a66bfd07ad476c643483f8fde6b9d104332f1136d431a0
2020-10-05T12:05:12.954+0300    DEBUG   Blob IDs: [sha256:613be09ab3c0860a5216936f412f09927947012f86bfa89b263dfa087a725f81 sha256:6e9e69112e314be909a2eb2ec84fcec50b75e60c7d49683f447172f3d4b74179 sha256:40b9d0cc8557e959846bbe79ba76853117bee1c15f8c8f1413e3e199ae6d473b sha256:2161476cf924c8fa4e2b73dca10b888496c7af92f41a00b44aad895fd85f709b sha256:40679b59771f6205e900311d81fb83453e2971ac4f6eca61d1bf6e4d14015fe7 sha256:012b8b7394a33be8d7c973ff6a9fcf15f9d3baa6c3054ceaaeb4039c27d58b0d sha256:9c6ca2e2ef7e63651c92672496201052cc3d7441776618489868e916c50b269b sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef sha256:178bd6f70eca8d7d18ed898b586b7e96d85f4b59671a64fa0bd4540fbc0e8b7a sha256:6adfc6bed4316444e52573f8dc1789c57e8e1e7ee7dc096a5d554a838d4e9b61 sha256:b01ecf879b6bdaa2d302ad9bbd556cfb67ac3478d8e1861767634e523c504a78]
2020-10-05T12:05:12.960+0300    INFO    Detecting RHEL/CentOS vulnerabilities...
2020-10-05T12:05:12.960+0300    DEBUG   redhat: os version: 7
2020-10-05T12:05:12.960+0300    DEBUG   redhat: the number of packages: 216
2020-10-05T12:05:12.987+0300    DEBUG   Detecting library vulnerabilities, path: srv/my-app/Gemfile.lock
2020-10-05T12:05:12.988+0300    INFO    Detecting ruby vulnerabilities...

internal-registry/some/image:master (centos 7.8.2003)
=============================================================================================================
Total: 0 (CRITICAL: 0)


srv/my-app/Gemfile.lock
=============================================
Total: 0 (CRITICAL: 0)

Output of trivy -v:

Version: 0.11.0

[simar7]

hi @pznamensky – thanks for the report. I tried to reproduce it locally but couldn't so I'll need a few more bits of information from you.

A few things of interest here:

1. What OS are you running Trivy on?
2. Do you have write permissions as the user running the Trivy process to create (write) a metadata.json file to disk?
   If so, could you verify what's inside of it and it's last modified date?
   For example, if you're on a macOS host:

$ cat /Users/<username>/Library/Caches/trivy/db/metadata.json
{"Version":1,"Type":1,"NextUpdate":"2020-10-09T12:37:23.034428891Z","UpdatedAt":"2020-10-09T00:37:23.034429191Z"}

$ ls -lrth /Users/<username>/Library/Caches/trivy/db/metadata.json
-rw-------  1 user  staff   113B  8 Oct 18:00 /Users/<username>/Library/Caches/trivy/db/metadata.json

I noticed even on the second run the following log output remains the same:

2020-10-05T12:04:53.488+0300    DEBUG   DB Schema: 1, Type: 1, UpdatedAt: 2020-10-01 00:36:30.496570788 +0000 UTC, NextUpdate: 2020-10-01 12:36:30.496570388 +0000 UTC

Which signals to me something has prevented writing to disk to update the metadata.json file.

1. Is your system clock out of sync? You could try running something like ntpdate -s time.nist.gov to force a sync.

[pznamensky]

Hi @simar7,
thanks for the quick reply.
But actually, I can't reproduce this too :)

2020-10-09T10:53:21.977+0300    DEBUG   DB update was skipped because DB is the latest
2020-10-09T10:53:21.977+0300    DEBUG   DB Schema: 1, Type: 1, UpdatedAt: 2020-10-09 00:37:23.034429191 +0000 UTC, NextUpdate: 2020-10-09 12:37:23.034428891 +0000 UTC

But it appeared twice: on our CI server and locally when I ran in debug mode.

What OS are you running Trivy on?

CentOS Linux release 7.8.2003 (Core)

Do you have write permissions as the user running the Trivy process to create (write) a metadata.json file to disk?

Yes, no doubt.
Though I guess it's not relevant right now because I can't reproduce this issue, but here are output of that commands:

~ $ cat ~/.cache/trivy/db/metadata.json
{"Version":1,"Type":1,"NextUpdate":"2020-10-09T12:37:23.034428891Z","UpdatedAt":"2020-10-09T00:37:23.034429191Z"}

~ $ ls -lrth ~/.cache/trivy/db/metadata.json
-rw------- 1 user user 113 Oct  9 10:50 /home/user/.cache/trivy/db/metadata.json

Is your system clock out of sync?

No, we're syncing the clock every several hours.

So I'll wait the next time the issue will reproduced on our CI and then check metadata.json permission and content.

[simar7]

Hmm that's interesting. Keep us posted!

[mozillazg]

@simar7 @pznamensky The root case is the cron job to update trivy db was failed during 10.1 ~ 10.7, so the trivy download an old db and try to download latest db again at every start time. read more: https://github.com/aquasecurity/trivy-db/issues/73

[knqyf263]

As @mozillazg explained, that is because the DB build was failing and Trivy downloaded an old DB every time. I think we should introduce a mechanism to cache the DB even when the DB build is failing.

[pznamensky]

Thank you all!
Should I close this issue, then?

[knqyf263]

Yes, please. I'll raise another issue to fix the root cause.