Harbor: [trivy] Failed to download vulnerability DB: API rate limit exceeded

[h8h]

What can we help you?

---------- Logs of report 739b5b7c-8b23-41e1-a8f2-c58486b593df ----------
2020-10-23T13:45:22Z [INFO] [/pkg/scan/job.go:325]: registration:
2020-10-23T13:45:22Z [INFO] [/pkg/scan/job.go:336]: {
  "uuid": "47ffc762-1518-11eb-b1e6-0242ac1a0009",
  "name": "Trivy",
  "description": "The Trivy scanner adapter",
  "url": "http://trivy-adapter:8080",
  "disabled": false,
  "is_default": true,
  "health": "healthy",
  "auth": "",
  "skip_certVerify": false,
  "use_internal_addr": true,
  "adapter": "Trivy",
  "vendor": "Aqua Security",
  "version": "v0.9.2",
  "create_time": "2020-10-23T10:12:39.627599Z",
  "update_time": "2020-10-23T10:12:39.627601Z"
}
2020-10-23T13:45:22Z [INFO] [/pkg/scan/job.go:325]: scanRequest:
2020-10-23T13:45:22Z [INFO] [/pkg/scan/job.go:336]: {
  "registry": {
    "url": "http://core:8080",
    "authorization": "[HIDDEN]"
  },
  "artifact": {
    "namespace_id": 1,
    "repository": "library/debian",
    "tag": "",
    "digest": "sha256:f4f8457fd057d0744bbd01d994e9ad1229f0bee5d063f1ab7a8568ef18c8ddf8",
    "mime_type": "application/vnd.docker.distribution.manifest.v2+json"
  }
}
2020-10-23T13:45:22Z [INFO] [/pkg/scan/job.go:156]: Report mime types: [application/vnd.scanner.adapter.vuln.report.harbor+json; version=1.0]
2020-10-23T13:45:22Z [INFO] [/pkg/scan/job.go:202]: Get report for mime type: application/vnd.scanner.adapter.vuln.report.harbor+json; version=1.0
2020-10-23T13:45:24Z [ERROR] [/pkg/scan/job.go:284]: check scan report with mime type application/vnd.scanner.adapter.vuln.report.harbor+json; version=1.0: running trivy wrapper: running trivy: exit status 1: 2020-10-23T13:45:22.934Z    INFO    Need to update DB
2020-10-23T13:45:22.934Z    INFO    Downloading DB...
2020-10-23T13:45:23.055Z    FATAL    failed to download vulnerability DB: failed to download vulnerability DB: failed to list releases: GET https://api.github.com/repos/aquasecurity/trivy-db/releases: 403 API rate limit exceeded for x.x.x.x. (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.) [rate reset in 50m29s]
: general response handler: unexpected status code: 500, expected: 200

I've uploaded about 20 images to harbor and then the scans failed.

I know what this issue is about.

But does every trivy scan really needs to download the DB?

I know that there is an option to skip the download, but I guess that I have to do it manually.

Is there an option to download the DB frequently (every day, every 12h)?

Thank you.

[lishinho]

It is really bad

[heww]

@h8h and @lishinho you can set a GitHub access token for the Trivy scanner to increase the rate limit to 5000 requests per hour. For more information see: https://goharbor.io/docs/2.1.0/install-config/configure-yml-file/

[h8h]

Algthough this is the solution, but if every project is so wasteful with GitHub resources, the server will be down soon.

Sorry IMHO this should not be the solution. The solution should be a pull on a regular time basis and not by every image push.

[heww]

Hi, @h8h I think that the Trivy adapter will update the vulnerability database from GitHub every 12hours, not every scanning. Is this right @danielpacak ?

[danielpacak]

Yes @heww Normally Trivy is caching the DB file downloaded from GitHub and reuses it for subsequent scans.

[h8h]

It seems that it not get cached

2020-10-23T13:45:24Z [ERROR] [/pkg/scan/job.go:284]: check scan report with mime type application/vnd.scanner.adapter.vuln.report.harbor+json; version=1.0: running trivy wrapper: running trivy: exit status 1: 2020-10-23T13:45:22.934Z    �[34mINFO�[0m    Need to update DB
2020-10-23T13:45:22.934Z    �[34mINFO�[0m    Downloading DB...
2020-10-23T13:45:23.055Z    �[31mFATAL�[0m    failed to download vulnerability DB:

I pushed serval images and than while scanning "downloading..." failed without redeploying the harbor docker stack.

With caching I guess I never ran into "403 API rate limit exceeded " or @danielpacak ?

[danielpacak]

👋 @h8h Indeed normally, if the persistence volume used by the Trivy adapter is not deleted for some reasons you should not bump into GitHub rate limit issues. As @heww mentioned and I confirmed in my previous comment, Trivy executable used by the adapter service detects whether the Trivy DB file is cached on a local file system and is up to date. It only tries to download the Trivy DB file from GitHub if it's not present or is outdated. Beyond that, the Trivy DB is updated every 12 hours by one of scheduled jobs.

That said, I'm wondering if:

• [ ] You've ever managed to download the Trivy DB file without getting the rate limit error in your environment?
• [ ] Is there any other app in your cluster that might reach out to GitHub APIs?
• [ ] Is there any process or configuration that might delete the Trivy DB from the PV created by the Trivy adapter service
• [ ] Could you provide more details about versions of Harbor and Trivy that you're using and the installation mode (Docker compose vs Helm chart?) You can find the version of the Trivy adapter in the Harbor UI under the Interrogation Service menu in the Trivy scanner details.

[danielpacak]

BTW, I was testing another thing and here's another confirmation that Trivy does skip the DB update with its default config. Search for DB update was skipped because DB is the latest in the following log output of the scanner adapter:

Nov 29 10:45:46 172.20.0.1 trivy-adapter[1686]: {"level":"debug","msg":"Updating status for scan job","new_status":"Pending","scan_job_id":"9f0b6d18ef8c4d722d46c1d2","time":"2020-11-29T10:45:46Z"}
Nov 29 10:45:46 172.20.0.1 trivy-adapter[1686]: {"expire":3600,"level":"debug","msg":"Updating scan job","redis_key":"harbor.scanner.trivy:store:scan-job:9f0b6d18ef8c4d722d46c1d2","scan_job_id":"9f0b6d18ef8c4d722d46c1d2","scan_job_status":"Pending","time":"2020-11-29T10:45:46Z"}
Nov 29 10:45:46 172.20.0.1 trivy-adapter[1686]: {"image_ref":"core:8080/library/nginx@sha256:c0978d1f3e5ef8a936537cf67bb155a581f56f93bbf8a345a650720e14d321bb","level":"debug","msg":"Started scanning","time":"2020-11-29T10:45:46Z"}
Nov 29 10:45:46 172.20.0.1 trivy-adapter[1686]: {"level":"debug","msg":"Saving scan report to tmp file","path":"/home/scanner/.cache/reports/scan_report_983178097.json","time":"2020-11-29T10:45:46Z"}
Nov 29 10:45:46 172.20.0.1 trivy-adapter[1686]: {"exit_code":0,"image_ref":"core:8080/library/nginx@sha256:c0978d1f3e5ef8a936537cf67bb155a581f56f93bbf8a345a650720e14d321bb","level":"debug","msg":"Running trivy finished","std_out":"2020-11-29T10:45:46.030Z\t\u001b[35mDEBUG\u001b[0m\tSeverities: HIGH,CRITICAL\n2020-11-29T10:45:46.038Z\t\u001b[35mDEBUG\u001b[0m\tcache dir:  /home/scanner/.cache/trivy\n2020-11-29T10:45:46.039Z\t\u001b[35mDEBUG\u001b[0m\tDB update was skipped because DB is the latest\n2020-11-29T10:45:46.039Z\t\u001b[35mDEBUG\u001b[0m\tDB Schema: 1, Type: 1, UpdatedAt: 2020-11-29 00:33:54.657934459 +0000 UTC, NextUpdate: 2020-11-29 12:33:54.657934059 +0000 UTC, DownloadedAt: 2020-11-29 10:24:22.54855719 +0000 UTC\n2020-11-29T10:45:46.090Z\t\u001b[35mDEBUG\u001b[0m\tVulnerability type:  [os]\n2020-11-29T10:45:46.122Z\t\u001b[35mDEBUG\u001b[0m\tArtifact ID: sha256:2562b6bef9769b637d47ec31284ac0f1a7facef498ab97753623b13c502cddce\n2020-11-29T10:45:46.122Z\t\u001b[35mDEBUG\u001b[0m\tBlob IDs: [sha256:f5600c6330da7bb112776ba067a32a9c20842d6ecc8ee3289f1a713b644092f8 sha256:546ff42febcb6104992d84b2d95beaa699f2952ede6bc4bdb996f02dda30f680 sha256:cf91f5e8589e08c68094796ab970a77ec5f49235dce040d706f2c3f492609961 sha256:4f5295ed788f7de301eb3e25bf72fcdcb5b731dab528495cfab6898bcdda3673 sha256:b56dcfc4eee5dc5435639d9603fbbf40474d282bec27edbb2796820039208dde]\n2020-11-29T10:45:46.122Z\t\u001b[34mINFO\u001b[0m\tDetecting Debian vulnerabilities...\n2020-11-29T10:45:46.122Z\t\u001b[35mDEBUG\u001b[0m\tdebian: os version: 10\n2020-11-29T10:45:46.122Z\t\u001b[35mDEBUG\u001b[0m\tdebian: the number of packages: 135\n","time":"2020-11-29T10:45:46Z"}
Nov 29 10:45:46 172.20.0.1 trivy-adapter[1686]: {"level":"debug","msg":"Removing scan report tmp file","path":"/home/scanner/.cache/reports/scan_report_983178097.json","time":"2020-11-29T10:45:46Z"}
Nov 29 10:45:46 172.20.0.1 trivy-adapter[1686]: {"level":"debug","msg":"Updating reports for scan job","scan_job_id":"9f0b6d18ef8c4d722d46c1d2","time":"2020-11-29T10:45:46Z"}
Nov 29 10:45:46 172.20.0.1 trivy-adapter[1686]: {"expire":3600,"level":"debug","msg":"Updating scan job","redis_key":"harbor.scanner.trivy:store:scan-job:9f0b6d18ef8c4d722d46c1d2","scan_job_id":"9f0b6d18ef8c4d722d46c1d2","scan_job_status":"Pending","time":"2020-11-29T10:45:46Z"}
Nov 29 10:45:46 172.20.0.1 trivy-adapter[1686]: {"level":"debug","msg":"Updating status for scan job","new_status":"Finished","scan_job_id":"9f0b6d18ef8c4d722d46c1d2","time":"2020-11-29T10:45:46Z"}
Nov 29 10:45:46 172.20.0.1 trivy-adapter[1686]: {"expire":3600,"level":"debug","msg":"Updating scan job","redis_key":"harbor.scanner.trivy:store:scan-job:9f0b6d18ef8c4d722d46c1d2","scan_job_id":"9f0b6d18ef8c4d722d46c1d2","scan_job_status":"Finished","time":"2020-11-29T10:45:46Z"}

[h8h]

I think I'm going to close this issue.

Today I've tried it with the current version 2.1.1 and it seems to work (scanning more than 20 images) without getting into the "API rate limit exceeded" error.

Thank you