Trivy: Trivy 0.2.0, run under docker-dind - gitlabCI - scan always ends with 'null' result

Created on 14 Nov 2019 · 24Comments · Source: aquasecurity/trivy

Trivy 0.2.0, run under docker-dind - gitlabCI - scan always ends with 'null' result

What did you expect to happen?
Conduct a scan

What happened instead?
Trivy does something, reports success and exits after 1 second

Output of run with -debug:

2019-11-14T18:51:04.070Z    DEBUG   cache dir:  /root/.cache/trivy
2019-11-14T18:51:04.074Z    DEBUG   This is the first run
[                    ] Downloading Lightweight DB file... 
[=>                  ] Downloading Lightweight DB file... 
[===>                ] Downloading Lightweight DB file... 2019-11-14T18:51:04.362Z  DEBUG   release name: v1-2019111418
2019-11-14T18:51:04.362Z    DEBUG   asset name: trivy-light.db.gz
[=====>              ] Downloading Lightweight DB file... 
[======>             ] Downloading Lightweight DB file... 2019-11-14T18:51:04.534Z  DEBUG   asset URL: https://github-production-release-asset-2e65be.s3.amazonaws.com/216830441/fa14f900-0709-11ea-9b7f-1f882f72ad9e?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20191114%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20191114T185104Z&X-Amz-Expires=300&X-Amz-Signature=4ae948a4ce0501f0edb9eb5585d397ef276d725f95f8ea2af8ade4659264494d&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Dtrivy-light.db.gz&response-content-type=application%2Foctet-stream
[========>           ] 
Downloading Lightweight DB file... 
[==========>         ]
Downloading Lightweight DB file... [============>       ] 
Downloading Lightweight DB file... [==============>     ] 
Downloading Lightweight DB file... [================>   ] 
Downloading Lightweight DB file... ==================> ] 
Downloading Lightweight DB file... [===================>] 
Downloading Lightweight DB file... [                    ] 
Downloading Lightweight DB file... [=>                  ] 
Downloading Lightweight DB file... [===>                ]
 Downloading Lightweight DB file... [=====>              ] 
Downloading Lightweight DB file... [======>             ]
 Downloading Lightweight DB file... [========>           ] 
Downloading Lightweight DB file... 019-11-14T18:51:05.858Z  INFO    Reopening vulnerability DB
2019-11-14T18:51:05.858Z    DEBUG   Vulnerability type:  [os library]
2019-11-14T18:51:08.814Z    DEBUG   OS family: alpine, OS version: 3.8.4
2019-11-14T18:51:08.814Z    DEBUG   the number of packages: 36
2019-11-14T18:51:09.612Z    DEBUG   the number of packages from commands: 26
2019-11-14T18:51:09.612Z    DEBUG   the number of packages: 36
2019-11-14T18:51:09.612Z    INFO    Detecting Alpine vulnerabilities...
2019-11-14T18:51:09.612Z    DEBUG   alpine: os version: 3.8
2019-11-14T18:51:09.612Z    DEBUG   alpine: the number of packages: 36

Output of trivy -v:

trivy version 0.2.0

Additional details (base image name, container registry info...):
checked image trivy --light alpine:3.8.4 -debug
dind image - docker:19.03-dind

result:

[
  {
    "Target": "myimage(alpine 3.8.4)",
    "Vulnerabilities": null
  }
]

scan outside gitlabci

alpine:3.8.4 (alpine 3.8.4)
===========================
Total: 1 (UNKNOWN: 0, LOW: 0, MEDIUM: 0, HIGH: 1, CRITICAL: 0)

+---------+------------------+----------+-------------------+---------------+
| LIBRARY | VULNERABILITY ID | SEVERITY | INSTALLED VERSION | FIXED VERSION |
+---------+------------------+----------+-------------------+---------------+
| musl    | CVE-2019-14697   | HIGH     | 1.1.19-r10        | 1.1.19-r11    |
+---------+------------------+----------+-------------------+---------------+

kinbug

Source

marcinbojko

All 24 comments

What's the musl version in your image?

$ docker run -it --rm your_image apk info -v | grep musl

knqyf263 on 14 Nov 2019

Hi @knqyf263
For to ease the test i am running trivy check in gitlabci against public image:
So, when running localy (ubuntu) or test using docker-dind localy I am using:

sudo trivy alpine:3.8.4
2019-11-15T07:30:39.433+0100    INFO    Reopening vulnerability DB
2019-11-15T07:30:39.887+0100    INFO    Detecting Alpine vulnerabilities...

alpine:3.8.4 (alpine 3.8.4)
===========================
Total: 1 (UNKNOWN: 0, LOW: 0, MEDIUM: 0, HIGH: 1, CRITICAL: 0)

+---------+------------------+----------+-------------------+---------------+--------------------------------+
| LIBRARY | VULNERABILITY ID | SEVERITY | INSTALLED VERSION | FIXED VERSION |             TITLE              |
+---------+------------------+----------+-------------------+---------------+--------------------------------+
| musl    | CVE-2019-14697   | HIGH     | 1.1.19-r10        | 1.1.19-r11    | musl libc through 1.1.23       |
|         |                  |          |                   |               | has an x87 floating-point      |
|         |                  |          |                   |               | stack adjustment imbalance,    |
|         |                  |          |                   |               | related...                     |
+---------+------------------+----------+-------------------+---------------+--------------------------------+

as you can see, this image has musl problems.

Running trivy inside gitlab ci against THE SAME image gives this one second scan then exits with 'null'

marcinbojko on 15 Nov 2019

docker run -it --rm alpine:3.8.4 apk info -v | grep musl
Unable to find image 'alpine:3.8.4' locally
3.8.4: Pulling from library/alpine
c87736221ed0: Pulling fs layer
c87736221ed0: Download complete
c87736221ed0: Pull complete
Digest: sha256:04696b491e0cc3c58a75bace8941c14c924b9f313b03ce5029ebbc040ed9dcd9
Status: Downloaded newer image for alpine:3.8.4
musl-1.1.19-r10
musl-utils-1.1.19-r10

marcinbojko on 15 Nov 2019

can you paste your gitlab-ci.yaml file. It is strange because i am running 0.2.0 in gitlab I have dozens of builds with this version already and it is working pretty well.

webmutation on 15 Nov 2019

👍1

Well, there is nothing strange there.
Trivy downloads, runs, scans, but result is 'null'.
DB is there, same size as runnin manually

 DOCKER_HOST: tcp://docker:2375
 DOCKER_DRIVER: overlay2
 DOCKER_TLS_CERTDIR: ""
 GIT_SUBMODULE_STRATEGY: recursive
docker_retag:
  retry: 2
  stage: docker_retag
  image: docker:stable
  services:
    - docker:$DOCKER_DIND
  before_script:
    - docker info
    - export IMAGE_VER=$(cat data/image)
    - export TAG_VER=$(cat data/tag)
    - export IMAGE_BASENAME=$(cat data/image_basename)
    - sh ./pipeline/trivy_init.sh
  script:
    - docker login xxx
    - docker login yyy
    - docker pull $IMAGE_VER:$TAG_VER
    - sh ./pipeline/trivy_run.sh
# if no failure - push
    - docker tag xxx
    - docker push yyy
  artifacts:
    expire_in: 1d
    paths:
      - data/
  only:
    - web
    - trigger

marcinbojko on 15 Nov 2019

Another sets of test 0.1.7 vs 0.2.0

docker_retag:
  retry: 2
  stage: docker_retag
  image: docker:stable
  services:
    - docker:$DOCKER_DIND
  script:
    - apk add --no-cache --update curl wget grep sed git
    - wget -q https://github.com/aquasecurity/trivy/releases/download/v0.1.7/trivy_0.1.7_Linux-64bit.tar.gz
    - tar zxf trivy_0.1.7_Linux-64bit.tar.gz
    - mv trivy /usr/local/bin
    - chmod +x /usr/local/bin/trivy
    - docker pull alpine:3.8.4
    - docker image ls
    - /usr/local/bin/trivy -debug alpine:3.8.4
  artifacts:
    expire_in: 1d
    paths:
      - data/
  only:
    - web
    - trigger

and 0.1.7 result

2019-11-15T15:09:14.227Z    DEBUG   Vulnerability type:  [os library]
2019-11-15T15:09:14.332Z    DEBUG   OS family: alpine, OS version: 3.8.4
2019-11-15T15:09:14.332Z    DEBUG   the number of packages: 16
2019-11-15T15:09:15.523Z    DEBUG   the number of packages from commands: 0
2019-11-15T15:09:15.523Z    DEBUG   the number of packages: 16
2019-11-15T15:09:15.523Z    INFO    Detecting Alpine vulnerabilities...
2019-11-15T15:09:15.523Z    DEBUG   alpine: os version: 3.8
2019-11-15T15:09:15.523Z    DEBUG   alpine: the number of packages: 16

alpine:3.8.4 (alpine 3.8.4)
===========================
Total: 1 (UNKNOWN: 0, LOW: 0, MEDIUM: 0, HIGH: 1, CRITICAL: 0)

+---------+------------------+----------+-------------------+---------------+--------------------------------+
| LIBRARY | VULNERABILITY ID | SEVERITY | INSTALLED VERSION | FIXED VERSION |             TITLE              |
+---------+------------------+----------+-------------------+---------------+--------------------------------+
| musl    | CVE-2019-14697   | HIGH     | 1.1.19-r10        | 1.1.19-r11    | musl libc through 1.1.23       |
|         |                  |          |                   |               | has an x87 floating-point      |
|         |                  |          |                   |               | stack adjustment imbalance,    |
|         |                  |          |                   |               | related...                     |
+---------+------------------+----------+-------------------+---------------+--------------------------------+

marcinbojko on 15 Nov 2019

0.2.0

docker_retag:
  retry: 2
  stage: docker_retag
  image: docker:stable
  services:
    - docker:$DOCKER_DIND
  script:
    - apk add --no-cache --update curl wget grep sed git
    - wget -q https://github.com/aquasecurity/trivy/releases/download/v0.2.0/trivy_0.2.0_Linux-64bit.tar.gz
    - tar zxf trivy_0.2.0_Linux-64bit.tar.gz
    - mv trivy /usr/local/bin
    - chmod +x /usr/local/bin/trivy
    - docker pull alpine:3.8.4
    - docker image ls
    - /usr/local/bin/trivy -debug alpine:3.8.4
  artifacts:
    expire_in: 1d
    paths:
      - data/
  only:
    - web
    - trigger

and 0.2.0 result

2019-11-15T15:14:44.248Z    DEBUG   Vulnerability type:  [os library]
2019-11-15T15:14:44.363Z    DEBUG   OS family: alpine, OS version: 3.8.4
2019-11-15T15:14:44.363Z    DEBUG   the number of packages: 16
2019-11-15T15:14:45.176Z    DEBUG   the number of packages from commands: 0
2019-11-15T15:14:45.176Z    DEBUG   the number of packages: 16
2019-11-15T15:14:45.176Z    INFO    Detecting Alpine vulnerabilities...
2019-11-15T15:14:45.176Z    DEBUG   alpine: os version: 3.8
2019-11-15T15:14:45.176Z    DEBUG   alpine: the number of packages: 16

alpine:3.8.4 (alpine 3.8.4)
===========================
Total: 0 (UNKNOWN: 0, LOW: 0, MEDIUM: 0, HIGH: 0, CRITICAL: 0)

marcinbojko on 15 Nov 2019

Hmm... in my environment, it works. I have no idea. What's your digest of the image in CI? Please paste the result of docker image ls.

Just in case, would you try the following?

    - docker image ls
    - docker run --rm alpine:3.8.4 apk info -v 
    - /usr/local/bin/trivy --reset
    - /usr/local/bin/trivy -debug alpine:3.8.4

knqyf263 on 15 Nov 2019

for 0.2.0 digest is

REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
alpine              3.8.4               dac705114996        8 months ago        4.41MB

for 0.1.7 digest is

REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
alpine              3.8.4               dac705114996        8 months ago        4.41MB

marcinbojko on 15 Nov 2019

👍1

I am stuck, can't find any other test to conduct

$ apk add --no-cache --update curl wget grep sed git
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/community/x86_64/APKINDEX.tar.gz
(1/10) Installing nghttp2-libs (1.39.2-r0)
(2/10) Installing libcurl (7.66.0-r0)
(3/10) Installing curl (7.66.0-r0)
(4/10) Installing expat (2.2.8-r0)
(5/10) Installing pcre2 (10.33-r0)
(6/10) Installing git (2.22.0-r0)
(7/10) Installing pcre (8.43-r0)
(8/10) Installing grep (3.3-r0)
(9/10) Installing sed (4.7-r0)
(10/10) Installing wget (1.20.3-r0)
Executing busybox-1.30.1-r2.trigger
OK: 22 MiB in 25 packages
$ wget -q https://github.com/aquasecurity/trivy/releases/download/v0.2.0/trivy_0.2.0_Linux-64bit.tar.gz
$ tar zxf trivy_0.2.0_Linux-64bit.tar.gz
$ mv trivy /usr/local/bin
$ chmod +x /usr/local/bin/trivy
$ docker pull alpine:3.8.4
3.8.4: Pulling from library/alpine
c87736221ed0: Pulling fs layer
c87736221ed0: Verifying Checksum
c87736221ed0: Download complete
c87736221ed0: Pull complete
Digest: sha256:04696b491e0cc3c58a75bace8941c14c924b9f313b03ce5029ebbc040ed9dcd9
Status: Downloaded newer image for alpine:3.8.4
docker.io/library/alpine:3.8.4
$ docker image ls
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
alpine              3.8.4               dac705114996        8 months ago        4.41MB
$ docker run --rm alpine:3.8.4 apk info -v
musl-1.1.19-r10
WARNING: Ignoring APKINDEX.adfa7ceb.tar.gz: No such file or directory
WARNING: Ignoring APKINDEX.efaa1f73.tar.gz: No such file or directory
busybox-1.28.4-r3
alpine-baselayout-3.1.0-r0
alpine-keys-2.1-r1
libressl2.7-libcrypto-2.7.5-r0
libressl2.7-libssl-2.7.5-r0
libressl2.7-libtls-2.7.5-r0
ssl_client-1.28.4-r3
zlib-1.2.11-r1
apk-tools-2.10.1-r0
scanelf-1.2.3-r0
musl-utils-1.1.19-r10
libc-utils-0.7.1-r0
$ /usr/local/bin/trivy --reset
2019-11-15T16:06:25.515Z    DEBUG   cache dir:  /root/.cache/trivy
2019-11-15T16:06:25.622Z    DEBUG   This is the first run
[                    ] Downloading Full DB file... [=>                  ] Downloading Full DB file... [===>                ] Downloading Full DB file... [=====>              ] Downloading Full DB file... 2019-11-15T16:06:25.928Z    DEBUG   release name: v1-2019111515
2019-11-15T16:06:25.928Z    DEBUG   asset name: trivy-light.db.gz
2019-11-15T16:06:25.928Z    DEBUG   file name doesn't match
2019-11-15T16:06:25.928Z    DEBUG   asset name: trivy.db.gz
[======>             ] Downloading Full DB file... 2019-11-15T16:06:26.105Z DEBUG   asset URL: https://github-production-release-asset-2e65be.s3.amazonaws.com/216830441/08bae900-07ba-11ea-98d3-341d413eecce?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20191115%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20191115T160626Z&X-Amz-Expires=300&X-Amz-Signature=6d3514b8a2e891f9dcc5215fb97c59629767c42ff2eb89134d1d912a2ba61ee7&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Dtrivy.db.gz&response-content-type=application%2Foctet-stream
2019-11-15T16:06:39.740Z    DEBUG   Vulnerability type:  [os library]
2019-11-15T16:06:39.845Z    DEBUG   OS family: alpine, OS version: 3.8.4
2019-11-15T16:06:39.845Z    DEBUG   the number of packages: 16
2019-11-15T16:06:40.413Z    DEBUG   the number of packages from commands: 0
2019-11-15T16:06:40.413Z    DEBUG   the number of packages: 16
2019-11-15T16:06:40.413Z    INFO    Detecting Alpine vulnerabilities...
2019-11-15T16:06:40.413Z    DEBUG   alpine: os version: 3.8
2019-11-15T16:06:40.413Z    DEBUG   alpine: the number of packages: 16

alpine:3.8.4 (alpine 3.8.4)
===========================
Total: 0 (UNKNOWN: 0, LOW: 0, MEDIUM: 0, HIGH: 0, CRITICAL: 0)

marcinbojko on 15 Nov 2019

I cannot reproduce it... What's $DOCKER_DIND?

knqyf263 on 15 Nov 2019

Oh, sorry I found. docker:19.03-dind

knqyf263 on 15 Nov 2019

👍1

Sorry, it's: 19.03-dind
I am using tagged version instead of stable/latest

marcinbojko on 15 Nov 2019

Yeah, I understand. These kinds of errors are the worst.
As a base I am using Centos7 (gitlab runners)
https://app.vagrantup.com/marcinbojko/boxes/centos7

marcinbojko on 15 Nov 2019

$ docker run --rm --privileged --name dind docker:19.03-dind
$ docker exec -it dind sh
/ # wget -q https://github.com/aquasecurity/trivy/releases/download/v0.2.0/trivy_0.2.0_Linux-64bit.tar.gz
/ # tar zxf trivy_0.2.0_Linux-64bit.tar.gz
/ # mv trivy /usr/local/bin
/ # chmod +x /usr/local/bin/trivy
/ # docker pull alpine:3.8.4
3.8.4: Pulling from library/alpine
c87736221ed0: Pull complete
Digest: sha256:04696b491e0cc3c58a75bace8941c14c924b9f313b03ce5029ebbc040ed9dcd9
Status: Downloaded newer image for alpine:3.8.4
docker.io/library/alpine:3.8.4
/ # docker image ls
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
alpine              3.8.4               dac705114996        8 months ago        4.41MB
/ # /usr/local/bin/trivy -debug alpine:3.8.4
...

alpine:3.8.4 (alpine 3.8.4)
===========================
Total: 1 (UNKNOWN: 0, LOW: 0, MEDIUM: 0, HIGH: 1, CRITICAL: 0)

+---------+------------------+----------+-------------------+---------------+--------------------------------+
| LIBRARY | VULNERABILITY ID | SEVERITY | INSTALLED VERSION | FIXED VERSION |             TITLE              |
+---------+------------------+----------+-------------------+---------------+--------------------------------+
| musl    | CVE-2019-14697   | HIGH     | 1.1.19-r10        | 1.1.19-r11    | musl libc through 1.1.23       |
|         |                  |          |                   |               | has an x87 floating-point      |
|         |                  |          |                   |               | stack adjustment imbalance,    |
|         |                  |          |                   |               | related...                     |
+---------+------------------+----------+-------------------+---------------+--------------------------------+

I tested on docker:19.03-dind, but the vulnerability was detected. Is my test wrong?

knqyf263 on 15 Nov 2019

Your test was perfect. Even on the same node (gitlab runner) doing THE SAME test, they are success. But through gitlab-ci it does zero.
So I suppose there is something related with gitlab/gitlab runner or it's settings which are blocking this.

marcinbojko on 15 Nov 2019

Well, for a time being I am back to 0.1.7, lightning fast scans will have to wait.
If I could provide extra-super-duper-debug info - let me know, and BIG thanks for support and Trivy!

marcinbojko on 15 Nov 2019

👍1

@knqyf263 sorry for spam, but I think I've found the reason ;)
So, for some reason, when using just defaults:

trivy -debug  alpine:3.8.4

under anything which is not dind at gitlab-ci, it works. I began to think, why 2 scans have almost everything in common, and I've found out- for SOME reason, severity default value is not casted/rendered properly in that env.
So, replacing 'get the default, i don't care setting' with:

trivy -debug -s "UNKNOWN,LOW,MEDIUM,HIGH,CRITICAL" alpine:3.8.4

when WE decide what's default - works.

alpine:3.8.4 (alpine 3.8.4)
===========================
Total: 1 (UNKNOWN: 0, LOW: 0, MEDIUM: 0, HIGH: 1, CRITICAL: 0)

+---------+------------------+----------+-------------------+---------------+--------------------------------+
| LIBRARY | VULNERABILITY ID | SEVERITY | INSTALLED VERSION | FIXED VERSION |             TITLE              |
+---------+------------------+----------+-------------------+---------------+--------------------------------+
| musl    | CVE-2019-14697   | HIGH     | 1.1.19-r10        | 1.1.19-r11    | musl libc through 1.1.23       |
|         |                  |          |                   |               | has an x87 floating-point      |
|         |                  |          |                   |               | stack adjustment imbalance,    |
|         |                  |          |                   |               | related...                     |
+---------+------------------+----------+-------------------+---------------+--------------------------------+

marcinbojko on 15 Nov 2019

👍1

Oh, what is the reason?

knqyf263 on 15 Nov 2019

Beats me, I am only a user here ;)

marcinbojko on 15 Nov 2019

No problem😄
If it is helpful for other users, it would be appreciated if you write down the detail. If no, let me close this one.

knqyf263 on 15 Nov 2019

👍1

It's up to you - but, just maybe - if we can keep this open for a week or so, to check if there will be some others with cases like this, it can save some time - especially when scan passes in CI and it can be discovered after several days.
Details are here: https://github.com/aquasecurity/trivy/issues/271#issuecomment-554460890

marcinbojko on 15 Nov 2019

Oh, I saw your comment before updating and overlooked the detail. Thank you for the explanation!

I think it's better to display target severities in a debug log. I'll work on it later.

knqyf263 on 15 Nov 2019

👍1

Thanks for the update. Dind is super buggy... its a pain. That is why i now build with kaniko.

I have trivy -s always set.
Ill try to remove it and see if i get the same

webmutation on 15 Nov 2019

👍1

Was this page helpful?

0 / 5 - 0 ratings

Related issues

Amazon Linux support

abedra · 4Comments

trivy --refresh shows error during redhat update

adel-s · 4Comments

Scan alpine image in air-gapped environments

knqyf263 · 3Comments

Crash following interrupted DB download

lizrice · 5Comments

cvss-vector

okazaki-hajime-ca · 6Comments