Trivy: Trivy in docker not able to scan local image since verison v0.10.0

[NasAmin]

Description

We use Trivy in our CI builds to scan local images. Since v0.10.0, trivy is not able to find the local images and expecting the image to exist in docker hub.

What did you expect to happen?
Expected trivy to scan local images.

What happened instead?
Trivy failed with the following error:
Command ran:
docker run --rm -v /var/run/docker.sock:/var/run/docker.sock -v $PWD:/tmp/.cache/ aquasec/trivy myimage:local
Where myimage:local was generated locally before running trivy.
Trivy failed with this error

 FATAL   unable to initialize a scanner: unable to initialize a docker scanner: 2 errors occurred:
        * unable to inspect the image (index.docker.io/library/myimage:local): Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.24/images/index.docker.io/library/myimage:local/json: dial unix /var/run/docker.sock: connect: permission denied
        * GET https://index.docker.io/v2/library/myimage/manifests/local: UNAUTHORIZED: authentication required; [map[Action:pull Class: Name:library/myimage Type:repository]]

Output of run with -debug:

docker run --rm -v /var/run/docker.sock:/var/run/docker.sock -v $PWD:/tmp/.cache/ aquasec/trivy --debug myimage:local2020-07-30T14:40:12.246Z        DEBUG   Severities: UNKNOWN,LOW,MEDIUM,HIGH,CRITICAL
2020-07-30T14:40:12.257Z        DEBUG   cache dir:  /home/appuser/.cache/trivy
2020-07-30T14:40:12.257Z        DEBUG   There is no valid metadata file: unable to open a file: open /home/appuser/.cache/trivy/db/metadata.json: no such file or directory
2020-07-30T14:40:12.257Z        INFO    Need to update DB
2020-07-30T14:40:12.257Z        INFO    Downloading DB...
2020-07-30T14:40:12.257Z        DEBUG   no metadata file
2020-07-30T14:40:12.788Z        DEBUG   release name: v1-2020073012
2020-07-30T14:40:12.788Z        DEBUG   asset name: trivy-light-offline.db.tgz
2020-07-30T14:40:12.788Z        DEBUG   file name doesn't match
2020-07-30T14:40:12.788Z        DEBUG   asset name: trivy-light.db.gz
2020-07-30T14:40:12.788Z        DEBUG   file name doesn't match
2020-07-30T14:40:12.788Z        DEBUG   asset name: trivy-offline.db.tgz
2020-07-30T14:40:12.788Z        DEBUG   file name doesn't match
2020-07-30T14:40:12.788Z        DEBUG   asset name: trivy.db.gz
2020-07-30T14:40:12.889Z        DEBUG   asset URL: https://github-production-release-asset-2e65be.s3.amazonaws.com/216830441/41262880-d25e-11ea-9f0d-69c6ece1083c?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20200730%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200730T143846Z&X-Amz-Expires=300&X-Amz-Signature=8962d7139933af30f139c0238307e1cefb4f262c886ef8dd8fbcb5f0301a5b97&X-Amz-SignedHeaders=host&actor_id=0&repo_id=216830441&response-content-disposition=attachment%3B%20filename%3Dtrivy.db.gz&response-content-type=application%2Foctet-stream
92.58 KiB / 17.57 MiB [>_____________________________________________________________] 0.51% ? p/s ?390.58 KiB / 17.57 MiB [->___________________________________________________________] 2.17% ? p/s ?713.58 KiB / 17.57 MiB [-->__________________________________________________________] 3.97% ? p/s ?917.58 KiB / 17.57 MiB [-->_____________________________________________] 5.10% 1.34 MiB p/s ETA 12s1.63 MiB / 17.57 MiB [---->_____________________________________________] 9.26% 1.34 MiB p/s ETA 11s2.06 MiB / 17.57 MiB [----->___________________________________________] 11.71% 1.34 MiB p/s ETA 11s2.46 MiB / 17.57 MiB [------>__________________________________________] 13.98% 1.42 MiB p/s ETA 10s2.85 MiB / 17.57 MiB [------->_________________________________________] 16.22% 1.42 MiB p/s ETA 10s3.24 MiB / 17.57 MiB [--------->_______________________________________] 18.42% 1.42 MiB p/s ETA 10s3.64 MiB / 17.57 MiB [---------->_______________________________________] 20.69% 1.46 MiB p/s ETA 9s4.02 MiB / 17.57 MiB [----------->______________________________________] 22.86% 1.46 MiB p/s ETA 9s4.42 MiB / 17.57 MiB [------------>_____________________________________] 25.13% 1.46 MiB p/s ETA 9s4.81 MiB / 17.57 MiB [------------->____________________________________] 27.40% 1.49 MiB p/s ETA 8s5.21 MiB / 17.57 MiB [-------------->___________________________________] 29.67% 1.49 MiB p/s ETA 8s5.61 MiB / 17.57 MiB [--------------->__________________________________] 31.93% 1.49 MiB p/s ETA 8s6.03 MiB / 17.57 MiB [----------------->________________________________] 34.30% 1.53 MiB p/s ETA 7s6.42 MiB / 17.57 MiB [------------------>_______________________________] 36.56% 1.53 MiB p/s ETA 7s6.87 MiB / 17.57 MiB [------------------->______________________________] 39.11% 1.53 MiB p/s ETA 7s7.29 MiB / 17.57 MiB [-------------------->_____________________________] 41.48% 1.56 MiB p/s ETA 6s7.72 MiB / 17.57 MiB [--------------------->____________________________] 43.93% 1.56 MiB p/s ETA 6s8.13 MiB / 17.57 MiB [----------------------->__________________________] 46.30% 1.56 MiB p/s ETA 6s8.55 MiB / 17.57 MiB [------------------------>_________________________] 48.66% 1.60 MiB p/s ETA 5s8.96 MiB / 17.57 MiB [------------------------->________________________] 51.01% 1.60 MiB p/s ETA 5s9.40 MiB / 17.57 MiB [-------------------------->_______________________] 53.48% 1.60 MiB p/s ETA 5s9.81 MiB / 17.57 MiB [--------------------------->______________________] 55.83% 1.63 MiB p/s ETA 4s10.24 MiB / 17.57 MiB [---------------------------->____________________] 58.30% 1.63 MiB p/s ETA 4s10.71 MiB / 17.57 MiB [----------------------------->___________________] 60.94% 1.63 MiB p/s ETA 4s11.12 MiB / 17.57 MiB [------------------------------->_________________] 63.30% 1.67 MiB p/s ETA 3s11.56 MiB / 17.57 MiB [-------------------------------->________________] 65.81% 1.67 MiB p/s ETA 3s12.00 MiB / 17.57 MiB [--------------------------------->_______________] 68.31% 1.67 MiB p/s ETA 3s12.45 MiB / 17.57 MiB [---------------------------------->______________] 70.86% 1.70 MiB p/s ETA 3s12.87 MiB / 17.57 MiB [----------------------------------->_____________] 73.23% 1.70 MiB p/s ETA 2s13.28 MiB / 17.57 MiB [------------------------------------->___________] 75.59% 1.70 MiB p/s ETA 2s13.71 MiB / 17.57 MiB [-------------------------------------->__________] 78.04% 1.73 MiB p/s ETA 2s14.16 MiB / 17.57 MiB [--------------------------------------->_________] 80.60% 1.73 MiB p/s ETA 1s14.64 MiB / 17.57 MiB [---------------------------------------->________] 83.33% 1.73 MiB p/s ETA 1s15.09 MiB / 17.57 MiB [------------------------------------------>______] 85.89% 1.76 MiB p/s ETA 1s15.55 MiB / 17.57 MiB [------------------------------------------->_____] 88.48% 1.76 MiB p/s ETA 1s16.00 MiB / 17.57 MiB [-------------------------------------------->____] 91.08% 1.76 MiB p/s ETA 0s16.47 MiB / 17.57 MiB [--------------------------------------------->___] 93.73% 1.80 MiB p/s ETA 0s16.92 MiB / 17.57 MiB [----------------------------------------------->_] 96.28% 1.80 MiB p/s ETA 0s17.38 MiB / 17.57 MiB [------------------------------------------------>] 98.93% 1.80 MiB p/s ETA 0s17.57 MiB / 17.57 MiB [----------------------------------------------------] 100.00% 2.12 MiB p/s 9s2020-07-30T14:40:22.179Z    DEBUG   Updating database metadata...
2020-07-30T14:40:22.179Z        DEBUG   DB Schema: 1, Type: 1, UpdatedAt: 2020-07-30 12:13:03.860403389 +0000 UTC, NextUpdate: 2020-07-31 00:13:03.860403189 +0000 UTC
2020-07-30T14:40:24.452Z        FATAL   unable to initialize a scanner:
    github.com/aquasecurity/trivy/internal/artifact.run
        /home/circleci/project/internal/artifact/run.go:72
  - unable to initialize a docker scanner:
    github.com/aquasecurity/trivy/internal/artifact.dockerScanner
        /home/circleci/project/internal/artifact/image.go:28
  - 2 errors occurred:
        * unable to inspect the image (index.docker.io/library/myimage:local): Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.24/images/index.docker.io/library/myimage:local/json: dial unix /var/run/docker.sock: connect: permission denied
        * GET https://index.docker.io/v2/library/myimage/manifests/local: UNAUTHORIZED: authentication required; [map[Action:pull Class: Name:library/myimage Type:repository]]

Output of trivy -v:

Version: 0.10.0

Additional details (base image name, container registry info...):
If we revert to trivy v0.9.0, the scan works successfully so something is broken in v0.10.0

Has there been any change that either affects detecting local images over the docker socket?

I would really appreciate if this has been paid attention as our builds are currently broken and as a workaround we have reverted to v0.9.0

Regards,

Nas

[NasAmin]

Tagging @mjjames here.

[knqyf263]

Please see here.
https://github.com/aquasecurity/trivy#others

[knqyf263]

Ah, I'm sorry it looks like you have a problem with the connection to docker.sock. Could you try docker run -u 0

[NasAmin]

@knqyf263 Yes using -u 0 works. So does this mean we have to run trivy as root when running in docker?

[mjjames]

we get:

• unable to inspect the image (index.docker.io/library/tenancyadmin:local): Error: No such image: index.docker.io/library/tenancyadmin:local
  * GET https://index.docker.io/v2/library/tenancyadmin/manifests/local: UNAUTHORIZED: authentication required; [map[Action:pull Class: Name:library/tenancyadmin Type:repository]]

if we use 0.9.0 we have no issues

[knqyf263]

@NasAmin Trivy ran as a root user before, but we changed it to a non-root user for security. If you want to use docker.sock, you have to run Trivy as root.

[knqyf263]

@mjjames If it still doesn't work with -u 0, could you open a new issue with the detail?

[mjjames]

I'm using docker for windows via WSL 2.
If i run:
docker run -u 0 --rm -v /var/run/docker.sock:/var/run/docker.sock aquasec/trivy tenancyadmin:local
I get
~FATAL unable to initialize a scanner: unable to initialize a docker scanner: 2 errors occurred: * unable to inspect the image (index.docker.io/library/tenancyadmin:local): Error: No such image: index.docker.io/library/tenancyadmin:local * GET https://index.docker.io/v2/library/tenancyadmin/manifests/local: UNAUTHORIZED: authentication required; [map[Action:pull Class: Name:library/tenancyadmin Type:repository]]~

should have been:
* unable to inspect the image (index.docker.io/library/authorizationas:local): Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.24/images/index.docker.io/library/authorizationas:local/json: dial unix /var/run/docker.sock: connect: permission denied * GET https://index.docker.io/v2/library/authorizationas/manifests/local: UNAUTHORIZED: authentication required; [map[Action:pull Class: Name:library/authorizationas Type:repository]]

It feels like same issue, as #579 as well. Everything was fine until 0.10.0

[mjjames]

I'm running Ubuntu 18.04. I believe @NasAmin is running 20.04

[knqyf263]

It says

Error: No such image: index.docker.io/library/tenancyadmin:local

so I feel v0.9.0 also should not work.

[dhananjaya-senanayake]

Had the same issue with the latest tag when running in Azure DevOps. Using the tag 0.9.2 fixed the failure

- script: |
    docker run --rm -v /var/run/docker.sock:/var/run/docker.sock \
    -v $HOME/Library/Caches:/root/.cache/ aquasec/trivy:0.9.2 --exit-code $(TRIVY_EXIT_CODE) \
    $(CONTAINER_REGISTRY)/$(WORKSPACE_REPOSITORY):latest
  displayName: scan workspace

[rahul2393]

Hello, everyone, I am creating PR to fix mounting to /var/run, but please don't mount to /root/ folder because of security purpose use /tmp/ for caches.

[mjjames]

It says

Error: No such image: index.docker.io/library/tenancyadmin:local

so I feel v0.9.0 also should not work.

My bad here, I've pasted the wrong message twice!

The message was:

• unable to inspect the image (index.docker.io/library/authorizationas:local): Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.24/images/index.docker.io/library/authorizationas:local/json: dial unix /var/run/docker.sock: connect: permission denied
  * GET https://index.docker.io/v2/library/authorizationas/manifests/local: UNAUTHORIZED: authentication required; [map[Action:pull Class: Name:library/authorizationas Type:repository]]

I get this from docker-compose

[rahul2393]

Ack, fix currently in the way

[knqyf263]

Hmm... It works in my environment even without -u 0. I'm not sure why I can't replicate it.

$ docker run --rm -v /var/run/docker.sock:/var/run/docker.sock -v $PWD:/tmp/.cache/ aquasec/trivy:0.10.0 alpine:3.10.2
alpine:3.10.2 (alpine 3.10.2)
=============================
Total: 5 (UNKNOWN: 0, LOW: 1, MEDIUM: 4, HIGH: 0, CRITICAL: 0)

[SGirousse]

Hmm... It works in my environment even without -u 0. I'm not sure why I can't replicate it.

$ docker run --rm -v /var/run/docker.sock:/var/run/docker.sock -v $PWD:/tmp/.cache/ aquasec/trivy:0.10.0 alpine:3.10.2
alpine:3.10.2 (alpine 3.10.2)
=============================
Total: 5 (UNKNOWN: 0, LOW: 1, MEDIUM: 4, HIGH: 0, CRITICAL: 0)

If do (on my local environment) :

docker run --rm -v /var/run/docker.sock:/var/run/docker.sock -v $PWD:/tmp/.cache/ aquasec/trivy:0.10.0 alpine:3.10.2

it is working fine but if I do :

docker login -u <username> -p <pwd> <private_azure_cr>
docker run --rm -v /var/run/docker.sock:/var/run/docker.sock -v $PWD:/tmp/.cache/ aquasec/trivy:0.10.0 <private_azure_cr>/<image>:<tag> 

then I got the error.

So something goes wrong when we need an authentication ?

[knqyf263]

@SGirousse Thanks, you're right. If you add -u root, it works, right?

[knqyf263]

@dhananjaya-senanayake Doesn't it work even with -u root?

[knqyf263]

@mjjames What's the permission of your /var/run/docker.sock? Could you specify the user with -u like -u foobar?

[SGirousse]

@SGirousse Thanks, you're right. If you add -u root, it works, right?

No, still the same error.

[knqyf263]

@SGirousse Could you tell me your environment and the permission of /var/run/docker.sock? Also, make sure it works with v0.9.2, please.

[knqyf263]

We just changed the user in the docker image, so it should work with -u root like:

docker run -u root -v /var/run/docker.sock:/var/run/docker.sock aquasec/trivy:0.10.0 --no-progress my_image

as it is the same as v0.9.2.

[SGirousse]

docker run --rm -v /var/run/docker.sock:/var/run/docker.sock -v $PWD:/tmp/.cache/ aquasec/trivy:0.9.0 <my_private_image>

=> success

docker run -u root --rm -v /var/run/docker.sock:/var/run/docker.sock -v $PWD:/tmp/.cache/ aquasec/trivy:0.9.0 <my_private_image>

=> failure

docker run --rm -v /var/run/docker.sock:/var/run/docker.sock -v $PWD:/tmp/.cache/ aquasec/trivy:0.9.2 <my_private_image>

=> failure

docker run -u root --rm -v /var/run/docker.sock:/var/run/docker.sock -v $PWD:/tmp/.cache/ aquasec/trivy:0.9.2 <my_private_image>

=> failure

sgirouss@LFR038345 MINGW64 ~/Work
$ ls -la /var/run/docker.sock\;C/
total 0
drwxr-xr-x 1 sgirouss 1049089 0 juil. 25 09:31 ./
drwxr-xr-x 1 sgirouss 1049089 0 juil. 25 09:31 ../

[knqyf263]

It seems that you're running Docker on Windows and it uses a different user. What if you add -u sgirouss?

[knqyf263]

We reverted the change and released v0.10.1 because we found Docker actions must be run by the default Docker user (root).
https://docs.github.com/en/actions/creating-actions/dockerfile-support-for-github-actions#user

I apologize for the inconvenience this has caused you.

[NasAmin]

@knqyf263 @rahul2393 Thanks very much for acting on it so fast. I can confirm, the latest version 0.10.1 now works as expected.

[knqyf263]

Thank you for telling us the issue quickly.