Argo-cd: Scan and report vulnerabilities in our CI pipeline

[edlee2121]

Summary

Need to regularly scan container images for vulnerabilities.

Motivation

We should not ship container images with vulnerabilities that have known fixes.

Proposal

This related issue has a lot of relevant discussion including what tools we might use.
https://github.com/argoproj/argo-cd/issues/2425

[dnascimento]

May you consider upgrade dex from dexidp/dex:v2.21.0 to dexidp/dex:v2.23.0 due to vulnerabilities in the alpine OS?

[jkleinlercher]

Our Security&Compliance guideline only allowes container images from sources, which often scan for security vulnerabilities and release new images as close in time as possible. This would mean, that new argocd images would be needed also if the underlying base image has some urgent security issues. Otherwise our security pipeline (artifactory and xray) would reject those images.
Don't know if argoproj can offer this service or if you can talk with companies like bitnami who seem to have some good experience with that?
The other way would be, that we (and maybe other companies) build the images ourselves, which is not very handy.

[jannfis]

I think in order to implement this, we'd need kind of a support strategy first. For example, do we support only the current release? Do we support N minor releases back? This will also define which set of images we'd need to scan and possibly act upon the results of those scans.

[saradhis]

I think in order to implement this, we'd need kind of a support strategy first. For example, do we support only the current release? Do we support N minor releases back? This will also define which set of images we'd need to scan and possibly act upon the results of those scans.

I suggest that we do N, N-1, and N-2 releases where N is, say 1.5.x. That, is we ignore the patch releases for this purpose.

[edlee2121]

We can start by doing this for new releases and go from there based on community needs.

[jannfis]

OK, some good news first: a re-scan of all images since v1.4.0 using Clair has not brought up relevant, unapproved CVEs with severity >= High that would affect any user of ArgoCD to this date.

All the reported CVEs with severity >= High are not relevant, since either a) they affect the Kernel which is void in containers or b) do not apply to use cases in ArgoCD (i.e. some are Git related, but only in combination with Visual Studio or WSL).

Questions:

1. Do we want/need to tackle the CVEs with severity < High as well? This probably would mean that we need to decide on another base image than Debian, tho. Their images are not updated on a very regular basis.

2. How do we want to integrate & schedule a regular scanning? I would suggest a two-fold approach: First, we integrate a Docker image build (without pushing it to the repo) for every PR into our CI pipeline and scan the resulting image using something like clair-local and clair-scanner. Second, we install the scanning infrastructure on some kind of production grade cluster, with a job for scanning already published images running at a regular interval. This job could then raise GitHub issues upon new findings, which would prompt us to handle accordingly and i.e. release a patch update for affected images.

[edlee2121]

That's great!

1. I'm thinking that we can stick with severity >= High for now.
2. Perhaps we can start with periodic scanning since most PRs won't change CVEs.

[jannfis]

After doing some more research, I think we should use a service to perform security scanning on our code base. I have evaluated Snyk by setting up a PoC for my repository clone and it seems to fulfil all of our needs:

• We can configure all relevant assets to be scanned (Docker images, code repositories, ...)
• It scans regulary at a definable interval (i.e. once per day)
• It notifies via e-mail upon new findings
• It can create GitHub PRs for fixing known vulnerabilities
• It not only statically scans Docker images, but also all of our Go and Node dependencies
• It provides a dashboard with an easy-to-grasp overview of current known vulnerabilites in our assets
• It is pretty much configurable for everything (severities, ignoring certain findings because of irrelevance, etc)
• It can be easily integrated into our GitHub CI pipeline so it will scan when someone wants to introduce new dependencies

Best of all, it's free for Open Source projects - with some of their enterprise features such as detailed reports and history dashboards stripped. I also see that Intuit is already a happy user of Snyk on their reference page.

Using a service would leave us more room to actually code, instead of maintaining another piece of infrastructure such as a Clair and possibly other tools for dependency scanning

[jannfis]

If above proposal is fine for everyone, I'd set up the organization in Snyk and request appropriate permissions on GitHub argoproj organization through Snyk. Someone with the right privileges would have to approve that request.

You can simply create accounts on Snyk by logging in with GitHub SSO. For that I can people to the organization, I'd need their e-mail addresses as associated with their GH account, you could send them to me via Slack DM (username @jannfis).

[edlee2121]

@jannfis Great idea! I just created the Argoproj org on Snyk and sent you invites since I already have most of your emails. I added a repo and container image as a PoC. Feel free to finish the configuration if we are happy with how it works.

[jannfis]

I think we're in a state now where we can work with the data produced by Snyk.

• For the start, we now scan Docker images v1.5.0, v.1.4.2 and latest. As releases progress, we will update this list in the configuration to contain the last 3 releases in their latest patch version.
• Dependency scanning for Go and NodeJS dependencies are in place. Unfortunately, Snyk only scans the primary branch (master in our case). I have seen they are planning to support scanning of other branches as well, so we might have a chance to scan dependencies in our release branches later.
• I have removed all the Kubernetes manifests from the scanner that are pure testing ones, so the only ones left are the ones we actually deliver.

On a side node, I'd really love if we implement static code scanning as well. I have conducted a simple PoC on my fork with the SaaS version of Sonarqube at https://sonarcloud.io, which is also free-to-use for Open Source projects. It provides a lot of detailed and valuable insight into our code, and has an awesome GitHub integration. It could help us greatly to improve code quality and to grow even better.

What do you think, @edlee2121?

[edlee2121]

Cool, the snyk dashboard looks a lot less cluttered now!

Please demo sonarcube to @alexmt and @jessesuen. If they like it, I'll set it up.
Perhaps this may make good demos for the next Argo CD community meeting.

[lcostea]

@edlee2121 is it possible to get access to the scan reports? This would be useful for us at Mambu to be able to see there are no high vulnerabilities on the versions we are using.
If snyk org access is not possible, I guess one way to do it would be to run a CI with snyk cli on tag creation, so this way we could see the pipelines results from GitHub directly. We are opened to other suggestions as well.
BTW, I see go.mod scanning is only available from cli (https://support.snyk.io/hc/en-us/articles/360003817417-Snyk-for-Golang), so I guess some pipelines would have to be created to keep scanning dependencies (after move from dep)