Argo-cd: Support for verification of GPG commit signatures

[jannfis]

Summary

Currently, ArgoCD does not support the verification of GPG commit signatures on the repositories it uses for syncing applications to clusters. ArgoCD should provide the option to verify commit signatures on a per-repository basis to enable the verification of the commits to be applied.

Motivation

In some sensitive environments, signing of Git commits using PGP/GPG and only considering the signed ones might be a security requirement.

Proposal

Since ArgoCD must know the public part of the keys used to sign commits in order to verify them, the public keys could be managed using the certificate management API introduced with v1.2. This would probably mean an additional ConfigMap and volume mount for argocd-server and argocd-repo-server.

As for handling unverified commits, other tools (i.e. Flux) just ignore them and don't treat them as a valid revision to sync to. We should handle it similarly, and only sync to successfully verified commits when the option is enabled in a repository.

I would be happy to implement this feature, if this is something we want to support in ArgoCD.

[jannfis]

@alexmt Is that something we should still work on, or is it something we will get from gitops-engine collaboration with Flux?

[stale[bot]]

Is this still relevant? If so, what is blocking it? Is there anything you can do to help move it forward?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[toonsevrin]

This feature would be great for hardened security!

[zephinzer]

This would be really useful for us too- our intended use case is to deploy only on commits done by pre-verified people

[jannfis]

Just for the record, I've started working on an implementation. Hope to have it finished for the next release after v1.5.

[toonsevrin]

@jannfis awesome!! Just an idea for a more extendible approach (I'm not sure how the argo internals work so this may need to be adapted a bit):

Add an "approved: false" label to the Application (or whatever the resource is that actually has the new commit sha) resource that is set to false whenever a new release is created.

Now we can add our own approval workflow (through controllers), eg. one that checks for a git signature, one that checks for multisig on tags with our own custom implementation etc. etc.

[jannfis]

@toonsevrin Although I like the idea behind your suggestion, I'd guess it would be a whole different tier than what I had in mind for this enhancement. In the first throw, I'd like to implement syncing an application only against Git commits that where signed with a key of a configurable range of allowed keys and implement basic key management via CLI/UI.

Your suggestion goes a little further, as to an approval for a sync from outside. Basically, I could imagine that there'll be a new field in the application CRD, which is used to specify whether approval is required and if yes, what kind (for example, single approval, n-of-m approval etc). Then, from the outside, you could modify the status of an application CR (i.e. argocd app approve or using ArgoCD API). This approval also would have to be cryptographically signed, i.e. a signature over the commit SHA value itself and attached to the "status" section of an application. So ArgoCD could verify the approval for the current Git commit SHA and then the sync would be allowed to trigger to the commit.

This could be used in combination with GPG commit signature, because the approvers could be sure that the commit is comming from a trusted source. Or it can be used on its own.

I'd be happy to discuss this further, because it could satisfy some requirements that may exist especially in heavily regulated corps.

[toonsevrin]

Hi @jannfis, you're right. Let's focus on the high value, low effort for now. The more advanced case could be implemented once there is interest for this.

I don't think you have to cryptographically sign the approvals percé. I'm not sure how granular we can go with kubernetes RBAC, but RBAC could probably just be used to protect the approval flow.

Also, regarding the usage in combination. The commit signature checker should just be another approver. Even if the repository can trick the other approvers with fake commits, the deployment will still fail due to this approver not approving it.

Thanks for implementing this btw.

[toonsevrin]

Oh also: If there are any organizations that may need this behavior in the mean time:

You can implement it yourself by always specifying the commit sha in the revision field of your Application. The commit sha acts like a checksum of your code so automatically verifies integrity.

Then you can write your own approver (a controller application) that tracks your git repository, and whenever a tag or commit is made, it checks your approval workflow. If the commit is approved, you update the revision field to the commit sha of the tag/commit.

Note: you will want to lock down (through rbac) argo so that this controller is the only agent that can edit the revision.

[toonsevrin]

Another argument against the advanced functionality:

If you are a large organization you can create a service that signs tags for you (with a private key that is generated once by the service itself). You can then add all sorts of requirements to the signature of this tag.

Once @jannfis finishes his implementation of signature validation, you simply add the public key of the service to argocd.

If one of my organizations ever needs this level of security, we will open-source our implementation here.

Conclusion: Let's not add the advanced modular approval functionality to the core of argocd.

[jannfis]

As a quick heads-up: I'm making good progress on this. So far, I've implemented all the primitives required for key management and signature validation as well as adaptions to the repository server and application controller as to enforce signature verification and handling according to its results.

Currently, to enable signature verification on a project level, one has to specify a list of valid (trusted) PGP key IDs in the project spec, like so:

spec:
  signatureKeys:
  - 4AEE18F83AFDEB23
  - ...

The key(s) specified must be in the GnuPG key ring known to ArgoCD (next step is to implement managing the keyring in a basic way using CLI, UI and API), otherwise signature verification will not succeed.

Now, if signature verification is enabled, the application controller will instruct the repository server to run a signature verification on the revision it wants to compare the live state to, and the repository server will then run a git verify-commit and pass the result back to the controller. The controller then decides whether the signature was a) good and b) made using a valid key (as configured in the project spec). If a + b are fulfilled, it will perform & try the sync, otherwise it will emit a ComparisonError and let the user know the reason (i.e. no signature on commit, bad signature, signature from an invalid key, ...).

The application controller will still use any non-signed commit for the comparison, and also will display the differences from a possible untrusted source. This was a design decision I made for no specific reason, but I'd like to hear opinions on that - it would also be possible to emit an error on the sync status and push the application into "unknown" state.

@toonsevrin I like the approach of an external entity performing the final signature once you need n-of-m or more complex form of authorization. I'm looking forward to see your project coming to life!

[toonsevrin]

Awesome!!

Quick note, please make the Spec

spec:
  signatureKeys:
  - key: 4AEE18F83AFDEB23
  - key: ...

to follow the kubernetes CRD guidelines. This is more future-proof (eg. if you want to add a name or expiry field or whatever in the future).

Regarding your note on the state with an untrusted source: OutOfSync is fine with me if that's what you are referring to. The most likely case when you are in this state is when you forgot to sign your commit/tag, which makes the resource permanently out of sync with your desired state (the unsigned commit). Anyways, this is indeed very subjective but not too important to me, as long as I can somehow find out that my resource is in a state where the signature of the latest commit is wrong.

And thanks! As I am the only developer and operator in most of my companies I doubt that will be very soon though :+1:

[jannfis]

Quick note, please make the Spec

spec:
  signatureKeys:
  - key: 4AEE18F83AFDEB23
  - key: ...

to follow the kubernetes CRD guidelines. This is more future-proof (eg. if you want to add a name or expiry field or whatever in the future).

Thanks for this suggestion @toonsevrin, it makes total sense. Will change the spec and adapt the code so that keys are specified as array of maps, instead as array of strings.

[toonsevrin]

@jannfis array of objects* ;) that way you can do better openapi validation

[jannfis]

Initial implementation was merged in PR #3242 right now :tada:

[toonsevrin]

Good job @jannfis !!