Flux: support triggering via webhook

[davidkarlsen]

Having to wait for flux to detect changes in the git repo is a bit of a drag / delay. It would be nice if flux could support being called via a webhook to trigger immediate refresh of the repo.

[squaremo]

We have a prototype of this in weave cloud, and the hooks are there in the daemon (i.e., it's just a matter of some engineering in weave cloud keeping it back).

We could also have these in the daemon. It's a bit trickier, because

1. you don't usually want to expose the flux API to the internet (so you've have to have some kind of ingress for just some routes)
2. it wouldn't necessarily deal with all the formats of payload, e.g., from github etc.

[davidkarlsen]

Great news - I guess it can be a bit nicer in weave cloud (UI etc).

It would make sense to have in the daemon for users running flux only / on-prem / no weave cloud. Since the git setup is really in the daemon the format should not matter much if it is just a signal to pull the configured repo (e.g. no need to parse payload).

[Morriz]

+1
it is how most CICD workflows are triggered...check out how drone.io does it: (example: github.com) you create an oAuth app with creds for drone to be able to create a deploy hook automatically (generating a unique token per webhook) for the repo it wants to register for. So no need to manually create a deploy hook for every repo, just configure for your git host (github or whatever) once.

It would be most elegant to create a dashboard service (with a nice UI showing all flux releases), which would be accessible to these app generated webhooks. That service would then publish those events to the flux instances, triggering them to do their thing. Maybe that is possible with hosted Flux, but I am not aware of the paid offering's options...

[fikriauliya]

Has it already in available open source version?

[warmfusion]

This feels like a real problem to support clear pipelines for CI/CD workflows. Our dev team wants to _know_ their deployment is activated as they would then execute secondary flows such as cache clears, routing behaviour or smoke/QA testing against the deployment.

Without having an explicit hook (and ideally a blocking wait) we can't use Flux for any deployment on our systems as the async delay reduces our options for automated hands-off continuous delivery and notification.

[squaremo]

Our dev team wants to _know_ their deployment is activated

This is less to do with webhooks and more with providing a layer of interpretation on the Kubernetes API.

But I do grasp the requirement: to be able to spin up tests, you need to know when the system under test is ready.

Without having an explicit hook (and ideally a blocking wait)

What would the wait wait for? To put my cards on the table: I think it's possible to give a practical answer, or a general answer, and very difficult to give a practical, general answer :-)

[warmfusion]

providing a layer of interpretation on the Kubernetes API.

True, but isn't that basically what Flux is for? Providing the bridge between deployment state and deployment intent?

What would the wait wait for

Maybe something like

• 'Status - In-Sync: Cluster consistent with $commit_ref' - Exit 0
• 'Status - Updating: Eventual consistency targeting $commit_ref' - Blocking
• 'Status - Error: Timeout reached whilst awaiting consistency. Review service state' - Exit 1
  
  
  
  • Timeout set by some fluxcli sync --timeout 60s type argument
  
  
• 'Status - Error: Cluster resources exhausted' - Exit 2

Isn't this covered by the existing features in the stack;

[derrickburns]

The interface should also include freezing the deployed configuration once it is consistent with a given commit. The freeze is necessary to ensure that tests can run without a change to the environment.

awaitAndFreeze(commitID)
runTests()
unFreeze()

[alanjcastonguay]

I fear feature creep away from the original scope. I understand the desire to avoid a potential several-minute lag between a new commit/image becoming available and the normal Flux poll cycle discovering it. Not by providing a new behaviour, but by jumping the clock and polling _now_. Which barely requires auth, as no directed change is possible (eg specific image/commit, locking), and no information is returned (eg whether an image was found/deployed). Those are already exposed in the API fluxctl consumes.

[squaremo]

I fear feature creep away from the original scope

For sure. Perhaps @warmfusion, @derrickburns, we could move this discussion to the mailing list. It seems like there's a bunch of related requirements and desiderata beyond simple webhooks, that need some curation.

[dotdc]

I'm also very interested by the feature, do you have any plan to implement this in the daemon?

Is it an option to expose an API endpoint like /api/v1/git-sync and protect it by using Basic, JWT, Oauth2.0...
In case the repository is commit intensive, it's maybe a good idea to consider a rate-limiting argument to set the number of seconds between 2 syncs, or something similar.

[alanjcastonguay]

I would like authentication (along with only exposing that one /route) to be the providence of an Ingress controller in front of Flux, with Flux documentation recommending an appropriate config.

Rate Limiting though... I suspect we really only want a maximum of one in-flight sync, rather than a request rate limit. If a sync is in-flight and one or more sync webhooks arrive, they should be batched into a single needs-another-sync-soon state that can be consumed/cleared. Is that equal to a golang buffered channel of length 1?

[squaremo]

Rate Limiting though... I suspect we really only want a maximum of one in-flight sync, rather than a request rate limit. If a sync is in-flight and one or more sync webhooks arrive, they should be batched into a single needs-another-sync-soon state that can be consumed/cleared. Is that equal to a golang buffered channel of length 1?

Yes, that and an non-blocking put; and in fact that's what fluxd does: https://github.com/weaveworks/flux/blob/master/daemon/loop.go#L143

[gled4er]

Hello,

@squaremo pointed me to #2211 and I wanted to confirm with the people on the thread if the NotifyChange can be used by Flux for notifying external system that Flux just made a change.

If this is not the case do you have an ETA when the functionality requested with this issue will be available? I don't see it included in any of the available milestones at the moment.

Is there design guidance we can follow to try to implement it?

Thank you!

[alanjcastonguay]

The /api/flux/v11/notify requires a json payload indicating the kind of notification (git/image) and information about the source.

curl -X POST http://127.0.0.1:3030/api/flux/v11/notify -d '{"kind":"git", "source":{"url":"[email protected]:alanjcastonguay/flux-get-started.git", "branch":"notmaster"}}'

ts=2019-07-29T20:46:20.1246356Z caller=loop.go:111 component=sync-loop event=refreshed [email protected]:alanjcastonguay/flux-get-started.git branch=master HEAD=8e4bcf418bbd0c13fb3bbb2303b3557665da2aa9

curl -X POST http://127.0.0.1:3030/api/flux/v11/notify -d '{"kind":"image", "source": {"Name":{"domain":"", "image":"stefanprodan/podinfo:1.6.1"}}}'

ts=2019-07-29T20:58:10.4301986Z caller=warming.go:95 component=warmer priority=stefanprodan/podinfo:1.6.1

But the client can't be trusted to specify the git repo or image name, and an Ingress controller or reverse-proxy is not in a position to iterate through various urls/images. There doesn't appear to be an method to _check all configured repos/images_ here.

The older /api/flux/v9/update-manifests also requires a json body but it's fixed for a ManualSync, no repository url needed. It's possible that an Ingress controller or HTTP proxy in front could accept any HTTP Post and override the body with this exact one. But it doesn't appear to try find new images, only new git commits.

curl -X POST http://127.0.0.1:3030/api/flux/v9/update-manifests -d '{"type":"sync", "spec": {}}'
"2cfc96ce-a8e0-4af8-3803-990aa62ca945"

ts=2019-07-29T21:19:57.8999575Z caller=loop.go:119 component=sync-loop jobID=2cfc96ce-a8e0-4af8-3803-990aa62ca945 state=in-progress
ts=2019-07-29T21:19:58.4225543Z caller=loop.go:131 component=sync-loop jobID=2cfc96ce-a8e0-4af8-3803-990aa62ca945 state=done success=true
ts=2019-07-29T21:19:58.9192476Z caller=loop.go:111 component=sync-loop event=refreshed [email protected]:alanjcastonguay/flux-get-started.git branch=master HEAD=8e4bcf418bbd0c13fb3bbb2303b3557665da2aa9

Neither of these are pleasant for use as-is as a webhook interface.

[proskehy]

Since I was looking for this functionality and couldn't find it, I took a look at how the above mentioned API methods work and added one that syncs the git repo without requiring any arguments. I'm not sure how it may fit in the larger scheme of Flux and what the maintainers have planned, so if this isn't something they'd like to merge, I hope that at least someone looking for this functionality might find it helpful.

With the above API in place, all you really need to do is add an Ingress that exposes it and preferably protect it in some way, at least with basic auth:

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: flux-git-webhook
  namespace: flux
  annotations:
    kubernetes.io/ingress.class: nginx    
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
    nginx.ingress.kubernetes.io/rewrite-target: /api/flux/v11/sync-git
    nginx.ingress.kubernetes.io/auth-type: basic
    nginx.ingress.kubernetes.io/auth-secret: flux-webhook
    nginx.ingress.kubernetes.io/auth-realm: "Authentication Required"
spec:
  rules:
  - http:
      paths:
      - path: /flux/sync-git
        backend:
          serviceName: flux
          servicePort: 3030

Now you can call whatever address your Ingress exposes on /flux/sync-git, spcifying the basic auth as user:pass@ in front of the address and it should work :)

[alanjcastonguay]

That's the same approach I had in mind, though I was really trying to use an existing API. I didn't find one and got distracted by other work.

[proskehy]

Me too, but unfortunately I didn't find one either. Though at least for the purposes of notifying Flux about new Docker images, I think I will be able to use /api/flux/v11/notify by simply calling it from a CI/CD pipeline.

[squaremo]

But the client can't be trusted to specify the git repo or image name, and an Ingress controller or
reverse-proxy is not in a position to iterate through various urls/images.

It was never the intention that an ingress or reverse proxy would call /notify directly, since _neither_ would they (in general) be able to verify the incoming payload. The way I imagined doing this was to have a small service that can be the webhook receiver -- which means verifying the payload, extracting the git URL and branch (or image repo), and then posting to /notify. The ingress or reverse proxy can then target that service.

[proskehy]

The way I imagined doing this was to have a small service that can be the webhook receiver

I was actually thinking about doing that, but I wasn't sure how to get started, since I'm not very familiar with writing stuff in Go. So just making that "dumb" method, which could theoretically only do very little harm was the way I chose.

It does sound interesting though and I would like to try help with it if that's something you're considering adding.

Edit: another thing is though, why the need to specify a repo in the first place? Flux can't watch multiple repos, can it? So I can really only see the usefulness of a "webhook client" for the image updates.

[squaremo]

why the need to specify a repo in the first place? Flux can't watch multiple repos, can it?

No, but conceivably the service invoking the webhook might only let you install a webhook for several git repos (e.g., for a team or account).

There is already a similar limitation with github: you can install a webhook for pushes to a particular repo, but you don't get to choose the refs you care about. That's why we pass the git repo and ref to the daemon in ../notify, so it can choose what to pay attention to; otherwise it would be constantly pinging the git service for updates, only to learn nothing has changed.

[squaremo]

I've said elsewhere, and it's worth repeating here: I think webhook receivers would make a nice contribution to fluxcloud*. It's not that I want to disown or deflect the premise of this issue -- my reasoning is that 1. because public endpoints ought to be separate from the flux API, they should be in another component, and 2. fluxcloud is another component that already takes the role of "adapting flux to the outside world".

(*NB I have not verified that the fluxcloud maintainers share my view)

[squaremo]

It would be nice if flux could support being called via a webhook to trigger immediate refresh of the repo

Please have a look at https://github.com/fluxcd/flux-recv and see if it will serve your purpose. That is built specifically to receive webhooks and pass them on to fluxd. Currently it understands GitHub, GItLab and Docker; it is a fairly simple matter to add support for another webhook source.