Rancher: Feature Request - Webhooks

[bscott]

Thinking Rancher should support Web hooks in the API, this would be ideal to take advantage of Docker Hub automated builds and give a CD type of a feel when deploying containers built by the DockerHub or other CI product.

[janeczku]

:+1:

[ibuildthecloud]

Do you want Rancher to be the consumer of webhooks or publish webhooks? What events in Rancher do you think should have webhooks?

[faddat]

For my purposes, Rancher would be the consumer of webhooks, updating containers when triggered with the webhook. Surely there are times when it would be good for rancher to publish webhooks, but I cannot think of one. I bet someone will, though!

[willseward]

:+1:

[wlan0]

:+1:

[lpgeiger]

+1 Tutum does this to great effect, very easily. https://support.tutum.co/support/solutions/articles/5000513815-triggers

[kaos]

See #52.

[hanselke]

:+1:

[sshipway]

:+1:

[ojkelly]

:+1:

[ShakataGaNai]

I would say that Rancher should both publish and consume Webhooks. For the consumption of webhooks, I'd use it for container/recipe lifecycle actions (deploy, redeploy, terminate) based on either my own application logic or something like Docker Hub.

As for publisher, everything that can be an option to publish, would be fantastic - like GitHub's model. If the hook exists, its easy to build a system that notifies Slack of container activity, integrate into my own DNS and even could notify VictorOps/PagerDuty when my critical containers go offline.

[sshipway]

A simple webhook receiver for scale up/down on a given stack.service would be nice.

Of course, the Rancher API exists which can do all of this, but simple webhooks for common integration points could be helpful.

The main thing we'd like is webhooks going out on events such as configuration changes (host add/del, service start/stop/add/del) and thresholded events (host CPU threshold, host lost contact, service degraded or down) for monitoring purposes.

[ojkelly]

The blocker for me, was having a webhook that would automatically redeploy containers. That is, pull the :latest image and redeploy.

I threw together a node server that could let me automatically redeploy containers for now, but first class integration would be ideal.

[ca0abinary]

Agreed. Also it would be nice to publish web hooks for events like degraded or otherwise failed services which could be used in a monitoring service or even just connected to something like slack.

[ShakataGaNai]

So I hate to mention this because "The other guy does it"-driven-development is never a Great Idea (TM), I want to provide one sample of something from tutum which I think relates as well: Their haproxy image has the ability to connect to the tutum API and in semi-real time add & remove instances from the HAproxy backend - for linked containers.

I think the Rancher/webhook translation could be (for the sake of a similar example) an HAproxy image that uses the regular API to register for webhooks notifications of containers coming/going that are of a certain classification (linked, or a specific tag/name/etc), to add/remove from the backend pool. Again, that'd be almost completely real time.

So haproxy has easy to use monitoring/up/down detection etc, but it could also be nginx which isn't as good at that.

[vincent99]

@ShakataGaNai it's entirely possible to build that against the Rancher API/metadata, but I'm not sure what you think that would buy us.

We already have a haproxy balancer with all those capabilities going the other way (pushing config down to the haproxy container), while being able to manage that the config actually matches across multiple instances on multiple hosts.

[sshipway]

@ca0abinary If you're using Nagios or MRTG, I have a plugin that works against the Rancher API for monitoring. Sure, its an active check rather than an active notification, but it does the job. The only problem is identifying degraded services, because the Rancher API does not (yet) have 'degraded' as a service state even though the web UI does ( #1052 fixes this )
Check plugin: https://github.com/sshipway/check_rancher

[JTurpin]

:+1:

[dbones]

+1

in my case it would be great to subscribe to rancher events.

[hulutter]

+1

[CBR09]

+1

[rvion]

+1

[hwinkel]

:+1:

[mugli]

So is this feature planned? Any hint when we can expect this to be added?

I'm new to rancher, maybe 'release/future' label is supposed to answer that, I'm not familiar with its semantics.

[ibuildthecloud]

This is a highly requested feature so we will add it. release/future just means "not 1.0." 1.0 is due at end of this month and we are past feature freeze. I'd really hope we could address this shortly after, right now we are narrowly focused on 1.0 so we haven't started looking post 1.0. I will say that this feature was supposed to be in 1.0 but we ran out of time.

[Ilya-Kuchaev]

+1

[bezelga]

I am really looking forward to this feature! :)

[nhmtns]

+1

[offroff]

:+1:

[ThatsNinja]

I spent the weekend cobbling some Rancher interaction onto a project I found on GitHub. Maybe this will help bridge the gap:
https://github.com/ThatsNinja/dockerhub-webhook-listener

Also, +1

[ysle]

:+1:

[tristanholl]

:+1:

[emlagowski]

:+1:

[sshipway]

It occurs to me that this feature could be provided via a template item, without too much difficulty. There already exists an (undocumented?) websocket for many events, and a periodic scan could be done for monitoring events; then a standard webhook URL (with optional credentials) configured via a passed env var to pass all events to as JSON (which can subsequently decide itself which to accept and which to drop). If only I had lots of free time to play with the idea...!

[andrewgbliss]

+1

[lewistaylor]

👍

[jjensn]

+1

[demohi]

👍

[mikepc]

+9

[mebyz]

+1

[jadsonlourenco]

+1

[roflmao]

+1

[dyre]

+1

[Gatux]

+1

[werbitzky]

+1

[wheredevel]

+1

[duchy]

+1
Hah, requested more than 1 year ago , any progress?
@deniseschannon ...?

[nfons]

+1

[Leen15]

+1

[mattidupre]

+1 (Awesome work with everything. Rancher is wonderful.)

[hernandev]

If you're looking for single command upgrade and confirm the upgrade, repulling the image on a service, i made a quick script while webhooks are not supported

made a quick script on a docker image do do the job from my ci
https://github.com/codecasts/rancher-deployer

Not really simple but for those using a CI/CD system, you can embed the variables there:

docker run --rm \
    -e RANCHER_URL="http://my-rancher.com:8080/v1/projects/1a5" \
    -e RANCHER_ACCESS_KEY="my_api_key" \
    -e RANCHER_SECRET_KEY="my_api_secret \
    -e RANCHER_STACK_ID="1e9" \
    -e RANCHER_STACK_NAME="myapp-production" \
    -e RANCHER_SERVICE_NAME="web" \
    codecasts/rancher-deployer:latest

[nfons]

@hernandev thats wonderful, i'll try this out

[jared-sage]

@hernandev Cheers m8, thanks, better than nothing.

[Leen15]

I made a wrapper for use rancher with webhooks while they are not yet supported.
If you need you can check here:
https://hub.docker.com/r/leen15/rancher-wrapper/

I'm using this wrapper with jenkins.

[rzachariah]

I'd like a webhook for stack deployments. I'd like to post to a slack channel when a rancher stack deployment begins and ends.

[spalladino]

Inspired on other solutions, we've built https://github.com/manastech/rancher-autoredeploy/, which sets up a small http server for listening requests from docker hub, and upgrades the matching services in the rancher environment using the API. Hope it helps until the feature is implemented!

[deniseschannon]

We will be starting with adding webhooks for scaling up/down services in our next release and look to expand to further use cases.

https://github.com/rancher/webhook-service

[soumyalj]

Tested with v1.4.0-rc2. The webhook feature to scale up/down services is implemented and is functional.

[jmreicha]

Is there documentation on how to use it? Been looking forward to this for a long time 💯

[mrajashree]

The webhook-service will have drivers to execute different actions. It has one driver for scaling up/down a service right now. We're adding more such drivers and even the community can add their own drivers. The steps on adding your own drivers will be explained in a blog to be released in a few days

[mrajashree]

Webhook-service 1.4: http://rancher.com/webhooks-in-rancher/
Docs on using webhook-service: http://docs.rancher.com/rancher/v1.4/en/cattle/webhook-service/
Steps for contributing: https://github.com/mrajashree/Webhook-blog/blob/master/Steps.md
We'd love to get contributions from the community

[JulianKingman]

Top of my wishlist for webhooks is redeployment based on commit to a git repo.

[mrajashree]

@JulianKingman redeployment of services? can you please explain a bit?

[mattidupre]

@mrajashree I think the workflow he's getting at is to be able to push a commit to github, then have a linked service automatically upgrade. That was the only thing I was missing when migrating from Tutum / Docker Cloud. Even without that, Rancher has been worth it x 10.

[JulianKingman]

@mattidupre that's exactly right. Other webhooks would be good, too, like dockerhub

[mnzaki]

I'm on rancher server 1.5.2, trying out the serviceUpgrade webhook listener, and I set up my docker registry to notify the trigger url as provided by rancher server. Unfortunately it doesn't work, docker registry2 does send the request, but I'm getting this in the rancher-server logs:

rancher_1  | time="2017-04-07T15:04:04Z" level=error msg="Error in request: Error Incomplete Docker Hub webhook response provided in executing driver for serviceUpgrade"

How would I go about debugging this further?

update: a little tcpdump-ing and I realized that Docker Hub webhook requests != docker registry webhook requests.
So I'm going to assume there's only support for Docker Hub currently, and that I would need a bit more plumbing to convert to the format that rancher webhooks expect.

upadate: related issue #8445

[mrajashree]

@JulianKingman we have already implemented the Docker Hub webhook in 1.5
https://docs.rancher.com/rancher/v1.5/en/cattle/webhook-service/#upgrading-a-service-based-on-docker-hub-webhooks
Is this what you were looking for?