Brigade: Temporary Services with Brigade

[philmay]

For some of the Brigade scripts we are running, we'd like to temporarily stand up supporting services that are available to our Brigade Jobs while our script is running. Once our Jobs have completed, we'd like to stop these supporting services as part of the cleanup process. Is this something that fits the Brigade approach to things (i.e., something Brigade might support), or would this be better handled elsewhere?

What we do now is use Helm to deploy these services and just leave them running, but some of our Brigade triggers are infrequent, and we'd rather not have unused services sitting around idle. Another option we've been kicking around is putting this service initiation into our custom Brigade gateway code – so in response to one of our external triggers, we'd deploy supporting services programmatically, then run Brigade, and then take the services down (haven't dug into the details on this yet).

Any thoughts?

Thanks!

[technosophos]

There are a few ways you could do this:

Option 1. Do this (using helm or kubectl in the brigade.js script:

events.on("someting", () => {
  const myname = "my-services"
  start = new Job("start-extra-services", "somebody/helm:latest")
  start.tasks = [`helm install -n ${myname} something`]
  start.run().then(
       // Do the rest of your stuff
  ).then(() ={
    stop = new Job("stop-extra-services", "somebody/helm:latest")
    stop.tasks = [`helm delete ${myname} --purge`]
    return stop.run()
  }).catch(() ={
    stop = new Job("stop-extra-services", "somebody/helm:latest")
    stop.tasks = [`helm delete ${myname} --purge`]
    return stop.run()
  })
})

Option 2: Use sidecars

Depending on what sorts of services you are standing up, one possible option is to run them as sidecar containers on the worker.

I have never tried this, though I believe @blimmer may have. I'm actually really interested in people's input about whether this is a useful pattern. If it is, I'd be inclined to making it easier to do.

Option 3: Directly create this in the brigade.js by importing @kubernetes/client-node

You can import the client-node library into brigade.js (we'll call that an "undocumented feature" ;-) ), and then create Kubernetes objects directly.

Now, I will be the first to say that this is sort of a violation of "the pattern", in which brigade.js is just a pipeline builder. But... I'm also the first to admit that just because that's "the pattern" doesn't mean any divergence is wrong.

Option 4: Create a separate gateway

Gateways are relatively easy to write (says the guy who writes them for a living). SO as you suggest, that is definitely an open route here. The one downside to this approach will be that your gateway will have to watch the worker to determine when the job is complete and things can be safely torn down. The brig client is actually a gateway implementation, and it does exactly that. So it's a pattern that can be accomplished.

Please keep me updated on what (if anything) you choose. This is one of those emerging areas of Brigade where we haven't yet developed a "best practice" because the use cases are still emerging.

[philmay]

Thanks for this advice, it's really helpful!

I was discussing option 4 on Friday with one of my co-workers (we already have a custom gateway running), and we came to the conclusion that we could do it, but that it was a bit complicated, and that it would be duplicating work that Brigade is already doing. We also discussed option 1, which seemed to be the simplest approach, so I started down that path. I should have option 1 working in short order.

Option 3 sounds interesting, and I may look into that as a possible next step after I get option 1 working with our system. I'll let you know what we do and our degree of success.

My co-worker also brought up an idea similar to option 2, but having the services be sidecars to our Brigade Job pods (the ones that actually need the services) rather than the worker. The idea of attaching them to the worker sounds interesting, so I'll bring that up with the team on Monday. I may reach out to @blimmer if we decide to look into it.

[blimmer]

Hey @technosophos and @philmay - I haven't tried out the sidecar option here - good luck!

[radu-matei]

Closing the issue, but referenced this for the documentation, as it is a really nice use case.
Feel free to reopen if you have additional questions!

Thanks!