Brigade: Success status is reported even when job pod fails

[omgjlk]

When there is a failure in the job pod, such as described in #111 , Brigade still provides back a success status. Surely this should be an error status.

[technosophos]

I see you're testing several things that are all closely related to some of the GitHub handling. There are a couple of in-flight features that we are working on to overhaul the upstream status part of Brigade. They'll be landing in sequence over the next few weeks, and will hopefully vastly improve the CI-like use-case for Brigade.

I apologize for the fact that you're bumping up against a few rough edges that we left when we tore out one system before we had its replacement done.

[omgjlk]

No worries! Y'all have been pretty responsive.

[technosophos]

We took another step toward making it possible to write your own GitHub status responses here: https://github.com/technosophos/github-notify

We're getting closer to having the big picture fixed for GitHub status messages. Ideally, we want to automate it rather than require script devs to have to manually code it to send status messages

[technosophos]

307 gets us a little bit further. Then I will have to re-focus on the GitHub notification piece.

[technosophos]

At this point, we need to decide whether the GitHub gateway is going to provide all of the notification support, or none of the notification support. Right now, it accepts the webhook and returns 200. Then when it schedules a worker, it sets the status to pending. Then it does nothing.

The project's brigade.js illustrates how to give more feedback via a Job. But it seems like we're sorta halfway between implementations. We need to cut the Gordian knot and just decide how much the github-gateway is going to do.

[mumoshu]

@technosophos I guess that it slightly relates to what I have been thinking for a while.

Then when it schedules a worker, it sets the status to pending. Then it does nothing.

The current implementation sounds reasonable enough to me.

But if we go further, my best bet is to slim-down gateway - make it notify nothing at first. while encouraging users to do notifications in project's brigade.js, and then introduce a kind of durable workflow engine to brigade(!).

---

Suppose a durable workflow engine as a stateful service to complete your DAG of jobs by handling:

• retrying / tolerating temporary brigade/node/job pod failure
• timing out
• notify on lifecycle events of tasks

Doing so will allow us to reliably provide notifications for example. No build remains pending when your (brigade component | brigade script | job pod) failed in the middle of processing.

We can of course implement a light-weight workflow-engine-like thing inside each brigade gateway. But I feel like it is just a reinvention of the wheel.

---

So, repeating my previous sentence, I'd suggest slim-downs of gateways, while deciding brigade's scope to not include a durable workflow engine. Then we can investigate integrations with another workflow-engine-likes to provide durability to brigade scripts. IMHO, the integration may end up including:

• A brigade gateway to invoke a durable workflow. For example, submit a argo workflow.
• A connector to allow external durable workflows to brig run $project -e $event_as_you_like.
  
  
  
  • It should be user's reposibility to make a single brig run idempotent, so that the workflow can retry it whenever necessary.
  
  
  • If the user don't want to retry the whole brigade script, he/she can freely split the script into multiple event handlers like events.on('step1', ...) and events.on('step2') so that the workflow can retry step1 and step2 independently.
  
  
• For example, reliably marking a github PR status to be failed when a brigade script failed could be done on the workflow engine, not on brigade alone. The workflow brig run -e buid_failed, brigade runs events.on('build_failed') to mark the PR status failed.

Let's keep Brigade doing one thing very well - scripting, not running durable workflows!

[mumoshu]

// Happy to create a dedicated github issue to discuss about the above. Sry I went too far from the original topic of this issue 😓

[technosophos]

We should create a second issue for that. It's an important thing that I know a few people have asked about, and I agree with you're core intuition: Brigade is about scripting.

[disha94]

I had a similar concern, like how do we update the status of brigade job failure/success on github??

[technosophos]

The main way is to use the github notifier. Here's an example from Brigade's brigade.js

https://github.com/Azure/brigade/blob/master/brigade.js#L121

Basically, the current GitHub gateway marks events as "pending", but never changes them. The Brigade script itself has to do that part.

We're working on a new GitHub gateway that uses the newly released GitHub Checks API (which is fantastic!). That will work a little differently. But it, too, will require the brigade.js to send GitHub status updates.

[omgjlk]

I was hoping you'd pick up the Checks API changes!

[radu-matei]

Closing as we're moving towards having the new Brigade GitHub App as the default.

Feel free to reopen if you have additional questions. Thanks!