Brigade: [Question] Where is the state managed?

[sathyatvrcbe]

Hi,
Let's assume that I have a brigade project with a brigade.js script. Assume that I have five jobs that have to be run sequentially and I have scripted the same in brigade.js. Also, assume that I trigger a build using the web hook and the brigade controller executes the build.

In this scenario, after executing two out of five steps of the build, the brigade controller gets killed due to some reason and a new brigade controller will be spawned by the deployments controller. In this situation, will the newly spawned controller have the ability to resume the build from step 3? Or will it restart the entire build from scratch? Or the user has to manually trigger a new build?

The answer to all the above three questions depends on how brigade controller is managing the state of the build. I am not seeing any DB dependency for brigade's k8 deployment. So, I can assume that the state is not persisted in DB. There are two other ways to manage the state: 1) The etc.d cluster and 2) The brigade controller's memory. Please let me know where the states of the builds are stored and how they are handled.

I have another orthogonal, but a related, question. Is it mandatory that at any point only one instance of brigade controller should run?

[radu-matei]

Your final question is discussed in #775.

• a new event is created
• the controller picks the event, reads the project, then creates a new worker
• the worker starts executing the brigade.js file (jobs are created)

All state is managed through Kubernetes - projects and builds are represented as Kubernetes secrets.

For example, whenever a new event is scheduled, when the controller handles it (starts creating a worker), it sets the status as accepted, so it (or potentially other controller instances) will not handle the same event again:

(see https://github.com/brigadecore/brigade/blob/3971ab0555369ed4f1fe99d25537f5f6e6bcbbe5/brigade-controller/cmd/brigade-controller/controller/handler.go#L28)

kind: Secret
metadata:
  labels:
    build: 01dqx0ke62p19hqp0wqvaxhp3q
    component: build
    heritage: brigade
    project: brigade-b9f6da279ec59af95d121d1e1e502ecf056504e267f2f7dfc747eb
    status: accepted

This also ties in to #775, and how we could do leader election.

To specifically answer your question, if the controller gets terminated during the execution, this shouldn't affect running jobs, since they are managed by a Brigade worker, which is per-event.

If the worker, however, gets terminated after running two of the five sequential jobs, the third job would not be started, and in the current state of Brigade, you would need to re-execute the build.

We continue to look into both #775 and #258 (which explores precisely this matter).

[sathyatvrcbe]

Consensus problem is already solved in K8s. Can't we use them to solve this leader election problem?

For example, instead of deploying the worker as pod, if we can deploy it as a deployment(with pod count as 1), the deployment will make sure that there is exactly one pod per deployment. This will make sure that the worker is resilient to failure.

And also if the states of the jobs can be offloaded to the config map, the worker becomes stateless. When it fails, it can get the state of the build from the config map. Since the consensus problem is already solved in the config maps, you don't need a separate leader election.

[radu-matei]

The main issue with offloading the state of the worker into a config map is knowing how to resume the execution of it.

Most Brigade scripts don't just start jobs that are completely independent of one another - you await jobs, pass state between jobs (from standard output, or shared volumes).
How and when the volumes are created, for example, and to what jobs they are attached, is handled by the worker - I expect resuming such a script not to be straightforward for the non-trivial use case.

That being said, I'm happy to go more in-depth into this issue, particularly into direct steps we could take - such as switching to a deployment with one pod for the worker, which would ensure the re-execution of the entire script in case of failure.

[krancour]

fwiw, I do think a lot of this discussion is good food for thought as we start to entertain the idea of a Brigade 2.

[sathyatvrcbe]

I understand it is complex and lots of internal details in implementing this. But, just given an idea how could be done in K8 environment since I have evaluated handful of workflow engines.

[krancour]

This question has been adequately addressed and is no longer relevant to 2.0 planning, so I am closing.