Brigade: Limiting the number of concurent builds

[dunglas]

Hi,

Is there a way to limit the number of concurrent builds (a queue mechanism)?
Thanks!

[bacongobbler]

I don't believe there is, but it'd be a nice enhancement!

[mumoshu]

@dunglas @bacongobbler Hi! Sounds like a great enhancement for me, too.

Would mind clarifying at which level(s) you'd like to limit the number of concurrency though?
Are you referring to only BUILD as you wrote so?

1. Brigade installation(You limit it so that your whole brigade installation could have up to N concurrent builds at a time)
2. Brigade project
3. Brigade build(A brigade event would trigger a brigade.js run, which may have up to N concurrent jobs at a time)

You seems like intended 3 but in case your purpose was to limit resource usage per brigade installation/project/worker, IMHO it would be worth considering.

For example, we could potentially utilize k8s resource quota to limit build concurrency.
You tie your brigade workers to specific namespaces w/ appropriate quotas while specifying resource limit/request to pods created via brigade jobs. No more than N = allowed_resource_per_ns / pod_requested_resource jobs would be executed concurrently.

[technosophos]

There is actually _a_ limit, but it's a limit to the number of workers a controller will start at once. And I don't think we expose configuration anywhere. And that limit might not be important in this case. It's really to prevent the controller from firing too many Kubernetes events.

To add on to the discussion above, another limit to consider would be to limit the total number of _jobs_ running at once. For example, one build could spawn many, many jobs. Right now we don't have a limit on that at all.

[disha94]

Hi, I have the same concern, if there is a way to limit no of concurrent builds, so has this issue been taken care of?

[technosophos]

No, but a +1 vote on the issue will make a difference. Our assumption has been that Kubernetes, not Brigade, was the thing that should handle throttling, so we haven't put any real effort into limiting the number of builds.

Are you mostly interested in limiting number of builds, or number of jobs? And are you thinking _overall_ limits, or limits of builds per project, etc?

For overall number of builds, I think we could manage this in the controller. Others would get tricky.

[disha94]

@technosophos , I am looking for limits of builds per project, so its like at a time only 1 build should be running of any project and branch. Is that possible?

[elvisace]

+1 crashed our cluster from too many requests coming in

[dgkanatsios]

So, I was taking a look at the controller code today and there might be two ways to actually tackle this.

-- Possible solution 1 --

• Inject some code here so as to get the total number of pods that are part of a Build (question: both worker and job Pods? Worker(s) only? Job(s) only? I suppose we can go down the Worker way, so that we would select Pods that have Labels component=build and PodPhase Running).
• If this number is below a threshold (set via controller ENV?), then we're good, proceed as usual to create Worker
• If not, return a certain error (e.g. an error starting with a pre-defined string, like Cannot create Build because there are too many Builds currently running
• this error will arrive here and will eventually be handled here. We can inject a piece of code that will
  
  
  
  • check if the error starts with the designated string
  
  
  • if so, it will
  
  
  • do c.queue.Forget(key) so that we remove the Secret key from the queue
  
  
  • call c.queue.AddAfter(key, duration) so that the Secret key will be re-added in the queue but will be processed at a later time
  
  
  • return
  
  
  • of course, we have to think of a reasonable/configurable value for the duration argument
  
  
  • else, it will continue normal error processing
  
  

-- Possible solution 2 --

Haven't thought this thoroughly, but this solution would require

• inject code into handler here so that if the threshold has been reached (or is about to be reached), controller will not create a new worker Pod
• we also need Pods monitoring in the informer so that we can see whether a Worker Pod is in the Completed Status
• when a Pod reaches the Completed Status, this implies that we have some "room" in the cluster to create new Pods. There, we could check if there are any Secrets with Labels["status"] != accepted and do c.queue.Add(key) on them

What do you think? I personally think that the first solution is simpler to implement. Needless to say, we'd need a lot of testing on that one.

[squillace]

How do we treat the namespace/rbac issues? In either solution?

BTW: I'm cool with an iterative solution: new pipelines can occur in the same ns/rbac domain as the original project at first, we can iterate to enable configuration in future, that kind of thing. anything that makes it support rbac properly but enables future config is fine by me.... baby steps!

[dgkanatsios]

Are you referring to #755? In both solutions, controller will continue to monitor/create Pods in the namespace it's supposed to (either in the one used today or in the separate one proposed by #755).

[squillace]

Yes. That's exactly what I mean. Perfect. I have no opinion on the implementation to choose; I leave that to the team. I just want to ensure that the end result gives the user what is expected. 👍

[radu-matei]

I think the desired solution would have to be a combination of the two.

The bit I'm not particularly comfortable with from the first solution is c.queue.AddAfter(key, duration) - in the right context, a build that was recorded at time T could be indefinitely starved from execution (in the interval from T to T + duration, previous builds finished the execution, and new ones were scheduled, and by the time the initial build is added again in the queue, we are already at the limit of concurrent builds.)

Also, let's leave the namespaced jobs discussion aside for now.
Thanks!

[dgkanatsios]

and by the time the initial build is added again in the queue, we are already at the limit of concurrent builds

in this case, it will be removed and re-added again with the proposed mechanism (queue.AddAfter). Though I understand your concern that if multiple Builds are queued all the time, there is a chance that this Build may be postponed and postponed again etc. But, again, if multiple Builds are supposed to be queued all the time and (in result of that) some Builds will not happen, this might signal an issue with current cluster capacity.

The main problem I have with second solution is the potentially wasted CPU cycles for the PodUpdated watcher/handler. Unless I'm missing something, Pods will be updated all the time and each time we need to

• get the Pod
• get its PodPhase
• check if it's Completed
• if that's the case, check if there are Secrets with Labels["status"] != accepted
• etc.

[radu-matei]

But, again, if multiple Builds are supposed to be queued all the time and (in result of that) some Builds will not happen, this might signal an issue with current cluster capacity.

In the context of manually setting the limit to 1 concurrent build for a project - reasonable requirement - (even if the cluster has enough capacity to run multiple builds) - we could end up in a situation where events for the same project are not handled in order - and with this approach, it might be a very common situation.

potentially wasted CPU cycles for the PodUpdated watcher/handler.

If we decide to handle the number of concurrent builds _per-project_, then we can use a labeled shared informer, which only executes when there are events happening for the worker pod for our specific build.

[dgkanatsios]

gotcha, you're talking about limits per project whereas I'm talking about limit per Brigade installation. What I also understand is that it's not 100% clear to me why we need this limit. The reason I referred to Brigade installation limits was that I imagined that this limit would protect cluster's health (so clusters with limited capacity would not have any Pending Pods etc).

I also used the same reasoning (Brigade installation limit) to suspect that there would not be any cases with limit == 1. But I _totally_ understand your concern, if we have a CI/CD pipeline (or others scenarios) it would make total sense to have limit set as 1. If, on the other hand, we had a Map-Reduce pipeline, ordering would not matter so limit could be > 1. However, I think that Brigade should not guarantee Brigade pipeline ordering - K8s is an eventually consistent system, so Build N+1 could run before Build N (rare, but could happen).

So, TL;DR -> agree that limit == 1 scenario would fail with Solution 1.

If we decide to handle the number of concurrent builds per-project, then we can use a labeled shared informer, which only executes when there are events happening for the worker pod for our specific build.

rough notes here to slowly start with the design

Informer would need to watch/list labeled pods indeed, they would need to be monitored for

• Label Component == job
• Existence of Labels i) build ii) jobName iii) project iv) worker

Only UpdateFunc would need to be created, it would

• get the Pod
• get its PodPhase
• check if it's Completed
• if that's the case, check if there are Secrets with Labels["status"] != accepted
• add these Secrets to the workqueue

WDYT?

[technosophos]

/cc @adamreese

[vdice]

As decided in the last Brigade Dev Meeting, as discussion/design is still ongoing for this feature, we're going to remove the v1.0 label while anticipating this would be one of the first features for a post-1.0 release.

We did bring up the possibility of adding in a change to expose/allow configuration of a Brigade-wide build limit, as it sounded like this would be of benefit for low engineering cost. If this sounds reasonable, a PR would definitely be in the cards for 1.0.

However, as it sounds like a project-level limit will be trickier to engineer, we'd prefer to bump to post-1.0.

cc @dgkanatsios who has spent a great deal of time/thought on this feature!

[vdice]

Removing the v1.1 label as we haven't seen much recent momentum on this issue (and we hope to cut v1.1 shortly).

[bnadeem]

Is there an update on this feature? If not, any workarounds? Thanks

[radu-matei]

@bnadeem - the current workaround is to limit the total number of pods in the Brigade namespace - see https://kubernetes.io/docs/tasks/administer-cluster/manage-resources/quota-pod-namespace/

[bnadeem]

Thanks @radu-matei , I did put a limit on the pods as suggested but instead of queuing the build, it simply fails it. In our use case, we want a max of 1 build happening at a time as we don't want more than one process updating our staging environment.

[technosophos]

For this case, you could do something like this:

• Write a job that creates a lock file in the cache, or if a lock is found, waits for the lock file to be deleted and then creates it, waits for the pipeline to end, and then destroys it
• Use that as a gate-keeper job for your pipeline

[krancour]

This is well covered by the 2.0 proposal, which has been ratified and is now guiding the 2.0 development effort. It probably doesn't make sense to track this as a discrete issue anymore.