Brigade: [proposal]: persist all build logs

[krancour]

Related to #1034

Currently, the only record of a build is its underlying Kubernetes secret (see #1036). Related-- the ability to inspect a build's results (view logs, etc.) depends entirely on the continued existence of the pods that implemented that build's jobs. This effectively presents users with a choice between accepting a cluster littered with completed pods or cleaning up periodically and losing a potentially valuable papertrail of build results-- which for some use cases in some industries may not be an option.

It seems that there would be a lot to be gained by streaming worker and job logs somewhere where they can be persisted. Doing so would enable us to clean up completed job pods more aggressively-- perhaps even having the worker do this immediately upon completion.

[radu-matei]

This request totally makes sense in the context of #1036.

One example of this already happening is when using the GitHub gateway (https://github.com/brigadecore/brigade-github-app/) - we're using sending all job logs to the GitHub API, where they are persisted, regardless of what happens with the pods, or even with the Kubernetes cluster.

I think the approach with no breaking changes (for 1.x, at least) would be to create a new component with a pluggable backend - however, if we are to maintain backwards compatibility, this component would have to scrape logs from the pods, as opposed to the Brigade controller / worker sending the logs to it.

What do you think?

[krancour]

I think the approach with no breaking changes (for 1.x, at least) would be to create a new component with a pluggable backend - however, if we are to maintain backwards compatibility, this component would have to scrape logs from the pods, as opposed to the Brigade controller / worker sending the logs to it.

My intuition says the easiest approach might be to use fluentd to forward logs from all relevant pods to... _someplace else_. I don't know where that someplace else is right now. i.e. We don't currently have a datastore other than k8s itself. I was imagining that in the 2.0 time frame, we'd identify one that meets the needs of multiple issues that are under the #1034 umbrella.

[vdice]

Assuming the fluentd/fluent-bit approach looks good, perhaps we can first add these components for log forwarding and if a configured datastore is supplied, ship logs off-cluster and delete worker/job pods. If none is supplied, we'd preserve the default of retaining worker/job pods.

[krancour]

ship logs off-cluster

They don't necessarily have to leave the cluster. They could, but don't have to. Whatever datastore we end up using, I think it would be good to have the option to run it locally on-cluster or point to a remote one-- e.g. a managed service, perhaps.

If none is supplied, we'd preserve the default of retaining worker/job pods.

This seems complicated to me. As an end user or even as an operator, I don't want to have to grapple with the equation of "if I configure a persistent logstore, Brigade will clean up after itself and if I don't, it won't." Can we just have it do the right thing always and then the only question becomes one of how flimsy vs how production grade is your datastore?

If it helps, can we look at this strictly as a breaking/2.0 change and say this is the way things work in the future?

[vdice]

They could, but don't have to. Whatever datastore we end up using, I think it would be good to have the option to run it locally on-cluster or point to a remote one-- e.g. a managed service, perhaps.

Good point.

Can we just have it do the right thing always and then the only question becomes one of how flimsy vs how production grade is your datastore?

Fair. FWIW, I am also eager to see the ever-growing number of completed worker/job pods disappear :) As suggested, perhaps this strictly being a 2.0/breaking change would be best to avoid complex/costly concessions for 1.x-compatibility.

[krancour]

btw, just in case anyone reading this issue might feel that simply tidying up the cluster (removing completed worker and job pods) isn't sufficient impetus for shipping/storing logs elsewhere, there's also this point that @vdice raised to me in a side channel the other day, which I think is worth committing to the public record here as part of the rationale for this improvement:

With log shipping, a Brigade setup would be much closer to supporting cluster scale-downs (either auto or manually) w/o loss of job history/details

This is such an excellent point. As things stand today, you can lose valuable logs just because you lost the node that the completed pods where scheduled to-- which is an event you may have no control over.

[krancour]

This is well covered by the 2.0 proposal, which has been ratified and is now guiding the 2.0 development effort. It probably doesn't make sense to track this as a discrete issue anymore.

[stavalfi]

@krancour if i don't care about any artifact of a build, is it safe to kill the pods using kubectl? what is the workaround to kill all pods of a build?

running nodes cost money. what is your recommandation (for now) to kill builds (all their pods) that are not needed?

[krancour]

@stavalfi

running nodes cost money

It does, but completed pods don't consume CPU or memory because their constituent containers are no longer running, and afaik should have no impact on the ability to either manually or automatically scale (scale _in_, in your case). Upon scaling in, all pods on a decommissioned node will be deleted (see here), regardless of whether they're running or completed.

Eviction of a running job pod obviously has a bearing on worker success, as the worker may error or time out while waiting for job completion. Eviction of a running worker pod might orphan running jobs.

Eviction of a completed job or worker pod has no impact on Brigade other than what has been discussed previously in this thread. i.e. When a completed job pod is deleted its logs are gone and no evidence remains that it ever ran. When a completed worker pod is deleted its logs are gone and the only evidence that remains would be the build secret that initially _triggered_ the worker.

(As an aside, in 2.0, which is currently under development, workers do not directly manage job pods-- rather they _request_ the creation of job pods and Brigade's new API coordinates their execution and monitors for completion-- meaning an interrupted worker still fails, but can't orphan its jobs because someone else is watching. But I digress.)

Bottom line is, I don't think you have anything to worry about, unless you're a neatnik like me and regard your growing pile of completed pods as unwanted clutter...

if i don't care about any artifact of a build, is it safe to kill the pods using kubectl? what is the workaround to kill all pods of a build?

... which seems to be the case.

This is what Brigade's vacuum component is for. It's a Kubernetes cron job that runs periodically and can be configured to delete all completed builds (and their associated worker and job pods) older than a certain age or in excess of a certain maximum count. In your case, it seems you could configure that vacuum to run frequently and delete all completed builds in excess of 0.

See here for configuration options.

[stavalfi]

@krancour

thanks again for the great comment.

I'm sorry for not being clear. I asked about running jobs and how can I stop them and their pods because if the user came to a conclusion that the build is not relavent, it's best to terminate the build instead of letting it run until it complete.

how can I find all the jobs that were triggered from the same build, and then, shutting them down while the build is in still running status?

[krancour]

@stavalfi this will kill a running build-- including all its jobs:

brig build delete <build id> --force