Pipeline: Create a binary for Affinity Assistant used as a placeholder pod

[jlpettersson]

This was requested in https://github.com/tektoncd/pipeline/pull/2630#discussion_r426566799

Expected Behavior

A small, minimal binary, e.g. an Go http service.

Actual Behavior

We currently use an Nginx image for this.

[vdemeester]

@bobcatfish as quickly discussed today, the hardcoded nginx image is most likely gonna be a blocker for openshift-pipelines :sweat_smile: We can use a flag for this and still using nginx but allowing openshift-pipelines (and other integrator) to customize the image.

[ImJasonH]

Does the image have to expose an HTTP service? Or just run indefinitely? Either way, a simple purpose-built Go binary + image sgtm.

Related #2562 #2714

[jlpettersson]

Or just run indefinitely?

Just run indefinitely. We could potentially use it for other things as well, if we want.

For this use case, it is just a pod with specific labels, so that taskRun-pods can configure podAffinity to attract that pod, so they are scheduled to the same Node.

[ImJasonH]

It could (not sure) be nice to reuse the nop image (see #2714) and give it two modes: exit successfully immediately no matter what (nop), or run indefinitely no matter what (affinity assistant).

[dlorenc]

It could (not sure) be nice to reuse the nop image (see #2714) and give it two modes: exit successfully immediately no matter what (nop), or run indefinitely no matter what (affinity assistant).

The run indefinitely option sounds a lot like the k8s pause container: https://www.ianlewis.org/en/almighty-pause-container

we could maybe reuse that one - it is already optimized to be tiny and run forever with 0 CPU footprint.

[dlorenc]

Source for it here: https://github.com/kubernetes/kubernetes/tree/master/build/pause