Test-infra: prow/mungegithub makefiles: get-cluster-credentials & non-gcloud

[BenTheElder]

See discussion here: https://github.com/kubernetes/test-infra/pull/4538#pullrequestreview-66093979

Prow and Mungegithub both have a number of rules that depend on get-cluster-credentials, we can probably either:

• Make it easier for this rule to be a no-op when used to deploy elsehwere
• Remove depending on this rule and rely on those working on kubernetes' infra running them manually prior to the currently dependent rules

/cc @stts @cjwagner @spxtr

[cjwagner]

I think the first option is preferable. These targets are a nice safety net to prevent deploying to the wrong clusters.

[BenTheElder]

Maybe we can just build up a command string then and set it at the top level so that you can do override it with something like:
make -C <foo> <target> GET_CLUSTER_CREDENTIALS_CMD="true"
It's not the most ergonomic, but it's probably preferable to some unrelated gcloud calls..

[spxtr]

I would caution against anything that looks like make -C <foo> <target> GET_CLUSTER_CREDENTIALS_CMD="true". Other users of prow have just made their own Makefiles as needed, since they also need to copy and modify cluster/*.yaml. Instead of making the Makefile more generic, we can add a section to the getting started guide that explains how to copy the Makefile and cluster/*.yaml. We can also add a comment over the get-cluster-credentials rule that explains that it's fine to edit it.

In the long run we need to think about deleting or at least reducing the Makefile in favor of some rules_k8s stuff.

[BenTheElder]

That's probably not the exact way to handle it, but having a makefile that only functions for one private instance checked in is kinda gross and we could probably figure out a reasonable way to also allow it to be used more generically. Most of the rest isn't very specific to our instance.

I'm not really a big fan of most of the targets depending on a target that overwrites your kubecfg either, but since I don't deploy mungegithub thus-far I'm OK leaving it as-is :-)

In the long run we need to think about deleting or at least reducing the Makefile in favor of some rules_k8s stuff.

We plan to do some makefile -> Bazel rules conversions this quarter, I'm not sure if mungegithub is worth it though. @mattmoor's demonstration of Prow with rules_k8s at the sig-testing meetup this week was great and I'd love to see all of our deploy / update be that clean :smile:

[spxtr]

but having a makefile that only functions for one private instance checked in is kinda gross and we could probably figure out a reasonable way to also allow it to be used more generically

Our cluster/*.yaml only function for one private instance, though. This is a recurring issue, that the makefile, the cluster config, and the unit tests for the config are all specific to our instance. I somewhat improved this for the cluster config by adding cluster/starter.yaml, but the problem persists for the others. I can open an issue to go into this in more depth, since it's been bothering me for a while now.

[mattmoor]

@BenTheElder @spxtr One of the reasons I broke up starter.yaml in my demo and re-aggregate them with k8s_objects was that it enables you to recycle pieces.

As a strawman, you could have:

cluster/
    starter/
        # all the files and BUILD targets
    mine/
        # overridden files
        # BUILD recycles //prow/cluster/starter:foo for things not overridden
        # but defines new objects for overridden bits (e.g. configmap).
    yours/
       # same idea, different overrides.

... you could also create directories that share common graduation paths, and directories for instances could just assemble pieces "off the shelf" (likely plus config).

The rules also work with rules_jsonnet if you wanted to head the route of templating, but I specifically decoupled to enable choice of templating tool (or none).

[kargakis]

I can't wait for the new declarative API that is built as part of the Declarative Application proposal to get ready for prime time.

[BenTheElder]

I think we can close this one by deprecating the makefile in favor of @mattmoor's work soon. we can provide a generic config and one specific to our deployment and only auth for our cluster. Something along those lines.

[BenTheElder]

Mungegithub is going away, and I plan to split the prow configuration into "reasonable for anyone to get started with" and "as used specifically for our production deployment with any idiosyncrasies we have". I'm hoping @cjwagner and @kargakis can help with this, possibly also @mattmoor xref https://github.com/kubernetes/test-infra/pull/4934

This should hopefully be a reasonable enough compromise.

[spxtr]

I plan to split the prow configuration into "reasonable for anyone to get started with" and "as used specifically for our production deployment with any idiosyncrasies we have".

That's the idea behind starter.yaml :)

[BenTheElder]

That's the idea behind starter.yaml :)

Yeah, That doesn't cover the whole process though. This issue is about the makefiles being specific to our cluster, which we don't seem to really have an equivalent for with starter.yaml. I want to handle the top-to-bottom build-and-deploy for both when we put the work into having rules_k8s.

[BenTheElder]

Mungegithub is headed for the graveyard at this point.
Edit: there are only two repos remaining using mungegithub as far as I know, istio, and kubernetes/kubernetes. Both are slated to use tide soonish.
/close