Service-catalog: Let's Make CI Better

[jeremyrickard]

We ran into this issue: https://github.com/kubernetes-incubator/service-catalog/issues/1962

The root cause of the issue is that changes were made in https://github.com/kubernetes-incubator/service-catalog/commit/2c22fc17a0cbc808f2e626f5d6c91dcd8047aa63 that were incompatible with the v0.14.0 release, however the helm chart was synch'ed to the chart repository. This resulted in people getting the updated _chart_, which changes the flags passed to the API server, but not a new docker image to go along with it.

This would suggest that the helm repo synch is occurring out of step with the image release. It's not at all clear from the service-catalog git repo how the synch even happens.

I think we should do two things:

1.) We should make whatever changes are needed now to make the helm repo sync only occur on a release.
2.) We should revisit the current state of the CI. Can we move to a model where everything that makes up the CI process (outside of secrets of course) is versioned and reviewable in the repo? This might be something like baselining all the travis stuff in the repo and defining any Jenkins jobs using something like Jenkins job builder plugin. Define the builds in YAML and have a job that rebuilds the jobs when they are changed. We used this quite effectively at my last company to keep the definitions of the Jenkins jobs in git so they were easily reviewable by people and there was no hidden build magic.

Ideally, we could get to a point where the service catalog team in general can fix build issues without needing outside intervention.

[MHBauer]

@viglesiasce halp, plz

[arschles]

Some more context. I'm gonna go meta here and do a small rant a well. Skip to the second paragraph if you want to spare yourself. I I got how CI works for the first 1.5 years I've been working on this project. Since then, I lost touch. That's partially because I haven't been that involved with that aspect of the project, but I can't catch up. Ideally I'd like to be able to get caught up by looking at code in this repo. In the past and today, I've tried to do so, but there are a lot of dark corners and dead ends (I assume because not all the code that drives our CI is in here? Except secrets, of course, like @jeremyrickard said). Vic is not intimately involved with this project, and we shouldn't expect him to come swooping in to fix our problems.

Concretely & related to this issue: I worked with @viglesiasce to get the helm repo sync script into the repo and working (many, many thanks for doing that work @viglesiasce), and I can't seem to find that script anymore (iirc it was called repo-sync.sh at the time). Is it still around? If so, where is it called from?

Anyway, I'm with @jeremyrickard. We have the technology y'all

[jeremyrickard]

We could make this a big umbrella effort and also work in _prow_.

[viglesiasce]

Would love to meet on this next week and come up with a plan for making this more transparent. We need to do this for the main charts repo as well.

The crux is that we need to be able to store the GCS key securely then run the sync script using that key.

For releasing new versions of the chart it should be done on tags not commits as it is now and the chart version needs to be updated before releases are tagged.

Apologies for the mess here.

[viglesiasce]

@arschles repo-sync.sh script is here:
https://github.com/kubernetes-incubator/service-catalog/blob/master/test/repo-sync.sh

[arschles]

thanks @viglesiasce - I'm not sure where it's called anymore, but perhaps I'm just out of touch.

Would love to meet on this next week and come up with a plan for making this more transparent. We need to do this for the main charts repo as well.

Thanks for setting that up too! I'll be on vacation, so just imagine me there lurking in the corner on zoom :)

Cheers!

[pmorie]

I'm very supportive of adopting prow.

[MHBauer]

@viglesiasce

For releasing new versions of the chart it should be done on tags not commits as it is now and the chart version needs to be updated before releases are tagged.

Is this a change we need to make in repo-sync.sh?

Q for the talk, As far as I can tell, we do not directly call repo-sync.sh during the build. Who is?_

[n3wscott]

MOVE TO PROW

[carolynvs]

Our Travis job for tagged commits now handles publishing the docker image and charts.