Go: x/build/{cmd/coordinator,app/appengine}: combine codebases

[dmitshur]

We've discussed and agreed wanting to merge app/appengine into cmd/coordinator. The motivation is to have a single codebase, share more code, fewer separate deployments, and tests with more inclusive coverage.

I didn't find an existing issue, so making this one to track the task.

/cc @bradfitz @andybons @toothrot

[bradfitz]

I regret that I have but four emojis to give to this bug.

[gopherbot]

Change https://golang.org/cl/199800 mentions this issue: app/appengine: don't exclude release-branch-only builders

[gopherbot]

Change https://golang.org/cl/199878 mentions this issue: app/appengine: hide release-branch-only builders when viewing master branch

[gopherbot]

Change https://golang.org/cl/208319 mentions this issue: app/appengine: remove ancient performance/benchmarking code

[gopherbot]

Change https://golang.org/cl/208320 mentions this issue: app/appengine, cmd/genbuilderkey: move key gen from App Engine to new tool

[gopherbot]

Change https://golang.org/cl/208322 mentions this issue: app/appengine: delete old "broke the build" code

[gopherbot]

Change https://golang.org/cl/208324 mentions this issue: app/appengine: delete more dead code

[gopherbot]

Change https://golang.org/cl/208397 mentions this issue: app/appengine: delete yet more dead code

[gopherbot]

Change https://golang.org/cl/208678 mentions this issue: app/appengine: remove unused tagHandler, document others, remove auth from init

[gopherbot]

Change https://golang.org/cl/208697 mentions this issue: app, gitmirror, maintner: use maintner for dashboard, not datastore

[bradfitz]

This is getting pretty close now.

Here's a suggested roadmap for finishing this:

• add an HTTP handler to the coordinator for removing build results. Currently there are only two things remaining that POST to the dashboard (the build.golang.org App Engine app): the coordinator itself, and x/build/cmd/retrybuilds. If we want to get to the coordinator knowing the world state (including keeping the work list in memory), we can't have retrybuilds reaching behind its back and writing to the datastore directly. So we should for now add a proxy handler to the coordinator that just proxies the /clear-results path to build.golang.org. Then change retrybuilds to use the farmer.golang.org hostname instead of build.golang.org.

• change the coordinator's findWork to not hit the dashboard but instead do the maintnerd GetDashboard API call itself. This'll need to start doing datastore Get calls (to either golang-org's GCP project w/ new service account, or migrate the data, which seems unnecessarily painful)

• modify the coordinator's /clear-results handler to do the datastore deletion itself. (And it can then also immediately start the replacement work, to rebuild whatever was cleared, without waiting for findWork to run)

• add a JSON handler to the coordinator that lists all the active post-submit builds

• drop the dashboard memcache code for buildingURLs (the blue gophers that render on the dashboard for in-progress builds) and instead hit the new JSON endpoint to figure out what's in progress. This then removes all memcache usage in the dashboard, which isn't (as easily) available in the new App Engine Go 1.12+ runtimes.

• update app/appengine from the App Engine datastore package to the Cloud (cloud.google.com/go/datastore) package. This is required to update to the App Engine Go 1.12 runtime. Do that to verify it all works.

• move the app/appengine UI code to a shared spot and make the coordinator also capable of running that handler. Now that it's using cloud.google.com/go/datastore, this should be trivial. Make an HTTP mux that routes to the dashboard UI handler based on Hostname. Say, if `hostname == "build.golang.org" || hostname == "farmer-ui-test.golang.org".

• create DNS record farmer-ui-test.golang.org to test that it works.

• change the build.golang.org DNS record.

• delete the App Engine app

[gopherbot]

Change https://golang.org/cl/210838 mentions this issue: app/appengine, cmd/coordinator: fetch active builds from coordinator, avoid memcache

[gopherbot]

Change https://golang.org/cl/210977 mentions this issue: app/appengine: migrate from App Engine to Cloud APIs, move to Go 1.13

[gopherbot]

Change https://golang.org/cl/210997 mentions this issue: app/appengine: do maintner & coordinator RPCs concurrently

[gopherbot]

Change https://golang.org/cl/210998 mentions this issue: app/appengine: explicitly mark the Log.CompressedLog field as not indexed

[gopherbot]

Change https://golang.org/cl/219120 mentions this issue: cmd/coordinator,cmd/retrybuilds: add wipe API to coordinator

[gopherbot]

Change https://golang.org/cl/221920 mentions this issue: cmd/coordinator: render partial build dashboard

[gopherbot]

Change https://golang.org/cl/222197 mentions this issue: cmd/coordinator/internal/dashboard: filter ErrNoSuchEntity

[toothrot]

I've been making some progress on this. I want to add a note to capture @dmitshur's comments in https://golang.org/cl/221920: we should improve how we handle static files.

I did a minimal amount. I think moving them into a static directory and using go generate as we do on other projects is a good solution, but we should somehow cause a build failure for a misconfiguration.

[gopherbot]

Change https://golang.org/cl/244137 mentions this issue: app/appengine: stop hiding some tested configurations for x repos

[gopherbot]

Change https://golang.org/cl/244398 mentions this issue: cmd/coordinator: restore partial support for -mode=dev

[gopherbot]

Change https://golang.org/cl/245277 mentions this issue: maintner/maintnerd/maintapi/version: support beta and RC release tags