Go: playground: play.golang.org reports 502 Server Error

[creachadair]

Advance apologies if this is not the correct issue tracker.

As of 28-Feb-2019 07:20 PST, visiting https://play.golang.org returns 502 Server Error for all requests. I have tried connecting from different access points and browsers, and as far as I can tell this is independent of those features. It looks like an AppEngine error page, but I no longer get the internal details.

Repro: Visit https://play.golang.org.
Expected: The playground.
Observed: 502 error response.

[agnivade]

/cc @dmitshur @andybons

[creachadair]

As of 08:23 PST the playground seems to be back online.

[dmitshur]

Thanks for the report. We are investigating it.

I've restarted and promoted a previously deployed version of the playground (also on 1.12, but before CL 163799), which is working for now.

I'll remove the Soon label since the playground is operational now, but keep this issue open to investigate further.

[bradfitz]

My theory is that Go 1.12's mandatory $GOCACHE caching is filling up the disk, which isn't very writable on App Engine.

Note how https://go-review.googlesource.com/c/playground/+/163798 set a $HOME for $GOCACHE.

In Go 1.11, if there was no $GOCACHE and no $HOME, the cache would be disabled. Now we have a cache writing to disk forever.

We should be making a temp dir per run and setting $GOCACHE to that, then nuking the temp dir after.

[dmitshur]

In Go 1.11, if there was no $GOCACHE and no $HOME, the cache would be disabled. Now we have a cache writing to disk forever.

In Go 1.11, there was a GOPATH and GOPATH/pkg, which could have suffered from the same problem, but it didn't. I'd like to figure out what's different about GOCACHE compared to GOPATH/pkg, any thoughts?

One idea is that GOPATH is located in /tmp, which perhaps is a safer place to write to on App Engine Flex? CL 163798 placed GOCACHE inside /home instead. Could that be significant?

Another thought is that the Go Playground doesn't allow new 3rd party packages to be imported, so GOCACHE (the build cache) should be mostly static for most packages, other than package main, the snippet itself being built.

But I'm not very familiar with the expected size of GOCACHE over time, perhaps it is indeed quite sizable just from snippets being built. I want to understand this better.

We should be making a temp dir per run and setting $GOCACHE to that, then nuking the temp dir after.

If so, do we need to do the same for $GOPATH (or just $GOPATH/pkg)?

Nuking the entire GOCACHE sounds expensive; I'd like to confirm it's necessary before committing to doing it.

[bradfitz]

In Go 1.11, there was a GOPATH and GOPATH/pkg, which could have suffered from the same problem, but it didn't

Neither of those apply. The Go standard library is already built & installed. And you can't put things in GOPATH so nothing is ever installed during build.

All artifacts before in sandbox's compileAndRun were already in a tempdir it made:

func compileAndRun(req *request) (*response, error) { 
        // TODO(andybons): Add semaphore to limit number of running programs at once.                                                                                                                       
        tmpDir, err := ioutil.TempDir("", "sandbox")
        if err != nil {
                return nil, fmt.Errorf("error creating temp directory: %v", err)
        }
        defer os.RemoveAll(tmpDir)
....
        exe := filepath.Join(tmpDir, "a.out")
        cmd := exec.Command("go", "build", "-o", exe, in)
        cmd.Env = []string{"GOOS=nacl", "GOARCH=amd64p32", "GOPATH=" + os.Getenv("GOPATH")}

We should just need to put the GOCACHE under that tmpDir too.

[gopherbot]

Change https://golang.org/cl/164519 mentions this issue: playground: put GOCACHE in tempdir

[dmitshur]

Thanks Brad, your analysis in https://github.com/golang/go/issues/30473#issuecomment-468373673 makes sense.

[dmitshur]

I ran a quick experiment to get a sense of GOCACHE disk utilization. I created a simple main.go file locally that just does fmt.Println("test 1") and ran it a few times with GOCACHE=$HOME/Desktop/gocache go run main.go.

Whenever the main.go file doesn't change, the size of the GOCACHE directory increases by 0.5~ KB after each go run.

Whenever the main.go file is modified (e.g., to print "test 2" instead of "test 1"), the size of GOCACHE directory increases by 5~ KB after the go run.

That confirms for me that it's not viable to keep the GOCACHE directory around forever in the playground, since it'll grow in size without bound.

[creachadair]

Thank you, @dmitshur and @bradfitz, for the quick diagnosis and fix.

[dmitshur]

Thank you for reporting this problem to us @creachadair!

[dmitshur]

The new version is deployed now, so this issue should not re-occur.