Go: x/playground: support third-party imports

[bradfitz]

It's time for the playground to support importing third-party packages.

We have https://proxy.golang.org/ now which is the hard piece.

It might have to assume @latest for now, unless we let people write a go.mod file somehow (magic comments either one per module version, or magic comments separating the textarea into N logical files, ala mime/multipart by easier to type?)

/cc @bcmills @dmitshur @ysmolsky

[bradfitz]

e.g. @neild posted this snippet recently: https://play.golang.org/p/1rR97pPojOd ... it'd be nice if that actually worked.

[bradfitz]

Well, that was easy...

bradfitz@go:~/src/golang.org/x/playground$ git di
diff --git a/sandbox.go b/sandbox.go
index 7493213..bf7a421 100644
--- a/sandbox.go
+++ b/sandbox.go
@@ -325,7 +325,7 @@ func compileAndRun(req *request) (*response, error) {
        exe := filepath.Join(tmpDir, "a.out")
        goCache := filepath.Join(tmpDir, "gocache")
        cmd := exec.Command("go", "build", "-o", exe, in)
-       cmd.Env = []string{"GOOS=nacl", "GOARCH=amd64p32", "GOPATH=" + os.Getenv("GOPATH"), "GOCACHE=" + goCache}
+       cmd.Env = []string{"GOOS=nacl", "GOARCH=amd64p32", "GO111MODULE=on", "GOPROXY=https://proxy.golang.org", "GOPATH=" + os.Getenv("GOPATH"), "GOCACHE=" + goCache}
        if out, err := cmd.CombinedOutput(); err != nil {
                if _, ok := err.(*exec.ExitError); ok {
                        // Return compile errors to the user.

There's a minor complication with supporting the legacy code.google.com/p/go-tour/pic/* packages in module mode. We currently pre-install them in the Docker container's $GOPATH, but module mode doesn't use that.

[dmitshur]

Agreed this would be nice, and is easier now than before.

There are no additional security concerns with this feature request. It's always been possible to accomplish this in a round-about way: by re-writing the snippet such that all third-party imports are inlined into the main program (similar to what bundle does). If the snippet does too much and takes too long to run, it'll still time out, etc.

ala mime/multipart by easier to type?

txtar format comes to mind.

One can also imagine a more featureful UI that allows adding extra files. For example, https://gist.github.com/ has an "Add file" button that adds another text box.

Maybe we can just have the playground run gists. _*flashbacks to 2016*_ 😱

There's also some overlap with https://github.com/dave/jsgo#how-it-works and perhaps some other playground-like projects (https://goplay.space/, etc.).

It might have to assume @latest for now

As long as we fix #31889, using @latest would at least return consistent results.

[bcmills]

One easy way to support non-latest versions would be to add comments to import statements:

import (
    "fmt"

    "gopkg.in/errgo.v2/errors"  // v2.1.0+incompatible
)

But that doesn't describe all of the transitive dependencies, and thus isn't particularly reproducible.
Might be better to write and store an explicit go.mod file; we could show it in a second column (perhaps initially hidden in a tray).

[bradfitz]

@bcmills, or how about this: if the user hits "play" or "share" on a package that uses a third-party import, we append the auto-generated go.mod to the end of the textarea buffer (using whatever magic delimiter syntax we come up with), so it's included in shares and reproducible.

e.g. user writes:

package main

import "github.com/davecgh/go-spew/spew"

func main() {
        spew.Dump(map[string]string{"foo": "bar"})
}

Then hits "Run" or "Share" and we first change it to:

package main

import "github.com/davecgh/go-spew/spew"

func main() {
        spew.Dump(map[string]string{"foo": "bar"})
}

# File: go.mod
module play
go 1.13
require github.com/davecgh/go-spew v1.1.1

[ysmolsky]

I love the idea. And it really comes to this: support multiple files in playground. I like the idea of using delimeters per file (txtar). I really wanted to implement support for multiple files, but I hate the idea of adding a box per each file - it does not feel right for the current UI - I tried multiple options there.

I support the idea of having multiple files specified by txtar, also main.go does not have to be prefixed by any file marker.

[gopherbot]

Change https://golang.org/cl/176317 mentions this issue: playground: support third-party imports (off by default for now)

[bradfitz]

The basic functionality (without any of the txtar etc multi-file splitting) is now done in CL 176317. It's off by default for now with an env flag (ALLOW_PLAY_MODULE_DOWNLOADS) set to false in the Dockerfile. That gives us a quick knob to turn it off later.

We also want to do some cleaning probably, so the disk doesn't fill. We'll probably want to hold a RWMutex while cleaning to avoid #31948

[bradfitz]

Actually, CL 176317 now also cleans up after itself (any downloaded modules) after each build.

[bcmills]

I would still rather we use a separate textarea instead of tacking the files together in the same buffer.
I don't really care whether that textarea goes below the Go source file or beside it.

I find it really useful to be able to press ⌃a ⌃c and paste the result directly into cat, and having to trim the file becomes particularly annoying as the files get longer.

[bradfitz]

@bcmills, we can start with a single buffer because it's a ton easier. We can add UI later, once somebody has time for that.

I find it really useful to be able to press ⌃a ⌃c and paste the result directly into cat, and having to trim the file becomes particularly annoying as the files get longer.

a) maybe playground snippets should only be snippets :)
b) I'll write you a splitter CLI tool you can pipe into. :)

[rsc]

Yes, please lets start with a single txtar buffer. We can worry about fancier UI later.

[Sajmani]

A discussion around whether to support multiple files in the playground is in issue #3806.

[AlexRouSg]

Would this be only for pure Go imports or would it also work for self contained cgo using packages?

[bradfitz]

Pure Go. It's still nacl for now. (See #30439 for removing nacl and moving the playground to a different type of sandbox, which might then support cgo)

[gopherbot]

Change https://golang.org/cl/176940 mentions this issue: playground: enable third-party imports

[akyoto]

This is amazing because it lets library authors add playground links to document their API.
People can test it without installing the library on their own machine.
A big thank you for this feature.

[unhashable]

@akyoto even better, examples in godoc could in principle automatically link to the playground.

[rogpeppe]

Thanks so much for doing this! Here's one issue: I just tried a snippet I was sharing yesterday and I get an error from vet, although the actual code works as expected:

The code is at: https://play.golang.org/p/cy1YkpSRtZJ
The vet errors I see seem like the third party deps haven't been pulled in before vet runs:

prog.go:9:2: cannot find package "github.com/heetch/confita" in any of:
    /usr/local/go/src/github.com/heetch/confita (from $GOROOT)
    /go/src/github.com/heetch/confita (from $GOPATH)
prog.go:10:2: cannot find package "github.com/heetch/confita/backend/file" in any of:
    /usr/local/go/src/github.com/heetch/confita/backend/file (from $GOROOT)
    /go/src/github.com/heetch/confita/backend/file (from $GOPATH)
prog.go:11:2: cannot find package "github.com/kr/pretty" in any of:
    /usr/local/go/src/github.com/kr/pretty (from $GOROOT)
    /go/src/github.com/kr/pretty (from $GOPATH)
Go vet exited.

[ysmolsky]

@rogpeppe, I see that @bradfitz already have deployed the new code. There are no vet errors when I run the snippet above.

EDIT: It might be an issue with playground.js not updated in the browser cache. When it happens it tries to call the old endpoint /vet, so that endpoint can fail.

[rogpeppe]

@ysmolsky Yes. When I hard-reloaded, the issue went away. Thanks!

[bradfitz]

Yeah, vet+modules required an update to the protocol and new JS client code.

[adrdev10]

Awesome 😊

[ysmolsky]

@bradfitz I would like to tackle multi files support in playground to enable go.mod generation and processing. Is it okay to copy txtar from go/internal to playground?

[dmitshur]

txtar is not in scope of #31761, so copying it (if that's the design we're implementing) sounds reasonable. If you're going to copy it 1:1 and don't need to make any modifications to it, I suggest copying via bundle so that there's less work of manually keeping it up to date. You can see an example of bundle usage to copy a package in https://github.com/golang/build/commit/2a37d0d3d741910977575a03a17a18d3bb02881d. It should also be placed it in an internal directory so outsiders don't start depending on it.

FWIW, there's also a copy of it at github.com/rogpeppe/go-internal/txtar, but not sure if it's viable to import a third-party package in x/playground.

Edit: That said, the playground already requires a couple of third-party modules, so maybe requiring github.com/rogpeppe/go-internal is acceptable? What do you think @bradfitz?

[bradfitz]

I have a local CL already in progress for multi-file support using github.com/rogpeppe/go-internal/txtar.

[gopherbot]

Change https://golang.org/cl/177043 mentions this issue: playground: support multiple input files in txtar format

[gopherbot]

Change https://golang.org/cl/177421 mentions this issue: playground: format go.mod files; fix paths in formatting errors

[peterhellberg]

Running https://play.golang.org/p/-LQxD_2TQWw results in the output:

prog.go:3:8: cannot find package "github.com/peterhellberg/gfx" in any of:
    /usr/local/go/src/github.com/peterhellberg/gfx (from $GOROOT)
    /go/src/github.com/peterhellberg/gfx (from $GOPATH)
Go vet exited.

@@@@@@****------
@@@@**@@--**----
@@**@@@@----**--
**@@@@@@------**
**------@@@@@@**
--**----@@@@**@@
----**--@@**@@@@
------****@@@@@@

Program exited.

Changing the code somewhat (by for example adding a new empty line) and running it does not output the error from go vet.

(This is probably the same issue that @rogpeppe was seeing)

[bradfitz]

@peterhellberg, deploying a fix now: https://go-review.googlesource.com/c/playground/+/177617

[toby1991]

The import doesn't use the latest package in github.com. Cannot specify the import package version.

[peterhellberg]

@toby1991 The playground caches builds, does it pull the new version of the package you are importing if you change the code in the playground?

[toby1991]

@toby1991 The playground caches builds, does it pull the new version of the package you are importing if you change the code in the playground?

Yeah, I realized that. And I've also change the code, it will rebuild. But the package is still not the latest. So I think the package fetching service also has caches? And the caches of the package fetching service is based on package name which is without the package version?

[thepudds]

@toby1991

Cannot specify the import package version.

Note that you can add a go.mod file on the playground, which then enables you to specify versions (for example using a tag or commit hash). The syntax looks like this:

-- go.mod --
module play.ground

require rsc.io/quote 19e8b97

Here is a complete example:
https://play.golang.org/p/NMWwk866tv5

[toby1991]

@thepudds Thank you very much! It works like a charm!!

[zulhfreelancer]

Hi everyone, can someone help me with this problem? https://play.golang.org/p/BuJWq6L4Mx2

I got the following errors after clicking 'Run':

go: finding github.com/zulhfreelancer/ethname 211ae2f
go: finding github.com/deckarep/golang-set v1.7.1
go: finding golang.org/x/net v0.0.0-20190613194153-d28f0bde5980
go: finding golang.org/x/crypto v0.0.0-20190611184440-5c40567a22f8
go: finding github.com/go-stack/stack v1.8.0
go: finding gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127
go: finding github.com/aristanetworks/goarista v0.0.0-20190607111240-52c2a7864a08
go: finding github.com/rs/cors v1.6.0
go: finding github.com/ethereum/go-ethereum v1.8.27
go: finding github.com/davecgh/go-spew v1.1.1
go: finding github.com/kr/pretty v0.1.0
go: finding gopkg.in/natefinch/npipe.v2 v2.0.0-20160621034901-c1b8fa8bdcce
go: finding github.com/kr/text v0.1.0
go: finding golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a
go: finding golang.org/x/text v0.3.0
go: finding golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2
go: finding github.com/kr/pty v1.1.1
go: finding golang.org/x/sys v0.0.0-20190412213103-97732733099d
go: finding golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3
go: downloading github.com/zulhfreelancer/ethname v0.0.0-20190614154357-211ae2fe0004
go: extracting github.com/zulhfreelancer/ethname v0.0.0-20190614154357-211ae2fe0004
go: downloading github.com/ethereum/go-ethereum v1.8.27
go: extracting github.com/ethereum/go-ethereum v1.8.27
go: downloading github.com/deckarep/golang-set v1.7.1
go: downloading golang.org/x/net v0.0.0-20190613194153-d28f0bde5980
go: downloading github.com/rs/cors v1.6.0
go: downloading github.com/go-stack/stack v1.8.0
go: downloading github.com/aristanetworks/goarista v0.0.0-20190607111240-52c2a7864a08
go: extracting github.com/go-stack/stack v1.8.0
go: extracting github.com/rs/cors v1.6.0
go: extracting golang.org/x/net v0.0.0-20190613194153-d28f0bde5980
go: extracting github.com/aristanetworks/goarista v0.0.0-20190607111240-52c2a7864a08
go: extracting github.com/deckarep/golang-set v1.7.1
# github.com/ethereum/go-ethereum/log
../gopath496381130/pkg/mod/github.com/ethereum/[email protected]/log/syslog.go:12:29: undefined: syslog.Priority
../gopath496381130/pkg/mod/github.com/ethereum/[email protected]/log/syslog.go:13:13: undefined: syslog.New
../gopath496381130/pkg/mod/github.com/ethereum/[email protected]/log/syslog.go:19:50: undefined: syslog.Priority
../gopath496381130/pkg/mod/github.com/ethereum/[email protected]/log/syslog.go:20:13: undefined: syslog.Dial
../gopath496381130/pkg/mod/github.com/ethereum/[email protected]/log/syslog.go:24:39: undefined: syslog.Writer
../gopath496381130/pkg/mod/github.com/ethereum/[email protected]/log/syslog.go:51:40: undefined: syslog.Priority
../gopath496381130/pkg/mod/github.com/ethereum/[email protected]/log/syslog.go:55:61: undefined: syslog.Priority

I added the block below because I want to force Go Playground to use my latest package version as previous package got some problems:

-- go.mod --
module play.ground

require github.com/zulhfreelancer/ethname 211ae2f

[bradfitz]

That package doesn't support GOOS=nacl GOARCH=amd64p32 (which the playground uses for now):

You can reproduce locally with:

dev:~ $ GO111MODULE=on GOOS=nacl GOARCH=amd64p32 go install github.com/ethereum/go-ethereum/log
go: finding github.com/ethereum/go-ethereum v1.8.27
go: downloading github.com/ethereum/go-ethereum v1.8.27
go: extracting github.com/ethereum/go-ethereum v1.8.27
go: finding github.com/go-stack/stack v1.8.0
go: downloading github.com/go-stack/stack v1.8.0
go: extracting github.com/go-stack/stack v1.8.0
# github.com/ethereum/go-ethereum/log
pkg/mod/github.com/ethereum/[email protected]/log/syslog.go:12:29: undefined: syslog.Priority
pkg/mod/github.com/ethereum/[email protected]/log/syslog.go:13:13: undefined: syslog.New
pkg/mod/github.com/ethereum/[email protected]/log/syslog.go:19:50: undefined: syslog.Priority
pkg/mod/github.com/ethereum/[email protected]/log/syslog.go:20:13: undefined: syslog.Dial
pkg/mod/github.com/ethereum/[email protected]/log/syslog.go:24:39: undefined: syslog.Writer
pkg/mod/github.com/ethereum/[email protected]/log/syslog.go:51:40: undefined: syslog.Priority
pkg/mod/github.com/ethereum/[email protected]/log/syslog.go:55:61: undefined: syslog.Priority

[lootek]

Hi, looks like I've got an issue with playground - it does not use the latest available version of 3rd party import.

In my particular case I try to use github.com/blang/semver which is apparently imported in playground in older version 3.5.1, while my dep-managed local project uses 3.6.1.

It's clearly visible as I got a particular case that still works in 3.5.1, but errors on 3.6.1 - see https://play.golang.org/p/wXQhQlvs30V

semver lib entry on https://index.golang.org/index is

{"Path":"github.com/blang/semver","Version":"v3.5.1+incompatible","Timestamp":"2019-04-15T13:54:39.107258Z"}

while I use the newest revision (still using dep in that project):

[[projects]]
  digest = "1:b6d886569181ec96ca83d529f4d6ba0cbf92ace7bb6f633f90c5f34d9bba7aab"
  name = "github.com/blang/semver"
  packages = ["."]
  pruneopts = "UT"
  revision = "ba2c2ddd89069b46a7011d4106f6868f17ee1705"

(...)

[[constraint]]
  name = "github.com/blang/semver"
  version = "3.6.1"

[peterhellberg]

@lootek It seems to be an issue with installing the latest tagged version:

Installing version v3.5.1

GO111MODULE=on go get -u 'github.com/blang/[email protected]'

go: finding github.com/blang/semver v3.5.1
go: finding github.com/blang v3.5.1
go: finding github.com v3.5.1
go: downloading github.com/blang/semver v3.5.1+incompatible
go: extracting github.com/blang/semver v3.5.1+incompatible



md5-424a60a1fa7f3db3c3a3a153d8e567e7



GO111MODULE=on go get -u 'github.com/blang/[email protected]'



md5-f163ad96c5242200cfd87b6d0e3dd2e4



go: finding github.com v3.6.1
go: finding github.com/blang v3.6.1
go: finding github.com/blang/semver v3.6.1
go: finding github.com/blang/semver v3.6.1
go get github.com/blang/[email protected]: github.com/blang/[email protected]: invalid version: module contains a go.mod file, so major version must be compatible: should be v0 or v1, not v3

Conclusion

v.3.5.1 is the last tagged version that doesn’t include a go.mod file, that is why it is picked. (Since the module needs to have a /v3 suffix)

[lootek]

oh, ok. I'll open an issue there then, thanks a lot!

(btw my case is a known bug on blang/semver: https://github.com/blang/semver/issues/55 )

[flyingmutant]

edit: I've changed the import path of the package to pgregory.net/rapid and released v0.3.0, the new playground links are now https://play.golang.org/p/ZdNjIGzT_EX and https://play.golang.org/p/lJmJvKRE_7H. The original issue can still be reproduced using the links below.

---

It looks like there is a bug: third-party imports sometimes work, and sometimes do not.

For example, https://play.golang.org/p/8hE61BnO8oB usually, but not always, fails with:

go: finding github.com/flyingmutant/rapid v0.2.2
go: downloading github.com/flyingmutant/rapid v0.2.2

Go build failed.

But https://play.golang.org/p/YrPAekuiiP4, which uses the same third-party package, appears to work:

=== RUN   TestCheck_parseDate
--- FAIL: TestCheck_parseDate (0.00s)
    prog.go:40: [rapid] failed after 4 tests: got back wrong date: (0, 0, 1)
        To reproduce, specify -run="TestCheck_parseDate" -rapid.seed=1257894000000000011
        Failed test output:
    prog.go:41: [rapid] draw y: 0
    prog.go:42: [rapid] draw m: 10
    prog.go:43: [rapid] draw d: 1
    prog.go:53: got back wrong date: (0, 0, 1)
FAIL

1 test failed.

[dmitshur]

The Playground largely supports third-party imports by now.

@bradfitz Should we close this issue now, or are there specific things that remain to be done here?

/cc @toothrot

[bradfitz]

Yeah, I think this is all done now!