Go: proposal: cmd/go: add project config to go.mod

[distinctdan]

The way things are today, go relies on environment variables for things like build architecture and os, as well as resolving private repos. This isn't good because they live outside of source control, so the build breaks when someone else clones the repo and tries to run go get or go build.

Instead, what about making these settings live in go.mod, similar to how node has a package.json? It might look like

env (
    GOOS linux
    GOPRIVATE github.com/myOrganization
)

My goal is to ensure it builds the same way every time for everyone that clones the repo, without them having to do any manual environment config.

[bcmills]

See previously #39005 (CC @dominikh @mvdan @jayconrod @matloob)

[bcmills]

See also #33985.

[mvdan]

Note that #39005 was not about altering the default build behavior, though. It was simply about listing the "build configurations" which are well supported by a module. This distinction makes me worry about this proposal.

[distinctdan]

Hmm, I hadn't considered different build configs. That is definitely a useful feature though. For security, I'm not experienced enough with Go to know all the options, but I'm thinking of just setting build options, not necessarily going as far as allowing shell scripts. Perhaps something like, where you have different available configs for the build:

buildConfig (
    default (
        Output myApp
        GOOS linux
        GOPRIVATE github.com/myOrganization
    )
    windows (
        Output myApp.exe
        GOOS windows
        // Inherits GOPRIVATE value from default
    )
)

[mvdan]

I'm still not convinced that a "default" makes sense. It would be very confusing for go build on Linux to produce a Windows binary, or to produce a binary in an unexpected directory, for example.

I think we should design the semantics (what do we want to list, and for what purpose?) before we worry about what file it would go under or with what syntax. The semantics is why #39005 was retracted, after all - it opened up the go tool to executing arbitrary code, and overall the idea gained no traction.

[distinctdan]

So I'm thinking that the build config section would be optional. Any values omitted would use the current behavior, so if an organization was running locally and had a mix of windows and linux machines, they could still omit the GOOS variable so that it would use the OS's default. My particular use case is we've got a small app that's being deployed to aws lambda, so it always needs to build for linux, and my org uses several private repos, so when I first tried to build the project, I got a bunch of errors and had to do a bunch of googling to figure out how to get it to work.

Coming from a web background, the npm and webpack ecosystems offer a lot more tooling. It's normal to have a default task that builds and runs in dev mode, often with hot code reloading, then you've got a task that builds in release mode. It's expected that things just work out of the box and that each module you import is self contained. Coming from that, it just seems strange to have to make global environment changes to get a local project to build, or to have to pass a bunch of flags every time. I've worked around this in the meantime by putting all my flags in a shell script that calls go build, but this seems like a pretty common use case, so I think it'd be nice if Go natively supported it.

[bcmills]

As an alternative, note that you could at least reduce the number of environment variables from many to just one, by setting the GOENV environment variable to point to a location within your repo.

[pickfire]

As an alternative, note that you could at least reduce the number of environment variables from many to just one, by setting the GOENV environment variable to point to a location within your repo.

Setting environment variable is not user friendly, it cannot be checked into the repo (it can through a file) and requires an extra step for new developers (new clone) to get started.

For example, one requires to source env.sh first before go build. Comparing to rust, one can just cargo build with the options in Cargo.toml.

[networkimprov]

I build app releases for Linux, Windows, and MacOS. Each release includes a platform specific tree of files beside the executable, and is packed into a TGZ or ZIP file. I built a shell script to do this, tho I could have used a makefile. I've essentially duplicated the script in multiple projects.

If there's a Go-native way to do this, I overlooked it.

[rsc]

In general go.mod is very narrowly scoped to information about dependency versions. It's not a catch-all for all possible configuration, in contrast to similar files in other systems, like package.json.

I would suggest that you check in a script with environment settings to be sourced, if you want your developers to be on the same page.

[rsc]

Based on the discussion above, this seems like a likely decline.

[distinctdan]

Even if this specific proposal is declined, the root problem still exists of building a project from source without having to do any manual configuration. I'm not locked in to my specific proposal, but I do think this needs to be addressed for go to be on par with other language ecosystems. It may be as simple as choosing a tool such as make, and updating the official docs to say something like "we recommend building with this tool in order to get consistent builds".

[ianlancetaylor]

@rsc already suggested checking in a script that sets environment variables. Is that sufficient, or would something else be required? Would it suffice to declare a convention for what the script should be called?

We seem to only be talking about environment variables. We currently already have a three level hierarchy for them: environment variables have default values set when Go is installed, which are overridden by values set by go env -w, which are overridden by values set in the environment when invoking the go tool. It sounds like you are suggesting that we need to another level in that hierarchy, in between go env -w and the user environment: module level environment variables. Does that sound right?

[distinctdan]

My main point would be that we need a single source of truth for build config that lives in source control. From someone who's relatively new to Go, it feels like kind of a mess right now, where your build might be pulling values from any of the sources you mentioned, none of which are in source control.

If the Go team doesn't want to go the route of package.json and do a full build file, I think it might be enough to update the docs to recommend creating a make file, or perhaps a shell script, to controls the build. Or, at least having a warning that Go relies on things outside your project's repo, and if you don't like that, then you need to implement a custom solution.

[ianlancetaylor]

Part of the reason we don't worry about this too much is that setting these environment variables is rarely necessary. Your example suggests setting GOOS=linux, but that will never be correct on a per-module basis. And setting GOPRIVATE is also not per-module, but rather a characteristic of the organization. So while it makes sense to put that kind of thing into source code control, the module does not seem like the right place for it.

[distinctdan]

Well, at least for my use case, it is always necessary to build with GOOS=linux, and I did have to manually set up GOPRIVATE when I first cloned the repo. Any top level project is probably going to have well-defined build targets. Like, if your company is deploying to aws, you always want to build the same way, regardless of whether the developer is on windows or linux.

You do have a point though about the os architecture not being applicable at the module level. Really, it's at the build target level, like you wouldn't expect imported modules to affect your build params in any way. However, for GOPRIVATE, it seems like that is still applicable to a module since that affects the module's imports.

[antichris]

Create a module-local go.env file, list your GOOS, GOPRIVATE and other (if any) Go environment adjustments there, build with GOENV=go.env go build. If you need scripting, custom targets (e.g. with tags), etc., use make. I don't think this is something that should be ported into Go.

I can see that someone coming from web background is used to tooling lock-in. Anyone with systems background is perfectly comfortable with creating a Makefile when additional configuration/tasks are needed in the build process. Using make (or one of the many alternatives) actually makes a project tooling-agnostic (apart from the selected build tool and its config), not even the language is set in stone: one can mix and match tools, libraries, components and languages in whatever way makes the most sense for their project's goals. I've actually seen people using make in their npm projects, which IMO suggests that they too have come to the above or quite similar conclusions.

[rsc]

Even if this specific proposal is declined, the root problem still exists of building a project from source without having to do any manual configuration.

Just to reply to this, that root problem is not something the go command aims to solve. Shell scripts and more complex build systems are possible solutions.

[rsc]

No change in consensus, so declined.