Go: proposal: Go 2: require specifying source file/package of variables/types/functions used

[yjiangnan]

Code readability matters. However, when reading Go code of others, names undeclared in the current file often come up, making it very difficult to easily understand code of others. Those names, often declared in another file in the package, would be much more easily understood by others if the source of declaration has been specified. A good example would be the python language:

from .somefile import AClass # notice the '.' that specifies the current directory. 
import .anotherfile as f
f.somefunction()

Then I can easily find out that AClass and somefunction are defined in other files and the names of the files. I can also easily see the structure/dependencies of the project by just reading the few starting lines.

I know Go 2 should maintain compatibility with Go 1. However, there are at least two options to help out the situation:

1. Add support for specifying the source as done in python so that programmers that value code readability would adopt this feature.
2. Produce compiling warnings for names defined in another file.

[DeedleFake]

I've gone back and forth on this type of thing. I've had multiple people who give me horrified looks when I tell them how scoping works for global identifiers in the same package, especially people coming from languages like Python or Node.js which handle modules on a per-file basis, but I've never actually had a problem with it in practice. In most cases, the 'correct' solution to this is to split whatever package your having trouble finding things in into multiple packages.

If someone is writing code that causes problems with this, forcing them to explicitly import each identifier rarely actually fixes it. It just results in an unnavigable maze of imports and, more importantly, makes it very difficult to refactor that code and clean it up in a lot of cases. Plus, I've found that the problem can often be circumvented via some basic usages of grep and/or a few other standard GNU tools. Maybe go doc can list the file that a global identifier was declared in for the current package when you look it up.

[yjiangnan]

@DeedleFake
I am not sure if the "'correct' solution" you suggest is correct at all. A lot of time, a programmer is reading codes that he cannot change and there could tens of files in the package.

I do not believe the problems you mentioned are problems, either. Programming language like Python does it all the time and I have never heard people complaining about it. Rather, it has been quite a nice experience for me.

When you suggest grep or go doc, I imagine you use go with command lines all the time. I happen to read some code from GitHub and your suggestion would simply add a lot of overhead.

From what you say, I would say that you are one of the type of person that do not really understand and appreciate readability. For a dedicated hacker, for sure, everything is possible to do. But that does not mean that everyone should do it the same way the hacker does. When you suggest users to use command line tools when reading code, that would really be a bad design of programming language in your mind.

[ianlancetaylor]

Currently there is one way to group Go code: in a package. This proposal would in effect introduce another grouping layer: a file within a package. I can't see any reason to introduce that layer. If you don't want multiple files in a single package, then use a single file. If you want to explicitly import from a different file in the same package, then put the two files in different packages.

[ianlancetaylor]

@yjiangnan Incidentally, please avoid comments like "I would say that you are one of the type of person that do not really understand and appreciate readability." Please stick to the issues and avoid ad hominem criticism. See https://golang.org/conduct. Thanks.

[networkimprov]

I concur that when reading code on github, it can be hard to locate the file where a package feature is defined. Github search sometimes helps, but I run into this often.

A simple solution is a go/doc-generated "package internals" text file, when there are more than e.g. 2 files in a package. It would have sections for types, variables, constants, and functions, then an alphabetical list in each section of _name_, _filename_\ _comment_. That is far more viable than a language change, but we need a specific, detailed description to move the idea forward.

PS: I use a rigorous naming method that (among many things) requires that package-level identifiers append the filename where they are defined. But most software engineers aren't that rigorous ;-)

[yjiangnan]

@DeedleFake @ianlancetaylor
I admit that my words might be too harsh. But please understand that that was really out of frustration when learning Go. Everything in Go looks wonderful except for the package management and namespaces. When DeedleFake said how he could do it, it really came from someone who is very familiar with the language and the problem at hand, in the position of a God-like all-knowing creature. However, most programmers are humble human beings with very limited brain capacity, especially for people new to the language. Using complex procedure for reading really poses a strong distraction and limits the energy to really understand the program logic.

Additionally, when you suggest "then use a single file", you are again arguing just from the perspective of a program writer. But for a lot of time, a programmer needs to read, understand, and use other people's code. Difficulties for understanding each other would simply mean that it would be more difficult for the Go community to build an integrated ecosystem, which is bad for the community as a whole in the end. That is why enforcing readability should really be a language feature and built into the design. Go enforces using defined variables and otherwise throws an error anyway.

I really hope that you could understand how frustrating it is for a Go learner when he sees a bunch of undefined names in a long piece of code he is struggling to understand. Facilitating the learning process is very important for growing the community for more code reviews, tests, and new packages, benefiting everyone using Go.

[ianlancetaylor]

@yjiangnan I appreciate the consideration about reading existing code, but I think that we disagree about readability. And, obviously, existing code already exists. We can not change the language so that almost all existing packages are broken. That will not happen no matter how desirable the language change may seem.

I agree that people looking at Go code need to understand that a single package can be composed of multiple files. It's worth noting that in Go, unlike some other programming languages, you know for sure that if an identifier is not defined in the current file, and is not one of the small number of predeclared identifiers that we expect all Go programmers to memorize, then it must be defined in some other file in the same package. That is the only possibility.

Adding an explicit intra-package import statement will tell you exactly which file defines the identifier. I guess that's an advantage, though to me it seems a small one. How does it help in practice? When I see an identifier not locally defined, I know it's in a different file. I can ask my editor to jump to the definition, or I can jump to the top of the file and look for the import statement there. It's not obvious to me that one approach is clearly better than the other.

Finally, you didn't address my comment about adding another grouping layer.

[yjiangnan]

@ianlancetaylor I am not sure how my two suggestions could break existing packages. It seems to me that you can always maintain backward-compatibility when adding new features.

If you always read Go code in an IDE, that may be fine. In that case, the Go tutorial should first teach users to install and use one Go IDE. In practice, such as when you read on GitHub, what you suggest is not possible. Additionally, when you use the "jump to definition" feature in an IDE, you still have to first focus and click and find the jump option and finally jump there. That involves much more physical work and mental focus than just looking at it. Finally, when you read someone else's code, you would not first read line by line. Instead, you want to first scan through it, understand its structure and get the large picture. Explicit import does exactly this for you. It also creates more namespaces and avoids name conflicts.

I do not believe you would have to add another grouping layer, at least not a very complicated one. If you can add other packages with several '/'s, how could it be so difficult to allow an additional step to get down to the file name?

[networkimprov]

Is github hiding my prev comment? The context is remote repositories where you cannot grep a directory or use an IDE. That's a tooling issue; suggest tooling solutions instead of debating language changes.

[ianlancetaylor]

I guess I don't understand this proposal. What precisely are you proposing?

[networkimprov]

@ianlancetaylor note that I am not the proposal author...

The problem I see is reading/navigating packages with many files on remote repos like github. It's often a challenge to discover which file defines a term. That's not a problem in single-file-module languages. And not a problem on local Go repos where you have standard tooling (IDE, grep).

So I suggest letting go/doc output a "package internals index" in ppl-friendly text. The repo owner would specify an index filename to be included/updated in the repo, perhaps via a go build command-line flag.

The go/doc output needs sections: _Types, Constants, Variables, Functions_
Each section needs an alphabetical list: _Term, Source file\ [Comments]_

There are other solutions via go/doc along similar lines.

EDIT: since it appears that the proposal author won't run with this idea, do you think it's worth filing a separate proposal?

[deanveloper]

I am not sure how my two suggestions could break existing packages. It seems to me that you can always maintain backward-compatibility when adding new features.

Okay so I have two files:

main.go

package main
func main() {
    helper()
}

helper.go

package main
func helper() int {
    return 5
}

That would not work under your paradigm, as helper is defined in a separate file from main.go. This change would break everything which calls functions or use package-scoped variables outside of their file.

The Go 1 compatibility guarantee states that any Go code written in Go 1 will always compile in any future version of Go 1. So my program written with Go 1.5 will still compile under Go 1.11

While Go 2 does not have a backward compatibility guarantee to Go 1, point of breaking the compatibility guarantee isn't to remake the language and break all Go 1 programs, but rather to give an opportunity to add small features which may break a few Go 1 programs (for instance, adding check and handle keywords break any programs which use those as identifiers).

[deanveloper]

Also, what about build scopes?

For instance:

main.go

package main
func main() {
    helper()
}

helper_linux.go

package main
func helper() string {
    return "helper on linux"
}

helper_windows.go

package main
func helper() string {
    return "helper on windows"
}

How would I know where to import the helper function from in an import statement?

[yjiangnan]

@deanveloper
All you need to do is to add helper.helper() (or helper/helper() or something similar) as alias to helper() in the compiler to maintain backward-compatibility. The compiler only needs to treat helper.helper() and helper() as the same. Go 2 obviously would need to come with a new compiler.

In your helper_linux.go and helper_windows.go example, it wouldn't work in the current Go implementation anyway. But it could work if you use helper_linux.helper() and helper_windows.helper() in my proposal.

[yjiangnan]

@ianlancetaylor
I am proposing adding support for aliases and finer-grain namespaces in the compiler. Specifically, you can do something like this:

import(
"ALongButMoreReadablePackageName:pkg"  // pkg would be an alias to ALongButMoreReadablePackageName
".localfile:lf"  // lf would be an alias to .localfile
)

function() // These three functions should be identical.
lf.function()
.localfile.function() // Possible to allow omitting the import if using this form

A funny thing in Go is, on one hand, it encourages limiting scopes of variables at the expense of readability by introducing something like:

if x := AlongStatementOrFunction(); x > 0{ 
// It takes efforts to get familiar with the semantics that it means 'if x > 0'
...
}

One the other hand, it extensively uses global variables affecting multiple files. That design philosophy really confuses me.

[deanveloper]

In your helper_linux.go and helper_windows.go example, it wouldn't work in the current Go implementation anyway. But it could work if you use helper_linux.helper() and helper_windows.helper() in my proposal.

Yes it would. Files ending in _linux.go only are compiled on Linux, and files ending in _windows.go only are compiled on Windows.

On Linux and windows it would compile. Compiling on Linux would include helper_linux.go and skip helper_windows.go, compiling on Windows would do the opposite.

Compiling on Mac (Darwin) would result in a compile error, as it would skip over both helper_linux.go and helper_windows.go, meaning that no helper() function is defined.

Also, this means you can't reference it as helper_windows.helper() because you shouldn't need to change that to helper_linux.helper() in order to compile on Linux. Remember that Go libraries are compiled from source, so if a Linux user wanted to compile your code (compiled on Windows), they would need to change all of your helper_windows to helper_linux

[deanveloper]

For more information about build constraints, you can read the following: https://golang.org/pkg/go/build/#hdr-Build_Constraints

[yjiangnan]

@deanveloper
I did not know that Go uses file names to specify OS-specific compiling options instead of using code such as using Macros in C++. That is a really surprising language feature.

However, in that case, you can keep it the old way for the code to maintain compatibility. My proposal does not disable this old feature. Rather, I only propose to add alias for additional support. You can even allow referring helper_linux.helper() on Mac if the programmer, let's say, does not care about other OSs and he is only grabbing someone else's code to use. It is only depends on what kind of logic you implement in the compiler. But you definitely do not have to break the old code.

[deanveloper]

However, in that case, you can keep it the old way for the code to maintain compatibility. My proposal does not disable this old feature. Rather, I only propose to add alias for additional support.

Making this proposal "optional" would be bad. If we're talking about simply referencing a variable, there should only be one way to do it. It hurts readability significantly for helper.helper() and helper() to mean the same thing.

You can even allow referring helper_linux.helper() on Mac if the programmer, let's say, does not care about other OSs and he is only grabbing someone else's code to use.

Saying helper_linux.helper() is a bad idea without a _linux in the caller's filename. The file won't compile unless targetting linux, so it should have a _linux suffix in the file's name.

---

The reason why scripting languages require you to import a variable from a file is because scripting languages revolve around files (scripts), so of course languages like Python, JavaScript, etc will require you to import variables from individual files. Languages like C, Java, and other compiled languages do not behave this way, and neither should Go. Go doesn't revolve around scripts, and scripts importing from other scripts. It revolves around packages, and I believe it should stay that way

[ianlancetaylor]

Thanks for the suggestion but we aren't going to adopt this proposal.

If you want the behavior seen in other languages, you can put your entire package into a single file. That will work fine. We aren't going to require people who choose to split up packages into multiple files to then explicitly import identifiers from those other files. We aren't going to provide an optional mechanism for that, either.

Thanks.