Go: syscall: Go Build Fails on Windows With Long Paths

[cretz]

Opening by request in #3358

What version of Go are you using (go version)?

go version go1.8beta2 windows/amd64

What operating system and processor architecture are you using (go env)?

Win64

What did you do?

Tried to compile several files, many with long names, in a folder via go build

What did you expect to see?

Successful compile

What did you see instead?

go build github.com/cretz/myproj: C:\dls\go1.8\go\pkg\tool\windows_amd64\compile.exe: fork/exec C:\dls\go1.8\go\pkg\tool\windows_amd64\compile.exe: The filename or extension is too long.

[bradfitz]

My guess is that problem is that the magic we added to the os package doesn't help the syscall package, since syscall (where the exec happens) doesn't use os.

Maybe we need to move the os package's support routines (building the UNC paths) into an internal support package usable by both os and syscall.

/cc @quentinmit

[bradfitz]

And /cc @alexbrainman

[cretz]

@bradfitz - I think this would be handled by the caller of Exec instead of in syscall directly since they are opaque strings and the difference between a file path and any other arbitrary string is determined by the caller. I do agree that this and the os package should share the same path-fixing code. I also agree it should be an internal function for now until more thought is given to expose (maybe os/exec.OSSafePath but it's really annoying to abstract a Windows -only issue).

[harshavardhana]

Why not have filepath.LongPath() function to be used for these situations?

[kardianos]

@harshavardhana If the go std lib handles windows long filepaths, which I think is great, I don't think it would be good to expose it anywhere except (maybe) in syscall, as this is a really platform specific issue where the fix doesn't need to affect API.

[bradfitz]

Yeah, no new public API.

[harshavardhana]

@harshavardhana If the go std lib handles windows long filepaths, which I think is great, I don't think it would be good to expose it anywhere except (maybe) in syscall, as this is a really platform specific issue where the fix doesn't need to affect API.

:+1:

[quentinmit]

We very consciously did not make syscall transparently handle long paths, because it would be confusing if the resulting syscall was called with different arguments than the caller intended.

Probably the right thing to do is to teach os/exec to use the same long path handling as os.

[bradfitz]

Actually, os/exec just uses os.StartProcess. So this is in the os package after all.

I'll send out a fix.

[gopherbot]

CL https://golang.org/cl/34725 mentions this issue.

[bradfitz]

Not trivial, it turns out. Somebody should tackle it during Go 1.9. https://golang.org/cl/34725 might be a good starting point. It has some debugging notes in it and a test.

[kardianos]

If I was to guess, the UNC meses with the quote logic; guess only.

[ALTree]

Note: #21145, just closed as a dup of this issue, has an handy Powershell script that creates a folder that triggers the problem.

[kardianos]

After reviewing #21145 I think the CL that Brad had started ( https://golang.org/cl/34725 ) is barking up the wrong tree. In that CL it attempts to fixup the process name to use the UNC path equivalent that the rest of the os functions can use. However, as noted in the CL, CreateProcess doesn't accept UNC paths for the process name.

Looking at this issue again the process name is short C:\dls\go1.8\go\pkg\tool\windows_amd64\compile.exe and less then 250 chars. This means that no long path fixup would be applied even if it could work.

I suspect that the error is actually referring to the argument length (Windows doesn't really differentiate between the two in error message).

From https://msdn.microsoft.com/en-us/library/windows/desktop/ms682425(v=vs.85).aspx

lpCommandLine [in, out, optional]
The command line to be executed. The maximum length of this string is 32,768 characters, including the Unicode terminating null character. If lpApplicationName is NULL, the module name portion of lpCommandLine is limited to MAX_PATH characters.

Two ways to fix this would be to instrument the Go compilers to take a new argument "-args-on-stdin=true" flag where the exec would read from stdin for args or pass the argument list file to the executable "@C:\Temparglist" where the given file is read off of disk and used as arguments.

[ianlancetaylor]

It would be nice to solve the general problem, but perhaps Windows does not provide a mechanism for that. If we can only solve the problem for cmd/go, then of course we should do that.

For the record, GCC and friends do this by treating any argument that starts with '@' as a response file. The file is turned into a sequence of white-space separated strings, and the response file argument is replaced by the resulting set of arguments. This seems to be a standard Windows concept, documented at https://msdn.microsoft.com/en-us/library/windows/desktop/aa367156(v=vs.85).aspx .

[ellismg]

I’m running into this issue and I’d like to contribute a fix.

If the maintainers would like me to teach the toolchain about response files, I’m happy to take a swing at it, but I have some questions:

1. It seems like it may be worthwhile to teach the flag package how to understand response files. If that is reasonable, my approach would be when looking for flags, also treat strings that start with @ specially. Is it reasonable to make that change or would it need to be guarded behind some check to ensure existing invocations which pass strings start start with @ in a position where a flag would be legal continue to work? It so, I assume adding a new function like AllowResponseFile(bool) is the way to go?
2. How do you want to do word splitting in the response file? Would we want to support multiple arguments per line in the response file or say that each argument is delimited by a newline. Regardless of what we choose, how does someone escape a character that would cause word breaking?
3. If adding this feature to the flag package is unacceptable should I just teach the compiler a new argument like -sourcefiles ?

[alexbrainman]

@ellismg before deciding how to fix this, can you provide me with steps to reproduce it.

Thank you

Alex

[ellismg]

@alexbrainman

Sure. In the case I'm hitting, the problem is just that we construct a command line that is too long (>32K characters) when we try to exec into the compiler.

This works:

C:\Users\Matt G. Ellis\go\src\github.com\ellismg\scratch>dir /B
empty.go
hello.go

C:\Users\Matt G. Ellis\go\src\github.com\ellismg\scratch>type empty.go
package main
C:\Users\Matt G. Ellis\go\src\github.com\ellismg\scratch>type hello.go
package main

import "fmt"

func main() {
    fmt.Printf("hello, world\n")
}
C:\Users\Matt G. Ellis\go\src\github.com\ellismg\scratch>go build

C:\Users\Matt G. Ellis\go\src\github.com\ellismg\scratch>

If you add a bunch of additional files (depending on how deep your source code is, you may need more copies of the dummy file, to construct a command line that is over the windows limit):

C:\Users\Matt G. Ellis\go\src\github.com\ellismg\scratch>for /l %i IN (1,1,2048) DO copy /y empty.go %i.g
o
[...lots of copy spew elided...]
````

Then `go build` will fail when trying to run compiler.exe

C:\Users\Matt G. Ellis\go\src\github.com\ellismg\scratch>go build
go build github.com/ellismg/scratch: C:\Gopkg\tool\windows_amd64compile.exe: fork/exec C:\Gopkg\tool\w
indows_amd64compile.exe: The filename or extension is too long.

Here's my system information:

C:\Users\Matt G. Ellis\go\src\github.com\ellismg\scratch>go version
go version go1.9 windows/amd64

C:\Users\Matt G. Ellis\go\src\github.com\ellismg\scratch>go env
set GOARCH=amd64
set GOBIN=
set GOEXE=.exe
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOOS=windows
set GOPATH=C:\Users\Matt G. Ellis\go
set GORACE=
set GOROOT=C:\Go
set GOTOOLDIR=C:\Gopkg\tool\windows_amd64
set GCCGO=gccgo
set CC=gcc
set GOGCCFLAGS=-m64 -mthreads -fmessage-length=0
set CXX=g++
set CGO_ENABLED=1
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
```

Locally, I did a hacky thing where I just taught compiler.exe about a -srcfiles argument that would read a response file and then passed every source file via that list. With that set of changes, I was able to build.

Nothing long path specific here, as @kardianos surmised. Just generating a very long command that ends up being over a Windows limit.

[ianlancetaylor]

Something like flag.(*FlagSet).ResponseFiles(bool) makes sense to me, with flag.ResponseFiles calling that method on CommandLine.

The response files should be parsed exactly as the Microsoft tools parse them. I believe they permit multiple options on a single line, and permit quoting with backslash, single quote, and double quote.

[alexbrainman]

the problem is just that we construct a command line that is too long (>32K characters)

@ellismg thank you for providing the instructions. I can confirm we have command line that is longer than 32K characters.

I do not see any other solution but feeding command line in a "response file". I just wander, maybe we should do it automatically. go command can measure length of a command line and use "response file" if it is longer than 32K. Than users don't even need to know about the "response file". Is that what you are proposing to do @ellismg ?

Alex

[ellismg]

Yes, that is my plan. My hope is to start working on the patches early next week.

[kardianos]

@ellismg Just so it is explicit, do you plan to focus changes to accept a "response file" in cmd/{go,compile} or flag or both?

[ellismg]

@kardianos. My plan was to teach flag about response files (as something you could opt into) and then opt cmd/{go,compile} into supporting response files and change the part of cmd/go to use a response file when invoking cmd/compile if the command would be too long otherwise.

I’m happy to take whatever approach you all want, I just care about being able to build projects with lots of .go files in them on windows.

[kardianos]

Will there be a function that takes an argument list ([]string) and converts it to a response file io.Reader? If so, where would that live? Again, not too worried, but I think it might be good to flesh out a few ideas on the issue tracker so if there are objections / design differences we can catch them before all the code is written.

[ellismg]

@kardianos As someone who is still very much a go neophyte I appreciate catching issues early. Let me spend a small amount of time coming up with a comprehensive change proposal. Should I share that here or is there some better place to track issues of this form?

[kardianos]

Track it here for now. Or you could create a new issue with something like "proposal: add response file parsing to flag package" and reference this issue in it. But here is fine.

[joelhandwell]

Got same issue with Golang 1.9.3

> go build -o terraform-provider-aws
go build github.com/terraform-providers/terraform-provider-aws/aws: C:\tools\go\pkg\tool\windows_amd64\compile.exe: fork/exec C:\tools\go\pkg\tool\windows_amd64\compile.exe: The filename or extension is too long.

[davelatimer]

any solution to this ? It is currently un-usable

[bradfitz]

@davelatimer, not yet.

[daniel-007]

Got same issue with Golang 1.10
baliance.com/gooxml/schema/soo/sml
go build baliance.com/gooxml/schema/soo/sml: D:devtools\Gopkg\tool\windows_amd64compile.exe: fork/exec D:devtools\Gopkg\tool\windows_amd64compile.exe: The filename or extension is too long.
my env:
windows10 64bit
go version go1.10 windows/amd64

[lachnerd]

still not working go1.9.5 windows/amd64
go build github.com/hashicorp/terraform/vendor/github.com/terraform-providers/terraform-provider-aws/aws: C:\Go\pkg\tool\windows_amd64\compile.exe: fork/exec C:\Go\pkg\tool\windows_amd64\compile.exe: Der Dateiname oder die Erweiterung ist zu lang.

[sodre]

@lachnerd I saw you are trying to compile the terraform-provider-aws on Windows. I believe, this does not work because because of the extremely long paths and not because of the golang packaging in conda-forge.

If you are wondering how hashicorp builds the windows dependencies, I believe they are are cross-compiling on Linux.

[arizvisa]

Why is this bug still open?

This issue has been in golang since compile started handing off the contents of entire packages via the commandline. Normally it was possible to just avoid using packages that included a ton of .go files but this past year a number of major projects have started including these larger packages, so it is unavoidable.

Where is the development for this fix to compile (and its callers) happening at so we can all help get this done and then the fix merged upstream faster?

[ALTree]

Why is this bug still open?

Because nobody has sent a (working) patch yet.

Where is the development for this fix to compile (and its callers) happening at so we can all help get this done and then the fix merged upstream faster?

Not sure what you mean. There is no "upstream": this is "upstream". Here is where you submit patches to fix this issue. See: https://golang.org/doc/contribute.html

[arizvisa]

@ALTree, question about bug being open was hypothetical and snarky. I'll avoid helping you with your semantics since you likely utilize the same resources as most internet normies and understand that being merged upstream means merged into the primary master branch like most.

Question was: where is the development for this fix to compile (and its callers) happening at so we can all help get this done and then merged? Is there anyone working on this? It's still tagged as "unassigned" and "help wanted". Maybe I fail searching through the mailing list archives, but there's no mention about this there either.

The issue with the 32k limit for UNICODE_STRING has already been identified, but there hasn't been any discussion on the proper fix short of ellismg's suggestion of adding an extra commandline parameter to compile and ensuring that its callers use it.

• Would it be better to add support for parsing '@'-prefixed files and expanding them into argv similar to gnu-related tools and microsoft's response files?
• Should support for this be added to the "flag" package since all commandline parsing on windows is effected?
• Or how about adding an argument that lets you specify paths to packages instead of expanding the paths to all .go files for a package?
• Or is ellismg's method of exposing an argument that reads source files from a whitespace delimited file the most likely solution to get merged?

Let's get this discussed and done, this oversight about the limitations of the windows environment during compilation for go has been around for too long now.

[bradfitz]

@arizvisa, no need for snark. The metadata on this bug is still accurate: nobody is working on this and it is still "help wanted". You're not missing any discussion elsewhere. This bug is the best place to discuss the details of a fix is this issue.

Should support for this be added to the "flag" package since all commandline parsing on windows is effected?

I'm afraid that that would delay the fix, adding documentation and API promises and more design.

I personally think the best way to get this fixed quickly and without much politics is doing something specific to cmd/compile only for now. We can make up whatever calling convention is most convenient (magic env variable to signal to tell cmd/compile get its os.Args from a flag file before flag.Parse?). Then maybe we can standardize later or support things that people might expect ("response files").

[bradfitz]

Okay, I got something working. I modified cmd/go to pass through the os/exec.Cmd.Args []string as JSON to stdin (to known trusted commands that are aware of it) and then added the decoding before flag.Parse() in cmd/compile.

I haven't tested it on Windows yet, but it works on Linux when I override the only-do-this-on-Windows part.

[gopherbot]

Change https://golang.org/cl/110395 mentions this issue: cmd/go, cmd/compile: pass long args via stdin on Windows to avoid length limits

[arizvisa]

@bradfitz. kk. I'll let you know its success in a short moment here.

[arizvisa]

kk. Awesome.

@bradfitz, so it took me a bit as I had to spot the code with prints. but the only issue that your patch had was that where you check for the tool being equal to "compile". We'll need to add the GOEXE suffix to the check (so it checks that filepath.Base is equal to compile.exe). At some point we can expand this extension for all the available tools anyways too.

Other than that, though. It works! Thanks!

(edited: to add clarification of the comparsion to "compile.exe")

[bradfitz]

@arizvisa, thanks for testing and noting the .exe bug. I've uploaded a new version and also using it with "link".

[arizvisa]

@bradfitz, hey man. I noticed that one of the reviewers of your patch suggested the idea of going the path of response files. I made a branch that goes down this route, and adds '@'-prefixed response files to the same place that your patch added your fix.

I essentially re-implemented the buildargv function (from libiberty in gcc) in golang, but at the moment it doesn't do recursive response files which I believe gnu-tools supports. It also doesn't explicitly test the tool that's being dispatched to as your patch does.

At the moment its dev'd against go1.10.1. If the community prefers this solution for some reason, though, I can rebase it against master and create a PR so it can be merged upstream. However, I could use another pair of eyes to sanity check how I'm escaping things when writing to the response file (since it's pretty common to have mistakes here).

The branch I'm referring to is at arizvisa/golang#GH-18468

[arizvisa]

Actually, another update. I didn't realize go1.10.2 came out, so I rebased it for that tag. lmk..

[alexbrainman]

@arizvisa if you want your change looked at, you have to use tools that this this repo uses - https://golang.org/doc/contribute.html on how to contribute. Or just create a PR here on master branch. It was @ianlancetaylor who suggested to use response files on https://golang.org/cl/110395. If you post your change here, maybe it will be prefered to CL 110395.

Alex

[arizvisa]

@alexbrainman, I don't want to step on bradfitz's toes as it was his original fix, I was just offering some direction as it was based on his original work and his response to my complaining. It works for me, but I also don't really want to maintain it or anything ;-) as I'm not really a developer.

Plus, I had realized that I implemented it from a GPL-licensed piece of software which I'm pretty sure is a licensing conflict.

[alexbrainman]

It works for me,

Sounds good to me. I gave Brad +2 now, so he can submit his CL.

I had realized that I implemented it from a GPL-licensed piece of software which I'm pretty sure is a licensing conflict.

It could well be.

Alex

[bradfitz]

I've submitted 17fbb83693d5d4b880bb128d7afdb137840f76ec (https://go-review.googlesource.com/c/go/+/110395). @arizvisa, please double check that it works for you?

I at least made it such that when it runs on our build farm, 10% of the compile/link invocations go through this code path, so it will get some exercise even for non-large build arguments.

[arizvisa]

@bradfitz, sorry to get back to you so late as I've been ill for a bit.

It seems to work without issue. So, thanks!

One thing not too important but worth thinking about in the future, the way you split lines in expandArgs just culls out the <CR>. I know it's dickheadish, but <CR> is actually a valid character in a filename. Regardless of that weirdness tho, bufio.Scanner might be able to be used in order to scan for newlines more effectively. That was the reason why I ripped buildargv out of libiberty anyways.

Regardless though. Thanks for getting this fixed!

[bradfitz]

<CR> is not really a valid character in a Windows filename per https://msdn.microsoft.com/en-us/library/windows/desktop/aa365247(v=vs.85).aspx which says...

Characters whose integer representations are in the range from 1 through 31, except for alternate data streams where these characters are allowed.

I don't think we need to care about "alternate data streams", which I assume are like Mac "resource forks"?

[arizvisa]

Ah, the api call suceeds. (Although I'm using touch to avoid writing a program).

[528] user@skinny ~/build/go1.10.2$ uname -a
MSYS_NT-6.3 skinny 2.10.0(0.325/5/3) 2018-02-09 15:25 x86_64 Msys
[529] user@skinny ~/build/go1.10.2$ touch $'hola\nmundo.'
[530] user@skinny ~/build/go1.10.2$ ls -l $'hola\nmundo.'
-rw-r--r-- 1 user None 0 May  6 16:02 'hola'$'\n''mundo.'
[531] user@skinny ~/build/go1.10.2$ echo hola*
hola
mundo.

Result of the file in the old cmd prompt.

C:\users\user\build\go1.10.2\src>dir /x ..\hola*
 Volume in drive C has no label.
 Volume Serial Number is FA87-0BF0

 Directory of C:\users\user\build\go1.10.2

05/06/2018  04:02 PM                 0 HOLAMU~1     hola?mundo.
               1 File(s)              0 bytes
               0 Dir(s)  13,513,748,480 bytes free

Then a streams which isn't related, plus I couldn't get a newline added as a stream (even using the api)

C:\users\user\build\go1.10.2>echo hi > stream:woot
C:\users\user\build\go1.10.2>dir /r stream
 Volume in drive C has no label.
 Volume Serial Number is FA87-0BF0

 Directory of C:\users\user\build\go1.10.2

05/06/2018  04:12 PM                 0 stream
                                     5 stream:woot:$DATA
               1 File(s)              0 bytes
               0 Dir(s)  13,508,710,400 bytes free

I'm not really a mac guy so my exp w/ resource forks are pretty spotty. but alternate data streams are typically used for things like metadata as they have no particular data structure. NTFS exposes some interesting default ADS which one can use to get some pretty interesting side effects out of software on windows. But as an example for a legitimate thing, browsers will add an ADS to a downloaded file so that a file can be noted as being "unsafe".

But regardless about platform differences/semantics, I'm just happy things are fixed and I just have tendencies to note side effects about things which is what that stripping of <CR> looked like to me. :)

The implementation of buildargv that I ripped attempted to parse out these things by handling quoting and backslash-escapes. However in order to use that implementation effectively, when I had to build the response file I spent most of the time trying to figure out how to properly quote, escape, and separate each argument in the response file so that buildargv would properly decode those characters that were explicitly specified which allows for their compiler to work independently of filesystem contraints. That's how the gnu folk handle it anyways.

Again though, thanks for fixing this!

(edited to fix some type-o's and the missing backticks around CR)

[simchev]

Is this supposed to be fixed in 1.10.2? I just upgraded to the latest version and I'm still getting the error.
Edit: I just saw the 1.11 milestone, how can I have access to this fix before the release?

[bradfitz]

Edit: I just saw the 1.11 milestone, how can I have access to this fix before the release?

You could patch it in and recompile Go if it's urgent.

But I suppose we could also backport it to Go 1.10.x if we do another one.

@gopherbot, please backport.

[gopherbot]

Backport issue(s) opened: #25292 (for 1.10).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://golang.org/wiki/MinorReleases.