Go: os: RemoveAll does not always work on very large directories

[mattfarina]

os.RemoveAll and other os operations (e.g., os.Rename) fail with large sets of files on Windows.

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

1.8.3

What operating system and processor architecture are you using (go env)?

Windows amd64 and Windows Subsystem for Linux amd64

What did you do?

An example repo can be found at https://github.com/mattfarina/go-test-windows-files. It's a repo with 10,000 files to be deleted to use as an example.

What did you expect to see?

I expect os.RemoveAll to delete all the files

What did you see instead?

Some of the files are deleted and then an error is thrown.

[mattfarina]

Note, using the system commands on windows and bash (WSL) work without issue. It's just in the os package.

[alexbrainman]

I tried running https://github.com/mattfarina/go-test-windows-files on my Windows XP and Windows 7. It works as expected - "go run main.go" prints "nil", and "files" directory is deleted. I tried running it couple of times, and it always succeeds. I tried both go1.8.3 and current tip.

Alex

[bradfitz]

@mattfarina, the question for Windows quirks is always: which virus scanner(s) are you using?

[mattfarina]

@bradfitz great question. I used Windows 10 Pro Creators Edition to reproduce this. Some Glide users ran into a problem and this was the underlying cause.

Windows Defender that came with the system by default is all I have for antivirus. The system has a minimal install. VSCode, git, go, and WSL are all that I installed to reproduce the issue.

This may be a new issue to Windows 10 Creators Edition. I initially reproduced the Glide issue on the k8s/helm project. I'd previously worked on helm under Windows Anniversary using Glide without issue.

Happy to supply other details that may help.

[alexbrainman]

Some of the files are deleted and then an error is thrown.

What is the error message?

Alex

[mattfarina]

@alexbrainman See the example at https://github.com/mattfarina/go-test-windows-files. Readme has that detail for this case.

[alexbrainman]

See the example at https://github.com/mattfarina/go-test-windows-files.

Thank you. But the "remove files: directory not empty" message looks strange to me.

Searching for "directory not empty" in Go repo, I find ENOTEMPTY. But I don't see how windows code in Go repo can return ENOTEMPTY. Can you try and see why your program returns "directory not empty" message? I am trying to understand which windows syscall fails and what error it returns.

Thank you.

Alex

[kumarharsh]

@alexbrainman just to clarify the error message, it's not remove files: ..., but this:

remove C:\Workspace...\path-to-project\folder1: The directory is not empty.
remove C:\Workspace...\path-to-project\folder2: The directory is not empty.
...
etc

[alexbrainman]

remove C:\Workspace...\path-to-project\folder1: The directory is not empty.

The "The directory is not empty" message must be ERROR_DIR_NOT_EMPTY. The only way you can get this message (as far as I know), if you are trying to delete directory with some files inside. Is it possible that some other program is creating files as you are running os.RemoveAll ?

Something like:

package main

import (
    "fmt"
    "io/ioutil"
    "os"
    "os/exec"
    "path/filepath"
    "time"
)

func runParent() error {
    tmpdir, err := ioutil.TempDir("", "ALEX")
    if err != nil {
        return err
    }

    exe, err := os.Executable()
    if err != nil {
        return err
    }

    cmd := exec.Command(exe, tmpdir)
    cmd.Stdin = os.Stdin
    cmd.Stdout = os.Stdout
    cmd.Stderr = os.Stderr
    err = cmd.Start()
    if err != nil {
        return err
    }

    // wait before clent starts creating new files
    time.Sleep(100 * time.Millisecond)

    err = os.RemoveAll(tmpdir)
    if err != nil {
        return err
    }

    return nil
}

func runChild(dir string) error {
    for i := 0; i < 10000; i++ {
        path := filepath.Join(dir, fmt.Sprintf("file%d.txt", i))
        f, err := os.Create(path)
        if err != nil {
            return err
        }
        f.Close()
    }
    return nil
}

func main() {
    if len(os.Args) == 1 {
        err := runParent()
        if err != nil {
            fmt.Printf("PARENT: %v\n", err)
        }
    } else {
        err := runChild(os.Args[1])
        if err != nil {
            fmt.Printf("CHILD: %v\n", err)
        }
    }
}

If I run this program on my Windows PC, I get this:

C:\>u:\test
PARENT: remove C:\DOCUME~1\brainman\LOCALS~1\Temp\ALEX711754103: The directory is not empty.

C:\>

Alex

[kumarharsh]

I'm facing this error through glide, the developer of which is the OP (matt). No idea what goes on inside glide itself, but apart from glide, no other program would be modifying or touching anything within the directory being deleted.

... except maybe Windows Defender ...

[djdv]

I can only give the not so helpful "works on my machine" response here, tested on W10(Ver:1703 Build:15063.447) with Go 1.8.3 and 1.9beta2. I have Windows defender disabled on this machine so maybe it's related. It might be worth running the tests with at least "Real-time protection" disabled since defender may have open handles when the files are fresh, although I'd imagine if this was the cause the results would be inconsistent.

[mattn]

I tried this on Windows7 64bit. And I got return nil.

I use Symantec Endpoint Protection. I'll try in later on Windows10 64bit.

[brendandburns]

@mattfarina

Check out:

https://glennsarti.github.io/blog/wsl-ruby-puppet/

For some instructions about disabling Defender on WSL files, it would be interesting to see if that fixes things on the WSL side...

[mattn]

I don't repro on Windows10. I use Windows Defender.

[thomastaylor312]

@brendandburns I just tried testing with the exclusion added and still get the issue. Both with the test repo from @mattfarina and trying it with glide. Let me know if I can test something else

[thomastaylor312]

I also just tried turning off real-time protection to no avail

[alexbrainman]

@kumarharsh and @thomastaylor312 if you can reproduce what @mattfarina described (the https://github.com/mattfarina/go-test-windows-files ), then you should be able to work out what is going on with os.RemoteAll.

For example, does this program https://github.com/mattfarina/go-test-windows-files/blob/master/main.go fails? What is the error message? Can you see any files remain in the "files" directory after program completes? Why did the program skipped them while deleting others? Did the program even attempted to delete them? Were the files in the "files" folder before you run the program? I would be interested in every error that occurs inside of os.RemoveAll, so I would be adding println everywhere to print every error. What are these errors? Can you see any errors related to the files that are left in the "files" directory.

Alex

[thomastaylor312]

@alexbrainman I will keep debugging it. But here are the answers to your questions:
Error message: remove files: directory not empty
Files deleted: It deletes about 5100 of them. I can't seem to find a pattern in which files are deleted and those that are stuck behind

I'll add some more debugging and see what I can find out

[alexbrainman]

Error message: remove files: directory not empty

Is the error message "directory not empty" or "The directory is not empty"?

Files deleted: It deletes about 5100 of them. I can't seem to find a pattern in which files are deleted and those that are stuck behind

There are 10001 files in that directory. So you are left with about 4901 remaining files. Out of remaining 4901 files, did os.RemoveAll even attempted to delete them? You can put println before each file is removed and log println output into a file, and than look in that file. If you discover that os.RemoveAll did attempt to delete them all, then each file deletion must have failed. What are the error messages for each failure? Again you could print them all with println.

Thank you.

Alex

[thomastaylor312]

@alexbrainman That is what I meant by "debugging" in this case. 🙂 I am planning on sprinkling println around the RemoveAll function to see what is going on. I will let you know my results as soon as I do.

As for the error message, I copy pasted it directly from the output. It is remove files: directory not empty

[mattfarina]

In the case the error remove files: directory not empty is in the pattern of remove [LOCATION]: directory not empty where [LOCATION] is the directory passed into os.RemoveAll.

In my case the example at https://github.com/mattfarina/go-test-windows-files removes a different number of files on each run. The last two runs removed 5118 and 5101 files.

I plan to merge in some changes to Glide this week to fix this. Hopefully it's just a temporary work around until we have a better fix.

@brendandburns Thanks for the Ruby WSL link. I'll dig into that a little more.

[thomastaylor312]

Ok, getting a little further on this. The names, err1 := fd.Readdirnames(100) call in RemoveAll is returning io.EOF. I am trying to figure out in Readdirnames and the functions it calls where the error is coming from. I haven't been able to yet

[mattn]

io.EOF is returned from:

https://github.com/golang/go/blob/6dcaa095c571988f2c1cfc9b914fb5128ab7c5ca/src/os/dir_windows.go#L66

And it is looked at:

https://github.com/golang/go/blob/5fea2ccc77eb50a9704fa04b7c61755fe34e1d95/src/os/path.go#L101-L106

When io.EOF is returned from Readdirnames, does this part hide err?

[thomastaylor312]

🤦‍♂️ I was looking in the wrong file for the error (unix instead of windows). I'll debug it some more tomorrow. Thank you for pointing me in the right direction (haven't hacked around in the standard library before)

[thomastaylor312]

Ok, so I have found that the error is occurring starting here. The syscall.ReadDirent call is returning 0, which happens twice when using the test repo like so:

Got to name range for test9997.txt
Got to name range for test9998.txt
got to buffer break 0
Got to name range for test9999.txt
got to buffer break 0

This break causes this line to return true and return an io.EOF. Based on what I print there when it gets the error, the length of names is 0 and n is 100.

I also tried printing out the value of n during each loop but I don't know what to make of the output or why it seems to start over:

Got to name range for test9998.txt
Value of n: 100
Value of n: 99
got to buffer break 0
Got to name range for test9999.txt
Value of n: 100
got to buffer break 0

Out of curiosity I tried changing this line from fd.Readdirnames(100) to fd.Readdirnames(-1) and I no longer got an error with RemoveAll. I have never really messed with raw kernel syscalls before so I am kind of at a loss for what I should try next.

[alexbrainman]

Ok, so I have found that the error is occurring starting here.

os/dir_unix.go file is not used on Windows. Are you running your test on Windows?

Alex

[thomastaylor312]

@alexbrainman This is WSL, so it is unix. I am running WSL on Windows 10. Sorry for the confusion on my part. In the first case when I tried this I was in dir_plan9.go but figured out that it was dir_unix.go that I needed to change

[bradfitz]

If Windows is implementing the Linux system calls in the wrong way (== any way different from Linux), then this is a bug in WSL, not Go.

But fortunately the WSL team is very quick to fix such bugs.

[alexbrainman]

@alexbrainman This is WSL, so it is unix. I am running WSL on Windows 10.

I don't have WSL installed. I won't be able to help you with that.
But it should not be much different - you just add println everywhere to see what is happening, and what errors happen. And then compare what you see with what API documentation promises. And tell us which API does not deliver on its promise.

I suspect @bradfitz knows who to ping about WSL bugs.

Alex

[thomastaylor312]

@alexbrainman That output I posted above is what I found out from putting println everywhere. I think it has something to do with the syscall I mentioned, so it may be good to check with the WSL team to see if there is a bug with how the syscall is implemented. @bradfitz any pointers on who I could talk to?

[brendandburns]

I can direct you to the WSL folks or you can file a bug here:

https://github.com/Microsoft/BashOnWindows

Let me know if you have trouble getting traction and I can help accelerate.

[thomastaylor312]

Thanks @brendandburns! I'll do a bit more debugging and then open up an issue

[zhengtg]

CustomRename function in windows also has bug.
if my project root directory not in c:\, glide get command get error.
it because move can't move directory between two partitions. eg：
cmd.exe /c move "C:\Users\adminAppData\Local\Temp\1624_30103" "D:/project/vendor"
拒绝访问。
移动了　 0 个目录。

[bradfitz]

/cc @jessfraz for WSL problem.

[jessfraz]

Sweet!

On Sep 23, 2017 18:24, "Brad Fitzpatrick" [e-mail] wrote:

/cc @jessfraz https://github.com/jessfraz for WSL problem.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/golang/go/issues/20841#issuecomment-331673659, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABYNbNmO3WnGnC0vmVvz8cVcoq7LdKvpks5slYU3gaJpZM4OIstj
.

[ianlancetaylor]

I think the fix is in https://golang.org/cl/62970 .

[alexbrainman]

CustomRename function in windows also has bug.
if my project root directory not in c:, glide get command get error.
it because move can't move directory between two partitions. eg：
cmd.exe /c move "C:\Users\adminAppData\Local\Temp\1624_30103" "D:/project/vendor"
拒绝访问。
移动了　 0 个目录。

@zhengtg you are correct, CustomRename function has a bug (I can reproduce it here - Windows move command refuses to move directory if source and destination are on different volumes), but CustomRename function is part of glide project, and you should use https://github.com/Masterminds/glide/issues to report problems with glide command.

Alex

[tyru]

This seems to occur again in go1.10beta1.

Sorry, deleted my comment because it was totally wrong.
After checking the source code, it appears this patch has not been applied yet to go1.9.1, go1.9.2, go1.10beta1.

[tyru]

For go1.10beta1, I confirmed this patch fixed the problem for my environment at least.

Environment

• go version go1.10beta1 linux/amd64
• OS name: Microsoft Windows 10 Pro (from systeminfo command)
• OS version: 10.0.15063 N/A Build 15063 (from systeminfo command)
• Ubuntu (WSL): 16.04.3 LTS (from lsb_release -a command)

[bradfitz]

@tyru correct, there is no fix for this yet, in any branch, and it will probably not be fixed in either Go 1.9.x or Go 1.10.x.

It's really Microsoft's job to fix this in WSL if they want to fully act like Linux.

But we might consider a fix for Go 1.11. The change above (https://golang.org/cl/62970) has merge conflicts and unaddressed feedback. Ping @raggi. :)

[jessfraz]

ping @jstarks

[jstarks]

Interesting. We (WSL team) will look into this.

[jstarks]

OK, that was fast. We believe this has been fixed in WSL in the recently released Windows 10 Fall Creators Update ("RS3", build 16299). Please let us know if you're able to reproduce this after updating to the latest Windows release.

[raggi]

Sorry, catching up with the comments on the original CL has been falling down my priority list. I won't be picking it up until January at the earliest.

It's important to point out here that the title is misleading and makes me sad that I mentioned Windows in the original bug report, because it's too easy to focus on that too much.

This behavior exists for many filesystems on many platforms. The spec allows for concurrent modification and for unbounded directory sizes. There is no tractable implementation for very large directories other than the behavior the original CL aims to fix. Platform doesn't really matter. You can almost certainly replicate the original bug with sufficiently large directories on both OSX and NFS today.

[bradfitz]

@raggi, thanks for reminding us of that. I remember you did mention that in the CL review comments. Will retitle.

[cwen0]

I also encounter this error when using os.RemoveAll, and I don't use WSL.

os : CentOS Linux release 7.3.1611 (Core) 
kernel: 3.10.0-514.el7
go version: go version go1.9.2 linux/amd64

Is there any good way to replace os.RemoveAll?

[davecheney]

@cwen0 please file a new issue. If it turns out it’s a duplicate we’ll link the two issues.

[cwen0]

ok, Thanks

[gopherbot]

Change https://golang.org/cl/121255 mentions this issue: os: when looping in RemoveAll, close and re-open directory

[gopherbot]

Change https://golang.org/cl/171099 mentions this issue: os: fix RemoveAll hangs on large directory