Go: mime/multipart: TempFile file hangs around on disk after usage in multipart/formdata.go

[nilslice]

Hey Go team,

I'd like to know if this is a bug or expected behavior. If it is a bug, I would like to write a patch to fix it.

---

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version go1.8 linux/amd64

What operating system and processor architecture are you using (go env)?

GOOS="linux"
GOARCH="amd64"

What did you do?

When uploading a file to a Go HTTP server, the call to req.ParseMultipartForm() subsequently calls readForm() and if the file size exceeds the maxMemory setting it creates a file on disk using the ioutil.TempFile() func, but the resulting file is never removed by the Go program. Unix behavior aside, based on the documentation for ioutil.TempFile(), the caller should remove the file once it is done using it. The file is only ever removed from the Go program if the io.Copy returns a non-nil error.

If possible, provide a recipe for reproducing the error.
A complete runnable program is good.

I would need to show a lot of code to demonstrate this, but you can see in the Go src here how the file is never removed: https://github.com/golang/go/blob/master/src/mime/multipart/formdata.go#L78

A link on play.golang.org is best.

What did you expect to see?

I expect that the temp file is removed after the file is copied to it's designated location on disk.

What did you see instead?

my /tmp directory is littered with multipart-XXXXXXXX files

[bradfitz]

When you're done with the multipart files, you can call:
https://golang.org/pkg/mime/multipart/#Form.RemoveAll

... which is accessible here:
https://golang.org/pkg/net/http/#Request.MultipartForm

But, yeah, maybe this could be more automatic. Maybe the multipart.File's concrete type could delete-on-Close.

Feel free to work on it!

[nilslice]

Thanks for the suggestions! I wasn't aware of Form.RemoveAll - good to know.

From my perspective, the complete process of dealing with HTML forms / other HTTP clients uploading files to my server shouldn't involve removing the temp files w/ RemoveAll. Based on the UX of a typical Go programmer, automatically cleaning up those files doesn't feel overly magical to me.

I'll work on it and submit if something clicks.

[kennygrant]

@nilslice still working on this?

We were hit by a similar problem (have small servers accepting file uploads, and tmp files filled the disk at times if not flushed regularly), and I too was unaware of Form.RemoveAll, but delete on close would be even better.

Happy to look at it if you're busy or leave it to you if you'd like to look at it.

[nilslice]

@kennygrant - I have a partial fix, but have been swamped with other work. If it's imperative to your project that it's patched, go for it! Otherwise, I'd still like to finish it. But no problem whatsoever if you want to get the patch in.

[kennygrant]

That's ok, I'll leave it to you if you have started. We have a workaround for now.

[Kagami]

But it's removed at the end of request?

https://github.com/golang/go/blob/release-branch.go1.10/src/net/http/server.go#L1553-L1555

[nilslice]

@Kagami - it does appear to make the call, but the last time I tested it the files were still on disk.

The blame shows that the code has been in the tree for ~7 years, well before this issue was originally filed: https://github.com/golang/go/blame/678dede7bc22a7cccfe78c42711478a9b1ecf4d3/src/net/http/server.go#L1553-L1555

Have you tried this to verify that the files are cleaned up after the request?

I'm trying to get my CL out this month or early next (taking a couple days off from work to do long put-off open source tasks I have queued up!).

[gopherbot]

Change https://golang.org/cl/113055 mentions this issue: mime/multipart: properly remove temp files in case of error

[lpar]

If this won't be fully fixed in 1.10, how about mentioning the need to call RemoveAll in the documentation for ParseMultipartForm?

[dvdscripter]

I'm no expert but you should be able to remove files without closing them?

...
file, _, err := r.FormFile("file")
if err != nil {
    return err
}
defer file.Close()
...

Leaves no "multipart*" files behind.

[agnivade]

@nilslice - Did you get a chance to work on this ? As per @Kagami's comment, it seems like the file is indeed removed.

Please also try with the 1.12beta1 to see if there has been any changes.

[nilslice]

@agnivade - I have not recently had time to wrap this up. If anyone would like to take it, or if it is indeed no longer an issue, that's awesome 😄

[chris3torek]

Note that this is still a problem with code generated by swagger. It's hard to work around in all cases. Success cases are easy enough, as your swagger parameters include the original HTTP request so you can do:

form := params.HTTPRequest.MultipartForm
err = form.RemoveAll()
// handle err, if you like
form.File = nil // in case the bug is fixed in the future

However, this doesn't handle every possible error path (specifically any where your own handler never gets called due to an error detected inside swagger itself).

[tooptoop4]

@bradfitz where is remove in net/http/#Request.MultipartForm ?