Go: io/ioutil: predictable suffix for TempFile
Right now ioutil.TempFile creates files with a given prefix as the basename. This is fine for the usual /tmp usage, where all the files in that directory are temporary files. However, for e.g. replacing a state file ~/.foo.json with a new one, what I'd want is writing to e.g. ~/.foo.<RANDOMNESS>.tmp and renaming that over ~/.foo.json. This way I can make e.g. backup and editor software ignore *.tmp files, and even clean old leftover temp files automatically.
gopherbot
All 35 comments
_Labels changed: added priority-someday, packagechange, removed priority-triage, go1.1maybe._
_Status changed to Accepted._
adg
on 25 Feb 2013
rsc
on 3 Mar 2014
I have run into this
See cmd/pprof/internal/tempfile has a nice function https://golang.org/pkg/cmd/pprof/internal/tempfile/#New
tempfile.New(dir, prefix, suffix string) (*os.File, error)
although imports of internal packages aren't allowed in >=Go1.6.
Yet
ioutil.TempFile(dir, prefix string)
and adds a suffix to the end of the path, and this messes up the intended suffix
My use case is for programs that need the suffix to try to select the opener by using the suffix/extension e.g
$ file pliesHello
pliesHello: ISO Media, MPEG v4 system, version 2
$ open pliesHello
cmd/pprof/internal/tempfile IMO provides a better API, since it even allows for DeferPath(path) deletion and no need to Close() then os.RemoveAll()
Any thoughts?
odeke-em
on 13 Jan 2016
@odeke-em maybe I don't understand the issue, but can't you just os.Rename the temp file once it's been created ?
davecheney
on 13 Jan 2016
Yap, same train of thought. For now that's exactly what am doing.
However, notice the number of steps that it takes:
- ioutil.TempFile(entryDir, prefix).
- defer tempFile.Close().
- defer os.RemoveAll(path.Join(entryDir, tempFile.Name()))
- and all the error checking.
os.Rename suffers from a few problems if the /tmp dir is not on the same device
unless we do byte by byte copying, please see https://github.com/rakyll/statik/pull/5.
It also isn't atomic.
Now let's compare that to using tempfile.New, and tempfile.DeferDelete and that's simpler.
I am all for us keeping things as they were if it keeps the code and language simpler and if the implementation is tricky, then I can implement some package or ways around it.
odeke-em
on 13 Jan 2016
Some tools (e.g., vips) rely on the extension to determine how to process the file. When exec such commands and feeding them temp files, being able to specify suffix is very important.
Right now I have to copy https://golang.org/pkg/cmd/pprof/internal/tempfile from an early go to a new package to achieve that.
hgl
on 23 Nov 2017
As a workaround, we just open sourced a tiny library which supports suffix, too: https://github.com/tink-ab/tempfile
JensRantil
on 26 Feb 2018
This is even more important for Windows systems where extensions mean something. For example, in one project we needed to have a ".bat" extension.
TomOnTime
on 2 Mar 2018
This seems like a good one for community day. Can we add a "help wanted" label?
TomOnTime
on 2 Mar 2018
I think I'm going to volunteer for this.
I'm new to contributing to this project, so hand holding is appreciated.
According to Contribution Guide, I should discuss the design a bit. Here is my design proposal:
- ioutil.TempFileExt() will be like TempFile() but take an extra parameter, the extension.
- The caller will have to include the entire extension, including the dot:
.batnotbat - refactor TempFile() to call TempFileExt(a, b, "")
- rename helper function nextSuffix() to nextRandText() to be more accurate.
- add unit tests
TomOnTime
on 2 Mar 2018
Thanks for your work @TomOnTime.
I'm not sure if I'm overengineering this, but I don't think it needs to have anything to do with extensions.
What if we use a more generic solution:
``
ioutil.TempFileTemplate(dir, "{{x}}file.bat")
````
Where{{x}}(mimicking the syntax introduced intext/template) is the random number that will be inserted by the function. The syntax and other restrictions (e.g., only one{{x}}` is allowed, otherwise panic) are debatable.
hgl
on 2 Mar 2018
The caller will have to include the entire extension, including the dot:
.batnotbat
If the caller does not provide an extension will bat just be appended, or are you saying that should be an error?
I'll vote for ioutil.TempFileExt(). path and path/filepath use *Ext() already. I think making it a template opens up too many error conditions.
adamdecaf
on 2 Mar 2018
The caller will have to include the entire extension, including the dot: .bat not bat
If the caller does not provide an extension will bat just be appended, or are you saying that should be an error?
I just mean that the the filename is made of prefix + random + extension, therefore callers will usually call TempFile() with a suffix that includes the "." if they want a ".".
Callers might assume the formula is prefix + random + "." + extension and they would be wrong.
TempFile(a, b) is simply a call to TempFileExt(a, b, "")
I think TempFileTemplate is a bit overly complex for the needs here.
TomOnTime
on 2 Mar 2018
FYI:
Technically the 3rd parameter is a suffix, not an extension. A better name is TempFileSuffix(p,n,s)
TomOnTime
on 2 Mar 2018
TempFileSuffix makes more sense. Thinking about this again I'd expect TempFileExt to include a . if I left it off.
adamdecaf
on 2 Mar 2018
Change https://golang.org/cl/98375 mentions this issue: ioutil: Add TempFileSuffix(), similar to TempFile()
gopherbot
on 3 Mar 2018
@ianlancetaylor. Did additional API for this get approved by the proposal process?
dsnet
on 3 Mar 2018
As far as I know this never went through the proposal process. I guess these days it probably should.
ianlancetaylor
on 3 Mar 2018
Makes sense. I'm new here so... what's next?
TomOnTime
on 3 Mar 2018
Makes sense. I'm new here so... what's next?
Welcome and thanks for interest in contributing :)
The formal process is here and is intended to ensure that new features are bring sufficient benefit to warrant expanding the API, and don't bring along with it major detriments.
Periodically (I think weekly?), a group of proposal reviewers scrub through issues in the proposal milestone and determine whether to approve/disapprove the suggested API change. However, there is a large backlog of proposals, so it may take time for them to get around to this one.
dsnet
on 3 Mar 2018
There's ioutil.TempDir already, that's one option if you want control over file suffixes.
We don't yet know how we'd want to add support for suffixes, or if we really do. One option instead of TempFile2 is to define that the current prefix is a pattern so that if there's a * that's where the randomness goes.
ioutil.TempFile("", "foo*.bat")
etc.
rsc
on 5 Mar 2018
Does @rsc's suggestion above address the use case here?
ianlancetaylor
on 19 Mar 2018
Does @rsc's suggestion above address the use case here?
The use of "*" as a marker for where to insert the random bits? Yes, that would work.
TomOnTime
on 20 Mar 2018
This would be a very welcome change guys, a huge functionality improvement.
Moataz-E
on 20 Mar 2018
OK, let's do what I wrote above: if "*" is in the string, then replace it with the random part. If there are multiple stars, use the first one (we have to pick something).
rsc
on 2 Apr 2018
I volunteer to write this. Seems like a good starter-project.
TomOnTime
on 2 Apr 2018
Personally, I like predictability and "following the spirit of existing API" of an original TempFileSuffix proposal more.
IMHO there are too many problems with *
- If only first
*is going to be replaced, how long should that part be, "enough to be unique" or some predefined minimum? - What if there are multiple
*, how long is long enough? - I want a file with
*in it (yes, exceptionally rare case), what do I do? Escape it?
tandr
on 3 Apr 2018
The new implementation replaces the first *. If you want an actual * in the file name, just include it after the first *.
TomOnTime
on 4 Apr 2018
@TomOnTime that way "Temp*.RANDOMPART.json" will still be impossible
tandr
on 4 Apr 2018
@TomOnTime that way "Temp*.RANDOMPART.json" will still be impossible
@tandr Agreed.We're making it difficult to do pathologically bad things.
TomOnTime
on 8 Apr 2018
Change https://golang.org/cl/105675 mentions this issue: io/ioutil: TempFile should permit user-specified suffix
gopherbot
on 8 Apr 2018
Any prediction of when this feature will be merged to go release?
felipeek
on 13 Jul 2018
@felipeek It should be in Go1.11beta1 already. You can try it out!
https://groups.google.com/forum/#!topic/golang-announce/m1Szl8Mu_Fs
Also, closed issues aren't monitored. If you find a bug please open a new issue.
adamdecaf
on 13 Jul 2018
Related issues
natefinch
·
3Comments
michaelsafyan
·
3Comments
myitcv
·
3Comments
rsc
·
3Comments
bradfitz
·
3Comments
Most helpful comment
There's ioutil.TempDir already, that's one option if you want control over file suffixes.
We don't yet know how we'd want to add support for suffixes, or if we really do. One option instead of TempFile2 is to define that the current prefix is a pattern so that if there's a * that's where the randomness goes.
etc.