Go: os: O_SYNC not utilized in os.OpenFile() on Windows

[aschmahmann]

While Linux systems are able to pass their low level flags directly into os.OpenFile on Windows os.OpenFile takes "invented values"

https://github.com/golang/go/blob/5d1a95175e693f5be0bc31ae9e6a7873318925eb/src/syscall/types_windows.go#L34-L48

These invented values are then checked against a subset of the features that Windows actually supports in syscall.Open

https://github.com/golang/go/blob/03aca99f476c34bad927410741251162181b6e16/src/syscall/syscall_windows.go#L273

Despite Windows supporting FILE_FLAG_WRITE_THROUGH (very close to O_SYNC on Linux) the Open function does not check for O_SYNC. This is unexpected behavior as developers would expect the O_SYNC flag on Windows to work since it can perform synchronous writes.

Some ways to resolve this unexpected behavior include:

• Adding a check for O_SYNC in syscall.Open in syscall_windows.go
• Having syscall.Open perform on Windows the same way it does on Linux, by passing in file flags directly
  
  
  
  • Note: This solution would also help with https://github.com/golang/go/issues/34681
  
  

[networkimprov]

cc @alexbrainman @zx2c4 @mattn
@gopherbot add OS-Windows

[alexbrainman]

Thank you for creating this issue.

I am not an expert here. But, I am not against mapping syscall.O_SYNC onto FILE_FLAG_WRITE_THROUGH.

Alex

[tsellers-r7]

Are there data loss concerns related to this issue. For example, expecting O_SYNC force writes to disk before returning but instead the OS is caching the writes?

[aschmahmann]

@tsellers-r7 my understanding is the answer is "sort of".

https://github.com/golang/go/blob/03aca99f476c34bad927410741251162181b6e16/src/os/file.go#L68-L81

The comment here states that not all flags are passed/utilized on every platform. However, if you assumed that O_SYNC worked on your platform (which wouldn't be unreasonable on Windows given the existence of FILE_FLAG_WRITE_THROUGH) and turned out to be incorrect you could experience unexpected data loss (as in https://github.com/dgraph-io/badger/issues/1084).

[alexbrainman]

However, if you assumed that O_SYNC worked on your platform (which wouldn't be unreasonable on Windows given the existence of FILE_FLAG_WRITE_THROUGH) and turned out to be incorrect you could experience unexpected data loss (as in dgraph-io/badger#1084).

I don't see any promise of data loss prevention in O_SYNC documentation.

It says

O_SYNC int = syscall.O_SYNC // open for synchronous I/O.

whatever that means.

Alex

[aschmahmann]

Fair enough, I guess I was referring to the Linux O_SYNC documentation http://man7.org/linux/man-pages/man2/open.2.html.

@alexbrainman you're absolutely correct that there aren't any Go contracts/documentation currently being violated. However, if a user isn't really sure what O_SYNC's synchronous I/O means they'll probably reach for something similar, like the link above.

[networkimprov]

The point of O_SYNC or write-through is to leave the file cache untouched, usually because the file to be written should not be added to the cache, for example in a backup scenario.

[ianlancetaylor]

@networkimprov That sounds like like O_DIRECT. O_SYNC means that writes will not return until the data has been sent to the underlying hardware. (Although modern smart disks will themselves buffer the data, so even that is not necessarily a guarantee.)

[networkimprov]

Oh, yes, O_SYNC makes write() become write() + fsync().

[zx2c4]

We have:

nix:

• O_SYNC
• O_DIRECT

win:

• FILE_FLAG_NO_BUFFERING
• FILE_FLAG_WRITE_THROUGH

Documentation on FILE_FLAG_NO_BUFFERING: https://docs.microsoft.com/en-us/windows/win32/fileio/file-buffering
Documentation on FILE_FLAG_WRITE_THROUGH: https://docs.microsoft.com/en-us/windows/win32/fileio/file-caching

Quick perusal makes me think that the closet approximation is actually O_SYNC=FILE_FLAG_WRITE_THROUGH, O_DIRECT=FILE_FLAG_NO_BUFFERING. Does this correspond with other folks' reading too?

Note that we'll probably never match the exact semantics, but we can find something that's reasonably close, I wouldn't mind making such a mapping in Go.

[networkimprov]

@ianlancetaylor is there a reason O_DIRECT wasn't included in pkg os?
As it happens, I now need it for a directory backup feature.

[ianlancetaylor]

The os package is intended to be portable across operating systems, and not even all Unix-like systems support O_DIRECT. On systems that do support it you can use syscall.O_DIRECT with os.OpenFile.

[networkimprov]

@gopherbot add "help wanted"

[jarifibrahim]

It should be easy to add the correct flags in the Open(..) syscall. @networkimprov @ianlancetaylor should I send a PR or is there a decision pending?

[ianlancetaylor]

You can send a pull request. It should only change the syscall package to handle O_SYNC on Windows. Thanks.

(Note that we just entered a release freeze so any change will be for the 1.16 release.)

[odeke-em]

Thank you everyone, moving to Go1.16 as per Ian's last comment https://github.com/golang/go/issues/35358#issuecomment-622500318 and given that the CL was sent 5 days ago and almost a month in the freeze. I've reviewed and posted feedback on the change though, so in August 2020 we'll hopefully merge it in.

[networkimprov]

The CL doesn't appear on this issue because the Fixes line in the comment should be #35358, not a URL.

[jarifibrahim]

@networkimprov Thank you. I've fixed it.

[gopherbot]

Change https://golang.org/cl/235198 mentions this issue: syscall: support O_SYNC flag for os.OpenFile on windows