Runtime: Add BinaryWriter.Write(int[]) and similar

[SoftCircuits]

Feature Suggestion

If I have an int array, I can write it to a binary file using the following code.

using (BinaryWriter writer = new BinaryWriter(File.Open(path, FileMode.Create)))
{
    for (int i = 0; i < array.Length; i++)
        writer.Write(array[i]);
    writer.Close();
}

But this approach does not seem optimal. An int array is stored in contiguous memory, so I don't understand why there isn't a way to write the entire array in a single call, as I could with byte[] or Span<byte> (or as I could with C or C++). There are workarounds, but I don't see why they are necessary.

How about adding overloads to BinaryWriter.Write() and BinaryWriter.Read() that accept int[] or Span<int> arguments, and perhaps other types as well. And maybe even FileStream could support the same?

This seems like one area where C# seems unnecessarily less efficient than, say, C or C++, and as long as the arrays are stored in contiguous memory, I don't see why this isn't a no-brainer.

[Symbai]

How about? https://docs.microsoft.com/en-us/dotnet/api/system.io.file.writeallbytes?view=netcore-2.2

File.WriteAllBytes(String, Byte[])

Or if you need to append an array:

using (var stream = new FileStream(path, FileMode.Append))
{
   stream.Write(array, 0, array.Length);
}

[SoftCircuits]

WriteAllBytes() does not accept int[].

[Symbai]

I'm sorry, I misread. You can use MemoryMarshal casting the int[] into byte[] and then write it all at once the usual way.

MemoryMarshal.Cast<int, byte>(array)

https://docs.microsoft.com/en-us/dotnet/api/system.runtime.interopservices.memorymarshal.cast?view=netcore-2.2

[SoftCircuits]

Yes, or MemoryMarshal.AsBytes(). But I'm not understanding why any workaround is necessary here. Why not just support int[]?

[Symbai]

I'm sorry that question is beyond my scope. I was just trying to help but it turned out you haven't asked about how rather than about why. Please ignore my posts then.

[SoftCircuits]

Yeah, I appreciate that. It's just that MemoryMarshal requires another nuget reference and has its own overhead (although very little, it appears). This was meant to be a feature suggestion.

[EgorBo]

MemoryMarshal is part of mscorlib (at least in .NET Core 3.0) and I don't think there should be any overhead.

[SoftCircuits]

I stand corrected. My class library is using .NET Standard.

Of course there's overhead. But, as mentioned, it appears to be very minimal. I also indicated there were workarounds. But I'm having trouble why we need to find workarounds here.

[mikedn]

Yeah, I appreciate that. It's just that MemoryMarshal requires another nuget reference and has its own overhead (although very little, it appears). This was meant to be a feature suggestion.

Even if it has overhead it's likely that if such BinaryWriter overloads will be added they'll simply use MemoryMarshal. There's no other way to implement such overload efficiently. Except when running on a big-endian platforms, which will require endianness conversion (AFAIR BinaryWriter is little-endian only).

[SoftCircuits]

Even if it has overhead it's likely that if such BinaryWriter overloads will be added they'll simply use MemoryMarshal. There's no other way to implement such overload efficiently. Except when running on a big-endian platforms, which will require endianness conversion (AFAIR BinaryWriter is little-endian only).

Why can't they just write unsafe code that takes the pointer to the integer array, and treat it as a byte array with four times as many elements? I hadn't thought about endianess on other platforms, but it doesn't seem like it's more of an issue than using MemoryMarshal.AsBytes() would be.

[mikedn]

Why can't they just write unsafe code that takes the pointer to the integer array, and treat it as a byte array with four times as many elements?

You can use unsafe code to treat it as a pointer to byte (byte*) but not as a byte array (byte[]). And you need a byte array to pass to the underlying stream.

Well, these days streams also accept Span<byte> and a span can be created from a byte*. But then it's not very different from using MemoryMarshal. In fact it's probably worse because using pointers will pin the array for the duration of the write.

[SoftCircuits]

You can use unsafe code to treat it as a pointer to byte (byte*) but not as a byte array (byte[]). And you need a byte array to pass to the underlying stream.

Well, assuming it eventually makes its way down to the WriteFile() Windows function, it seems like this should be straight forward. But, yes, I guess a stream can be other things like MemoryStream. As you say, it accepts Span<byte>. So why not Span<int>?

[mikedn]

So why not Span?

Streams work with bytes. They're not concerned with how to convert from more complex types to bytes, that's the concern of writers, formatters, serializers etc.

[JeremyKuhne]

If we get enough feedback and samples that show this is useful (measuring perf benefits, how broadly useful, etc.) I'm not opposed to this. Given that BinaryWriter is little endian we could potentially add significant optimization if we're currently running on a little endian system.

If we add overloads for any other primitives is that we should probably add them all. We would need to do span and array overloads to facilitate languages that don't have span. That would be 20 new APIs.

[SoftCircuits]

@JeremyKuhne As far as usefulness, it must surely be a common task to want to write an array other than a byte array to a file. And int array in particular. Maybe I do more binary file writing that most, but this has come up for me a number of times.

[Gnbrkm41]

why int array in particular? From what I've seen, binary data are always passed around various APIs as either pointer to a byte, a byte array or span of byte.

[SoftCircuits]

@Gnbrkm41 The point is that safe C# doesn't support pointers, and so the efficient way I would have handled this in C or C++ is not an option here. It's a limitation of the language.

An int array in particular because I strongly suspect that writing an int[] is the most common array type next to byte[]. At least, this has come up for me on a number of occasions. But, yes, it would be nice to do the same thing with longs and maybe other types as well. (Not string[] because that does not store all data contiguously and the same type of optimizations would not be possible.) But I'll start with int[] and go from there. :)

[Joe4evr]

The point is that safe C# doesn't support pointers

That's exactly what Span<T> has been made to solve.

[SoftCircuits]

@Joe4evr If Span<int> was supported by BinaryWriter.Write(), then we'd be good to go.

[GrabYourPitchforks]

That's exactly what Span<T> has been made to solve.

There's no "safe" API to go between Span<byte> and Span<int>. You'd have to go through the MemoryMarshal or Unsafe classes. Both classes are considered equivalent to using the unsafe keyword. Think of it as akin to the Marshal APIs which take IntPtr as a parameter. You don't need to write the unsafe keyword to call these APIs, but you are nevertheless performing operations on raw pointers and need to use caution. If you're disciplined in how you use the APIs then they'll allow you to accomplish your scenario with minimal overhead.

If you're trying to do this the "safe" way, you could allocate a new byte[] and use Array.Copy(Array, int, Array, int, int) to copy data between the int[] and byte[]. That pattern incurs a copy but would be fully verifiable and type-safe. You'd still be responsible for any necessary endianness fixups.

[SoftCircuits]

If you're trying to do this the "safe" way, you could allocate a new byte[] and use Array.Copy(Array, int, Array, int, int) to copy data between the int[] and byte[]. That pattern incurs a copy but would be fully verifiable and type-safe. You'd still be responsible for any necessary endianness fixups.

The whole point of using Span<T> is really a performance one. Have to copy arrays here would be inconsistent with the objective.

[GrabYourPitchforks]

So if you're willing to use unsafe-equivalent code, then the solution was already given:

myBinaryWriter.Write(MemoryMarshal.AsBytes(myArray)); // projects a T[] as a ROS<byte> and writes it

That pattern also works with sbyte[], short[], ushort[], int[], uint[], MyBlittableStruct[], and so on. So it's far more flexible than adding just the one overload asked for here.

[SoftCircuits]

So if you're willing to use unsafe-equivalent code, then the solution was already given:

So then why add Span? I would expect that to be more performant than MemoryMarshal. Certainly, easier to use. Span was added, so why not Span<int>?

[GrabYourPitchforks]

Span<byte> is the way to represent arbitrary buffers of binary data. It makes sense to add a Span<byte>-consuming overload to an API that deals with streaming data.

Contrast this against Span<T> (for T != byte), which is the way to represent contiguous blocks of elements of type T. With _very_ few exceptions, the Framework isn't in the business of assuming how such data should be projected as a buffer of arbitrary binary data. That's generally a concern left to the application developer.

But once you project the data as a Span<byte> - either using one the Framework-provided methods or using your own logic - then you can use the API BinaryWriter.Write(ReadOnlySpan<byte>) to write the data regardless of how you ended up projecting it.

I think this actually shows the great power of gluing all of these APIs together. Instead of adding an API that would be used by a single consumer, we have a more generalized pattern that allows both you and anybody who has variations of this scenario to accomplish this in a single line of code using existing APIs.

[vadimkantorov]

A problem with ROS is that it doesn't support lengths > 2Gb...

[SoftCircuits]

I'm curious what happened to this. It was closed, and no one else seemed to see it from my perspective. But then I see it was added to the "5.0 milestone". Does that mean there's still hope?

BTW, I've programmed a lot of assembly language, C and C++. Over the decades, you form ideas about how to do things efficiently in a language. This suggestion is about my frustration over why C# can't provide similar efficiency in an accessible way.