Csvhelper: CSVWriter failing to complete second to last row, but column is partially complete!

[rcalderoni]

This is a strange one.

When I export my List<dynamic> to JSON it is all accounted for, no missing rows or data.

When I write it to CSVWriter the last row is missing, and the second to last row is half-complete, MID-COLUMN.

I determined that it is not crashing on the row itself because I took my record set, duplicated it and did a CONCAT on the clone, thus I can see at the end of the first set it has no issue writing the final 2 rows.

The column that it has been hanging on has the value of "1" where the identical row/column from the first set that writes successfully has a value of "15". It's literally crapping out in the dead middle of the column for some reason.

In another instance of failure to complete row the full column value should be "1970" but it stops dead at "19", never writing any more columns or rows after that.

So either something is jacked up, or I am not using the streams correctly... something in ASP.NET Core MVC is terminating the stream? Malware on my localhost or bad hardware? Very wonky.

public IActionResult Export()
{
    var ms = new MemoryStream();
    var writer = new StreamWriter(ms);

    var csv = new CsvWriter(writer);

    csv.Configuration.QuoteAllFields = true;

    var records = ReportService.ExportAllRecords();

    csv.WriteRecords(records);

    return File(ms.ToArray(), "text/csv", $"ExportAll{DateTime.Now.ToShortDateString()}.csv");
}

THE JSON of records: https://jsonblob.com/97c1ff24-d7d1-11e8-839a-1f7510819126

THE CSV turned XLSX because Github hates CSV (see row 18 and 26 should be identical, you can see where it crapped out, and row 19 is present, but row 27 is empty) ExportAll10%2F24%2F2018 (11).xlsx

[JoshClose]

Try

csv.WriteRecords(records);
writer.Flush();

[rcalderoni]

Thanks, that did it. So it was a race condition? It was returning before it was done? Was something running in another thread?

[JoshClose]

No. A StreamWriter has a buffer that is written to. When that buffer is full, it gets flushed to the stream. At the end of writing, you need to flush the writer so anything in the buffer gets written to the stream. If your writer is in a using block, it will get flushed automatically at the end of the block.

[MiroJ]

I am using ver 12.3.2 (from NuGet) and I still see the same issue, even with Flush() and FlushAsync(). My code is very similar:

try
{
    using var stream = new MemoryStream();
    using var writer = new StreamWriter(stream);
    using var csv = new CsvWriter(writer);

    csv.WriteRecords(list);

    csv.Flush(); // or await csvFlushAsync

    var bytes = stream.ToArray();

    return File(bytes, "text/csv", "mapxpress-export.csv");
}
catch (Exception)
{
    return null;
}

[JoshClose]

You need to flush the StreamWriter, not the CsvWriter.

writer.Flush();

WriteRecords is already flushing the CsvWriter buffer for you, so you can remove that.

[MiroJ]

You are absolutely right!
As if you can't be :)

Thanks!

[JoshClose]

No problem. I get things wrong all the time, so please challenge me if you think I'm wrong.

[Bassist067]

The StackOverflow quiestion didn't lead me to answer, but this thread did. We need to flush the streamWriter after filling stream and before copying it to file.

Thanks guys. You are the best.

[JoshClose]

@Bassist067 If you understand what is going on under the hood, it makes sense and becomes intuitive. Streams are buffered, meaning only a portion of the data is in the stream. As you write to a stream, the streams buffer fills up. Once that buffer is full, that buffer is flushed. The stream could be attached to a file or a network. If you were writing a single char at a time and it was sending that across the wire on a network, that would be really bad. This is why streams buffer the data. When you're done writing, you need to flush the remaining data that is sitting in the buffer. This will happen automatically if you call Dispose or are using a using(stream) { } block on the stream.

[JoshClose]

I have a "prerequisites" section in the documentation that goes over some of these things also. https://joshclose.github.io/CsvHelper/examples/prerequisites/streams