Discordchatexporter: PlainText and CSV formats don't preserve multiline messages properly.

[tyzoid]

Presently, there are only two text-based formats for the export of messages, PlainText and CSV. (HTML requires a browser to view, and generates a lot of extra boilerplate taking up unnecessary space).

Both of these have problems when it comes to dealing with multiline messages (shown below). The CSV export cause newlines in messages to be replaced with a comma, but doesn't escape commas within messages, leading to an inability to separate lines back out. Further, the plaintext export doesn't distinguish between multiple messages from the same person, or one multi-line message.

CSV:

Author;Date;Content;Attachments;
tyzoid#0358;05-Oct-18 12:08 PM;"1: Multiline message test, 2: 2nd line, 3: 3rd line";;
tyzoid#0358;05-Oct-18 12:10 PM;"1: Multiline message with commas,, 2: How will it cope?, 3: More commas,,,, comma,,, lol";;
tyzoid#0358;05-Oct-18 12:10 PM;"Multiline message with quotes "" test,, Comma too, lol";;

PlainText:

==============================================================
Guild:    Direct Messages
Channel:
Topic:
Messages: 3
Range:    05-Oct-18 12:00 AM ->
==============================================================

[05-Oct-18 12:08 PM] tyzoid#0358
1: Multiline message test
2: 2nd line
3: 3rd line
1: Multiline message with commas,
2: How will it cope?
3: More commas,,,, comma,,, lol
Multiline message with quotes " test,
Comma too, lol

[Tyrrrz]

Do you have a suggestion on how this should be fixed?

[tyzoid]

For the plaintext, I'd recommend delimiting each message, for example:

==============================================================
Guild:    Direct Messages
Channel:
Topic:
Messages: 3
Range:    05-Oct-18 12:00 AM ->
==============================================================

[05-Oct-18 12:08 PM] tyzoid#0358
1: Multiline message test
2: 2nd line
3: 3rd line
1: Multiline mess
[05-Oct-18 12:08 PM] tyzoid#0358age with commas,
2: How will it cope?
3: More commas,,,, comma,,, lol
[05-Oct-18 12:09 PM] tyzoid#0358
Multiline message with quotes " test,
Comma too, lol

For CSV, there's a standard way to embed newlines, which is to surround the field in quotes, and put a literal newline in the CSV. One potential issue with this is that it could possibly break compatibility with tools/scripts that don't support the full CSV standard, as it means one message might span multiple lines.

[DashLt]

Why not just replace newlines with '\n', and escape actual mentions of '\n'?

[tyzoid]

@DashLt That works, but it's a bit clunky. Plus, you've still got the issue that messages may be merged in plaintext.

For CSV, I'd highly recommend following spec instead of encoding newlines in other ways:
https://tools.ietf.org/html/rfc4180#page-3

From section 2:

   6.  Fields containing line breaks (CRLF), double quotes, and commas
       should be enclosed in double-quotes.  For example:

       "aaa","b CRLF
       bb","ccc" CRLF
       zzz,yyy,xxx

[Tyrrrz]

Fixed on master.
Messages are not grouped in PlainText, lines are not replaced in CSV.

[IntrepidVector]

Exporter still appears to have issues with exporting commas to CSV file type. Not an export option I plan to use going forward (HTML and plaintext appear to both work great), but thought you would like to know.

[Tyrrrz]

What's the exact problem?

[IntrepidVector]

I may have been hasty to attribute it to commas, but something is definitely going weird with CSV files. It may be that OpenOffice has some weird default import values. It was treating every space in certain messages as a cell delimiter.

With that setting unchecked, the only issue I'm still seeing with CSV files is that it's exporting some long messages that contain line breaks to several rows with blank rows in between.

[Tyrrrz]

That definitely doesn't look right. Is there any chance you can open it in a different application? Or maybe you can send the snippet that's causing issues, as long as there's no sensitive information.

[IntrepidVector]

I can definitely experiment with opening it in Excel or Google Sheets and
sure, I can send you some of the text that's exhibiting this. I'll get back
to you with what I find and with some sample text.

On Tue, Nov 20, 2018, 2:20 PM Alexey Golub <[email protected] wrote:

That definitely doesn't look right. Is there any chance you can open it in
a different application? Or maybe you can send the snippet that's causing
issues, as long as there's no sensitive information.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/Tyrrrz/DiscordChatExporter/issues/102#issuecomment-440416130,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AOhvAc03ASgJ0tl8PQ5QrA7O9Z-qEeNrks5uxGQDgaJpZM4XKihY
.

[IntrepidVector]

Tried it in Google Sheets. Same strange formatting occurs. Included here are:

• screenshot of how text appears in HTML view (correct)
• snippet of CSV that is 'misbehaving' (renamed to .txt so github will let me upload)
• same snippet in plaintext (correct)

textsample.txt
csv_sample.txt

[sauravexodus]

@DashLt That works, but it's a bit clunky. Plus, you've still got the issue that messages may be merged in plaintext.

For CSV, I'd highly recommend following spec instead of encoding newlines in other ways:
https://tools.ietf.org/html/rfc4180#page-3

From section 2:

   6.  Fields containing line breaks (CRLF), double quotes, and commas
       should be enclosed in double-quotes.  For example:

       "aaa","b CRLF
       bb","ccc" CRLF
       zzz,yyy,xxx

Not related to this issue, but this helped me save the day ;D. Thank you so much