Discordchatexporter: 100% CPU usage when writing chat log to file
The client jumps to CPU 100% for several minutes when exporting a large channel (18K messages) to CSV (macOS, Mono).
The final file is just 2,3 MB and Go parses this in under a second.
terorie
All 17 comments
Works perfectly with large channels and CSV here. What are the specs of your Mac and which version of DCE are you using?
Yudi
on 15 Mar 2019
In fact, it does get close to 100% for a few minutes when the file is being generated. So… Can confirm.
Yudi
on 15 Mar 2019
@RenanYudi Just compiled master, Visual Studio Community 2017
Imo, Mac specs don't matter much (CSV serializing needs no CPU). MBP Mid 2015 Core i7 2,8 GHz
Channel is 461525479460044800, which you can access with invite
https://discordapp.com/invite/cMHemg8 if you want to try yourself
terorie
on 15 Mar 2019
I have a suspicion https://github.com/Tyrrrz/DiscordChatExporter/blob/77bf32aa5a811a4515a8eae23c8ca9b26cd07c24/DiscordChatExporter.Core/Services/ExportService.cs#L50-L58 is the slow part
terorie
on 15 Mar 2019
@RenanYudi I ended up writing a (streaming) dumper in Go that uses no CPU at all
https://github.com/terorie/discord-dump
Only does guilds in JSON for now, will add all the features of this repo.
Hope it's useful for the author.
terorie
on 15 Mar 2019
I'm guessing the CPU utilization is caused by the new markdown AST parser. In what version did it start to happen?
Tyrrrz
on 15 Mar 2019
100% utilization starts to get significantly longer with 2.10
Yudi
on 15 Mar 2019
Then the 100% utilization will likely be unavoidable for HTML exports but it can at least be improved so it doesn't block UI. The other export formats may need to be brought outside the templating engine to improve the performance. I was hoping having them all under the same engine would make maintaining them convenient, but I guess it hurts performance too much.
Tyrrrz
on 15 Mar 2019
I don't know C# enough to contribute, but I have an idea how to fix this. How about using a streaming interface with implementations for JSON, CSV and HTML and avoiding templating?
The pseudocode looks like this:
interface MessageWriter {
void SetWriter(Writer wr)
void Begin() // Writes header (Like <body><div id="messages">)
void WriteMessage(Message)
void Close() // Writes footer (Like </div></body>)
}
void dumpMessages(File file, Message[] msgs) {
MessageWriter w = getDefaultMessageWriter();
w.SetWriter(file);
w.Begin();
for (Message msg : msgs) {
w.WriteMessage(msg);
}
w.Close()
}
Even if it makes the code a bit less clean, HtmlWriter::WriteMessage could either escape and write the content by hand (all non-ASCII => HTML numerical escape, writing using String.format) or use a HTML template per message.
terorie
on 16 Mar 2019
Also @Tyrrrz would you mind if I take your HTML templates and put them in https://github.com/terorie/discord-dump (MIT license)? I'd give you attribution in source and program help of course.
terorie
on 16 Mar 2019
It used to be a streaming interface without templating, a long time ago: https://github.com/Tyrrrz/DiscordChatExporter/blob/d958f613a377d0a1a4f5b2f41071529df9ebb88b/DiscordChatExporter.Core/Services/ExportService.Html.cs#L252-L365
By looking at the file you can probably guess why we ended up using templates. But even then, the templating doesn't induce that much of an overhead, it seems it's mostly the markdown parsing which will not go anywhere anyway.
Tyrrrz
on 16 Mar 2019
You can use any file in this repository in accordance with GPLv3 so yes.
Tyrrrz
on 16 Mar 2019
@Tyrrrz https://github.com/nginx/nginx/blob/910f330ad0caa49fe901e9426ef00d95d45ba32c/src/http/modules/ngx_http_autoindex_module.c#L446 It might look bad but even Nginx does it
I think hardcoding is worth the performance, would you agree?
terorie
on 18 Mar 2019
Of course. I'll look into it when I get more time...
Tyrrrz
on 27 Mar 2019
Tyrrrz
on 10 Apr 2019
Nice and fast again!
terorie
on 10 Apr 2019
Just to clarify, it doesn't completely solve the problem, but it drastically reduces the impact. I will look into optimizing the exporting process even further in the future.
Tyrrrz
on 10 Apr 2019
Related issues
Latias4Ever
·
4Comments
chibiusagi
·
4Comments
makeworld-the-better-one
·
3Comments
fynnay
·
6Comments
billyon
·
3Comments