Discordchatexporter: Unexpected exception when calling ResolveUrl

[deyab]

This error only happens with --media switch

To test, join wsb discord; here's an invite: https://discord.gg/wallstreetbets

It's very busy server, so I am trying to download by quarters. Something like:
.\DiscordChatExporter.Cli.exe export -t "your_token" -c 615491272836251658 -o "C:\discord_export" -f HtmlDark --after "2019-10-01 00:00" --before "2020-01-01" --media

it throws an error:
Exporting channel 'trading / snp500-spx-es-spy-only'... 13.85 %<input>(137,89) : error : Unexpected exception when calling ResolveUrl

I do get the same error if I am running the GUI version as well. Thanks!

[Yudi]

Have you updated to 2.21.1?

[deyab]

Yes. Using v 2.21.1 on Windows 8.1 x64

C:\tools\DiscordChatExporterCLI_v2.21.1>dotnet --info
  It was not possible to find any installed .NET Core SDKs
  Did you mean to run .NET Core SDK commands? Install a .NET Core SDK from:
      https://aka.ms/dotnet-download

Host (useful for support):
  Version: 3.1.5
  Commit:  65cd789777

.NET Core SDKs installed:
  No SDKs were found.

.NET Core runtimes installed:
  Microsoft.AspNetCore.App 3.1.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 3.1.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.WindowsDesktop.App 3.1.5 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

[Yudi]

I'm able to go past 13.85%
I'll wait for the export to finish

[Yudi]

[deyab]

¯\_(ツ)_/¯

I ran it again, and it finished for me as well. Thanks for the help. I will try a couple other time period and if it goes well, I will close the issue. Might have been some transient network issue at that time because I tried thrice before logging the bug,

[deyab]

Is there a debug flag? To increase verbosity? Or some log that I can look at? I am persistently getting this error for other time periods, and knowing what failed would be helpful.

I will look into this more... try the download later, in different chunks etc. One of the issue is the size of the downloads. The server got so busy lately 15 days worth of logs are approaching 0.5 GB FWIW (including the media)

[deyab]

Unfortunately, the issue persists, and since I can't find any debug options (short of compiling the code myself and looking through the code) I will leave this open for now. I tried to export another channel which is much lighter in traffic, and got the same error. That's a private server, and although I can add people, I don't have the explicit permission of the channel owner.

On my side, I am on Comcast 100mb download plan, very stable connection.

Going back to the wsb server, this is what I got, over 24 hours:

command:
C:\tools\DiscordChatExporterCLI_v2.21.1>.\DiscordChatExporter.Cli.exe export -t "a_token" -c 615491272836251658 -o "C:\tools\DiscordChatExporterCLI_v2.21.1" -f HtmlDark --after "2020-04-01 00:00" --before "2020-07-01" --media

Each output is from separate invocation:
Exporting channel 'trading / snp500-spx-es-spy-only'... 17.29 %<input>(127,101) : error : Unexpected exception when calling ResolveUrl

Exporting channel 'trading / snp500-spx-es-spy-only'... 49.59 %<input>(127,101) : error : Unexpected exception when calling ResolveUrl

Exporting channel 'trading / snp500-spx-es-spy-only'... 47.01 %<input>(128,101) : error : Unexpected exception when calling ResolveUrl

Exporting channel 'trading / snp500-spx-es-spy-only'... 49.59 %<input>(127,101) : error : Unexpected exception when calling ResolveUrl

Exporting channel 'trading / snp500-spx-es-spy-only'... 49.59 %<input>(127,101) : error : Unexpected exception when calling ResolveUrl

Exporting channel 'trading / snp500-spx-es-spy-only'... 49.59 %<input>(127,101) : error : Unexpected exception when calling ResolveUrl

From the other, much, much less traffic channel:

C:\tools\DiscordChatExporterCLI_v2.21.1>.\DiscordChatExporter.Cli.exe export -t "a_token" -c channel_id -o "C:\tools\DiscordChatExporterCLI_v2.21.1" -f HtmlDark --media
Exporting channel 'Text Channels / general'... 75.15 %<input>(127,101) : error : Unexpected exception when calling ResolveUrl

C:\tools\DiscordChatExporterCLI_v2.21.1>.\DiscordChatExporter.Cli.exe export -t "a_token" -c channel_id -o "C:\tools\DiscordChatExporterCLI_v2.21.1" -f HtmlDark --media
Exporting channel 'Text Channels / general'... 75.15 %<input>(127,101) : error : Unexpected exception when calling ResolveUrl

[Tyrrrz]

It should definitely ignore HTTP errors, so that implies something else. Looking at it, it seems to crash consistently around 49.59%.
Can you try running with something like --after "2020-05-15" --before "2020-07-01" (approx. half the range) and see if it crashes in the beginning as well?

[deyab]

The second private server is much easier to deal with because of lighter traffic. It always abort at a very specific spot. Everything is exported before someone posting a message with this link. And the url loads if I click on it

I will look into the staggered export of the wsb server as you suggested a bit later in the day. Thanks!

[Tyrrrz]

Can you copy-paste the link here?

[deyab]

The link is nothing special I think, but here it is:
https://medium.com/macoclock/we-need-to-talk-about-airpods-pro-4bbd2533e031

[deyab]

Try this. Quick & consistent failure:

C:\tools\DiscordChatExporterCLI_v2.21.1>.\DiscordChatExporter.Cli.exe export -t "a_token" -c 615491272836251658 -o "C:\tools\DiscordChatExporterCLI_v2.21.1" -f HtmlDark --after "2020-02-15 00:00" --before "2020-02-16" --media
Exporting channel 'trading / snp500-spx-es-spy-only'... 15.52 %<input>(127,101) : error : Unexpected exception when calling ResolveUrl

[Tyrrrz]

Try this build:

https://github.com/Tyrrrz/DiscordChatExporter/actions/runs/182487470

[deyab]

Been testing it out. Flawless. Thanks so much!