Signal-desktop: Standalone-App stops working (crash) during import of backup

[Wasseranomalie]

• [x] I have searched open and closed issues for duplicates

---

Bug description

Created first backup with the chrome-extension v0.44.1 and tried to import it into the new standalone-applikation. It crashed during the procedure resulting in the "something went wrong during the import"-screen after starting the applikation again. Unfortunately I can't see anything useful in the debug-log, but I'll post it anyway. Neither updating the chrome-extension to 0.44.2 and backing up again nor reinstalling the standalone-applikation did solve the issue. (It seems that deinstalling chrome v1.x does not clear the applikation-data, on first start after installing it I directly landed at the "something went wrong"-screen again)

Steps to reproduce

• Create backup with 0.44.2
• Import backup with new v1.0.34
• Wait for import to complete
• (I know, something else must be different since it seems to work for many users)
  Actual result: Crash after some time waiting for the import to complete
  Expected result: Import successful

Screenshots

Platform info

Operating System: Windows 10 64Bit
Browser: Electron v??

Signal version: 1.0.34

Link to debug log

https://gist.github.com/anonymous/bc476b51311a60eda9581293bd81c575
(Latest crash seems to be after timestamp 2017-11-01T23:14:01.349Z )

Edit: Debug-log from export:
https://gist.github.com/anonymous/17debc9137a18772b395376eff39e2b6
According to the extension the backup was successful (not like in #1660 where the export seems to fail)
(Latest backup starts at timestamp 2017-11-01T23:34:43.739Z )

[scottnonnenberg]

On of your conversations was extremely large. When you next try to do the import, watch the CPU and memory usage in process manager. I suspect that the application is having trouble with a conversation of that size.

[Wasseranomalie]

The memory-usage right before the crash is about 3700MB (2 processes combined), the CPU usage on my Laptop about 90%, on desktop about 20%

[mlsxlist]

@scottnonnenberg after #1616 was fixed with 0.44.7 for me I am now running into this error on import with 1.0.36. Thanks for your great support. In the terminal I see:

{"name":"log","hostname":"myhost","pid":9552,"level":30,"msg":"Done importing 4110 messages for conversation [REDACTED]~`³","time":"2017-11-08T05:25:18.490Z","v":0}
tcmalloc: large alloc 1157627904 bytes == 0x11ee56007000 @ 
[9552:1108/052519.367096:FATAL:memory_linux.cc(35)] Out of memory.
#0 0x00000342c4ee <unknown>
#1 0x00000343f06b <unknown>
#2 0x000003411ade <unknown>
#3 0x0000033a5f24 operator new()
#4 0x7f5c16535e99 std::string::_Rep::_S_create()
#5 0x7f5c16536b0b std::string::_Rep::_M_clone()
#6 0x7f5c16536bb0 std::string::reserve()
#7 0x7f5c16536e3a <unknown>
#8 0x7f5c16536f0e std::string::append()
#9 0x000001ef1ffa <unknown>
#10 0x00000156fc63 <unknown>
#11 0x0000015731be <unknown>
#12 0x0000017ad2d7 <unknown>
#13 0x0000017d50c8 <unknown>
#14 0x0000017d4f43 <unknown>
#15 0x0000017ac92f <unknown>
#16 0x0000017d58dd <unknown>
#17 0x0000016c6d97 <unknown>
#18 0x000001659a1b <unknown>
#19 0x000001dca1f1 <unknown>
#20 0x00000342093e <unknown>
#21 0x000003442bac <unknown>
#22 0x000003442e58 <unknown>
#23 0x0000034431bb <unknown>
#24 0x00000342539a <unknown>
#25 0x000003442947 <unknown>
#26 0x0000033fe170 <unknown>
#27 0x000003403565 <unknown>
#28 0x000003432443 <unknown>
#29 0x7f5c1c866064 start_thread
#30 0x7f5c15c9d62d clone

Aborted

[scottnonnenberg]

@mlsxlist Would you be willing to provide the full log? Please also talk about how the memory usage of the process changes as the import progresses.

One other thing comes to mind - do you have any particularly large attachments in your export?

[mlsxlist]

@scottnonnenberg please see the log here:

https://gist.github.com/55a0cae2b386d9ea6b121989f0b42885

I observed the memory during import. During import the overall memory usage of the signal processes goes up to 4-5 GB before it crashes.

For me it is not clear which conversation is imported at the time of the crash. The larges attachments I see the Signal folder from where I import the message are videos with around 85 MB.

[scottnonnenberg]

@mlsxlist Thanks. One thing to try is to update your Chrome app to 0.44.10, released today, export again then see if that import works. We do some extra work to reduce the on-disk impact of messages which failed to send.

[mlsxlist]

@scottnonnenberg when I export again with 0.44.10 the export never ends and signal hangs on the _Please wait while we export your data..._ screen. In the log I see

https://gist.github.com/mlsxlist/6da32f9c4032d03b846c84b09468d943

[scottnonnenberg]

0.44.11 has been out for most of today - what do you see when you export with that version?

[mlsxlist]

Thanks. Export worked with 0.44.11. Did not see the update.

[mlsxlist]

@scottnonnenberg I get the same out of memory error as before. I observed the memory and did the import after a reboot of the laptop with 16GB of physical memory. With signal-desktop running there was 13GB of free memory. When I started the import I could see memory going down from 13GB to 0. signal-desktop ate up all memory until the process was terminated by the OS.

The folder that holds the exported data from the chrome app 0.44.11 has a size of 3,4 GB. Strange that signal-desktop needs a factor 3 more memory for the import.

https://gist.github.com/ac8890909dd3e4b2dc1259e4855fc2bf

[scottnonnenberg]

@mlsxlist Can you take a look at all the .json files in your export directory? How big is the largest one?

[mlsxlist]

@scottnonnenberg The largest .json file is db.json with a size of 4MB. The largest messages.json has a size of 2.6MB and is a group conversation.

[scottnonnenberg]

@mlsxlist Hm. We should definitely be able to load those files no problem. Can you also find your largest attachment?

[mlsxlist]

@scottnonnenberg the largest attachment is a mp4 file with a size of 84.8 MB

[scottnonnenberg]

Signal Desktop 1.0.38 was released today. It has improvements for large-scale import. Take a look!

[mlsxlist]

@scottnonnenberg thanks. Also 1.0.38 runs out of memory for me.

{"name":"log","hostname":"myhost","pid":9567,"level":30,"msg":"Done importing 4110 messages for conversation [REDACTED]~`³","time":"2017-11-15T05:24:38.418Z","v":0}
{"name":"log","hostname":"myhost","pid":9567,"level":30,"msg":"messages done saving via puts scheduled","time":"2017-11-15T05:24:38.419Z","v":0}
tcmalloc: large alloc 1157627904 bytes == 0x34fc3c8de000 @
[9567:1115/052439.097854:FATAL:memory_linux.cc(35)] Out of memory.

0 0x00000342c4ee (/opt/Signal/signal-desktop (deleted)+0x342c4ed)

1 0x00000343f06b (/opt/Signal/signal-desktop (deleted)+0x343f06a)

2 0x000003411ade (/opt/Signal/signal-desktop (deleted)+0x3411add)

3 0x0000033a5f24 (/opt/Signal/signal-desktop (deleted)+0x33a5f23)

4 0x7fbb3c23ee99 std::string::_Rep::_S_create()

5 0x7fbb3c23fb0b std::string::_Rep::_M_clone()

6 0x7fbb3c23fbb0 std::string::reserve()

7 0x7fbb3c23fe3a

8 0x7fbb3c23ff0e std::string::append()

9 0x000001ef1ffa (/opt/Signal/signal-desktop (deleted)+0x1ef1ff9)

10 0x00000156fc63 (/opt/Signal/signal-desktop (deleted)+0x156fc62)

11 0x0000015731be (/opt/Signal/signal-desktop (deleted)+0x15731bd)

12 0x0000017ad2d7 (/opt/Signal/signal-desktop (deleted)+0x17ad2d6)

13 0x0000017d50c8 (/opt/Signal/signal-desktop (deleted)+0x17d50c7)

14 0x0000017d4f43 (/opt/Signal/signal-desktop (deleted)+0x17d4f42)

15 0x0000017ac92f (/opt/Signal/signal-desktop (deleted)+0x17ac92e)

16 0x0000017d58dd (/opt/Signal/signal-desktop (deleted)+0x17d58dc)

17 0x0000016c6d97 (/opt/Signal/signal-desktop (deleted)+0x16c6d96)

18 0x000001659a1b (/opt/Signal/signal-desktop (deleted)+0x1659a1a)

19 0x000001dca1f1 (/opt/Signal/signal-desktop (deleted)+0x1dca1f0)

20 0x00000342093e (/opt/Signal/signal-desktop (deleted)+0x342093d)

21 0x000003442bac (/opt/Signal/signal-desktop (deleted)+0x3442bab)

22 0x000003442e58 (/opt/Signal/signal-desktop (deleted)+0x3442e57)

23 0x0000034431bb (/opt/Signal/signal-desktop (deleted)+0x34431ba)

24 0x00000342539a (/opt/Signal/signal-desktop (deleted)+0x3425399)

25 0x000003442947 (/opt/Signal/signal-desktop (deleted)+0x3442946)

26 0x0000033fe170 (/opt/Signal/signal-desktop (deleted)+0x33fe16f)

27 0x000003403565 (/opt/Signal/signal-desktop (deleted)+0x3403564)

28 0x000003432443 (/opt/Signal/signal-desktop (deleted)+0x3432442)

29 0x7fbb4256f064 start_thread

30 0x7fbb3b9a662d clone

Aborted

[mlsxlist]

@scottnonnenberg after a reboot to make sure that as much free memory as possible is available the import succeeded. Thanks, Scott for you help.

[scottnonnenberg]

@Wasseranomalie It seems, in another thread, that you were able to import successfully. I'm going to close this bug, happy to re-open it if anyone else experiences this.