Monica: Unable To Export Data

[pqhf5kd]

Describe the bug
I'm trying to export my data (just to keep a backup) using the Export to SQL button from the hosted MonicaHQ instance but I'm just getting a blank page. Some weeks back I was getting an error but I didn't submit a ticket at the time.

Screenshots
Clicking the button beings to load https://app.monicahq.com/settings/exportToSql, it finishes and literally a blank page is returned (no source code)

[mattdp]

@asbiin I am experiencing this issue as well, and it makes me nervous about continuing to put data into the system. Thank you for all your help building Monica!

[DieselDuz42]

I, too am experiencing this issue.

I dropped an email on Sept 9th and have gotten no response.

10/17/19
Still, no response ended up deleting all of my contacts in the system to unsubscribe.

[TomGranot]

This is fixed by now, at least on 2.4.* and 2.5.* which I ran today.

[mattdp]

@tomgs I don't believe this is working - I tried on PC Firefox and Mac Chrome. I have 2K+ contacts so I would guess it's timing out for people with large databases and working for smaller ones.

@asbiin I'd be happy to investigate further - I've done a bit of work with databases in the past, but know very little about PHP / how projects are structured - can you point me in the right direction as to where this function is located in the codebase, and I'll try to do some research from there?

[TomGranot]

@mattdp Ah, sorry about that, my database is indeed smaller than yours.
Per your question:

The route that specific settings pages sits on is /settings/export, and it is redirected in the router here:
https://github.com/monicahq/monica/blob/c90354c07aa1e302405d6e224c4281e419fa2f09/routes/web.php#L238

The settings controller is defined here:
https://github.com/monicahq/monica/blob/10e73e35b6160bf8a427ddfabb958970297cfb45/app/Http/Controllers/SettingsController.php#L36

If I'm not mistaken, than this is the thing that actually triggers the correct view inside that file:
https://github.com/monicahq/monica/blob/10e73e35b6160bf8a427ddfabb958970297cfb45/app/Http/Controllers/SettingsController.php#L236-L243

This is that view:
https://github.com/monicahq/monica/blob/master/resources/views/settings/export.blade.php#L1-L51

And this is the button that triggers what is eventually giving you an empty page:
https://github.com/monicahq/monica/blob/master/resources/views/settings/export.blade.php#L41

If I'm not mistaken, this is where the export gets triggered:
https://github.com/monicahq/monica/blob/684ce6bc56e7bed079bdf8b1fdacb352e937e964/app/Console/Commands/ExportAll.php

It relies on this class to do its thing:
https://github.com/monicahq/monica/blob/684ce6bc56e7bed079bdf8b1fdacb352e937e964/app/Jobs/ExportAllAsSQL.php

But I can't really pinpoint the failing point without your input, since for me it's seemingly working.

[mattdp]

Thank you so much, @tomgs , for taking your time to outline this to me! I will check this out when I have a few free hours.

[TomGranot]

My pleasure @mattdp. I'll actually be attending an Hacktoberfest later this week (Wedensday) so if the hours match up we might be able to hack on it together. I'd really like to see this fixed, as this is one of the most awesome things about Monica - their SQL export actually exports all the things.

[mattdp]

TLDR: @djaiss @asbiin - within five seconds of submitting this comment I started an export on the hosted site. Odds are that's the only export with a close timestamp. Any chance you could send us whatever logging you get from that export failing, so we can move towards a fix during Tom's Hacktoberfest time Wednesday?

--

Very cool @tomgs - want to let me know by emailing monicaissue at matthewfdupont.com what time/time zone you'll be hacking to see if we can sync?

Since I use the hosted version, I'm wondering what if anything I can do short of spinning up an instance of self-hosted with a few thousand contacts to help diagnose if the issue. I'd guess, as I'm sure you've already considered, that either

a) the export function you highlighted errors out somewhere along the way for something memory- or variable-size related
b) the export function you highlighted has a max query time that fails, or
c) the function works fine but the page returns before it's complete, and there's no way to hand off a payload that's a few seconds late to the frontend.

But I haven't been a dev for a few years so might be missing something obvious here.

[djaiss]

It doesn't work in production indeed.
I've been trying to understand what's going on exactly, but I fail. I think I will have to put a lot of traces in place to see what's going on here. It works locally and in small instances, but at Monica's size, it just goes right into a 500 after a while.

I'll push a new PR soon with logs everywhere to understand this problem better. Sorry for the problems.

[asbiin]

Hi @mattdp , and thank you @tomgs for your expertise!
As @djaiss just mentionned, we are aware of the situation, but it's mindblowing right now as there is no error log at all. We are investigating...

[mattdp]

Thanks @djaiss and @asbiin , please let us know when the trace is in place—if you have any logs you can send about the issues once those traces are in place, I'm happy to try and dig a bit on what they mean in terms of changes needed.

I really appreciate the attention to this issue! It makes me much more comfortable building out long-term workflows relying on Monica when export works.

[djaiss]

I've added some logs in production to understand what's going on.
So the problem in production is still unknown, but at least we see that it’s the contacts table that causes the problem. The script stops at the contacts table somehow. Going to investigate more.

[TomGranot]

Hola! Awesome to see all the attention around this. Hope this pans out!

@Mattdp, will contact.

[TomGranot]

Well, Drip died on the address you gave me, but I'll be on this on
Wednesday, circa 2100 or 2200 GMT+2. No worries if that doesn't work for
you;) I'd stress test it locally with dummy contacts.

[TomGranot]

Ah, just noticed it's an actual email there in your comment! I just went directly to the website. Emailing now.

[djaiss]

The export process has been refactored and has been merged, and finally it works in production again. I hope it works for all users too. My vision is that exporting data is essential and core to the product.
We'll tag a new version soon.

[mattdp]

@djaiss just checked and it works for me - thank you so much @djaiss ! I feel way more comfortable investing time in fleshing out my data in Monica with that in place, and appreciate all the effort to diagnose.