Easy-digital-downloads: File download log migration failing (GDPR)

[mintplugins]

Bug Report

Expected behavior

When you do an upgrade for https://github.com/easydigitaldownloads/easy-digital-downloads/issues/6548, it should be batched and work for any number of download logs.

Actual behavior

A few customers with stores containing a large number of file downloads attempt to do the upgrade routine, it fails with this error message:

PHP Fatal error: Allowed memory size of
> 134217728 bytes exhausted (tried to allocate 133883136 bytes)

Steps to reproduce the behavior

1) Create a store with a large number of download logs (like 80,000) on version 2.9.1 or earlier.
2) Attempt to do the upgrade routine from https://github.com/easydigitaldownloads/easy-digital-downloads/issues/6548
3) It should fail with the error message above.

Information (if a specific version is affected):

EDD Version (or branch): 2.9.1 being migrated to 2.9.2

[boluda]

Same problem trying to reset the shop and running other recounting tools, similar to #3633

[mintplugins]

I believe this is caused by this query, which seems to query every single download log, and attempts to store them in a single wp_option. For this customer in particular, that would be a query of over 80,000 logs.

https://github.com/easydigitaldownloads/easy-digital-downloads/blob/5e9baeaddb3e093a51669cb2fffbe0e52d8d2d46/includes/admin/upgrades/classes/class-file-download-log-migration.php#L193-L208

We should reduce that, and instead of querying for every log on the first load, we should only query for 50 at a time, possibly using the wp_query offset.

[mintplugins]

This is ready for testing. In order to test you'd have to test it in a store that hasn't done the upgrade yet.

For our affected customers, they weren't able to do the upgrade so for them they can do the upgrade still (because the nag still shows in the admin notice).

P.S. another affected customer here:
https://secure.helpscout.net/conversation/618087125/85686/

[pippinsplugins]

@JJJ @mintplugins I'm getting an SQL error when running this:

WordPress database error You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'AND taxonomy = 'edd_log_type'' at line 1 for query SELECT term_taxonomy_id FROM wp_17_term_taxonomy WHERE term_id = AND taxonomy = 'edd_log_type' made by do_action('wp_ajax_edd_do_ajax_export'), WP_Hook->do_action, WP_Hook->apply_filters, edd_do_ajax_export, EDD_File_Download_Log_Migration->pre_fetch, QM_DB->query

[pippinsplugins]

Note: this is for a site that has zero download logs.

[JJJ]

@boluda You're correct about that.

We've made a few improvements to this (in what will become EDD3.0) but there is still more to do. The per_step approach here belongs in the base class so all extensions can follow suit, but that's outside the scope of this specific issue unfortunately, and probably deserves its own.

[boluda]

Thanks @JJJ :)
Should I create another issue? It's not a priority to reset the shop, but I like to recalculate the stats every couple of months, to get more accurate reports.

[JJJ]

@boluda if you don't mind, that would be really helpful actually. Otherwise, @sunnyratilal or I will get around to it.

Related: if Sunny and I did our jobs right, reports in EDD3.0 will require a lot less (hopefully none!) manual recalculation.

[boluda]

Sure thing, @JJJ I'm on it, creating the issue right now.

Related: I'm sure you are doing a GREAT job in EDD3.0, big fan of your work (and documentation), happy you joined Pippin's team.