Easy-digital-downloads: 3.0 - Migration: Collection of improvements

[cklosowski]

This is a catch-all issue as I work through the migration routine to capture any necessary issues to fix. They will be included in a single issue to in order to allow us to test this easier and more quickly.

• [x] Pending orders are having a 'completed' date added to them.
• [x] WPCLI count was incorrect in the results, however, in my admin I have 101898 orders. I suspect that this is due to the new process of creating a 'refund' order for each refunded 'sale'.

Old Records: 101898
New Records: 104962

• [x] Remove any unnecessary calls to instantiate objects.
• [x] Multiple PHP notices and warnings related to array types, null items, etc.
• [x] Disable caching of the new DB class while migrating. - There is a real chance we could overwhelm a caching layer with all the new writes when migrating, so we should avoid writing every new item to the cache, knowing full well, the next item is just going to overwrite it again.
• [x] v30_migration method in the WPCLI class does not exist (as suggested by the admin notice about the upgrade)
• [x] Fix whitespace issue in order migrations for when refunds are present.
• [x] Fix SQL query in CLI to get all log entries (performance/failure)
• [x] Update child payment IDs with new parent order IDs

A few of these should help improve the performance of the migration as well, but if we need to we can do a deeper dive into the profiling of the migration to reduce the time to migrate.

[cklosowski]

On a fresh 2.9.x -> 3.0 migration I had no errors in the logs. Also we are already excluding the sales logs, so the count issue is coming from somewhere else.

[cklosowski]

Somewhere around 117,000 log entries, this fails in CLI as well. The query to get the logs starts to fail getting all the logs (likely due to the 3 JOINs that are being run). We'll need to find a way to optimize this.

[cklosowski]

Ok I've made a few improvments, going to run this through the EDD site data again and see how it does.

[cklosowski]

Just ran into a REALLY weird case that we'll have to account for I think.

It seems gravity forms, in some configuration (or bug, I'm not sure) used to create empty orders in EDD, for something as simple as a support form. Our old /contact-developer form submitted multiple Orders to EDD that are abandoned and have no products or gateways on them.

These were from around 2014 and earlier, from what I can see.

[cklosowski]

So I did a bunch of microtime logging, and over 100k orders, the first and last items were still processing in the sub 0.2 second range and a bulk of them sub 0.1 seconds. I'm not sure there is much more we can do to speed that up. At this point I'm down to testing the data validity as well as ridding the PHP error logs of entries for malformed or bad data.

[cklosowski]

One last PHP error to clean up. it only came up twice during our entire migration of the EDD site data locally:
[13-Mar-2019 17:04:59 UTC] PHP Notice: Undefined index: cart_details in /app/wp-content/plugins/easy-digital-downloads/includes/admin/upgrades/v3/class-data-migrator.php on line 375

Need to dig in and see if there is something I can do here to prevent this one.

[cklosowski]

I also still have a few weird orders that are 'pending' that are being brought in with a completed date, so I'll have to dig into those as well.

[cklosowski]

Ok the pending order issue is due to the value going into the DB as 0000-00-00 00:00:00 and when it comes out, it's getting set to now when parsed.

Also I noticed that all orders migrated over had a date_created as the time of the migration. Do we want to allow for this to be overridden so it can be the date the post was created from the custom post type?

@easydigitaldownloads/core-team

[pippinsplugins]

date_created definitely needs to be accurate per the original records.

[cklosowski]

@pippinsplugins I agree, and after diffing in, the ones having the date_created issue are ones that have 0000-00-00 00:00:00 listed as their post_date_gmt column in the original data. So that's why they are getting the date of the migration. One thing we could try to do to account for those, is see if this post_date_gmt column is the default value, and if so, estimate the post_date_gmt based off teh post_date by applying the difference in the post_modified_date and post_modified_date_gmt

[JJJ]

I agree with Pippin, that retaining the original data seems right here.

Curious though, why do the previous orders have empty post_date values? We should try to trap some obvious/fast conditions like these, but every condition in the iteration is the slower it’ll go.

[cklosowski]

@JJJ I'm not 100% positive. It's not all, just some throwing the data off, so i need to do a quick little bit to try and account for that case.

[cklosowski]

This should handle the logic to determine the Post Date GMT:
```
// Account for a situation where the post_date_gmt is set to 0000-00-00 00:00:00
$date_created_gmt = $data->post_date_gmt;
if ( '0000-00-00 00:00:00' === $date_created_gmt ) {

        $date_created_gmt  = new DateTime( $data->post_date );
        $modified_time     = new DateTime( $data->post_modified );
        $modified_time_gmt = new DateTime( $data->post_modified_gmt );

        $diff = $modified_time_gmt->diff( $modified_time );

        if ( 1 === $diff->invert ) {
            $date_created_gmt->add( new DateInterval( 'PT' . $diff->h . 'H' ) );
        } else {
            $date_created_gmt->sub( new DateInterval( 'PT' . $diff->h . 'H' ) );
        }

        $date_created_gmt = $date_created_gmt->format('Y-m-d H:i:s');
    }

````

[cklosowski]

Results with this logic:

$date_created      = '2019-02-16 09:19:38';
$date_created_gmt  = '0000-00-00 00:00:00';
$post_modified     = '2019-02-16 09:19:38';
$post_modified_gmt = '2019-02-16 15:19:38';


string '2019-02-16 15:19:38'

And then if we go for a positive GMT offset:

$date_created      = '2019-02-16 09:19:38';
$date_created_gmt  = '0000-00-00 00:00:00';
$post_modified     = '2019-02-16 09:19:38';
$post_modified_gmt = '2019-02-16 01:19:38';

string '2019-02-16 01:19:38'

[cklosowski]

Also we'll have to account for minutes since you can have a .5 offset.

[mintplugins]

Just a note that Carbon was changed to \Carbon\Carbon, which fixed a migration error for me where a customer had notes attached.

Ultimately this will be changed to EDD\Utils\Date, but I haven't had a chance to re-run the migration with that yet.

[pippinsplugins]

I've run a test migration and everything appeared to come through properly!

I need to test again with a store that has refunded orders.

[mintplugins]

Just ran another migration and had this failure on a few discount codes, which caused the migration to get stuck.
<strong>WordPress database error:</strong> [You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near &#039;WHERE meta_key = &#039;legacy_discount_id&#039; AND meta_value = 29811&#039; at line 1]<br /><code>SELECT edd_discount_id FROM WHERE meta_key = &#039;legacy_discount_id&#039; AND meta_value = 29811</code></p></div><div id="error"><p class="wpdberror">

The discount itself (29811) was a post with the post_type edd_discount.

The only other place I could find 29811 in the original (pre-migrated) database was in the wp_postmeta table under _edd_payment_meta with the following value.
I've removed the PID of this customer here on GitHub, but the length of the strings are the same.

a:15:{s:6:"amount";s:2:"79";s:4:"date";s:19:"2012-08-09 16:16:00";s:5:"email";s:28:"[email protected]";s:3:"key";s:32:"33b8d01d6e1dec0fa014e529811dc6a1";s:8:"currency";s:3:"USD";s:9:"downloads";s:51:"a:1:{i:0;a:2:{s:2:"id";i:952;s:7:"options";a:0:{}}}";s:9:"user_info";s:179:"a:5:{s:2:"id";i:2091;s:5:"email";s:28:"[email protected]";s:10:"first_name";s:4:"xxxx";s:9:"last_name";s:12:"xxx xxxxxxxx";s:8:"discount";s:21:"BUY1G2FREE (-$158.00)";}";s:12:"cart_details";s:48:"a:1:{i:0;a:2:{s:2:"id";i:952;s:5:"price";d:79;}}";s:7:"user_id";s:4:"2091";s:7:"address";s:0:"";s:9:"address_2";s:0:"";s:5:"state";s:0:"";s:4:"city";s:0:"";s:3:"zip";s:0:"";s:7:"country";s:0:"";}

[JJJ]

SELECT edd_discount_id FROM WHERE

Looks like the database table is missing. It should be between the FROM and WHERE.

If we could narrow down that call, and use wp_debug_backtrace_summary() on it, that would be awesome.

[mintplugins]

@JJJ Which table should I be looking for?

[JJJ]

Which table should I be looking for?

I did a quick search for legacy_discount_id and my best guess is it's inside the find_legacy_id() method.

edd_discountmeta is now edd_adjustmentmeta, so I think that needs to be changed.

[mintplugins]

I was able to migrate the Mint Themes database with https://github.com/easydigitaldownloads/easy-digital-downloads/commit/cf03cc215bc3cec75f634bd4bd4545fa8716f9ea in place. 🥁🎉

[mintplugins]

I have found a couple of issues post migration:

• All order IDs are blank in the new order_id column
• Products set to "draft" during migration result in orders with no products attached. The use case here is that I used to sell a product that I no longer do. Payments with that product exist, but I have the product itself set to "draft".

[cklosowski]

@mintplugins if these two items above are still the case, can we get new issues created?