Easy-digital-downloads: Move CSV Import Functionality to Core

[SDavisMedia]

After discussing it in the team meeting and chatting with @pippinsplugins, this functionality is going to shift from extension to EDD core. This is where it begins. :+1:

[pippinsplugins]

This should include the following:

• payment import
• product import
• customer import (maybe)

Along with this introduction, I think we should make export and import from EDD site A to EDD site B seamless. If export a purchase history file from one site, I should be able to directly import it into another site without hassle.

[pippinsplugins]

I'm going to start working on this.

[pippinsplugins]

This is now up to date with the latest release/2.6

[pippinsplugins]

I have the base product import working well. Now just need to go through and handle all of the field mapping.

[pippinsplugins]

Ok this could use some initial testing for the product import section.

For testing, please start with a product export from EDD and then import that. Once that's working well, we can move to other file structures too.

Some notes:

1. UI is still rough.
2. Only product import is functional.
3. Automatically mapping CSV columns to product fields is not yet supported so for now all fields have to be mapped manually.
4. Category and tag fields are not working yet and am not sure why.
5. Featured image doesn't work yet.
6. Custom fields don't work yet.

[SDavisMedia]

My initial test with two sites on issue/3620 is being met with this on a white screen during the import:

All I'm doing is exporting CSV from EDD core, uploading the CSV file for import, and when I click the Import button, that's what I get.

[SDavisMedia]

Scratch that... it worked from my live site to my local. No clue why it did that from local to live.

Still testing... more to follow.

[SDavisMedia]

Ignoring the fields you mentioned above, the import worked perfectly! No issues at all and all the product meta is correct. 🎉

Would be cool to have a link to the downloads table here: http://glui.me/?i=gehkxgpydo4xiqs/2016-05-10_at_9.16_PM.png/

[SDavisMedia]

Actually, it looks like Author -> Product Author didn't work.

[pippinsplugins]

@SDavisMedia I think that white screen might have been due to a JS file being cached.

[pippinsplugins]

Import complete message now includes a link to the relevant list table.

[michaelbeil]

It looks like we should add the Needs Testing label here and also create a PR for issue/3620(bring branch up to date) against release/2.6.

https://github.com/easydigitaldownloads/easy-digital-downloads/compare/release/2.6...issue/3620

[pippinsplugins]

It does need testing but it's also not nearly done, thus why I hadn't added it :)

[pippinsplugins]

Featured image import now works. Images can be specified by:

• Filename.ext - File must exists somewhere inside the wp-content/uploads/ directory for this to work. Can be in any sub folder.
• http://site.com/wp-content/uploads/2016/05/filename.ext - URL pointing to file in uploads folder.
• http://some-external-site.com/file.ext - An external image file that will be downloaded to the WP media library

[pippinsplugins]

The product import is ready for another round of testing!

[SDavisMedia]

Getting the same behavior as this again: https://github.com/easydigitaldownloads/easy-digital-downloads/issues/3620#issuecomment-218356870

I'm exporting from a local MAMP site to a _brand new_ local MAMP site. Literally nothing has been done to the new site previously.

[pippinsplugins]

@SDavisMedia Force reload the page. It's a JS caching issue (as far as I can tell).

[SDavisMedia]

@pippinsplugins JS cached where? It could only be browser and that's been cleared. Trying to import to a site that was just created about an hour ago.

Could the data itself cause this? I'm attaching my compressed CSV file.
edd-export-downloads-05-25-2016.csv.zip

[pippinsplugins]

Do you have SCRIPT_DEBUG on?

The CSV file shouldn't cause that. It's happening because the browser is loading the uploaded file instead letting the JS read it. So far I've only seen that when the browser is loading a cached version of the JS file.

[SDavisMedia]

It was definitely because of SCRIPT_DEBUG not being true. No clue why there'd be a cached file from this new site.

[pippinsplugins]

Browsers cache asset files even without caching plugins active.

[SDavisMedia]

Got it.

For some reason, I cannot get the system to import more than 19 products. @michaelbeil experienced the same issue but he was using my CSV file. I've deleted some downloads and added new ones on the export site, keeping the total over 19. Tried a new export/import and it stops at 19 again.

Here's a sample CSV with 20 downloads:
20-items.csv.zip

[michaelbeil]

With a new set of 20 downloads in a CSV, only 19 were imported.

[pippinsplugins]

@SDavisMedia @michaelbeil could you try again please?

[pippinsplugins]

Validation of empty / non-CSV files has been restored.

[pippinsplugins]

Payment importing now works!

[SDavisMedia]

@pippinsplugins The issue with only importing 19 products is now fixed! :+1:

Not sure why no mapping is happening for me.

[pippinsplugins]

No mapping? Could you elaborate?

[SDavisMedia]

@pippinsplugins So far today, my fields are not automatically mapping when I import the CSV. For each test I've done, I'm setting them all manually.

The exported CSV is coming from a site also running issue/3620 but not the most recent changes from the last 2 hours. I'll pull down on the export site again to see if that makes a difference.

[pippinsplugins]

Ah, the magic mapping. That's not fully set up yet.

[SDavisMedia]

Okay all good [as far as we can be] with product exporting/importing at the moment.

---

Payment and Customer exporting:

With both of them, I found myself thinking I _had_ to choose a date range (Payment export) and make a Download section (Customer export) in order to get the CSVs. Obviously, that wouldn't work to get all data in one CSV for both.

I only realized I could get all the data in both by just trying without using those filters. It worked, of course, but I can see people being stuck there. So that's a UI thing we may need to be more clear on.

Payment import:

Honestly had no idea that the Payments export and import functionality would be capable of creating my customer records as well. I thought I had to handle importing customers on my own. So that worked. However, I'll have to test again because secondary email addresses (from the new feature) on my customer records from the export site were considered new customers on the import site. New customer records were created for those email addresses. I guess that makes sense for the overall 2.6 release, though.

[pippinsplugins]

Exporting: for now, these will probably be opened as separate issues. I want to address each of them but probably not for beta 1 tomorrow. Will try for 2.6 still though. We need to first figure out _exactly_ which changes need to be made.

Import: yep! It will create customers along the way :) That won't work until we have everything merged into release/2.6 since some of the code is still in separate issue branches.

[SDavisMedia]

@pippinsplugins Cool cool! Outside of that, everything was pretty smooth. :+1:

[pippinsplugins]

Great!

[michaelbeil]

All works well here after a git pull and a product import. I can now import over 19 products. We should be good to go for beta 1 tomorrow.

Export/Import UI referenced by @SDavisMedia above could use some improvements.

[pippinsplugins]

A few items I know we haven't finished yet:

• [x] Help text for the payment importer after uploading the file
• [x] Links to Import / Export tools page from the Reports > Exports page
• [x] Variable price support for payment import
• [x] Something funky is going on with payment keys. The keys appear to work fine but searching for a key after import fails
• [x] Some UI love from @SDavisMedia

[pippinsplugins]

Also need:

• [ ] some form of error logging to report failed imports on a per-row basis

[SDavisMedia]

While testing Payments export/import, Products (raw) doesn't appear to be working properly. Some Products are being created upon import with names like Product Name / Product Name / Product Name. http://glui.me/?i=ugkcvhqymvgarls/2016-05-26_at_12.44_PM.png/

Here's my CSV if you need to take a look.
payments-may-26-1212.csv.zip

[michaelbeil]

Ran the same test with @SDavisMedia's CSSV and ran into the same issues.

[pippinsplugins]

Tweaks to make:

1. Add alternating row colors
2. Add some padding to select menus
3. Add help text (see #4641 )
4. Support CSVs without header rows (currently fails)
5. Show first row value next to each column for CSV Column
6. Show which fields are required
7. Put product field on left, CSV on right
   8, Make product fields fixed. No need to select the field.
8. Test with RTL

[pippinsplugins]

Adjusted import UI:

It's functional for both product and payment import.

[pippinsplugins]

In issue/3620

[pippinsplugins]

RTL works:

[michaelbeil]

I think it might be good to bold the column titles "Payment Field" and "CSV column".

[pippinsplugins]

Live data previews are in!

Please test in issue/3620 branch.

[SDavisMedia]

I'll give feedback as I go. First thing I notice on the Download import is this setting:

I definitely think the label needs tweaking. Something along the lines of "Check this box if blah blah." Blah blah being the least likely scenario for CSVs and header rows. For example, if you suspect that 80% of CSVs will have a header row, then "Check this box if your CSV file does not have a header row."

[cklosowski]

@SDavisMedia I can add a tooltip once this is ready and merged and I go through and add the helpful data to it.

[SDavisMedia]

Yup! And Pippin also stated that it will be checked by default as well. So all good there!

[michaelbeil]

Looking good here.

[SDavisMedia]

Decimal places on on Total Earnings preview data could be trimmed:

[SDavisMedia]

Just finished a product export/import and wow... what a smooth process. Super intuitive and the visual feedback from the Data Preview column is golden. That was amazingly easy. Two thumbs up.

Gonna do payments now.

[SDavisMedia]

Payment export/import was smooth as well. There was a little confusion on some of the mapping. I'm sure it'll improve with tool tips but just to keep record, I'll explain below. This was a CSV from EDD core.

---

Starting the list with Currency Code and me not having anything for it was discouraging right out of the gate. I lost a little confidence in my ability to map properly (and that the import would work) before I even got to make a successful match. Just something to think about for the inexperienced users.

I didn't map Customer ID. Wasn't sure what it should be.

Why aren't IP Address and Mode in the export?

Payment Number will need a strong tool tip since it can be a number of things and _sometimes_ match the Parent Payment ID itself.

No clue what Subtotal would be but as a user, it's not something I was comfortable ignoring considering it's extremely relevant to payment data. I had to ignore it, though, because there was nothing to map it to. Confidence killer.

I mapped "User" to User ID but I'm pretty sure that was wrong. Seems that should be in the CSV export file.

[pippinsplugins]

1. Currency code: added to payment export.
2. Customer ID. This one is tricky. Most of the time it should not be mapped, but sometimes it needs to be. See https://github.com/easydigitaldownloads/easy-digital-downloads/issues/4625. The main place it is needed is when going from EDD to EDD and customers are being imported that have multiple email addresses.
3. IP added to export.
4. Payment number: :+1:
5. Subtotal. This will be helped by a tool tip. It would be the purchase total before taxes usually.
6. User ID can be an email, username, or actual user Id.

[SDavisMedia]

Payment import affected by https://github.com/easydigitaldownloads/easy-digital-downloads/commit/195455b4d92c719114cdcc99efd82931fe45f75e

Haven't tested yet but I'd imagine the product import is affected as well.

[SDavisMedia]

That fixed it. I'll go back and verify product earnings data preview is still good shortly.

[SDavisMedia]

Payment import mapping is much better.

Not all of my payments imported successfully but I'm not sure why. I can't find several of the payment IDs in my history. I'm attaching the CSV file I imported.

payments.csv.zip

[SDavisMedia]

Maybe there's something up with my data. I remember this happening with my first round of testing some weeks back. Pretty sure @michaelbeil testing with my CSV and had the same issue. I'll need to test with new data.

[michaelbeil]

You are remembering correctly @SDavisMedia.

[pippinsplugins]

@SDavisMedia from that CSV, do you happen to know of at least one example that didn't import?

[SDavisMedia]

@pippinsplugins It's reeaaally hard to say because it's looking like the IDs aren't matching up either.

What I mean is the very first product, ID 7359, did not import. The first one to import was row 215, ID 6271. But the CSV file's information for 6271 does not match ID 6271 data in my system after the import.

So not only is it hard to tell what didn't import, it's also hard to tell if my imports actually match my exports.

This data has been accumulating since 2014. Plenty of products and customers have been deleted. Not sure if that affects the export/import data in a negative way.

[pippinsplugins]

Note: IDs will not match before and after import.

What's the name of the product that didn't work?

[SDavisMedia]

Well I figured that was the case but then I wondered what's the point of mapping them? I guess for data relationship or something?

Anyway, again it's hard to say because some things are not matching. I simply can't tell what wasn't imported... I can only tell that the total number of records doesn't match up.

So let's look at the differences between one individual record that I've identified on both sites. The first product in my CSV is a transaction from June 6, 2016 (2016-06-06). I only have two transactions from that date in my export and they're pretty much identical purchases.

Notice both transactions are for $60. The payment records are pretty much identical so we can just look at the most recent one.

My "Commissions" product is $100 with an _excluded_ 20% tax, making it $120. On that transaction, a discount code was used titled discount and it is for 50% off. That puts the transaction at $60.

---

Now moving over to that same record on my import site. Things I'm noticing:

1. The first record displaying in my export site (the one I too screenshots of above) is the _last_ record on my import site (notice the pagination, the 3rd record details, etc.):

Should it be reversed like that? My entire import site purchase history is backwards, showing 2014 records first.

1. Focusing on the total price, it's wrong displaying at $170 on the table. Clicking through to the actual record, it changes even more:

I can't really tell what the math is doing but I do know the import site is not accounting for tax settings or discounts that don't exist within its system.

---

Definitely having a hard time understanding what I'm seeing so I can't really answer these questions clearly.

Might be worth it to export from a site with 3 transactions, all different dates, and all with taxes and discounts. Then import to a site with no taxes or discounts and observe the price details and record ordering, if that makes sense.

[pippinsplugins]

@SDavisMedia There shouldn't be an option to map payment / product IDs. Can you show me a screenshot of the mapping?

1. Import order.

Looks like you didn't map the Purchase Date. If the purchase date is mapped, the order should be retained. I'm going to double check that we are importing them in the correct order--not immediately sure.

1. Taxes / discounts. I'm going to start testing!

[pippinsplugins]

Definitely found some problems.

The bottom four purchases were exported and then imported back in, so the bottom four should match the top four. The amounts are definitely wrong.

[pippinsplugins]

@SDavisMedia you were correct about the export order. They were reversed.

[SDavisMedia]

My latest payment import attempt failed about a third of the way through. Here are the logged errors:

Jun 09, 20:55:23    PHP Notice: Indirect modification of overloaded property EDD_Payment::$address has no effect in /Applications/MAMP/htdocs/localhost/edd-testing/wp-content/plugins/easy-digital-downloads/includes/admin/import/class-batch-import-payments.php on line 342
Jun 09, 20:55:23    PHP Notice: Indirect modification of overloaded property EDD_Payment::$address has no effect in /Applications/MAMP/htdocs/localhost/edd-testing/wp-content/plugins/easy-digital-downloads/includes/admin/import/class-batch-import-payments.php on line 342
Jun 09, 20:55:23    PHP Notice: Indirect modification of overloaded property EDD_Payment::$address has no effect in /Applications/MAMP/htdocs/localhost/edd-testing/wp-content/plugins/easy-digital-downloads/includes/admin/import/class-batch-import-payments.php on line 342
Jun 09, 20:55:23    PHP Notice: Undefined offset: 1 in /Applications/MAMP/htdocs/localhost/edd-testing/wp-content/plugins/easy-digital-downloads/includes/admin/import/class-batch-import-payments.php on line 516
Jun 09, 20:55:25    PHP Notice: Undefined offset: 1 in /Applications/MAMP/htdocs/localhost/edd-testing/wp-content/plugins/easy-digital-downloads/includes/admin/import/class-batch-import-payments.php on line 516
Jun 09, 20:55:25    PHP Notice: Undefined offset: 1 in /Applications/MAMP/htdocs/localhost/edd-testing/wp-content/plugins/easy-digital-downloads/includes/admin/import/class-batch-import-payments.php on line 516
Jun 09, 20:55:25    PHP Notice: Undefined offset: 1 in /Applications/MAMP/htdocs/localhost/edd-testing/wp-content/plugins/easy-digital-downloads/includes/admin/import/class-batch-import-payments.php on line 516
Jun 09, 20:55:25    PHP Notice: Undefined offset: 1 in /Applications/MAMP/htdocs/localhost/edd-testing/wp-content/plugins/easy-digital-downloads/includes/admin/import/class-batch-import-payments.php on line 516
Jun 09, 20:55:27    PHP Notice: Undefined offset: 1 in /Applications/MAMP/htdocs/localhost/edd-testing/wp-content/plugins/easy-digital-downloads/includes/admin/import/class-batch-import-payments.php on line 516
Jun 09, 20:55:28    PHP Notice: Undefined offset: 1 in /Applications/MAMP/htdocs/localhost/edd-testing/wp-content/plugins/easy-digital-downloads/includes/admin/import/class-batch-import-payments.php on line 516
Jun 09, 20:55:28    PHP Notice: Undefined offset: 1 in /Applications/MAMP/htdocs/localhost/edd-testing/wp-content/plugins/easy-digital-downloads/includes/admin/import/class-batch-import-payments.php on line 516
Jun 09, 20:55:28    PHP Notice: Undefined offset: 1 in /Applications/MAMP/htdocs/localhost/edd-testing/wp-content/plugins/easy-digital-downloads/includes/admin/import/class-batch-import-payments.php on line 516
Jun 09, 20:55:29    PHP Notice: Undefined offset: 1 in /Applications/MAMP/htdocs/localhost/edd-testing/wp-content/plugins/easy-digital-downloads/includes/admin/import/class-batch-import-payments.php on line 516
Jun 09, 20:55:30    PHP Notice: Undefined offset: 1 in /Applications/MAMP/htdocs/localhost/edd-testing/wp-content/plugins/easy-digital-downloads/includes/admin/import/class-batch-import-payments.php on line 516
Jun 09, 20:55:30    PHP Notice: Undefined offset: 1 in /Applications/MAMP/htdocs/localhost/edd-testing/wp-content/plugins/easy-digital-downloads/includes/admin/import/class-batch-import-payments.php on line 516
Jun 09, 20:55:30    PHP Notice: Undefined offset: 1 in /Applications/MAMP/htdocs/localhost/edd-testing/wp-content/plugins/easy-digital-downloads/includes/admin/import/class-batch-import-payments.php on line 516
Jun 09, 20:55:32    PHP Notice: Undefined offset: 1 in /Applications/MAMP/htdocs/localhost/edd-testing/wp-content/plugins/easy-digital-downloads/includes/admin/import/class-batch-import-payments.php on line 516
Jun 09, 20:55:32    PHP Notice: Undefined variable: customer_by_email in /Applications/MAMP/htdocs/localhost/edd-testing/wp-content/plugins/easy-digital-downloads/includes/admin/import/class-batch-import-payments.php on line 414
Jun 09, 20:55:32    PHP Notice: Trying to get property of non-object in /Applications/MAMP/htdocs/localhost/edd-testing/wp-content/plugins/easy-digital-downloads/includes/admin/import/class-batch-import-payments.php on line 427
Jun 09, 20:55:32    PHP Fatal error: Call to a member function create() on a non-object in /Applications/MAMP/htdocs/localhost/edd-testing/wp-content/plugins/easy-digital-downloads/includes/admin/import/class-batch-import-payments.php on line 441

[pippinsplugins]

Another notice I'm seeing:

PHP Notice: Undefined index: price_id in /Users/pippin/Sites/edd/wp-content/plugins/easy-digital-downloads/includes/admin/reporting/export/class-batch-export-payments.php on line 167

[pippinsplugins]

@SDavisMedia ready for another test!

[pippinsplugins]

Looks like the email metadata isn't getting saved completely during import.

Payment History:

Dashboard widget and error log:

[pippinsplugins]

@easydigitaldownloads/core-devs could I get one more round of tests on this? Primarily want to verify my fix for the email issue above.

[mintplugins]

Just had a crash on import. 500 internal server error in console from ajax. Digging in to try and find the cause...

Sidenote about possible UX imporvements. If you export from EDD, I feel like the CSV value names should match up perfectly when importing - currently they don't. Is it possible for us to make them match up? Exact Title -> Exact Title

[mintplugins]

Here is my error log (which I cleared prior to importing). These should all be related to the import.
error-log-upon-import-edd.txt

To confirm, I tested on branch issue/2.6 up to https://github.com/easydigitaldownloads/easy-digital-downloads/commit/27d96af946b3eb099ba1ca481731f52b43f970f5

[pippinsplugins]

@mintplugins I can update some of the column names to match, but a few will need to be left as is for backwards compatibility reasons. For example, we really don't like the Amount column's name in the export and would love for that to be Total (like the import), but changing it Total could break existing users data export / import routine for various accounting systems. So one change at a time.

I'll push up some name changes on the Import side in a few minutes.

A few notes on some of the columns above:

1. Payment Number and Payment ID are not the same thing. They're very similar but not quite. The Payment ID is the actual ID field from the wp_posts table. Payment Number is the sequential order number assigned to the payment when sequential ordering is enabled.
2. When importing an EDD export, the Purchased Product(s) column should be mapped to the Products (Raw) column. This is noted in the updated import documentation. Could you take a look at that and let me know if it's clear enough?

[pippinsplugins]

Currency Code I'd like to leave as is since it makes it more clear that we are asking for the code and not the symbol.

Address Line 1 and Address Line 2 I'd like to leave as sell since it reads more clearly than having Address and then Address (line 2). Removes possible confusion about whetherAddressshould be the full address or just part of it. Example: doesAddress` mean street, city, postal, country?

[pippinsplugins]

Some changes pushed up: https://github.com/easydigitaldownloads/easy-digital-downloads/commit/74fe3e9e3f97925502332653a87afefc4b930cb3

[pippinsplugins]

@mintplugins could you test your same import again and let me know if the errors are gone?

[pippinsplugins]

See #4692 for future adjustments of column names.

[mintplugins]

Import worked flawlessly now! 👍

Re: Column names and backwards compatibility: Understood. Will continue that conversation here https://github.com/easydigitaldownloads/easy-digital-downloads/issues/4692

[pippinsplugins]

@mintplugins great!

I'm going to go ahead and close this as complete!