Woocommerce-services: Spike: Determine which WCS flows need E2E tests

[c-shultz]

We would like to get all important WCS flows under E2E tests this quarter so we have a strong foundation of tests to build on when we need to refactor and build new functionality.

The goal of this Spike is to check for flows that are not currently covered. We should also check for flows that affect site customers (e.g. automated taxes and live rates).

Note: it's not the same codebase, but while we're at it, we should also include the wc-admin onboarding prompt in this Spike.

Output:

• GitHub issues in WCS to create flows (with [Type] E2E )
  
  
  
  • Include as much detail as possible so test writer covers important functionality
  
  
• GitHub issues to improve E2E test infrastructure (if needed)
• Master thread post on the wcship P2 summarizing results and linking all open GitHub issues

[c-shultz]

@Ferdev, already created some good issues for this, so these should be included in the master thread:

Tests:
https://github.com/Automattic/woocommerce-services/issues/1915
https://github.com/Automattic/woocommerce-services/issues/1918
https://github.com/Automattic/woocommerce-services/issues/1917
https://github.com/Automattic/woocommerce-services/issues/1916

Infrastructure Improvements:
https://github.com/Automattic/woocommerce-services/issues/1914
https://github.com/Automattic/woocommerce-services/issues/1898

[c-shultz]

~For getting started with Looker, there's some good in the comments here: https://github.com/Automattic/woocommerce-shipping-issues/issues/22~ disregard 😄 too many issues open

[harriswong]

I agree to focusing on identifying and testing the critical flows, but not every use-cases. End to end test takes long time to run and it also has a high maintenance cost. I vote for the "test pyramid" approach where we focus more on the bottom of the pyramid than the top.

[c-shultz]

As part of this, let's also look at what Proton has done to run core tests in Docker to make debugging tests much easier. We should determine how easy this would be for us to adopt and how much value we think we can get from it.

[c-shultz]

+1 to focusing on the critical flows. Let's also update the linked issues as part of this to be really clear about what steps/flow the E2E should follow, so none of our PRs get lost in testing every possible use case.

1915

1918

1917

1916

[Ferdev]

hey @c-shultz, was this comment intended for another issue? https://github.com/Automattic/woocommerce-services/issues/2002#issuecomment-613142168

[c-shultz]

Lol. yep, I'm not sure what happened there. I had a few issues open at the same time.

[harriswong]

This is based on the information already shared by @Ferdev and @c-shultz.

User flow:

A user trying to accomplish a goal and how this user click through WCS. Some of these user flow are the following. If any of these fails, then the user can't buy label and that means WCS is not working properly:

• Purchasing label (done, https://github.com/Automattic/woocommerce-services/blob/develop/tests/e2e-tests/specs/admin/create-shipping-label.test.js)
• Reprinting label (https://github.com/Automattic/woocommerce-services/issues/2010)
• Refunding label (https://github.com/Automattic/woocommerce-services/issues/2011)
  
  
  
  • Purchasing label again after refund
  
  
• Track label (https://github.com/Automattic/woocommerce-services/issues/1915)
• WCS settings (https://github.com/Automattic/woocommerce-services/issues/1916)
• WCS banner from WC (https://github.com/Automattic/woocommerce-services/issues/2012)

Order status

How the user interact with WCS also depends on the order status. New order and existing order should provide different user flow. Some of the following overlaps the user flow listed above. But I wanted to list them out and consolidate after.

New order

• Click create shipping label, click buy shipping label [done]
• Shipping label modal
  
  
  
  • Filling the form and click "buy shipping label"
  
  
  • Test 1 item
  
  
  • Test multiple items with multiple packages (https://github.com/Automattic/woocommerce-services/issues/2017)
  
  
• Manual create order (https://github.com/Automattic/woocommerce-services/issues/2013)
  
  
  
  • Click create shipping label, but manual enter addresses
  
  
  • Repeat above test cases
  
  

Existing order

• Shipping label model
  
  
  
  • Should display Tracking
  
  
• Shipment Tracking (https://github.com/Automattic/woocommerce-services/issues/2014)
  
  
  
  • View details -> should popup
  
  
  • Request refund
  
  
  • Reprint
  
  

Plugin status

This page is used for debugging and would help troubleshoot when creating a support ticket.
wp-admin/admin.php?page=wc-status&tab=connect

• Make sure status is green (https://github.com/Automattic/woocommerce-services/issues/1917)
• Make sure print button works (https://github.com/Automattic/woocommerce-services/issues/1918)

Tax

• Automated tax. Should support only 1 city tax calculations, where they have nexus in. (https://github.com/Automattic/woocommerce-services/issues/2015)

Live rates

• As user, go to product page, select product, add to cart. Test shipping rates are shown. (https://github.com/Automattic/woocommerce-services/issues/2016)

Infrastructure

• Create a "wait for selector with text" function, move it in our util folder. (https://github.com/Automattic/woocommerce-services/issues/2009)