Openfoodnetwork: [Spree Upgrade][Flaky Spec] ./spec/features/admin/shipping_methods_spec.rb:86

[sigmundpetersen]

Description

This spec is unreliable:
./spec/features/admin/shipping_methods_spec.rb:86

Failing often, seen in:
https://semaphoreci.com/openfoodfoundation/openfoodnetwork-2/branches/2-0-stable/builds/216
https://semaphoreci.com/openfoodfoundation/openfoodnetwork-2/branches/2-0-stable/builds/229

Context

Our Semaphore builds fail regularly because of these unreliable tests. If you encounter this in your build, please comment on this issue or add 👍

Error:

1) shipping methods as an enterprise user creating a shipping method
     Failure/Error: expect(page).to have_content I18n.t('editing_shipping_method')
       expected to find text "translation missing: en.editing_shipping_method" in "Logged in as : [email protected] Account Logout Store DASHBOARD ORDERS PRODUCTS REPORTS ORDER CYCLES ENTERPRISES CUSTOMERS Editing Shipping Method NEW BACK TO SHIPPING METHODS LIST NAME DESCRIPTION CATEGORY DELIVERY PICK UP TAGS local ✖ CALCULATOR CALCULATOR Weight (per kg) Weight (per kg) Flat Percent Flat Rate (per order) Flexible Rate Flat Rate (per item) Price Sack PER KG: CATEGORIES SHIPPINGCATEGORY #31 ZONES GLOBALZONE Hubs FIRST DISTRIBUTOR SECOND DISTRIBUTOR MANAGE UPDATE or CANCEL"
     # ./spec/features/admin/shipping_methods_spec.rb:112:in `block (3 levels) in <top (required)>'

[mkllnk]

I looked through the last 20 Semaphore builds and could find only one that failed because of this spec. The error message is different as well:

  1) shipping methods as an enterprise user creating a shipping method
     Failure/Error: expect(page).to have_flash_message message
       expected to find flash message #Shipping method "Teleport" has been successfully created!
     # ./spec/features/admin/shipping_methods_spec.rb:114:in `block (3 levels) in <top (required)>'

The customers spec seems to be much more urgent.

[mkllnk]

Matt tried to solve spec/features/admin/shipping_methods_spec.rb:86 through 6e77afbf35644dc24519d18be6ae7e66287d36e7 but it's still happening:
https://semaphoreci.com/openfoodfoundation/openfoodnetwork-2/branches/2-0-stable/builds/229

  1) shipping methods as an enterprise user creating a shipping method
     Failure/Error: find('.flash', visible: false).text(:all).strip

     Capybara::ElementNotFound:
       Unable to find css ".flash"
     # ./spec/support/request/web_helper.rb:45:in `flash_message'
     # ./spec/features/admin/shipping_methods_spec.rb:113:in `block (3 levels) in <top (required)>'

[mkllnk]

I can't get it to fail locally. On Semaphore, I got it to fail once in 130 spec runs. It did fail on it's own. So it's not another spec interfering. Anyone having any other ideas?

[mkllnk]

I'm trying to narrow it down. I haven't seen any fail in 600 runs with this patch:

diff --git a/spec/features/admin/shipping_methods_spec.rb b/spec/features/admin/shipping_methods_spec.rb
index e81d904b2..b23d529c5 100644
--- a/spec/features/admin/shipping_methods_spec.rb
+++ b/spec/features/admin/shipping_methods_spec.rb
@@ -107,8 +107,10 @@ feature 'shipping methods' do
         expect(page).to have_css '.tag-item'
       end

+      sleep 1
       click_button I18n.t("actions.create")

+      sleep 1
       expect(page).to have_content I18n.t('spree.admin.shipping_methods.edit.editing_shipping_method')
       expect(flash_message).to eq I18n.t('successfully_created', resource: 'Shipping method "Teleport"')

I'm going to try those sleeps individually. If we have no other clue, the sleep is worth the trade-off to make the spec stable, I think.

[mkllnk]

Removing the first sleep, keeping the second fails. I suspect it has to do with the tags input. We had problems with that before.

Removing the second sleep looks promising. 400 runs :green_heart: