Openfoodnetwork: Stripe not accepting multiple credit cards, can not process order

[chezaorchard]

Description

Customer attempting to check out - completes credit card details for stripe payment - details are not accepted. Customer attempts this with three different credit cards, from two different issuing banks, all return errors. See below errors:

ING debit card - your card has been declined.
ING debit card - your card does not support this type of purchase.
ANZ credit card - your card has been declined.

Customer was unable to process order and contacted hub.

NB. Notes from Customer

Expected Behavior

Customer credit card details should have been accepted and transaction processed without error.

Actual Behaviour

Customer could not check out, order was unable to be made.

Steps to Reproduce

Not able to reproduce

Animated Gif/Screenshot

Workaround

Hub manually processed the transaction for the customer.

Severity

bug-s1: a critical feature is broken: checkout, payments, signup, login

Your Environment

• Version used:
• Browser name and version: Safari
• Operating System and version (desktop or mobile): Desktop

[luisramos0]

This could be a problem with the stripe account of the hub? Is the hub able to receive payments from other customers?

[kirstenalarsen]

yes @luisramos0 the hub is receiving payments from other customers. They are having a lot of occurrences of #5774 as well - every time there is a failed payment.

I have just had a look and the two affected customers had emails --REMOVED email address--- and ---REMOVED email address--- - there are completed orders for them in the system as the hub put them through (using their own credit card). The customers were unable to complete their own orders

I cannot find any record of these customers in Stripe i.e. I can't see the failed payments. I'm not sure if it is possible that OFN didn't get as far as talking to Stripe?. The payment method is Stripe. I don't know if there is a security problem posting the actual stripe method / code here so will send in pm

They are both regular customers - unlikely to be major user error

For one of them (Jennie) we can see all these orders that she has not completed - I don't know how many of them were tried. When I search stripe for those order numbers I get nothing - no record of them in Stripe
REMOVED screenshot
Where there are errors of failed payments it looks like most of the errors are, which I think is unusual

I have now been looking at Stripe and I am seeing a lot of these - including for other customers - [just realised this is not the whole payment - this is the missing bit as reported in #5774 ]

They are showing as failed in Stripe and ---removed screenshot---

--

[kirstenalarsen]

I am making this an S1 because we have customers unable to checkout

[luisramos0]

I see checkout errors in bugsnag but the errors look valid:

[kirstenalarsen]

There seems to be a significant increase this week @luisramos0 and the reports from customers are that there is money on the card. It seems suspicious - they are regular customers who have ordered plenty of times

[luisramos0]

First I did a frequency analysis in AUS live, these are the number of refused payments rejected based on "insufficient funds" per day with message "Your card has insufficient funds.":

This is for do_not_honor error type with message: "Your card was declined."
do_not_honor:

This is for error transaction_not_allowed with message: "Your card does not support this type of purchase."
transaction_not_allowed:

For these 2 last error types, the stripe docs say:
"The card has been declined for an unknown reason."
"The customer needs to contact their card issuer for more information."
here: https://stripe.com/docs/declines/codes

I looked at one of the two occurrences of "Your card does not support this type of purchase" (user first name J.) and it looks the card was being used previously in OFN yes but now it's failing with this error. I think they need to contact the bank.

Another user J. that Kirsten mentions has paid successfully previously with a mastercard but not with the visa that is now failing. The visa has actually already failed in the past and failed again with error "Your card was declined.". Looks like they need to use the mastercard that worked before or contact their bank regarding the visa card.

So far this is looking like not-a-bug... what do you think?

[filipefurtad0]

Hi all,

I've tried to reproduce this issue in staging but found payments always succeeded with valid cards.

The only thing which strikes me and may cause confusion to the user is the discrepancy between errors related to invalid cards, observed in hubs configured with Stripe Connect. This appears to be a bug.

For example. if an error is triggered (for example, due to an expired card) and afterwards a different invalid card is used then, the error shown to the user is that of the first card. I could not reproduce this for hubs configured with stripe-SCA.

The gif below shows introducing 4 invalid cards, followed by a valid card. The errors are displayed above, in the red banner:

These attempts appear in Stripe's dashboard:

So this apparently does not prevent the user from using a valid card, but may cause confusion if the card being used is invalid or the payment is denied.

Tested cards, in stripe SCA and Connect, respectively in different staging servers:

• Valid cards

5200828282828210 Mastercard (debit)
4000000560000004 tok_be pm_card_be

• Invalid Cards

4000000000000002 Charge is declined with a card_declined code. (displays a correct error message)
4000000000000127 Charge is declined with an incorrect_cvc code (displays an incorrect error message)
4000000000000069 Charge is declined with an expired_card code. (displays a correct error message)
4000000000009995 Charge is declined with a card_declined code -> insufficient_funds (displays an incorrect error message)

[kirstenalarsen]

i was just attempting to do something else which led me to be trying to checkout using our credit card to try and create a test order and I have managed to magically recreate this issue! Well done me. I have checked the numbers multiple times and they are definitely correct. I have a few hunches to explore but just documenting here.

This is firefox

I am now trying in Chrome with dev tools open - failed again.
Same error as above.

This is dev tools

I have checked amount of money on the card and I have called and spoken to the bank. There is no hold on the card. Other transactions are going through. I am about to try buying something on a different website

I am wondering if this could be connected to the particular stripe account, but I can't see how it would know that this is 'our' card. Just checking now . .

Yep - have confirmed that I also CAN'T use the card to checkout at a different shop on OFN. I could checkout on that same shop with a different card.

I have now just gone to CERES Fair Food (where I can easily cancel my order) and placed an order using this card. The problem is absolutely, comprehensively US

[kirstenalarsen]

NB. Using a different card to complete that test order I now get the error that my order needs to be at least 50c. I think that is coming from Stripe as OFN can take orders <50c (I just put one through for zero using different payment method)

Does this mean that we never even hit Stripe with this? I think maybe as I am not seeing failed transactions in the Stripe account. There is no record of any of this activity

Then I tried to do it in Chrome (same shop and order as before) with a different card and an order >50c and now I get this error

and these are all my errors in dev tools

[kirstenalarsen]

I have now just gone into a different browser, different order, same shop, different card - and created an order that is showing up in Stripe. Definitely correct stripe account and stripe is working.

[kirstenalarsen]

The plot thickens . . one of those orders has gone through - as having been paid using the Stripe payment method but with balance due of $0.40 i.e. no payment was taken. And I got a warning message at the OFN end that order couldn't be less than 50c

No record of this in Stripe at all

[luisramos0]

hey Kirsten, dont you think we should separate the issues? One maybe around payments below 0.5cents, the other is the generic error messages from stripe about cards they may not be blocked by the bank. Maybe we are getting them mixed up because between our system, the stripe systems and the banks systems there may be a rule to decline payments below 0.5.

[filipefurtad0]

Thanks for these insights @kirstenalarsen.

Agree with @luisramos0 , this looks like a separate issue - created #5803 ; Moved my previous comment here as well.

[kirstenalarsen]

yeah i didn't think the 50c thing was this issue, I was just reporting what happened. I agree separate issue is good idea - thanks @filipefurtad0 and @luisramos0

[kirstenalarsen]

I would really like to know if we are seeing this problem (the main one, that this issue is about) anywhere - is it showing in our logs / bugsnag etc? Is there a trace of me doing this yesterday that we can check if is happening to other people, on top of the ones we know about? If it is, then this should go back to S1

[luisramos0]

yes, there are a couple of entries every now and then about failed cards which basically means users do try to user cards that dont have enough funds or are blocked by stripe or their banks for some unknown reason, the statistical analysis I did here shows v3 didnt change the numbers:
https://github.com/openfoodfoundation/openfoodnetwork/issues/5785#issuecomment-661038832
I repeated the queries now and the numbers for these 2/3 days since then are similar.

We could investigate if some of these are valid cards that work in other websites but why? Is it worth it not trusting Stripe?

Stripe is very clear about the do_not_honor error here:
https://support.stripe.com/questions/decline-code-do-not-honor
"the only way to resolve this is to ask your customer to reach out to their card-issuing bank to look at the transaction and determine why it was declined. Customers should provide the bank with the charge amount, date/time, and the company name for assistance."

Wednesday seems to be the day with more do_not_honor errors, yesterday 22July we had 6, previous weeks we had 7 and 5 respectively (wednesdays). Other days of the week show 1 or 2 of these, but also many days without errors.
I checked the total volume for yesterday and I see 164 credit cards payments, 6 errors is 3.5% of the payments, but on the 19th, 20th and 21st there were none of these and over 100 payments in each of these days, so, 0%.

I looks like we have bigger problems to solve. I'd downgrade this investigation to S3. What do you think @kirstenalarsen?

[kirstenalarsen]

Can you clearly see the errors in the database around the time that I filed this issue @luisramos0 ? My attempts are definitely included in your numbers above? Because there is no record in stripe of the problem - it is not showing as a failed payment or do not honour in stripe, it doesn't get that far. This is why it worries me - because i'm not sure that we're picking it up at all, and therefore we have no idea how many people it's affecting?

I am not keen to spend more time chasing bugs than we have to. However given that in my case we know that there is nothing wrong with the card, it does point to there being a problem at OFN.

Those numbers you are giving me for do-not-honour errors are for Aus only? Wednesdays hey . . what is happening on wednesdays . . @chezaorchard @emilyjeanrogers ?

[luisramos0]

Can you clearly see the errors in the database around the time that I filed this issue @luisramos0 ?

I think you mean your tests on the 22nd July, right?
I tried to find them on the database and I couldn't. I now remember that there are two steps in the payment process:
1 - on the browser, call to stripe to add the card details to stripe (or get a new payment token if the card is already there)
2 - on our server, call to stripe to take the payment
The stats I took from the DB are only for the second step.
We have improved things recently regarding step 1 and added a bugsnagJS alert when there are problems with step 1.
BUT, there are two things that are now bothering me:

• I cant find your test payments in bugsnagJS
• I know we have seen a few new bugsnagJS errors since PR #5070 but I see that in this PR we have not tested the fact that we were getting bugsnag alerts when appropriate (only tests the console).

This is the relevant code.
I am not sure we have bugsnagJS activated for any staging environment so I just tested in katuma prod.
I run the test from #5070:
"We can test the logging by checking out with a test stripe card but inserting a invalid expiry date. It shows a red message but you can click checkout anyway and check the console log."
And I dont see any bugsnagJS alert for katuma live. I think we need to fix that so that we get a bugsnagJS alert for these errors.

it does point to there being a problem at OFN.

It's not just OFN, it's a combination of OFN with the OFN Stripe account with the Hub stripe account. All that influences the Stripe algorithms and can make cards be blocked.
Anyway, I think we can improve logging by making sure we get the bugsnag alert for all these errors. I am not convinced we will find bugs on our side that will make more payments be accepted.

The numbers are for AUs only, I checked the DB now, the numbers are proportional to the number of sales, Wednesdays and Sundays are the big days in AUS.

ping @openfoodfoundation/testers could you double check that the test in #5070 does not trigger a bugsnagJS alert? If that's the case I think we should open a specific issue to get that sorted.

[filipefurtad0]

Hi @luisramos0 and @kirstenalarsen ,

Yes, I remember that one.. It introduced console-logging, so I don't think errors on bugsnag were checked.

I looked for errors in staging-FR in the dates the tests were performed, but unfortunately bugsnag it only keeps data from two months ago (from May onward, in this case), and the 5070 was in March.

Also found no errors in devops-notifications channel in Slack, on the dates this PR was tested:

I tried to trigger bugsnag now, with cards

• 4000008400001629 - 3D auth card, card_declined failure code.
• 4000000000000002 - card_declined code
• 4000000000009995 - Charge is declined with a card_declined code. The decline_code attribute is insufficient_funds.

Which appear on Stripe dashboard:

But not on bugsnag-UK (where it was just tested). BugsnagJS would appear on the console - right? These are the three errors:

So, in summary, yes - we need an issue to add bugsnagJS alerts on failing Stripe-SCA payments.

[luisramos0]

@filipefurtad0 thanks for your feedback!
are you aware that BugsnagJS is a different project on bugsnag and also on slack?
All BugsnagJS projects are in the AUS bugsnag account and the slack channel is devops-notifications-js

[filipefurtad0]

No I was not - @luisramos0 . Thank you for pointing that out.
I can't seem to find that channel in Slack, though. Perhaps an invitation only channel?

[filipefurtad0]

Thanks for adding me @luisramos0 :tada:

I just ran these three invalid cards (as posted above) on a stripe-SCA configured hub, at staging-AU, but observed nothing on the channel nor on bugnsnag JS Australia. Would this bugsnag be the right place to look?

Here's the registry on Stripe.

Do you consider these tests conclusive? Or are there specific test cases which could trigger the expected warnings?

[luisramos0]

I think the test in #5070 (card expiry date) is the one we need to use here: for BugsnagJS. It's the different between testing the step 1 or the step 2 I mention above (you can verify this by looking at the network tab in dev tools: the request that must fail is the one to stripe.com, not to our checkout endpoint).

I am not sure BugsnagJS is configured in staging AU.

In your test case I think the alerts should go to the normal (backend) bugsnag.

I think I need to take this on myself, configure bugsnagJS in a staging environment and test the different cases.

[luisramos0]

I can replicate the bugsnagJS problem locally, it looks like we are not really getting all the javascript error in bugsnag.
This is the new issue: https://github.com/openfoodfoundation/openfoodnetwork/issues/5858
I made it S2 as this one. I am not sure about that.

[luisramos0]

One idea that came up today was to make the javascript call to Stripe a bit more resilient like we did for the checkout javascript code recently in #5202
Basically we would do something like 5202 (use catch instead of response.error) here:
https://github.com/openfoodfoundation/openfoodnetwork/blob/6e1dd78e9f83101dea78ab5fa67d0e3724815e09/app/assets/javascripts/darkswarm/services/stripe_elements.js.coffee#L14

It's a very easy improvement, maybe we could do it now...

[kirstenalarsen]

seems like a good idea