Openfoodnetwork: Validation email error with an old account with bug before email confirmation work

[myriamboure]

Description

We have one user in France who created her account when there was a bug with private shopfronts. We discovered the bug with her actually. Today her account still have the following issue:

• she still appears as a "guest user" even though she has an account, she logs in, etc.
  
• the hub manager created a subscription for her but the subscription is failing, with the given error message
  
  (the message says that 5 recurring orders were planned for that OC, and 4 were treated successfully, 1 failed du to error:
  R377155761: this email is already associated to a user and has already been validated. Please login to continue, or use another email.)

Also strangely if I look at the failing order R377155761 here is what I see

so it is in "cart" status but with "balance due" which is somehow strange...

We are stuck, we can give no support, she doesn't have another email she can use, and anyway I tried that workaround but it's failing also (see #2840)

Expected Behavior

She has validated her email so she shouldn't be a guest anymore and not treated as such. She can get subscriptions.

Steps to Reproduce

1. You need to connect as a super admin on French instance and see in customer the status for that user.
2. You can check the subscription failing status from the order R377155761

Context

The user is trying to get subscription working... but it doesn't work for that particular customer.

Severity

There is no workaround, but it concerns only one user, so I would say S3

Your Environment

• Version used: v1.18
• Browser name and version: Chrome
• Operating System and version (desktop or mobile): Ubuntu 18

Possible Fix

• Can we resend some confirmation link? Or check in data base the status for that user and fix it?

[myriamboure]

@Matt-Yorkley you might have an idea where that can come from maybe?

[myriamboure]

I just did another test: I added that user as a manager of a demo hub, it worked without problem, so the email is definitely validated, but still act as a guest account for some aspects of it... (marked as "guest checkout" and blocking subscription)

[myriamboure]

Ok, this user is stuck. I know it concerns only one user, but there is no workaround, we have trying everything we could imagine with Rachel, and the inconsistency guest/account seems like there is a tech bug. Does it qualify as a S2? We have really no workaround... we need someone to look into DB to try to understand the problem. I try it, if you disagree retrograde, I will just have to try to make her wait and not use subs for herself...

[daniellemoorhead]

Just so we're tracking the conversation for update purposes on here :)
@pacodelaluna @HugsDaniel @luisramos0 over to you all to get into it (potentially using @mkllnk's suggestion)

[lin-d-hop]

UK are also experiencing this issue with one shopfront.
There is no workaround for the user.
In particular it is one of our FA users and they only came onto OFN for subs... so this is a real headache that ONLY this user is experiencing this problem :-/

@myriamboure with this affecting a user in UK and France can we up to s2?

[luisramos0]

@daniellemoorhead that conversation on slack was for #2925
I am not in the email party, Matt and Maikel and Hugo have been covering that area lately.

[daniellemoorhead]

Ok, seems I got mixed up on my S2s...oops!

So @Matt-Yorkley @mkllnk @HugsDaniel will one/more of you pick up this S2 and get it sorted?

[mkllnk]

I haven't looked at the data yet, but I have an idea of what could be going on. Let's get the terminology first:

• Guest: A guest user is somebody who doesn't have a user account. They never registered, never entered a password or are just not logged in. This is different to somebody who signed up, but is not confirmed yet.
• Registered user: They submitted the form to register a user account. So they entered an email address and a password. The email address needs to be confirmed via a link before the user can log in and use the account.
• Customer: A customer is somebody who ordered through a shop. They can be either a guest or a registered user which is has been confusing.

A subscription is associated to a customer, but only works for customers who are registered users. And I guess in this case we have a subscription associated to a guest customer account. There is also a registered user with the same email address. But for OFN, they are two different customer records. One is a guest and one is a registered user. And since the guest record was created anonymously, OFN does not assume that they are the same person.

[mkllnk]

I looked at the French data and can confirm that the subscription is associated to a guest customer. There are many different ways how we can fix it. Here are some thoughts.

Create a new customer record with the registered user

The admin interface doesn't let you create two customer records with the same email address. Since we have a guest customer already, we can't create another customer record ourselves. But the user should be able to log in and order from the shop. That should create a second customer record.

I tried that locally, but ran into other problems. That user already ordered yesterday and if I now try to visit the shop, it crashes. The order from yesterday can't be displayed, because the code can't fine the prices any more. Very bad and very distracting. Another S2?

app/models/spree/variant_decorator.rb:89:in `price_with_fees'
app/serializers/api/variant_serializer.rb:19:in `price_with_fees'
app/helpers/injection_helper.rb:98:in `inject_json_ams'
app/helpers/injection_helper.rb:30:in `inject_current_order'
app/views/layouts/darkswarm.html.haml:42:in `_c6d2238a3bcf573720820adc8c63780a'

But that order proves that creating another order as user doesn't create a new customer record. It seems to think that the guest record is enough.

Allow to create multiple customers with the same email address

We could change the code to allow the creation of a registered user customer with the same email address of another customer. This could be very confusing, but it would allow to resolve this issue. It would also be correct, because email addresses can change owners. If an email address is given to another person (like a phone number), we should create a new record. This would only be possible if we can deactivate the old account first.

I need to look into the last pull requests working on this field to understand what the latest agreement was. What should happen when an existing guest customer decides to sign up? Why does the subscription fail for guest customers. The payment method is "Check". So if there is not automated payment, how is it supposed to work? I find it really difficult to find the information at the moment.

Make the guest customer a registered customer

A developer can change the association in the database:

c = Customer.find(887) # Perrine's guest customer account associated to the subscription
u = Spree::User.find(828) # Perrine's registered user account
c.user = u
c.save

This could be the easiest and quickest fix. But I'm still thinking about data integrity. It means that we have a registered customer record that is associated with guest orders. That's probably okay, but who knows, we have a lot of unanticipated data integrity problems at the moment.

[kirstenalarsen]

might also be useful to visit the discussion here where this potential issue was identified I think, and there was some discussion about pros/cons of different ways to deal with it. Might be unrelated, if so ignore me :)

[mkllnk]

The problem

I think I understand now what's happening.

• Perrine orders from a shop as a guest. A customer record is created.
• The shop manager creates subscription with the current customer record. This record is a guest record.
• 4 subscription orders are successful. Everything works.
• Perrine signs up for a user account and confirms their email address. The customer record is not updated and is still a guest record.
• When Perrine tries to buy more as guest, the email address is already in use. Perrine has to log in to order from the shop.
• The subscription tries to create the fifth order. As usual it tries to create a guest order. But ordering as guest is not allowed if there is a registered user with the same email address. The order fails and remains in the cart state.

The solution

We need a subscription that is associated to a registered user customer. Our own data validation rules restrict us to only one customer record per email address for each shop. In order to work within these constrains, we can only update the customer record to associate it to the registered user (code above). But this is not a permanent solution. This will happen again.

I would like to propose to allow two customer records per email address. One for a guest and one for a registered user. In this case the solution here would be to create a new customer record and a new subscription for the new registered customer. The subscription for the guest customer can be stopped.

What to do for now

We should run the code above to update the current customer record. I don't think it will be a problem, especially since we know that it's the same person. It is easy to revert as well. I'm not sure what will happen to the subscription though. I don't know if it will try to create the order again and if that is desired. Since the customer ordered yesterday, we probably don't want two orders. That's why I think this should be done after talking to the shop manager and possibly to the consumer. Everybody has to check that they get what they want.

Quick fix

Here is the code again to associate the user to the customer record:

c = Customer.find(887) # Perrine's guest customer account associated to the subscription
u = Spree::User.find(828) # Perrine's registered user account
c.user = u
c.save

And in case you want to revert it:

c = Customer.find(887)
c.user = nil
c.save

[myriamboure]

Thank you @mkllnk for the investigation, I think the porblem might be related to https://github.com/openfoodfoundation/openfoodnetwork/issues/2338. We were supposed in this solution to be able to have two records with the same email in a given shop, but apparently it has not been implemented as expected.
One question though:

I would like to propose to allow two customer records per email address. One for a guest and one for a registered user.

If I have a subscription running with the guest account and a second record is created for the registered account, the subscription will keep on with the guest account, no? If I want to switch it to the customer account I have to stop the subscription and create a new one with the registered account so does it mean I will see twice the name of the customer when I first choose the customer in setting up the subscription? I will see twice the name so I need at least to know which one is the guest and which one is the registered customer...

All in all I find all that very confusing... with the level of complexity we have in our processes I'm wondering if we shouldn't just remove the guest checkout possibility. Or accept to just change it from guest to registered and associate previous guest orders to it.

@mkllnk about your argument on email address that can be realocated, I'm still pretty dubious, how does other systems manage that? If I buy on Amazon, I die and my email then is used some time later by another person, when this person tries to access Amazon the site will say the address already exists, will ask "new password" and then the person will be able to access my previous orders, etc.

Can an email address really be reallocated to someone else??? I think it is important to answer that question first... and I would like to know how other system do. There is no way IMO to protect from that... unless the system is notified that the email address has changed owner by some central authority... The email is used by so many system as unique identifier, so why is it if the email is not a satisyfing UNIQUE identifier?

For quick fix I'm happy that we just associate the user to the customer record so the user. Can you do it @mkllnk ? Or should I ask someone else? Let's do it, I'll pause right now her subscription so she won't be affected straight away.

But we will hit the problem again so we need to find a more durable solution in parallel...

[myriamboure]

Ok the impact of this bug is worse that I thought:

• a hub manager adds a list of email in the customer list because they want to setup subscriptions for them. When you add a customer in the list, no validation email is sent, nothing. So if I setup a subscription straight away, and an order is made automatically, the record will be a guest record.
• then the user wants to shop, he has to login to shop as the shop has been setup that way, BUT when she logs in, she can't update the order as she is not recognized as the same record, now she is registered user, but the subscription was made with the guest record...

THIS IS A MESS !

I told the user to hold on, we need to find a durable solution from now it seems or the subs feature is unusable. @mkllnk @Matt-Yorkley you both have worked on it so might be more familiar with potential solutions... would like to hear also @luisramos0 inputs about how such problems are handled in other systems maybe?

[myriamboure]

Trying to list options to solve that annoying bug:
1- We need guest records to be associated with registered user when it is created, so the subs is "transferred" to the registered account and reconciliation happens correctly.
_This would be the smoother solution, best UX, would avoid confusion. But there is still a risk that some guest orders are made with error emails, and have data discrepancy. The only way to avoid that is to force login before buying, which is what most website do (like Amazon, etc.)_
2- We don't allow subscription to be run on unregistered user. When I setup a subscription for an unconfirmed user, it triggers a confirmation email, and the subscription can only be effective when the user has clicked on the link.
_I find it confusing for the user, to receive a confirmation email from a system I don't know anything about (they don't know about OFN they know about their hub name...)_
3- Remove guest checkout option and just force any user to login to be able to complete checkout. So we know we will have no wrong guest orders associated with error emails. But it means that if I add a customer in my customer list, this needs to trigger an email saying something like "hub X has invited you to join OFN to be able to buy in their shop, click on confirmation link below to accept and setup your password".

I know guest checkout is something people like, but it's causing so many issues, I would vote to go for option 3 personally. Do you see other options?

[mkllnk]

Duplicate records

The pull request solving #2338 only added the label to guest customer records. It didn't change any logic. The description says that there were duplicate customer entries, but I don't know how that could happen. The uniqueness constraint was added in 2015 ed941e211d771526c06cd1e50878311c25c83978.

If I have a subscription running with the guest account and a second record is created for the registered account, the subscription will keep on with the guest account, no? If I want to switch it to the customer account I have to stop the subscription and create a new one with the registered account so does it mean I will see twice the name of the customer when I first choose the customer in setting up the subscription? I will see twice the name so I need at least to know which one is the guest and which one is the registered customer...

Yes, that is how I imagined it would work.

On email address re-use

about your argument on email address that can be realocated, I'm still pretty dubious, how does other systems manage that? If I buy on Amazon, I die and my email then is used some time later by another person, when this person tries to access Amazon the site will say the address already exists, will ask "new password" and then the person will be able to access my previous orders, etc.

Most systems don't deal with re-allocated email addresses. I'm not sure about Amazon, but in most online shops somebody could take over your account after they inherited your email address.

Email addresses are recycled by most email providers. It made the news when people realised that it is very common practice. One exception is Google's Gmail. They explicitly say that they are not recycling any addresses.

Email recycling is a known problem and companies like Facebook already worked on a solution and there is a standard now.

Quick fix

I have updated the customer record. It's now associated to the registered user.

Inviting users to OFN

We definitely have a broken process here. And we talked about this a lot before we released subscriptions. We must have forgotten along the way that this is still not solved.

If a shop requires logged in users, they can't invite guest users. It's bad that our UI allows that. We talked about enforcing users to log in for subscriptions, but that is a problem as well. Some shops want to be able to create subscriptions without the customer needing to do anything. They are talking to a customer without computer on the phone. That's why we allowed shop-managed guest subscriptions.

I totally agree that we miss the feature to create registered user accounts for customers. As you said, they need a different welcome email explaining who invited them to what.

Transition from guest to registered user

I find this one really difficult. On one hand it would be really convenient to just associate a new user account with all the old data associated to that email address. On the other hand it can violate people's privacy. Even though it's a rare case, you can't say: "I'm innocent to 95%".

What do others do? You mentioned Amazon not allowing guest orders at all. Shopify does allow guest orders and keeps the guest history separate from a new account's history. We could probably find more examples for either practice.

Is a subscription so complicated to re-create? Should we give a shop manager the power to re-assign a subscription from one customer to the other?

[luisramos0]

Should we give a shop manager the power to re-assign a subscription from one customer to the other?

I dont think so, sounds complicated.
And converting guest user to registered user is also complicated. I wouldn't go that way.

What if:

• we remove guest user checkout from OFN?
• if not, what if we make it impossible to create subscriptions for guest users?

[mkllnk]

we remove guest user checkout from OFN?

• We lose a few shoppers who want to try it out and don't want to create yet another account.
• When creating orders for a new user from the admin interface, we always need to create a user account first.

what if we make it impossible to create subscriptions for guest users?

• We need to create user accounts for people.

Any other implications @sstead?

[myriamboure]

Change guest account to registered account

Shopify does allow guest orders and keeps the guest history separate from a new account's history.

Can't we just do that? Associate the new user to the guest user, BUT not associate the guest orders with it, like move the association of guest orders to some form of general guest user, like a trash user ;-) I say that because I guess an order needs to be associated to a user.

As a hub I could invite customers, and create subs even on guest customers, when they finally register they are recognized and their account becomes registered, so from now on they can see their orders, etc. The UX would be ok, even though the first subs I will have won't be saved in my customer account...

The problem of recycled emails is not fully solved, but not more than it is today, if my email is recycled, the person who inherits it can ask "forgot password" anyway, so I don't really see what it changes from what we do know... if we want to address that problem we need to change the way we work today.

Removing the possibility to checkout as guest

@luisramos0

What if:
- we remove guest user checkout from OFN?
- if not, what if we make it impossible to create subscriptions for guest users?

You can already create subscriptions for guest users, the problem is that when those users register, it makes the subs crash (as it did for Perrine here) as it is associated with the guest account but there seems to be a conflict that @mkllnk mentioned above "But ordering as guest is not allowed if there is a registered user with the same email address. The order fails and remains in the cart state."

I know if might discourage some users to have to create an account to be able to buy, but it's always a design question. If customer can shop, put product in the cart, and you are only asked to login when you checkout, you have already "catched" somehow your customer, it's not as bad as asking to register to see the content of a shop. I don't think it would be too problematic...

If we move toward that direction, it means when you invite a customer in your shop, if should be an invitation that the customer has to accept buy confirming the link and setting up the password, so it's just another confirmation email, and like managers are invited, customers will be "pending" until they accept.
If another shop invite the same customer, they would still have to accept to join that hub customer list, but technically they would be already a registered user. So I would not consider that confirmation only as a way to validate the user email and create a registered user, but also as an opt-in to join the hub shopper list (which is good in term of GDPR). It's not the same as for managers, you can't have a list of all registered customer that you can pick into for your customer list ;-)

So technically, as a hub I wouldn't be able to create a subscription until a user has accepted to join my hub. Which doesn't sound like illogical to me. It will be marked as pending and the hub manager should be able to resend the invitation in case the customer deleted it, or didn't pay attention.

So those are the two options I see then... what do you think ?

To sum up:
- Option 1a: when registered user with same email address of guest user, the guest user account is "transformed" into the registered user account and guest orders remain attached. If recycled email, account can be hijkacked, but just as it can today.
- Option 1b: when registered user with same email address of guest user, the guest user account is "transformed" into the registered user account but guest orders are not attached to it. If recycled email, account can be hijkacked, but just as it can today. We avoid some risk linked to wrong emails being used in guest checkout, so it might be safer.
- Option 2: we get rid of the possibility to checkout as guest, introduce the obligation to login at checkout for unregistered/unlogged users, and when a hub add a customer in customer list, an invitation/confirmation email is sent and user remains "pending" until it is confirmed. Subscription can only be created when invitation is accepted.

@sstead @kirstenalarsen @mkllnk @luisramos0 @Matt-Yorkley ? Do you see other options? It's not an easy decision, but it is crucial we state on that for the future development of our product.

[mkllnk]

Option 1b sounds good to me. I'm not sure yet what exactly that means on the data level. We may copy the customer record or disconnect the old orders. It may even be good to never associate anonymous guest orders to a customer record. If the shop wants to see all orders of one email address, they can use the filter. Do they need a customer record?

[sineadfenton]

Guest Account to Normal Account

[luisramos0]

@sineadfenton these trello links are not public.

[sineadfenton]

public now - we use this just in the UK but hopefully you can see and interact with it now @luisramos0

[myriamboure]

@mkllnk you probably need other devs answer on this, but my 2 cts is that it is fine if guest orders don't have a customer record, like are linked to no customer. @Matt-Yorkley @sauloperez @luisramos0 do you agree ? It would be good to move forward then as I see UK linked another case, and we have users just waiting for this to be solved to use again subscription, they are just blocked.

[kristinalim]

From @mkllnk:

I would like to propose to allow two customer records per email address. One for a guest and one for a registered user. In this case the solution here would be to create a new customer record and a new subscription for the new registered customer. The subscription for the guest customer can be stopped.

I like this option too. Subscriptions and orders created by the hub before a user registers will not be exposed to the user by default, unless the subscriptions and orders are explicitly assigned to the registered user (if implemented).

We could then think of the data of the guest customer record as the hub's own data, which the hub alone manages, not the person behind the customer record.

But ordering as guest is not allowed if there is a registered user with the same email address. The order fails and remains in the cart state.

And then this restriction can be removed too?

Option 1b sounds good to me. I'm not sure yet what exactly that means on the data level. We may copy the customer record or disconnect the old orders. It may even be good to never associate anonymous guest orders to a customer record. If the shop wants to see all orders of one email address, they can use the filter. Do they need a customer record?

I prefer this option too, even without considering that it seems to be the fastest solution. Then as mentioned, we'll probably need some indicator in admin views for whether a customer email is for a guest or for a registered user.

[kristinalim]

Agh, sorry! I accidentally closed the issue.

[myriamboure]

I see contradictions in what you say @kristinalim but maybe I don't fully understand. For me if we allow two accounts per email then it's not possible to move a guest account (attached to one email) to a validated account. What we suggest in 1b is that, the user account becomes validated, it's not a second account attached to the same email. There can only be one account with one email (if I understand) but the status changes. Also it's logical then that if you are not guest anymore, you cannot buy as a guest with the registered email.
About shall we associated old orders to an account to start with or not, and disconnect or not when guest becomes registered user, this is more a tech decision, up to you.

From the discussions above it seems people prefer to keep guest accounts and not get rid of them. I have no objections but if guests accounts cause so much trouble I think we should consider also simplify and get rid of it. People do have to create an account to buy on Amazon and it doesn't prevent them from buying. If we keep the possibility to put items in cart and only login at checkout, I would consider that possibility. @lin-d-hop @sauloperez @kirstenalarsen @luisramos0 what do you think about the option 2, ged rid of guest accounts? Hard to get data on that, probably there is a significant number of customers who checkout as guests today, but would they stop buying if they had to register?

[mkllnk]

@myriamboure Guess what I just found? Our preferred solution 1b is implemented already. This really affects only old user accounts that have not been associated with their customer records yet. We need to run a database migration to fix it. Here are some stats for French production:

select count(customers.*) from customers join spree_users on customers.email = spree_users.email where customers.user_id is null;
-- 319 guest customer records that should be associated to a user

select count(distinct customers.email) from customers join spree_users on customers.email = spree_users.email where customers.user_id is null;
-- 305 people affected

select count(id) from subscriptions where customer_id in (select customers.id from customers join spree_users on customers.email = spree_users.email where customers.user_id is null);
-- 0 subscriptions affected (subs total: 22)

[myriamboure]

Great @mkllnk that's a good news ! Does it mean that the user can safely create new customers and subscription for them, and when those customers decide to create an account, their subscription will be automatically attached to the new registered account? So yes we have to fix that (I know there are similar problems in UK so that migration is needed everywhere I guess) but good news is that we can tell subs user they can freely go already and setup subs, right?

[mkllnk]

Yes. This bug affects only people who signed up for a user account before May 2018.

[sineadfenton]

Subscription Payments not processed

[myriamboure]

@sineadfenton did you update to latest version in UK ? I think so, and in that case this should be fixed. Can you check with @Matt-Yorkley if the migration worked correctly on old user accounts ? Else it might be another case, open a new issue in that case.