Wp-calypso: [SIGNUP]: move the user step from the beginning to end of the Onboarding flow

[adamziel]

What is this task about?

This is one of four tasks outlined in pa1Ax5-e-p2 that relate to reducing the dropoff in the onboarding/signup flow.

Let's create an AB test variation (50/50) that will show the user registration step (username/email/password) as the first step for one group of users and as the last step for the other group of users.

The test should run on the main onboarding flow only.

It looks like something like this has been attempted before: https://github.com/Automattic/wp-calypso/pull/30696

This task may or may not involve extensive investigation and testing:

• checking how we store the user step details
• whether or not we can add the user step _after_ the plans step.
• updating the e2e tests to ensure we only test against the current variant
• running the concept behind design/UX folks for a sanity check/more ideas

Some initial investigative research will probably be required. In that case, please post your findings to +zeldap2, and feel free to split this issue into smaller ones to assist with planning.

User impact/business justification

Currently, we lose many users at the account creation step. We want to remove as many barriers to onboarding as possible, including allowing users to preview a WordPress.com site without deciding whether to commit to creating an account.

Dependencies

• [x] Post the test details on +marketingtestingp2 and X-post +marketingacquisitionp2
• [x] Post a feature update to +wpcomhappyp2

Further reading

pa1Ax5-e-p2
https://github.com/Automattic/wp-calypso/blob/master/client/lib/abtest/README.md

[andrewserong]

Just writing up some notes / todos from some initial exploration in my local environment:

• [ ] The user step currently only shows the login button if the user step is the first step in the flow. This caused the issue Ramon flagged in the older PR: https://github.com/Automattic/wp-calypso/pull/30696#issuecomment-462560219. But we should be able to remove this positionInFlow check to re-enable it: https://github.com/Automattic/wp-calypso/blob/2597939d435780b41db2ef648020735911bfc610/client/blocks/signup-form/index.jsx#L700
• [ ] Test that re-enabling that login link doesn't break anything
• [ ] Moving the user step after the plans step means that there's an error thrown in the console on the plans step stating: Uncaught (in promise) Failed to start an authenticated Happychat session: No current user found. This means that there isn't the Question mark button in the bottom right, and users can't initiate a chat at this step, because they don't yet have a user account. For the purposes of this test, we could gracefully suppress this error, but a longer term solution might be to use a temporary userId of some kind at this point? (Just flagging that this could be explored further).

[andrewserong]

Removing the positionInFlow check displays the login URL correctly, and the generated link is: http://calypso.localhost:3000/log-in?redirect_to=http%3A%2F%2Fcalypso.localhost%3A3000%2Fstart%2Fplans — this link attempts to go back to the previous step, which isn't too terrible, and perhaps the logic there could be improved.

However, when I click on that link and log in, I'm redirected to http://calypso.localhost:3000/start/site-type. Attempting to go to http://calypso.localhost:3000/start/plans directly also redirects back to the site-type presumably because the step dependencies aren't there. Checking state.signup confirms that the progress, steps, dependencyStore are all as if we're attempting to start from scratch. 🤔

This feels like it might be intentional behaviour (e.g. logging in clears out state), so I'll investigate how logging in works, next.

For further context, it looks like when the user step was previously the last step in the sign up flow, the log in link was hidden. However, if someone enters an existing email address into the registration form, it displays a login link for them anyway, which also has the same problem as above. And I feel like having a signup flow that doesn't let you proceed with an existing an account is a deal breaker, so I think it's worth seeing if we can get the login behaviour to work correctly.

[andrewserong]

I've created a draft PR for this A/B test, and listed the caveats I've discovered so far. I think the issues I've mentioned are acceptable for the purposes of an A/B test, but would be things for us to fix if we wish to permanently move the user step to the last position. Here's the draft PR: https://github.com/Automattic/wp-calypso/pull/34770/

Interestingly the logging in / redirect issue that I encountered was a long running issue back when the user step was previously in the final position of the flow. Here are some relevant issues / PRs:

• The initial fix: https://github.com/Automattic/wp-calypso/pull/25690
• The reason the fix had to be rolled back (Redux state for sign up not persisting across logged out / logged in boundary): https://github.com/Automattic/wp-calypso/pull/25893#issuecomment-405734228
• The issue has been around since early 2017 at least: https://github.com/Automattic/wp-calypso/issues/10738

My draft PR isn't _quite_ ready for review just yet — I ran into the broken transition between flows issue again (https://github.com/Automattic/wp-calypso/issues/32506) while testing in Firefox. I'd like to get closer to the bottom of this on Monday before switching the PR over to ready for review.

Once the PR seems stable, I'll write up a P2 post on marketingtestingp2 and wpcomhappyp2 to announce the AB test. And of course, I'm happy for any feedback on the way that I'm approaching this, if anyone has any strong opinions or good ideas!

[andrewserong]

I spent a decent amount of time investigating the broken transition issue today. On the one hand, this issue would likely be resolved if we pursued changing the removeUnneededSteps logic as in: https://github.com/Automattic/wp-calypso/pull/34211, however that approach also raised further issues related to the Back button behaviour.

I think the main issue with this A/B test is to ensure that moving from the A/B test variant to any other sign up flow starts again from scratch: (e.g. the user goes to the WordPress.com homepage and selects a different button to start the signup process). I will continue to investigate the progress state and what's going on there.

Also, just a note for myself, I need to add AB Test overrides to the Woo e2e tests in addition to the ones in wp-signup-spec.js to ensure they're all using the default variant.

[andrewserong]

I've rearranged where the A/B test fires from, so that it's called in flows.js instead of flows-pure.js, which now has the e2e canaries passing on the PR. The full-desktop and full-mobile e2es are failing though, and while I don't _think_ that's because of this PR, I'll investigate further.

In looking into the BSOD issue, what's happening locally when switching A/B test values is that the defaultFlowName is swapping, and then the Signup class' componentWillMount checks what position we are in the current flow, and then redirects to the beginning of that flow. If you do the swap from the user step, this means that the user step is added to the progress state, and in an unexpected position. So when you switch flows after selecting the online-store site type, everything in the progress state is off by one index as to what's expected, and removeUnneededSteps destroys the progress state again, causing the BSOD.

An idea I have to try is to call reset on the SignupFlowController to clear out the progress state when we land at the first position in the flow. Currently I've tried doing this in the Signup controller on componentWillMount, however it looks like the user step's componentDidMount method still has time to fire and call saveSignupStep adding the user step to the progress state, causing the above issue. But I _think_ I'm getting closer!

Will keep on digging tomorrow!

[andrewserong]

Made good progress today — I fixed the BSOD issue, which was caused by the progress state having items in it in an unexpected order as described in the comment above. A simple solution is to prevent the user step from rendering altogether by setting resumingStep in the Signup component's state before registering the redirect. This will prevent the Signup component from rendering the current step, which was then calling saveSignupStep, which caused the issue.

Next up, I've been working on transferring the signup and progress state from a logged out user to a logged in user so that after a user logs in, we can pop them in the final step of the flow, instead of redirecting back to the beginning. To do this will involve _not_ clearing out the persisted state for the user in client/blocks/login when the rebootAfterLogin method is called, and then in client/state/initial-state.js load the state from the persisted logged out state if there is no existing state for the logged in user (in this case, we know that we're going from a logged out state to a logged in state, so it should be safe to transfer the data).

I've hacked this together very loosely and have it very close to working properly locally, and it will enable us to restore the fix for this logged out to logged in behaviour from last year in: https://github.com/Automattic/wp-calypso/pull/25690

Since it's all hacked together right now, tomorrow I'll work on cleaning it up and work toward a reviewable PR.

[andrewserong]

I've resolved the issue of transferring state from a logged out session to a freshly logged in session, cleaned up my code a fair bit, and it's now _tentatively_ ready for review (and any feedback if there are concerns / issues with the way I've gone about this!): https://github.com/Automattic/wp-calypso/pull/34770 — I still need to write a test to cover the changes to initial-state.js, but otherwise I think it's very close.

There's some overlap between this A/B test and the passwordless signup: https://github.com/Automattic/wp-calypso/pull/34768 — just flagging here that it would make sense for the tests to be run sequentially instead of concurrently, so I'm happy to rebase this PR against the outcome of the other test when the time comes.

[andrewserong]

Made a number of small fixes based on feedback today, and rebased against master. I'll finish off the tests I've been working on today on Monday, and left a note that I'm keen for any other feedback / concerns surrounding the fairly global changes to initial-state.js (in that it doesn't _just_ affect signup): https://github.com/Automattic/wp-calypso/pull/34770

[andrewserong]

I've split out the changes to the initial state to its own PR to merge that in before the A/B test PR (to keep that PR smaller, and reduce the potential impact of any one deployment): https://github.com/Automattic/wp-calypso/pull/34982

Once that PR lands, I'll rebase the A/B test PR and make it ready for a final round of review.

[andrewserong]

Bit of an edge case, but while testing the A/B test PR I got stock on the loading screen if I do the following steps:

1. Go to complete the /start signup flow and log in at the final step.
2. You should be redirected to the plans step.
3. Go to a flow that _doesn't_ have that particular plans step, e.g. /start/premium — I got stuck on the "Awesome" loading screen.

I'll investigate this further...

[andrewserong]

Just an update from the end of the week.

• The above issue (https://github.com/Automattic/wp-calypso/issues/34099#issuecomment-516747762) likely won't affect us for a while, due to the recent fix to update those other flows to also use the generic plans step, so I think it can be safely left for the time being.

• The changes to initial-state.js have been deployed separately this week in prep for this A/B test: https://github.com/Automattic/wp-calypso/pull/34982

• P2 posts announcing the A/B tests have been posted p8Eqe3-JD-p2 and p7DVsv-72k-p2

• e2e tests are passing on the A/B test

• Investigated the impact of the incoming import signup flow A/B test. We've decided to have that test run on only the control of _this_ A/B test, so that the option to switch to the import flow is never present on an incompatible flow.

• The remaining tasks are to rename the online-store-onboarding flow to begin with the word ecommerce so that it's compatible with the PBD behaviour of _not_ being private if the site is an online store: https://github.com/Automattic/wp-calypso/pull/34888#discussion_r309110084, and to do a final manual check / PR review.

At this stage, it looks like we should be able to merge and deploy this sometime early next week, assuming we don't run into any other issues!

[andrewserong]

This A/B test is now live. More info in: p8Eqe3-JD-p2

I'll leave this issue open and "In Progress" until the test is concluded.

[andrewserong]

This test has concluded, with control winning out. More info in p8Eqe3-JD-p2 and the A/B test has been reverted in https://github.com/Automattic/wp-calypso/pull/35552