Woocommerce-admin: Onboarding: Not enough data in tracks to determine progress

[joshuatf]

Describe the bug
In determining whether someone has started the profiler, how far they got, and if they completed it, we have a few shortcomings with our current tracking methods.

Check out the https://github.com/Automattic/looker/pull/531 branch to see more.

Start

We're currently using storeprofiler_welcome_clicked to determine if the profiler was started. This may not be accurate if the start step is skipped, however. I'm thinking using a lose query LIKE '%wcadmin_storeprofiler_% will be enough to resolve this point.

Progress/Step

Individual steps are not tracked. We previously thought that the page_view would take care of this, but it is not triggered if the path does not change, but properties do (e.g., the step property changes but we're still on the dashboard/profiler page, no event is tracked).

We should probably record an event when a step is completed.

Completion

The current check to determine completion is via the storeprofiler_store_theme_choose event. While this is currently true(ish) depending on cart items, it would be much better to have an event that actually checks when the profiler is marked complete.

To Reproduce
Steps to reproduce the behavior:

1. Already have plugins installed and connected.
2. Walk through the onboarding profiler.
3. Note you skip the welcome step.
4. Complete the profiler.
5. Check events to see if you can determine progress made.

Expected behavior
Expected to be able to derive percentage of started/completed onboarding profiles in addition to where users may be dropping off.

[pmcpinto]

Hey @joshuatf it's great to have you taking a look at our event tracking performance. I have a couple of questions:

We're currently using storeprofiler_welcome_clicked to determine if the profiler was started. This may not be accurate if the start step is skipped, however. I'm thinking using a lose query LIKE '%wcadmin_storeprofiler_% will be enough to resolve this point.

But if the user skips the step we won't trigger the usage tracking auth message, so we're not tracking them, correct? Or maybe I'm missing something here.

Individual steps are not tracked. We previously thought that the page_view would take care of this, but it is not triggered if the path does not change, but properties do (e.g., the step property changes but we're still on the dashboard/profiler page, no event is tracked).
We should probably record an event when a step is completed.

I'm assuming that a step is completed when they click "continue" or "complete/proceed". Your idea is to include an event for any pageview or something like a backend event to confirm the step completion?

While this is currently true(ish) depending on cart items, it would be much better to have an event that actually checks when the profiler is marked complete.

In this case, it would be a backend event, correct?

[joshuatf]

But if the user skips the step we won't trigger the usage tracking auth message, so we're not tracking them, correct? Or maybe I'm missing something here.

It may be possible that they've already opted in through WCS. May be a little odd, but checking for any storeprofiler_* events covers our bases here.

I'm assuming that a step is completed when they click "continue" or "complete/proceed".

I agree that this should be when an event is marked complete, however there is no event in place for this yet. I think we may have been relying on the page view tracking built into WC Admin, but unfortunately this doesn't get tracked since a "page change" doesn't actually occur (we're always on the "dashboard" page).

In this case, it would be a backend event, correct?

We could make this a backend event or frontend; both should work okay. It's mostly a matter of explicitly tracking when it happens instead of relying on the theme step as the final step.

Using the theme step to determine completion is problematic since if we ever add anything after (as we did with the cart) and if we ever move it (as we currently plan to) we can't be certain they completed the profiler. Adding an explicit storeprofiler_complete event will make our data more clear and future-proof it.

[pmcpinto]

Ok thanks for the clarification. So just to summarize, the idea is to add a loose event for the start step and a completed event for every single step in the profiler and for when all the profiler steps are completed, correct?

[joshuatf]

the idea is to add a loose event for the start step

I think for our analytics we can just check to see if any storeprofiler_ events have been tracked (i.e. any step or action has been taken in the profiler so we know they've at least seen the screen). I think tracking one individual step is a little too risky and may not give accurate results if we start users at different parts of the process.

completed event for every single step in the profiler and for when all the profiler steps are completed, correct?

Yes to both of these.

[pmcpinto]

@joshuatf it could also be useful to have something similar for when a task is completed:

wc_admin_task_completed
Event prop: task name

What do you think?

[joshuatf]

Good idea @pmcpinto - was added in https://github.com/woocommerce/woocommerce-admin/pull/3726. Closing this issue.

[pmcpinto]

@joshuatf it could also be useful to have something similar for when a task is completed:
wc_admin_task_completed
Event prop: task name
What do you think?

@joshuatf can you confirm if this event tracking was also included? I'm not seeing it on Tracks

[joshuatf]

Sorry I misread your earlier comment, @pmcpinto. New issue opened here https://github.com/woocommerce/woocommerce-admin/issues/3907