Wp-calypso: Can Sybil be used for the FAB shown in Calypso.
Currently, Sybil (which is an automated system for suggesting Support Docs to users) is only used on https://wordpress.com/help/contact. Elsewhere the FAB is used - which (AFAIK) is hard coded and therefore more time consuming to update to provide relevant answers as our product changes.
Why
- Consolidating 2 different systems for managing support doc suggestions reduces the ongoing maintenance workload
- Sybil is managed and updated by HEs - so this is something that can be done without the need for Dev work or commits to Calypso.
jordesign
All 14 comments
I'd love to understand some numbers behind this proposal. Like:
How many searches result in a customer contacting Happiness with Sybil on https://wordpress.com/help/contact compared to the number of searches within the FAB search popup which result in a customer contacting Happiness.
We're talking about a serious investment of time and I just want to make sure that before we go down that road that the proposed solution is measurably better than what we have now.
cc @kristenzuck @kurt213
davemart-in
on 21 Sep 2020
Thanks for the ping @davemart-in! I'll take a closer look at this later on this week and share my findings here :)
kristenzuck
on 21 Sep 2020
How many searches result in a customer contacting Happiness with Sybil on https://wordpress.com/help/contact compared to the number of searches within the FAB search popup which result in a customer contacting Happiness.
I'm not exactly sure what you're asking (and I don't know what FAB is either!), but for Sibyl:
Users click on 6000+ sibyl suggestions each month for WPCOM, and that's from it living in the contact form alone. It could certainly use some love (hasn't been updated or audited in years), but even so, it's working well - that's an impressive stat for something that only appears in one place.
How many of those end up contacting support anyway? There's no direct way to tell. We think it prevents about 5000 queries total each month. Some of those people certainly contact support later for other questions, but Sibyl helps reduce how many questions they have or clears out the easier steps in a customer question.
Where do we find FAB stats?
gracie
on 21 Sep 2020
@jordesign question for you - you mentioned this
Elsewhere the FAB is used - which (AFAIK) is hard coded and therefore more time consuming to update to provide relevant answers as our product changes.
What's an instance of when that was problematic? I'm wondering if it would be wise to test FAB next to Sibyl and see if they're both serving relevant and time-sensitive suggestions. Right now, I can see it's serving up the editor deprecation doc when I write, "How can I write a post" into the question mark search field. Did we have to manually request that be added, or is it working intuitively? Granted, since these results show a support doc's title instead of "Are you looking for how to publish a post in the new editor?" we lose that translation into the customer's perspective and language in FAB, and I see that as a poignant downside to it. I'd love to see its stats.
If we want to expand Sibyl and see its stats are performing better than what's in FAB now, I'd like to request that PS make it a Q4 or Q1 goal to give WPCOM Sibyl a good scrub + seek out some dev help in polishing it, then pursue expanding it.
gracie
on 21 Sep 2020
I don't know what FAB is either
Sorry for using an acronym here! FAB stands for Floating Action Button and in this case we're talking about the blue help button in the bottom right corner of most screens:
I'm not exactly sure what you're asking
@gracie I guess what I'm asking is that before we start looking to incorporate Sibyl into the blue help button popup, I'd love to better understand how the Sibyl user flow compares to the blue help button popup user flow in terms of providing self-help (thus preventing the need to contact support). My guess is that this project would end up be a large or even XL project, so I'd love to just gather some data around it to help us make a more informed decision.
davemart-in
on 21 Sep 2020
@gracie I guess what I'm asking is that before we start looking to incorporate Sibyl into the blue help button popup, I'd love to better understand how the Sibyl user flow compares to the blue help button popup user flow in terms of providing self-help (thus preventing the need to contact support).
Super. I agree with you.
gracie
on 21 Sep 2020
Note: I removed @sarahcada's comment because it contained sensitive data and links, and messaged in Slack to move to a P2
apeatling
on 25 Sep 2020
Thank you, @apeatling. I appreciate it!
Reposting answers to specific questions:
I guess what I'm asking is that before we start looking to incorporate Sibyl into the blue help button popup
I'll expound on what Gracie said earlier in this thread. "Users click on 6000+ sibyl suggestions each month for WPCOM", and don't return to the contact form where their question is still in the contact form. Another stat to look at is that in August, out of 6000+ people who clicked a Sibyl suggestion, only 3000+ returned within a week for a question (either the same one they asked initially or another question entirely), which means that the other 50% probably got their answer.
For comparison of what the users experience, while both do a great job pulling up docs that may be helpful, the questions/entries crafted for Sibyl anticipate the possible ways a user will ask their question. Sibyl presents several versions of a question, lets the user pick which one they actually mean, and gives links to the most specific area we can think of so that they can get their answer quickly. For the FAB and My Home support UI, if the understandably short titles of the support docs don't make the user feel like the specific answer they're looking for isn't there, then they won't feel helped. I can share comparison screenshots and further commentary on this in a P2.
By the way, there seem to be history of the FAB using Sibyl suggestions. It's possible that the length of the questions that forced the team to change it, I鈥檓 unsure and currently unable to dig up the history in dev convos. Perhaps Dave might be more aware.
As for maintenance, there's a control area where any HE can add, edit, and delete/unapprove entries anytime without dev intervention. It would be wonderful if the FAB or the My Home support UI can have a similar area.
sarahcada
on 28 Sep 2020
Thanks for sharing @sarahcada!
there seem to be history of the FAB using Sibyl suggestions
This is news to me. Does anyone know who could be pinged for more info on this? @kristenzuck, @gracie any thoughts?
As for maintenance, there's a control area where any HE can add, edit, and delete/unapprove entries anytime without dev intervention. It would be wonderful if the FAB or the My Home support UI can have a similar area.
Yep. Understood. This makes sense!
I'd still be interested to look at funnel data between the two options. @kurt213 is this something that you'd be able to add to your list? Basically, I'm interested in knowing:
A) Of the people who view the FAB UI and perform a search, how many go on to click "Contact us" and submit a contact form or chat with us?
B) Of all the people who view https://wordpress.com/help/contact how many go on to submit the contact form?
davemart-in
on 28 Sep 2020
@davemart-in got this added to my list. Part A funnel is complete. Currently looking into calculating part B.
Quick question on part A:
how many go on to click "Contact us" and submit a contact form or chat with us?
I can see the contact form submission but can't see an option to chat? Not sure if it's a plan limitation/there is an alternative way for a user to reach chat?
kurt213
on 29 Sep 2020
@kurt213 There are plan limitations on chat (@jordesign do you know what plan is required?). There are also availability limitations. I believe this is what you should see if chat is available:
davemart-in
on 29 Sep 2020
@davemart-in @kurt213 Currently an active paid plan is required (and relies on chat availability). This is changing shortly (to Premium or higher) but you should end up with access with any existing paid plan
jordesign
on 7 Oct 2020
There are plan limitations on chat (@jordesign do you know what plan is required?). There are also availability limitations. I believe this is what you should see if chat is available:
Currently an active paid plan is required (and relies on chat availability). This is changing shortly (to Premium or higher) but you should end up with access with any existing paid plan
Additionally if you are an admin on someone else's site with a paid plan, you can get chat access, there are also presales and precancellations.
kristenzuck
on 7 Oct 2020
I've completed an initial funnel analysis of these two journeys pcr1an-9-p2
A) Of the people who view the FAB UI and perform a search, how many go on to click "Contact us" and submit a contact form or chat with us?
B) Of all the people who view https://wordpress.com/help/contact how many go on to submit the contact form?
kurt213
on 8 Oct 2020
Related issues
rickybanister
路
3Comments
ehg
路
3Comments
lamosty
路
3Comments
jeherve
路
3Comments
aduth
路
3Comments
Most helpful comment
I'm not exactly sure what you're asking (and I don't know what FAB is either!), but for Sibyl:
Users click on 6000+ sibyl suggestions each month for WPCOM, and that's from it living in the contact form alone. It could certainly use some love (hasn't been updated or audited in years), but even so, it's working well - that's an impressive stat for something that only appears in one place.
How many of those end up contacting support anyway? There's no direct way to tell. We think it prevents about 5000 queries total each month. Some of those people certainly contact support later for other questions, but Sibyl helps reduce how many questions they have or clears out the easier steps in a customer question.
Where do we find FAB stats?