Foundation.mozilla.org: [User Testing] Simple User Testing

[taisdesouzalessa]

Based on our conversation from the design meeting, we will be performing a simple user testing in our site. After that we will gather our leanings and apply to see the results.

Here is a list generated on the meeting:

• [x] a. Usertesting.com requirements
• [x] b. Matthew to get access again
• [x] c. Make a short list of specific things to test … sketch basic naive test
• [x] d. Pick something simple (focus on the quality of the test)

I broke down (c. and d.) in a series of steps:

• [x] 1. What we want to learn about? Goal of the research (everything revolves around that)
• [x] 2. Depending on 1: we can choose moderated or unmoderated tests Usertesting.com just allows unmoderated tests.
• [x] 3. What’s the audience? Who will we test (prepare for screener questions)?
• [x] 4. Draft the screener and the user testing questions
• [x] 5. Get feedback from Becca and Matthew / iterate on user testing questions
• [x] 6. Do a pilot test with friends/family to make sure the questions are clear.
• [x] 7. Do the real test on Usertesting.com
• [x] 8. Gather videos/analyze data
• [ ] 9. Iterate based on test
• [ ] 10. Measure results (before and after) test - can we detect an improvement in any metric?
• [ ] 11. Share case study with our team and other teams

Anything else you feel it's missing? @xmatthewx @kristinashu @beccaklam @natalieworth @sabrinang

[taisdesouzalessa]

The goal of this presentation is to highlight options for testing and to decide which one to choose. Feel free to add more options too, just keep the same formula (add to the presentation) for easier comparison to existing proposals.

https://docs.google.com/presentation/d/1Sm4yeKbhMFGfOKUje57ONoNzUP9GCGJOzsHP11Xe9D8/edit?usp=sharing

Next step: after we decide on this item, we move on to drafting screener and user test questions.

[kristinashu]

Excellent @taisdesouzalessa! I've added in my vote.

[beccaklam]

This is great @taisdesouzalessa ! I added my vote as well :) Would love to help with designing the tests too!

[taisdesouzalessa]

Thanks for the votes there Kristina and Rebecca. I met with Matthew and we are going for the "Proposal #3: Campaigns". Yay!

Here is the initial draft for the script: https://docs.google.com/document/d/1WaY_aUoNOw2z7cmQ5u_D6kpawp1Iybk_WsHAr0WtF2A/edit?usp=sharing

@beccaklam please take a look there, add comments and modify things you think it makes sense, let's build this test together! @xmatthewx any feedback there is welcomed! After we refine the script, Becca and I could do a pilot test with friends/family to make sure the questions are solid before we move on with the real test.

[taisdesouzalessa]

After feedback from Matthew and Becca (thank you both!), here is the new version of the script:
https://docs.google.com/document/d/1WaY_aUoNOw2z7cmQ5u_D6kpawp1Iybk_WsHAr0WtF2A/edit?usp=sharing

@beccaklam @xmatthewx please let me know what you think of this new version, more focused on the task we want to learn more about and based on what we currently have on the page.

I also tweaked the goal of the proposal number 3 (slide deck) to reflect our latest dicussions.

[taisdesouzalessa]

@beccaklam @kristinashu @xmatthewx I sent you an email through usertesting.com so you can play the test in your computers. I am not sure how this sharing feature works so please let me know if you run into any issues.

To see the test, go to:

1. Select Tasks tab
   

1. Preview Test on the left sidebar (you'll need to install the UserTesting Recorder first, the pop up will guide through that)
   

Let me know what you think (for the campaigns user testing screen, I used a screenshot of the page after petition is signed and applied it to a CMS template, so users can click on the menu and the little banner after the main copy). The only thing that is not faithful to the real page is that the "share" buttons don't stay fixed on top (and they are not clickable) but since we are not testing them, I don't think it is relevant.

[beccaklam]

Hey @taisdesouzalessa, the test looks good :) Great job designing it, it's so exciting to see your hard work coming to fruition. My only feedback is to simplify the language a bit and to make it more explicit that the user is landing on a post-sign petition page. I'll leave some suggestions in your google doc draft.

[kristinashu]

This is great Tais! Super simple. We'll have to celebrate after the first test!

[taisdesouzalessa]

Tested the test with a non-Mozillian (step 6) and made the necessary tweaks. Test is ready to launch 🚀.

@xmatthewx I can't place this order at the moment. Could you please place the order or give me the authorization to do so? We are testing only in 1 person at the moment, correct?

[taisdesouzalessa]

We received a video from 1 user! I added notes to the test and asked designers to add their own notes so we can compare them and move on to next stage.

[taisdesouzalessa]

NEXT STEPS (related to item 9 of the list: "iterate on test").

We should perform the same test again, with tweaks on the script and screening.

From conversation with Matthew:
• Filter for a less tech savvy user at screening (reason: tech savvy users are very familiar with the hamburger menu and they can easily find info that less tech savvy users may not find).
• Let’s be less leading (reason: we felt that, by asking people "where would you find more info about Mozilla" we are kind of leading them to search for more info instead of allowing them to surface their interest on learning more).

• After they “complete petition”, but before we direct them to find somewhere “learn more about Mozilla” … let’s be more open ended.
  1) “Now that you’ve completed the survey, are you curious about anything? Where might you look next? (talk out loud, don’t click just yet)”
  2) Now go there, describe what you see.
  3) Go back to the petition page. If you wanted to learn more about Mozilla, where would you look? (if you already did this in the prior, tell us where else you might go...)

We should also take a look at the comments designers left on the UserTesting platform to do tweaks based on what was felt after analyzing the video

[kristinashu]

@beccaklam as Tais' UX partner, I'm going to assign this to you. Can you please review it and see if anything else needs to be done?

[beccaklam]

@kristinashu Looks like next steps are to test again and this time be less 'leading' in our questions with a less tech savvy audience. Wondering if we should test now or later?

[beccaklam]

Ideas:

• To further test campaigns retention we could ask what users would do next on the post sign page and see where they end up
• We could decide where specifically we want users to go after the post sign page and design a flow to see if the users follow it
• Analyze where users are coming from and where they drop off and test to see where users go on those drop off pages

[beccaklam]

Closing this ticket for now. Summary ticket here.