Bedrock: [Experiment] Global FxA CTA on www for Fx users

[ejregithub]

Description: Implement a test showing a Fx Account CTA in the Global Navigation to users visiting mozilla.org w. a Firefox browser instead of the current 'Download Firefox' CTA.

manapage

[hoosteeno]

@alexgibson can you review the implementation ideas in the experiment plan and let me know what questions you have?

@sghoseWI can you review this experiment plan and let me know how we can make it stronger?

[hoosteeno]

I met with Alex, Craig and Mark to discuss. This is almost ready to pull into delivery.

[alexgibson]

Some implementation notes based on our discussion yesterday:

• We're going to use a simple "Get a Firefox Account" button in place of the download button in the main navigation. This will link off to accunts.firefox.com with some utm params specific to this experiment (params TBD).
• If there's space, we will also include a small "What is this?" type link underneath the button, that links to the /firefox/accounts/ page, where people can learn more about the benefits of a having an account.
• This "Get a Firefox Account" button will be shown to all Firefox users, irrespective of being signed-in to an account or not. We may follow up with a further experiment in the future if this proves successful, to show a third state for signed-in users. For the purposes of this experiment, that third state is not needed.
• Because we're keeping this experiment simple and lightweight with just two states, running this experiment site wide on all pages shouldn't be difficult.
• The button will be shown only to Firefox desktop visitors. iOS and Android users (who can't use the FxA signup flow), will still see the regular download button.

[alexgibson]

Based on the above, I think I can get started on this now, and we can sweat the design and copy details when this is up on demo.

[alexgibson]

I pushed an WIP version of this to demo, using existing strings and regular button styles.

https://bedrock-demo-agibson.oregon-b.moz.works/en-US/

The FxA button text reads "Create a Firefox Account" and the small link reads "Learn More", but we can choose any copy we decide works best.

For Protocol pages, the FxA button uses the standard secondary product themed button (blue outline):

For Pebbles and Sandstone pages, the FxA button uses the older green outline secondary button, to remain consistent with thoe pages:

Open questions:

• Are we OK with this design wise? (button colors, link placement)
• Are we OK with the current strings (Is there any shorter text we can use?)
• Do we need a control group for this experiment, what should that be?
• What FxA link params do we need to pass over to accounts.firefox.com?
• What GA data attributes do we need on the FxA button?
• What GA data attributes / params do we need for the small "Learn More" link?
• Do we need to worry about replacing the download button in the sticky sub navigation, on pages such as /firefox/?

/cc @hoosteeno @claychaffin @mrkwvr

[sghoseWI]

@hoosteeno -- the experiment plan generally looks great to me. Only suggestions I have are adding in a sample size estimate needed in each treatment group and the control group (along with the corresponding desired confidence/error rate levels). Also would be good to add what specific countries/states we are going to run this in.

http://www.raosoft.com/samplesize.html

^ We can use this to estimate the sample size needed in each group -- and also get an idea of how long we need to keep the experiment open. I can get these numbers for us -- I just need to know what the overall population for our experiment is. Am I correct in thinking that this is the sum number of users in the specific countries/states we are experimenting in?

[alexgibson]

I'm actually curious about what the control group should do here, is there a benefit of having a control group for this experiment? The default experience (which is usually the control) is just to show a regular download button, but this is an entirely different action to signing up for an account. I guess it depends what we want to get out of this experiment. If we want to know that a) this FxA cta is effective and b) downloads don't really suffer overall, then can we know both of these things without a control group?

[sghoseWI]

Hi @alexgibson, my thinking is that we would want the control to be whatever standard behavior we want to at least continue to meet (i.e. number of downloads or account sign ups) -- after adding the FxA cta and controlling for whatever independent variables we are interested in factoring in. I think we would still need a control group for this experiment so we can have a baseline and statistically analyze if there is a significant change observed in our experiment from that baseline after the feature change is implemented. This should hopefully help us understand if the FxA cta is effective -- while testing to make sure there is no meaningful decline in our important core metrics like number of account sign ups.

[hoosteeno]

@sghoseWI I agree. Can you go ahead and update the experiment plan with your recommendations for segment size and treatment duration, as well as anything else you think makes it more measurable?

@alexgibson and @sghoseWI, regarding instrumentation...

1. for the prior FxA experiment on /new, we instrumented the form with a couple GTM parameters (https://github.com/mozilla/bedrock/pull/6173/files#diff-60e3efec8b76a589e39ecba1d079edafR8). Can you specify what those should be so we can track this easily in GA?

2. For amplitude, what do you think about the following params:
   entrypoint=mozilla.org-globalnav
   utm_source=www.mozilla.org
   utm_medium=referral
   utm_campaign=globalnav
   utm_content=[urlencoded button copy]

[ejregithub]

Let’s be sure we pass this by Alex Davis, folks on the Accounts team.

[vincejoyiv]

Hey @alexgibson here's 2¢ from a design point of view.

First, the learn more link is pretty tight to the mega menu when it's open.

I'd recommend a couple things: Make the font size a little smaller, align it to the right edge of the button, nudge it up a few pixels.

Here's what I did to achieve that result

Second, on the /firefox pages, its creating a large vertical gap between the main nav and the subnavigation below

Adjusted version looks like this

[vincejoyiv]

@alexgibson @hoosteeno One more thing to note: On this page, the learn more link doesn't do anything, which can be a frustrating user experience. I don't really have a solution for this yet, but if this moves past being just an experiment we'll have to address it.

[alexgibson]

@alexgibson @hoosteeno One more thing to note: On this page, the learn more link doesn't do anything, which can be a frustrating user experience. I don't really have a solution for this yet, but if this moves past being just an experiment we'll have to address it.

I think the most appropriate solution here is to hide the button and link on this page, as the page manages it's own state for prompting people to sign up. We don't want to compete with that and create a confusing UX.

[hoosteeno]

I think the most appropriate solution here is to hide the button and link on [benefits] page

Agree. I'll add to mana.

[hoosteeno]

I've consulted with Alex Davis and Lauren Niolet, who are the stakeholders listed in the experiment plan, and I feel comfortable proceeding with this experiment once we've got all our implementation details worked out.

[sghoseWI]

@hoosteeno -- those Amplitude parameters looks good to me.

[alexgibson]

Ok, here are the updates made so far based on feedback:

• I've hidden the download button / FxA cta on the /firefox/accounts/ page.
• I've added utm params to the FxA button for Amplitude (as per @hoosteeno's suggestions)
• I've updated based on design feedback from @vincejoyiv.
• I've included the proposed copy courtesy of @claychaffin.

I've pushed this all to demo: https://bedrock-demo-agibson.oregon-b.moz.works/en-US/

From a design perspective, my only feedback is that the "Check out the Benefits" small link feels _really_ out of place here. The navigation was not really designed around having anything more than a single button top right, and the addition of this link makes everything feel more cluttered and noisy, no matter how much we tweak at it. Interested in @vincejoyiv's thoughts here.

Things still to do:

• I need to know what GA dataLayer.push() calls to make for each variation of the experiment (control and FxA button). Because this is a site wide experiment, we can't really use versioning based on URL params (e.g. ?v=1, ?v=2), as this would create a large amount of noise, impact overall site performance, and also risk interfering with other single-page experiments. I'll need to store a cookie instead to determine which version a visitor should see.
• I still need to configure the experiment, and for this I need to know what % of Firefox traffic should be etered into each cohort.

[vincejoyiv]

@alexgibson I agree 100% with your analysis of the design re: that "benefits" link. I was working under the impression that it was a requirement for the experiment, but I'm here to question that.

I'd be interested in 86ing the link and seeing what happens. Do folks click on the button? and do they sign up for accounts? if not, maybe we try _just_ a link as a future experiment.

My personal hypothesis is that folks fail to sign up for an account because they don't understand what it is. Not so much because they can't figure out where or how to do so.

@hoosteeno I'd be interested to hear your thoughts here

[alexgibson]

Today's updates:

• Implemented experiment cohorts:
  
  
  
  • 50% of Firefox users the the FxA button
  
  
  • 50% of Firefox users see the download button.
  
  
• Each cohort has a dataLayer.push() associated with it in GA:
  
  
  
  • FxA cohort:
    
    window.dataLayer.push({ 'data-ex-variant': 'accounts-button', 'data-ex-name': 'global-fxa-button' });
  
  
  • Download button cohort:
    
    window.dataLayer.push({ 'data-ex-variant': 'download-button', 'data-ex-name': 'global-fxa-button' });
  
  
• Added data attributes to the small link underneath the FxA button:
  
  
  
  • data-link-type="nav"
  
  
  • data-link-name="Check out the Benefits".
  
  
• Experiment respects Do Not Track (DNT).

I've pushed this all to demo. You'll need to test in a Firefox browser with DNT disabled to be entered into the experiment. Depending on which cohort you get we store a cookie (1 day expiration), so if you would like to test the other cohort you'll need to clear that cookie and refresh.

https://bedrock-demo-agibson.oregon-b.moz.works/en-US/

[alexgibson]

Marking this as blocked until we get final experiment params.

[sghoseWI]

Just chatted with @alexgibson -- confirming here that I think it would be wise for us to leave the experiment running for 1 week. We only need ~17,000 sessions in both the treatment and control group -- which we can reach very quickly given daily traffic numbers. But allowing the experiment to run for a week will allow us to control for hourly/daily differences observed over the course of the experiment days by randomly sampling from more robust session data. 50% traffic to both groups will be more than enough -- but if there is a compelling reason to reduce the experimental group traffic that I'm not aware of, it shouldn't be a problem as we should still have sufficient samples to analyze post-experiment.

[alexgibson]

Thanks @sghoseWI - I'm going to mark this experiment as unblocked and we'll move foreward with the parameters defined in https://github.com/mozilla/bedrock/issues/6629#issuecomment-454400854

@hoosteeno if you have any feedback on the above please add it, else we'll move forward with merging this once it has been code reviewed.

[hoosteeno]

This looks great to me.

[sghoseWI]

Based on some new information I've received-- requesting that a hold be placed on this card for now (meeting with @hoosteeno to discuss today).

For more context:

• Talking to Daniel, he thinks that the control group should be users with the FxA download button on the global nav instead of existing FxA CTAs elsewhere on the site.

• The argument is that in this experiment, we need to make sure that both the treatment group and the population group have the same underlying population. And that we are trying to test whether the FxA CTA produces more business value than the original downloads button for existing users — not whether this specific FxA button is converting at a higher rate than other FxA ctas to different populations across the site (or through channels like first run)
  - Also, the 34% conversion rate on firstrrun is way too high of a success target
  - Daniel thinks that even a small % of account sign ups generated through the FxA CTA button is a win vs. the status quo (so maybe our benchmark can be the 6% downloads conversion rate from the existing download button)

• On a separate point, I'm seeing that there are roughly ~1.5M daily en-us sessions for firefox users per day in GA. Now that I have a bit better understanding of the site, I actually think that there is no need to direct 50% of traffic to the experimental group for a full week. We can just do 1% over the week and we will have plenty of data to randomly sample from to get representative results. This way we can reduce unnecessary risk/costs associated with too large of an experimental group if things don't go quite as planned.

Some outstanding points to consider:

• Are the majority of users who are unnecessarily re-downloading firefox doing so from global nav or the FxA CTA lower on the page? (if the lower one, then maybe we should focus another experiment on just that population)
• Do we need to experiment on only those with the most recent version of firefox? Because existing users get an upgrade browser cta when opening old versions of Firefox anyways… so we think that we could arguably get more business value by directing all firefox users on the page to the FxA CTA (and for that reason, maybe we should expand the scope of the experiment to all existing firefox users — not just most recent version)
• Should the global nav bar with the downloads/FxA CTA button be persistent while scrolling? As of now it disappears upon scrolling — this can potentially be another useful experiment too.
  
  
  
  • Finally, are we instrumenting the experiment such that we can properly track not just clicks on the FxA CTA button — but also whether clicking that button leads to ACTUAL account sign-ups afterwards?
  
  

[sghoseWI]

@hoosteeno and I talked through each of the above points and we have a plan forward

@alexgibson, here's what we decided as it pertains to experiment parameters:

• Let's still run the experiment for the entirety of 1 week as originally planned
• Instead of 50% traffic assigned to the experimental group, let's assign 10%
• The control group will now be those users in en-us with a current browser (the remaining 90% of relevant traffic) who see the original downloads button. This will ensure we are comparing the same populations across groups.

Let me know of any questions -- otherwise, I think we can safely remove the hold once the experiment parameters are adjusted. I'll update the wiki to reflect these points.

[alexgibson]

@sghoseWI @hoosteeno this sounds good to me, thanks.

One question: If we only want to target current Firefox users, how do we class "current"? Do we just include the current release version, or are we also interested in people on pre-release version as well? (Beta, Dev Edition, Nightly).

I think this is my last question before we can move forward.

[sghoseWI]

My understanding is we just want to target the current release version. Thanks @alexgibson!

[hoosteeno]

@alexgibson we can target >= 57, but it looks like you've coded it to target latest release, and that gives us plenty of data, so that works.

[alexgibson]

This is blocked on code review.