Bedrock: [Experiment] "How To Install" content on /thanks page

[hoosteeno]

Mana Page
https://mana.mozilla.org/wiki/pages/viewpage.action?pageId=90734715

Current copy

| Estimates | |
| ------------- | ------------- |
| TL | 3 days |
| PM | 1 hour |
| ENG | 5 days |
| SEO | |
| ANA | |
| DES | 1/2 day |
| COPY | 5 days |

We should add instructional content explaining to Chrome/Windows downloaders what to do next, in order to _install_ their new copy of Firefox.

We should measure whether the download:install ratio changes for people who see this messaging.

We should compare the install rate to two other segments: Chrome/Windows downloaders who get the current experience (a newsletter signup form) and Chrome/Windows downloaders who get a generic "thanks" message (which was the treatment prior to the newsletter signup form).

---

:yellow_heart: Success Criteria :yellow_heart:

• [ ] A percentage of Chrome/Windows downloaders see some clear instructions for how to install Firefox; a percentage of Chrome/Windows downloaders see the current newsletter signup form; a percentage of Chrome/Windows downloaders see a generic "thanks" message
• [ ] We can measure the install rate of these segments of downloaders and can see installs happening from each segment

:heavy_exclamation_mark: Risks :heavy_exclamation_mark:

• [ ] We may see that the FxA signup rate goes down

Tasks

• [ ]

[claychaffin]

The copy deck can be found here: https://docs.google.com/document/d/1TYsgHFCMgj7xQiveNRAf-m73KH-c9DBYCt6jMPdC-aM/edit?usp=sharing

[alexgibson]

From an experiment & copy perspective, this seems very nearly complete enough to start work on.

One remaining question: do we need any design input here on how the copy should be presented? Are we showing any images to accompany the information, or do we think copy alone is enough for this experiment?

[alexgibson]

I'm going to make a start on this, whilst we iron out the last details.

[alexgibson]

A WIP of this is now up on demo:

URL: https://bedrock-demo-agibson.oregon-b.moz.works/en-US/firefox/download/thanks/

The experiment targets en-US and Chrome only, with each variation set to 33% of traffic.

• https://bedrock-demo-agibson.oregon-b.moz.works/en-US/firefox/download/thanks/?v=a (control)
• https://bedrock-demo-agibson.oregon-b.moz.works/en-US/firefox/download/thanks/?v=b (install copy)
• https://bedrock-demo-agibson.oregon-b.moz.works/en-US/firefox/download/thanks/?v=c (app store buttons, aka no-newsletter)

Each variation fires a unique dataLayer.push() on page load:

    dataLayer.push({
        'data-ex-variant': 'control',
        'data-ex-name': 'new-fx-download-thanks-install'
    });

    dataLayer.push({
        'data-ex-variant': 'install',
        'data-ex-name': 'new-fx-download-thanks-install'
    });

    dataLayer.push({
        'data-ex-variant': 'app-store-buttons',
        'data-ex-name': 'new-fx-download-thanks-install'
    });

@sghoseWI @hoosteeno let me know if this looks ok, or if you would like any adjustments. Thanks

[sghoseWI]

@alexgibson this looks great to me! Just made some edits to the Mana page to reflect the changes that were made.

[vincejoyiv]

@claychaffin a note on the copy as it stands in the demo: The phrase "Firefox installer" doesn't actually in the Chrome download bar. In my version of Chrome it says "Firefox 65.0.dmg".

Possibly just removing the quotes would be a workable solution, where the final text could be: "Finish setting up Firefox by clicking the Firefox installer in Chrome’s download bar."

See the screenshot for reference.

[vincejoyiv]

@alexgibson @claychaffin as far as a graphic treatment goes: the obvious thing to do is something like the screenshot below. However, I'd want us to be very careful, because including a screenshot of UI that is _not_ the actual UI can lead to some pretty bad results. If folks click the image expecting something to happen, it could be frustrating when nothing does.

I could spend more time on this, but I'm not sure if it's best to block the experiment to create a new image from scratch.

[claychaffin]

@hoosteeno @vincejoyiv @alexgibson If possible, I'd like to do an A/B test on the supporting copy as part of this experiment-

Proposed copy test:
Copy A: Finish setting up Firefox by clicking the Firefox installer in Chrome’s download bar.
_vs_
Copy B: Click on the Firefox installer in Chrome’s download bar to finish setting up Firefox.

We use imperative sentences like these a lot; I'd love to know which imperative sentence structure performs better- - result>action vs action>result--- in hopes of improving results on the "Thanks" page, and to learn and ideally apply it to other pages.

[alexgibson]

Ok, so:

• We have a proposed image (thanks @vincejoyiv! Can you please provide a link so i can download it?).
• We also have a new copy variation proposed by @claychaffin, which would bump this experiment up to a total of 4 variations (control, install-copy-a, install-copy-b, no-newsletter).

@hoosteeno thoughts on the above 2 items? Do you think the image is suitable enough for this experiment, and is the extra copy variation ok to add?

Edit: I do share some of @vincejoyiv's concern here in that people might try and click the image, as opposed to the actual download bar in Chrome. If we did include the image, perhaps our copy could be a little more descriptive e.g. "Finish setting up Firefox by clicking the Firefox installer in Chrome’s download bar at the bottom of the screen"?

[hoosteeno]

Re: the image, we agreed to modify to make it a simplified picture of a (Chrome/Windows) browser with an arrow pointing to the location the downloaded file will show up in the footer

Re: the copy, an additional variation seems good

@alexgibson we also talked about the need to implement stub attribution on this, since that's the only way we'll be able to measure install rates. Is it possible to trigger stub attribution without actually setting utm parameters in the URL?

[pmac]

Or perhaps if they click the image add a big down pointing arrow at the bottom left of the screen :)

[pmac]

Is it possible to trigger stub attribution without actually setting utm parameters in the URL?

Yes. We'd have to decide whether a user would be selected for this experiment in the attribution collection code, and the add some attribution value indicating their involvement and get it signed and ready for inclusion on the bouncer link when/if they get there. But I believe that's quite doable, right @alexgibson ?

[pmac]

The phrase "Firefox installer" doesn't actually in the Chrome download bar. In my version of Chrome it says "Firefox 65.0.dmg".

@vincejoyiv that's because you're on a Mac and the people in this experiment will all be on Windows and will therefore get the Stub Installer and not the full download on this page.

[alexgibson]

Yes. We'd have to decide whether a user would be selected for this experiment in the attribution collection code, and the add some attribution value indicating their involvement and get it signed and ready for inclusion on the bouncer link when/if they get there. But I believe that's quite doable, right @alexgibson ?

I had not envisaged doing any custom stub attribution code for the basis of this experiment. That's decidedly more work than just letting the attribution code "do it's thing", but if we really wanted to avoid using UTM params on the URL, then anything is possible yeah.

[alexgibson]

Ok, based on our team time discussion yesterday I've re-architected this experiment to the following:

Target audience: Chrome visitors on Windows only.
Target locale: en-US only.
Base URL: https://bedrock-demo-agibson.oregon-b.moz.works/en-US/firefox/new/

There are now a total of 4 variants, set to 25% of traffic each:

• https://bedrock-demo-agibson.oregon-b.moz.works/en-US/firefox/new/?v=a
• https://bedrock-demo-agibson.oregon-b.moz.works/en-US/firefox/new/?v=b
• https://bedrock-demo-agibson.oregon-b.moz.works/en-US/firefox/new/?v=c
• https://bedrock-demo-agibson.oregon-b.moz.works/en-US/firefox/new/?v=d

Landing on any of the variants above, I've routed around our usual stub attribution code to make a custom auth request with the following values (this avoids using same site UTM params):

{
    utm_source: 'www.mozilla.org',
    utm_medium: 'download_button',
    utm_campaign: 'download_thanks_page',
    utm_content: 'download_thanks_install_experiment-v-{a|b|c|d}',
    referrer: document.referrer
}

I guessed these values for now, but we can set them to anything we like.

Each of the above variant URLs will also link directly to our 4 /thanks/ page variations:

• https://bedrock-demo-agibson.oregon-b.moz.works/en-US/firefox/download/thanks/?v=a
• https://bedrock-demo-agibson.oregon-b.moz.works/en-US/firefox/download/thanks/?v=b
• https://bedrock-demo-agibson.oregon-b.moz.works/en-US/firefox/download/thanks/?v=c
• https://bedrock-demo-agibson.oregon-b.moz.works/en-US/firefox/download/thanks/?v=d

I still need to thoroughly test all this logic, but I'm floating it early here to get feedback on how this shapes up in order to meet our needs.

[alexgibson]

@hoosteeno @sghoseWI ^

[hoosteeno]

Thanks @alexgibson, great work. So someone who lands in /new with en-US Windows Chrome gets split to one of 4 variants of the page, and their variant determines the message they'll see and the attribution parameters we'll set.

Do those variants also have different GA event instrumentation? I can imagine someone wanting to be sure that the variations each drove appx. the same number of clicks, since we'll be wanting to assert something about any difference in the number of installs.

I have a bit of additional logic to suggest:

if utm_content is missing or empty { 
  add our experiment utm_content parameter // this is the only one we *really need* in order to identify these sessions, and it's often empty
  if any other utm parameters are missing or empty {
    add our other experiment parameters // all 4 have to be present to get into stub attribution
  }
}
else {
  don't clobber any utm parameters if we can't modify utm_content
}

Something like this would ensure that we can measure the maximum number of sessions, without clobbering any of them. Let me know if you want to talk it out.

@sghoseWI, Alex's work above is a good, expedient way to distribute this experiment, but it will reduce the overall experiment population slightly since some part of our target audience does not download on the /new page. You may wish to update the experiment plan to anticipate only that part of our target audience that downloads after landing on /new.

[alexgibson]

@hoosteeno make sense to me, I'll work on incorporating these changes 👍

[alexgibson]

Ok, I've updated the logic based on @hoosteeno's suggestions above:

The experiment will only run if all of the following conditions are met:

• User is on Windows
• User is on Chrome
• utm_content param is not already present when landing on the /new URL.

If the above three conditions are met OK, the user is redirected to one of the 4 variants. Each of those variants will:

• Make a stub attribution auth request, explicitly setting only the utm_content value required by our experiment. All other existing utm_* values will be honored, only falling back to our experiment parameters if not present in the URL.
• Once that is done, a GA dataLayer.push() event is then also made:

window.dataLayer.push({
    'data-ex-name': 'download_thanks_install_experiment',
    'data-ex-variant': 'v-' + variant
});

@hoosteeno does this match what you propose?

Note: re clobbering existing stub attribution data. This will only be avoided for people landing directly on /new with UTM params in the URL. If someone landed on a different page (e.g. the homepage) and then navigated to /new before downloading, they would have already made a stub attribution request, and we would still overwrite their data. In theory though this should be ok (i think), since the check for utm_content should filter those people out.

Perhaps this is acceptable, but another option could be to check for the existence of a stub attribution cookie, and only make a custom request if that cookie does not exist. Let me know what you think.

[alexgibson]

Marking this is blocked until we have final imagery.

[alexgibson]

Also noting, if we want to run this experiment on /new/ for en-US, this experiment is likely blocked by the Win10 taskbar funnelcake finishing (https://github.com/mozilla/bedrock/issues/6586)

[ejregithub]

@alexgibson - according to Su Hong in the bug for the funnelcake, the experiment has more than enough (about 157,534) enrollees. It looks like the test was flipped OFF about 5 days ago. We can now remove that and TC).

[alexgibson]

PR to remove the funnelcake test: https://github.com/mozilla/bedrock/pull/6796

[mrkwvr]

@alexgibson sketch file with new illustration
Firefox Hub (How-To-Install variation on -thanks @ a97ed0a).sketch.zip

[claychaffin]

@hoosteeno Hey Justin, Vince and I refined and added "Need help... ?" copy to the copy doc, so please be sure to take a look. @mrkwvr @alexgibson @vincejoyiv

[hoosteeno]

We've had some new guidance about how experimentation relates to FxA signups, and I want to bring those into this experiment. I'm sorry for the late change -- things are moving fast.

Let's go with 3 variations: The control, the install copy (copy variation A only), and the app-store buttons aka no-newsletter.

control: 50%
install copy A: 25%
app-store: 25%

Why?

• @sghoseWI says a 25% split will give us enough data in appx. a week to be confident in the results, which is fast enough.
• Re-allocating copy variation B's traffic to the newsletter signup minimize the FxA impact of this experiment. We can test the hypothesis of copy variation B another time.

I've updated the experiment plan with these changes. Thanks.

[alexgibson]

Ok, I've reworked the experiment like so:

There are now three variants:

• https://bedrock-demo-agibson.oregon-b.moz.works/en-US/firefox/download/thanks/?v=a (control 50%)
• https://bedrock-demo-agibson.oregon-b.moz.works/en-US/firefox/download/thanks/?v=b (install copy 25%)
• https://bedrock-demo-agibson.oregon-b.moz.works/en-US/firefox/download/thanks/?v=c (app-store 25%)

For the install copy variant I have:

• [x] Updated to the latest copy.
• [x] Added the browser illustration that Mark provided
• [x] Added the help link as suggested by Clay and Vince.

With all of that, I think we're good to move forward here and get this into code review.

Does that sound good to you, @hoosteeno?

[hoosteeno]

Yes, this sounds great. :shipit:

[alexgibson]

Ok in testing on demo, I'm happy with the current code. Flagging this for code review.

[ejregithub]

@amychurchwell - please pick up this review when ready.

[amychurchwell]

@stephaniehobson is reviewing this issue! (removing my assignment.) 👍

[sghoseWI]

Hi Team -- The screenshot below is from a report I made in GA. I think I may have found an instrumentation problem. The downloads #s for the v=c arm are way higher than either of the other two arms. Does anyone have thoughts on whether there might be an issue on the backend?

[alexgibson]

@sghoseWI I've taken a look at the experiment and as far as I can see everything seems to be functioning as expected. The sessions in your screenshot look correct too (variant a has roughly double the number of sessions compared to b and c).

I'm not sure what the "Firefox Download Counter" field is associated with here. When people land on /download/thanks/, the download is initiated automatically on page load. I suspect this some GA configuration issue. Perhaps @hoosteeno can help here?

[sghoseWI]

@alexgibson Thanks for checking - I did some more digging and after chatting with @hoosteeno, we agree with your assessment!