Caseflow: Determine frontend analytics approach (discussion on 6/28)

[amprokop]

What kind of analytics should we capture across our apps, and what tools should we use? We currently have Google Analytics running, but it's seldomly used at the moment and not set up with our React SPAs.

[NickHeiner]

Reader is having similar discussions about this as well. @abbyraskin, would you like to merge our discussion with this one? Or maybe we need to figure out what we want on our own first?

We should at least have Google Analytics. It's trivial to implement. We'll get information like page views for free. We can also track specific events, although I think a tool like Mixpanel is better suited to in-depth user tracking. I'm guessing that getting Mixpanel approved would be a huge lift, though.

Then there are things like Snowplow, which provide detailed, fine-grained, custom analytics. Snowplow is open source, and we could implement it entirely on our own in AWS. That can be a lot of work, though, and probably is not worth the effort, given that we have direct personal access to many users.

[abbyraskinUSDS]

I'm fine with turning the conversation into a broader team meeting if we think it'll be useful. The design team is also discussing the topic this week. Is this Wisdom Wednesday Worthy?

I think the answer to the original question posed on this thread will depend a bit on the individual projects, but it would be great to align on Caseflow-wide goals, tools, lessons learned/things to avoid (e.g., storing PII in GA), and maybe some general guidelines around implementation.

[joofsh]

I think we can safely capture the product info we need within GA alone (avoiding PII). I don't see enough value in switching to something like Snowplow.

Things that I believe should track for all Caseflow products:

• all url changes as the user navigates within mini-SPAs (dispatch is lacking this for sure)
• any sort of core submit success/failure events

Beyond that, I think what events we'll want to track will vary between products. Reader may have very specific things they want to track like how often a user selects more than 1 category filter. That's useful product information and doesn't contain any PII. For products like Certification & Dispatch that have a flow, it might be valuable to track how often users click to go back a page.

[NickHeiner]

Also, what about Hotjar? I have no idea if it's good; I just found it from the ads on the lodash docs.

[avrildavid]

@amprokop - what google analytics do we already have in place for the Certification V2 NYRO pilot? Is there anything else that should be added?

cc @wturne @laurjpeterson

[Chingujo]

Designers have also been documenting some product questions in the following design issue, so far @abbyraskin for Reader and @shellicious for Certification.

#205 Product Questions for Analytics

[NickHeiner]

Other things that may be useful, if we can use them:

• https://heapanalytics.com/
• https://www.kissmetrics.com

[Chingujo]

My two cents in case its helpful.

Kissmetrics offers more refined digital marketing (conversions, marketing funnels and etc). Also good for tracking paths across multiple devices (I think all of our users are desktop). When I used both it was usually google analytics to store session data and Kissmetrics to do more detailed DM analysis.

Where most of the questions were basic user, session (URL and paths to error messages that JD mentioned), and UI data, mostly did Google Analytics into a reporting tool - thinking more downstream on how we use data - like Analytics Canvas, Tableau. Or an even more simpler heat map tool like CrazyEgg for UI related insights.

Heap is more of an end to end solution that bypasses the above need by offering event visualization and built in reporting. Only used this for a few months but looked promising because you don't have to jump through hoops for more advanced tracking and lets you push existing dataLayers. Implementation wise they give a pre built script that can be modified as necessary like GA.

"Free Solutions"

Not sure if pricing is a concern so the "Free" softwares solutions so far listed would be:

Google Analytics/GTM - Free
Heap Analytics - free for 5,000 session and 50k sessions for badge
Hotjar - Basic at 2,000 page views per day

[NickHeiner]

@Chingujo have you worked with Mixpanel?

[Chingujo]

@NickHeiner

I have but not an expert. Sat through a demo and created tracking plans for it. Implementation for websites in DM context.

Some things to consider:

Pros:

• [ ] You can run it with Google Analytics (JS doesn't interfere or shouldn't when I did both)
• [ ] Event tracking emphasis over page views. The nice thing is that it is pretty hands off
• [ ] Works with Ruby (thought haven't done technical implementation)
• [ ] "Codeless tracking feature" literally just install the JS library and enabling auto track - then tracks everything (should be checked)
• [ ] Free version

Cons:

• [ ] Steeper learning curve (not the most friendly UI)
• [ ] There is a property limit for UI interactions
• [ ] Sometimes slow
• [ ] Not that great at tracking percentage behavioral changes
• [ ] Free version is only 60 days of historical data

[NickHeiner]

Ok, thanks for those insights.

[NickHeiner]

The in-person discussion for this will happen directly after standup on Friday, June 9th.

[amprokop]

@NickHeiner @Chingujo — did this happen?

[NickHeiner]

Not yet. We were waiting for @cmgiven to get back. I'm going to reschedule it now that he's here.

[nickheiner-usds]

Discussion Notes

• Algorithmically identifying which docs are most important, for instance, is something we can look at after having a baseline of data
• It would be nice to have finer-grained analytics than GA provides. It would be nice to see the full user journey instead of events with less context.

Questions we want to be able to answer

• For certification, why are people cancelling?
• https://github.com/department-of-veterans-affairs/appeals-design-research/issues/205
• Behavior based on role (judge, lawyer, etc)
• @abbyraskin @gnakm please post more here?

Problems with GA

• Data is locked up in GA
• No PII can go to GA
• Build for a consumer website, not a tool like ours

Plan going for forward

• Use data from app db (Chris + product people)
• Make Google Analytics work (Shane)
• If there are still questions we can't answer, look into something more complicated. (product people)