Html5-boilerplate: Use localStorage for Google Analytics tracking when available

[davidmurdoch]

TL;DR:

(function(b,o,i,l,e,r){b.GoogleAnalyticsObject=l;b[l]||(b[l]=
function(){(b[l].q=b[l].q||[]).push(arguments)});b[l].l=+new Date;
e=o.createElement(i);r=o.getElementsByTagName(i)[0];
e.src='//www.google-analytics.com/analytics.js';
r.parentNode.insertBefore(e,r)}(window,document,'script','ga'));
ga('create','UA-XXXXX-X',{'storage': 'none','clientId':localStorage.getItem('gaClientId')});
ga(function(t){localStorage.setItem('gaClientId',t.get('clientId'));});
ga('send','pageview');

---

The Source:
http://stackoverflow.com/questions/4502128/convert-google-analytics-cookies-to-local-session-storage/19207035?noredirect=1#19207035

Google Analytics Docs:
https://developers.google.com/analytics/devguides/collection/analyticsjs/domains#disableCookies

We could use Modernizer.localstorage to check for localStorage support and fallback to cookies if its not available. Though I'm not sure if we want to lock in Modernizr as a dependency.

Why?
Because Google doesn't need to send their cookie to your server for every single request to your domain (or theirs, for that matter).

[alrra]

Though I'm not sure if we want to lock in Modernizr as a dependency.

Maybe it would be best to just add it to the docs ?

Also, ping @mathiasbynens.

[drublic]

Thanks for optimizing the snipped, David. As @alrra I think we are good with adding it to the docs.

[davidmurdoch]

The credit doesn't belong to me; this was brought to my attention by @elmerbulthuis. Though I wouldn't really consider this an optimization of the _snippet_ itself, per se --- it is more of an optimization of the web as a whole :-p.

I wonder how many bytes could be saved, globally, if everyone adopted the localStorage solution.

[mathiasbynens]

I’m obviously a big fan of this solution. The only problem with including it by default in the boilerplate is the one @davidmurdoch mentioned: we need to feature-test for localStorage first. This can be done by using Modernizr or by adding a small piece of standalone code but either way it’s going to slightly increase the page size. Then again, it will save many bytes in the long run, since no cookies will be sent in the request headers for any resources on the affected domain.

[mathiasbynens]

Something like this:

(function(b,o,i,l,e,r){b.GoogleAnalyticsObject=l;b[l]||(b[l]=
function(){(b[l].q=b[l].q||[]).push(arguments)});b[l].l=+new Date;
e=o.createElement(i);r=o.getElementsByTagName(i)[0];
e.src='//www.google-analytics.com/analytics.js';
r.parentNode.insertBefore(e,r)}(window,document,'script','ga'));
(function(){var a=(function(){var c=new Date,b;try{
localStorage.setItem(c,c);b=localStorage.getItem(c)==c;
localStorage.removeItem(c);return b&&localStorage}catch(d){}}());
ga('create','UA-XXXXX-X',a?{storage:'none',clientId:a.gaId}:{});
ga(function(b){a.gaId=b.get('clientId')});ga('send','pageview')}());

(It uses the localStorage feature test taken from http://mathiasbynens.be/notes/localstorage-pattern.)

It seems to work fine after some quick tests. I’ll create a PR after testing this more extensively. (Help welcome, of course!)

[davidmurdoch]

FYI: this has the potential to save about 33 raw bytes (headers/cookies aren't compressed) per round-trip for each request to the affected domains.

@mathiasbynens' current inline-feature-detect solution is 130 compressed bytes∗ larger (obviously this will be different for each unique page, but it gives us a rough idea). So, we should probably see if we can golf this down a bit more.

I'd personally like to see the gzipped diff down to a 65 bytes and will give it a shot myself soon. :-)

_∗using this deflator: http://www.vervestudios.co/projects/compression-tests/snippet-deflator_

[davidmurdoch]

318 GZIPped bytes (our current version is 248 GZIPped bytes):

(function(l,e){GoogleAnalyticsObject='ga',(window.ga||(ga=function(l,e){(ga.q=ga.q||[]).push(arguments)})).l=+new Date,l=document.createElement('script'),l.src='//www.google-analytics.com/analytics.js',(e=document.getElementsByTagName('script')[0]).parentNode.insertBefore(l,e);ga('create','UA-XXXXX-X',(function(l,e){try{l=(localStorage[ga.l]=ga.l)==ga.l;localStorage.removeItem(ga.l);return l}catch(l){}}())?{storage:'none',clientId:localStorage.clientId}:{});ga(function(l,e){localStorage.clientId=l.get('clientId')});ga('send','pageview')}())

This isn't tested very well, so I'll still need to do that. But its a start.

And unfortunately, the localStorage test is probably compromised in some browser somewhere, since I got rid of setItem and getItem calls and used some other golfing "tricks".

That's all I got for now. :-)

[mathiasbynens]

It just occurred to me we’ve been gzipping the snippet itself, which is kinda pointless. Gzip results depend on the rest of the document (i.e. the HTML source if it’s inlined in a document, or the rest of the JavaScript file if it’s part of one). Maybe comparing gzipped sizes of just the snippet is not the best way to measure this?

Your snippet looks nice. Good catch re-using the ga.l timestamp instead of generating a new one!

And unfortunately, the localStorage test is probably compromised in some browser somewhere, since I got rid of setItem and getItem calls and used some other golfing "tricks".

If that’s the case, it would be a dealbreaker IMHO.

We can replace document.getElementsByTagName('script')[0] with document.scripts[0] when Firefox < 9 support is not an issue anymore.

[davidmurdoch]

@mathiasbynens GZIPping just the snippet will approximate the _minimum_ byte savings from compression. So it is not entirely a moot point. In nearly all cases, the compression ratio for the snippet will increase as the page size increases.

Still need to test! I've added the getItem and setItem calls back in and managed to get it down to 309 bytes:

+function(l,e){(ct=this[GoogleAnalyticsObject='ct']||function(l,e){(ct.q=ct.q||[]).push(arguments)}).l=+new Date,l=document.createElement('script'),l.src='//www.google-analytics.com/analytics.js',(e=document.getElementsByTagName('script')[0]).parentNode.insertBefore(l,e);try{localStorage.setItem(ct.l,ct.l),l=localStorage.getItem(ct.l)-ct.l,localStorage.removeItem(ct.l)}catch(l){};ct('create','UA-XXXXX-X',l?{}:{clientId:localStorage.clientId,storage:'none'}),ct(function(l,e){localStorage.clientId=l.get('clientId')}),ct('send','pageview')}()

• I'm now using an IIFE that uses a + sign instead of wrapping parentheses.
• I'm also using localStorage.clientId instead of localStorage.gaId as clientId saves some bytes.
• Using this instead of window saved 1 more byte (in combination with moving the GoogleAnalyticsObject assignment).
• Changing ga to ct (ct is more prevalent) saved one more byte (this is probably not worth the confusion).
• Getting rid of the function call and reusing l for the localStorage check by assigning it to 0 on success saved a bunch of bytes.

Again, this needs lots more testing.

[drublic]

@davidmurdoch Any updates on the tests yet? Can we write down a test flow for this so others can help test?

[davidmurdoch]

Sorry i've been MIA, I got put on a high-priority, 6 month project and haven't been able to devote much time to anything else.

The easiest (and dumbest) way to test this is to just replace your analytics code with this new code and see if you get any odd fluctuations in numbers and browsers versions. I've done this myself and haven't seen anything that sticks out. However, I don't' have many oldie visitors anyway.

Another way would be to load this experimental analytics script in a generated iframe (so as not to interfere with the stable analytics snippet) and call _trackPageview from there, under a different GA account, of course. Then you just need to compare the data after a week or so.

I can't promise that I can work on a drop-in snippet for testing this any time soon; if someone else wants to take ownership of these ideas while I go back into hiding please go right ahead. :-)

[drublic]

I've just started a test for http://drublic.github.io/css-modal/. I got 97k page views last months but wildly spread around browser.

The numbers:

1. Chrome 44.01%
2. Firefox 34.38%
3. Internet Explorer 8.86%
4. Opera 5.26%
5. Safari 4.01%
6. Android Browser 2.22%

Let's wait and see. I got the "normal" statistics running in parallel.

Apart from that I think the code needs some more updates for readability (80 chars per line and where to insert the identifier).

I'll get back to this test in around a week.

[drublic]

I am a bit early but my findings are pretty stable at the moment. Unfortunately I see big difference in the number of visitors for both accounts.

The default implementation shows 2,964 unique visits for March 13th to 17th.
The local storage based shows 756 unique visits for the same time frame.

There might be three possible reasons:

• my implementation of the snipped is corrupted
• loading the iframe is blocked by browsers
• the local storage integration of the snipped is broken

Currently I don't see any mistakes in my code here: http://drublic.github.io/css-modal/test-gau-localstorage.html (which is the iframe that has been integrated in the site).

Also I haven't experienced iframes being blocked by browsers or pages. Has anyone an idea if this might happen?

Which leads me to the solution that the local storage GUA snipped has bugs. I have not looked into what the problems might be.
Can we develop an unminifed variant for further testing and minimize after we could a working solution?

Also I'd opt for descoping this from HTML5BP v5.0 and release it with 5.1 if we find a solution. What do you guys think?

[alrra]

Also I'd opt for descoping this from HTML5BP v5.0 and release it with 5.1

@drublic :+1: (added issue to the v5.1.0 milestone).

[tomfuertes]

If you're numbers are that off it's probably the fact you have to supply a default clientId when calling ga('create', w/ storage:'none'.

https://developers.google.com/analytics/devguides/collection/analyticsjs/domains#disableCookies

[davidmurdoch]

Just blogged about this issue on my site, here: Google Async Analytics using LocalStorage and set up a test page here: http://davidmurdoch.com/google-async-analytics-using-localstorage-test/.

Please read, share, and test.

(note: if you find any typos or errors on those pages, let me know over on twitter @pxcoach.

[philipwalton]

Hey, sorry to get here a little bit late to the party. I work on the Google Analytics team, and I wanted to comment and offer my thoughts on this issue.

First of all, I don't think it's a good idea for the H5BP project to recommend a Google Analytics tracking snippet that's functionally different from the officially recommended one. People will probably assume they're the same, and if they're actually not, it'll cause confusion. If the Google Analytics documentation claims GA supports some feature and it doesn't because someone's using a different snippet, that will probably lead to some pretty hard to debug issues (especially if H5BP doesn't make it obvious that the snippets are different).

If there's something that GA could do better, we'd love to evolve with the needs of the community rather than diverge from it. (BTW, feel free to ping or cc me on any GA related Github issues.)

Anyway, here's the main problem with localStorage and why GA doesn't offer it as the default storage mechanism:

localStorage is scoped to location.origin whereas cookies can be scoped to a top level domain. Cookie storage allows analytics.js to do subdomain tracking out of the box, and this wouldn't be possible with localStorage. In addition, if parts of your site are HTTP and other parts are HTTPS, that would also fail (and by fail I mean the storage isn't shared, so you'd lose the client ID and GA would treat it as a separate session). While it's true that these aren't concerns for most GA users, I still think it'd be bad to offer this proposed snippet as a drop-in-replacement due to the feature-loss I just described.

That being said, based on this issue and @davidmurdoch's blog post, we're going to try to prioritize building an officially supported localStorage mechanism. Currently the storage parameter only supports the options cookie and none, but we'd like to add a third localStorage option, so users who don't need subdomain or cross-scheme tracking can opt in. I don't know when this will be added, but I can update this issue when/if it is.

Does this seem reasonable to everyone?

[alrra]

@philipwalton Thanks for the comment!

Does this seem reasonable to everyone?

Cc: @davidmurdoch, @mathiasbynens

[mathiasbynens]

That being said, based on this issue and @davidmurdoch's blog post, we're going to try to prioritize building an officially supported localStorage mechanism.

:+1:

Please update the issue when this gets added. Thanks!

[davidmurdoch]

@philipwalton, :+1: Excellent news! However, you don't need to _try_ to build it, we already did! :-p (I kid, I kid).

I'll go ahead and update my blog post with this news, and create a GitHub repo with the unofficial localStorage tracking code, making sure to emphasize its shortcomings. Thanks!

[jonathantneal]

:+1: but it also seems like the future web needs some kind of topLevelStorage. Glad the option will be made available. With that in mind, and when the snippet gets in, what might the preference be for h5bp?

[davidmurdoch]

@jonathantneal, we had globalStorage in Firefox, which did cross scheme, port, and sub-domain storage. Firefox was the only one to implement it, and it has since been marked obsolete. :-(

[alrra]

@davidmurdoch Thank you so much for opening this issue and digging into it, we sincerely appreciate it!

@philipwalton Thanks again for joining the discussion, and like @mathiasbynens said, please keep us updated!

[alrra]

and create a GitHub repo with the unofficial localStorage tracking code, making sure to emphasize its shortcomings.

@davidmurdoch's repository is https://github.com/davidmurdoch/ga-localstorage (although it is not yet updated).

[davidmurdoch]

I just published the "Google Analytics using localStorage" script to npm: https://www.npmjs.org/package/ga-localstorage

The https://github.com/davidmurdoch/ga-localstorage repo has also been updated with the code.

[caesarsol]

Hello, have you read this SO comment?

http://stackoverflow.com/questions/4502128/convert-google-analytics-cookies-to-local-session-storage/19207035#comment-44767913

I'd be curious to know what you all think.

[philipwalton]

@caesarsol I think that's a really bad idea. As I described in my comment, cookies and localStorage don't have the same restrictions, so swapping them out for every single script that runs on the page is extremely risky.

[caesarsol]

hello @philipwalton, thanks for the response but maybe I explained poorly, i was referring to this comment by SO user _smhmic_:

This might violate GA TOS! Here is a secondhand quote from a GA Team member, taken from this article: "Using HTTP State Management mechanisms" (read: localStorage) "to propagate cookie state is a circumvention of our privacy safeguards. Doing so violates the Google Analytics Terms of Service". My interpretation of this is that GA employs cookies and not localStorage because more users are familiar with the concept of cookies and how to clear them; thus, GA's use of cookies is a privacy feature. – smhmic

[philipwalton]

Using HTTP State Management mechanisms" (read: localStorage) to propagate cookie state is a circumvention of our privacy safeguards. Doing so violates the Google Analytics Terms of Service

Hmmm, I don't think this is true. There are opt-out features that GA provides (e.g. Chrome extensions) that don't rely on the implementor using cookies. I think the point of this section of the TOS is that you can't create a mechanism by which someone who's using an official "do not track" extension will _still_ be tracked.

I can look into it further and I'll update this thread is my assumptions turn out to be false.

[davidmurdoch]

Update:

It is not against TOS to use localStorage to store the ClientID; it is now officially supported by Google: https://developers.google.com/analytics/devguides/collection/analyticsjs/cookies-user-id#using_localstorage_to_store_the_client_id

Note: if you have to support (extremely) old browsers (like iOS5 and FF4) their example snippet may fail (see: https://github.com/Modernizr/Modernizr/blob/master/feature-detects/storage/localstorage.js).