Privacybadger: Privacy Badger runs out of storage space

[ghostwords]

Following up on https://github.com/EFForg/privacybadger/issues/1238#issuecomment-334373499 (and https://github.com/EFForg/privacybadger/issues/1588#issuecomment-332410569).

Unchecked runtime.lastError while running storage.set: QUOTA_BYTES quota exceeded

After running out of space, any changes the user or Privacy Badger itself make fail to survive browser/extension restarting. Yellowlist updates set domains to cookieblocked but then these domains revert to their previous state (blocked, allowed, or just missing from action_map). Badgers in this state learn about new domains during browsing, but forget all new learning after getting restarted.

We could add unlimitedStorage to the manifest, if it doesn't trigger new permissions warnings.

We will also greatly reduce storage usage by implementing #1446.

Need to migrate out-of-sync yellowlist entries. Anything else that we could fix via migrations?

[ghostwords]

@terrorist96 We could work around (for now anyway) the saving data issue in your Privacy Badger by removing all those pubnub.com entries (https://github.com/EFForg/privacybadger/issues/1318#issuecomment-297056299).

[ghostwords]

The following will remove all pubnub.com domains from your action_map:

(function () {
  let am = badger.storage.getBadgerStorageObject('action_map');
  for (let domain in am.getItemClones()) {
    if (window.getBaseDomain(domain) == "pubnub.com") {
      am.deleteItem(domain);
    }
  }
}());

[terrorist96]

Thanks, but I'd rather retain them for now so I can test the solution you guys come up with.

[ghostwords]

According to https://developer.chrome.com/apps/permission_warnings#nowarning, unlimitedStorage should not trigger a permissions warning. Confirmed by setting up a local update server (https://github.com/EFForg/privacybadger/pull/1532#issuecomment-318107919) and forcing an update.

[ghostwords]

@terrorist96 What are your most populous domains at this point? Is pubnub.com still way ahead of everything else?

[ghostwords]

@terrorist96 Could you run the following in your background page to (temporarily, as your storage is still maxed out) fix your yellowlist discrepancies, and then run the two queries (https://github.com/EFForg/privacybadger/issues/1238#issuecomment-334651508, https://github.com/EFForg/privacybadger/issues/1588#issuecomment-334607309) from before to see if all yellowlisted domains have gotten fixed by the following code?

(function () {
  let storage = badger.storage,
    action_map = storage.getBadgerStorageObject('action_map'),
    ylist = storage.getBadgerStorageObject('cookieblock_list');

  // (re)add all domains on the yellowlist
  _.each(Object.keys(ylist.getItemClones()), function(domain){
    ylist.setItem(domain, true);
    var baseDomain = window.getBaseDomain(domain);
    if(action_map.hasItem(baseDomain) &&
       _.contains([constants.BLOCK, constants.COOKIEBLOCK],
                  action_map.getItem(baseDomain).heuristicAction)){
      storage.setupHeuristicAction(domain, constants.COOKIEBLOCK);
    }
  });

  // reblock all blocked domains to trigger yellowlist logic
  let blocked = storage.getAllDomainsByPresumedAction(constants.BLOCK);
  for (let i = 0; i < blocked.length; i++) {
    let domain = blocked[i];
    badger.heuristicBlocking.blacklistOrigin(
      window.getBaseDomain(domain), domain);
  }
}());

[terrorist96]

@terrorist96 What are your most populous domains at this point? Is pubnub.com still way ahead of everything else?

Yep.

    "base": "pubnub.com",
    "count": 29950,

Num. 2:

    "base": "googlevideo.com",
    "count": 1548,

Ok, I ran your code. I then re-ran the two codes that you linked to and both returned undefined.

[ghostwords]

Nice, thank you! Running that code should have fixed all those yellowlisted domains in your Badger, at least until you restart your browser.

[terrorist96]

Ok cool. This way I can confirm the issue is fixed when you push out the next update. :smiley:

Edit: just confirmed. Vidible, for example was yellowlisted for me. I then restarted my browser and it's back to being redlisted now. So I now eagerly anticipate the next update so we can confirm that others that may have been affected by this issue will have the issue fixed. Thanks!

[jawz101]

I kinda want to see what all @terrorist96 's storage.js file looks like these days :) It's probably logged every privacy problem on the Internet today.

[terrorist96]

:rofl: