Nixpkgs: Firefox 1548973 Hotfix (addons disabled due to expired intermediate certificate)

[deliciouslytyped]

Installing the hotfix:

I didn't want to turn on user studies and it didn't seem to be pulling the hotfix immediately anyway, so I did some digging and found the hotfix installer at https://normandy.cdn.mozilla.net/api/v1/recipe/.

Search for "name": "Hotfix: Update XPI signing intermediate [Bug 1548973]", and then install the xpi from the addonUrl field. I don't know how supported this method is but it worked for me. I cannot guarantee user support, so if you can't figure things out yourself/don't consider yourself a power user it's probably a bad idea to rely on this That said, strictly on topic issues (this isn't really a mozilla issue tracker) will be handled on a best effort basis.

Example url: https:// storage.googleapis.com /moz-fx-normandy-prod-addons/extensions/[email protected]

Edit: see later in thread for some vague clarification from Mozilla about community workarounds, which FWIW is more or less a reasonable under the circumstances. Likely to attempt to reduce support load. TL;DR: caveat emptor. (nothing has changed)

Edit2: Fixed typo: addonIUrl -> addonUrl

Edit3: For those coming to this issue from other sources: Nixpkgs currently runs firefox 66. For those on an older version where the normandy xpi does not work (unsure if user error, or version incompatibility - have not tested it myself) a more involved workaround that is mentioned in the issue tracker is to extract the certificate from the XPI and manually import it into the certificate store. E.g. https://bugzilla.mozilla.org/show_bug.cgi?id=1549078#c71

[deliciouslytyped]

The issue can be followed at https://bugzilla.mozilla.org/show_bug.cgi?id=1548973 (see also various other links hanging off this as "Depends On" and "Blocks", linked in the comments, etc )
->

• https://discourse.mozilla.org/t/certificate-issue-causing-add-ons-to-be-disabled-or-fail-to-install/39047/15
• https://twitter.com/mozamo

[alexxbb]

Thank you so much! I've been looking for this!

[The-SamminAter]

Thanks, it worked!

[LYY]

It worked for me, Thanks!

[deliciouslytyped]

~Slight~ clarification of support status https://bugzilla.mozilla.org/show_bug.cgi?id=1549078#c36

Edit: correction: https://bugzilla.mozilla.org/show_bug.cgi?id=1549061 is linked somewhere, and so is https://blog.mozilla.org/addons/2019/05/04/update-regarding-add-ons-in-firefox/

The latter says: "There are a number of work-arounds being discussed in the community. _These are not recommended as they may conflict with fixes we are deploying_. We’ll let you know when further updates are available that we recommend, and appreciate your patience. (May 4, 15:01 EST)"

Therefore my suggestion is to periodically check the issues for additional information, caveat emptor, etc etc.

[Nuclearfarts]

THANK YOU. the way Mozilla has been handling this is stupid. this exists, why not give people the link instead of asking them to browse unprotected for SIX FUCKING HOURS

[gak1234]

Almost 18 hours unpatched, here. I couldn't enable studies - toggling the checkbox made NO change to prefs.js so studies are vaporware on my install for some reason. Just give me the stinkin' patch, Mozilla.

[07416]

Installing the hotfix:

I didn't want to turn on user studies and it didn't seem to be pulling the hotfix immediately anyway, so I did some digging and found the hotfix installer at https://normandy.cdn.mozilla.net/api/v1/recipe/.

Search for "name": "Hotfix: Update XPI signing intermediate [Bug 1548973]", and then install the xpi from the addonIUrl field.

* addonUrl field

[deliciouslytyped]

Fixed typo.

[deliciouslytyped]

Added edit mentioning a second workaround method that may or may not work if the XPI doesn't work.

[captainepoch]

How can I uninstall this once Mozilla fixed the issue? I don't see anything at about:addons or about:studies...

[deliciouslytyped]

Oh, that's a good question. I just assumed it would show up in about:addons - but it makes some sense that it doesn't. I'll see if I can find anything since I will need that myself.

[captainepoch]

Oh, that's a good question. I just assumed it would show up in about:addons. I'll see if I can find anything since I will need that myself.

For clarification, I have Studies and Telemetry disabled in my profile

[deliciouslytyped]

The addon seems to be listed in about:debugging but that does not seem to provide a removal interface.

[captainepoch]

The addon seems to be listed in about:debugging but that does not seem to provide a removal interface.

I deleted the .xpi from the profile directory and it seems to remove the add-on, but I'm not sure if it's removed completly or not... And it seems to not store data, as far as I can see.

Edit 1:
As far as I can tell, once you run the .xpi file you can delete from your profile, everything seems to keep working.

Also, I downloaded the .xpi and unzip it. I see it injects a cert and then forces a re-verification of the add-ons' signature.

The file api.js has the data, so... Copy and paste the file content here, maybe someone with more experience can tell more about this:

/* eslint no-unused-vars: ["error", { "varsIgnorePattern": "skeleton" }]*/
ChromeUtils.defineModuleGetter(this, "XPIDatabase", "resource://gre/modules/addons/XPIDatabase.jsm");

var skeleton = class extends ExtensionAPI {
  getAPI(/* context */) {
    return {
      experiments: {
        skeleton: {
          async doTheThing() {
            // first inject the new cert
            try {
              let intermediate = "MIIHLTCCBRWgAwIBAgIDEAAIMA0GCSqGSIb3DQEBDAUAMH0xCzAJBgNVBAYTAlVTMRwwGgYDVQQKExNNb3ppbGxhIENvcnBvcmF0aW9uMS8wLQYDVQQLEyZNb3ppbGxhIEFNTyBQcm9kdWN0aW9uIFNpZ25pbmcgU2VydmljZTEfMB0GA1UEAxMWcm9vdC1jYS1wcm9kdWN0aW9uLWFtbzAeFw0xNTA0MDQwMDAwMDBaFw0yNTA0MDQwMDAwMDBaMIGnMQswCQYDVQQGEwJVUzEcMBoGA1UEChMTTW96aWxsYSBDb3Jwb3JhdGlvbjEvMC0GA1UECxMmTW96aWxsYSBBTU8gUHJvZHVjdGlvbiBTaWduaW5nIFNlcnZpY2UxJjAkBgNVBAMTHXNpZ25pbmdjYTEuYWRkb25zLm1vemlsbGEub3JnMSEwHwYJKoZIhvcNAQkBFhJmb3hzZWNAbW96aWxsYS5jb20wggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQC/qluiiI+wO6qGA4vH7cHvWvXpdju9JnvbwnrbYmxhtUpfS68LbdjGGtv7RP6F1XhHT4MU3v4GuMulH0E4Wfalm8evsb3tBJRMJPICJX5UCLi6VJ6J2vipXSWBf8xbcOB+PY5Kk6L+EZiWaepiM23CdaZjNOJCAB6wFHlGe+zUk87whpLa7GrtrHjTb8u9TSS+mwjhvgfP8ILZrWhzb5H/ybgmD7jYaJGIDY/WDmq1gVe03fShxD09Ml1P7H38o5kbFLnbbqpqC6n8SfUI31MiJAXAN2e6rAOM8EmocAY0EC5KUooXKRsYvHzhwwHkwIbbe6QpTUlIqvw1MPlQPs7Zu/MBnVmyGTSqJxtYoklr0MaEXnJNY3g3FDf1R0Opp2/BEY9Vh3Fc9Pq6qWIhGoMyWdueoSYa+GURqDbsuYnk7ZkysxK+yRoFJu4x3TUBmMKM14jQKLgxvuIzWVn6qg6cw7ye/DYNufc+DSPSTSakSsWJ9IPxiAU7xJ+GCMzaZ10Y3VGOybGLuPxDlSd6KALAoMcl9ghB2mvfB0N3wv6uWnbKuxihq/qDps+FjliNvr7C66mIVH+9rkyHIy6GgIUlwr7E88Qqw+SQeNeph6NIY85PL4p0Y8KivKP4J928tpp18wLuHNbIG+YaUk5WUDZ6/2621pi19UZQ8iiHxN/XKQIDAQABo4IBiTCCAYUwDAYDVR0TBAUwAwEB/zAOBgNVHQ8BAf8EBAMCAQYwFgYDVR0lAQH/BAwwCgYIKwYBBQUHAwMwHQYDVR0OBBYEFBY++xz/DCuT+JsV1y2jwuZ4YdztMIGoBgNVHSMEgaAwgZ2AFLO86lh0q+FueCqyq5wjHqhjLJe3oYGBpH8wfTELMAkGA1UEBhMCVVMxHDAaBgNVBAoTE01vemlsbGEgQ29ycG9yYXRpb24xLzAtBgNVBAsTJk1vemlsbGEgQU1PIFByb2R1Y3Rpb24gU2lnbmluZyBTZXJ2aWNlMR8wHQYDVQQDExZyb290LWNhLXByb2R1Y3Rpb24tYW1vggEBMDMGCWCGSAGG+EIBBAQmFiRodHRwOi8vYWRkb25zLm1vemlsbGEub3JnL2NhL2NybC5wZW0wTgYDVR0eBEcwRaFDMCCCHi5jb250ZW50LXNpZ25hdHVyZS5tb3ppbGxhLm9yZzAfgh1jb250ZW50LXNpZ25hdHVyZS5tb3ppbGxhLm9yZzANBgkqhkiG9w0BAQwFAAOCAgEAX1PNli/zErw3tK3S9Bv803RV4tHkrMa5xztxzlWja0VAUJKEQx7f1yM8vmcQJ9g5RE8WFc43IePwzbAoum5F4BTM7tqM//+e476F1YUgB7SnkDTVpBOnV5vRLz1Si4iJ/U0HUvMUvNJEweXvKg/DNbXuCreSvTEAawmRIxqNYoaigQD8x4hCzGcVtIi5Xk2aMCJW2K/6JqkN50pnLBNkPx6FeiYMJCP8z0FIz3fv53FHgu3oeDhi2u3VdONjK3aaFWTlKNiGeDU0/lr0suWfQLsNyphTMbYKyTqQYHxXYJno9PuNi7e1903PvM47fKB5bFmSLyzB1hB1YIVLj0/YqD4nz3lADDB91gMBB7vR2h5bRjFqLOxuOutNNcNRnv7UPqtVCtLF2jVb4/AmdJU78jpfDs+BgY/t2bnGBVFBuwqS2Kult/2kth4YMrL5DrURIM8oXWVQRBKxzr843yDmHo8+2rqxLnZcmWoe8yQ41srZ4IB+V3w2TIAd4gxZAB0Xa6KfnR4D8RgE5sgmgQoK7Y/hdvd9Ahu0WEZI8Eg+mDeCeojWcyjF+dt6c2oERiTmFTIFUoojEjJwLyIqHKt+eApEYpF7imaWcumFN1jR+iUjE4ZSUoVxGtZ/Jdnkf8VVQMhiBA+i7r5PsfrHq+lqTTGOg+GzYx7OmoeJAT0zo4c=";
              let certDB = Cc["@mozilla.org/security/x509certdb;1"].getService(Ci.nsIX509CertDB);
              certDB.addCertFromBase64(intermediate, ",,");
              console.log("new intermediate certificate added");
            } catch (e) {
              console.error("failed to add new intermediate certificate:", e);
            }

            // Second, force a re-verify of signatures
            try {
              XPIDatabase.verifySignatures();
              console.log("signatures re-verified");
            } catch (e) {
              console.error("failed to re-verify signatures:", e);
            }
          }
        }
      }
    };
  }
};

[wideocean]

+1
This "fix" described here works for me too.
However, there is no obvious way to uninstall it?

I guess that when the official patch rolls out, we can uninstall Firefox completely and just reinstall it, right?

[deliciouslytyped]

Ok so I couldn't find the original source code for the XPI, would be nice if someone could post where to find it, but it's basically readable if you extract the xpi. I don't know what the content.js does, otherwise this is the code:

/* eslint no-unused-vars: ["error", { "varsIgnorePattern": "skeleton" }]*/
ChromeUtils.defineModuleGetter(this, "XPIDatabase", "resource://gre/modules/addons/XPIDatabase.jsm");

var skeleton = class extends ExtensionAPI {
  getAPI(/* context */) {
    return {
      experiments: {
        skeleton: {
          async doTheThing() {
            // first inject the new cert
            try {
              let intermediate = *omitted*
              let certDB = Cc["@mozilla.org/security/x509certdb;1"].getService(Ci.nsIX509CertDB);
              certDB.addCertFromBase64(intermediate, ",,");
              console.log("new intermediate certificate added");
            } catch (e) {
              console.error("failed to add new intermediate certificate:", e);
            }

            // Second, force a re-verify of signatures
            try {
              XPIDatabase.verifySignatures();
              console.log("signatures re-verified");
            } catch (e) {
              console.error("failed to re-verify signatures:", e);
            }
          }
        }
      }
    };
  }
};

Some similar patches can be found linked from https://bugzilla.mozilla.org/show_bug.cgi?id=1549061

Basically, I guess it looks safe to just delete the addon as stated by @captainepoch but I'm not a firefox dev so YMMV.

Edit: Oops, some duplicated work. :)

[deliciouslytyped]

Specifically, relevant discussion can be read at https://phabricator.services.mozilla.com/D29940 , I have not read through it.

[deliciouslytyped]

https://news.ycombinator.com/item?id=19828631 links https://news.ycombinator.com/item?id=19827428 links http://kb.mozillazine.org/Uninstalling_extensions#Uninstalling_manually suggests the manual deletion method is fine. Who knows how up to date that article is though.

[brunoais]

@captainepoch
let intermediate = that's a certificate encoded in base64.
let certDB = This is internal API. In this case, it is loading the module that handles certificates.
certDB.addCertFromBase64( Adding the base64 encoded certificate to the browser's certificates.
XPIDatabase.verifySignatures(); Regenerating the certificates cache.

You can see both comments. They explain what is being done in less detail.

[Jake5319]

The googleapi link to intermediate hotfix as in OP, pops window "normandy.cdn.mozilla.net Firefox prevented this site from asking you to install software on your computer"

Then if I resend url I get another popup "The add-on could not be downloaded because of a connection failure"

Any ideas?

Edit: here is complete url with ++ h++ps://storage.googleapis.com/moz-fx-normandy-prod-addons/extensions/[email protected]

browser console shows 1557070136986 addons.xpi WARN Download of https://storage.googleapis.com/moz-fx-normandy-prod-addons/extensions/hotfix-update-xpi-intermediate%40mozilla.com-1.0.2-signed.xpi failed: [Exception... "Certificate issuer is not built-in." nsresult: "0x80004004 (NS_ERROR_ABORT)" location: "JS frame :: resource://gre/modules/CertUtils.jsm :: checkCert :: line 163" data: no] Stack trace: checkCert()@resource://gre/modules/CertUtils.jsm:163
onStopRequest()@resource://gre/modules/addons/XPIInstall.jsm:2078

[deliciouslytyped]

You must have something different with your configuration and/or have something causing the site to be untrusted. I don't know, sorry. There seem to be ways to get past the prevention of installation from untrusted sites but I don't feel comfortable recommending that. You need to figure out why your Firefox doesn't trust the path.

Edit: there seem to be other people on the internet with your issue.

I guess you could download the XPI manually with curl or wget or something (right click -> save link as?), I'm just looking at your browser very suspiciously. This is basically bypassing a check it's doing and is unhappy about for some reason.

Edit 2: As long as you haven't disabled signature verification I guess it's fine.

[deliciouslytyped]

It looks like firefox doesn't like the certificate the addon is signed with for some reason?

[Jake5319]

Thank you stefano-m!!!

[pauljt]

Just in case it helps future visitors, the fix (same solution as the hotfix) has already shipped now (about 3 hours ago in 66.0.4). See release notes for details and known outstanding issues: https://www.mozilla.org/en-US/firefox/66.0.4/releasenotes/

[vcunat]

The master and 19.03 branches have Firefox 66.0.4 now, so I assume hotfixes aren't needed anymore.

[camy011]

Just wanted to throw this fix out there since I am on an older version of Firefox 56.0.2 which does not currently have a fix. You can manually import the certificate found in the xpi. To do this you will want to download the xpi from the below link possibly with another browser like IE if you ware having trouble.
https:// storage.googleapis.com /moz-fx-normandy-prod-addons/extensions/[email protected]
Rename as .zip or open with a zip program.
Navigate to experiments\skeleton and open api.js in a text editor.
Copy everything inside quotes after let intermediate = " until the next "
Create a new text file in notepad and add the certificate header/trailer like so and paste the value in between.

-----BEGIN CERTIFICATE-----
MIIHLTCCBRWgAwIBAgIDEAAIMA0GCSqGSIb3DQEBDAUAMH0xCzAJBgNVBAYTAlVTMRwwGgYDVQQKExNNb3ppbGxhIENvcnBvcmF0aW9uMS8wLQYDVQQLEyZNb3ppbGxhIEFNTyBQcm9kdWN0aW9uIFNpZ25pbmcgU2VydmljZTEfMB0GA1UEAxMWcm9vdC1jYS1wcm9kdWN0aW9uLWFtbzAeFw0xNTA0MDQwMDAwMDBaFw0yNTA0MDQwMDAwMDBaMIGnMQswCQYDVQQGEwJVUzEcMBoGA1UEChMTTW96aWxsYSBDb3Jwb3JhdGlvbjEvMC0GA1UECxMmTW96aWxsYSBBTU8gUHJvZHVjdGlvbiBTaWduaW5nIFNlcnZpY2UxJjAkBgNVBAMTHXNpZ25pbmdjYTEuYWRkb25zLm1vemlsbGEub3JnMSEwHwYJKoZIhvcNAQkBFhJmb3hzZWNAbW96aWxsYS5jb20wggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQC/qluiiI+wO6qGA4vH7cHvWvXpdju9JnvbwnrbYmxhtUpfS68LbdjGGtv7RP6F1XhHT4MU3v4GuMulH0E4Wfalm8evsb3tBJRMJPICJX5UCLi6VJ6J2vipXSWBf8xbcOB+PY5Kk6L+EZiWaepiM23CdaZjNOJCAB6wFHlGe+zUk87whpLa7GrtrHjTb8u9TSS+mwjhvgfP8ILZrWhzb5H/ybgmD7jYaJGIDY/WDmq1gVe03fShxD09Ml1P7H38o5kbFLnbbqpqC6n8SfUI31MiJAXAN2e6rAOM8EmocAY0EC5KUooXKRsYvHzhwwHkwIbbe6QpTUlIqvw1MPlQPs7Zu/MBnVmyGTSqJxtYoklr0MaEXnJNY3g3FDf1R0Opp2/BEY9Vh3Fc9Pq6qWIhGoMyWdueoSYa+GURqDbsuYnk7ZkysxK+yRoFJu4x3TUBmMKM14jQKLgxvuIzWVn6qg6cw7ye/DYNufc+DSPSTSakSsWJ9IPxiAU7xJ+GCMzaZ10Y3VGOybGLuPxDlSd6KALAoMcl9ghB2mvfB0N3wv6uWnbKuxihq/qDps+FjliNvr7C66mIVH+9rkyHIy6GgIUlwr7E88Qqw+SQeNeph6NIY85PL4p0Y8KivKP4J928tpp18wLuHNbIG+YaUk5WUDZ6/2621pi19UZQ8iiHxN/XKQIDAQABo4IBiTCCAYUwDAYDVR0TBAUwAwEB/zAOBgNVHQ8BAf8EBAMCAQYwFgYDVR0lAQH/BAwwCgYIKwYBBQUHAwMwHQYDVR0OBBYEFBY++xz/DCuT+JsV1y2jwuZ4YdztMIGoBgNVHSMEgaAwgZ2AFLO86lh0q+FueCqyq5wjHqhjLJe3oYGBpH8wfTELMAkGA1UEBhMCVVMxHDAaBgNVBAoTE01vemlsbGEgQ29ycG9yYXRpb24xLzAtBgNVBAsTJk1vemlsbGEgQU1PIFByb2R1Y3Rpb24gU2lnbmluZyBTZXJ2aWNlMR8wHQYDVQQDExZyb290LWNhLXByb2R1Y3Rpb24tYW1vggEBMDMGCWCGSAGG+EIBBAQmFiRodHRwOi8vYWRkb25zLm1vemlsbGEub3JnL2NhL2NybC5wZW0wTgYDVR0eBEcwRaFDMCCCHi5jb250ZW50LXNpZ25hdHVyZS5tb3ppbGxhLm9yZzAfgh1jb250ZW50LXNpZ25hdHVyZS5tb3ppbGxhLm9yZzANBgkqhkiG9w0BAQwFAAOCAgEAX1PNli/zErw3tK3S9Bv803RV4tHkrMa5xztxzlWja0VAUJKEQx7f1yM8vmcQJ9g5RE8WFc43IePwzbAoum5F4BTM7tqM//+e476F1YUgB7SnkDTVpBOnV5vRLz1Si4iJ/U0HUvMUvNJEweXvKg/DNbXuCreSvTEAawmRIxqNYoaigQD8x4hCzGcVtIi5Xk2aMCJW2K/6JqkN50pnLBNkPx6FeiYMJCP8z0FIz3fv53FHgu3oeDhi2u3VdONjK3aaFWTlKNiGeDU0/lr0suWfQLsNyphTMbYKyTqQYHxXYJno9PuNi7e1903PvM47fKB5bFmSLyzB1hB1YIVLj0/YqD4nz3lADDB91gMBB7vR2h5bRjFqLOxuOutNNcNRnv7UPqtVCtLF2jVb4/AmdJU78jpfDs+BgY/t2bnGBVFBuwqS2Kult/2kth4YMrL5DrURIM8oXWVQRBKxzr843yDmHo8+2rqxLnZcmWoe8yQ41srZ4IB+V3w2TIAd4gxZAB0Xa6KfnR4D8RgE5sgmgQoK7Y/hdvd9Ahu0WEZI8Eg+mDeCeojWcyjF+dt6c2oERiTmFTIFUoojEjJwLyIqHKt+eApEYpF7imaWcumFN1jR+iUjE4ZSUoVxGtZ/Jdnkf8VVQMhiBA+i7r5PsfrHq+lqTTGOg+GzYx7OmoeJAT0zo4c=
-----END CERTIFICATE-----

Rename the file extension to .crt
In Firefox navigate to Options -> Privacy & Security -> Certificates -> View Certificates -> Authorities -> Import...
It will show up under Mozilla Corporation as signingca1.addons.mozilla.org

[vcunat]

56 surprises me. I'd expect 52 ESR to be more likely (we even still have it in latest nixpkgs), but both are unmaintained upstream and thus unlikely to receive official fixes.

[brunoais]

@camy011 Quite curious here.
What things prevent you from using the latest firefox? Is it some extension that is still not ported to webextension and no good alternative?

[vcunat]

@brunoais: I remember threads about this: here and perhaps here.

[brunoais]

@vcunat That's what I needed. Thanks. Makes sense.

[camy011]

@camy011 Quite curious here.
What things prevent you from using the latest firefox? Is it some extension that is still not ported to webextension and no good alternative?

Originally I didn't upgrade because I didn't want my addons disabled. The goal being to allow enough time for addons to update and alternatives to become available. I actually went through and installed the latest FF side-by-side yesterday and got everything working to the point that I don't need my old copy.

Some addons that don't have an alternative that I was using though:
Classic theme restorer
Menu Wizard (hide unused/reorder menu options)
Roomy bookmarks toolbar

Most of what I wanted from these addons can be done using the userChrome.css so it doesn't matter too much. Tabs on bottom, moving "Select All" and "Copy" to the top of the right click menu, hiding context menu options (if I care enough to get around to it), hiding text for bookmarks to fit more on the bar.

I also used classic theme restorer to move my file menu to my bookmarks bar to help compact the browser. While there isn't an easy way to do that it looks like I can move my bookmarks toolbar to my file menu bar which gives almost the same effect (top vs middle row).

[hargobind]

In pre-Quantum FF (versions 56, 55, etc.), I tried following some of the instructions in this thread, but didn't have success with extensions working again.

Finally I found this thread on Reddit which basically goes through the same process mentioned by @camy011 about extracting and installing the certificate, but it also includes the last important step about running a command needed to re-verify the signatures of all add-ons.

[vcunat]

It appears even fix for 52 ESR is planned, though I still can't see a new 52.x release on the download site.

For users who cannot update to the latest version of Firefox or Firefox ESR, we plan to distribute an update that automatically applies the fix to versions 52 through 60.