K6: Sign release binaries and installers for Windows

[na--]

As illustrated by this recent issue, we might benefit from signing our binary releases with a valid key for Windows.

To be honest, I'm not sure if that would prevent or even decrease future antivirus false-positive results, but even if we just make that "_Do you want this app from an unknown publisher to make changes to your PC_" warning dialog less scary, it's probably worth it.

Some resources on the topic:

• https://docs.microsoft.com/en-us/windows-hardware/drivers/dashboard/get-a-code-signing-certificate
• https://www.rahulpnath.com/blog/code-signing-in-azure-devops-pipeline/
• https://stackoverflow.com/questions/252226/signing-a-windows-exe-file
• https://docs.microsoft.com/en-us/windows/desktop/SecCrypto/cryptography-tools
• https://docs.microsoft.com/en-us/windows/desktop/SecCrypto/signtool

Potentially connected issues:

• Use Azure Pipelines for CI https://github.com/loadimpact/k6/issues/959:
  Since it's a Microsoft service, it's worth investigating whether there is some sort of a better way to do binary signing there. I'm somewhat worried about just uploading our future code-signing key to a random CI service, so maybe they have some sort of a key enclave or managed signing or something that adds a bit of security and transparency to the process.
• Chocolatey installation (this new PR https://github.com/loadimpact/k6/pull/1027 and the current windows installation instructions)
  Worth investigating if chocolatey, being a package manager, would actually benefit from this whole binary code-signing business. In the Linux world, it's not the binaries that are signed, but the different rpm/deb/apk/pacman/etc. packages...

[majkinetor]

Worth investigating if chocolatey, being a package manager, would actually benefit from this whole binary code-signing business

It wouldn't at this moment AFAIK. Furthermore, Chocolatey Gallery has integration with Virus Total so packages are as safe as it can get because they are not going to be approved if there are more then couple of false positives on entirety of 70+ antivirus engines.

In the case of latest version there is one false positive result:

Previous version has 2 results.

This is the list of AV's used, there is no Windows Defender tho.

[imiric]

After a couple of days of diving into the Windows certificate world :sweat_smile:, I finally managed to produce a signed binary and installer. :tada:

You can see the CI changes and download the signed MSI from here.

I opted to not use any third-party GH Actions for this because of security reasons:

• this one uses MS' signtool.exe but doesn't allow specifying a password, presumably because it expects a blank one... plus it's written in JS...
• And the other uses osslsigncode, which has a noble goal of open sourcing this functionality and allowing it to be cross-platform, but we should probably use the suggested MS tool for this to avoid any potential issues, and the windows-latest GHA image already comes bundled with signtool.exe, so we don't need the cross-platform support.

Anyway, here's how it looks like in Windows:

However, having a signed installer doesn't get rid of the "Windows protected your PC" warning dialog... :sob:

Now it shows the same screen just with the correct publisher information, whereas before it showed "Unknown publisher":

This SO answer and this linked one suggest that in order to remove this warning a publisher can either "gain" trust with SmartScreen over time, presumably by a lot of users choosing to ignore and run this signed installer anyway, but I don't see any guarantees of when or if that will eventually happen... Or, we can outright "buy trust" with an EV certificate, which the current one that Robin provided is not. These are usually much more expensive than regular certs and it's a plain scam if you ask me, but perhaps it might be worth it as a shortcut to get rid of this warning.

@robingustafsson @na-- @MStoykov What do you think?

[imiric]

Here's an idea of the time and number of installs it takes for SmartScreen to gain trust in a publisher without an EV cert:

For me, it took 16 days and some 500 to 2000 installs before SmartScreen trusted my brand-new certificate.

And the other answer for that question mentions that even an EV cert is not guaranteed to get rid of this warning, but it probably speeds things up.

Given this, I'm inclined to suggest that we start using this non-EV cert from v0.30.0 and just wait for MS to trust it. I reckon we can hit ~2K installs in a couple of weeks.

[na--]

Hmm not sure if we should buy into the MS EV certificate racket. Even this basic signing gets rid of the other "unknown publisher" warning, and puts Load Impact AB in the blue dialog in the last screenshot, so it's definitely better than nothing and may be good enough for now...

Regarding signtool.exe - I think you made the right decision. osslsigncode seems interesting and I'm glad it exists, but since we're already running on a windows VM for other reasons, we have no need to use it.