K6: PKI extension to crypto module [bounty: $650]

[robingustafsson]

We want to extend the k6 crypto module with support for PKI crypto. This will mean adding functionality to generate cryptographically strong random numbers, read/parse x.509 certificates, read/parse PEM encoded keys, signing/verifying and encrypting/decrypting data. We want to support PKCS#1 version 1.5, PKCS#1 version 2 (also referred to as PSS and OAEP), DSA and ECDSA.

Related issues:

• https://github.com/loadimpact/k6/issues/637: Use AES/ECB/PKCS5Padding in k6
• https://github.com/loadimpact/k6/issues/725: expose Crypto.getRandomValues() or nodejs' crypto
• https://github.com/loadimpact/k6/issues/822: Error on using 'crypto' functions inside browserified NodeJS module

Requirements

Besides the user-facing JS APIs detailed below a completed bounty must also include tests and docs.

Generate cryptographically strong random numbers

Proposal for JS API

import { randomBytes } from "k6/crypto";

let rndBytes = randomBytes(numBytes); // returns a byte array

Relevant links:

• https://golang.org/pkg/crypto/rand/
• https://nodejs.org/api/crypto.html#crypto_crypto_randombytes_size_callback

Parsing x.509 encoded certificates

Proposal for JS API (shorthand):

import { x509 } from "k6/crypto";

let issuer = x509.getIssuer(open("mycert.crt"));
let altNames = x509.getAltNames(open("mycert.crt"));
let subject = x509.getSubject(open("mycert.crt"));

Proposal for JS API (full):

import { x509 } from "k6/crypto";

let certData = open(“mycert.crt”);
let cert = x509.parse(certData);

The Certificate object returned by x509.parse() should return a an Object with the following structure:

{ subject:
   { countryName: 'US',
     postalCode: '10010',
     stateOrProvinceName: 'NY',
     localityName: 'New York',
     streetAddress: '902 Broadway, 4th Floor',
     organizationName: 'Nodejitsu',
     organizationalUnitName: 'PremiumSSL Wildcard',
     commonName: '*.nodejitsu.com' },
  issuer:
   { countryName: 'GB',
     stateOrProvinceName: 'Greater Manchester',
     localityName: 'Salford',
     organizationName: 'COMODO CA Limited',
     commonName: 'COMODO High-Assurance Secure Server CA' },
  notBefore: 'Sun Oct 28 2012 20:00:00 GMT-0400 (EDT)',
  notAfter: 'Wed Nov 26 2014 18:59:59 GMT-0500 (EST)',
  altNames: [ '*.nodejitsu.com', 'nodejitsu.com' ],
  signatureAlgorithm: 'sha1WithRSAEncryption',
  fingerPrint: 'E4:7E:24:8E:86:D2:BE:55:C0:4D:41:A1:C2:0E:06:96:56:B9:8E:EC',
  publicKey: {
    algorithm: 'rsaEncryption',
    e: '65537',
    n: '.......' } }

Relevant links:

• https://golang.org/pkg/crypto/x509/

Signing/Verifying data (RSA)

Proposal for JS API (shorthand version):

import { x509, createSign, createVerify, sign, verify } from "k6/crypto";
import { pem } from "k6/encoding";

// alternatively this can be called like:
// x509.parse(open("mycert.crt")).publicKey();
let pubKey = x509.parsePublicKey(pem.decode(open("mykey.pub")));
let privKey = x509.parsePrivateKey(pem.decode(open("mykey.key.pem"), “optional password”));

export default function() {
    let data = "...";

    // one of "base64", "hex" or "binary" ("binary" being the default).
    let outputEncoding = "hex";

    // for PSS you need to specify "type": "pss" and the optional "saltLength": number option, if options is empty or not passed to sign/verify then PKCS#1 v1.5 is used.
    let options = {...};

    // Signing a piece of data
    let signature = sign(privKey, "sha256", data, outputEncoding, options);

    // Verifying the signature of a piece of data
    if (verify(pubKey, "sha256", data, signature, options)) {
        ...
    }
}

[LOWER PRIO] Proposal for JS API (full version):

import { x509, createSign, createVerify } from "k6/crypto";
import { pem } from "k6/encoding";

// alternatively this can be called like:
// x509.parse(open("mycert.crt")).publicKey();
let pubKey = x509.parsePublicKey(pem.decode(open("mykey.pub")));
let privKey = x509.parsePrivateKey(pem.decode(open("mykey.pem"), "optional password"));

export default function() {
    let data = "...";

    // one of "base64", "hex" or "binary" ("binary" being the default).
    let outputEncoding = "hex";

    // for PSS you need to specify "type": "pss" and the optional "saltLength": number option, if options is empty or not passed to sign/verify then PKCS#1 v1.5 is used.
    let options = {...};

    // Signing a piece of data
    let signer = createSign("sha256", options);
    signer.update(data, [inputEncoding]);
    let signature = signer.sign(privKey, outputEncoding);

    // Verifying the signature of a piece of data
    let verifier = createVerify("sha256", options);
    verifier.update(data, [inputEncoding]);
    if (verifier.verify(pubKey, signature)) {
        ...
    }
}

Relevant links:

• https://golang.org/pkg/crypto/rsa
• https://golang.org/pkg/encoding/pem/

Signing/Verifying data (DSA)

The API would be the same for sign/verify as for RSA, the type of encryption used would be inferred by the keys used, so by the following lines:

let pubKey = x509.parsePublicKey(pem.decode(open("mykey.pub")));
let privKey = x509.parsePrivateKey(pem.decode(open("mykey.pem"), "optional password"));

Relevant links:

• https://golang.org/pkg/crypto/dsa/

Signing/Verifying data (ECDSA)

The API would be the same for sign/verify as for RSA, the type of encryption used would be inferred by the keys used, so by the following lines:

let pubKey = x509.parsePublicKey(pem.decode(open("mykey.pub")));
let privKey = x509.parsePrivateKey(pem.decode(open("mykey.pem"), "optional password"));

Relevant links:

• https://golang.org/pkg/crypto/ecdsa/

Encrypt/Decrypt data (RSA)

Proposal for JS API:

import { x509, encrypt, decrypt } from "k6/crypto";
import { pem } from "k6/encoding";

// alternatively this can be called like:
// x509.parse(open("mycert.crt")).publicKey();
let pubKey = x509.parsePublicKey(pem.decode(open("mykey.pub")));
let privKey = x509.parsePrivateKey(pem.decode(open("mykey.pem"), "optional password"));

export default function() {
    let data = "...";

    // one of "base64", "hex" or "binary" ("binary" being the default).
    let outputEncoding = "hex";

    // for OAEP you need to specify "type": "oaep" and the optional "hash": "sha256" (default) and "label": string options, if options is empty or not passed to encrypt/decrypt then PKCS#1 v1.5 is used.
    let options = {...};

    // Signing a piece of data
    let encrypted = encrypt(pubKey, data, outputEncoding, options);

    // Verifying the signature of a piece of data
    let plaintext = decrypt(privKey, encrypted, options));
}

Relevant links:

• https://golang.org/pkg/crypto/rsa

[robingustafsson]

Link to bountysource: https://www.bountysource.com/issues/69000729-pki-extension-to-crypto-module-bounty-350

[bookmoons]

Submitted a PR for randomBytes #922 to try to start this off.

[bookmoons]

When I try to make a method under crypto crypto.x509.parse(), it doesn't receive the ctx context.Context object that all the crypto methods do. Is there some switch I should be flipping to make that work?

[na--]

Can you share some of the code, since I'm not sure exactly what you're trying to do. In general, the k6 code that binds Go methods to the goja JS runtime checks whether the first argument is a context or context pointer and passes that.

[bookmoons]

Put up an example here:
https://github.com/bookmoons/k6/commit/f337350c3a5c73d0bd8c92ad96b4409a2d185879

The target JS API is like this, with a subnamespace crypto.x509:

import { x509 } from "k6/crypto";

const pem = "pem-encoded-certificate";
const certificate = x509.parse(pem);

In Go I've tried to do this by adding a struct field to Crypto. But methods under that new struct don't seem to receive the Context. It shows this error:

TypeError: Could not convert function call parameter pem-encoded-certificate to context.Context

There's a test that can be run to see the error:

go test -race github.com/loadimpact/k6/js/modules/k6/crypto

[MStoykov]

In order for the context to be bounded you need to call common.Bind (as @na-- has linked ) for crypto and all other modules it happens because their part of the index and the get bound when required,

I would recommend that you just make x509 to be k6/crypto/x509 and for people to have to import that. Although this goes against what @robingustafsson required as API, so maybe some revisiting of the proposed API is required.

You can probably hack around it with some changes to common.Bind ? Maybe having Crypto (the struct type) implement some method like Bind and it can take some ... arguments (return an error) and bind the x509 field it holds ... and than keep that one .. but I don't know what the arguments need to be :)

[bookmoons]

I would recommend that you just make x509 to be k6/crypto/x509 and for people to have to import that.

Thanks guys. If there are no objections, I'll head in this direction.

[robingustafsson]

The proposed change in API is totally fine with me.

[bookmoons]

I have something toward this, with a certificate parsing successfully. But I'm having a casing issue.

The output object ends up snake cased:

{ 'signature_algorithm': 'sha1WithRSAEncryption' }

The API wants the standard JavaScript thing of camel casing:

{ signatureAlgorithm: 'sha1WithRSAEncryption' }

Is there some way to achieve that?

[bookmoons]

Found the way to do this with tags. Good feature.

[bookmoons]

Submitted the certificate parsing piece of this in #1014.

[bookmoons]

This has become larger than I can do under the bounty amount. Not sure how valuable it is, but I'm setting a target of $650 to take it through the rest.

[robingustafsson]

@bookmoons Thanks for letting us know. I've raised the bounty to $650 now so you can complete the project.

[bookmoons]

Thank you very much.

[plambrechtsen]

I'm also interested in this bug as it would be nice if I could generate signed JWTs other than HS256. There is RS256/PS256/EC256 signing methods and many oauth flows require a RS256 or PC256 signed JWT.
https://github.com/dgrijalva/jwt-go or https://github.com/dvsekhvalnov/jose2go have all the necessary libraries to generate everything you need.

[na--]

@plambrechtsen this seems a bit out of scope for this issue. Can you create a new issue about it, linking to this issue? It would also need some evaluation, since I'm not sure which JWT parts would be better implemented as Go code and which parts should be implemented as a JS library that uses the Go crypto primitives @bookmoons has added in the PR.

[plambrechtsen]

Hi @na-- thanks for that. As you can see I just created a new issue :)

[bookmoons]

A question about data encoding. In signing for example we have this interface:

const signer = createSign("SHA256");
signer.update(data, encoding);

Elsewhere encoding is hex base64 binary. So this seems (?) to make data a binary value.

Does it make sense to also allow string messages for signing and encryption?

const message = "Sent from my private space station orbiting Jupiter."
const signature = sign(priv, "SHA256", message);

[bookmoons]

Maybe that doesn't really belong. Trying to fit decoding into decryption adds some complication. If we're treating them as primitives I guess people can handle encoding and decoding externally.

[bookmoons]

bookmoons/sign has signing for all 3 cryptosystems ready to submit when the current PR is settled.

[bookmoons]

bookmoons/encrypt has encryption ready to submit at the right time.

[na--]

Thank you, @bookmoons, for the awesome contribution! Unfortunately, as @robingustafsson has told you, we decided to wait a bit and further evaluate and benchmark some things before we proceed to expand the Go JavaScript API with the crypto functions. So, we'll close this issue in order to pay out the full bounty to you, but we'll not merge the code in https://github.com/loadimpact/k6/pull/1025 just yet.

One of the main things we want to investigate is if we can fix the handling of binary data (https://github.com/loadimpact/k6/issues/1020, https://github.com/loadimpact/k6/issues/873) before we add any further crypto functions. They heavily depend on it, and lugging plain integer arrays is probably very inefficient and definitely very cumbersome.

Also, since having a custom k6-specific crypto API won't help k6 with the adoption of JS libraries from the broader JS ecosystem that depend on either the WebCrypto API or on the Node.js crypto module, we want to benchmark and evaluate where exactly we fall short on performance when polyfills for those are used. Depending on what we find, we may end up re-writing the crypto module to be a Go polyfill for parts of the Web crypto API or something like that...