Distribution: Docker registry: token signed by untrusted key with ID

[kyleburnett]

I'm starting the docker registry using an image built from the following Dockerfile:

FROM registry:2

COPY config.yml cmd/registry/config.yml
RUN mkdir /usr/local/share/ca-certificates/thinknode.dev/
COPY ssl/* /usr/local/share/ca-certificates/thinknode.dev/

VOLUME ["/var/lib/registry"]
EXPOSE 5000
ENTRYPOINT ["registry"]
CMD ["cmd/registry/config.yml"]

where the ssl directory contains the files thinknode.dev.crt,thinknode.dev.key, and thinknode.dev.key. The config.yml looks like this:

version: 0.1
storage:
    cache:
        layerinfo: inmemory
    filesystem:
        rootdirectory: /tmp/registry-dev
    maintenance:
        uploadpurging:
            enabled: false
http:
    addr: :5000
    debug:
        addr: localhost:5001
auth:
  token:
    realm: https://thinknode.dev:3061/v2/token/
    service: thinknode.dev:5243
    issuer: thinknode.dev:3061
    rootcertbundle: /usr/local/share/ca-certificates/thinknode.dev/thinknode.dev.crt
redis:
  addr: localhost:6379
  pool:
    maxidle: 16
    maxactive: 64
    idletimeout: 300s
  dialtimeout: 10ms
  readtimeout: 10ms
  writetimeout: 10ms

My container is being started using the following command:

docker run -p 5000:5000 tn-registry

where I've named the docker image built using the Dockerfile above "tn-registry".

So, the issue I'm having is with the json web token. I'm trying to implement the token auth method. The auth server is written in Node and uses the npm module jsonwebtoken to generate a token. The issue seems to be coming from the way in which I am generating the fingerprint to use as the kid in the JWT header. The docker registry reports the following error.

time="2015-08-03T19:34:27Z" level=error msg="token signed by untrusted key with ID: \"34GP:OUVV:ZKGB:RTW6:6VDP:7NJR:JPVV:F7ZR:HZZZ:IKMM:VOX3:LXQQ\""

This happens when I try to issue the docker login command. Naturally, the Docker client reports a 401 Unauthorized error from the registry.

Others who have written Docker registry seem to be using the docker/libtrust library. I need something similar for Node - specifically, something like the KeyID function that converts a key into the fingerprint that the docker registry is expecting (see here). Is there such a thing?

If not, then I need to figure out whether my code is doing the right thing (though I suspect that it is not based on what the Docker registry is reporting):

var crypto = require('crypto');
var forge = require('node-forge');

// ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
// Local variables

var lookup = "ABCDEFGHIJKLMNOPQRSTUVWXYZ234567";

// ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
// Local functions

function base32encode(bytes) {
    var start = 0;
    var end = start + 5;
    var str = "";
    // We must convert each 5-byte chunk into an 8 character base32 string
    while (start < bytes.length) {
        // Collect 5-byte chunk
        var vals = [];
        for (var i = start; i < end; ++i) {
            if (i >= bytes.length) {
                vals.push(0);
            } else {
                vals.push(bytes[i]);
            }
        }
        // Character by character...
        var char1 = vals[0] >>> 3;
        var char2 = (vals[0] & 7 << 2) + (vals[1] >>> 6);
        var char3 = vals[1] & 62 >>> 1;
        var char4 = (vals[1] & 1 << 4) + (vals[2] & 240 >>> 4);
        var char5 = (vals[2] & 15 << 1) + (vals[3] & 128 >>> 7);
        var char6 = vals[3] & 124 >>> 2;
        var char7 = (vals[3] & 3 << 3) + (vals[4] & 224 >>> 5);
        var char8 = (vals[4] & 31);
        var chars = [char1, char2, char3, char4, char5, char6, char7, char8];

        // Compose String
        for (var j = 0; j < chars.length; ++j) {
            str += lookup[chars[j]];
        }

        start += 5;
        end += 5;
    }
    return str;
}

function fingerprint(str) {
    var print = "";
    for (var i = 0; i < 48; ++i) {
        print += str[i];
        if ((i + 1) === 48) {
            return print;
        } else if (i % 4 === 3) {
            print += ":";
        }
    }
}

function keyIdFromCryptoKey(pem) {
    var cert = forge.pki.certificateFromPem(pem);
    var asn1 = forge.pki.certificateToAsn1(cert);
    var der = forge.asn1.toDer(asn1);
    var buf = new Buffer(der.getBytes(), 'binary');
    var hash = crypto.createHash('sha256').update(buf).digest();
    var str = base32encode(hash);
    var print = fingerprint(str);
    return print;
}

The argument that I'm passing here to keyIdFromCryptoKey is the contents of the thinknode.dev.crt file.

Can you point me in the right direction?

I should also mention that the registry and auth servers are sitting behind nginx. The configuration for that looks like this:

upstream thinknode-api {
  server localhost:3060;
}

upstream docker-registry {
  server localhost:5000;
}

server {
  listen       3061;
  server_name  thinknode.dev;

  proxy_set_header Host       $http_host;    # Required for Docker client
  proxy_set_header X-Real-IP  $remote_addr;  # Passthrough real client IP

  client_max_body_size 0;        # Disable any limits to avoid HTTP 413 for large image uploads
  chunked_transfer_encoding on;  # Required to avoid HTTP 411

  ssl on;
  ssl_certificate /usr/local/share/ca-certificates/thinknode.dev/thinknode.dev.crt;
  ssl_certificate_key /etc/ssl/private/thinknode.dev.key;

  location / {
     proxy_pass http://thinknode-api;
  }
}

server {
  listen       5243;
  server_name  thinknode.dev;

  add_header 'Docker-Distribution-API-Version' 'registry/2.0' always;

  proxy_set_header Host       $http_host;    # Required for Docker client
  proxy_set_header X-Real-IP  $remote_addr;  # Passthrough real client IP

  client_max_body_size 0;        # Disable any limits to avoid HTTP 413 for large image uploads
  chunked_transfer_encoding on;  # Required to avoid HTTP 411

  ssl on;
  ssl_certificate /usr/local/share/ca-certificates/thinknode.dev/thinknode.dev.crt;
  ssl_certificate_key /etc/ssl/private/thinknode.dev.key;

  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header X-Forwarded-Proto $scheme;
  proxy_set_header X-Original-URI $request_uri;
  proxy_set_header Docker-Distribution-Api-Version registry/2.0;

  location / {
    proxy_pass http://docker-registry;
  }
}

and the docker version information is:

Client version: 1.7.1
Client API version: 1.19
Go version (client): go1.4.2
Git commit (client): 786b29d
OS/Arch (client): linux/amd64
Server version: 1.7.1
Server API version: 1.19
Go version (server): go1.4.2
Git commit (server): 786b29d
OS/Arch (server): linux/amd64

[dmp42]

ping @jlhawn

[jlhawn]

hi @kyleburnett Thanks for opening this issue. Sorry for the delay in my response. At first glance it looks like this might be your problem:

function keyIdFromCryptoKey(pem) {
    var cert = forge.pki.certificateFromPem(pem);
    var asn1 = forge.pki.certificateToAsn1(cert);
    var der = forge.asn1.toDer(asn1);
    var buf = new Buffer(der.getBytes(), 'binary');
    var hash = crypto.createHash('sha256').update(buf).digest();
    var str = base32encode(hash);
    var print = fingerprint(str);
    return print;
}

It looks like this function is getting the DER bytes of a certificate and not of the public key from the certificate.

I'm not much of a Javascript programmer, but I took a quick look at the node-forge x509 code and rsa code and think something like this might be what you want:

var asn1 = forge.pki.publicKeyToAsn1(cert.publicKey);

instead of:

var asn1 = forge.pki.certificateToAsn1(cert);

I haven't looked too closely at your fingerprint/base32 code but it looks pretty straightforward. If you can, compare the key IDs that your javascript code generates against the ones that the libtrust golang code would generate given the same certificate. Let me know if this solves your issue.

[jlhawn]

Also, the line from your build file that copies in the certificate, you should consider changing it to:

COPY ssl/thinknode.dev.crt /usr/local/share/ca-certificates/thinknode.dev/

One of the goals of the JWT auth mechanism is to not have to use any secrets for authorization so you should only need the certificate bundle file to indicate the trusted public keys and not have to keep around the private key (only the token signing server should need the private key).

[dmp42]

@kyleburnett do you feel Josh's answer is addressing your concerns?

[kyleburnett]

Sorry for the delayed response. Getting a Go program up and running with the libtrust library helped us compare what libtrust was doing vs what our code was doing. There was an error my base32 encoding function. Thanks for the help. :)

[kyleburnett]

Here is the algorithm for generating the kid from .cert and .key files in javascript for people having similar issues.

var bluebird = require('bluebird');
var crypto = require('crypto');
var forge = require('node-forge');
var fs = require('fs');

var data = {};

var alphabet = "ABCDEFGHIJKLMNOPQRSTUVWXYZ234567";

function base32encode(value) {
    var skip = 0;
    var bits = 0;
    var output = '';

    // Iterate over bytes
    var i = 0;
    while (i < value.length) {
        var v = value[i];
        if (typeof v == 'string') {
            v = v.charCodeAt(0);
        }

        // Set current bits
        if (skip < 0) { // We have a carry from the previous byte
            bits |= (v >> (-skip));
        } else { // No carry
            bits = (v << skip) & 248;
        }

        // Produce a character if there is enough data, otherwise, get more data
        if (skip < 4) {
            output += alphabet[bits >> 3];
            skip += 5;
        } else {
            skip -= 8;
            i++;
        }
    }

    // Consume any remaining bits left
    output += (skip < 0 ? alphabet[bits >> 3] : '');
    return output;
}

bluebird.bind(data).then(function() {
    return fs.readFileAsync('/path/to/certificate/certificate.crt');
}).then(function(crt) {
    this.crt = crt;
    return fs.readFileAsync('/path/to/key/example.key');
}).then(function(key) {
    this.key = key;
}).then(function() {
    var cert = forge.pki.certificateFromPem(this.crt);
    var asn1 = forge.pki.publicKeyToAsn1(cert.publicKey);
    var der = forge.asn1.toDer(asn1);
    var buf = new Buffer(der.getBytes(), 'binary');
    var hash = crypto.createHash('sha256').update(buf).digest();
    var base32 = base32encode(hash.slice(0, 30));

    // Create key id (fingerprint)
    this.kid = '';
    for (var i = 0; i < 48; ++i) {
        this.kid += base32[i];
        if (i % 4 === 3 && (i + 1) !== 48) {
            this.kid += ":";
        }
    }
});