Dockerode: Pulling private repository from AWS fails using AWS auth code

Created on 12 Apr 2018 · 12Comments · Source: apocas/dockerode

When using the auth code from the AWS

public static getDockerAuth(privateRepo: string) {
const auth = {};
const configFile = ${process.env.HOME}/.docker/config.json;
if (existsSync(configFile)) {
const authFileContent = readFileSync(configFile);
const authJson = JSON.parse(authFileContent.toString());
if (authJson.auths[privateRepo]) {
auth['authconfig'] = {key: authJson.auths[privateRepo].auth};
}
}
return auth;
}

then using dockerode with creds recieved from getDockerAuth fail

import * as Dockerode from 'dockerode';
let globalDocker = new Dockerode();

private static async dockerPullWrapper(image: string, options = {}): Promise {
let s: string = await globalDocker.pull(image, options).then(
(stream) => {
return new Promise((resolve, reject) => {
this.globalDocker.modem.followProgress(stream, onFinished, onProgress);

                function onFinished(err, data) {
                    if (err || (data && data.error)) {
                        reject(`Failed to pull image due to ${err || data.error}`);
                    }
                    console.log('--> pulled with data ');
                    console.log(data);
                    resolve(image);
                }

                function onProgress(event) {
                    process.stdout.write('.');
                }
            });

    }).catch((err) => {
            throw new Error('Failed to pull image');
        });

return s;
}

a couple of issues arise in this case:

onFinished - is sent with empty err argument when triggered yet the data.error exists
you get an authentication error when trying to pull the image using the authconfig

Any idea why?

Source

eladco-at-comigo

Most helpful comment

@EladComigo
Alright, here is my proof-of-concept image pulling code, tested on Dockerode 2.5.5 and aws-sdk 2.229.1.
It can be used as a script, and you need to supply an image with repository identifier and a tag for it to work properly. Please note that image names without preceding repositories are thought to be contained in public docker registry by docker daemon, and to pull ECR images you do need to prefix it with the ECR host (like 400293112113.dkr.ecr.eu-west-1.amazonaws.com/your-image-name:1.2-image-tag).

let AWS = require('aws-sdk');
let Dockerode = require('dockerode');
let url = require('url');
let imageName = process.argv[2];

async function getToken() {
    let ecr = new AWS.ECR();
    let data = await ecr.getAuthorizationToken().promise();
    let authInfo = data.authorizationData[0];
    let [user, pass] = Buffer.from(authInfo.authorizationToken, 'base64').toString().split(':');
    return {
        username: user,
        password: pass,
        serveraddress: url.parse(authInfo.proxyEndpoint).host
    };
}

async function pullImage(img) {
    let dockerClient = new Dockerode();
    let authconfig = await getToken();

    console.log(`Pulling ${img}`);
    let pullStream = await dockerClient.pull(img, {
        authconfig: authconfig
    });

    let result = await new Promise((resolve, reject) => {
        dockerClient.modem.followProgress(pullStream, (err, out) => {
            if (err) {
                reject(err);
                return;
            }
            resolve(out);
        });
    });

    console.log(result);

}

pullImage(imageName);

DenSpirit on 26 Apr 2018

👍7

All 12 comments

I've managed to push to private AWS ECR repository by inspecting the way Docker CLI encodes authorization. I expect pull to work the same.

Your best bet is to retrieve credentials from Amazon yourself and perform certain transformations on them, and then pass the result to Dockerode.

Using AWS SDK for Javascript, you can retrieve authorizationToken and proxyEndpoint strings. authorizationToken will be a Base64-encoded string of "user:pass" pair, separated by colon. You need to extract username and password from that string, and then construct the following object:

{
    username: username, // the part before colon in authorization token from amazon
    password: password,  // the part after colon
    serveraddress: "xxx.dkr.ecr.aa-region-1.amazonaws.com" // A host, and only host, part of proxyEndpoint
}

If you pass that object as authconfig to Dockerode, it will encode it as Base64 and pass in one of the headers, just like Docker CLI does.

If you insist on using local docker config.json file, perform similar transformations to the entry in auths object, using key as serveraddress and auth value as Base64-encoded string from above.

DenSpirit on 20 Apr 2018

👍1

Thank you @DenSpirit for your response, though trying that lead did not work for me,
I have decoded the AuthToken that I get from AWS using this function:

public static async getAuthToken(): Promise {
try {
const ecr: AWS.ECR = new AWS.ECR();
const auth: PromiseResult = await ecr.getAuthorizationToken().promise();
return auth.authorizationData;
} catch (e) {
throw new Error (Failed to retrieve AuthToken for AWS, error: ${e});
}
}
note: using ... to reduce text
which returns data that includes AWS:sxr65sds...s6s7s code - then I take this AWS token and decode it base64 - I get this information:
{ payload: '1IARQ...1wTv5kZdIIaIUwR+7C0Xr7zWMrARUAYNHa0=',
datakey: 'AQEBAHh+dS+BlN...LJ6NwkqM=',
version: '2',
type: 'DATA_KEY',
expiration: 1524542681 }
I saw in the AWS that these can be decrypted using special KMS abilities - so I tried that:

public static async getDecryptedCode(awsCode: string, endpoint: string, callback: Function) {
console.log(awsCode);
let buf = Buffer.from(awsCode, 'base64');
let tokens = JSON.parse(buf.toString());
let myKMS = new KMS({endpoint: endpoint});
let params: KMS.Types.DecryptRequest = {CiphertextBlob: buf}; // Buffer, GrantTokens: [tokens['payload'], tokens['datakey']] };
let c: AWS.Request = await myKMS.decrypt(params, (err, data) => {
if (err) console.log(err, err.stack); // an error occurred
else console.log(data); // successful response
callback();
});
}

but I still get the unautherized exception:
{ UnknownError: Unauthorized
at Request.extractError (/home/eladc/projects/comigo/comigo-infra/node_modules/aws-sdk/lib/protocol/json.js:48:27)
at Request.callListeners (/home/eladc/projects/comigo/comigo-infra/node_modules/aws-sdk/lib/sequential_executor.js:105:20)
at Request.emit (/home/eladc/projects/comigo/comigo-infra/node_modules/aws-sdk/lib/sequential_executor.js:77:10)
at Request.emit (/home/eladc/projects/comigo/comigo-infra/node_modules/aws-sdk/lib/request.js:683:14)
at Request.transition (/home/eladc/projects/comigo/comigo-infra/node_modules/aws-sdk/lib/request.js:22:10)
at AcceptorStateMachine.runTo (/home/eladc/projects/comigo/comigo-infra/node_modules/aws-sdk/lib/state_machine.js:14:12)
at /home/eladc/projects/comigo/comigo-infra/node_modules/aws-sdk/lib/state_machine.js:26:10
at Request. (/home/eladc/projects/comigo/comigo-infra/node_modules/aws-sdk/lib/request.js:38:9)
at Request. (/home/eladc/projects/comigo/comigo-infra/node_modules/aws-sdk/lib/request.js:685:12)
at Request.callListeners (/home/eladc/projects/comigo/comigo-infra/node_modules/aws-sdk/lib/sequential_executor.js:115:18)
message: 'Unauthorized',
code: 'UnknownError',
statusCode: 401,
time: 2018-04-23T16:04:41.601Z,
requestId: undefined,
retryable: false,
retryDelay: 92.36757013579879 }

eladco-at-comigo on 23 Apr 2018

@EladComigo, you should not decode the password further than that, just do it once (to get AWS:sxr65sds...s6s7s) and then use AWS as username and the sxr65sds...s6s7s part as password string for further steps.

DenSpirit on 24 Apr 2018

@DenSpirit I have tried using these creds:

Trying to retrieve docker myApplication : 0.0.88 from AWS
{ username: 'AWS',
password: 'eyJwYXlsb2FkIjoiTFF5YmNHL0RST1ZKS...I6MTUyNDcwNDY2M30=',
serveraddress: 'https://400293112113.dkr.ecr.eu-west-1.amazonaws.com' }
(node:31990) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 1): Error: Failed to pull image, due to : Error: (HTTP code 401) unexpected - Get https://registry-1.docker.io/v2/library/myApplication/manifests/0.0.88: unauthorized: incorrect username or password

And as you can see I got an error claiming that the username and password are incorrect

eladco-at-comigo on 25 Apr 2018

@EladComigo
Try using only the host part for serveraddress, omit the protocol (like 400293112113.dkr.ecr.eu-west-1.amazonaws.com).

DenSpirit on 25 Apr 2018

@DenSpirit, got the same result.

eladco-at-comigo on 26 Apr 2018

let AWS = require('aws-sdk');
let Dockerode = require('dockerode');
let url = require('url');
let imageName = process.argv[2];

async function getToken() {
    let ecr = new AWS.ECR();
    let data = await ecr.getAuthorizationToken().promise();
    let authInfo = data.authorizationData[0];
    let [user, pass] = Buffer.from(authInfo.authorizationToken, 'base64').toString().split(':');
    return {
        username: user,
        password: pass,
        serveraddress: url.parse(authInfo.proxyEndpoint).host
    };
}

async function pullImage(img) {
    let dockerClient = new Dockerode();
    let authconfig = await getToken();

    console.log(`Pulling ${img}`);
    let pullStream = await dockerClient.pull(img, {
        authconfig: authconfig
    });

    let result = await new Promise((resolve, reject) => {
        dockerClient.modem.followProgress(pullStream, (err, out) => {
            if (err) {
                reject(err);
                return;
            }
            resolve(out);
        });
    });

    console.log(result);

}

pullImage(imageName);

DenSpirit on 26 Apr 2018

👍7

@DenSpirit, Thanks! it works, the prefix is what was missing, Thank you for the time invested and the patience

eladco-at-comigo on 29 Apr 2018

👍1

I have tried above sample to pull image from aws but it throws following error now. I have tried authConfig with key approach mentioned in the doc that also doesn't work.

{ Error: (HTTP code 404) unexpected - pull access denied for XXXXX.dkr.ecr.us-west-2.amazonaws.com/XXXXXXX, repository does not exist or may require 'docker login': denied: Your Authorization Token is invalid.
at /Volumes/data/dev/projects/test/node_modules/docker-modem/lib/modem.js:296:17
at IncomingMessage. (/Volumes/data/dev/projects/test/node_modules/docker-modem/lib/modem.js:323:9)
at IncomingMessage.emit (events.js:203:15)
at IncomingMessage.EventEmitter.emit (domain.js:448:20)
at endReadableNT (_stream_readable.js:1143:12)
at process._tickCallback (internal/process/next_tick.js:63:19) reason: undefined, statusCode: 404, json: null }

Mac OS: 10.14.4
Node version : v10.18.0
Docker version 19.03.5, build 633a0ea
docker-compose: "0.23.2"
aws-sdk: "2.364.0"

I am going to try docker login approach via cli as mentioned here https://docs.aws.amazon.com/AmazonECR/latest/userguide/Registries.html#registry_auth

harissarwar on 30 Jan 2020

I tried the docker login before running the node script that uses dockerode (that was my first approach) and it doesn't work.
The above approach parsing the docker json config file and passing that as a { username, password, serveraddress } option object doesn't work either -> Just fails with Get https://xxxx.dkr.ecr.eu-west-1.amazonaws.com/v2/nginx/manifests/stable-alpine: no basic auth credentials
The other approach with the { key } object fails the same way as @harissarwar

I am using it with buildImage and am referring to a base image from a private repo in my Dockerfile

OzTK on 30 Jan 2020

@OzTK docker login command will not help in Dockerode's case because Dockerode does not inspect docker config files for authentication data.
In above sample, what did await getToken() return? getToken in this case is responsible for retrieving ECR credentials in form that is acceptable for Dockerode. It's unlikely the token was invalid right after it was issued. It might indicate that you lack permissions to access repository, that you modified the sample, or that you kept issued token long enough for it to expire.

DenSpirit on 3 Feb 2020

@OzTK - I came across this today. It's not documented at all, but you can pass through an 'registryconfig' option to docker.build(). It's an object that gets mapped through to the 'X-Registry-Config' header on the docker API request:

{
  "docker.example.com": {
    "username": "janedoe",
    "password": "hunter2"
  },
  "https://index.docker.io/v1/": {
    "username": "mobydock",
    "password": "conta1n3rize14"
  }
}

To transform it from the authconfig object used elsewhere, I used:

    const registryconfig = {
      [authconfig.serveraddress]: {
        username: authconfig.username,
        password: authconfig.password,
      },
    };