Docker-node: Please add `yarn`(awesome npm alternative)

[nodkz]

Will be cool if yarn was integrated to latest node@6 images.
https://yarnpkg.com/

Yarn: A new package manager for JavaScript

[pesho]

It may be a bit premature to decide that yet, but I was thinking about the same yesterday. Given the overwhelmingly positive response to the Yarn release (~9300 GitHub stars in just one day, ~1600 upvotes on HN) this might become a very common request.

I've done a quick test, and adding Yarn to the -slim image (via npm install -g yarn) adds 21.1 MB to the size.

[nodkz]

Current official node image sizes:

node    slim    4d8411ec1363    13 days ago 210.3 MB
node    onbuild 275dae106538    13 days ago 653.9 MB

Yep, Yarp adds 10% to size for node:slim but:

• disk space is a most cheaper thing
• time till run is a most expensive 👟🚀

Agreed that needs to wait some time (a couple of hours 😂) till it became a new standart.

List of most fat packages in yarn. May be some dependencies can be removed. Or bundled with tree-shaking.

6,6M    ./core-js
4,8M    ./lodash
1,7M    ./es5-ext
1,7M    ./node-gyp
928K    ./babel-runtime
548K    ./github

208K    ./bl
244K    ./diff
216K    ./hawk
220K    ./json-schema
244K    ./request
256K    ./sshpk
244K    ./tough-cookie

156K    ./agent-base
168K    ./babel-types
104K    ./dashdash
116K    ./fstream
124K    ./gauge
104K    ./har-validator
112K    ./hoek
100K    ./jodid25519
172K    ./mime-db
168K    ./node-emoji
176K    ./readable-stream
104K    ./tar
184K    ./tweetnacl

[chorrell]

It's probably something we could add in the future if there was enough demand for it, but I'm pretty ambivalent about it at this point since it's so new. I'd prefer to keep the images fairly simple, especially the slim variant, since the main point is to use it as a base image and install dependencies etc on top of it.

For installation, I would be more inclined to use the Yarn Debian package repository (under Linux):

https://yarnpkg.com/en/docs/install

That more closely follows the best practices for official images (pgp signed packages):

https://github.com/docker-library/official-images#image-build

[pesho]

For installation, I would be more inclined to use the Yarn Debian package repository (under Linux):

https://yarnpkg.com/en/docs/install

That more closely follows the best practices for official images (pgp signed packages):

https://github.com/docker-library/official-images#image-build

Tried that yesterday. Currently doesn't work on our images, because their packages depend on an installed recent nodejs DEB package (which should probably be relaxed to Recommends or Suggests).

[chorrell]

Ugh, ok. Maybe at some point they'll pgp sign the tarball.

[pesho]

It looks like there may be soon a ~ 2 MB bundled release which is CLI-compatible: yarnpkg/yarn#888. If that works, I'm definitely +1 on including it.

[Daniel15]

Currently doesn't work on our images, because their packages depend on an installed recent nodejs DEB package (which should probably be relaxed to Recommends or Suggests).

This would break the out-of-the-box experience for people that are just getting started. Without Node.js being installed, apt-get install yarn then yarn will show an error about nodejs not being found.

Most people are on Ubuntu 16.04 which has Node.js 4.2.6 in the repo, and the NodeSource repo can be added to get Node 6.

[pesho]

Without Node.js being installed

The absence of a nodejs package does not indicate that Node.js is not installed. There are many popular alternative installation methods, such as nvm, nave, n, make install, binary tarballs, and so on.

[pesho]

@Daniel15 a Recommends relation would also not break the out-of-the-box experience, as recommended packages are installed by default on both Ubuntu and Debian. But it would allow those who have installed Node in a different way to opt out of the DEB package.

[Fishrock123]

Are we talking about shipping with Yarn? Because that's not a thing in core yet and I'm not sure that is a good idea at this time.

[Starefossen]

@Fishrock123 I don't think any of Docker WG members are suggesting we should add Yarn at this point, but that might change if Yarn becomes a popular request by the masses.

[chorrell]

Yes, that's my understanding too: this is just about adding yarn to the docker images.

[andrewmclagan]

Any timeline for this?

[Starefossen]

@andrewmclagan at them moment no, there is a similar discussion over at nodejs/node#9161.

[jeremyzahner]

If you guys would tag your releases on here we could fork the repo and use our custom images (including yarn). But since you don't, it's pretty hard to establish a workflow based on your official image (which is sad).

[Starefossen]

@jeremyzahner This is how you can use the official image with yarn – today:

FROM node:6.9.1
RUN npm install -g yarn

...

I agree that it would be more convenient to have it pre installed, but it is not a blocker from basing on the official image.

[chorrell]

@jeremyzahner I don't understand your "tag your releases" comment. Can you elaborate? That is, I don't understand how this fits with your desired workflow.

[Starefossen]

The issue would be to run one-off commands like this since yarn is not installed by default:

$ docker run --rm node:argon yarn install

[Daniel15]

Currently doesn't work on our images, because their packages depend on an installed recent nodejs DEB package (which should probably be relaxed to Recommends or Suggests).

This has been fixed, by the way. The Debian package should be suitable for you to use now 😄 The repo is GPG signed.

[kachkaev]

Hey guys, would it be possible to prioritize this issue somehow? It is the most :+1:-d in the whole project at the moment :–)

I would love to use yarn in GitLab CI to test and build react apps. Today's lack of an official node-yarn image keeps me and others on NPM. I know it is possible to use yarn locally and then NPM in CI, but that feels a bit dangerous. A couple of unofficial images exist and can help, but they a probably not the best choice for anything serious.

[Starefossen]

There is nothing preventing you from making your own image and publishing it to Docker Hub if you want to use yarn today:

FROM node:6-alpine

ENV YARN_VERSION 0.20.0

ADD https://yarnpkg.com/downloads/$YARN_VERSION/yarn-v${YARN_VERSION}.tar.gz /opt/yarn.tar.gz

RUN yarnDirectory=/opt/yarn && \
    mkdir -p "$yarnDirectory" && \
    tar -xzf /opt/yarn.tar.gz -C "$yarnDirectory" && \
    ln -s "$yarnDirectory/dist/bin/yarn" /usr/local/bin/ && \
    rm /opt/yarn.tar.gz

As it stands today yarn is not an official part of Node.js, we do not have tests etc. to make sure that we maintain compatibility with yarn from version to version. Yarn is not yet reached a v1.0 stability level which makes breaking changes a problem.

[jeremyzahner]

@kachkaev We already did so: https://hub.docker.com/r/jshmrtn/node-yarn/

[kachkaev]

Good news! There is now an image by yarn developers:
https://hub.docker.com/r/yarnpkg/node-yarn/

There are not that many tags at the moment (e.g. alpine version is missing), but this is still a great step forward! Related PR: https://github.com/yarnpkg/yarn/pull/2721

Only after @Starefossen replied to my comment I realised that this issue is in nodejs/docker-node repo instead of docker-library/official-images or similar. Now I also think that this is probably not the best place to ask for yarn support in docker, as that package manager comes from a completely different source than node+npm. @nodkz how about closing the issue here and continuing the conversation in another repo, e.g. yarnpkg/node-yarn?

[Daniel15]

There are not that many tags at the moment (e.g. alpine version is missing)

The current node-yarn image is based off Debian Jessie, not Alpine. I can add an Alpine image if there's demand for it, but generally I haven't seen a lot of demand as most people are already using other Debian/Ubuntu packages as part of their build system anyways.

[pesho]

I tried creating a (new) PR adding Yarn to the official docker-node images, but unfortunately all available installation methods still have blocking issues. A major goal for me is to keep the size increase small, otherwise such a PR has a very low chance of being accepted.

Here is a list of the available methods, with pros/cons:

• Single-file .js Webpack bundle

  • (pro) Small size (only 3.2 MB), has good chance of being accepted here
  • (con) Not signed, thus unsuitable for an official Docker image

• Tarball

  • (pro) Signed
  • (con) Large size (~14 MB after extraction) because it's unbundled

• DEB package

  • (con) Large size (~14 MB after extraction) because it's unbundled
  • (con) Only signed if using the APT repository, which is not a good option if we want a fixed version
  • (con) Would be difficult to use in the -alpine image

• NPM package

  • (con) Large size

@Daniel15 do you think any of these could be improved?

[Daniel15]

do you think any of these could be improved?

I can add a GPG signature for the single-file .js bundle, if you think that's the easiest approach. Currently our release scripts only sign tar.gz files (https://github.com/yarnpkg/releases/blob/gh-pages/nightly/api/sign_releases.php#L34-L44), but that's easy to change.

I filed an issue to sign the standalone .js files: https://github.com/yarnpkg/yarn/issues/2728

Large size (~14 MB after extraction) because it's unbundled

We should be able to reduce the size of the tarball / Deb package. The bundled .js file is smaller because it only contains the code that's actually needed, whereas the tarball contains a bunch of stuff from node_modules that's not needed at runtime (such as readmes for the dependencies, example files, documentation, etc). @thejameskyle was working on reducing the size of the tarball by analyzing the JS files that are actually being used (similar to what Webpack is doing for the single-file .js bundle) but I'm not sure what the current status is.

Only signed if using the APT repository, which is not a good option if we want a fixed version

You can install a fixed version using the repository. Example:

apt-get install yarn=0.20.3-1

Would be difficult to use in the -alpine image

Someone recently built an Alpine package for Yarn; see https://github.com/yarnpkg/yarn/issues/1326

[pesho]

I can add a GPG signature for the single-file .js bundle, if you think that's the easiest approach. Currently our release scripts only sign tar.gz files (https://github.com/yarnpkg/releases/blob/gh-pages/nightly/api/sign_releases.php#L34-L44), but that's easy to change.

Yes, this sounds great!

We should be able to reduce the size of the tarball / Deb package. The bundled .js file is smaller because it only contains the code that's actually needed, whereas the tarball contains a bunch of stuff from node_modules that's not needed at runtime (such as readmes for the dependencies, example files, documentation, etc). @thejameskyle was working on reducing the size of the tarball by analyzing the JS files that are actually being used (similar to what Webpack is doing for the single-file .js bundle) but I'm not sure what the current status is.

Why not simply use the already available Webpack-ed bundle in the tarball?

[Daniel15]

Why not simply use the already available Webpack-ed bundle in the tarball?

Haha, that's an option that I didn't even consider! It's an interesting idea.

One of the issues we have with that is that stack traces become useless as they all point to the yarn.js file. We did try using source maps with the standalone .js file, but the source map is way too big and as a result it uses a huge amount of RAM. The source map file is also extremely large.

By the way, not sure if you saw it but I also edited my comment to say that you can install a specific version of Yarn when using the Debian repo:

apt-get install yarn=0.20.3-1

will always install version 0.20.3.

[pesho]

Haha, that's an option that I didn't even consider! It's an interesting idea.

One of the issues we have with that is that stack traces become useless.

Are they completely useless indeed? The bundle is not "uglified" so the names and formatting are mostly preserved. Only the line numbers and filename should be different.

By the way, not sure if you saw it but I also edited my comment to say that you can install a specific version of Yarn when using the Debian repo

Yup, I saw it :) Still, a method which works with both Debian and Alpine image variants would be preferable.

[nodkz]

@nodkz how about closing the issue here and continuing the conversation in another repo, e.g. yarnpkg/node-yarn?

@kachkaev to much feedback still here, so let keep this issue open.

[nodkz]

My current solution for building docker image for production (good hint for starting your own). This solution is used a half year and rewritten several times.

Run yarn build-docker, which calculates hash of package.json and yarn.lock and, if image does not exists, creates node-modules image app-kz/node-modules:production-[HASH] (if image exists, skip this build step for speed). After that used this node-modules image for creation app image with name app-kz/production:170214-1357-30c37cc ([YYMMDD]-[hhmm]-[commitHash])

package.json

{
  ...
  "scripts": {
    "build-docker": "BABEL_ENV=build ./node_modules/.bin/babel-node ./bin/run deploy/docker/buildAppImage",
  },
  ...

run.js

Script which runs other js files with some logging info

import chalk from 'chalk';

function format(time) {
  return time.toTimeString().replace(/.*(\d{2}:\d{2}:\d{2}).*/, '$1');
}

async function run(fn, options) {
  const start = new Date();
  console.log(chalk.bold.yellow(`[${format(start)}] Starting '${fn.name}'...`));

  let result;
  let hasError = false;
  try {
    result = await fn(options);
  } catch (err) {
    console.log(chalk.bgRed.bold.white(`Failed '${fn.name}': ${err.message}`));
    console.error(err.stack);
    hasError = true;
  }

  const finish = new Date();
  const time = finish.getTime() - start.getTime();
  if (hasError) {
    console.log(chalk.bgRed.bold.white(
      `[${format(finish)}] Finished '${fn.name}' after ${time} ms`
    ));
  } else {
    console.log(chalk.bold.green(
      `[${format(finish)}] Finished '${fn.name}' after ${time} ms`
    ));
  }
  return result;
}

if (process.mainModule.children.length === 0 && process.argv.length > 2) {
  delete require.cache[__filename];
  const module = require(`./${process.argv[2]}.js`).default;
  run(module);
}

module.exports = run;
module.exports.default = run;

config.js

import path from 'path';
import cp from 'child_process';

export const regionECR = 'eu-west-1';
export const remoteRepo = `000000000000.dkr.ecr.${regionECR}.amazonaws.com`;
export const repositoryName = 'repo-kz';
export const rootDir = path.resolve(__dirname, '../../');
export const buildEnv = 'production';
export const webpackBuildDir = path.resolve(rootDir, `./build/${buildEnv}`);

export function isDockerImageExists(imageNameWithTag) {
  const imageId = cp.execSync(`docker images -q ${imageNameWithTag}`).toString();
  return imageId && imageId.length > 0;
}

export function getContainerInfo(imageNameWithTag) {
  return cp.execSync(`docker images ${imageNameWithTag}`).toString();
}

buildNodeImage.js

/* eslint-disable no-use-before-define, camelcase, no-multi-str */

import cp from 'child_process';
import fsExtra from 'fs-extra';
import hashFiles from 'hash-files';
import chalk from 'chalk';
import {
  repositoryName,
  rootDir,
  buildEnv,
  isDockerImageExists,
  getContainerInfo,
} from '../config';

const nodeVersion = '6.9.4';
const nodeSlimImage = `node:${nodeVersion}-slim`;
const nodeBuildImage = `node:${nodeVersion}-onbuild`;
const imageName = `${repositoryName}/node-modules`;
const ramDiskName = `docker_${new Date().toJSON().slice(0, 19).replace(/[^0-9]+/g, '')}`;
const dockerContextFolder = `/Volumes/${ramDiskName}`;
console.log(imageName);

export default function build_DockerContainer_With_NodeModules() {
  const imageTag = calcPackageHash(buildEnv);
  const imageNameWithTag = `${imageName}:${imageTag}`;

  if (!isDockerImageExists(imageNameWithTag)) {
    try {
      console.log(`Mounting RAM disk ${ramDiskName}`);
      mountRamDisk();

      console.log(
        chalk.magenta('Installing node_modules for debian '
                    + `via Docker container (${nodeBuildImage} 650MB) to the RAM disk`)
      );
      npmInstallViaFatContainer(buildEnv);

      console.log(chalk.magenta(`Creating slim docker image (from ${nodeSlimImage}): ${imageNameWithTag}...`));
      createDockerContainer(imageNameWithTag);
    } finally {
      console.log(`Unmounting RAM disk ${ramDiskName}`);
      unmountRamDisk();
    }

    console.log(chalk.magenta('Docker container with NODE_MODULES successfully builded.'));
  } else {
    console.log(chalk.magenta('Docker container with NODE_MODULES already exists.'));
  }
  console.log(chalk.magenta(getContainerInfo(imageNameWithTag)));
  console.log(chalk.magenta('You can run container via command:'));
  console.log(chalk.bold.magenta(`docker run -i -t --rm ${imageNameWithTag} /bin/bash`));

  return {
    name: imageName,
    tag: imageTag,
    nameWithTag: imageNameWithTag,
  };
}

function createDockerContainer(imageNameWithTag) {
  const dockerfile = [
    `FROM ${nodeSlimImage}`, // 210MB container only with node
    'WORKDIR /src',
    'COPY . /src',
  ];
  fsExtra.outputFileSync(`${dockerContextFolder}/Dockerfile`, dockerfile.join('\n'));

  cp.execSync(`docker build \
    -t ${imageNameWithTag} \
    ${dockerContextFolder}`,
    {
      cwd: dockerContextFolder,
      stdio: [0, 1, 2],
    }
  );
}

function npmInstallViaFatContainer() {
  fsExtra.emptyDirSync(dockerContextFolder);
  fsExtra.copySync(`${rootDir}/package.json`, `${dockerContextFolder}/package.json`);
  fsExtra.copySync(`${rootDir}/yarn.lock`, `${dockerContextFolder}/yarn.lock`);
  const npmInstallCmd = (
    'wget https://yarnpkg.com/install.sh && chmod +x install.sh && ./install.sh --nightly && rm -f install.sh '
      + '&& PATH=$PATH:~/.yarn/bin/ ' // add yarn to paths
      + '&& yarn add string-width --production ' // needed for build bcrypt module and runtime
      + '&& yarn install --production ' // install app modules
      + '&& rm -rf ~/.yarn ' // remove cached modules and yarn itself for reducing container size
  );

  // IMPORTANT: `node:onbuild` is 650MB container with build scripts
  // needed for build `fsevent` and `crypto` bin-files for debian
  cp.execSync(`docker run \
             -t --rm -v ${dockerContextFolder}:/src \
             ${nodeBuildImage} \
             /bin/bash -c "cd /src && ${npmInstallCmd}"`,
    {
      cwd: dockerContextFolder,
      stdio: [0, 1, 2],
    }
  );
}

function calcPackageHash(env) {
  const hash = hashFiles.sync({
    files: [
      `${rootDir}/package.json`,
      `${rootDir}/yarn.lock`,
    ],
  });

  return `${env}-${hash}`;
}

function mountRamDisk() {
  cp.execSync(`diskutil erasevolume hfsx ${ramDiskName} \`hdiutil attach -nomount ram://1200000\``,
    { stdio: [0, 1, 2] }
  );
}

function unmountRamDisk() {
  cp.execSync(`diskutil unmountDisk force ${ramDiskName}`,
    { stdio: [0, 1, 2] }
  );
}

buildAppImage.js

/* eslint-disable no-use-before-define, camelcase */

import cp from 'child_process';
import del from 'del';
import fs from 'fs';
import fsExtra from 'fs-extra';
import chalk from 'chalk';
import run from '../../run';
import buildNodeImage from './buildNodeImage';
import {
  repositoryName,
  buildEnv,
  webpackBuildDir,
  isDockerImageExists,
  getContainerInfo,
} from '../config';

const imageName = `${repositoryName}/${buildEnv}`;

export default async function build_DockerContainer_With_APP() {
  let buildDirStat = fs.statSync(webpackBuildDir);
  if (!buildDirStat) {
    cp.execSync('yarn build', { cwd: webpackBuildDir, stdio: [0, 1, 2] });
  }

  buildDirStat = fs.statSync(webpackBuildDir);
  if (!buildDirStat) {
    console.log(chalk.magenta.bold('Firstly bundle app via command: `yarn build`'));
    throw new Error(`Webpack folder with bundled code does not exist: ${webpackBuildDir}`);
  }

  const nodeImage = await run(buildNodeImage);
  if (!nodeImage) {
    throw new Error('Cannot build app container. NodeModules container not provided.');
  }

  const imageTagFallback = buildDirStat
                    .mtime
                    .toJSON()
                    .slice(0, 19)
                    .replace('T', '-')
                    .replace(/[^-0-9]+/g, '');
  const revision = fs.readFileSync(`${webpackBuildDir}/revision.txt`, 'utf8').toString().trim();
  const imageTag = revision || imageTagFallback;
  const imageNameWithTag = `${imageName}:${imageTag}`;

  console.log(`Creating app docker image: ${imageNameWithTag}...`);

  if (!isDockerImageExists(imageNameWithTag)) {
    console.log('Remove source map files for avoiding vulnerability of code');
    del.sync([`${webpackBuildDir}/public/**/*.map`]);

    createDockerfile(nodeImage.nameWithTag);
    cp.execSync(`docker build \
      -t ${imageNameWithTag} \
      ${webpackBuildDir}`,
      { cwd: webpackBuildDir, stdio: [0, 1, 2] });

    console.log(chalk.magenta('Docker container successfully builded.'));
  } else {
    console.log(chalk.magenta('Docker container already exists.'));
  }
  console.log(chalk.magenta(getContainerInfo(imageNameWithTag)));
  console.log(chalk.magenta('Command for enter to container:'));
  console.log(chalk.bold.magenta(`docker run -i -t --rm ${imageNameWithTag} /bin/bash`));
  console.log(chalk.magenta('Command to start server on port 5000:'));
  console.log(chalk.bold.magenta(`docker run -i --rm -p 5000:5000 ${imageNameWithTag}`));

  return {
    name: imageName,
    tag: imageTag,
    nameWithTag: imageNameWithTag,
  };
}

function createDockerfile(modulesImage) {
  const dockerfile = [
    `FROM ${modulesImage}`,
    'WORKDIR /src',
    'COPY . /src',
    'EXPOSE 5000',
    'CMD ["node", "/src/server.js"]',
  ];
  const filename = `${webpackBuildDir}/Dockerfile`;
  fsExtra.outputFileSync(filename, dockerfile.join('\n'));

  return filename;
}

[SimenB]

Someone recently built an Alpine package for Yarn; see yarnpkg/yarn#1326

That package doesn't really help, as that package will install its own version of Node, ignoring the already installed one.

Nice to see some movement here, though! 😄

[Daniel15]

@pesho - .js release files are now signed, I've signed all previous releases too. 👍

[pesho]

New proposal PR created: #337

[chorrell]

The latest version of yarn is now being added via https://github.com/nodejs/docker-node/pull/337

Here's the Docker Hub PR: https://github.com/docker-library/official-images/pull/2703

[kachkaev]

Looks like yarn was added to all official node images – perfect! README on Docker Hub does not mention yarn so it may be confusing, but the recipes themselves do include it! 🎉 🎉 🎉

[Starefossen]

Yes, there is a pending PR to update documentation.

[styfle]

@Starefossen This still seems to be missing from the docs.

I was surprised when I looked at the environment variables for a recent node app to see "YARN_VERSION":"0.21.3". I checked the docs on Docker Hub and there was nothing mentioning Yarn. Eventually I found this issue.

Which PR is supposed to add the docs?

[Starefossen]

@styfle nodejs/docker-node#345 added this to the README of _this_ repository, I'll look into why that has not been propagated onto Docker Hub. Thank you 🙂

[romandragan]

Why yarn version is so much outdated? It's still 0.24.6 while latest published is already 0.27.5, which has many minor bug fixes.

[SimenB]

Because of a miscommunication in when to update it. Next release will have it updated

[styfle]

@Starefossen @SimenB Any idea why the readme on Docker Hub is still outdated*?

*Notice no mention of yarn

[SimenB]

I don't know how that file is updated...

[chorrell]

It requires a separate PR to https://github.com/docker-library/docs

They don't automatically pull in our README.md or docs unfortunately.

[styfle]

@chorrell Thanks, I submitted a PR in https://github.com/docker-library/docs/pull/964

[SimenB]

@romandragan FWIW 8.2.0 will have a updated Yarn installation. We'll update the others when a new version of node is released for them