Docker-node: `yarn` not found in `node:alpine`

[beldur]

I'm using the node:alpine image in a gitlab-ci build.
Since a few minutes ago I get the following error in gitlab-ci:

$ yarn config set cache-folder .yarn
/bin/sh: eval: line 50: yarn: not found
ERROR: Job failed: exit code 127

My .gitlab-ci.yaml:

build:
  image: node:alpine
  stage: build
  script:
    - yarn config set cache-folder .yarn
    - yarn
    - yarn build

[mmcgarr]

I'm seeing the same in circleci

```
Step 6/9 : RUN yarn
---> Running in a8731d6619d5
/bin/sh: yarn: not found
The command '/bin/sh -c yarn' returned a non-zero code: 127

[rtorino]

I'm experiencing the same issue.

[Scukerman]

Same here.
Need to revert ae26d217bca933e72ac5f8de946681c1b5f8b37e.

[thrawny]

Same for the normal node:9.8.0

~ » docker run -it --rm node:9.8.0 sh  
# yarn
sh: 1: yarn: not found

[fabriciocolombo]

Same here with node:carbon

[nicolas-duvauchel]

This was caused by this commit: https://github.com/nodejs/docker-node/commit/ae26d217bca933e72ac5f8de946681c1b5f8b37e

Given the way the symlink is done mkdir -p /opt/yarn should have been deleted

[mmcgarr]

node:9.7-alpine image seems unaffected

[SimenB]

@PeterDaveHello @chorrell

[tredston]

Seems to affect all supported tags except chakracore.

[michalpiekarski]

The problem is with mkdir -p /opt/yarn (related to missing /opt directory see #648).
The correct command should be mkdir -p /opt.

[cecton]

latest is affected I think:

The link /usr/local/bin/yarn is broken, it targets /opt/yarn/bin/yarn which doesn't exist. But /opt/yarn/yarn-v1.5.1 exists.

root@5706d42db0c6:/# ls /usr/local/
CHANGELOG.md  LICENSE  README.md  bin  etc  games  include  lib  man  sbin  share  src
root@5706d42db0c6:/# ls /usr/local/bin
node  nodejs  npm  npx  yarn  yarnpkg
root@5706d42db0c6:/# /usr/local/bin/yarn
bash: /usr/local/bin/yarn: No such file or directory
root@5706d42db0c6:/# ls /usr/local/bin
node  nodejs  npm  npx  yarn  yarnpkg
root@5706d42db0c6:/# stat /usr/local/bin/yarn
  File: '/usr/local/bin/yarn' -> '/opt/yarn/bin/yarn'
  Size: 18              Blocks: 0          IO Block: 4096   symbolic link
Device: 31h/49d Inode: 62          Links: 1
Access: (0777/lrwxrwxrwx)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2018-03-16 10:19:06.000000000 +0000
Modify: 2018-03-16 10:19:06.000000000 +0000
Change: 2018-03-16 11:13:53.243600600 +0000
 Birth: -
root@5706d42db0c6:/# /usr/local/bin/yarn
bash: /usr/local/bin/yarn: No such file or directory
root@5706d42db0c6:/# /usr/local/bin/n
node    nodejs  npm     npx
root@5706d42db0c6:/# /usr/local/bin/yarn
bash: /usr/local/bin/yarn: No such file or directory
root@5706d42db0c6:/# /opt/yarn/bin/yarn
bash: /opt/yarn/bin/yarn: No such file or directory
root@5706d42db0c6:/# ls /opt/yarn
yarn-v1.5.1
root@5706d42db0c6:/# ls /opt/yarn/yarn-v1.5.1
LICENSE  README.md  bin  lib  package.json
root@5706d42db0c6:/# ls /opt/yarn/yarn-v1.5.1

[lingz]

Broke our CI as well this morning!

[SimenB]

Should #639, #647 and #648 all be reverted, or can someone send a new PR fixing it?

[elboletaire]

Same here, with every version of node. We've already tried carbon and latest. Both fail with the yarn command.

[SimenB]

This affects 9.8.0, 9.8, 9, latest, 8.10.0, 8.10, 8, carbon, 6.13.1, 6.13, 6, boron, 4.8.7, 4.8, 4, argon and all variants (alpine, slim, onbuild, wheezy, stretch)

[cecton]

@SimenB I volunteer to make a PR. I will remove the mkdir -p and replace by a symlink where the yarn has been decompressed.

[SimenB]

@cecton thanks!

[pesho]

I will be able to fix that in several hours. If anyone can take care of it earlier, please do.

[SimenB]

I asked for a revert in https://github.com/docker-library/official-images/pull/4131, but I think they are all on the west coast of the US, so probably asleep.

[ghindle]

Adding the following to my Dockerfile as a temporary fix helped. Remember YMMV!
RUN mkdir -p /opt/yarn/bin && ln -s /opt/yarn/yarn-v1.5.1/bin/yarn /opt/yarn/bin/yarn

[cecton]

Close to finish....

[chorrell]

It's ok, I have a branch with the required changes. Will PR it in a minute. Will test locally as well.

[cecton]

@chorrell @SimenB please have a look ^_^ I put so much effort into it, I would be glad to contribute to the project.

[florentchauveau]

The image node:9.7-alpine can be used as a workaround until this is fixed.

[n1ru4l]

Latest working version seems to be node:8.9.4 and node:8.9.4-alpine

[slawiko]

@n1ru4l I've tested it too - it works

[ragelo]

Same with node:8?

Step 7/9 : RUN yarn install
> /bin/sh: 1: yarn: not found

[slawiko]

@ragelo it seems yes. Just specify more concrete version (8.9.4-alpine was working for me)

[vrch]

node:boron is also broken apparently

[andrioid]

node:6 also broken

[montoias]

node:boron & node:carbon are broken also

[PeterDaveHello]

We're fixing it in #650

[SimenB]

Please don't spam down the issue with what images are broken, full list here: https://github.com/nodejs/docker-node/issues/649#issuecomment-373686322 (feel free to correct me if any are missed there, though!)

[ragelo]

@SimenB can it be related to current issue? Why the node has 4.2.6 version inside container?
Image: node:8.9.4, before it was node:8.

> yarn build
yarn run v1.5.1
warning You are using Node "4.2.6" which is not supported and may encounter bugs or unexpected behavior. Yarn supports the following semver range: "^4.8.0 || ^5.7.0 || ^6.2.2 || >=8.0.0"
$ cd "$(npm prefix)" && ./.npm-scripts/build
sh: 1: npm: not found

UPD:
I've tested it with cached node:8.9.1, the script works as expected.

[SimenB]

@ragelo I think (and hope), it's a separate issue. Mind opening up a dedicated issue with a reproduction? I cannot reproduce it

$ docker run --rm node:8.9.4 node --version
Unable to find image 'node:8.9.4' locally
8.9.4: Pulling from library/node
4176fe04cefe: Pull complete
851356ecf618: Pull complete
6115379c7b49: Pull complete
aaf7d781d601: Pull complete
936f8420f2e4: Pull complete
eab82fe5fcf4: Pull complete
ba2f0c7b6af8: Pull complete
70f351c27b65: Pull complete
Digest: sha256:5afc7736a71bcf24281d9dbff878c771106e0791d56949b1a4e8d27c50424283
Status: Downloaded newer image for node:8.9.4
v8.9.4
$ docker run --rm node:8.9.4 yarn versions
yarn versions v1.3.2
{ http_parser: '2.7.0',
  node: '8.9.4',
  v8: '6.1.534.50',
  uv: '1.15.0',
  zlib: '1.2.11',
  ares: '1.10.1-DEV',
  modules: '57',
  nghttp2: '1.25.0',
  openssl: '1.0.2n',
  icu: '59.1',
  unicode: '9.0',
  cldr: '31.0.1',
  tz: '2017b' }
Done in 0.04s.

[devwfs]

This issue broke our pipeline and we did a workaround as follows:

/opt/yarn/yarn-v1.5.1/bin/yarn install

[orfin]

@PeterDaveHello @chorrell
it seems that this PR: https://github.com/nodejs/docker-node/pull/647
is not backward-compatible and broke Docker Images (e.g. 8.10.0-alpine)

All of our build pipelines are now stuck due to errors such as:

/bin/sh: yarn: not found

```npm ERR! code EEXIST
npm ERR! Refusing to delete /usr/local/bin/yarn: /opt/yarn/bin/yarn symlink target is not controlled by npm /usr/local/bin

Currently, as a workaround one can install yarn via apk.

Example:

in Dockerfile

FROM node:8.10.0-alpine

RUN apk add --no-cache git openssl ca-certificates yarn
```

IMHO tagging model of official node Docker images is broken. It must not be possible to change image of already created image, instead new tag should be used.

[navarroaxel]

This repo needs better unit tests :tea:

[kylecordes]

@navarroaxel I thought the same thing, already looked at the PR, and can see that it adds tests that yarn is actually available. :-)

[oyvindwe]

IMHO tagging model of official node Docker images is broken. It must not be possible to change image of already created image, instead new tag should be used.

@orfin A bit off topic, but I could not agree more. I believed the one of the selling points of Docker was to create repeatable builds. Currently this is only possible using @<tag> notation in Dockerfile, which is not very user developer friendly.

[JustDoItSascha]

I think it's a little bit awkward that changes will be commited to existing tags other than "rc", "nightly" or similar. In my opinion a version like 9.8.0 should never ever be changed. When changes have to be made it's 9.8.1. Many many ci pipelines will be broken today with many many hours of work of devops to find the solution.

[sgargan]

A version label should be cast in stone, particularly for something as widely used as this. Lots of wasted hours and heads scratched over this today.

[n1ru4l]

Finding the root of the issue was not that hard... 😅

However I agree that the tagged versions that represent a full semver specifier should never be modified

[Kosta-Github]

@n1ru4l depends on how you specify the semver for the node images:
should it be inline with the nodejs version? IMHO: yes

but then you might also want to have security patches for the underlying OS for this nodejs version to somehow show up (automatically)?

[davidblurton]

I would be very interested in reading a blog post on how this mistake happened and what steps you've taken to ensure that it won't happen again.

[Hypnosphi]

@Kosta-Github I think it would be enough to match corresponding minor version of node

[joar]

@davidblurton Here's a draft: https://github.com/nodejs/docker-node/pull/650

[n1ru4l]

@Kosta-Github I forgot about the security patches 😅

[chorrell]

@davidblurton As I see it there were few issues:

1. We didn't have a basic test of yarn which would have caught this early (fixed in #650)
2. We should not have immediately pushed the changes to the docker hub. Instead, the changes to how yarn is installed/linked should have been grouped with a Node.js (and therefor docker tag) update.

[SimenB]

Also note that the tag is updated automatically by docker hub to account for security fixes in the underlying OS, so the only way to truly lock it down is to use the id of the image, not the tag

[oyvindwe]

A docker image is an artifact of its own, and thus needs it own version. The image version may not need to be a full semver though, only a build increment. node:alpine-8.6.0_16 for the 17th release of node-alpine-8.6.0 might be sufficient. node:alpine-8-6 and node:alpine-8 might still be mutable "tags".

[ojab]

As @SimenB already written, image id is immutable identifier of the docker image, i. e. instead of

$ docker pull node:9.8.0-alpine
9.8.0-alpine: Pulling from library/node
Digest: sha256:1fbcd77d0eb2af765d24ae4758b5a94b0d55c6002a15a4431d55795a449ebd3d
Status: Image is up to date for node:9.8.0-alpine

you should use

$ docker pull node@sha256:1fbcd77d0eb2af765d24ae4758b5a94b0d55c6002a15a4431d55795a449ebd3d

[LaurentGoderre]

There's a good reason for tags not being immutable. One of the main benefit of Docker is that you inherit security fixes to underlying images. Sure there's a chance that a change to the base image might break a build but it's a trade-off that I personally gladly take to get the security benefits.

[robinclaes]

Happening to our builds too...!

[oyvindwe]

@ojab as I wrote in my first comment, I am perfectly aware how to lock down a version. However, I find the use of “tag” misleading. In version control systems, a tag usually points to a given revision or commit (possibly on a specific branch). When a “tag” is updated, it is more like a VCS branch. The main problem with using the digest notation is that you don’t immediately see which tag it originated from.

Other repositories as Maven, NPM, Yum, etc., contain versioned artifacts. I do not see why Docker should not.

[oyvindwe]

@LaurentGoderre There exist syntaxes for specifying that you want the latest version (at some level) for semver. But providing people the possibility to easily lock down the version would be nice. You never know when even a security patch breaks your production system.

[LaurentGoderre]

We are staying consistent with the other docker official images. Maybe there will be a solution to this issue in the future (in the form of a lockfile or something)

[Shahor]

Hey guys,

thanks for the quick fix <3

That being said we continue having the problem when building our images :(
Is there something peculiar that should be done to get back to work properly?

[SimenB]

You need to wait for the image to be published. It's currently being built

[oyvindwe]

@LaurentGoderre consistency is a good thing. :) Repeatable builds provides even better consistency. My comment is not limited to the node image. Where is the right place to raise this issue?

[CalebKing3]

I'm happy we were not the only one experiencing this issue 😅 , reverting to a earlier tag was our solution but kudos to the community on this form for providing suggestions.

[cecton]

I'm happy we were not the only one experiencing this issue 😅 , reverting to a earlier tag was our solution but kudos to the community on this form for providing suggestions.

I think the whole world ended up here when they realized their builds were not passing lol!

[djbingham]

The fix seems to have made its way through the system. I've just had a GitLab CI build pass using the node:alpine image.

[danielwhatmuff]

@SimenB @LaurentGoderre - PR with changes to prevent this sort of thing happening again - https://github.com/nodejs/docker-node/pull/658

[rarkins]

I wrote up a blog post to explain how/why this happened, why it's not completely unexpected and can happen again, and what you can do to protect against it in future: https://renovateapp.com/blog/docker-mutable-tags

In summary:

1. Docker tags can change at any time. They are mutable and that is by design, not fault
2. Even a Docker tag that looks like semver (e.g. node:8.10.0) still has the properties of a tag and can be changed/updated at any time, like happened here
3. There are valid reasons for the image a tag points to to be changed (such as refactoring the Dockerfile or a security patch to the base), so it's not a good idea to insist that a tag that looks like a semver should never be changed
4. The only way to ensure your build is reproducible/consistent is to use Docker digests as the identifier instead of the tag
5. You can conveniently still include a Docker tag in the digest for human readability, e.g. node:8.10.0-alpine@sha256:a55d3e87802b2a8464b3bfc1f8c3c409f89e9b70a31f1dccce70bd146501f1a0
6. If you were using digests for your node base images instead of tags, you not only wouldn't have "accidentally" upgraded to the new broken image in the first place, and even if you had updated on purpose then you could have immediately rolled back the digest in your Dockerfile once you discovered the break, instead of waiting hours
7. Beyond that, these digests are a little "developer-unfriendly" to update by hand, but open source tools to automate it exist

[cecton]

I believe there is a much simpler solution: if you use Docker for your work, host your own registry and update the images when you need to.

[oyvindwe]

@cecton no build time Internet is good for private projects, but not open source development. Also, it does not solve the problems with not having a repeatable build when debugging.

[oyvindwe]

That should read: “no build time Internet dependencies”

[cecton]

Yes actually, you made a good point... I was more thinking about professional projects.

[jwineman]

Fixed by adding

RUN apk add --no-cache --virtual .build-deps \
    ca-certificates \
    wget \
    tar && \
    cd /usr/local/bin && \
    wget https://yarnpkg.com/latest.tar.gz && \
    tar zvxf latest.tar.gz && \
    ln -s /usr/local/bin/dist/bin/yarn.js /usr/local/bin/yarn.js && \
    apk del .build-deps

from https://hub.docker.com/r/jfyne/node-alpine-yarn/dockerfile to my Dockerfile.