Node: Tarball created on Mac throws errors when extracted on Linux

Created on 11 Sep 2018 · 30Comments · Source: nodejs/node

Version: 10.10.0
Platform: Linux sgallaghp50.sgallagh.rht 4.17.19-200.fc28.x86_64 #1 SMP Fri Aug 24 15:47:41 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Subsystem: Tarball

Download http://nodejs.org/dist/v10.10.0/node-v10.10.0.tar.gz on Linux.
Run tar xvf node-v10.10.0.tar.gz

Results:

tar: Ignoring unknown extended header keyword 'SCHILY.dev'
tar: Ignoring unknown extended header keyword 'SCHILY.ino'
tar: Ignoring unknown extended header keyword 'SCHILY.nlink'
tar: Ignoring unknown extended header keyword 'SCHILY.dev'
tar: Ignoring unknown extended header keyword 'SCHILY.ino'
tar: Ignoring unknown extended header keyword 'SCHILY.nlink'
tar: Ignoring unknown extended header keyword 'SCHILY.dev'
tar: Ignoring unknown extended header keyword 'SCHILY.ino'
tar: Ignoring unknown extended header keyword 'SCHILY.nlink'
tar: Ignoring unknown extended header keyword 'SCHILY.dev'
tar: Ignoring unknown extended header keyword 'SCHILY.ino'
tar: Ignoring unknown extended header keyword 'SCHILY.nlink'
tar: Ignoring unknown extended header keyword 'SCHILY.dev'
tar: Ignoring unknown extended header keyword 'SCHILY.ino'
tar: Ignoring unknown extended header keyword 'SCHILY.nlink'
tar: Ignoring unknown extended header keyword 'SCHILY.dev'
tar: Ignoring unknown extended header keyword 'SCHILY.ino'
tar: Ignoring unknown extended header keyword 'SCHILY.nlink'
tar: Ignoring unknown extended header keyword 'SCHILY.dev'
tar: Ignoring unknown extended header keyword 'SCHILY.ino'
tar: Ignoring unknown extended header keyword 'SCHILY.nlink'
tar: Ignoring unknown extended header keyword 'SCHILY.dev'
tar: Ignoring unknown extended header keyword 'SCHILY.ino'
tar: Ignoring unknown extended header keyword 'SCHILY.nlink'
tar: Ignoring unknown extended header keyword 'SCHILY.dev'
tar: Ignoring unknown extended header keyword 'SCHILY.ino'
tar: Ignoring unknown extended header keyword 'SCHILY.nlink'
tar: Ignoring unknown extended header keyword 'SCHILY.dev'
tar: Ignoring unknown extended header keyword 'SCHILY.ino'
tar: Ignoring unknown extended header keyword 'SCHILY.nlink'
tar: Ignoring unknown extended header keyword 'SCHILY.dev'
tar: Ignoring unknown extended header keyword 'SCHILY.ino'
tar: Ignoring unknown extended header keyword 'SCHILY.nlink'
tar: Ignoring unknown extended header keyword 'SCHILY.dev'
tar: Ignoring unknown extended header keyword 'SCHILY.ino'
tar: Ignoring unknown extended header keyword 'SCHILY.nlink'
tar: Ignoring unknown extended header keyword 'SCHILY.dev'
tar: Ignoring unknown extended header keyword 'SCHILY.ino'
tar: Ignoring unknown extended header keyword 'SCHILY.nlink'
tar: Ignoring unknown extended header keyword 'SCHILY.dev'
tar: Ignoring unknown extended header keyword 'SCHILY.ino'
tar: Ignoring unknown extended header keyword 'SCHILY.nlink'
tar: Ignoring unknown extended header keyword 'SCHILY.dev'
tar: Ignoring unknown extended header keyword 'SCHILY.ino'
tar: Ignoring unknown extended header keyword 'SCHILY.nlink'
tar: Ignoring unknown extended header keyword 'SCHILY.dev'
tar: Ignoring unknown extended header keyword 'SCHILY.ino'
tar: Ignoring unknown extended header keyword 'SCHILY.nlink'
tar: Ignoring unknown extended header keyword 'SCHILY.dev'
tar: Ignoring unknown extended header keyword 'SCHILY.ino'
tar: Ignoring unknown extended header keyword 'SCHILY.nlink'
tar: Ignoring unknown extended header keyword 'SCHILY.dev'
tar: Ignoring unknown extended header keyword 'SCHILY.ino'
tar: Ignoring unknown extended header keyword 'SCHILY.nlink'

It appears that in 10.10.0, Node.js may have started using BSD tar on a mac to create the tarball. It would be much better if it were to use GNU tar, as this will produce a tarball with only the standard headers.

build

Source

sgallagher

👍4

Most helpful comment

Calling this solved. As per, nodejs/build#1777, nightlies as of today use Linux to generate all non-binary assets (source, headers, docs). This will apply to Node 12+. Older versions will stick with the current path. I understand this may not be 100% satisfactory because it doesn't change Node 8 or 10, but I hope it's good enough so we don't have to mess too much with older build processes because breakage (even obscure breakage) is not something we're keen to tempt.

rvagg on 18 Apr 2019

👍2

All 30 comments

/cc @nodejs/build @nodejs/release

MylesBorins on 11 Sep 2018

I can't reproduce and I have the same version of Fedora as in the OP

targos on 11 Sep 2018

For the record, I'm using tar-1.30-6.fc29.x86_64

I just realized I'm still reporting a Fedora 28 kernel, but I'm on the Fedora 29 development branch.

sgallagher on 11 Sep 2018

Sorry. I can indeed reproduce (tar xvf node-v10.10.0.tar.gz | grep Ignoring" to see the same output).

I doesn't really throw an error, though (the exit code is 0). Is there an issue with the extracted files?

targos on 11 Sep 2018

There's no actual error, but it's noisy. I noticed it because I have a script that goes in and sanitizes the tarball to eliminate all the content that isn't actually redistributable (such as some of the OpenSSL algorithms). My script suppresses stdout but leaves stderr so I can see if anything goes wrong.

It never printed these messages before, so I assume something changed. In general, warnings aren't a great user experience and I'd be concerned about using a non-standard TAR because it might lead to actual incompatibilities in the future.

sgallagher on 11 Sep 2018

The tarball is created on Mac but that has been the case for previous releases as well.

@nodejs/build Did anything change on the release machine (osx1010-release-tar) between v10.9.0 and v10.10.0?

This is what was executed to create the file:

tar -cf node-v10.10.0.tar node-v10.10.0
gzip -c -f -9 node-v10.10.0.tar > node-v10.10.0.tar.gz

targos on 11 Sep 2018

It seems this happens when you use BSD tar instead of GNU tar.

mscdex on 11 Sep 2018

I think we moved from building on 10.10 to 10.11... potentially with
another provider.

On Tue, Sep 11, 2018, 11:06 AM mscdex notifications@github.com wrote:

It seems this happens when you use BSD tar instead of GNU tar.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/nodejs/node/issues/22805#issuecomment-420307295, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAecV8wXD3gqEVK3G-tLOAUqSmbppZYMks5uZ9FkgaJpZM4WjPTo
.

MylesBorins on 11 Sep 2018

I don't see an obvious difference between the 10.10 and 10.11 hosts (will update with details).

https://ci-release.nodejs.org/computer/release-requireio-osx1010-x64-1/
https://ci-release.nodejs.org/computer/release-macstadium-macos10.11-x64-1/

# 10.10
Result: bsdtar 2.8.3 - libarchive 2.8.3
# 10.11
Result: bsdtar 2.8.3 - libarchive 2.8.3

The job seems to run make tar-upload.

gibfahn on 11 Sep 2018

@gibfahn I suspect that on 10.10, gnu tar was also installed and probably owned the tar name. Updating to 10.11 may have replaced this with BSD tar.

sgallagher on 11 Sep 2018

But Node 10.x releases are still done with 10.10:

https://ci-release.nodejs.org/job/iojs+release/3740/console

Skipping osx1011-release-tar for Node.js 10
Triggering iojs+release » osx1010-release-tar

targos on 11 Sep 2018

But Node 10.x releases are still done with 10.10:

We had some trouble with release-requireio-osx1010-x64-1 as recorded in:
https://github.com/nodejs/build/issues/1484
https://github.com/nodejs/build/issues/1486
Some there might have been an general software update

refack on 11 Sep 2018

P.S. might also be coming from the new npm verions (https://github.com/nodejs/node/pull/22591)

refael@refaelux:/mnt/d/code/temp$ tar xzvf node-v10.10.0.tar.gz 2>&1 | grep -C10 Ign
node-v10.10.0/deps/npm/test/npm_cache/index-v5/a6/c9/
node-v10.10.0/deps/npm/test/npm_cache/index-v5/a6/c9/dbe7acc04b7cea7b99c68f3821cf6d7f49caba301e383b31143bd9670f28
node-v10.10.0/deps/npm/test/npm_cache/index-v5/77/52/
node-v10.10.0/deps/npm/test/npm_cache/index-v5/77/52/b7394cb3a27a44448e3ac4d6834e73ca79c301c7108c94b0ba23ca97b71c
node-v10.10.0/deps/npm/test/npm_cache/index-v5/19/db/
node-v10.10.0/deps/npm/test/npm_cache/index-v5/19/db/491ce5fd3b8f5521f62a3d208c8b91325f3f8fe347347649e2e7cb8c10b2
node-v10.10.0/deps/npm/test/npm_cache/content-v2/sha512/
node-v10.10.0/deps/npm/test/npm_cache/content-v2/sha512/76/
node-v10.10.0/deps/npm/test/npm_cache/content-v2/sha512/f3/
node-v10.10.0/deps/npm/test/npm_cache/content-v2/sha512/f3/e8/
tar: Ignoring unknown extended header keyword 'SCHILY.dev'
tar: Ignoring unknown extended header keyword 'SCHILY.ino'
tar: Ignoring unknown extended header keyword 'SCHILY.nlink'
node-v10.10.0/deps/npm/test/npm_cache/content-v2/sha512/f3/e8/2d6a0b75ad5cebd09177d93f572dbd8b877ee9f1505b2e84e03810fa0412e4904060be7d2a4df4221b1bb92884db886826bf8cd26634667a2d103a999438
node-v10.10.0/deps/npm/test/npm_cache/content-v2/sha512/76/39/
tar: Ignoring unknown extended header keyword 'SCHILY.dev'
tar: Ignoring unknown extended header keyword 'SCHILY.ino'
tar: Ignoring unknown extended header keyword 'SCHILY.nlink'
node-v10.10.0/deps/npm/test/npm_cache/content-v2/sha512/76/39/4b378512c68bf209b433e06b71df27a45f7e7be35f174a0f83bce7799628b74dbe993c18b1c12e899a1ed7b159470b382180d1f0a5c4098ac6092cda1a8f
node-v10.10.0/deps/npm/test/npm_cache/_logs/2018-08-29T02_01_27_486Z-debug.log

refack on 11 Sep 2018

@gibfahn I suspect that on 10.10, gnu tar was also installed and probably owned the tar name. Updating to 10.11 may have replaced this with BSD tar.

@sgallagher type -a tar and type -a gtar produced the same results on both machines, this is the only tar in the path.

gibfahn on 12 Sep 2018

Just for the record:

10.10.0:

[sgallagh@sgallaghp50:shm]$ curl http://nodejs.org/dist/v10.10.0/node-v10.10.0.tar.gz --output - |tar zx  > /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  8 34.2M    8 2855k    0     0  14.8M      0  0:00:02 --:--:--  0:00:02 14.7Mtar: Ignoring unknown extended header keyword 'SCHILY.dev'
tar: Ignoring unknown extended header keyword 'SCHILY.ino'
tar: Ignoring unknown extended header keyword 'SCHILY.nlink'
tar: Ignoring unknown extended header keyword 'SCHILY.dev'
tar: Ignoring unknown extended header keyword 'SCHILY.ino'
tar: Ignoring unknown extended header keyword 'SCHILY.nlink'
tar: Ignoring unknown extended header keyword 'SCHILY.dev'
tar: Ignoring unknown extended header keyword 'SCHILY.ino'
tar: Ignoring unknown extended header keyword 'SCHILY.nlink'
tar: Ignoring unknown extended header keyword 'SCHILY.dev'
tar: Ignoring unknown extended header keyword 'SCHILY.ino'
tar: Ignoring unknown extended header keyword 'SCHILY.nlink'
tar: Ignoring unknown extended header keyword 'SCHILY.dev'
tar: Ignoring unknown extended header keyword 'SCHILY.ino'
tar: Ignoring unknown extended header keyword 'SCHILY.nlink'
tar: Ignoring unknown extended header keyword 'SCHILY.dev'
tar: Ignoring unknown extended header keyword 'SCHILY.ino'
tar: Ignoring unknown extended header keyword 'SCHILY.nlink'
tar: Ignoring unknown extended header keyword 'SCHILY.dev'
tar: Ignoring unknown extended header keyword 'SCHILY.ino'
tar: Ignoring unknown extended header keyword 'SCHILY.nlink'
tar: Ignoring unknown extended header keyword 'SCHILY.dev'
tar: Ignoring unknown extended header keyword 'SCHILY.ino'
tar: Ignoring unknown extended header keyword 'SCHILY.nlink'
tar: Ignoring unknown extended header keyword 'SCHILY.dev'
tar: Ignoring unknown extended header keyword 'SCHILY.ino'
tar: Ignoring unknown extended header keyword 'SCHILY.nlink'
tar: Ignoring unknown extended header keyword 'SCHILY.dev'
tar: Ignoring unknown extended header keyword 'SCHILY.ino'
tar: Ignoring unknown extended header keyword 'SCHILY.nlink'
tar: Ignoring unknown extended header keyword 'SCHILY.dev'
tar: Ignoring unknown extended header keyword 'SCHILY.ino'
tar: Ignoring unknown extended header keyword 'SCHILY.nlink'
tar: Ignoring unknown extended header keyword 'SCHILY.dev'
tar: Ignoring unknown extended header keyword 'SCHILY.ino'
tar: Ignoring unknown extended header keyword 'SCHILY.nlink'
tar: Ignoring unknown extended header keyword 'SCHILY.dev'
tar: Ignoring unknown extended header keyword 'SCHILY.ino'
tar: Ignoring unknown extended header keyword 'SCHILY.nlink'
tar: Ignoring unknown extended header keyword 'SCHILY.dev'
tar: Ignoring unknown extended header keyword 'SCHILY.ino'
tar: Ignoring unknown extended header keyword 'SCHILY.nlink'
tar: Ignoring unknown extended header keyword 'SCHILY.dev'
tar: Ignoring unknown extended header keyword 'SCHILY.ino'
tar: Ignoring unknown extended header keyword 'SCHILY.nlink'
tar: Ignoring unknown extended header keyword 'SCHILY.dev'
tar: Ignoring unknown extended header keyword 'SCHILY.ino'
tar: Ignoring unknown extended header keyword 'SCHILY.nlink'
tar: Ignoring unknown extended header keyword 'SCHILY.dev'
tar: Ignoring unknown extended header keyword 'SCHILY.ino'
tar: Ignoring unknown extended header keyword 'SCHILY.nlink'
tar: Ignoring unknown extended header keyword 'SCHILY.dev'
tar: Ignoring unknown extended header keyword 'SCHILY.ino'
tar: Ignoring unknown extended header keyword 'SCHILY.nlink'
100 34.2M  100 34.2M    0     0  26.0M      0  0:00:01  0:00:01 --:--:-- 26.0M

vs 10.9.0:

[sgallagh@sgallaghp50:shm]$ curl http://nodejs.org/dist/v10.9.0/node-v10.9.0.tar.gz --output - |tar zx  > /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 34.1M  100 34.1M    0     0  28.8M      0  0:00:01  0:00:01 --:--:-- 28.8M

I mostly wanted to rule out that something changed on my end with our version of GNU tar. But it's definitely on the Node.js end.

sgallagher on 12 Sep 2018

👍2

This occurs in BSD tar when multi-byte characters appear in filenames within the archive, apparently. Or at the very least, that is one way to trigger it:

$ uname -sr
Darwin 17.7.0
$ touch a
$ tar cvf a.tar a
a a
$ strings a.tar | grep SCHILY
$ touch 未
$ tar cvf 未.tar 未
a 未
$ strings 未.tar | grep SCHILY
23 SCHILY.dev=16777220
25 SCHILY.ino=4302147341
18 SCHILY.nlink=1

apaprocki on 21 Sep 2018

This is still an issue with the node v10.11.0 tarball.

mscdex on 30 Sep 2018

/cc @nodejs/build can we get osx using gnu tar in our infra?

On Sat, Sep 29, 2018, 8:49 PM mscdex notifications@github.com wrote:

This is still an issue with the node v10.11.0 tarball.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/nodejs/node/issues/22805#issuecomment-425684914, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAecV9-3quo67zd1k83mF_MXEOHLMtU7ks5ugBUdgaJpZM4WjPTo
.

MylesBorins on 30 Sep 2018

This is still an issue with the node v10.13.0 tarball.

mscdex on 10 Nov 2018

/cc @nodejs/build

On Sat, Nov 10, 2018, 5:37 AM mscdex <[email protected] wrote:

This is still an issue with the node v10.13.0 tarball.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/nodejs/node/issues/22805#issuecomment-437557698, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAecV_GY_c8zt9gsWrQeFAz8RjwRWGNhks5utlf3gaJpZM4WjPTo
.

MylesBorins on 10 Nov 2018

I'm pretty sure we never installed a non-standard tar on any release machines. It's very possible that there was an automatic update, it's kind of hard to control those on macOS.

As far as I can tell, there's just an annoyance here, not an actual error? Is it worth our while having to jump through the hoops required to get gnu-tar installed on all our release machines? And why are we determining gnu-tar to be correct in not knowing these extensions? Maybe it needs to adapt to bsd-tar?

rvagg on 11 Nov 2018

FWIW I've encountered some systems with older versions of (gnu) tar that will actually exit with a non-zero exit code and produce error messages when extracting the tarballs, despite the contents being extracted seemingly correctly. So I'd classify it as more than an annoyance.

mscdex on 11 Nov 2018

OK, so can we identify particular files that are causing problems? Maybe we can deal with it from that end.

rvagg on 14 Nov 2018

As noted earlier by @refack the messages are displayed when unpacking files from npm. Specifically, it seems like it's files/directories located in subdirectories within 'deps/npm/test/npm_cache/_cacache/content-v2/sha512'.

mscdex on 14 Nov 2018

This continues to be a problem with tarballs on nodejs.org, including the other release lines now (e.g. v8.15.0 released today).

Can we revert back to using GNU tar on the machine(s) generating tarballs? As far as installation goes, it's as simple as doing brew install gnu-tar if homebrew is installed (the binary is available as gtar but could be aliased) or brew install gnu-tar --with-default-names to have tar instead of gtar.

mscdex on 26 Dec 2018

/cc @nodejs/build can we get this fixed?

MylesBorins on 3 Jan 2019

👍1

@gdams might be able to look at this, but he needs to get the additional AIX machine added first.

mhdawson on 10 Jan 2019

Adding
--warning=no-unknown-keyword
to your tar extraction command is a low-effort workaround to suppress the nagging. You may need to move the option to the beginning or end of the command depending on your parameters.