Distributions: N: Skipping acquire of configured file 'main/binary-arm64/Packages' as repository 'https://deb.nodesource.com/node_6.x xenial InRelease' doesn't support architecture 'arm64'

[vielmetti]

• Version: v4.2.6
• Platform: Linux armv8hello.local.lan 4.4.0-38-generic #57-Ubuntu SMP Wed Sep 7 10:19:14 UTC 2016 aarch64 aarch64 aarch64 GNU/Linux

I'm looking to install Node v6 on ARMv8 (aarch64) on Ubuntu. I see at https://nodejs.org/en/download/current/ that there's a version compiled at 6.9.1. So far so good.

However, when installing from apt-get I see

N: Skipping acquire of configured file 'main/binary-arm64/Packages' as repository 'https://deb.nodesource.com/node_6.x xenial InRelease' doesn't support architecture 'arm64'

Is there some other repository I can go to to get node@6 on xenial? thanks.

[vielmetti]

Looks like this was previously commented on at #262 - not sure if a policy has changed since then.

[chrislea]

There haven't been any policy changes at this time. We currently only support armhf for ARM architectures. If there is sufficient demand we may re-evaluate at some point in the future.

[mmoll]

What is the "official" way to show that demand? ;)

[chrislea]

@mmoll You just did. No promises, and we don't currently have arm64 support planned, but we are always paying attention to what people are asking for. Thanks for letting us know!

[mmoll]

BTW, if any questions should arise, I'm running ARMv7 (ODroid C1s) and ARMv8 (ODroid C2s) build nodes for theforeman.org, we're using pbuilder (managed by Puppet, triggered by Jenkins) to build our packages and that's working quite well.

[David-Noble-at-work]

I'll add my vote for arm64 support

[ontoport]

I just stumbled about the same issue and would very much appreciate this architecture (arm64) being supported in near future. I'm working with ODROID C2-s.

[EmbeddedAndroid]

Voting for arm64 support as well!

[WhyKickAmooCow]

Voting for arm64

[ThomasKaiser]

What do you guys think about simply installing the armhf variant on arm64 distros? This will work on every 32-bit or 64-bit Debian/Ubuntu (though not outdated Raspberries with armv6l CPU core):

dpkg --add-architecture armhf
curl -s https://deb.nodesource.com/gpgkey/nodesource.gpg.key | apt-key add -
echo 'deb [arch=armhf] https://deb.nodesource.com/node_6.x xenial main' > /etc/apt/sources.list.d/nodesource.list
apt update && apt install nodejs:armhf

I know that people fear performance drops when using armhf binaries on arm64 architecture but the tests we ran show that's negligible.

[mmoll]

@ThomasKaiser multiarch could also be disabled for whatever reasons and there might be problems with native extensions (e.g. node-sass).

[ThomasKaiser]

So this needs some testing :)

multiarch could also be disabled for whatever reasons

Do you know of any arm64 based OS image doing that? IMO this would be a rather stupid move since still some huge packages like eg Firefox or Chromium show a lot of instabilities/crashes when using their native architecture while armhf is working just fine (can only speak for Banana Pi M64, Pine64, ODROID-C2 and Orange Pi PC 2 though)

[mmoll]

Do you know of any arm64 based OS image doing that?

No, but nothing would hold anybody off that.

some huge packages like eg Firefox or Chromium show a lot of instabilities/crashes when using their native architecture

At least when I tested chromium about two months ago on aarch64 (Odroid C2) it worked fine.

[ThomasKaiser]

At least when I tested chromium about two months ago on aarch64 (Odroid C2) it worked fine.

Slightly off-topic but do you get Octane V2 to run without freezing?

I reference this Chromium issue for another another reason: memory constraints (see also here). Mini servers with aarch64 CPUs but just 512MB RAM like NanoPi NEO 2 are reality now, a lot more will follow soon and arm64 is simply fat:

root@pine64:~/node-v6.10.1-linux-arm64/bin# du -sh node /usr/bin/nodejs
28M node
19M /usr/bin/nodejs
root@pine64:~/node-v6.10.1-linux-arm64/bin# file node /usr/bin/nodejs
node:            ELF 64-bit LSB executable, ARM aarch64, version 1 (GNU/Linux), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, BuildID[sha1]=502250b41ce34687398105159f01d63d9c9d30cf, not stripped
/usr/bin/nodejs: ELF 32-bit LSB executable, ARM, EABI5 version 1 (GNU/Linux), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, for GNU/Linux 3.2.0, BuildID[sha1]=98758978a7952f54a2836f8c532a458d40d68c0e, stripped

On a related note: I already thought about adding a variant to Armbian's build system to combine 64-bit kernel with a 32-bit armhf userland for exactly that reason. While this is bad since performance will decrease somewhat for sure it could be a good idea since anything memory intensive that could cause swapping/paging would benefit clearly from applications not running out of physical memory. On ODROID-C2 this might never be an issue due to 2GB DRAM there and even swapping hurts not that much when running off the fast eMMC we can use there. But it will kill a NEO 2 running off an average SD card (since they suck totally when it's about random IO, this could be considered 'swapping to death')

[vielmetti]

It would be a bad idea to mislabel an armhf binary as aarch64 and expect it to work everywhere. While many of the single-board ARMv8 based computers also run armhf binaries, there are also powerful data center class systems in place now (and more in a pipeline) that are 64-bit only.

How are the ARM binaries being built now? If the build process is sufficiently automated it should be a small matter of CI integration to get aarch64 build/test on Packet's fast 2A servers (96 cores, 128G memory). @chrislea is there some part of the project team that's in charge of CI that I could engage with?

[ThomasKaiser]

It would be a bad idea to mislabel an armhf binary as aarch64 and expect it to work everywhere.

Just for the record: Nobody is talking about this but we speak about multiarch that allows armhf binaries inside an arm64 Debian/Ubuntu distro with CPU cores brought up in AArch64 state so it doesn't matter whether the CPU is '64-bit only' or not (64-bit would mean only AArch64 but no AArch32 -- please compare with 'Naming conventions' and 'Boot_modes' here.

[mmoll]

@ThomasKaiser very brief, as this is really off-topic: at the moment I have only headless aarch64 build slaves without X, so no idea. Regarding swap: I use iSCSI over Gigabit ethernet as block storage, so the situation is a bit different.

[ThomasKaiser]

@mmoll Ok, understood.

Just for the record: This is Octane 2.0 benchmark for nodeJS running in Xenial 16.04.2 on Pine64 with kernel 4.9.0 (cpufreq scaling and throttling disabled and running with boring 864 MHz only to prevent any 'strange effects'). Same numbers:

armhf (full octane-v2-armhf.txt log):

Richards            : 1869
DeltaBlue           : 3547
Crypto              : 2046
RayTrace            : 3960
EarleyBoyer         : 2590
RegExp              : 314
Splay               : 1033
SplayLatency        : 5232
NavierStokes        : 2150
PdfJS               : 1248
Mandreel            : 1176
MandreelLatency     : 2880
Gameboy             : 2299
CodeLoad            : 1327
Box2D               : 1547
zlib                : 5203
Typescript          : 3194
----
Score (version 9): 2030
 duration  130.539  seconds

arm64 (full octane-v2-arm64.txt log):

Richards            : 2016
DeltaBlue           : 3266
Crypto              : 2458
RayTrace            : 4210
EarleyBoyer         : 2359
RegExp              : 303
Splay               : 557
SplayLatency        : 3952
NavierStokes        : 2642
PdfJS               : 1284
Mandreel            : 1421
MandreelLatency     : 2798
Gameboy             : 2266
CodeLoad            : 1215
Box2D               : 1468
zlib                : 6152
Typescript          : 2858
----
Score (version 9): 1971
 duration  132.897  seconds

Edit: So by trusting into passive benchmarking attempts there is no advantage running 64-bit Binaries (I've to admit that I'm not a passive benchmarking fan for obvious reasons. Might be interesting whether some people could just give it a try, compare both attempts and let vmstat 3 monitor what's happening in reality. I did this now just to realize that this benchmark is rather lightweight and mostly single-threaded so the numbers above are worthless anyway)

[ThomasKaiser]

Memory consumption running arm64 variant living in /usr/local/:

root@pine64:~# vmstat 10
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 1  0      0 1931816   3776  62388    0    0   106    29  251  291  1  0 97  1  0
 1  0      0 1745008   3776  70980    0    0   854     0  269   65 23  1 76  0  0
 1  0      0 1728980   3776  71132    0    0    15     0  312   66 27  0 73  0  0
 1  0      0 1707016   3776  71132    0    0     0     0  294   50 26  0 74  0  0
 1  0      0 1513940   3776  71148    0    0     0     0  340   91 26  1 73  0  0
 1  0      0 1493408   3776  71180    0    0     2     0  336   72 28  0 72  0  0
 2  0      0 1479908   3776  71184    0    0     0     0  339  117 27  0 73  0  0
 1  0      0 1459832   3776  71192    0    0     0     0  337   73 28  0 72  0  0
 1  0      0 1456404   3776  71192    0    0     0     0  264   25 25  0 75  0  0
 2  0      0 1451088   3776  71192    0    0     0     0  342   52 29  0 71  0  0
 1  0      0 1425128   3912  71256    0    0    16     1  604  224 37  0 62  0  0
 1  0      0 1415104   3912  71256    0    0     0     0  410   91 32  0 68  0  0
 1  0      0 1370308   3912  71256    0    0     0     0  282   35 26  0 74  0  0
 1  0      0 1373912   3912  71264    0    0     0     0  282   36 26  0 74  0  0
^C

Now switching back to the armhf variant:

root@pine64:~# export PATH=/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
root@pine64:~# hash -r
root@pine64:~# which npm
/usr/bin/npm
root@pine64:~# npm install benchmark-octane -g
/usr/bin/benchmark-octane -> /usr/lib/node_modules/benchmark-octane/bin/benchmark-octane
/usr/lib
└── [email protected] 

root@pine64:~# echo 3 > /proc/sys/vm/drop_caches
root@pine64:~# benchmark-octane

Memory consumption with armhf:

root@pine64:~# vmstat 10
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 1  0      0 1950960   2240  59972    0    0   102    53  588  652  6  1 93  1  0
 1  0      0 1887284   3160  70660    0    0  1160     0  296  108 22  1 76  1  0
 1  0      0 1877004   4692  71400    0    0   225     0  388  173 27  0 73  0  0
 1  0      0 1873060   4692  71408    0    0     0    22 2113 3713 26  0 69  5  0
 1  0      0 1737404   4692  71432    0    0     0     0  294   51 26  0 74  0  0
 1  0      0 1699376   4692  71524    0    0     9     0  343   86 28  0 72  0  0
 2  0      0 1688028   4692  71528    0    0     0     0  388  155 28  0 71  0  0
 1  0      0 1680288   4692  71528    0    0     0     0  289   48 26  0 74  0  0
 1  0      0 1677928   4692  71528    0    0     0     0  265   28 25  0 75  0  0
 1  0      0 1677944   4692  71528    0    0     0     0  266   30 25  0 75  0  0
 1  0      0 1669844   4692  71532    0    0     0     0  491  128 36  1 63  0  0
 1  0      0 1710340   4692  71532    0    0     0     0  410  131 31  0 69  0  0
 1  0      0 1709308   4692  71528    0    0     0     0  304   50 27  0 73  0  0
 1  0      0 1685516   4692  71532    0    0     0     0  277   36 25  0 74  0  0
^C

Still same performance numbers (did a kernel update to 4.10 in between) but memory consumption 'slightly' differs. With the arm64 binaries the benchmark eats up to 663MB (2049724 - 1370308) while armhf needs just 363MB (2049724 - 1677928). What am I'm doing wrong here?

(full armbianmonitor -u debug log)

[vielmetti]

Thanks for the pointer to Octane. Similar test results on the Packet 2A server. (96 core, Cavium).

Richards            : 5955
DeltaBlue           : 6056
Crypto              : 6356 
RayTrace            : 8524
EarleyBoyer         : 5776
RegExp              : 877
Splay               : 5414
SplayLatency        : 10116
NavierStokes        : 6183
PdfJS               : 3087
Mandreel            : 3978
MandreelLatency     : 7463
Gameboy             : 7517
CodeLoad            : 3331
Box2D               : 2929
zlib                : 14755
Typescript          : 5842
----
Score (version 9): 5283
 duration  70.897  seconds

[ThomasKaiser]

@vielmetti your results with the Cavium confirm single-threaded behaviour of this Octane benchmark. ~2000 score on the most boring 64-bit ARM SoC around (Allwinner A64) running at 864 MHz and just 2.5 times faster on your 96 core beast.

Anyway: the interesting question for me is still whether there's a performance drop or not when running NodeJS armhf binaries instead of the arm64 variants. At least with this benchmark the memory footprint difference is that huge that it might even matter on your Cavium machine with IIRC 128GB DRAM. Since memory requirements might start to limit the number of parallel processes.

I got yesterday some feedback in a developer channel confirming memory requirements somewhat but pointing also to some tunables.

[vielmetti]

@ThomasKaiser The memory requirements are interesting and relevant. If there's any kind of benchmarking you might suggest to exercise all of the cores at once - which might be frankly as simple as Octane + GNU Parallel. Let me try that first.

% seq 1 96 | parallel nodejs ./run.js > ~/octane-96.out &

shows a max of about 53G of memory used when 96 jobs are running at once, or 537G per each. So this exercise, at least, doesn't suggest that the memory footprint matters as much as it might. That's not really as much of a surprise as it might be, though I am aware of e.g. a similar Java workload on ARMv8 where we are memory bound even on a 128G machine because of the desire to scale out sideways.

[ThomasKaiser]

In the meantime I tried it with v7.8.0 (V8 version : 5.5.372.43, with v6.10.1 before it was V8 version 5.1.281.95) as suggesteted by @apritzel.

With arm64 maximum memory consumption is now at 593MB (70MB less compared to 663MB with v6.10.1/5.1.281.95 before -- but still way too much to fit into 512 MB DRAM) and with armhf it's now 370MB (7MB more compared to 363MB before) while overall scores are slightly faster but still identical (2066 vs 2057).

arm64:

Richards            : 1956
DeltaBlue           : 3332
Crypto              : 2434
RayTrace            : 4201
EarleyBoyer         : 2309
RegExp              : 331
Splay               : 706
SplayLatency        : 3162
NavierStokes        : 2616
PdfJS               : 1302
Mandreel            : 1584
MandreelLatency     : 3566
Gameboy             : 2740
CodeLoad            : 1382
Box2D               : 1466
zlib                : 6206
Typescript          : 3019
----
Score (version 9): 2066
 duration  127.332  seconds

root@pine64:~# vmstat 10
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0      0 1942836   9064  55628    0    0    50   111  143  205  2  2 94  2  0
 1  0      0 1837796   9200  74264    0    0  1882     0  199  170 10  1 89  0  0
 1  0      0 1767224   9204  74220    0    0     6  1082  703  850 26  0 72  1  0
 1  0      0 1749172   9204  74332    0    0     3   730 2127 3725 26  0 74  0  0
 1  0      0 1542220   9580  74636    0    0    65     0  325   76 26  1 73  0  0
 1  0      0 1491580  10120  74816    0    0    69     0  367  103 29  0 71  0  0
 1  0      0 1471200  10120  74832    0    0     0     0  340  105 27  0 73  0  0
 1  0      0 1442612  10120  74852    0    0     0     0  394   88 31  0 69  0  0
 1  0      0 1442612  10120  74852    0    0     0     0  264   26 25  0 75  0  0
 1  0      0 1441744  10120  75036    0    0    13     0  265   28 25  0 75  0  0
 1  0      0 1635948  10120  74960    0    0     0     0  578  175 40  1 59  0  0
 1  0      0 1607272  10120  74964    0    0     0     0  446  133 32  0 67  0  0
 1  0      0 1549968  10120  74976    0    0     0     0  327   60 27  0 72  0  0
 1  0      0 1470608  10120  75016    0    0     0     0  272   32 25  0 75  0  0
^C

armhf:

Richards            : 1876
DeltaBlue           : 3385
Crypto              : 2036
RayTrace            : 3635
EarleyBoyer         : 2529
RegExp              : 329
Splay               : 1657
SplayLatency        : 2764
NavierStokes        : 2155
PdfJS               : 1267
Mandreel            : 1283
MandreelLatency     : 3682
Gameboy             : 2442
CodeLoad            : 1478
Box2D               : 1521
zlib                : 5292
Typescript          : 3151
----
Score (version 9): 2057
 duration  130.932  seconds

root@pine64:~# vmstat 10
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0      0 1962500   1816  43372    0    0    99   137  347  526  6  1 91  2  0
 1  0      0 1907012   2724  69464    0    0  2705     0  185  217  5  0 94  0  0
 2  0      0 1869936   3264  73052    0    0   412     0  315   92 25  1 75  0  0
 1  0      0 1868920   3264  73052    0    0     0     1  316   89 26  0 74  0  0
 1  0      0 1850104   3264  73056    0    0     0     0  296   51 26  0 74  0  0
 1  0      0 1711864   3264  73264    0    0    16     0  320   82 26  1 73  0  0
 1  0      0 1689096   3264  73272    0    0     0     0  375  128 28  0 71  0  0
 2  0      0 1670548   3396  73560    0    0    41     0  397  142 29  0 70  0  0
 1  0      0 1675516   3396  73560    0    0     0     0  292   31 27  0 73  0  0
 1  0      0 1675580   3396  73560    0    0     0     0  265   28 25  0 75  0  0
 3  0      0 1777816   3396  73760    0    0    26     0  426  133 31  1 68  0  0
 1  0      0 1788656   3396  73760    0    0     0     0  468  148 34  0 66  0  0
 1  0      0 1787240   3396  73760    0    0     0     0  356   78 29  0 71  0  0
 1  0      0 1755536   3396  73760    0    0     0     0  282   43 25  0 74  0  0
 0  0      0 1929512   3528  73956    0    0    33     0  211   43 18  0 82  0  0
^C

I think looking at 'combined results' is misleading so let's take a closer look:

                      arm64   armhf
Richards            : 1956    1876
DeltaBlue           : 3332    3385
Crypto              : 2434    2036
RayTrace            : 4201    3635
EarleyBoyer         : 2309    2529
RegExp              : 331     329
Splay               : 706     1657
SplayLatency        : 3162    2764
NavierStokes        : 2616    2155
PdfJS               : 1302    1267
Mandreel            : 1584    1283
MandreelLatency     : 3566    3682
Gameboy             : 2740    2442
CodeLoad            : 1382    1478
Box2D               : 1466    1521
zlib                : 6206    5292
Typescript          : 3019    3151

Some of the tests get a minor performance improvement when running as arm64: Crypto (19%), RayTrace (15%), SplayLatency (14%) NavierStokes (21%), Mandreel (23%), Gameboy (12%) and zlib (17%)

Some others perform neutral and the biggest win when running armhf is with 'Splay' test where 32-bit code shows 2.3 times better numbers (benchmark descriptions).

While it would be still interesting how armhf/arm64 performance numbers and memory requirements differ with different CPU implementations (especially Cavium as 'true server' design) I think I will focus on another attempt first: Using a whole 32-bit userland and compare memory footprint on a 512 MB board (since that's the real use case for me: getting Node-RED run somehow on mini servers where performance doesn't matter that much but memory footprint a lot more and especially swapping/paging would be a disaster)

[ThomasKaiser]

And now running Octane 2.0 some numbers (especially Splay and SplayLatency) look totally different. Only armhf results now:

Richards            : 1875
DeltaBlue           : 3441
Crypto              : 2062
RayTrace            : 3597
EarleyBoyer         : 2488
RegExp              : 329
Splay               : 1291
SplayLatency        : 5727
NavierStokes        : 2152
PdfJS               : 1250
Mandreel            : 1286
MandreelLatency     : 3685
Gameboy             : 2432
CodeLoad            : 1476
Box2D               : 1466
zlib                : 5264
Typescript          : 3169
----
Score (version 9): 2109
 duration  131.155  seconds

Difference: now used npm install benchmark-octane2 -g and benchmark-octane2. This is [email protected] vs. [email protected] before.

[vielmetti]

@ThomasKaiser - Node-RED will run on very memory constrained systems. I'd look at http://nodered.org/docs/hardware/raspberrypi especially where it says

node --max-old-space-size=256 red.js

[ThomasKaiser]

Node-RED will run on very memory constrained systems

Sure, but all Rasperries run a 32-bit userland anyway while those now emerging way more interesting ARM boards with 64-bit CPUs (NanoPi NEO 2, NEO Plus 2 and even the low-end Pine64) come with 64-bit distros and especially this test here shows that memory footprint is totally different when comparing V8 in armhf and arm64 flavour. And that's the reason I still question using arm64 binaries with NodeJS by default (weighing some performance improvements with memory). But so far this is more backed by a feeling and a benchmark (I don't trust in that much) than real facts.

So still hoping for others simply giving it a go and rolling out the armhf binaries on a test or even productive machine to compare with real workloads.

[mmoll]

I still question using arm64 binaries with NodeJS by default

I'm on the exact opposite side.

[chrislea]

Hello everybody here - we've added support for arm64 in our main repository as of today. You can read the announcement here. Enjoy.

[ThomasKaiser]

Hmm... making some noise led to the wrong result ;)

Anyway: if people are happy wasting memory for questionable performance increases I'm happy with. Anyone stumbling accross this and concerned about memory constraints: just follow the 32-bit Armbian for NEO 2 reference above. And also just as reference an example how to deal with NodeJS on ARMv6-ARMv8: https://github.com/ThomasKaiser/install-iot-stuff/commit/b223b2711e4b5665b966ed0deaa300d0fe160b2a

BTW: Funny that only Rasperries are mentioned in your arm64 announcement: RPi 3 might be the only 64-bit capable ARM SoC condemned to run with 32-bit ARMv6 code forever (therefore having exactly no 64-bit memory constraints problems) ;)