Node: AIX build failure

[richardlau]

Looking into this. Opening for investigation/tracking.

Refs: https://github.com/nodejs/node/pull/22237#issuecomment-434048630

https://ci.nodejs.org/job/node-test-commit-aix/18405/nodes=aix61-ppc64/consoleFull

First few errors in build log:

01:12:48 In file included from ../src/env.h:27:0,
01:12:48                  from ../src/connection_wrap.h:6,
01:12:48                  from ../src/connection_wrap.cc:1:
01:12:48 ../src/aliased_buffer.h:27:22: error: expected type-specifier
01:12:48            typename = std::enable_if_t<std::is_scalar<NativeT>::value>>
01:12:48                       ^
01:12:48 ../src/aliased_buffer.h:27:22: error: expected '>'
01:12:48 In file included from ../src/env.h:27:0,
01:12:48                  from ../src/connection_wrap.h:6,
01:12:48                  from ../src/connection_wrap.cc:1:
01:12:48 ../src/aliased_buffer.h:65:31: error: template argument 3 is invalid
01:12:48                  v8::Uint8Array>& backing_buffer)
01:12:48                                ^
01:12:48 In file included from ../src/env.h:27:0,
01:12:48                  from ../src/connection_wrap.h:6,
01:12:48                  from ../src/connection_wrap.cc:1:
01:12:48 ../src/aliased_buffer.h:123:41: error: template argument 3 is invalid
01:12:48      Reference(AliasedBuffer<NativeT, V8T>* aliased_buffer, size_t index)
01:12:48                                          ^
01:12:48 ../src/aliased_buffer.h:163:31: error: template argument 3 is invalid
01:12:48      AliasedBuffer<NativeT, V8T>* aliased_buffer_;
01:12:48 

cc @nodejs/platform-aix

[addaleax]

Looks like this comes from https://github.com/nodejs/node/pull/23808, on which CI was run but not on AIX?

[richardlau]

https://github.com/nodejs/node/pull/23808 was the most recent change to aliased_buffer.h.

[richardlau]

Looks like this comes from #23808, on which CI was run but not on AIX?

@refack @nodejs/build-infra Any ideas why the CI for https://github.com/nodejs/node/pull/23808#issuecomment-432730964 (https://ci.nodejs.org/job/node-test-pull-request/18121/) didn't run on AIX?

[richardlau]

Also from the build log https://ci.nodejs.org/job/node-test-commit-aix/18405/nodes=aix61-ppc64/consoleFull it looks like gcc 4.8.5 is being used, which doesn't look right:

00:44:45 + . ./build/jenkins/scripts/select-compiler.sh
00:44:45 + [  != DONT ]
00:44:45 + SELECT_ARCH=AIXPPC
00:44:45 + [ -z ]
00:44:45 /tmp/jenkins6046268851601114322.sh[29]: test: 0403-004 Specify a parameter with this command.
00:44:45 + [ AIXPPC = PPC64LE ]
00:44:45 + [ AIXPPC = S390X ]
00:44:45 + [ AIXPPC = AIXPPC ]
00:44:45 + echo Setting compiler for Node version  on AIX
00:44:45 Setting compiler for Node version  on AIX
00:44:45 + [  -gt 9 ]
00:44:45 + which gcc
00:44:45 + which g++
00:44:45 + which g++
00:44:45 + export CC=ccache /opt/freeware/bin/ccache/gcc CXX=ccache /opt/freeware/bin/ccache/g++ CXX_host=ccache /opt/freeware/bin/ccache/g++
00:44:45 + echo Compiler set to default at 4.8.5
00:44:45 Compiler set to default at 4.8.5

[richardlau]

I think this error suggests the script is being run with the default korn shell instead of bash?

/tmp/jenkins6046268851601114322.sh[29]: test: 0403-004 Specify a parameter with this command.

@nodejs/build-infra Can someone check this on the AIX CI machines?

[richardlau]

This is what the equivalent select-compiler.sh output on AIX is supposed to look like in a recent working build off master:

https://ci.nodejs.org/job/node-test-commit-aix/18415/nodes=aix61-ppc64/consoleFull

06:23:05 + . ./build/jenkins/scripts/select-compiler.sh
06:23:05 ++ '[' '' '!=' DONT ']'
06:23:05 ++ case $NODE_NAME in
06:23:05 ++ SELECT_ARCH=AIXPPC
06:23:05 ++ '[' -z ']'
06:23:05 +++ python tools/getnodeversion.py
06:23:05 ++ NODE_VERSION=12.0.0
06:23:05 +++ echo 12.0.0
06:23:05 +++ cut -d . -f 1
06:23:05 ++ NODEJS_MAJOR_VERSION=12
06:23:05 ++ '[' AIXPPC = PPC64LE ']'
06:23:05 ++ '[' AIXPPC = S390X ']'
06:23:05 ++ '[' AIXPPC = AIXPPC ']'
06:23:05 ++ echo 'Setting compiler for Node version 12 on AIX'
06:23:05 Setting compiler for Node version 12 on AIX
06:23:05 ++ '[' 12 -gt 9 ']'
06:23:05 ++ export LIBPATH=/home/iojs/gmake/opt/freeware/lib:/home/iojs/gcc-6.3.0-1/opt/freeware/lib/gcc/powerpc-ibm-aix6.1.0.0/6.3.0/pthread/ppc64:/home/iojs/gcc-6.3.0-1/opt/freeware/lib
06:23:05 ++ LIBPATH=/home/iojs/gmake/opt/freeware/lib:/home/iojs/gcc-6.3.0-1/opt/freeware/lib/gcc/powerpc-ibm-aix6.1.0.0/6.3.0/pthread/ppc64:/home/iojs/gcc-6.3.0-1/opt/freeware/lib
06:23:05 ++ export PATH=/home/iojs/gcc-6.3.0-1/opt/freeware/bin:/opt/freeware/bin/ccache:/home/iojs/jdk8/jre/bin:/usr/bin:/etc:/usr/sbin:/usr/ucb:/usr/bin/X11:/sbin:/usr/java6/jre/bin:/usr/java6/bin
06:23:05 ++ PATH=/home/iojs/gcc-6.3.0-1/opt/freeware/bin:/opt/freeware/bin/ccache:/home/iojs/jdk8/jre/bin:/usr/bin:/etc:/usr/sbin:/usr/ucb:/usr/bin/X11:/sbin:/usr/java6/jre/bin:/usr/java6/bin
06:23:05 +++ which gcc
06:23:05 +++ which g++
06:23:05 +++ which g++
06:23:05 ++ export 'CC=ccache /home/iojs/gcc-6.3.0-1/opt/freeware/bin/gcc' 'CXX=ccache /home/iojs/gcc-6.3.0-1/opt/freeware/bin/g++' 'CXX_host=ccache /home/iojs/gcc-6.3.0-1/opt/freeware/bin/g++'
06:23:05 ++ CC='ccache /home/iojs/gcc-6.3.0-1/opt/freeware/bin/gcc'
06:23:05 ++ CXX='ccache /home/iojs/gcc-6.3.0-1/opt/freeware/bin/g++'
06:23:05 ++ CXX_host='ccache /home/iojs/gcc-6.3.0-1/opt/freeware/bin/g++'
06:23:05 ++ echo 'Compiler set to 6.3'
06:23:05 Compiler set to 6.3

We should be using gcc 6.3 on AIX for master (https://github.com/nodejs/node/blob/master/BUILDING.md#aix).

[refack]

I fixed a bug in the build script around 4AM EDT at @joyeecheung's ping on IRC

tl;dr the default shell on AIX is not bash so some of the tests in select-compiler.sh weren't evaluated correctly, and the job ended up compiling with gcc4.8.

I'm closing this as resolved, but if anyone sees one of those fails here is the explanation.

[refack]

@refack Any ideas why the CI for #23808 (comment) (ci.nodejs.org/job/node-test-pull-request/18121) didn't run on AIX?

There were infra issues, so the AIX job was disabled for a few days. I turned it back on last night.

[richardlau]

@refack Any ideas why the CI for #23808 (comment) (ci.nodejs.org/job/node-test-pull-request/18121) didn't run on AIX?

There were infra issues, so the AIX job was disabled for a few days. I turned it back on last night.

Is that logged anywhere? That means that for the few days it was disabled none of the CI's were running on AIX.

[refack]

Is that logged anywhere? That means that for the few days it was disabled none of the CI's were running on AIX.

I'm not sure if I sent out a notification on IRC...
There was a notice that one machines whet offline for manual testing https://github.com/nodejs/build/projects/2

P.S. Anyway as policy whenever we (the Build WG) take a platform offline, we take responsibility to chase down regressions afterwards, if any arise.

[mhdawson]

@refack I don't quite understand, the last update to https://github.com/nodejs/build/commits/master/jenkins/scripts/select-compiler.sh was 27 days ago
based on the history?

[refack]

@mhdawson AFAICT this is what happened:

1. select-compiler.sh has always been silently failing on AIX and picking up gcc4.8. My assumption is that it's because it uses bash features, but the the aix Jenkins job was calling the default sh which is not bash.
2. During the last few days we landed several PRs with code that actually requires gcc4.9.
3. The AIX job was offline during the last few days
4. When I brought it back online (2018-10-28 ~10PM EDT) it started surfacing the issue with the wrong gcc version.
5. At 2018-10-29 ~4AM EDT Joyee pinged the build IRC channel
6. I made the following change; (adding an explicit bash shebang #!/usr/bin/bash -xe)
7. Everything works (that is select-compiler.sh picks gcc6 and the code in master compiles and passed the test suite).

[mhdawson]

Ok thanks, got it. I suspect it was just using the wrong compiler since the last update I made. I had checked the release jobs carefully, but I guess I needed to check all of them (one of the reasons I was cautious about changing the script).

Separately, I think we should open an issue in the build repo any time we remove a platform from one of the jobs so that it gets maximum visibility.