Node: V8 CI is not Python 3 compatible

[targos]

Job: https://ci.nodejs.org/job/node-test-commit-v8-linux
It is now permanently red for the master branch.

There are at least two issues:

1. ~The Python version installed on centos7-ppcle hosts doesn't include bzip2 support:
   https://github.com/nodejs/node/issues/25789#issuecomment-546895578
   https://stackoverflow.com/questions/12806122/missing-python-bz2-module
   https://ci.nodejs.org/job/node-test-commit-v8-linux/2592/nodes=centos7-ppcle,v8test=v8test/console~ Fixed

2. gclient doesn't yet support Python 3
   https://github.com/nodejs/node/issues/24512#issuecomment-546897615
   https://ci.nodejs.org/job/node-test-commit-v8-linux/2592/nodes=benchmark-ubuntu1604-intel-64,v8test=v8test/console

[targos]

/cc @nodejs/platform-ppc for 1.

/cc @nodejs/build for 2.

Is Python 3 the default Python on test-nearform_intel-ubuntu1604-x64-1 ?

[rvagg]

Is Python 3 the default Python on test-nearform_intel-ubuntu1604-x64-1 ?

It shouldn't be. I believe python is always Python 2 on Linuxes, you have to opt-in to python3. There's very few places where we're forcing python3.

21:48:44 + ./configure
21:48:46 Node configure: Found Python 3.5.2...

I don't know why it's selecting that, my understanding of configure is that if python2.7 is found that it'll use that:

$ ssh test-nearform_intel-ubuntu1604-x64-1 which python2.7
/usr/bin/python2.7

There's also something wrong with this:

21:48:46 WARNING: C++ compiler (CXX=g++, 5.5.0) too old, need g++ 6.3.0 or clang++ 8.0.0

select-compiler.sh was supposed to fix that last week but it doesn't seem to be working: https://github.com/nodejs/build/blob/74a34e2d9ee5dc4fb945e7c9a5eff388e78227a6/jenkins/scripts/select-compiler.sh#L136-L141

I think it's to do with the switch to case and the match needs to allow for preceding characters. Probably *ubuntu1604-*64*. The $nodes in this instance should be benchmark-ubuntu1604-intel-64 I believe but ubuntu1604-64 on node-test-commit-linux. Can someone get that sanity check that and get a PR up? I'm off for the day.

[rvagg]

@nodejs/python FYI ^ "gclient doesn't yet support Python 3", yay

_(EDIT: Just an FYI, this isn't a critical dependency, we just need to be able to run it in our infra for v8 builds)_

[sam-github]

FYI @targos I'm working on rebuilding python-3.7 with bzip2 support on the *-osuosl-centos7-ppc64-le-* hosts.

[richardlau]

@AshCripps has re-ansibled the test-osuosl-centos7-ppc64-le-* hosts and the bzip2 issue looks to be resolved. https://ci.nodejs.org/job/node-test-commit-v8-linux/nodes=centos7-ppcle,v8test=v8test/2622/console is now failing with a different error:

09:49:49 Failed to fetch file gs://chromium-clang-format/942fc8b1789144b8071d3fc03ff0fcbe1cf81ac8 for v8/buildtools/linux64/clang-format. [Err: Traceback (most recent call last):
09:49:49   File "/home/iojs/build/workspace/node-test-commit-v8-linux/nodes/centos7-ppcle/v8test/v8test/deps/v8/_depot_tools/gsutil.py", line 190, in <module>
09:49:49     sys.exit(main())
09:49:49   File "/home/iojs/build/workspace/node-test-commit-v8-linux/nodes/centos7-ppcle/v8test/v8test/deps/v8/_depot_tools/gsutil.py", line 186, in main
09:49:49     clean=args.clean)
09:49:49   File "/home/iojs/build/workspace/node-test-commit-v8-linux/nodes/centos7-ppcle/v8test/v8test/deps/v8/_depot_tools/gsutil.py", line 134, in run_gsutil
09:49:49     gsutil_bin = ensure_gsutil(force_version, target, clean)
09:49:49   File "/home/iojs/build/workspace/node-test-commit-v8-linux/nodes/centos7-ppcle/v8test/v8test/deps/v8/_depot_tools/gsutil.py", line 113, in ensure_gsutil
09:49:49     target_zip_filename = download_gsutil(version, instance_dir)
09:49:49   File "/home/iojs/build/workspace/node-test-commit-v8-linux/nodes/centos7-ppcle/v8test/v8test/deps/v8/_depot_tools/gsutil.py", line 70, in download_gsutil
09:49:49     u = urllib.urlopen(url)
09:49:49   File "/usr/local/lib/python3.7/urllib/request.py", line 222, in urlopen
09:49:49     return opener.open(url, data, timeout)
09:49:49   File "/usr/local/lib/python3.7/urllib/request.py", line 525, in open
09:49:49     response = self._open(req, data)
09:49:49   File "/usr/local/lib/python3.7/urllib/request.py", line 548, in _open
09:49:49     'unknown_open', req)
09:49:49   File "/usr/local/lib/python3.7/urllib/request.py", line 503, in _call_chain
09:49:49     result = func(*args)
09:49:49   File "/usr/local/lib/python3.7/urllib/request.py", line 1387, in unknown_open
09:49:49     raise URLError('unknown url type: %s' % type)
09:49:49 urllib.error.URLError: <urlopen error unknown url type: https>
09:49:49 ]

[cclauss]

$ python3
>>> import urllib
>>> urllib.urlopen('https://nodejs.org/en/').read() # does that print HTML?

>>> import ssl # does that work?

[richardlau]

On test-osuosl-centos7-ppc64_le-1:

$ `python3`
>>> `import urllib`
>>> `urllib.urlopen('https://nodejs.org/en/').read()` # does that print HTML?

>>> import urllib
>>> urllib.urlopen('https://nodejs.org/en/').read()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'urllib' has no attribute 'urlopen'
>>>

>>> `import ssl` # does that work?

>>> import ssl
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.7/ssl.py", line 98, in <module>
    import _ssl             # if we can't import it, let the error propagate
ModuleNotFoundError: No module named '_ssl'
>>>

[richardlau]

ModuleNotFoundError: No module named '_ssl'

So given we're building Python 3.7 from source on centos7-ppc64 this is most likely missing openssl-devel (in a similar fashion to missing bzip2-devel) based on https://stackoverflow.com/questions/5128845/importerror-no-module-named-ssl?

cc @AshCripps @sam-github

[sam-github]

Seems reasonable, @AshCripps can you try adding openssl-devel to the deps, right after bzip2-devel?

[AshCripps]

Ive added openssl-devel to the playbooks https://github.com/nodejs/build/pull/2044 and its now installing on the machine, I just need someone to disable the machines on jenkins so I can rebuild python on them

[richardlau]

@AshCripps has reansibled the centos7-ppcle test hosts. Rerunning https://ci.nodejs.org/job/node-test-commit-v8-linux/2625/ and centos7-ppcle now fails in a similar fashion to benchmark-ubuntu1604-intel-64 (i.e. 2. from the OP). So I think we're good now for centos7-ppcle insofar as the version of Python 3 is concerned.

[sam-github]

closing - @targos feel free to reopen if we missed something, but it looks like this fixes the host setup part of the problem.

[targos]

Thanks for the host setup!
I'd like to keep this open as long as there are issues with that job, though.

[sam-github]

@targos sorry, I closed the wrong issue, yes, this should stay open until 2. is fixed.

[targos]

Issue to follow for the migration of Chromium to Python 3: https://bugs.chromium.org/p/chromium/issues/detail?id=942720

I don't know if there is a more specific one for depot_tools.
Edit: found https://bugs.chromium.org/p/chromium/issues/detail?id=984182

[targos]

@nodejs/build would it be possible to keep benchmark-ubuntu1604-intel-64 and centos7-ppcle in the job but allow the to fail (have a green or yellow node-test-commit-v8-linux if they fail)?

[rvagg]

not something I can think of how to do without trying to throw far too much bash at it, @richardlau any bright ideas?

[sam-github]

Is the problem that they have py3? Should we remove it until v8 is py3 ready?

[richardlau]

not something I can think of how to do without trying to throw far too much bash at it, @richardlau any bright ideas?

We could try the Text Finder plugin to look for the specific error message in the console and mark the build unstable if found but its documentation suggests it's for downgrading build status (whereas failure->unstable would be an upgrading of status).

https://issues.jenkins-ci.org/browse/JENKINS-26188 is requesting something equivalent to Travis CI's "allow_failures" but hasn't been implemented.

[cclauss]

__cmd || true__ is the equivalent of Travis’s __allow_failures__. You get the output of __cmd__ but the test continues / passes.