Node: ./configure fails after Python version default changed

[codebytere]

• Version: master
• Platform: Darwin
• Subsystem: build

https://github.com/nodejs/node/commit/b2ccbb27dc98dfd771d7d5592e139e2001baa56c seems to have caused ./configure to fail locally.

Before:

node on git:master ❯ ./configure                                                      8:50PM
Node configure: Found Python 2.7.10...
INFO: Using floating patch "tools/icu/patches/64/source/common/putil.cpp" from "tools/icu"
INFO: Using floating patch "tools/icu/patches/64/source/i18n/dtptngen.cpp" from "tools/icu"
INFO: configure completed successfully

After:

node on git:master❯ ./configure                                                       8:50PM
Node configure: Found Python 3.7.4...
INFO: Using floating patch "tools/icu/patches/64/source/common/putil.cpp" from "tools/icu"
INFO: Using floating patch "tools/icu/patches/64/source/i18n/dtptngen.cpp" from "tools/icu"
No receipt for 'com.apple.pkg.DeveloperToolsCLILeo' found at '/'.
No receipt for 'com.apple.pkg.DeveloperToolsCLI' found at '/'.
gyp: No Xcode or CLT version detected!
Error running GYP

No changes were made to XCode, and i have command line tools on my machine already. Reverting this commit locally resolves the issue, but i'm not sure why it would have caused it to begin with?

cc @cclauss perhaps?

[richardlau]

@codebytere Could you try applying the changes from https://github.com/nodejs/node-gyp/pull/1890 to tools/gyp/pylib/gyp/xcode_emulation.py?

[cclauss]

You should still be able to control which Python make uses with PYTHON=python2

I am interested in the outcome of Richard’s 1890 experiment above.

Also, nodejs/node-gyp#1939 (merged) that really focuses on issues related to No Xcode or CLT version detected!.

If you have recently upgraded to macOS Catalina then please provide a review of nodejs/node-gyp#1940.

[ZYSzys]

@codebytere Could you try applying the changes from nodejs/node-gyp#1890 to tools/gyp/pylib/gyp/xcode_emulation.py?

It was still fail after applying https://github.com/nodejs/node-gyp/pull/1890

index 9a58df4782..610f90de21 100644
--- a/tools/gyp/pylib/gyp/xcode_emulation.py
+++ b/tools/gyp/pylib/gyp/xcode_emulation.py
@@ -20,6 +20,8 @@ import sys
 import tempfile
 from gyp.common import GypError

+PY3 = bytes != str
+
 # Populated lazily by XcodeVersion, for efficiency, and to fix an issue when
 # "xcodebuild" is called too quickly (it has been found to return incorrect
 # version number).
@@ -1407,7 +1409,7 @@ def XcodeVersion():
   except:
     version = CLTVersion()
     if version:
-      version = re.match(r'(\d+\.\d+\.?\d*)', version).groups()[0]
+      version = ".".join(version.split(".")[:3])
     else:
       raise GypError("No Xcode or CLT version detected!")
     # The CLT has no build information, so we return an empty string.
@@ -1453,6 +1455,8 @@ def GetStdoutQuiet(cmdlist):
   job = subprocess.Popen(cmdlist, stdout=subprocess.PIPE,
                          stderr=subprocess.PIPE)
   out = job.communicate()[0]
+  if PY3:
+    out = out.decode("utf-8")
   if job.returncode != 0:
     raise GypError('Error %d running %s' % (job.returncode, cmdlist[0]))
   return out.rstrip('\n')
@@ -1463,6 +1467,8 @@ def GetStdout(cmdlist):
   Raises |GypError| if the command return with a non-zero return code."""
   job = subprocess.Popen(cmdlist, stdout=subprocess.PIPE)
   out = job.communicate()[0]
+  if PY3:
+    out = out.decode("utf-8")
   if job.returncode != 0:
     sys.stderr.write(out + '\n')

The error shows:

 ~/Projects/node ./configure                                  [13:50:00]
Node configure: Found Python 3.7.4...
INFO: Using floating patch "tools/icu/patches/64/source/common/putil.cpp" from "tools/icu"
INFO: Using floating patch "tools/icu/patches/64/source/i18n/dtptngen.cpp" from "tools/icu"
Traceback (most recent call last):
  File "./configure", line 25, in <module>
    import configure
  File "/Users/zhangyongsheng/Projects/node/configure.py", line 1755, in <module>
    run_gyp(gyp_args)
  File "tools/gyp_node.py", line 54, in run_gyp
    rc = gyp.main(args)
  File "tools/gyp/pylib/gyp/__init__.py", line 547, in main
    return gyp_main(args)
  File "tools/gyp/pylib/gyp/__init__.py", line 532, in gyp_main
    generator.GenerateOutput(flat_list, targets, data, params)
  File "tools/gyp/pylib/gyp/generator/make.py", line 2181, in GenerateOutput
    part_of_all=qualified_target in needed_targets)
  File "tools/gyp/pylib/gyp/generator/make.py", line 773, in Write
    extra_mac_bundle_resources, part_of_all)
  File "tools/gyp/pylib/gyp/generator/make.py", line 879, in WriteActions
    env = self.GetSortedXcodeEnv()
  File "tools/gyp/pylib/gyp/generator/make.py", line 1864, in GetSortedXcodeEnv
    additional_settings)
  File "tools/gyp/pylib/gyp/xcode_emulation.py", line 1772, in GetSortedXcodeEnv
    additional_settings)
  File "tools/gyp/pylib/gyp/xcode_emulation.py", line 1683, in _GetXcodeEnv
    if XcodeVersion() >= '0500' and not env.get('SDKROOT'):
TypeError: '>=' not supported between instances of 'tuple' and 'str'

[richardlau]

@ZYSzys Please try https://github.com/nodejs/node-gyp/pull/1939 as well then.

[cclauss]

The code before nodejs/node-gyp#1932 might have been eating exceptions that are now appearing. This is safer in the long term but might cause hiccups in the near term.

[Trott]

You should still be able to control which Python gets used with PYTHON=python2

For some reason, that doesn't seem to work for me. Or at least it still reports finding Python 3.7.1, suggesting it's still using it (and getting me the same results).

$ PYTHON=python2 ./configure
Node configure: Found Python 3.7.1...
INFO: Using floating patch "tools/icu/patches/64/source/common/putil.cpp" from "tools/icu"
INFO: Using floating patch "tools/icu/patches/64/source/i18n/dtptngen.cpp" from "tools/icu"
No receipt for 'com.apple.pkg.DeveloperToolsCLILeo' found at '/'.
No receipt for 'com.apple.pkg.DeveloperToolsCLI' found at '/'.
gyp: No Xcode or CLT version detected!
Error running GYP
$ 

To anticipate the obvious question:

$ python2 --version
Python 2.7.15
$ 

[Trott]

After applying https://github.com/nodejs/node-gyp/pull/1890/commits/19398a4b0d88581e819b697445771a6e69759cd5.patch, I get similar results to those reported by @ZYSzys.

Applying https://github.com/nodejs/node-gyp/pull/1939.patch on top of that does not apply cleanly, but the patch-merge-conflict to sort out is pretty straightforward I think, so I did that. Now ./configure works, but make fails:

  File "./gyp-mac-tool", line 673
    max_value_length = len(max(filelist.items(), key=lambda (k,v):len(v))[1])
                                                            ^
SyntaxError: invalid syntax

[cclauss]

I was inaccurate when I wrote:

• “You should still be able to control which Python ~gets used~ with PYTHON=python2”

I should have instead written:

• “You should still be able to control which Python make uses with PYTHON=python2“

nodejs/node-gyp#1939 Is a rewrite of this logic and would be my preferred solution.

Fixed in #30146, `lambda (k,v):len(v) is indeed a Python 3 syntax error which flake8 can spot automatically. #30143 reconfigures flake8 to find syntax errors even in our dependencies.

[bnoordhuis]

FWIW, python2 configure.py or python2.7 configure.py can be used as a workaround.

[sam-github]

FTR, Ben's comment above is the right way to force configure to use a specific python executable, ./configure has never respected a PYTHON env variable.

[cclauss]

https://github.com/nodejs/node/pull/30262#issuecomment-549938564 is correct. We have cleaned up the nodejs/node-gyp's __gyp/pylib/gyp/xcode_emulation.py__ file especially in nodejs/node-gyp#1890 nodejs/node-gyp#1895 nodejs/node-gyp#1932 and nodejs/node-gyp#1939 but we have not brought those changes back to this repo's __tools/gyp/pylib/gyp/xcode_emulation.py__ file which is required to fix this issue. PR on its way...