Meson: Wrong cpu detection for 'x86_64' windows

[sz-CHEN]

In ‘x64 Native Tools Command Prompt for VS 2017’, the cpu detected by meson is 'x86' ？

**********************************************************************
** Visual Studio 2017 Developer Command Prompt v15.9.16
** Copyright (c) 2017 Microsoft Corporation
**********************************************************************
[vcvarsall.bat] Environment initialized for: 'x64'

D:\opt\glib-2.62.2\_build>meson .. --buildtype=release --prefix="D:\opt\glib\"
The Meson build system
Version: 0.52.0
Source dir: D:\opt\glib-2.62.2
Build dir: D:\opt\glib-2.62.2\_build
Build type: native build
Project name: glib
Project version: 2.62.2
C compiler for the host machine: cl (msvc 19.16.27034)
C linker for the host machine: link 19.16.27034
C++ compiler for the host machine: cl (msvc 19.16.27034)
C++ linker for the host machine: link 19.16.27034
Host machine cpu family: x86
Host machine cpu: x86

And I tried in vs2013 , 2015, got the same result.

[scivision]

I don't see this with VS2019--I didn't try with VS2017.

` The Meson build system Version: 0.52.0 Source dir: C:\meson\test cases\common\1 trivial Build dir: c:\temp\11 Build type: native build Project name: trivial test Project version: undefined C compiler for the host machine: cl (msvc 19.23.28106.4) C linker for the host machine: link 19.23.28106.4 Host machine cpu family: x86_64 Host machine cpu: x86_64

[sz-CHEN]

Is there any way to overwrite host_machine?

[scivision]

The way I know of to control host_machine is by cross compiling. However that shouldn't be necessary here.

Is the correct CPU family detected from Powershell or regular windows command prompt instead of Visual Studio prompt?

Does anyone else reading this see the same issue from VS2017 or older?

[lazka]

glib CI uses VS2017 and it works there, but it's also using meson 0.49.2: https://gitlab.gnome.org/GNOME/glib/-/jobs/489274

[scivision]

hmm, is this related to #6134...

[jpakkane]

If you want to debug this, adding choice print statements in environment.py function detect_cpu and detect_cpu_family is fairly simple.

Also note that this may be related to the Python you are using. If you have a 32 bit Python on a 64 bit machine, it _might_ misdetect the CPU in use. Something to keep in mind.

[scivision]

Looking inside environment.py, and using VS2019 command prompt on 64-bit Windows with Python 3.8 32-bit, this issue could happen if environment variable PROCESSOR_ARCHITEW6432 is not defined

@sz-CHEN what do you get from your Visual Studio command prompt for

echo %PROCESSOR_ARCHITEW6432%

[scivision]

More specifically, here's what one gets on a 64-bit Windows system using Python 64-bit vs. 32-bit.
It does not matter if using Visual Studio command prompt or Powershell or whatever:

# 64-bit Python
>>> os.environ['PROCESSOR_ARCHITECTURE']
'AMD64'

# 32-bit Python
>>> os.environ['PROCESSOR_ARCHITECTURE']
'x86'

[scivision]

@sz-CHEN you can determine if you're using 32-bit or 64-bit Python by:

python -c "import sys; print(sys.maxsize > 2**32)"

• 32-bit Python will print False
• 64-bit Python will print True

reference

[scivision]

The above note may be the key to fix this issue. We are now waiting for feedback from @sz-CHEN to run these checks above.

[sz-CHEN]

@scivision Thanks for your help, the following is what I have tried.

**********************************************************************
** Visual Studio 2017 Developer Command Prompt v15.9.16
** Copyright (c) 2017 Microsoft Corporation
**********************************************************************
[vcvarsall.bat] Environment initialized for: 'x64'

C:\Program Files (x86)\Microsoft Visual Studio\2017\Community>D:

D:\>cd opt\glib-2.62.2

D:\opt\glib-2.62.2>python -c "import sys; print(sys.maxsize > 2**32)"
True

D:\opt\glib-2.62.2>python
Python 3.8.0 (tags/v3.8.0:fa919fd, Oct 14 2019, 19:37:50) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> os.environ['PROCESSOR_ARCHITECTURE']
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'os' is not defined
>>> import os
>>> os.environ['PROCESSOR_ARCHITECTURE']
'AMD64'
>>> quit()

D:\opt\glib-2.62.2>echo %PROCESSOR_ARCHITEW6432%
%PROCESSOR_ARCHITEW6432%

D:\opt\glib-2.62.2>echo %PROCESSOR_ARCHITECTURE%
AMD64

It is 64bit python, but the variable %PROCESSOR_ARCHITEW6432% was not defined but % PROCESSOR_ARCHITECTURE% is defined as 'AMD64'.
And I tried set %PROCESSOR_ARCHITEW6432% as 'AMD64', but got the same result.

Then, I modified the 'environment.py', add print("compiler.target:"+compiler.target) in function detect_windows_arch, and found that: compiler.target:x86, and the compiler is defined from def detect_machine_info(compilers: typing.Optional[CompilersDict] = None) -> MachineInfo: called as machines.build = detect_machine_info()

C compiler for the host machine: cl (msvc 19.16.27034)
C linker for the host machine: link 19.16.27034
C++ compiler for the host machine: cl (msvc 19.16.27034)
C++ linker for the host machine: link 19.16.27034
compiler.target:x86
compiler.target:x86
Host machine cpu family: x86
Host machine cpu: x86

[sz-CHEN]

I tried print the compilers in different function.

OrderedDict([('c', <VisualStudioCCompiler: v19.16.27034 `cl`>), ('cpp', <VisualStudioCPPCompiler: v19.16.27034 `cl`>)])
detect_cpu_family
OrderedDict([('c', <VisualStudioCCompiler: v19.16.27034 `cl`>), ('cpp', <VisualStudioCPPCompiler: v19.16.27034 `cl`>)])
detect_windows_arch
OrderedDict([('c', <VisualStudioCCompiler: v19.16.27034 `cl`>), ('cpp', <VisualStudioCPPCompiler: v19.16.27034 `cl`>)])
arch: amd64
compiler.target:x86
detect_cpu
OrderedDict([('c', <VisualStudioCCompiler: v19.16.27034 `cl`>), ('cpp', <VisualStudioCPPCompiler: v19.16.27034 `cl`>)])
detect_windows_arch
OrderedDict([('c', <VisualStudioCCompiler: v19.16.27034 `cl`>), ('cpp', <VisualStudioCPPCompiler: v19.16.27034 `cl`>)])
arch: amd64
compiler.target:x86

[scivision]

To summarize sz-CHEN's findings:

• using 64-bit Python
• Visual Studio 2017 19.16
• PROCESSOR_ARCHITEW6432 was not defined, but even defining it as AMD64 does not help

I think it may be helpful to trace line by line to see which exact line makes this x86 decision. Does any dev have vs2017 to try or sz-CHEN. You can insert a breakpoint() statement to help

[mczarnek]

Thank you for looking into this scivision, this is indeed the issue I've found too causing issue #6134

I ran this within meson itself:
message('host_machine.cpu_family():'+host_machine.cpu_family())
gstreamer uses this to determine the version of wine to use, when it believes I'm using x86, this causes meson to pick the wine32 version of wine

Otherwise, same issue with python 2.7.17

>>> os.environ['PROCESSOR_ARCHITECTURE']
'AMD64'
>>> os.environ['PROCESSOR_ARCHITEW6432']
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Python27\lib\os.py", line 425, in __getitem__
    return self.data[key.upper()]
KeyError: 'PROCESSOR_ARCHITEW6432'

[scivision]

OK @mczarnek I posted some questions in that issue

For @sz-CHEN, can you also copy and paste what Meson outputs from configuring a trivial meson.build containing only project('foo', 'c')

I suppose an issue may be that you only have Visual Studio compilers--well let's just try it and see what happens from regular (non-visual studio) command prompt.

[lb90]

Hi! I have the same problem compiling GLib with VS 2019 (using x64 Native Tools Command Prompt for VS 2019). I have tried debugging detect_cpu_family in environment.py with pdb.set_trace(). Here's what turned out:

• detect_windows_arch() calls detect_native_windows_arch()
• detect_native_windows_arch() rightly detects "amd64" inspecting the PROCESSOR_ARCHITECTURE environment variable.
• Control flow returns to detect_windows_arch() and runs that checks and immediately x86 is returned

[lb90]

In fact the first if succeds.

I have compiler.id == 'msvc' and compiler.target == 'x86'. But is it right to have compiler.target as 'x86'? Afterall I am not cross compiling. But sure, msvc can compile x86_64 and x86

[scivision]

Yes that's the question I have, I need someone more experienced with Meson and VS to know what the outcome should be--is it that VS has the "wrong" target configuration or that Meson should understand
https://github.com/mesonbuild/meson/issues/6134#issuecomment-550465629

[scivision]

@nirbheek do you have insight or know who might into this issue? It seems with any version of Visual Studio, Meson thinks an x86 target should be built when a 64-bit target is desired.

[lb90]

If I debug on 0.51.2 compiler.target is 'x64'

[lb90]

The problem is this: https://github.com/mesonbuild/meson/blob/4c96aa34cb415ca654c0571280386880e9d14ca3/mesonbuild/environment.py#L993

The RE doesn't match. So match is None and target gets set to 'x86'.

[lb90]

But why doesn't it match?

(Pdb) lookat
'Microsoft (R) C/C++ Optimizing Compiler versione 19.23.28106.4 per x64\nCopyright (C) Microsoft Corporation. Tutti i diritti  sono riservati.\n\n'
(Pdb) lookat.split('\n')[0]
'Microsoft (R) C/C++ Optimizing Compiler versione 19.23.28106.4 per x64'

[lb90]

Ahhh OMG. The problem in the RE is that 'for' (an english word) is not present in other languages :)

[lb90]

Here's the relevant commit: https://github.com/mesonbuild/meson/commit/a107eab2949cf9750ed88a634118b31666c16c7a

[scivision]

Assuming English language environment has been an issue previously, maybe on some issues still open.

It seems time to specifically have a non-English CI environment. Maybe a couple (Latin & non-Latin alphabets)

[scivision]

@sz-CHEN if you switch back to Meson 0.51.2 does that work, but 0.52.0 is broken? are you on a non-English language Windows computer? (like https://github.com/mesonbuild/meson/issues/6128#issuecomment-550490425)

[lb90]

Is there a cl.exe command line option to force output in en_US?

[scivision]

set VSLANG=1033

or change codepage

If that helps, the question would be how to do that from Meson...

[scivision]

I think Meson's output is always in English, so depending on general opinion, if either of those work for this issue, maybe that would help avoid this issue in other places in Meson.

[lb90]

set VSLANG=1033

Great! That works :)

[lb90]

BTW, I think we should throw an exception here:

https://github.com/mesonbuild/meson/blob/4c96aa34cb415ca654c0571280386880e9d14ca3/mesonbuild/environment.py#L997

[scivision]

@jpakkane the solution here for a non-English locale appears to be setting environment variable to force Visual Studio to English output.

Another solution (besides removing the word "for" in the Meson regex code) would be per-language regex.

Since Meson's output is always English (?) maybe detecting this situation (non-English locale, use of MSVC) and setting this environment variable to force English MSVC output from the internal Python code is one possible fix?

[scivision]

yes I think an exception would be a better idea there. We'll surely have less confused users who might otherwise just give up in those cases.

[sz-CHEN]

I'm on a Chinese langauage Windows computer.
After set VSLANG=1033 it works.
Thanks for all of you.

[jfinkhaeuser]

Setting VSLANG=1033 does not seem to change my cl output, either on the Windows cmd or using a bash e.g. from MSYS or something.

I should add that I have this problem on the VS Community edition 19.23.28106.4

I think the regular expression may be too brittle. Why not change it to this?

 match = re.search('.* (x86|x64|ARM|ARM64)$', lookat.split('\n')[0]) 

• You're already in a block that has found microsoft in the output.
• The regex is fairly explicit in looking for line-ending specifiers with a preceding space, which should be display language independent.

[jfinkhaeuser]

Ah, I see master is already ahead.

[lb90]

Hi Jens! Just for information purpose, could you post here the first line of cl.exe /? ?

[jfinkhaeuser]

Sure - took a while to get there (different/work computer and all that), but it's attached. Language is German, no matter the value of VSLANG.

I think the different detection patter in master is doing it right.

[lb90]

Hi @jfinkhaeuser! Were you able to find a workaround in the meantime?

[lb90]

I don't know if that does matters but I have the Visual Studio English Language Pack installed. In Visual Studio Installer, go to the 'Language Pack' tab and check the 'English' checkbox. Maybe this helps

[jfinkhaeuser]

That might be something I'll have to try. I usually configure everything in English, but don't really run VS - I just need the compiler :) I'll have a look.

For the time being, it's not super urgent - we need Windows builds mainly for running tests; the target platforms are mobile. So for now, we've just switched to 32-bit builds. But longer term, it'd be good to see this fixed.

Which really boils down to: is there an ETA for the detection code in master?

[lb90]

@nirbheek using the VSLANG envvar is not a reliable solution. Can we change the regular expression in some way to work for all locales?

[jfinkhaeuser]

The code in master uses a different regex, that should be ok.

[lb90]

Jens are you sure? Because I had this issue using meson from git.

[lb90]

Here's the link: https://github.com/mesonbuild/meson/blob/master/mesonbuild/environment.py#L996

[jfinkhaeuser]

That might've been changed since last I looked. I saw the entire if 'Microsoft' in out or 'Microsoft' in err: block factored out into a function that just looked for a line ending in one of the CPU patterns, without the leading for.

Sorry for the confusion!