Conda-build: conda build with Visual C++ Compiler for Python 2.7+

[0zeroth]

With a default install of Visual C++ Compiler for Python 2.7+ and the only requirements in meta.yaml {{ compiler('c') }}, I see conda build failing with "Did not find VS in registry or in VS90COMNTOOLS env var - exiting".

Trying to debug conda-build, I'm looking at conda_build\windows.py - in particular the msvc_env_cmd function. Tracing the code, it appears it should fall through to the case:

build_vcvarsall_cmd(distutils_find_vcvarsall(9))

I can confirm that my base conda environment (which uses python 3.6) can install setuptools; import distutils and find_vcvarsall(9) successfully.

However, when I inspect the file _build_env\etc\conda\activate.d\vs2008_compiler_vars.bat it appears some other logic has been used - in particular, the lines below. I am confused, because looking at the code it appears the function appends lines to msvc_env_lines - and it do the following in sequence:

msvc_env_lines.append('set DISTUTILS_USE_SDK=1')
msvc_env_lines.append('set MSSdk=1')
msvc_env_lines.append('set "VS_VERSION={}"'.format(version))
msvc_env_lines.append('set "VS_MAJOR={}"'.format(version.split('.')[0]))
msvc_env_lines.append('set "VS_YEAR={}"'.format(VS_VERSION_STRING[version][-4:]))
msvc_env_lines.append('set "CMAKE_GENERATOR={}"'.format(VS_VERSION_STRING[version] +
                                                            {'64': ' Win64', '32': ''}[bits]))

However, when I inspect the generated file _build_env\etc\conda\activate.d\vs2008_compiler_vars.bat something is generating logic _in the middle of that sequence_ that appears to loop over the registry and check environment variables - ie it appears this script is trying to duplicate the logic contained in setuptools+distutils:

:: Set env vars that tell distutils to use the compiler that we put on path
SET DISTUTILS_USE_SDK=1
SET MSSdk=1

:: http://stackoverflow.com/a/26874379/1170370
SET platform=
IF /I [%PROCESSOR_ARCHITECTURE%]==[amd64] set platform=true
IF /I [%PROCESSOR_ARCHITEW6432%]==[amd64] set platform=true

if defined platform (
set "VSREGKEY=HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\VisualStudio\9.0"
)  ELSE (
set "VSREGKEY=HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\VisualStudio\9.0"
)
for /f "skip=2 tokens=2,*" %%A in ('reg query "%VSREGKEY%" /v InstallDir') do SET "VSINSTALLDIR=%%B"

if "%VSINSTALLDIR%" == "" (
   :: Look in VS90COMNTOOLS
   set "VSINSTALLDIR=%VS90COMNTOOLS%"
)

if "%VSINSTALLDIR%" == "" (
   ECHO "Did not find VS in registry or in VS90COMNTOOLS env var - exiting"
   exit 1
)


echo "Found VS2008 at"
echo "%VSINSTALLDIR%"

SET "VS_VERSION=9.0"
SET "VS_MAJOR=9"
SET "VS_YEAR=2008"

set "MSYS2_ARG_CONV_EXCL=/AI;/AL;/OUT;/out"
set "MSYS2_ENV_CONV_EXCL=CL"

:: other things added by install_activate.bat at package build time

SET "CMAKE_GENERATOR=Visual Studio 9 2008 Win64" 
CALL "%VSINSTALLDIR%\..\..\VC\vcvarsall.bat" amd64 

However, I really don't understand how the activate scripts hang together, or where vs2008_compiler_vars.bat fits into the picture.

I've tried setting about every environment variable mentioned in these scripts to no avail: are there any other steps I can take to try and debug this problem?

[Slightly off-topic, would it not fix everything if there was a way to specify the absolute path to vcvarsall.bat - then no matter what problems a user was having, they could always override with the absolute path?]

[0zeroth]

Possibly this is related: https://github.com/conda/conda-build/issues/2263#issuecomment-356718517

[mingwandroid]

You haven't actually shown how conda build fails. Please include the output.

[mingwandroid]

However, I really don't understand how the activate scripts hang together, or where vs2008_compiler_vars.bat fits into the picture.

You must remember that the Anaconda Distribution includes a lot more than just Python code. There are also many C and C++ libraries.

However, when I inspect the generated file _build_env\etc\conda\activate.dvs2008_compiler_vars.bat

This file is not generated. bld.bat is generated and that contains the various set commands.

Compiling Python itself requires a C compiler, and so we use ours (on Windows, MS's!) and our activate scripts are run during the build env creation process at build-time. On Unix platforms, during the build of Python, the details of the compiler and compilation flags are copied into a sysconfigdata.py file, but the source of this information is the compiler activate scripts. On Windows distutils behaves quite differently; it is basically hard-coded with very few tweak parameters beyond DISTUTILS_USE_SDK and MSSdk.

[0zeroth]

meta.yaml snippet:

build:
  number: 0
  script:
    - echo "Hello"

requirements:
  build:
    - {{ compiler('c') }}
  host:
    - python
  run:
    - python 

building with conda build --python . fails with

conda build --python 2.7 meta.yaml
<snip>

  environment location: C:\***\conda-bld\tmp_pkg_1532324996873\_h_env


The following NEW packages will be INSTALLED:

    certifi:        2018.4.16-py27_0
    pip:            10.0.1-py27_0
    python:         2.7.15-he216670_0
    setuptools:     39.2.0-py27_0
    vc:             9-h7299396_1
    vs2008_runtime: 9.00.30729.1-hfaea7d5_1
    wheel:          0.31.1-py27_0
    wincertstore:   0.2-py27hf04cefb_0

<snip setting a bunch of environment variables,hen set a bunch of environment variables 
(including `c_compiler=vs2008, cxx_compiler=vs2008, vc=9)>

"c:\ProgramData\Miniconda3\Scripts\activate.bat" ***\conda-bld\tmp_pkg_1532324996873\_h_env"
<snip - setting the path>
set CONDA_MAX_SHLVL=2
SET DISTUTILS_USE_SDK=1
<snip - the rest of the vs2008_compiler_vars.bat script>
ECHO "Did not find VS in registry or in VS90COMNTOOLS env var - exiting"

I guess what has me confused is it doesn't appear that vs2008_compiler_vars.bat would ever work as it checks VS90COMNTOOLS and the registry - neither of which (as far as I can tell) are set by VC++ for Python. I had assumed that somewhere distutils would be used to find the right path, but if vs2008_compiler_vars.bat is not generated, I'm at a loss as to what to try next.

So, I guess I'm wondering:

• does conda-build support Visual Studio C++ for Python 2.7
  * if so, is there something that needs to be set as environment variable or registry entry to hint to conda-build about where to find it
  * if not, what _is_ supported for Python 2.7 (I tried to find what conda-forge uses on their build agents, but haven't got very far). (I did try some other Visual Studio 2008 installs, but they didn't work - with so many moving parts I don't think that tells me much.)

Given the endless pain Visual Studio + Python 2.7 seems give everyone, I'm inclined to just give up, but I feel like there's probably just something simple I'm missing.

[mingwandroid]

We do not use Visual Studio C++ for Python 2.7, we use Visual Studio 2008. If you want to add support for that then a PR would be welcome. That PR would involve changes to: https://github.com/AnacondaRecipes/aggregate/tree/master/vs2008

[0zeroth]

Thanks for clarifying - I will stick to Visual Studio 2008, I only want to use what everyone else is using!

Also, it appears that supporting this would require trying to duplicate the distutils logic in a .bat script, which seems daunting!

[msarahan]

They're roughly equivalent, but unfortunately some of the paths differ, and it's not trivial. Sorry for the trouble. What Appveyor does is to install VS2008 express edition, and then VC for Python 2.7, and finally there's some patches that conda-forge developed to get things to work right. It's kind of a mess.

[greglandrum]

It's interesting to read here that VC++ for Python 2.7 is not supported when the conda-build documentation (https://conda.io/docs/user-guide/tutorials/build-windows.html) claims that it is:

The MS Visual C++ Compiler for Python 2.7 and the Microsoft Windows SDK for Windows 7 and .NET Framework 4 are both reasonably well tested. Conda build is carefully tested to support these configurations, but there are known issues with the CMake build tool and these free VS 2008 and 2010 alternatives. 

The docs should probably be updated.

[kdheepak]

I'm having this exact same issue on Travis-CI with Windows. I'm assuming in order to use Visual Studio C++ for Python 2.7 I would just need to set the VS90COMNTOOLS environment variable?

@msarahan you said:

What Appveyor does is to install VS2008 express edition, and then VC for Python 2.7, and finally there's some patches that conda-forge developed to get things to work right.

On Travis, would I have to do the same thing? Specifically this is the error I'm getting:

%SRC_DIR%>for /F "skip=2 tokens=2,*" %A in ('reg query "HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\VisualStudio\9.0" /v InstallDir') do SET "VSINSTALLDIR=%B" 
%SRC_DIR%>if "" == "" (set "VSINSTALLDIR=" ) 
%SRC_DIR%>if "" == "" (
ECHO "Did not find VS in registry or in VS90COMNTOOLS env var - exiting"  
 exit 1 
) 
"Did not find VS in registry or in VS90COMNTOOLS env var - exiting"
Traceback (most recent call last):
  File "C:\tools\miniconda3\envs\helics-build-environment\Scripts\conda-build-script.py", line 10, in <module>
    sys.exit(main())
  File "C:\tools\miniconda3\envs\helics-build-environment\lib\site-packages\conda_build\cli\main_build.py", line 456, in main
    execute(sys.argv[1:])
  File "C:\tools\miniconda3\envs\helics-build-environment\lib\site-packages\conda_build\cli\main_build.py", line 447, in execute
    verify=args.verify, variants=args.variants)
  File "C:\tools\miniconda3\envs\helics-build-environment\lib\site-packages\conda_build\api.py", line 208, in build
    notest=notest, need_source_download=need_source_download, variants=variants)
  File "C:\tools\miniconda3\envs\helics-build-environment\lib\site-packages\conda_build\build.py", line 2314, in build_tree
    notest=notest,
  File "C:\tools\miniconda3\envs\helics-build-environment\lib\site-packages\conda_build\build.py", line 1444, in build
    windows.build(m, build_file, stats=build_stats, provision_only=provision_only)
  File "C:\tools\miniconda3\envs\helics-build-environment\lib\site-packages\conda_build\windows.py", line 342, in build
    check_call_env(cmd, cwd=m.config.work_dir, stats=stats, rewrite_stdout_env=rewrite_env)
  File "C:\tools\miniconda3\envs\helics-build-environment\lib\site-packages\conda_build\utils.py", line 374, in check_call_env
    return _func_defaulting_env_to_os_environ('call', *popenargs, **kwargs)
  File "C:\tools\miniconda3\envs\helics-build-environment\lib\site-packages\conda_build\utils.py", line 354, in _func_defaulting_env_to_os_environ
    raise subprocess.CalledProcessError(proc.returncode, _args)
subprocess.CalledProcessError: Command '['cmd.exe', '/c', 'conda_build.bat']' returned non-zero exit status 1

I'm able to install Visual Studio C++ for Python 2.7 but not Visual Studio 2008. Any suggestions on how I would go about using Visual Studio C++ for Python 2.7. Everything I've tried when setting VS90COMNTOOLS has failed to work.

[kdheepak]

For what it is worth, I am still getting this error, and I don't really know how to fix it. Is there a way to use a more modern compiler for a Python 2.7 build?