Vcpkg: Visual Studio build of vcpkg omits 11 source files, 152 test cases & 346 assertions

[claremacrae]

Describe the bug

• vcpkgtest.vcxproj does not include all .cpp files in toolsrc\src\vcpkg-test
• not all tests are compiled and run via Visual Studio

Environment

• OS: Windows 10 Pro 2004 (OS Build 19041.329)
• Compiler: see below

    Microsoft Visual Studio Community 2019
    Version 16.6.5
    VisualStudio.16.Release/16.6.5+30320.27
    Microsoft .NET Framework
    Version 4.8.04084

    Installed Version: Community

    Visual C++ 2019   00435-60000-00000-AA319
    Microsoft Visual C++ 2019

To Reproduce

Steps to reproduce the behavior:

A. Run all tests in Visual Studio, from vcpkg.sln

1. git checkout 0896cfcc301843b455bf14497a15b0e775735198
2. Open toolsrc\vcpkg.sln in Visual Studio
3. Do a Debug build
4. Run vcpkg-test

The output ends with:

===============================================================================
All tests passed (2616 assertions in 103 test cases)

B. Run all tests from a CMake-created build

1. git checkout 0896cfcc301843b455bf14497a15b0e775735198
2. For example, open the folder in CLion
3. Do a Debug build
4. Run vcpkg-test, from the console, or without using CLion's Catch2 adapter, so you see the total number of tests:

The output ends with:

===============================================================================
All tests passed (2962 assertions in 155 test cases)

Expected behavior
The same number of tests run, regardless of build method.

Additional context

Perhaps someone has forgotten to add new test source files to vcpkgtest.vcxproj?

When built from CMake, these test source files are loaded:

toolsrc\src\vcpkg-test\arguments.cpp
toolsrc\src\vcpkg-test\binarycaching.cpp
toolsrc\src\vcpkg-test\binaryconfigparser.cpp
toolsrc\src\vcpkg-test\catch.cpp
toolsrc\src\vcpkg-test\chrono.cpp
toolsrc\src\vcpkg-test\commands.build.cpp
toolsrc\src\vcpkg-test\commands.create.cpp
toolsrc\src\vcpkg-test\dependencies.cpp
toolsrc\src\vcpkg-test\files.cpp
toolsrc\src\vcpkg-test\hash.cpp
toolsrc\src\vcpkg-test\json.cpp
toolsrc\src\vcpkg-test\manifests.cpp
toolsrc\src\vcpkg-test\mockcmakevarsprovider.cpp
toolsrc\src\vcpkg-test\optional.cpp
toolsrc\src\vcpkg-test\paragraph.cpp
toolsrc\src\vcpkg-test\plan.cpp
toolsrc\src\vcpkg-test\specifier.cpp
toolsrc\src\vcpkg-test\statusparagraphs.cpp
toolsrc\src\vcpkg-test\strings.cpp
toolsrc\src\vcpkg-test\stringview.cpp
toolsrc\src\vcpkg-test\system.cpp
toolsrc\src\vcpkg-test\uint128.cpp
toolsrc\src\vcpkg-test\update.cpp
toolsrc\src\vcpkg-test\util.cpp

In Visual Studio, these test source files are loaded:

toolsrc\src\vcpkg-test\arguments.cpp
toolsrc\src\vcpkg-test\catch.cpp
toolsrc\src\vcpkg-test\chrono.cpp
toolsrc\src\vcpkg-test\dependencies.cpp
toolsrc\src\vcpkg-test\files.cpp
toolsrc\src\vcpkg-test\mockcmakevarsprovider.cpp
toolsrc\src\vcpkg-test\paragraph.cpp
toolsrc\src\vcpkg-test\plan.cpp
toolsrc\src\vcpkg-test\specifier.cpp
toolsrc\src\vcpkg-test\statusparagraphs.cpp
toolsrc\src\vcpkg-test\strings.cpp
toolsrc\src\vcpkg-test\update.cpp
toolsrc\src\vcpkg-test\util.cpp

[claremacrae]

/CC @strega-nil for info...

[strega-nil]

This is the issue with having two build systems... I'll fix it

[claremacrae]

Hi @strega-nil

This is the issue with having two build systems... I'll fix it

It’s maybe more to do with one of those systems being manually maintained, and people having to remember to do that.

Since the changing text in the .vcxproj - and maybe filter files too - only contains file-names, here is a CMake idea for a permanent [*] fix, to prevent anyone ever needing to manually maintain lists of files in these files in future:

1. Write code to teach the CMakeLists.txt files to generate the sections of VS configuration files that list files
2. Enable CMake to generate the content of the VS files with updated file list spliced in, and all the other text in the files retained, unchanged.
3. Make CMake actually only overwrite the existing VS files if their content would be different (to prevent the annoyance of VS keeping on saying the project/solution has changed, every time CMake runs.

This way, after a source file is added, removed or renamed, as soon as CMake is rerun, the VS config will be updated, and show up as a diff in git. Usually I suspect those files would then be committed as part of the changeset that did the adding/moving/renaming.

By only auto-generating the file lists, and not all the compiler config, etc, this idea allows for all other VS configuration changes to be made in the IDE, as currently.

[*] Permanent until the VS team changes the formatting of the vcxproj/filter files (speaking from personal experience of about 15 years of maintaining scripts that do just this sort of thing to VS files!)