Vega-strike-engine-source: Figure out and fix Windows compilation

[stephengtuggy]

So guys. Last year, I literally spent 2 months trying to build Vega Strike on Windows 10! One of the most frustrating experiences of my developer life.

One thing I CAN tell you, is that Visual Studio 2019 is a no-go as a development environment. The required third-party dependencies just hadn't been updated to support VS2019. VS2017 is probably the latest version that will work.

We will also (probably) want to try clang as an alternate compiler on Windows. Perhaps llvm as well?

And finally, TBH, CMake may just not be up to the task any more. It is a venerable tool, and has served us well for years. But now, it might be time to look at other options? What do you guys think? I have a couple of suggestions for things we could try, if you guys want to hear them.

[stephengtuggy]

Relates to #46

[stephengtuggy]

Relates to #52

[BenjamenMeyer]

I don't think there's a good cross-platform replacement for CMake. AutoTools are a no-go on Windows. QMake is nice but Qt is looking to drop it in favor of CMake; even their qbs is being dropped in favor of CMake.

Starting with VS2017 sounds good as far as Windows goes.

As to changing compilers - I'm more in favor of supporting multiple compilers - so supporting GCC, Clang, LLVM, and VC at the same time. It tends to help C++ code as it clears out bugs through the different behaviors. Right now, we just need to focus on one at a time, which looks like GCC. Clang will be a follow-on; LLVM can be proposed but won't likely happen until after CLang. VC will come with the Windows support; but first we have to have a CI system that will support Windows so that we can keep from breaking it as we move forward.

[royfalk]

From Wikipedia: The Clang project provides a language front-end and tooling infrastructure for languages in the C language family (C, C++, Objective C/C++, OpenCL, CUDA, and RenderScript) for the LLVMhttps://www.llvm.org/ project.
So that's one thing less to worry about.

---

From: Benjamen Meyer [e-mail]
Sent: Tuesday, April 7, 2020 7:41:17 PM
To: vegastrike/Vega-Strike-Engine-Source [e-mail]
Cc: Subscribed [e-mail]
Subject: Re: [vegastrike/Vega-Strike-Engine-Source] Figure out Windows build scheme, possibly from scratch :-P (#53)

I don't think there's a good cross-platform replacement for CMake. AutoTools are a no-go on Windows. QMake is nice but Qt is looking to drop it in favor of CMake; even their qbs is being dropped in favor of CMake.

Starting with VS2017 sounds good as far as Windows goes.

As to changing compilers - I'm more in favor of supporting multiple compilers - so supporting GCC, Clang, LLVM, and VC at the same time. It tends to help C++ code as it clears out bugs through the different behaviors. Right now, we just need to focus on one at a time, which looks like GCC. Clang will be a follow-on; LLVM can be proposed but won't likely happen until after CLang. VC will come with the Windows support; but first we have to have a CI system that will support Windows so that we can keep from breaking it as we move forward.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHubhttps://github.com/vegastrike/Vega-Strike-Engine-Source/issues/53#issuecomment-610495258, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ACQWBEASJ2CGDUCL2K3PH7DRLNJS3ANCNFSM4MDIAYWA.

[stephengtuggy]

I don't think there's a good cross-platform replacement for CMake. AutoTools are a no-go on Windows. QMake is nice but Qt is looking to drop it in favor of CMake; even their qbs is being dropped in favor of CMake.

Fair enough. Mozilla mach might be an option though. Maybe. I still need to find out a little more about it.

On a different topic: the process of collecting third-party dependencies is different on every system. It is a headache and a half. CMake will build them, once you have them. But how do you find the right libraries/downloads in the first place? Esp. on Windows.

How about using some sort of cross-platform package manager? E.g. fpm :-P

Starting with VS2017 sounds good as far as Windows goes.

:+1:

As to changing compilers - I'm more in favor of supporting multiple compilers - so supporting GCC, Clang, LLVM, and VC at the same time. It tends to help C++ code as it clears out bugs through the different behaviors. Right now, we just need to focus on one at a time, which looks like GCC. Clang will be a follow-on; LLVM can be proposed but won't likely happen until after CLang. VC will come with the Windows support; but first we have to have a CI system that will support Windows so that we can keep from breaking it as we move forward.

Sure, sounds good.

And CircleCI does support Windows now, last I checked. If that helps.

[stephengtuggy]

From Wikipedia: The Clang project provides a language front-end and tooling infrastructure for languages in the C language family (C, C++, Objective C/C++, OpenCL, CUDA, and RenderScript) for the LLVMhttps://www.llvm.org/ project. So that's one thing less to worry about.

OK. LLVM is not actually a compiler, it seems. Crap. I remembered that wrong then. :-P

[BenjamenMeyer]

CMake can retrieve things from FTP/HTTP too. So we if we didn't vendor, we could have a repo/artifact storage that has copies. If it's not available in the system then CMake could download it.

Alternative is using some kind of cross-platform package manager, but then we may have to vendor/package that ourselves too on all the platforms at least for the CI system.

Certainly open to ideas; best bet may be to vendor right now though.

[nabaco]

What about cmake -G "Visual Studio 2017". I've seen it used in other projects, might it be relevant to us?

[stephengtuggy]

CMake can retrieve things from FTP/HTTP too. So we if we didn't vendor, we could have a repo/artifact storage that has copies. If it's not available in the system then CMake could download it.

Hmm. Didn't know that. That's good to know.

Alternative is using some kind of cross-platform package manager, but then we may have to vendor/package that ourselves too on all the platforms at least for the CI system.

Certainly open to ideas; best bet may be to vendor right now though.

From a DevSecOps perspective, I am always concerned about vendoring, because then your copy of the vendor's software doesn't stay up to date. At least, not unless you REALLY stay on top of it. All too often, these out-of-date libraries end up with known security holes, which can cause some huge problems for your machine as a whole.

[nabaco]

I've been trying to tackle this task lately. Running cmake on windows works like a charm for me, the main problem is finding the libraries.

I have found the following library so far:

• freeglut
• openal
• libpng
• SDL
• vorbis
• zlib

And am currently stuck on Math lib 😞

Any help will be appreciated.

[BenjamenMeyer]

FYi - there's a few parallel repositories that probably have the stuff you need for building/running on Windows. We may need to do updates, but they're there. See the Wiki and the repo list.

[nabaco]

@BenjamenMeyer I have looked at the repos, and it seems like they have the compiled DLL/EXE files. I don't that's enough, as I need the header files too for compilation.

The current process I'm using is (in PowerShell):
mkdir build && cd build cmake ../engine cmake --build .
And then I to fix the failures.

Currently I commented out the MATH lib in CMakeLists.txt, and I'm failing on a UTIL lib.
Once I'll be able to compile at least once, I'll make a PR for the adapted CMakeLists.txt.
The current changes I'm thinking about are:

1. Adding HINTS for all the places a package needs to be found. This allows setting the library path in an environment variable. Based on this.
2. Adding a download command for the libraries, based on this and this. This will make it much easier to integrate into the CI, and will make it much easier for new developers to bring up a dev environment on windows.

[BenjamenMeyer]

@nabaco okay; thanks for taking a look. If they don't help with this, then we might just want to archive those repos as we get things taken over. I'd honestly like to see us reduce the number of repositories any way as I think most of the repositories are things that should be in the main repo. (The Asset repositories are obviously ones that need to be separated out.)

[stephengtuggy]

@nabaco Wish I could remember where I found the Math and Util libraries. I found them somewhere. But that was on a different development machine. :-(

So now you are experiencing the pain I felt, as I tried to get past the CMake stuff to actually building the VS project on Windows. It was... incredibly frustrating.

I ended up refactoring the CMakeLists.txt quite extensively. I never finished getting it to build, so I never submitted a PR. But if you like, you can take a look at what I did have, here: https://github.com/stephengtuggy/Vega-Strike-Engine-Source/tree/refac/cmake-on-windows

[nabaco]

@pheonixstorm any chance you can help with this?

[nabaco]

Anyone willing to help, as @www2000 has suggested, you can use Windows VMs on linux:
https://developer.microsoft.com/en-us/microsoft-edge/tools/vms/

[nabaco]

So far I had some progress.
I'm using VCPKG to deal with the dependencies, and it handles most issues - makes life much easier.
I merge into my testing branch #103, and replaced the FindGTK3.cmake with this one, as it was tested with VCPKG.

According to this thread in the forums, math is unnecessary, so for now I removed the check for it from CMakeLists.txt

For some reason VCPKG libpng didn't work out of box, so I have to fix it from find_package(PNG) to find_package(libpng), and it seems to pass for now, even though it writes that it didn't find it.

Currently it fails with:

CMake Error: The following variables are used in this project, but they are set to NOTFOUND.

Please set them or make sure they are set and tested correctly in the CMake files:

UTIL_LIB (ADVANCED)

    linked by target "vegastrike" in directory C:/Users/travis/build/nabaco/Vega-Strike-Engine-Source/engine

[nabaco]

I have passed configuration stage, after moving UTIL into a guard:

+IF(UTIL_LIB_FOUND)
+       SET(TST_LIBS ${TST_LIBS} ${UTIL_LIB})
+ELSE(UTIL_LIB_FOUND)
+       message("!! Util Lib not found")
+ENDIF(UTIL_LIB_FOUND)

-SET(TST_LIBS ${TST_LIBS} ${UTIL_LIB})

[nabaco]

Build fails with the following two repetetive errors:

c:\users\travis\build\nabaco\vega-strike-engine-source\engine\src\gnuhash.h(61): error C2953: 'stdext::hash_compare': class template has already been defined [C:\Users\travis\build\nabaco\Vega-Strike-Engine-Source\build\OPcollide.vcxproj]

  c:\users\travis\build\nabaco\vega-strike-engine-source\engine\src\gnuhash.h(56): note: see declaration of 'stdext::hash_compare'

c:\users\travis\build\nabaco\vega-strike-engine-source\engine\src\faction_generic.h(5): fatal error C1083: Cannot open include file: 'boost/shared_ptr.hpp': No such file or directory [C:\Users\travis\build\nabaco\Vega-Strike-Engine-Source\build\OPcollide.vcxproj]

Any help?

[nabaco]

I have commented the problematic function in gnuhash.h, as it should be supplied by STD.

Regarding boost, I have changed the path to it in CMakeLists.txt and it seem to remedy it.

Now I'm solving actual compilation issues.

[stephengtuggy]

@nabaco You may find that the 64-bit Windows environment isn't recognized by our code. Only 32-bit Windows. Let us know if you find that to be the case.

If so, the problem seems to lie in posh.h. I tried fixing it by combining the VS version of posh.h with modifications from other versions of the file that I found in other projects. However, I'm still not sure if I got it right.

FWIW, I believe the file I came up with is here

[nabaco]

@nabaco You may find that the 64-bit Windows environment isn't recognized by our code. Only 32-bit Windows. Let us know if you find that to be the case.

Well, damn. That's very important information, thank you. I'll re-download the dependencies and try to recompile.

If so, the problem seems to lie in posh.h. I tried fixing it by combining the VS version of posh.h with modifications from other versions of the file that I found in other projects. However, I'm still not sure if I got it right.

The is an issue on 32-bit too?
I see these errors: Vega-Strike-Engine-Source\engine\src\posh.h(567,1): fatal error C1189: #error: POSH cannot determine target CPU

There were a bunch of errors regarding #include "config.h" not found. I removed all those includes from the source files. If I understand correctly, it's relevant to non-MSVC compilers, and are included by the -include config.h directive to the compiler. With that directive, the pre-processor inserts the include file manually into each file.

[nabaco]

The following files got compiled so far:

• asteroidgen.exe
• mesh_tool.exe
• replace.exe
• trisort.exe

[nabaco]

Attaching log with errors:
output.log

It seems that most issues are with std::hash.

vegasettings.exe failed on the linking stage, but that might be due to previous errors.

[nabaco]

gnuhash.h removal filed #119 separately.

Switching to win32 removed issues with Posh. I think we should update it to v1.3, and maybe apply the changes from @stephengtuggy.

Now the errors that I need to deal with:

• [x] engine\src\cmd\key_mutable_set.h(41): error C2923: 'std::multiset': 'std::_Tree<_Traits>::_Compare' is not a valid template type argument for parameter '_Pr' (it repeats quite a lot I have to say)
• [x] engine\src\python/python_class.h(240): error C2653: 'Python': is not a class or namespace name - I'll try to compile with Py3 and see if it'll help
• [x] engine\src\python/python_class.h(240): error C3861: 'reseterrors': identifier not found - I think that's due to the previous one
• [ ] MSVC\14.16.27023\include\unordered_map(133): error C2280: 'std::hash<_Kty>::hash(const std::hash<_Kty> &)': attempting to reference a deleted function - I think it might be related to the first error

For more details go here

What bothers me most, why don't we get this issues with GCC/Clang? It makes me think that still something is off with the dependencies/build infrastructure.

[nabaco]

So the python issue was due to config.h not being included by MSVC, and the HAVE_PYTHON not being defined.
I don't know whether I should attempt MSVC to find/include config.h, or define HAVE_PYTHON.

IMHO, I think we should remove HAVE_PYTHON guard all-together since nowadays we don't compile without python anyway...

[BenjamenMeyer]

yes, we're pretty closely tied with the Python libs; not to say we couldn't add something for another language binding but I don't see that happening (possible, but not practical).

That said, MSVC should probably be finding/including config.h - there's probably other stuff that depends on that and it should be generated by CMake, no? So that seems like the more critical and underlying error that if not solved here is going to break elsewhere too.

[stephengtuggy]

What bothers me most, why don't we get this issues with GCC/Clang? It makes me think that still something is off with the dependencies/build infrastructure.

That's what I'm afraid of as well.

I spent several months last year assuming that the problem was in the CMake files. Perhaps they weren't laid out in enough of a cross-platform compatible way. I.e. using -D for defines, instead of using the CMake standard define syntax.

Unfortunately, I came up short. I don't know that I ever really got past the issues that we're stuck on now. Something does seem to be wrong with the dependencies/build infrastructure. But I don't know exactly what.

[nabaco]

@BenjamenMeyer thanks to your tip, I found the needed flag for MSVC to include config.h
/FI

Trying to compile now

[nabaco]

I'm having a real hard time with the hashes, receiving odd errors from different places. Though what's odd, on linux everything compiles ok.

[nabaco]

I think I have found a potential culprit. MSVC doesn't support C++11 anymore. It compiles with C++14 by defult.
https://docs.microsoft.com/en-us/cpp/build/reference/std-specify-language-standard-version?view=vs-2019

I guess I should try compiling with an older compiler.

[nabaco]

Aparantly compiling with STD-C++14 passes both with GCC and Clang, so I'm assuming that that's not the issue.

[stephengtuggy]

@nabaco Based on my experience with Visual Studio 2019, I would indeed recommend that you use Visual Studio 2017 or older, if you can. I thought I mentioned that at some point?

[stephengtuggy]

Sorry to hear you're having so much trouble.

[BenjamenMeyer]

We officially support only VS 2017 right now.

[stamasd]

I have begun some work on trying to get VS compiled on Windows under MSYS2, using the mingw32 compiler system. I have made m running into compilation errors.some progress - managed to get the source configured under cmake, that took a bit of tweaking - but now I'm running into compilation errors.
I have logs that I can attach, and also notes on what I needed to do to get cmake to configure it under mingw32.
Attached is the make output of the current error

make.2.txt

[BenjamenMeyer]

@stamasd my best guess is a missing header; though @nabaco could probably tell you if he's seen that before. It's certainly trying to build under the Win32 configuration as opposed to a Linux or Mac configuration.

Since you're using MingW32/MSYS2, could you please capture the whole run using script and do a clean build? We'll need to see what CMake found, how other parts are compiling, etc too.

Thanks!

[stamasd]

You mean this? OK, but there isn't much there.
typescript.txt

[stamasd]

And including the output of ccmake, then make
typescript2.txt

[stamasd]

Probably best to move discussions regarding my attempts to https://github.com/vegastrike/Vega-Strike-Engine-Source/issues/208

[BenjamenMeyer]

Looking at https://github.com/vegastrike/Vega-Strike-Engine-Source/pull/463/checks?check_run_id=2746482085 I can see part of the issue with compile time... it's building all the dependencies too; which also means we don't get any benefit from 3rd parties having signed their own windows builds :( and probably why we get more flags from virus checkers too.

I wonder if there is a way we can install official distributables from the our dependencies to (a) speed things up and (b) take advantage of them having signed anything themselves....

[stephengtuggy]

@BenjamenMeyer The problem with that, as I understand it, is that the dependencies have to be built with the exact same version of Visual C++ as Vega Strike itself is built with. Otherwise, they won't be binary-compatible.

What we can probably do, is to cache the dependency build artifacts from one Windows CI run, so that they can be used by the next Windows CI run, rather than building from scratch each time. Assuming that the VC++ version hasn't changed.

[BenjamenMeyer]

@stephengtuggy the same version of the VS C/C++ libraries and same multi-threading settings yes; but not necessarily the same version of VC itself. They decoupled that in VC 2012.

[stephengtuggy]

@BenjamenMeyer Fair enough.

[stephengtuggy]

@BenjamenMeyer Take a look at PR #477

[stephengtuggy]

@BenjamenMeyer Can this issue be closed? Vega Strike is now building on Windows; we have added a Windows build to the CI; and now that Windows CI build is caching dependencies. I think we're done here.