Team-comtress-2: Linux builds

[Ristovski]

Last time I tried to make a Linux build of the leak, I managed to compile the client and server .so libs but ran into some linking issues with protobuf (seemed like some weird ABI compatibility problem).

Valves build system is /horrible/. To get anything working on Linux you first have to patch several Makefiles, normalize some filenames (Valve really likes to #include without regard to case sensitivity), outright fix some warnings/bugs (gotta love -Werror) and even add missing includes - have fun grepping the source tree to find all the paths.

I am currently on vacation but I can probably share my findings/patches when I get back.

Have you made any progress yourself?

_PS: Out of curiosity I built an optimized version of libtogl.so (valves DX->OpenGL translation layer lib) and it made no difference in performance (it did break rendering a bit tho, no idea why). Just in case you got that idea as well._

[maximsmol]

I actually propose we write a Python script to replace VPC (the Makefile/vsproj generation utility). On my branch I had to do extremely extensive changes to get clean debug-release separation (the unpatched sources will fail the build whenever you change build configuration but do not do a full rebuild). I also broke all linux builds since I had no way of testing my changes on that platform (though it should be easy to fix the other platforms' VPCs, all the Makefiles are unpatched).

And if we, for example, want to support more than 2 different build configurations, it will be a similar titanic effort. VPC is just not very flexible. I, for one, want the option to turn FPO, Retail, and CRT on and off at will, without having to rebuild the entire project every single time.

An alternative is to build VPC (sources are in external/vpc/utils/vpc) and patch it. I do not really like this because writing build systems in C++ is going to be torturous.

See #10

[Ristovski]

I've actually thought of seeing how hard it would be to convert the whole build system to something nicer like Meson (which can also generate VS projects btw, for the Windows folks).

I guess the first step would be to see how chaotic/dumb the actual directory structure is (seems to be very messy, with include dirs scattered all over the place).

I might eventually look into this but first we need a reproducible Linux build that works, which is a completely different beast and requires fixing up the actual source code (which I will get to once back from vacation in ~2 weeks).

But yes, using anything other then VPC would be great.

[nosoop]

I'd propose having a Python script that parses the VPC scripts into one or more Ninja build scripts.

The main draw is that it's a bit less invasive of a change compared to redoing the entire build in Meson (which itself is a full build system on top of Ninja), but any method that drops the need to use Visual Studio is good in my opinion.

[maximsmol]

I have an almost working VPC parser on my fork

[Demez]

https://github.com/quiverteam/QuiverProjectCreator Here's a vpc replacement i made a while ago, the scripts are pretty different from the original scripts, but still recognizable. You can run the vpc converter included to convert all vpc scripts to the qpc script format.

[JJL772]

ABI issues can be solved by passing the -fabi-compat-version=2 flag to GCC, which will force GCC to emit symbols compatible with the ABI found in GCC 4.9. here is an example of how to integrate that with this project.

If you want to stay with VPC, you can look at this, which is a mildly improved version of VPC for Linux, although Quiver Project Creator (QPC) is probably a better alternative.

[Thalass]

I'm no coder, but I am a linux user and I hereby volunteer to test things once things get to that stage. I run Mint, but have no problem installing other distros for testing purposes. If that helps haha

[Kartoffelsaft]

I've been trying to get some sort of build working, and I don't really know where to start. Since issue #10 mentions makefiles, I assume that those are located in linux_sdk/.

Upon initially running make it complains about a couple of things:

1. it requires the x86-64 instruction set (weird, considering tf2 is 32-bit)
2. the target doesn't support -fcf-protection=full

the first error seems to have gone away using the compatibility flag mentioned by @JJL772. Changing -march from pentium3 to core2 could sometimes get rid of the error too, but depending on what was in USER_CFLAGS it sometimes wouldn't.

After doing some research on the second issue, I found that more recent versions of Ubuntu, -fcf-protection=full is applied by default unless explicitly turned off. I guess this makes sense, as it seems like a lot of Valve's linux stuff is based on old Ubuntu versions. The -fcf-protection=none can be added to get rid of the error. I would make a pull request to add that flag, but I don't think I should do that without having a fully working build system to go with it.

At this point I'm stuck on a File name too long error. The whole output I'll put at the bottom of this post. I've got a few questions though:

1. Is this the makefile that I should be looking at? I get the impression that it is, but I just want to make sure
2. This makefile has GAME_DIR=~/valve/hl2bin/ in it, but I don't have a ~/valve/ directory. The comments in the file seem to imply that it is read from, and not written to. Do I need some bin files in there? if I do, how would I generate/get them? (I'm guessing it would be a similarly complicated process as building the game itself)
3. Forgive me if this is a stupid question, but what is VPC? I get that it is a build script system, but I just can't find like any other info on it. Is it valve/source engine specific? What makes it so bad that a python/meson build system would be better? What does the acronym even stand for? Basically all of my searches lead to "Virtual Private Cloud", which is clearly not what I'm looking for.

if [ -z "/usr/local/bin/gcc" ]; then echo "Compiler not defined."; exit; fi
if [ ! -d . ];then mkdir .;fi
cd .
make -f Makefile.vcpm CC=/usr/local/bin/gcc CPLUS=/usr/local/bin/g++ CPP_LIB="/usr/local/lib/libstdc++.a /usr/local/lib/libgcc_eh.a" BUILD_DIR=. BUILD_OBJ_DIR=./obj SOURCE_DIR=.. SHLIBLDFLAGS=-shared -Wl,-Map,vcpm_map.txt -Wl SHLIBEXT=so CLINK=/usr/local/bin/gcc CFLAGS=""-fcf-protection=none -fabi-compat-version=2"  -mtune=i686 -march=pentium3 -mmmx -O3 -fpermissive -D_LINUX -DNDEBUG -Dstricmp=strcasecmp -D_stricmp=strcasecmp -D_strnicmp=strncasecmp -Dstrnicmp=strncasecmp -D_snprintf=snprintf -D_vsnprintf=vsnprintf -D_alloca=alloca -Dstrcmpi=strcasecmp -Usprintf=use_Q_snprintf_instead_of_sprintf -Ustrncpy=use_Q_strncpy_instead -UPROTECTED_THINGS_ENABLE" LDFLAGS="-lm -ldl ~/valve/hl2bin//bin/tier0_i486.so ~/valve/hl2bin//bin/vstdlib_i486.so" ARCH=i486 GAME_DIR=~/valve/hl2bin/ MOD_CONFIG=hl_ReleaseSDKWin32 NAME=server XERCES_INC_DIR=/home/alfred/tmp/xerces-c-src_2_6_0/include XERCES_LIB_DIR=/home/alfred/tmp/xerces-c-src_2_6_0/lib
make[1]: Entering directory '/home/kartoffelsaft/Documents/code_projects/team-comtress-2/linux_sdk'
make[1]: abi-compat-version=2  -mtune=i686 -march=pentium3 -mmmx -O3 -fpermissive -D_LINUX -DNDEBUG -Dstricmp=strcasecmp -D_stricmp=strcasecmp -D_strnicmp=strncasecmp -Dstrnicmp=strncasecmp -D_snprintf=snprintf -D_vsnprintf=vsnprintf -D_alloca=alloca -Dstrcmpi=strcasecmp -Usprintf=use_Q_snprintf_instead_of_sprintf -Ustrncpy=use_Q_strncpy_instead -UPROTECTED_THINGS_ENABLE: File name too long
make[1]: stat: abi-compat-version=2  -mtune=i686 -march=pentium3 -mmmx -O3 -fpermissive -D_LINUX -DNDEBUG -Dstricmp=strcasecmp -D_stricmp=strcasecmp -D_strnicmp=strncasecmp -Dstrnicmp=strncasecmp -D_snprintf=snprintf -D_vsnprintf=vsnprintf -D_alloca=alloca -Dstrcmpi=strcasecmp -Usprintf=use_Q_snprintf_instead_of_sprintf -Ustrncpy=use_Q_strncpy_instead -UPROTECTED_THINGS_ENABLE: File name too long
make[1]: *** No rule to make target 'abi-compat-version=2  -mtune=i686 -march=pentium3 -mmmx -O3 -fpermissive -D_LINUX -DNDEBUG -Dstricmp=strcasecmp -D_stricmp=strcasecmp -D_strnicmp=strncasecmp -Dstrnicmp=strncasecmp -D_snprintf=snprintf -D_vsnprintf=vsnprintf -D_alloca=alloca -Dstrcmpi=strcasecmp -Usprintf=use_Q_snprintf_instead_of_sprintf -Ustrncpy=use_Q_strncpy_instead -UPROTECTED_THINGS_ENABLE'.  Stop.
make[1]: Leaving directory '/home/kartoffelsaft/Documents/code_projects/team-comtress-2/linux_sdk'
make: *** [Makefile:89: vcpm] Error 2

[micwoj92]

@Kartoffelsaft https://developer.valvesoftware.com/wiki/Valve_Project_Creator

[Ristovski]

I managed to get both the client and server built:

ls -lh ./obj_client_tf_linux32/release/client.so
-rwxr-xr-x 1 rafael rafael 439M Sep 15 22:00 ./obj_client_tf_linux32/release/client.so
ls -lh ./obj_server_tf_linux32/release/server.so
-rwxr-xr-x 1 rafael rafael 382M Sep 15 23:01 ./obj_server_tf_linux32/release/server.so

Since I did this with the original leak, I will have to rebase my changes on top of comtress master (which will take time since even the initial commit isn't clean)

Anyways, I wasted way too much time on this already, so I might rebase and open a PR somewhere this week.

[maximsmol]

The build-linux branch has enough patches to get to a runnable binary. Some more stuff I have locally rn (not ready to be pushed cause these changes are sus) gets me through the loading screen but I get a crash when connecting to Steam.

[Ristovski]

The build-linux branch has enough patches to get to a runnable binary. Some more stuff I have locally rn (not ready to be pushed cause these changes are sus) gets me through the loading screen but I get a crash when connecting to Steam.

Just tried building from build-linux and it failed.

Here are some of the problems/remarks:
1) 063b32226708095fb510ca7440af754789895a65 introduced weird NixOS symlinks inside thirdparty/protobuf-2.5.0, mainly compile and test-driver. These prevented building protobuf, had to go and revert those files.

2) Step 3 of the Linux build readme, (the only instance where autoconf/autoreconf is required) has a missing autoreconf step.

3) Various weird build/linking related issues such as:

---- COPYING TO [snip]/team-comtress-2/dedicated_main/ [release] ----
rm: cannot remove '[snip]/team-comtress-2/dedicated_main/': Is a directory

I suggest testing the Linux builds inside a clean environment, this can then further be automated ala CI with something like Docker.