Team-comtress-2: x64 bits support

[FriskTheFallenHuman]

Is your feature request related to a problem? Please describe.
One thing that would be good for team comtress 2 is x64 builds, this allows the engine use more resources of the current
machine and it could improve the performance and stability.

Describe the solution you'd like
The leak's groups.vgc has a group for x64 bits, another mention of it is on source_dll_win32_base.vpc

Describe alternatives you've considered
nope

Additional context
iirc this is only used for mac os builds.

[melvyn2]

Merging #113 into this issue:

Is your feature request related to a problem? Please describe.
Slightly; Memory usage runs dangerously close to the 32-bit limit, especially on macOS due to texture memory allocation issues.

Describe the solution you'd like
Overhaul 32-bit specific code so that it does not rely on a specific memory size, and add 64-bit code with #ifdefs in areas that are not possible to generalize.

Describe alternatives you've considered
Fix the memory fragmentation issues on the platforms on which this is an issue.

Additional context
Very High texture quality is disabled on macOS for this reason.
ValveSoftware/Source-1-Games#2238 contains a lot of pertinent information.
Additionally, both CS:GO and Garry's Mod have been ported to 64-bits.

[Yttrium-tYcLief]

this allows the engine use more resources of the current machine

This is a common misconception. The only thing that a 64-bit client would enable is going past the 4GB RAM limit. It would not improve performance, or utilize your hardware more efficiently. Considering TF2 rarely approaches even half of that limit, it's probably not worth the considerable amount of effort that would be involved. Resources are best spent elsewhere.

[maximsmol]

Replacing RFC with "help wanted" because we REALLY do want x64 builds, but this isn't something me or coms want to work on because it's a giant pain

[Zionner]

I've been doing some experimenting with compiling for x64. Not preserving functionality like use of intrinsic functions yet, just a simple full build. To that end I've gotten most of the way through the process, but I've hit a bit of a roadblock in that tf2 seems to use a custom version of the steamworks sdk, which we need x64 binaries for.

[mastercoms]

What seems custom about it? Are you sure it's not just using an older version?

[Zionner]

Looking through the files included in public/steam, it's got a mixture of header files that can't be found in any single version of the public SDKs. An example of this is: "isteamcontroller.h" was introduced in V125 of the public steam API, but the file "isteamgamestats.h" was removed in V118, so there's no public version where these two files are in the same SDK version.

Edit: I've also run a search through more modern versions of the API, and there's no direct port of the old functionality lost from "isteamgamestats.h" - There may be a way to reproduce the same functionality in there somewhere, but simple fixes, like the functionality just being moved into different files doesn't seem to be a thing.

Edit 2: Quick update, I've found a version of steam_api64 that works for most of what we need for steam, but we're still missing x64 binaries for steamnetworkingsockets. I'm still investigating.

[Zionner]

New update:

I've managed to get the dedicated server to compile, but so far I've been unable to build an x64 version of vphysics.dll.

The client needs an x64 version of steamnetworkingsockets in order to compile (Or it requires a rewrite to remove reliance on that dll).

[treacherousfiend]

I don't think removing the reliance on steamnetworkingsockets is possible, as technically even opening a map via the console starts up a server.

[Zionner]

@treacherousfiend I was thinking on this, and wouldn't the same functionality be available via the more modern steam API versions? Ever since they made steam relays publicly usable, wouldn't this essentially reproduce what steamnetworkingsockets used to do?

Trying to update to a modern version of the steam API would introduce other problems (namely, game stats) but it's a thought!

[treacherousfiend]

I wouldn't know that, i'm mostly just going off of some basic source engine knowledge, so removing anything network related from tf2 seems like a bad idea.
I want to say that the dedicated server is even just a headless TF2 exe

[Zionner]

You'd think so, but thus far, the dedicated server hasn't required any linking to steamnetworkingsockets. I'm not saying it won't somewhere down the chain, but it's not linking it in the same way the client does at least.

[treacherousfiend]

It'll be something to look into, might end up being an optimization of its own depending on how its implemented.

[Zionner]

Wanted to a add a new update on progress here: I've managed to compile a broken 64 bit dedicated server. We load all necessary dlls for the server to run, but we can't properly load assets.

Current obstacles:

• VPhysics builds are completely broken. I only got it to compile by removing offending code causing errors. This will almost definitely just break things, but getting VPhysics builds to work is another task in itself.

• DmxLoader is broken, as asset serialization & deserialization relies on the sizeof certain objects, which contain pointers, so the size of those types are no longer correct. I hacked past it by only saving 32 bits of pointers allocated for these objects so we could get compiling, but obviously this will just straight up break, so we need a real solution for this problem.

• There are several other places where compile time assert sizeofs are failing, but I've not yet investigated the consequences of these asserts yet, but the assumption is these are also breaking.

• The Client still needs an x64 solution for steamnetworkingsockets.lib/dll

• Need to write a whole TON of vpc scripts for x64.

[Zionner]

Update: Been out for most of the week, so haven't been able to work on this too much. However it seems as though we are now successfully loading assets on the dedicated server. Now we're crashing due to some memory allocations in the console. So slowly making progress!

[melvyn2]

The Client still needs an x64 solution for steamnetworkingsockets.lib/dll

CS:GO should have a 64-bit copy of steamnetworkingsockets, but I haven't checked.

[Zionner]

CS:GO is 32 bit as well. Even if it wasn't it would need to include .lib files, which it doesn't. =(

[mastercoms]

I'm pretty sure CSGO is only 32 bit on Windows.

https://www.phoronix.com/scan.php?page=news_item&px=CSGO-64bit-Linux

[scarface52]

Does this matter? It's from the latest update.

[micwoj92]

Does this matter? It's from the latest update.

Yeah, this makes TF2 only offline :(

[mastercoms]

No, it doesn't. They just updated to the new Steamworks API for Steam datagrams, which has been included in the main API. We can't really use new APIs unless we update all of our Steam usage, which is a huge feat.

[Zionner]

Unfortunately, it's increasingly looking like it may be necessary to update the steam API used.

I've gotten the dedicated server to the point where we're trying to create a server using the steam libraries. Right now it crashes somewhere inside the current version of steam_client64.dll.

I'll continue working on it, see if I can spot anything.

[melvyn2]

@Zionner for testing, you might want to use Goldberg emulator instead of legit steamworks API for testing. On linux we used it when steamworks API wasn't working, and it did contain all the features TF2 needed (afaik).

[Zionner]

Another Update for you guys: I got past the steamworks problem, by disabling inventory support. I'll look into using Goldberg as @melvyn2 suggested, when I need to use that API.

I'm at the point where I'm just slowly fixing up 64-bit to 32-bit data loss, as they come up. We have started loading entities on the dedicated server though, so I'm hoping that we will have a first test of how broken this stuff is soon. :P

[Zionner]

It's been a while since I updated this, but here's where I'm at:

There's some nasty corruption going on somewhere in the studio model data, that's holding up progress. Due to various reasons I've not been able to spend quite as much time on this the past few weeks, but I am still trying to figure out the problem, and will update here when I make progress.

[Zionner]

It's been quite a while since I've update this, so here's a new one:

I've been in/out of the hospital for the past month, and this will probably continue into January. I still hope I can find some time to continue on this, but not much progress is being made at the moment.

[dimhotepus]

@Zionner Hope you are in a good health and mood!

Feel free to publish your x86-64 branch here. As a one who tried to port Source 2007 on x86-64 (and even get main menu and gui running successfully :)) can help you a bit.

Main problems to solve:

• Critical: missing required ivp sdk, hence, no way for vphysics on x86-64. Need to reverse engineer existing steam version for Linux (check hypothesis it has debug symbols inside), or implement missing functionality. Looks like included ivp sources are from Source 2007 (outdated, but better than nothing).
• Critical: a lot of pointer truncations in studiomdl and everywhere. Well, Valve engineers like to store pointers as ints for optimization purposes :(. Even found whole struct with pointer packed into int in the soundsystem. Such structures can be stored and restored from disk, hence, x86-64 and x86 clients / saves may become incompatible.
• Critical: some structures use size_t and stored / restored from disk, so on x86-64 we should use unsigned int to be portable. Also some APIs are x86 only - ex. consuming / returning pointers as ints / unsigned ints. If we rewrite to x86-64, than we break ABI. So special care should be done to allow playing with x86-64 clients on x86 servers and vice versa.
• Missing Bink Video and Miles Sound System dependencies in x86-64. I've found some old SDK and restored required libs from headers, also found x86-64 versions of Bink Video and Miles Sound System (MSS) that run pretty well. But some MSS plugins are missed for x86-64 - at least mp3 (resolvable).
• Tier0 memory allocators, engine heaps, etc. all require usage of size_t and ptrdiff_t as memory sizes, indices and so on.
• Stack unwinding, debuggers, etc in tier0 use x86-isms and should be ported to x86-64.
• Tier1 collections often return iterators-pointers that should become of uintptr_t or intptr_t types. In other case we got truncations on x86-64.
• Inline asm should be replaced with C++ in some places, or extracted to MASM sources, as MSVC in x86-64 mode doesn't support inline asm.
• Some Windows APIs, like GetWindowLong/SetWindowLong are deprecated and should be replaced with x86-64 counterparts.
• vgui should use intptr_t or uintptr_t instead of int for pointers to action handlers, as it stores address of handler.
• Also very likely mods become broken and simple recompilation for x86-64 wouldn't help. Need to port favorite mods, too :).

Unfortunately, if vphysics issue will not be solved, there is no sense in porting to x86-64.

[mastercoms]

missing required ivp sdk, hence, no way for vphysics on x86-64. Need to reverse engineer existing steam version for Linux (check hypothesis it has debug symbols inside), or implement missing functionality. Looks like included ivp sources are from Source 2007 (outdated, but better than nothing).

https://github.com/HurricanePootis/team-comtress-2/commits/vphysics-rev

[int-72h]

To chime in about the reversed vphysics-
it's partially complete. Most of it works but ragdoll physics is the only thing that doesn't iirc.

[dimhotepus]

To chime in about the reversed vphysics-
it's partially complete. Most of it works but ragdoll physics is the only thing that doesn't iirc.

May be it makes sense in finish reverse engineering?

Personally i'm not strong in assembler, and looks like you are using some high level Hydra tool. But still, if help is needed, ready to jump in.

[dimhotepus]

For anybody who interested in porting to x86-64:

1. Must read 64 bit porting lessons here: https://www.viva64.com/en/l/. A lot of useful information from guys who are doing 64 bit for almost 2 decades.
2. Search how to and enable all warnings in your compiler, remove warnings suppression from the sources. Compiler is your friend, think seven times before suppressing something. If you think compiler is faulty, rethink again, ask compiler devs, rewrite code to not issue the warning at all.
3. Use static analyzers integrated into your compiler (clang-tidy, msvc prefast, etc.). Show a tons of things, saves you from sleepless nights debugging crazy issues on customer machines.
4. Use third-party static analyzers, like Coverity and PVS-Studio. They are written by people with significant knowledge in language standards and compiler internals, and they save your time. If you think only enthusiast devs make mistakes, read this: https://www.viva64.com/en/inspections. Personal favorite - Source SDK analysis results - https://www.viva64.com/en/b/0229/
5. Use Clang sanitizers (https://hpc-wiki.info/hpc/Compiler_Sanitizers) to catch problems in runtime.
6. Last but not the least - valgrind.

[int-72h]

To chime in about the reversed vphysics-
it's partially complete. Most of it works but ragdoll physics is the only thing that doesn't iirc.

May be it makes sense in finish reverse engineering?

Personally i'm not strong in assembler, and looks like you are using some high level Hydra tool. But still, if help is needed, ready to jump in.

The one doing almost all of the reversing is crackbomber - I'm not doing any reversing (just working out the bugs for the linux port).

[dimhotepus]

Small tip: add /windows to vpc flags for project generation, and generated Visual Studio solution will have both Win32 and x64 platforms. See https://github.com/mastercomfig/team-comtress-2/blob/master/external/vpc/utils/vpc/main.cpp#L644

[dimhotepus]

Also interesting to note, that vpc's $WIN32 platform doesn't work for x86-64 build on Windows. You should explicitly mark vpc item as $WIN64, like $WIN32 || $WIN64 or just $WIN64. It leads to a huge amount of changes in vpc scripts to support x86-64 build, because almost all $WIN32 code is also $WIN64.

For example, let's see how Microsoft defines _WIN32 in x64 builds: _WIN32 Defined as 1 when the compilation target is 32-bit ARM, 64-bit ARM, x86, or x64. Otherwise, undefined. (https://docs.microsoft.com/en-us/cpp/preprocessor/predefined-macros). So made a hypothesis that it is easier to fix vpc project generation for x86-64 by assuming $WIN32 is also enabled when $WIN64 platform defined, instead of manually replacing almost all $WIN32 (~116 cases) with $WIN32 || $WIN64.

Summary:
VPC platform | Was before fix | Become after fix
------------ | ------------- | ---------------------------
$WIN32 | x86 only | both x86 and x86-64
$WIN64 | x86-64 only | x86-64 only
$WIN32 or $WIN64 | both x86 and x86-64 | both x86 and x86-64
$WIN32 and !$WIN64 | x86 only | x86 only

In order to be compatible with original vpc behavior we can do this under special command line flag, smth
like /definewin32forwin64.

Pros: less changes in original vpc scripts, easier to reason about for regular windows dev (as Windows SDK headers do).
Cons: automated solution can miss some edge cases, changes in vpc can lead to regressions.

What do you think, should we change VPC, or just replace $WIN32 (~116 cases) with $WIN32 || $WIN64 where needed?

[melvyn2]

The $WINDOWS VPC macro seems to be defined for both arches.

[dimhotepus]

The $WINDOWS VPC macro seems to be defined for both arches.

Thanks! Checked, it does.

Replacing $WIN32 (~116 cases) with $WINDOWS looks like an easier & safer task than changing VPC, even if i personally prefer the latest one :).

[dimhotepus]

Now it is possible to have both old x86 and new x86-64 bins as as single pack.
Player can just select which one he will run.

[Zionner]

Since there's been some movement here, I thought I'd update you all on my end:

I've been going through chemo for the past month, and will be for the next few months. So I've not had much time, or energy to work on this.

I'll do my best to at at least get a branch up of my changes. A couple of the changes are iffy (Specifically, the changes around studiomtl serialization) But 99% of the changes have been fixing pointer truncations, replaced some (not all) of the ASM & SIMD stuff, and fixing x64 compiler errors. Last I worked on it, x64 was compiling (Server side only), you could start a server a begin loading a map. The map loading would crash trying to deal with animations.

It's worth noting that the x64 changes are on a pretty old version of comtress (See my fork for the last time it was update).

[dimhotepus]

Since there's been some movement here, I thought I'd update you all on my end:

I've been going through chemo for the past month, and will be for the next few months. So I've not had much time, or energy to work on this.

I'll do my best to at at least get a branch up of my changes. A couple of the changes are iffy (Specifically, the changes around studiomtl serialization) But 99% of the changes have been fixing pointer truncations, replaced some (not all) of the ASM & SIMD stuff, and fixing x64 compiler errors. Last I worked on it, x64 was compiling (Server side only), you could start a server a begin loading a map. The map loading would crash trying to deal with animations.

It's worth noting that the x64 changes are on a pretty old version of comtress (See my fork for the last time it was update).

Thank you for the update!

What do you think about me cherry-picking your commits into the https://github.com/mastercomfig/team-comtress-2/pull/554 (with all the credits, of course)? A lot of work has been done by you, so it makes sense to not loose any bit of it.

P.S. Wish you to fight against illness and win. Waiting for your final PR approval in a next few months :)

[Zionner]

I've committed the changes I managed to get through at: https://github.com/Zionner/team-comtress-2/tree/x64

[dimhotepus]

I've committed the changes I managed to get through at: https://github.com/Zionner/team-comtress-2/tree/x64

Nice work. Like how commits are atomic and easy to follow and cherry-pick.

[dimhotepus]

Progress update:

Managed to compile most of the bins in x86-64 mode on Windows.
Thanks to @Zionner for huge job of porting the things on the server part.

Now updates for client/server:

• VPhysics still need to be fully reverse-engineered (ragdolls) before port is ready. May be help with when port will be blocked only by this. Resolvable?
• Required socket, steam_api and steamnetworkingsockets libs with back x64 binaries to build client, server, engine, inputsystem, etc. Will merge with @Zionner fixes, so some dependencies will be satisfied. Resolvable?
• Required to find lib/bin for libpng to build in x64. Will use thirdparty/libpng-1.5.2 as a reference. Resolvable.
• Required to find lib/bin for SDL 2.0.2. Will usethirdparty/SDL2as a reference. Found one, now use precompiled. Resolvable.
• Required to find lib/bin for open_vr <= 0.9.1. Found one, now use precompiled. Resolvable.
• Required to find lib/bin for bink (binkw64). Found one, now use precompiled. Resolvable. Need to check, likely engine doesn't depend on it at all, so just remove.
• Will add new missed dependencies here.

Missed x64 dependencies:

• Open - should be either precompiled into binaries and provided inside source tree (less control), or compiled from source during build (better).
• Proprietary - try to not use at all (binkw64, mss64, steam_api, ...). If not possible - headers / libs / binaries provided inside source tree.

[melvyn2]

libpng is in the source tree, check the vphysics-rev branch on how to compile it (check build.sh).

[int-72h]

Good news on vphysics - @CRACKbomber has said that he's continuing the reversing.