Wrye-bash: Linux Support

[Utumno]

Bash should be able to run on linux. Opening this to start investigating.

• [ ] Case sensitive filesystem - oopsie woopsie, Bash has absolutely no notion of that
• [ ] Proton awareness - so that the next bullet point is obsolete for many (most?) users
• [ ] A (better) way to configure paths, e.g. of the documents folder - ref #572, we do not want to continue adding to bash.ini (because of #250)
• [x] app buttons code is tightly coupled with _winreg win32api - Linux does not really use app buttons code anymore, but that just points to us needing to rethink these entirely (#570)
• [ ] hunt down all hardcoded backslashes (duh) - ref 2e65a933e1a53b582a7d3c30600171f7ad62ad6c
• [x] move winreg from bass and to an environment detection module - cross platform
• [x] WindowsError, duh:http://stackoverflow.com/q/1210118/281545, http://stackoverflow.com/a/8890190/281545, http://stackoverflow.com/a/2678797/281545, http://stackoverflow.com/a/22027543/281545 -> edit done see #252
• [x] this (traceback solved in a9751915e101bf0ba6261e66dce1ff5d640f3e5d but check also warns) :

$ python2 /mnt/win/Users/MrD/Desktop/ZZZ/wrye-bash/Mopy/Wrye\ Bash\ Launcher.pyw -o /mnt/win/GAMES/TESIV/Oblivion No translation file for language: en 01:33:47 AM: Warning: Mismatch between the program and library build versions detected. The library used 3.0 (wchar_t,compiler with C++ ABI 1008,wx containers,compatible with 2.8), and wxPython used 3.0 (wchar_t,compiler with C++ ABI 1002,wx containers,compatible with 2.8). Traceback (most recent call last): File "/mnt/win/Users/MrD/Desktop/ZZZ/wrye-bash/Mopy/Wrye Bash Launcher.pyw", line 88, in <module> bash.main() File "/mnt/win/Users/MrD/Desktop/ZZZ/wrye-bash/Mopy/bash/bash.py", line 363, in main import bosh File "/mnt/win/Users/MrD/Desktop/ZZZ/wrye-bash/Mopy/bash/bosh.py", line 55, in <module> import balt File "/mnt/win/Users/MrD/Desktop/ZZZ/wrye-bash/Mopy/bash/balt.py", line 666, in <module> import windows File "/mnt/win/Users/MrD/Desktop/ZZZ/wrye-bash/Mopy/bash/windows.py", line 31, in <module> from ctypes.wintypes import MAX_PATH File "/usr/lib/python2.7/ctypes/wintypes.py", line 23, in <module> class VARIANT_BOOL(_SimpleCData): ValueError: _type_ 'v' not supported

Of course the biggest hurdle is the dlls... If it proves easy we can compile them to .sos so much the better
All DLLs dropped by rewriting relevant code in Python:

• [x] libloadorder - #295
• [x] libbsa - #339
• [x] libloot-python - #431
• [x] CBash - #530

Blockers:

• [ ] #250 - if we want to deploy to a single installation point in /usr or /opt, we need profiles
• [ ] #460 - most distros don't package the libs we need for py2 anymore
• [ ] #572 - weird setups and paths are even more common on Linux, we need to solve this once and for all (and in a profiles-friendly way, no adding to bash.ini!)

[nycz]

i've been thinking about the problem with case (in)sensitive paths a bit since that is the by far biggest hurdle to get bash running on linux. the only decent option i can come up with is generating and keeping a dict of lower-case-to-mixed-case path pairs that bash's internal path system deals with.

basically this:

1. something wants to access a file in Directory1/dir2/DIR3/File_Name
2. change the whole path to lower case
3. list all items in the root path on disk, change them to lower case and find directory1
4. add the unchanged case version of directory1 (let's say DIRectory1) to the output path
5. go to DIRectory1 and repeat until you have the full path

i have no idea if this is as slow as it sounds, but it should be fairly easily cached. also, i'm thinking this should only be done this way on case-sensitive file systems so hopefully windows shouldn't be affected. one fairly significant problem would of course be case-sensitive file conflicts, for example if someone manually adds the directory Actors where actors already exists, but i don't see an easy way around that. my best bet would be to add a check for duplicate files/directories or something.

[Utumno]

if someone manually adds the directory Actors where actors already exists, but i don't see an easy way around that. my best bet would be to add a check for duplicate files/directories or something.

And that's where I stopped reasoning about that

What I think is a better approach is to refactor "bash's internal path system" till it's absolutely clear where the fine line between the path handling and the system calls limit lies - pathlib has separate mixins for abstract paths and concrete paths and Pitrou certainly though deep about that ;)

So help needed in refactoring - I filled up an issue ( #368 ) and my current wip branch contains a new data structure that is very related to the problem (LowerDict) - a very tricky merge I have not found the time to push to dev. Keep in mind that it took me couple of years to finally realize what's the right way to address that (simpler) problem - there are many details that I can't really convey but LowerDict is the tool for that job.

Keep also in mind that refactoring in Path class led to a _dramatic_ decrease in memory: https://bethesda.net/community/post/371559

So this is not an easy problem. Bottom line we should revisit this once we've switched to pathlib - and that's way, way ahead. Meanwhile there's a lot to be done.

[nycz]

first of all i hope i didn't come across as arrogant or sounding like i had the silver bullet solution or anything D: i know things are a lot more complicated than they might seem for someone not intimately familiar with the source code and i don't expect this to be solved easily or quickly. this is, however, more or less the one thing that keeps bash from working on linux so the sooner it's done (with or without my help) the sooner i can use (and by extension develop) bash properly on my own computer so i'm eager to do what i can to help, specifically with this.

with regards to your branch and LowerDict, is that so tightly tied to future Path changes that it's not meaningful for me to start diving deep into Path system and pathlib yet?

[Utumno]

first of all i hope i didn't come across as arrogant or sounding like i had the silver bullet solution or anything D:

Haha, not at all, same here

this is, however, more or less the one thing that keeps bash from working on linux so the sooner it's done (with or without my help) the sooner i can use (and by extension develop) bash properly on my own computer so i'm eager to do what i can to help, specifically with this.

Preferably _with_ your help - dunno of anyone else that actually uses Bash on pure linux :P

with regards to your branch and LowerDict, is that so tightly tied to future Path changes that it's not meaningful for me to start diving deep into Path system and pathlib yet?

It's certainly meaningful. I really don't know how to guide you though - the hypothesis that paths are case insensitive is _implicit_ in the code. I think you should start with the hypothesis that case insensitive paths are indeed unique (that is ignore cases like Data/Textures and Data/textures) and investigate if all works - tests...

Bash's APIs _are_ case aware however (both Path and LowerDict) - that's a good thing

I will merge Asap but meanwhile, having a look in there, helping (once merged) to clearly name attributes that use case insensitive data structures (prepend ci_) then hunting down Path (start with cs property uses) is a possible course of action.

But also probably tackling other better defined refactoring issues will make you more familiar with the code (and help with current dev) - for instance here's one I would like someone to take off my plate: #372

[BeermotorWB]

This may a bit exotic, and may be outside of the scope of Wrye Bash as an application, but there has been some work done on an overlay FUSE-mountable filesystem called ciopfs.

The TL;DR of it is it will take a directory (e.g. "/usr/local/Windows Games/skyrim/data/") and overlay it as a case-insensitive union/overlay mount at some other path (e.g. /home/soandso/skyrim/data/), all in userland so it doesn't require escalation of privileges to mount it.

The good news about this is there is (or was) already a ciopfs package for Ubuntu in the main repository. I'm not sure about its availability on other distros, specifically Fedora which is the only distro with any kind of Wayland movement at the time of this writing.

I'm not saying this is the panacea for this particular issue, but it may feed some ideas.

[Utumno]

Good to know @BeermotorWB

[Ember2528]

Recently case insensitive directory support was added to the Linux kernel for ext4 filesystems. It's not a complete solution since people use all kinds of filesystems, but can help with part of that problem

[Utumno]

@Ember2528 all good but we have a more pressing matter at hand - namely wx3 and python3 - hands needed

[Utumno]

Got a new answer on MAX_PATH queries that seems worth investigating https://stackoverflow.com/a/62796371/281545

[monyarm]

Tried running Wrye Bash on linux, and found a few things that need to be changed in order for it to even launch
pull all of the windows specific classes in env.py under a

if _os.name == "posix":
    pass
else:

change the ctypes import to :

from ctypes import byref, c_wchar_p, c_void_p, POINTER
try:
    from ctypes import Structure, windll, wintypes
except (ImportError,ValueError): # we're on linux
    Structure = None
    windll = None
    wintypes = None

As importing from ctypes throws an ValueError under linux

[llde]

I think WB on linux is supposed to work only under wine (and it works well, mostly)

[Infernio]

This is the issue for eventually getting WB to natively run on Linux, without wine.
The current status of this issue is that we're keeping it in the back of our heads and trying hard not to introduce any assumptions that only hold on Windows, but WB isn't even close to running properly on Linux yet and it's not a focus either. Our current focus is #460, which will help this issue as well with so many distros now dropping py2.

As @monyarm noticed, env.py is currently Windows-specific. It's not supposed to be and eventually we want to refactor it into a package that provides two implementations, one for Linux and one for Windows.

[Utumno]

Thanks @monyarm - @llde no linux support is a goal, albeit an experimental one, as a case sensitive filesystem makes no sense for a game developed on windows. But if the user knows what they are doing Bash should eventually run in linux

EDIT: ninj'ed :P

@Infernio I had a branch somewhere making actually env import from _windows that's the goal (and it will be the _windows/_linux.py that will blow with import error but yeah, 480, 312, 460 ...)

[Infernio]

Was scratching my head why code like this broke on Linux:

dict_of_paths = {}
# dict construction etc. here
some_val = dict_of_paths[some_path] # blows on Linux

Then I remembered:

[infernio@inf-desktop ~]$ python3
Python 3.9.1 (default, Dec 13 2020, 11:55:53) 
[GCC 10.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.path.normcase('/home/infernio/.local/share/Steam/steamapps/common/Oblivion/Data/Oblivion.esm')
'/home/infernio/.local/share/Steam/steamapps/common/Oblivion/Data/Oblivion.esm'

So Path.__hash__ doesn't work because currently Path._s == Path._cs on Linux, always :P

[Infernio]

We can't just use lower() either - normcase does more on Windows: https://docs.python.org/3/library/os.path.html#os.path.normcase

Edit: we can't even put it into env and change the behavior by OS - at least the usage in bsa_files really does need the other part of normcase's behavior.
Edit 2: see 8b6aecb91b8b45003c228635c1f601a2e26a4c57