Wrye-bash: Load order handling fixups and enhancements

[Utumno]

• [x] rename menus Load -> Active Plugins etc (https://github.com/wrye-bash/wrye-bash/commit/d22017e9e710b321f5c2f141a85d9239101ee364)
• [x] _fixLoadOrder should check for duplicates (https://github.com/wrye-bash/wrye-bash/commit/a18e4cb7490e42d1bbd5bc9b1430d93267718fa6)
• [x] rework #256 to skip over invalid load orders (https://github.com/wrye-bash/wrye-bash/commit/4d432b986952a512aa55cbd4253366835e6a49cd)
• [x] DOCS!
  
  
  
  • [x] mention that WB fixes most invalid load orders automatically - done in 230adfd7eeeafa12a975eb2622f033da4eb91e59.
  
  
  • [ ] mention Bash breaks conflicts in timestamp games
  
  
• [ ] list active mods rename
• [ ] list all mods menu add
• [x] Drop _modTimesChange ? (32705cd78568a04a8407d125eb00cd5d9b2a8a1b)

• [ ] detect plugins txt / loadorder txt deletion and restore cached if available (or if we lock load order, #210)

  def _plugins_txt_modified(self):
  +        exists = self.plugins_txt_path.exists()
  

  If does not exist restore/save our cached active/loadorder if any !!

• better error handling:
  
  
  
  • [x] warn on fixups and only on fixups (espm flip we should not deprint (do we still ?) - unhide should not deprint...) - actually we should completely remove the deprints from the
    
    games.py module - the client should compare and handle that - partly done in 48427d33a953804bb5084492645983645f59964f - at least fixups bookkeeping separated from fix logic
  
  
  • [x] backup plugins.txt and loadorder on fixups - done in 230adfd7eeeafa12a975eb2622f033da4eb91e59.
  
  
• Tests !
  
  
  
  • make sure I always have valid lists
  
  
  • make sure I always return cached values when I pass them
  
  
• [x] PEP8 load_order.py (https://github.com/wrye-bash/wrye-bash/commit/be33be98b23bb1c1aa7326f9d6a14d77069fb01b)

__fixLOAD ORDER check what happens with desync and flip esm in skyrim

Skyrim - handle desync with plugins txt: COMMENT

esl and co:

• [x] Creation Club files. The order of these is specified by the *.ccc file for each game. If that file is missing then it falls back on a hardcoded list. -> __add this hardcoding to bash__ - done in d136bade777db841cc192f43328fb43a36926749.

308/309 -> new issue: #578

[Infernio]

Regarding this point:

revisit master/children warnings - add a big fat one for esm or esl flagged files that have an esp as master

We should show a warning for any master file (according to in_master_block, i.e. ESM-flagged or with .esm/.esl extension) that has a non-master file (i.e. one where in_master_block is false) as a master.

So, basically:

• For every plugin in the load order:
• If load_order.in_master_block is True for that plugin:
• For every plugin that it has a master:
• If load_order.in_master_block is False for that master:
• Print a big, fat warning.

[Utumno]

We do have a warning for out of order masters - a separate color and text - there is a piece of strange code for detecting circular masters too

Let's finish with esls first I'd say

[Infernio]

This isn't for out of order masters, it's for ESMs that have ESPs as masters. That's problematic because it forces the ESPs to load in the wrong block.

But yeah, ESL support should get finished first.

[Utumno]

Since Bash will not load the esp in the master's block those _will_ be out of order (the esl child _will_ be loaded in the masters block) so some warning will be issued - we just need to offer a more serious one for the esl case and probably a menu item to correct those (somehow). Anyway I may be missing something been programming for a while on this
Over and out :P

[Utumno]

Some priorities:

• [x] add docstrings in games.py (6cee9084fe1f4b4aeadd599294ea9f447f764ba1)
• [x] rename games.py (duh) - was meant to handle, well, game handling but proved not a good idea. _games_load_order.py? - done in 78ed0234bb4ed961f1071573d77374952b8aac8a.
• [x] traceback in https://github.com/wrye-bash/wrye-bash/issues/260#issuecomment-535924659 - points to some bug in there - done in ce78fc0b194987d6979e03028b8d16d5a647aea2.
• [x] saved active plugins lists must not be saved along the UI settings in BashSettings.dat !!! That needs some backwards compat code to be removed in 308 - done in 27fdc4d74909b4327419a5cfea90f5c951edbc0e.

See also first post

[Infernio]

LO handling seems to have some issues in FNV (probably also FO3, but I haven't tried that). When I install some mods (via BAIN) with 'Lock Load Order' turned on, I get the

Load order has changed outside of Bash
and has been reverted to the one saved in Bash. You can hit
Ctrl + Z while the mods list has focus to undo this

popup, despite everything having been done through Wrye Bash.

[Infernio]

818117f750e0f46970462bdf492201ae24c35568 fixes the traceback mentioned in the comment above. Probably not the cleanest solution, but it's better than crashing on boot :)

[Utumno]

LO handling seems to have some issues in FNV (probably also FO3, but I haven't tried that).

Hmm I did have that myself and not on FO3/FNV - probably some edge case in locking (complex code, actually a stress test of the load order API)

[Infernio]

I've almost implemented the CCC fallback in its entirety, just waiting for someone to send me the most up-to-date Fallout4.ccc contents. I'd also like to push another task/request here:

• [x] Add the You can't reorder the following mods behavior to Skyrim (LE) masters as well. No reason to allow reordering those DLCs. Don't know how safe it would be for the earlier games (TTW and Nehrim come to mind). - done in d9f1bee7125cd47694d6c8892154e50903f73247.

As a side effect, doing that would also close the last TODO in #433, finishing that issue off for good.

[Infernio]

CCC fallback is done in 575f1f0a5b4e42cda26911ccf69ba8a69f51074d. I hardcoded it into games.py, because importing bush in there felt... wrong to me :P

[Utumno]

Oh thanks! It does worry me that this should be kept up to date - a way of reading this from the game exe would be ideal but I don't even know what this would entail

because importing bush in there felt... wrong to me :P

it would be, still this is getting big - when I started there was only Oblivion and Skyrim :)

[Infernio]

If we read it from the exe, we'd probably have to keep an offset up to date since that could change per game version.
And that would likely be far harder for future maintainers than hardcoding it like this, so this is probably the best solution :(

[Infernio]

I updated the CCC fallback to the latest game versions in f6941e9d86e7b95c30e6e509b5a615db688dea31. We could probably make a branch out of all the LO commits on nightly and merge it soon, they've been there for somewhere between 4-8 weeks now with no issues reported.

[Utumno]

Yep - good enough is good enough!

[Infernio]

Alright, that's enough for dev :P Cherry-picked a whole bunch more commits that have been on for a month or more, because it was getting bloated to no end.
Still, this only helps attack the commit number, not how much code is on nightly. Especially 15-wx3-upgrade is becoming an annoying dependency for a lot of branches (specifically localize.py, since it isn't just a simple rename).

[Infernio]

LO handling seems to have some issues in FNV (probably also FO3, but I haven't tried that). When I install some mods (via BAIN) with 'Lock Load Order' turned on, I get the

Load order has changed outside of Bash
and has been reverted to the one saved in Bash. You can hit
Ctrl + Z while the mods list has focus to undo this

popup, despite everything having been done through Wrye Bash.

So, I've managed to find one case where this is replicable, even in Skyrim:

1. Ensure that Lock Load Order is enabled
2. New Mod > A.esp
3. Right Click A.esp > Add ESM Flag
4. Make an installer containing this ESM-flagged plugin
5. Right Click A.esp > Remove ESM Flag, move it down in your load order a bit
6. Install the installer you created in step 3, overriding the regular A.esp with the ESM-flagged A.esp

This will show the popup that LO was changed outside Wrye Bash. What probably happens is that BAIN wants to move the now ESM-flagged plugin up into the master block, but doesn't call load_order.Unlock(), so you get the popup and WB undoes the change.
Since the plugin does end up in the right place when you switch to the Mods tab, WB probably does some fixup when it realizes that the plugin is now outside the master block, and this fixup correctly calls load_order.Unlock().

Edit: ead3701f8d765166880a5ce7cd34bfa1f5ec5a4d fixes this, but I'm not sure it's the right approach.
Edit 2: The commit above also fixes the same issue with the Anneal command, so that's a good sign 😛

[Utumno]

Yep - keep in mind that those esm flips are very hacky code - for one the mtime/size cached values do . not change when you do an esm flip so this may be missed in an update - there are yak/hack comments all over that.