Community-patch-dll: Freeze on barbarian turn.

[tu-79]

_1. Mod version (i.e Date - 4/23):_

2018-01-21

_2. Mod list (if using Vox Populi only, leave blank):_
FlagPromotions
InfoAddict
Improved City View

_3. Error description:_

Freeze on barbarian turn.

_4. Steps to reproduce (optional):_

Logs.zip
AutoSave_0216 AD-1530.zip

This time I have a recent savegame. I just spread religion in one aztec city and killed a hunnic dromon.

Supporting information:
Please note that you can attach .zip files by dragging-and-dropping them. If possible, zip up all supporting data and post that way.

1. Log files (always attach your Logs folder, located at My Documents/My Games/Sid Meier's Civilization 5. Make sure you have enabled logging before experiencing an error! Go here to find out how: http://forums.civfanatics.com/showthread.php?t=487482):

2. Save game (always attach a save that was made a turn before the error; located at My Documents/My Games/Sid Meier's Civilization 5/ModdedSaves):

3. CvMiniDump.dmp file (attach if experiencing a game crash. Located at Program Files/Steam/steamapps/common/Sid Meier's Civilization V):

4. Screenshots (optional):

[Jotaro31400]

yes i've got the same issue i will join my log soon.

[Infixo]

Is FlagPromotions affecting save files? If not, can you load the save without it and see if the problem exists?

[Padre19]

Same issue for me without FlagPromo.

[ilteroi]

we're not making progress. i can't reproduce it.

once more: i need a save where all i need to do is to press "end turn". otherwise any unit i move in a different way than you influences the game state and we'll never catch this.

[tu-79]

But then I need to manually save just before hitting NEXT button. Because I don't know if I'm going to be able to reproduce exactly all my steps in the last turn.

EDIT. Anyway, if reloading solves the problem, I wish to continue this game, it's the first time in a while that I'm ahead. And I want to see if I can stop than hunnic invasion with such a small army.

[LoneGazebo]

@ilteroi I think I caught the freeze:

DangerPlots.cpp Line 972

m_lastResults.push_back(std::make_pair(unitStats, iPlotDamage));

It crashed here with access invalid due to location choice during a barb turn - stack was a simple path lookup for a unit.

[LoneGazebo]

If I can get more info I will.

[ilteroi]

crash or freeze?

anyway, i think your callstack is shot or you simply ran out of memory.

more info definitely appreciated.

[tu-79]

I have 8GB RAM. Map is small. I almost never had freezes before. And that other time was a bug that only affected Windows 7 machines. I'll try to provide more info, but I'm a bit busy now looking for a school for my son. My wife won't appreciate that I spend so much time 'playing'.

[ilteroi]

no worries, i was talking to G :)

[LoneGazebo]

Freeze. I can’t get it to replicate and I didn’t realize what I’d caught until I’d already de-linked the debugger. :( Bugger all!

[Eriktheread]

Byzantines, huge map, 14 civs. Getting this every 1-4 turns. Unplayable right now. Other mods: Wonder race, unit path, city limit viewer.
Save:

AutoSave_0112 BC-0075.zip

Logs:
Logs.zip

[XplosiveLun]

Here's another case. Freeze twice in recent turns. Unusual because it used NOT to happen in a chain. One coincident I discovered is it may be related to Barbarian capturing a city state.

• Mongolia
• Standard
• 8 players
• Communitas map

Mods:

• Unit path viewer
• JFD's cultural Diversity (core + soundtrack)
• Historical religion icon

Saves & logs:
Mongo AutoSave 100-102.zip
Logs.zip

[ilteroi]

the freeze probably occured to me as well, but when i stepped into it none of the threads were in the game core DLL. that's bad. means it's either a latent vanilla bug we managed to trigger somehow, or possibly a lua issue.

[LoneGazebo]

@Infixo is it possible that this is related to some new lua you've written? I've now had the freeze happen to me twice while in the civilopedia...it feels like an infinite loop is happening in lua, that's my guess.

[ilteroi]

can you maybe build a version without the recent UI changes so that we can verify the lua theory?

[Infixo]

UI Lua is not executed during AI turns, is it? At least I have not seen any PlayerDoTurn events or similar, in EUI.
Do we have a save file with just VP that causes problems? I’m running test games all the time and have not encountered this issue. Occasional CTD if there is a bug, but not the freeze.
The only weird case I had recently was with 3/4UC project when FireTuner started to loose InGame context and crashed. I’m asking about clean save file because e.g. UnitPathViewer is not fully compatible with VP - Akamal privately fixed it.
Civilopedia - do you refer to the situation that the cursor changed into „processing” and stayed? if so, this is not freeze, the cursor normally works (there was a normal bug with one of fields, that I have fixed); if something else - let me know on which article, and I’ll check

[Infixo]

Ok, I'm at the computer so I can check. I am basically changing:

• CityView.lua
• CityView_small.xml & CityView.xml
• EUI_tooltip_library.lua
• Civilopedia.lua

Plus one-timers:

• ChooseInternationalTradeRoutePopup.lua (turn counter)
• TopPanel.lua, GPList.lua & GPList.xml (gp turns fix)

Any version from releases before Dec, 15th should do.

[Infixo]

I ran the file from @Eriktheread. 20 turns manually, no problems.
But I cannot run in auto-play. I get CTD in DangerPlots::ResetDangerCache(const CvPlot* pCenterPlot, int iRange), stack is like SetAIAutoPlay->SetAllPlotsVisible->AddKnownAttacker.
I don't know if this is related at all, maybe some other bug, but definetely a bad sign.

Edit. Interesting comment in the code: "//open up a slot if required - it seems Really Advanced Setup does not leave observer slots but sets them all to closed". And then it goes into the CTD section (he uses RASv15).

[ilteroi]

interesting.

i did recently make a change to AddKnownAttacker, so there might be a bug there. will check but that's not the problem we're looking for.

the freeze doesn't seem to happen during autoplay and it doesn't seem to be reproducable so that points to a UI issue. don't know why it would only occur during the barbarian turn though.

[deadstarre]

I autosave every turn. First time I froze was on turn 35, appearing during barbarian turn but when i rebooted and got back into the game, turn 36s auto save was there - as if turn 35 had fully completed, even though itd appeared to freeze. I loaded up the turn 36 save expecting to continue on, and it froze again before I could click start from the the portrait loading screen. so in that way, i did confirm a freeze occurring not during barbs turn, hopefully that means something to anybody. unfortunately I no longer have either save

[Infixo]

Re:CTD Internal table m_DangerPlots is not initialized (NULL).
I've made a temp fix, and it's working.

Freeze could be due to Lua script, but I can hardly imagine an UI because nothing is executed without the player action. Also - entire UI is available during auto-play - you can go into every city, open city view, etc. And as you say - somehow auto-play is immune to the problem?
On the other hand - FlagPromos and UnitPathViewer are constantly monitoring all units and have a ton of code that is executed automatically. Plus they use persistent storage which is well... not the most reliable thing in Civ5. I would really like to see a report from a clean VP install, without these mods.

[Eriktheread]

If we havnt found this by the time I start a new game Ill load one up without the path viewer.

I also autosave every turn. Dunno if its related.

[LoneGazebo]

The two times it froze for me I was doing auto play and had the civilopedis open.

[LoneGazebo]

*Civilopedia (my phone hates this word)

[Infixo]

But what article did you open? Civilopedia doesn't do anything by itself. Until you click something, nothing happens, it just displays static text.

[Infixo]

Anyway, I tried also tu_79's file. Played 20 turns manually, then auto-played, nothing happens.

[LoneGazebo]

@Infixo I think I was browsing wonders?

[ilteroi]

path viewer is probably not to blame, tu79's save doesn't use it.

regarding autoplay i really meant unattended autoplay.

[Infixo]

Maybe we should announce a bug hunt. Looking for a save file from clean VP game, no mods, that has that issue and it is repeatable.

If this UI related then basically everyone should get that problem. But it is still rather very rare. I’m leaning towards memory problems and they can be accelerated by mods. We should also ask people how many turns they played until freeze happened.

[Legoras]

Sorry for interrupting, but I reading EUI thread and there sometimes people have identical freeze, with no other mods, even without Vox. So maybe it is Only because EUI ?

[Infixo]

@Legoras Can you give a link?

[Eriktheread]

It might just be anecdotal, but Ive noticed it happening more frequently when I have another section of the UI open, like a city screen or the tech tree. Current game was smooth up to turn ~100, now its freezing every 4 or 5 turns.

[Legoras]

@Infixo
I only read last 6 pages EUI thread, I think will be more there. 1.29 freeze, sometimes people back to 1.28 and it fixed, but sometimes 1.28 also crash. I dont know which version Vox use [as base]

https://forums.civfanatics.com/threads/enhanced-user-interface.512263/page-137#post-14797954 [freeze]
https://forums.civfanatics.com/threads/enhanced-user-interface.512263/page-139#post-14837841 [hang]
https://forums.civfanatics.com/threads/enhanced-user-interface.512263/page-140#post-14865657 [crash]
https://forums.civfanatics.com/threads/enhanced-user-interface.512263/page-141#post-14885503 [crash]
https://forums.civfanatics.com/threads/enhanced-user-interface.512263/page-142#post-14923931 [freeze]

Update:
https://forums.civfanatics.com/threads/enhanced-user-interface.512263/page-136#post-14757103 [ freeze]
https://forums.civfanatics.com/threads/enhanced-user-interface.512263/page-136#post-14796185 [freeze]

[LoneGazebo]

IIRC there is a limited amount of memory that the UI can access at any one time - if that amount is exceeded, it can cause issues. At one point bc1 had to remove some value caching methods to overcome this issue. I believe that there is chance that some recent additions to the code are causing this. @Infixo I'm not saying it is bad code, it's not, but rather that the UI can only handle so much without freezing.

[Infixo]

32-bit Lua on 64-bit machines can access up to 1GB of memory (due to some weird technicalities it is less than on 32-bit machines). I suppose it shares memory with Civ5.exe which is also 32-bit. Maybe if people are using many mods or running games on big maps then it really hits some memory issues. Let's see if anybody with clean VP will have problems.

[LoneGazebo]

I'm getting pretty consistent crashes that happen in the lua exe now. Something's up.

[Infixo]

In what configuration? Auto-play or manual? Post a save file, pls.

[LoneGazebo]

Auto-play. Right now it's inconsistent, and the minidumps are varying between lua and civ5 exe. Hmm...

[Infixo]

Don't know what to say. I am auto-playing 300+ turns game on 1-26 code + my usual UI mods. Not a single problem.

[ryanmusante]

I had a freeze on turn 366 on marathon speed during barbarian turn. I didn't have logging enabled this time. Before the freeze, task manager had executable using 1.9 GB. I am using additional UI mods as seen.

[bigcat88]

Maybe this will help, this minidump from yesterday. I have disabled autosaves.
Almost clear VP, only Enchanced Naval , really advanced setup, ige and wonder race.

CvMiniDump.zip

[LoneGazebo]

I've been going through the code from 1-14, and - in the DLL - the only questionable spots I've seen are in CvPlot:

getNumUnitsOfAIType
getFirstUnitOfAITypeSameTeam
getFirstUnitOfAITypeOtherTeam

I think they're probably fine, but 'while' statements are the biggest culprits for hangs in the DLL.

[LoneGazebo]

also GetNumSpecificFriendlyUnitCombatsAdjacent is new and has a while statement in it.

[LoneGazebo]

Other than that, the only major new code changes are @Infixo's lua and @ilteroi's rewrite of the dominance zones logic. Neither of which update during the barbarian turn. I'm stumped.

[ilteroi]

let's do this systematically. create a version with only the c++ changes and a version with only the lua changes and see which one causes the freezes.

[Infixo]

Some other functions use while(pUnitNode != NULL) but these use only while (pUnitNode). Theoretically it should be the same, but it's always safer to check directly for NULL?

[LoneGazebo]

@ilteroi you can do that pretty easily- grab the current 'official' version and replace (6) EUI compatibility with it: https://mega.nz/#!zV9SkJqB!r-ZfpBs_Oyq4Is2-trvDSUqC9ZsOTn3GMHCyDfjBC78

[Infixo]

To test anything you need a file that causes problems every time not sometimes. Otherwise you won’t know if lack of problems is a result of ’sometimes’ or the fix. Do we have such file?

[LoneGazebo]

@Infixo not a reliable one, no.

[Infixo]

Do you actually have any freeze/crash problems? Because I have none. Couldn’t even reproduce anything from 2 saves attached here.
@LoneGazebo You tracked something yesterday. Was it something else?

[LoneGazebo]

@Infixo I'm not sure, I haven't been able to replicate since yesterday.

[ricanuck]

Got the barbarian freeze yesterday, 26-1, ONLY playing CP core plus EUI 1.28g (which never presented a problem). So you can discard VP extra components, the bug is part of the core CP.
I am convinced the bug is inside Infixo's changes, but I have no proof (call it dev intuition if you want, or ignore it... your call).

Edit: LUAs I use from the CP core that do not conflict with EUI:

5\MODS(1) Community Patch\LUA

27/01/2018 05:53 PM

.
27/01/2018 05:53 PM ..
08/10/2015 11:42 AM 10,324 Bombardment.lua
23/01/2017 04:44 PM 92,008 CultureOverview.lua
09/10/2015 08:10 PM 155,102 CultureOverview.xml
02/06/2015 05:46 PM 33,228 DeclareWarPopup.lua
02/06/2015 08:27 PM 14,902 DeclareWarPopup.xml
15/07/2015 06:20 PM 21,863 Demographics.lua
28/10/2016 03:39 PM 66,216 DiscussionDialog.lua
28/10/2016 03:39 PM 10,536 DiscussionDialog.xml
19/10/2016 12:11 AM 64,232 EspionageOverview.lua
02/12/2015 09:58 PM 37,909 EspionageOverview.xml
12/01/2018 03:04 PM 16,681 GPList.lua
09/09/2016 12:37 PM 5,456 NaturalWonderPopup.lua
24/02/2016 07:53 PM 5,992 PuppetCityPopup.lua
28/07/2017 12:11 PM 29,635 ReligionOverview.lua
17/11/2015 06:21 PM 17,483 TradeRouteHelpers.lua

[LoneGazebo]

@ricanuck I don't think Infixo has touched any of those LUA files.

[Infixo]

@ricanuck Please provide a save file.

I am working with (6a) EUI and (6b) Non-EUI components plus (6c) as copy. No Lua changes in core CP.
Even Civilopedia is unchanged in CP.

Also, none of my changes (but one) contain any loops. And that exception was made on Jan 8th to a function that is literally never called because VP doesn't use OrPreReqTechs (findPathLength, I made it to fix Wonder Planner).

[LoneGazebo]

@Infixo @ilteroi i'm leaning towards it being a bug in the functions mentioned above

getNumUnitsOfAIType
getFirstUnitOfAITypeSameTeam
getFirstUnitOfAITypeOtherTeam

or in the rework of the tacticalmap zones. I don't really see anything else in the DLL that could cause it.

[Infixo]

Yeah, I "vote" for getFirstUnitOfAITypeSameTeam. It is the only function called around barbarians that was changed in 1-21.

CvUnit* pRescueUnit = pNewTarget->getFirstUnitOfAITypeSameTeam(BARBARIAN_TEAM, UNITAI_SETTLE);
if (!pRescueUnit)
pRescueUnit = pNewTarget->getFirstUnitOfAITypeSameTeam(BARBARIAN_TEAM, UNITAI_WORKER);
if (pRescueUnit)
m_iUnitToRescue = pRescueUnit->GetID();

I have no idea WHY it could cause problems however.
Except that NULL checks. Also, ALL other while loops in CvPlot are constructed as if (cond) do...while(cond) Then again - idk why it would matter, but these are differences.

[Infixo]

And while we're at this, there is a minor bug, 2nd call should be UNITAI_WORKER:

// bonus for captured civilians (settlers and workers, not missionaries)
int iBonus = pPlot->getNumUnitsOfAIType(UNITAI_SETTLE) * 3 + pPlot->getNumUnitsOfAIType(UNITAI_SETTLE) * 2;

[LoneGazebo]

Yup re: typo in second one. The thing is, the 'barbarian turn' thing is misleading - stuff that happens on the global/game level takes place during that phase as well, so it's not technically barbarian. It's either something global or something related to the human player's turn start.

[ricanuck]

I am continuing the same game now, just reloaded from the last save yesterday, 50 turns and nothing happened, could not reproduce the freeze of last night... this is one of "those", nasty mfing hard to catch buggies... so no savegame.
OK, so if not LUA, last changes to DLL... again, I'm using Core only.

[tu-79]

To add something. G is rigth, it occurs during Barbarian turn, but at the moment there wasn't any barbarians, unless some city state was suffering an invassion I wasn't aware of.

[ilteroi]

ok guys i see a lot of speculation here. let's recap:

• last known good version was 1-14
• it's a freeze, callstacks as far as available did not point to modded code
• probably a deadlock, not an infinite loop (check cpu load next time)
• it's not reproducable
• it seems to occur during turn change
• it occurs without additional mods

correct?

what about EUI? do we have any feedback from non-EUI players?

anyway, it seems we need more testing. the fastest way would be to create multiple versions and ask people on the forums to test them and report back. to minimize confusion, it's probably best to have only two versions and iterate if necessary:

• DLL 1/14 with latest LUA
• DLL 1/26 with old LUA

[MoiMagnus]

I would add that some players reported a correlation with auto-save.
(ex: https://www.reddit.com/r/civvoxpopuli/comments/7tffig/game_crashes_every_10_turns_during_autosave/).
But it might be a different issue than this one.

[Infixo]

The post from bigcat88 strongly suggests that 1-14 was ok and 1-21 was not.
https://forums.civfanatics.com/threads/new-beta-version-january-26th-1-26.627497/page-4#post-15011166

[Infixo]

@ilteroi Can you explain deadlock using an example? Are they easy to happen in Civ5 code?

[bigcat88]

It is a deadlock. I cant press alt+tab, only can open task manager or windows button. No CPU usage after that.

[ilteroi]

@Infixo : a deadlock is a common multithreading pitfall, when two threads are a waiting for each other endlessly. they are nasty because they are often latent and become manifest only later after subtle timing changes in completely unrelated code.

so it's no use trying to find a bug in our modded code. we can only try to find the change that triggered the deadlock and build a workaround.

[LoneGazebo]

@ilteroi there were some unit promotions added by another user in 1-14 - mind taking a peek and seeing if the code is okay? I’m away all day.

[Infixo]

:) I am quite familiar with theory and was asking about Civ5 situation. I am not familiar with the code as you, so basically e.g. which parts are multithreaded? Is it only engine or the DLL as well? what internal ‚resource’ or operation could possibly trigger deadlock? Etc. Basically things specific to Civ5.

[LoneGazebo]

The UI is threaded on its own thread relative to the game DLL. So if there are any routines being called by the UI that break this link it can cause problems.

[LoneGazebo]

Even if the lua isn’t changed, you did modify a bunch of vanilla lua hooks in the dll that would affect this.

[MoiMagnus]

Have you example were the freeze appeared out of an autosave ?
The user in (ex: https://www.reddit.com/r/civvoxpopuli/comments/7tffig/game_crashes_every_10_turns_during_autosave/) seemed to only have freeze during autosave.

(And has the same settup : VP with EUI and no other mods)

[Infixo]

People are actually saying that autosave is done correctly, they can reload and continue. It has to be something done directly after autosave, like end-turn things or preparation for the next turn, when „barbarian turn” icon is still on the screen.

[MoiMagnus]

People are actually saying that autosave is done correctly, they can reload and continue. It has to be something done directly after autosave, like end-turn things or preparation for the next turn, when „barbarian turn” icon is still on the screen.

Yes, the user was saying that it was succeeding, but that it only froze turns of autosave. (but it might be a biais of observation)

The fact that the autosave succeed does not mean that it is not guilty. It can be a failure of the "autosave" to give back the hand to the main thread after finishing the save. (In case auto-save also have its own thread)

[HungryForFood]

Are you still looking for a save file? I have one where all you need to do is to click next turn, and the only save game incompatible mod was Historical Religions Complete.

An observation I had was that it did not freeze every time. However, if I moved the camera around and spam clicked the ground after clicking next turn, the freeze was much more likely.

[Infixo]

Can we post a debug version with AI-perf.csv log enabled and info from OutputDebugString (not sure if the latter is possible)? It would give more info about how far the game progresses and where it stops.

@HungryForFood I would like one. I have not yet had the freeze on my comp.

[ilteroi]

ah, ok. basically there are two main threads in civ, on is the UI thread and the other is the GameCore (DLL) thread. the problem is that all lua is executed by the UI thread, but it does make calls into the DLL, which is not threadsafe at all (except for a few places, check calls to GetGameCoreLock / ReleaseGameCoreLock)

however, there are most definitely other locks used in the 3D engine and so on, so this is not the only place where you can run into problems.

[bigcat88]

can u get and use EUI from 1-14 version? if yes - which folders replace? if no - I will try no eui version for test but it will be pain for me :)

[Infixo]

@bigcat88 The problem happens even when using only (1) CP.
Here are the files:
(6a) from VP 1-14.zip

[bigcat88]

Maybe there are 2 different problems. Even when I turn autosaves off, it is not ok sometimes. I just want to test first with old eui, then if all is still bad - without eui. Just be sure that problem is in dll.
thanks - will continue my hanging game and write if it will hang/crash

[LoneGazebo]

3919

[LoneGazebo]

Is JFD's comment at end a confirmation that it is the LUA? I think it might be. Anyone else with freezes getting errors in their lua.logs?

[bigcat88]

No, just look after freeze at lua.log - no errors. My last 4 lines is usual:
[708856.921] TurnProcessing: Hiding TurnProcessing
[708933.718] WonderRace: Oda Nobunaga is building Taj Mahal in Kyoto (4 turns)
[708943.906] WonderRace: Nebuchadnezzar II is building Taj Mahal in Babylon (5 turns)
[708943.906] WonderRace: Notification fired: [COLOR_RED]Nebuchadnezzar II[ENDCOLOR] has begun constructing Taj Mahal in Babylon

I am in progress of looking what causes this...

[Infixo]

@bigcat88 Can you post the save file that causes the freeze?

[bigcat88]

I trying to find the turn in game where it will cause a freeze every time or maybe place where will be 1 freeze every 5-6 turns, to test changed eui/old eui/no eui. And it is a problem. I even set autosaving on each turn to catch them. Maybe I need start a new game just for test without random seed. In current game I can play for 90 minutes without any freezes and then 2 freezes on 2 turns. Still cant find what causes this(after last freeze I turned off WonderRace). Patient :)

[HungryForFood]

@Infixo Here's the save. It doesn't happen every time, but you can move the camera around after clicking next turn to make it more likely. And the logs too if you're interested.

My mod list in the order the game displays, all of which except VP and Historical Religions Complete claims to be save-game compatible. I managed to recreate the freeze when using just VP and HR Complete to load.

(1) Community Patch (v. 88)
Historical Religion Complete (BNW or GK) (v. 45)
Ingame Editor (v. 39)
Quick Turns (v. 10)
Tectonic Map Script (v. 4)
(2) Community Balance Overhaul (v. 13)
(4) C4DF - CBP (v. 11)
(3) City-State Diplomacy Mod for CBP (v. 27)
(5) More Luxuries - CBO Edition (5-14b) (v. 155)
(6a) Community Balance Overhaul - Compatibility Files (EUI) (v. 1)
UI - Show XP In Military Overview (v. 2)

I'm using the 1/26 version of VP. Autosave was on and set to every turn. I was playing in windowed mode.

[LoneGazebo]

Loaded it and trying it now.

[LoneGazebo]

CvGameCore_Expansion2.zip

Test for me please. The slight hang at the end of the barb turn is your game autosaving. Otherwise it should be working. IF this works...the bug was my fault. If it doesn't help...someone else is to blame! :)

[HungryForFood]

Nope, freeze is still there. What do you mean by slight hang though, did it proceed to next turn without having to relaunch Civ 5?

Curiously the AI turn went a bit differently, Pocatello declared friendship even though he didn't before. Does swapping the DLL like this affect the RNG somehow?

I was playing the game while waiting for your reply, and from my observation, it never froze as long as I do nothing while the AI turns are processing. No camera movement, no clicking, etc.

Same goes for the save. No camera move = no freeze. Violently moving the camera = freeze.

[LoneGazebo]

No freeze for me period if autosave is set to 1 per 10 turns.

[Albertolona]

@Infixo

Have game settings been eliminated?

There was a problem before where animations where causing problems and it wasn't being reproduced since modders were running with it switched off, so perhaps try with different settings.

[HungryForFood]

So I turned off autosave and loaded my save again, and I couldn't get the freeze.

Yep, definitely something to do with autosave. I turned it back on, and can trigger the freeze again.

[LoneGazebo]

@ilteroi any possibility that the method by which your new tacticalmapanalysis stuff is saved is causing an issue with saving/loading the game? I noticed it's a new kstream format. Twice I loaded hungry's game and it froze on me while loading. That's definitely new.

[ilteroi]

what do you mean? i didn't change the savegame format (afaik)

also, when it freezes, what does the callstack say?

[Infixo]

@Albertolona Which settings do you mean? Quick movement / quick combat? Something else?

[Infixo]

@HungryForFood Why do you have like a half a million of these in GameCore.log:
[616915.218] LOAD ERROR: Type not found for hash: 4294967295

Edit. The timestamp of the file is much earlier, so probably from a different session. But weird, anyway.

[HungryForFood]

I have no idea what it means. Nothing seems off with my game.

[ryanmusante]

For me, to trigger a freeze is to "_quick-save_" and than immediately click "_exit to desktop_" from the menu. I don't know if this is because of my wine+steam configuration or because of the aforementioned bug.

[bigcat88]

Hm, played for 7 hours. Got two crashes(not repeatable, looks like crash happens after autosave. first crash 100% was after autosave, now will start civ and look - was game saved or not after second)(not freezes).
And first crash was as I remember invalid eip register(I have no debug symbols for civ5 so cant say more, and it is just a minidump). Second the same, but in stack there present "right" address of return(so corrupted stack by 1 parameter?) that points to civ5 code(u will see if load in windbg).
Attach 2 dumps, second dump was last. Logs folder only for second dump, I forgot to save log folder for first crash.

Logs2.zip

Dumps.zip

[bigcat88]

But if look at message_debug log that lines tells me that autosave was present. And I looked in autosave folder, there is present autosaves with time creation equal to dump file.
[766123.843] DBG: UpdateMoves() : player 63 Crom running AutoMission (AI_TACTMOVE) on Lancer id=15309
[766123.843] DBG: CheckPlayerTurnDeactivate() : auto-moves complete for Crom
[766123.843] DBG: changeNumGameTurnActive(-1) m_iNumActive=0 : setTurnActive() for player Crom
[766128.953] DBG: Game Turn 516
[766128.968] DBG: changeNumGameTurnActive(1) m_iNumActive=1 : setTurnActive() for player 0 Catherine

@Gazebo Can you post a debug symbols(pdb file) for dll(1-26)(if u have it) and say where I can download debug symbols for CivilizationV_DX11 module(if Firaxis provide them) Just interesting what is that function. I never compiled civ5 dll, and I am too lazy to run VM with studio to do that(and I am not sure that I have the same studio version, and if not I need to install new vm for specified version)

[Infixo]

@LoneGazebo How can I make a debug version, i.e. not FINAL_RELEASE. I removed FINAL_RELEASE from preproc defines and got 135 compile errors, all from CvAsserts. Wrong variable names, wrong function names, non-existing variables, etc. I've fixed them and still cannot create a .dll - linker says there are 4 missing external symbols.

[LoneGazebo]

Don’t undefine final release like that. Debug/release is set by the compiler. There’s just s drop down toggle in VS.

[LoneGazebo]

@ilteroi you are correct I was thinking of something from earlier. There is no change to kstream in tactical analysis map.

@infixo try disabling the trade route length UI functionality. The trade system code is very fragile and it might be the cause of tyis.

[Infixo]

@LoneGazebo TR length? I've added GetTradeRouteTurns(), and it was included in 1-14.
Re: debug What drop down? I am checking project properties and I have only Mod and Mod2.
Also, those asserts have usually simple typo errors. They are not "compilable" no matter the settings. It seems that simply nobody ever tried to compile them actually. In normal settings preprocessor changes them into (void)0.

[ilteroi]

if you're not using FINAL_RELEASE, you need to link against libraries which aren't included in the SDK from Firaxis. but it doesn't matter, simply use DEBUG and RELEASE like normal people do :)

and yes: there are too many compile flags and 98% of them don't work or break the build or cause hair loss or a combination thereof.

[Infixo]

My observations.

1. The save file from @HungryForFood creates freeze every single time I use it. I have 2-turn autosaves and just run auto-play.
2. The debugger shows ntdll.dll!774ff901. Every single time.
3. The freeze doesn't happen when I don't do anything with cursor. I run in Windowed mode, when the cursor is off the window - the game plays. When I move the cursor into the window and start moving around just displaying tooltips from cities and tiles (NOT clicking anything NOR moving camera), it freezes in a usual place.
4. I've enabled ai-perf.log and added some outputs. It shows:

, 3, CvPlayer::doTurn(), Turn 168, Barbarians,,,, 0.006118
, 2, CheckPlayerTurnDeactivate(), Turn 168, Monaco: activate next player,,, 0.016512
, 1, CheckPlayerTurnDeactivate(), Turn 168, processing player Monaco,, 0.016823
, 1, CheckPlayerTurnDeactivate(), Turn 168, processing player Crom,, 0.000002
, 1, CvGame::update(), Turn 168, finished CheckPlayerTurnDeactivate(),, 0.000000
, 0, CvGame::update(), Turn 168, finished updateMoves(), 0.017004
, 0, Tactical Analysis Map, Turn 168, Barbarians, 0.022543
, 0, Tactical AI, Turn 168, Barbarians, 0.142012
, 0, Homeland AI, Turn 168, Barbarians, 0.000000
, 0, AI_unitUpdate, Turn 168, finished AI HomelandAI update, 0.000000
, 0, updateMoves(), Turn 168, AI_unitUpdate() called for player Crom, 0.000037
, 1, CvGame::update(), Turn 168, finished UpdatePlayers(),, 0.000000
, 2, CheckPlayerTurnDeactivate(), Turn 168, Crom: activate next player,,, 0.000006
, 1, CheckPlayerTurnDeactivate(), Turn 168, processing player Crom,, 0.000918
, 1, CvGame::update(), Turn 168, finished CheckPlayerTurnDeactivate(),, 0.000000
, 0, CvGame::update(), Turn 168, finished updateMoves(), 0.001078

When the game progresses the log shows next:

, 0, AI_unitUpdate, Turn 167, finished AI HomelandAI update, 0.000000
, 0, updateMoves(), Turn 167, AI_unitUpdate() called for player Crom, 0.000038
, 1, CvGame::update(), Turn 167, finished UpdatePlayers(),, 0.000000
, 2, CheckPlayerTurnDeactivate(), Turn 167, Crom: activate next player,,, 0.000011
, 1, CheckPlayerTurnDeactivate(), Turn 167, processing player Crom,, 0.001344
, 1, CvGame::update(), Turn 167, finished CheckPlayerTurnDeactivate(),, 0.000000
, 0, CvGame::update(), Turn 167, finished updateMoves(), 0.001511
  << this is not executed >>
, 0, Connections/Gold, Turn 168, Byzantium, 0.001019
, 2, AI_ considerAnnex,,, 0.000090
, 1, CvPlayerAI::AI_doTurnPre, Turn 168, Byzantium,, 0.000118
, 1, Tactical Analysis Map, Turn 168, Byzantium,, 0.004144

[ilteroi]

finally some progress!

but i have a hard time parsing this ... can you add some more details?

also why does the second log snippet refer turn 167 when it should be "progressing" after turn 168?

[Infixo]

Because on 168 it stopped. The previous 7 turns look exactly the same, just different turn number.

This part of code is like this (CvGame::Update()):

updateMoves();
AI_PERF_FORMAT("AI-perf.csv", ("CvGame::update(), Turn %03d, finished updateMoves()", GC.getGame().getElapsedGameTurns()));
if(!isPaused()) // And again, the player can change after the automoves and that can pause the game
{
    updateTimers();
    UpdatePlayers(); // slewis added!
    { AI_PERF_FORMAT("AI-perf.csv", ("CvGame::update(), Turn %03d, finished UpdatePlayers()", GC.getGame().getElapsedGameTurns())); }

Basically after performing the entire player Turn, the log should contain ", 1, CvGame::update(), Turn 168, finished UpdatePlayers(),, 0.000000". But the game ofc doesn't go that far. "Connections/Gold" is just a first of many other lines, which is also not reached.

[LoneGazebo]

@infixo glad you could get it to freeze. I never could.

[LoneGazebo]

Also did you use the debug dll I posted above?

[Infixo]

@LoneGazebo No, normal 1-26 with ai-perf enabled. I will try out the one you posted.

[Infixo]

Ok, this is the long shot, but I found something in tactical zones.

CvTacticalAnalysisMap::AddNewDominanceZone(CvTacticalDominanceZone& zone)
m_IdLookup[zone.GetZoneID()] = m_DominanceZones.size();
m_DominanceZones.push_back(zone);
return &m_DominanceZones.back();

CvTacticalAnalysisMap::UpdateZoneIds()
m_IdLookup.clear();
for (size_t i = 0; i < m_DominanceZones.size(); i++)
m_IdLookup[m_DominanceZones[i].GetZoneID()] = i;

~Lets say size() is 5. In the first function we assign size(), so basically there is going to be a map entry that points to 5. All entries will point to 1..5, when adding consecutive zones. In the 2nd function that is (I think) recreating the mapping, we never assign size(). It will point to 4. So the mapping will be to 0..4.
Idk if there could be any consequences at all.~

Edit. Ok, push_back() increases size(), so size() is 4 in the 1st case actually. Scratch that.

[Infixo]

@LoneGazebo Tested with your DLL. Still freezes, in the same place.
Curious thing is that the freeze doesn't happen immediately. The save starts at turn 161 and it usually freezes on 167. I got once 168. But 6 turns go smoothly, even with cursor movements, clicking, showing tooltips, etc. And then suddenly it freezes. Also, all 7 auto-saves are created (I changed to 1-turn to see if there's any difference).
It's as if something was building up, counters, memory problems, whatever.

[LoneGazebo]

Do you have VerySleepy CS installed? Install it, hook it into the .exe of the game as soon as you hit the next turn button on a turn you know will freeze, let it run while the game is frozen, then end the test and look at the cycle results.

[Infixo]

No, I don't have that tool. I'll try it out.
In the meantime, pls. tell me exactly what normal people do to get a debug DLL. I undef'ed NDEBUG and defined _DEBUG - still getting linker errors (2 missing externals).

[LoneGazebo]

[LoneGazebo]

See that 'Release' drop down menu toggle? Set it to debug.

G

[Infixo]

I don't have "Release" over there but only "Mod". And "Configuration Manager...".
I am using VS Express 2012 and project files from master, unmodified.

[LoneGazebo]

a.) Get VS 2013 Express
b.) https://msdn.microsoft.com/en-us/library/wx0123s5.aspx

[Infixo]

VerySleepy results. I started on turn 165 and ran VS for ~40 seconds. The game froze at turn 166 this time, afer ~15 sec. So the results are ~15sec of the gameplay and ~25sec of the freeze.

freeze 40s gameplay.zip

[Infixo]

Lots of data. Why some functions are in green color? Any advise on what to look for?

[LoneGazebo]

I think I see it. Sort by inclusive to find loops - higher .s count = more cycle time.

[LoneGazebo]

CvGameCore_Expansion2.zip

[LoneGazebo]

try this DLL.

[Infixo]

Minor events in GetYieldPerTurnFromMinors?

[LoneGazebo]

Yup.

[LoneGazebo]

For some reason those hooks are misbehaving. Not sure why, I've put dozens of hooks in the DLL. I'm going to rewrite those functions for JFD and just give him an array to store the values in, instead of trying to do it with hooks.

[LoneGazebo]

It'll break savegames, so test for me (I just set it to return 0 instead of looping) to see if it actually fixes the freezes.

[Infixo]

IT'S WORKING!
I've auto-played 20 turns, doing all crazy cursor moves, including opening CityView, Civilopedia, etc. No freeze, no crash, game behaves normally.
Edit. Ran also 2nd time to make sure it is not some weird luck. Still working good.

[LoneGazebo]

Okay. I must have overloaded the lua hook method. Blegh. That’s new. Everyone else is off the hook. I’m the baddie.

In the meantime turning off BALANCE_CORE_JFD in the custommodoptions.xml is a workaround. I’m away but that could be useful information for the forum. @jfdaly I’ll be reworking that lua function for you.

[Infixo]

This VerySleepy is a freaking awesome tool. We should've started with that :)

[JFDaly]

Alrighty.

[LoneGazebo]

I have been, but I couldn’t get it to freeze!

[ryanmusante]

This is an epic thread. I've felt emotional highs and lows during the past seven days reading it.

[tu-79]

And my email has never been so busy... was watching the thread.
Glad you did it.