Tmpe: Stuck cims

[aubergine10]

Got a train station that regularly gets hoards of stuck cims:

Reset stuck cims (via mod options > maintenance tab) sets them free:

Would be nice if we could work out what's causing it.

Will add savegame and logs later.

[aubergine10]

Save: https://steamcommunity.com/sharedfiles/filedetails/?id=2056376792

[ninjamods]

Savegame: https://steamcommunity.com/sharedfiles/filedetails/?id=2057580865

Error: System.NullReferenceException: Object reference not set to an instance of an object
at TrafficManager.Manager.Impl.ExtCitizenInstanceManager.StartPathFind (uint16,CitizenInstance+,TrafficManager.API.Traffic.Data.ExtCitizenInstance+,TrafficManager.API.Traffic.Data.ExtCitizen+,UnityEngine.Vector3,UnityEngine.Vector3,VehicleInfo,bool,bool) <0x00c16>
at TrafficManager.Custom.AI.CustomCitizenAI.CustomStartPathFind (uint16,CitizenInstance+,UnityEngine.Vector3,UnityEngine.Vector3,VehicleInfo,bool,bool) <0x00146>
at ResidentAI.StartPathFind (uint16,CitizenInstance+) <0x007d5>
at ResidentAI.SetTarget (uint16,CitizenInstance+,uint16,bool) <0x0096c>
at (wrapper dynamic-method) HumanAI.ArriveAtDestination_Patch1 (object,uint16,CitizenInstance+,bool) <0x0020e>
at (wrapper dynamic-method) HumanAI.ArriveAtTarget_Patch1 (object,uint16,CitizenInstance+) <0x000b3>
at (wrapper dynamic-method) HumanAI.SimulationStep_Patch0 (object,uint16,CitizenInstance+,CitizenInstance/Frame+,bool) <0x02013>
at (wrapper dynamic-method) ResidentAI.SimulationStep_Patch1 (object,uint16,CitizenInstance+,CitizenInstance/Frame+,bool) <0x0032a>
at CitizenAI.SimulationStep (uint16,CitizenInstance+,UnityEngine.Vector3) <0x001f9>
at TrafficManager.Custom.AI.CustomHumanAI.CustomSimulationStep (uint16,CitizenInstance+,UnityEngine.Vector3) <0x00649>
at (wrapper dynamic-method) CitizenManager.SimulationStepImpl_Patch0 (object,int) <0x0047b>
at SimulationManagerBase`2.SimulationStep (int) <0x0004c>
at CitizenManager.ISimulationManager.SimulationStep (int) <0x00024>
at SimulationManager.SimulationStep () <0x00693>
at SimulationManager.SimulationThread () <0x0018a>

TMPE.log
output_log.txt

[aubergine10]

@ninjamods That error isn't in the logs you attached - run the game again until error occurs and attach the log files at that point

[krzychu124]

Hi @ninjamods your logs are "empty". It looks like you attached logs after you shared savegame. You should load savegame, wait/provoke error then close game and collect logs. Log files are always cleaned every time you start the game :/
We will check savegame anyways ;)

Edit: I forgot to click _Add comment_ lol
Edit2: Loading savegame might be painfull [>10k assets]

[aubergine10]

Note to self: Check title for typos when creating issues lol

[aubergine10]

Here's my logs btw, for save linked in https://github.com/CitiesSkylinesMods/TMPE/issues/825#issuecomment-612318559:

output_log.txt
TMPE.log

Notably they were all waiting for path, but it's only tiny population (about 5k cims in the city) and there was no buildup of PFs that I could see:

Info 564.5636497: UtilityManager.RemoveStuckEntities() called.
Info 564.5655368: UtilityManager.RemoveStuckEntities(): Waiting for all paths.
Info 564.5663157: UtilityManager.RemoveStuckEntities(): Resetting citizen instances that are waiting for a path.
Info 564.5668581: Resetting stuck citizen instance 20 (waiting for path)
Info 564.5674219: Resetting stuck citizen instance 2017 (waiting for path)
Info 564.5679532: Resetting stuck citizen instance 2101 (waiting for path)
Info 564.5684711: Resetting stuck citizen instance 2170 (waiting for path)
Info 564.5689951: Resetting stuck citizen instance 2225 (waiting for path)
etc...

Also, I've tried at least three different train stations in that spot and all had the same issue so it's unlikely to be asset-caused (unless they were all made from same template and that template has some issue?).

[ninjamods]

Reuploaded! This error happens at start for this game every time, so I'm pretty sure it's the same one.

TMPE.log
output_log.txt

[ninjamods]

TMPE.log
output_log.txt

More detailed logs attached!

[aubergine10]

Narrowed down to this block of code:

            Citizen[] citizensBuffer = Singleton<CitizenManager>.instance.m_citizens.m_buffer;
            ushort parkedVehicleId = citizensBuffer[instanceData.m_citizen].m_parkedVehicle;
            ushort homeId = citizensBuffer[instanceData.m_citizen].m_homeBuilding;
            CarUsagePolicy carUsageMode = CarUsagePolicy.Allowed;
            var startsAtOutsideConnection = false;
            ParkingAI parkingAiConf = GlobalConfig.Instance.ParkingAI;

[krzychu124]

@aubergine10 instanceData can be null there (missing asset?).
Look at log:

Info 676.6934940: Loading Data from New Load Routine! Length=4161932
Info 694.3635829: Loading State from Config
Info 694.3647886: Loading 46834 extended citizens
Info 694.4278915: Loading 57528 extended citizen instances

[krzychu124]

Ok, it looks like incorrectly loaded building(missing subscription/removed from workshop)?

Building info is always set when asset is correctly loaded. @aubergine10 I think we can add nullcheck there and log that situation. There is no info about that asset used by this instance of building because info was not set, obviously

[aubergine10]

@krzychu124 What should happen if null though? return false or Reset(ref extInstance)?

[krzychu124]

Good question :D I'll check what is going on there, maybe we could set path as invalid (soft/hard).

[krzychu124]

I tested your savegame @aubergine10 and I noticed similar behaviour when I was debugging broken nodes _(AKA Luke, vehicles are despawning)_. This time m_path of citizen is constantly changing unlike m_path of vehicle like before. Have you tried broken nodes detector?

It's weird because it seems that path can be found (pf success counter rapidly goes up -> x thousands per minute, it's not normal, btw) but it is invalidated for some reason before citizen start moving.

[ninjamods]

Interesting, do I use this: https://steamcommunity.com/sharedfiles/filedetails/?id=1777173984&searchtext=broken+node to test for broken nodes?

[krzychu124]

Yeah that one, but your error was already fixed in Labs and that error seem to be completely unrelated to "stuck cims" issue ;)
I'll update Stable today.

[aubergine10]

I've run Broken Nodes detector multiple times, it found nothing. Also ran the broken PT checker and ghost nodes checker to no avail.

Is there somewhere we can determine why the paths are being invalidated?

[krzychu124]

It will be pita but I have to follow call stack :/

[aubergine10]

Does BND scan only transit networks? Could it be some other type of network causing the issue (quays, etc)? Since SH quays can now have ped paths.

[aubergine10]

So, thanks to Krzychu's investigations, it turns out my stuck cims problem is most likely the result of an ancient Viking settlement under my train station

Image from krzychu:

Looks like Move It, Anarchy or some other mod (or asset) is leaving building intestines scattered around the map.

[aubergine10]

fuuukkk....

Like, there's an entire era of human history buried in them thar hills...

[aubergine10]

When zooming in to the overlay, a lot of the text moves over actual (wanted/proper) segments/nodes.

Part of the issue debugging this is the overlay is just splatting stuff everywhere. I think it might be easier to debug if we had better tooling; will create new issue for that. EDIT: #844

[TesseractToby]

@aubergine10 - I can add another save game to the list with this same issue (I believe).
My PFs is up to 1.5k and rising, grinding everything to a halt, but I've not been able to find anywhere like your train station which may be causing this specific issue.
Included is my output_log and tmpe logs following a search for broken nodes/lines and a reset of stuck sims (which was performed after saving the game, I've left it with the issue growing for review).

Tesseract_TMPE.log
Tesseract_output_log.txt

Save game can be found at: https://steamcommunity.com/sharedfiles/filedetails/?id=2066275717

Edit I found the transport building having the issues. In the save attached, it's the Intercity Bus Station at Elm Hills district. Disabling that building causes the PFs to drop to normal levels.

[krzychu124]

@TesseractToby it seems that vanilla also can't handle that situation properly, so it might be game bug 😕
@aubergine10 I've noticed something weird. Why almost all stuck cims have dog??

[aubergine10]

I think dogs just stand out from the crowd so we notice them more.

I've pretty much confirmed the cause of the ancient viking villages... It's due to Move It mod being used on any station that has underground paths/networks.

I'm not sure if it's direct result of moving the building, or side-effet - eg. when building is bulldozed, the bulldozer might not be recognising the viking ruins as belonging to the building.

[krzychu124]

I think that pedestrian path nodes have to be at specific distance to PF sees them as connected. Maybe you moved only visible nodes and segments, but those invisible are still at original position and PF followed internal path without checking connections?

[aubergine10]

Maybe you moved only visible nodes and segments, but those invisible are still at original position and PF followed internal path without checking connections?

Specifically it was the _station building_ I moved. It seems Move It mod isn't properly moving _some_ paths that are internal (or otherwise part of) that station. I'm not sure if the station has sub-buildings that Move It forgets to move or something like that?

I used @kianzarrin network detective mod and all of the problematic segments are ped paths.

I suspect a side-effect of this issue, particularly for detailers, is consumption of the node/segment/lane limits, because they'll have vast numbers of these "viking ruins" lurking under their city considering how much they use Move It mod.

I'm still not quite sure if the Move It bug is limited to specific vs. all train stations. Maybe just those with certain kinds of internal path? Maybe it's something to do with sub-buildings / or path-containing props? While I've only noticed it with train stations, it could potentially be any kind of building containing internal paths.

I think that pedestrian path nodes have to be at specific distance to PF sees them as connected.

For ped paths, this is certainly the case. If within certain proximity to another path/node they seem to auto-connect somehow.

I'm not sure how it would break pathfinder though - if it thinks they're valid it will include them in the path, cims should still walk along them. Maybe it's the AI that's getting confused? Maybe there is some special case for Pedestrian Connection Inside segments?

I'm not sure how we could ever auto-detect the problem segments. While they have some obvious characteristics, there doesn't seem to be any reliable way to determine if they are associated with a building or not. Looking at the structs in ILSpy, I don't see any obvious reference to a 'parent building' (maybe there is one I didn't spot?) - my guess is that once plopped, the game doesn't really care. Although if that were the case, how would bulldozing the building clear up the internal paths? There must be some sort of linkage somewhere? If there is a linkage, maybe that's what's confusing the pathfinder/AI?

[aubergine10]

Just noticed another situation, this time with "internal roads" of a ferry harbor:

Note the road has two sets of nodes/segments:

I replaced the original road (part of the harbor building) with a different road using Klyte45's Touch This mod. It seems that the nodes/segments of the original road are retained, so it's like two roads on top of each other.

Using Alt+Click/Drag (Move It mod) to select the node of the replacement road and drag it:

As you can see, there is a 'ghost segment' that I assume in the original road. Somehow the road mesh isn't showing but the segment is still there.

I suspect this too will cause issues for pathfinder/AI.

[krzychu124]

This issue: https://github.com/CitiesSkylinesMods/TMPE/issues/825#issuecomment-616860983 is caused by ghost nodes/segments there. Probably something happened and intercity bus station was not moved correctly leaving some nods and segments confusing path finder. Basically, path finder created invalid path -> it should not connect disabled to not disabled node if they are not asset internal.

Similar issue (stuck cims) may happen on very wide roads (>64f). It's vanilla code.

Cim will request new path, he will get new one, but because of string max distance value _connection not longer than 64f_ (PF doesn't know about it) is invalidated by cim. This will create infinite loop.

For cyclists situation is a little bit better, because they also invalidate path but released bike is releasing also citizen(cim is not really cycling but bike is used by cim -> citizenAI is superior AI over bikeAI, unlike for other vehicleAIs where vehicle is managing citizen units inside)

I've increased that limit and didn't spot any issues, cims and cyclist can use very wide roads. No despawn or stuck occurred at a time of tests