CKAN stuck at loading available modules

[NicLars04]

Background

CKAN Version: 1.26.0

KSP Version: 1.4.3.2152

Operating System: Windows 10 x64

Have you made any manual changes to your GameData folder (i.e., not via CKAN)? -> Yes

Problem

What steps did you take in CKAN?
-> selecting 1.4 as new compatible KSP game version

What did you expect to happen?
-> Game change the settings to an other compatible mode

What happened instead?
-> CKAN (freez) stuck at loading available modules in Loading Modules

Screenshots:
->

[NicLars04]

Its loading like that (several times(>1)) over an hour

[DasSkelett]

Which version(s) did you mark as compatible?

[NicLars04]

marked compatible version: 1.4

[NicLars04]

a noteable thing to ad is that that occured right after letting it udate

[DasSkelett]

Does it work now?
Else try to delete the ckan.exe and replace it with a manually downloaded v1.26.

[randomguyingithub]

try to move ckan folder outside. It seems compatible data file doesn't work with new ckan(the ckan won't even appear for me)
then replace the registry with the old one and do things again
(not sure if this is correct but it fixed for me)

[NicLars04]

No, it doesn't work either way, i tried so far everything including new install.
But anyway (Ikd if this is interresting full), when it's at the step "loading available modules" it doesn't require any of my capacities and i cant controle it anymore (e.g.: fullscreen, closing), I have to close ist via the task manager.

i still dont know how to fix this, cause every attempt end in the same result, there each step seems the same.

[DasSkelett]

Could you you try to run ckan.exe update in the command line? See if that works.

Edit: Can you zip the registry.json and upload it here? I want to take a look at it.

[randomguyingithub]

Don't tell me it's the old pop-up behind ckan problem

[NicLars04]

This is the registry.json :
registry.zip

still with the "ckan.exe update" it wont work(it didnt changed anything).

btw Grüße aus Deutschland

[DasSkelett]

Grüße zurück!

The registry works perfectly fine for me, even if I don't refresh after replacing the existing one.
Neither can I spot something wrong.
Can you try the complete clean and reinstall process?

[NicLars04]

the question is how should i do it. Every time i'm runing CKAN it will automaticly go into this endless loading status.

[DasSkelett]

Oh, yeah...
Another command to try first: ckan.exe repair registry (set the --ksp flag if this KSP install is not you default).
After that (and if the first command doesn't work):
1) Go into the CKAN/history folder and search for a .ckan file which looks valid (a lot of mods listed under "depends"). As recent as possible. Copy that one into the directory where your ckan.exe is (and rename it to something easy if you want, like "mods.ckan").
2) Do only _Phase 2_ of the Clean and reinstall process.
3) Run ckan.exe install -c mods.ckan (or whatever you choose as a file name).
4) Perform _Phase 4_.

[jupiterbjy]

I'm having same issue, and looks even more awesome when performed clean install. This grey screen is how it looks like in first startup of CKAN after clean install. After force closing this not responding process, I now then have exact same problem with NicLars04.

I tried to fix my backed up CKAN's registry and result is:

It just crashes. After 1.26 update, few CKAN-installed mods like "DeployableEngiens" and "Wildblue Tools" are shown as AutoDetect and was removed from registry for unknown reason.

Performing clean install CKAN with v1.25.4 without update is also crashing with exception as following:

System.ArgumentOutOfRangeException: 인덱스 및 길이는 문자열 내의 위치를 참조해야 합니다.
매개 변수 이름: length (Rough translation: The index and length must refer to the position within the string. Parameter name: length)
   위치: System.String.Substring(Int32 startIndex, Int32 length)
   위치: System.Linq.Lookup`2.Create[TSource](IEnumerable`1 source, Func`2 keySelector, Func`2 elementSelector, IEqualityComparer`1 comparer)
   위치: System.Linq.GroupedEnumerable`3.GetEnumerator()
   위치: System.Linq.Enumerable.ToDictionary[TSource,TKey,TElement](IEnumerable`1 source, Func`2 keySelector, Func`2 elementSelector, IEqualityComparer`1 comparer)
   위치: System.Linq.Enumerable.ToDictionary[TSource,TKey,TElement](IEnumerable`1 source, Func`2 keySelector, Func`2 elementSelector)
   위치: CKAN.NetFileCache.GetCachedFilename(Uri url, Nullable`1 remoteTimestamp)
   위치: CKAN.NetFileCache.IsMaybeCachedZip(Uri url)
   위치: CKAN.GUIMod.UpdateIsCached()
   위치: CKAN.Main.<>c__DisplayClass253_0.<_UpdateModsList>b__1(CkanModule m)
   위치: System.Linq.Enumerable.WhereSelectListIterator`2.MoveNext()
   위치: System.Collections.Generic.HashSet`1.UnionWith(IEnumerable`1 other)
   위치: CKAN.Main._UpdateModsList(Boolean repo_updated, IEnumerable`1 mc)
   위치: CKAN.Main.PostUpdateRepo(Object sender, RunWorkerCompletedEventArgs e)
   위치: System.ComponentModel.BackgroundWorker.OnRunWorkerCompleted(RunWorkerCompletedEventArgs e)

From these results, I think CKAN is accessing some files outside of KSP folder or otherwise there's no reason for old version of CKAN to break after update. If so, tell us which files' being accessed.

[randomguyingithub]

I ran repair registry cmd, I got gravityturncontinued conflict with gravityturn error

[NicLars04]

exactly how jupiterbjy discribed, i have the same issues and same results as he presented.

[DasSkelett]

@NicLars04 did you try the second part of my comment?

1) Go into the CKAN/history folder and search for a .ckan file which looks valid (a lot of mods listed under "depends"), as recent as possible. Copy that one into the directory where your ckan.exe is (and rename it to something easy if you want, like "mods.ckan").
2) Do only _Phase 2_ of the Clean and reinstall process.
3) Run ckan.exe install -c mods.ckan (or whatever you choose for file name).
4) Perform _Phase 4_.

[NicLars04]

ehm... i tried but i am not that good in this cmd and programming stuf, so i am a bit strugelling how to do it. Could anyone explain it to me in more details please, i would realy appreciate it. 👍

(ps.: i dont understand how the end of step 2 shuold work, then i have all exept for the downloads deleted)

[NicLars04]

CKAN Version: v.1.25.4 ###

Ok, So i have just installed the older version and know i am getting new errors like from jupiterbjy. If all dont work, Is there any way to get it working in an older version?

[jupiterbjy]

FIXED IT:

Run regedit.exe, find CKAN Folder located under 'HKEY_CURRENT_USER\Software\CKAN' or search CKAN.

Remove that CKAN Folder itself, and launch CKAN regardless of it's version.

And BAM. It works. Tested on v1.26.0.

I'm quite surprised CKAN does use registry.. To developers, please make sure to deploy hotfix that remove/fix actual registery, not registry.json file.

Have a (un)safe flight everyone!

[DasSkelett]

@jupiterbjy thanks for the clue.
I suspect the KSPBuilds key to be the problematic one.

@NicLars04, scratch the reinstallation. Please try to delete the registry keys, one at a time.
1) Run regedit.exe (via Win+R combination, or search for it tín the start menu)
2) Navigate to HKEY_CURRENT_USER\Software\CKAN
3) Right click on KSPBuilds -> Delete
4) Run CKAN, see if it works.
5) If not, delete DownloadCacheDir and run CKAN again
7) Repeat with another entry in the registry until CKAN works again.
6) Please tell us which key got it to work for you again.

[jupiterbjy]

I still got v1.25.4 CKAN on Laptop with untouched registy, I'll give result in 1 - 2 hours.

FYI, it's safe to assume KSPBuild is cause because after CKAN regenerated all keys, all other keys are same, except install directory key which I moved last year and still not changed.

[jupiterbjy]

Data previous v1.26 (collapsed)

{
"builds": {
"464": "0.23.5.464",
"559": "0.24.2.559",
"642": "0.25.0.642",
"705": "0.90.0.705",
"830": "1.0.0.830",
"842": "1.0.2.842",
"861": "1.0.4.861",
"1024": "1.0.5.1024",
"1028": "1.0.5.1028",
"1172": "1.1.0.1172",
"1174": "1.1.0.1174",
"1180": "1.1.0.1180",
"1183": "1.1.0.1183",
"1196": "1.1.0.1196",
"1203": "1.1.0.1203",
"1209": "1.1.0.1209",
"1215": "1.1.0.1215",
"1224": "1.1.0.1224",
"1228": "1.1.0.1228",
"1230": "1.1.0.1230",
"1250": "1.1.1.1250",
"1260": "1.1.2.1260",
"1289": "1.1.3.1289",
"1473": "1.2.0.1473",
"1479": "1.2.0.1479",
"1485": "1.2.0.1485",
"1486": "1.2.0.1486",
"1489": "1.2.0.1489",
"1494": "1.2.0.1494",
"1499": "1.2.0.1499",
"1500": "1.2.0.1500",
"1509": "1.2.0.1509",
"1517": "1.2.0.1517",
"1520": "1.2.0.1520",
"1523": "1.2.0.1523",
"1532": "1.2.0.1532",
"1539": "1.2.0.1539",
"1540": "1.2.0.1540",
"1546": "1.2.0.1546",
"1548": "1.2.0.1548",
"1553": "1.2.0.1553",
"1563": "1.2.0.1563",
"1564": "1.2.0.1564",
"1569": "1.2.0.1569",
"1574": "1.2.0.1574",
"1576": "1.2.0.1576",
"1583": "1.2.0.1583",
"1584": "1.2.0.1584",
"1586": "1.2.0.1586",
"1604": "1.2.1.1604",
"1622": "1.2.2.1622",
"1727": "1.2.9.1727",
"1730": "1.2.9.1730",
"1737": "1.2.9.1737",
"1738": "1.2.9.1738",
"1743": "1.2.9.1743",
"1750": "1.2.9.1750",
"1758": "1.2.9.1758",
"1764": "1.2.9.1764",
"1773": "1.2.9.1773",
"1781": "1.2.9.1781",
"1790": "1.2.9.1790",
"1796": "1.2.9.1796",
"1800": "1.2.9.1800",
"1804": "1.3.0.1804",
"1836": "1.3.1.1836",
"1847": "1.3.1.1847",
"1855": "1.3.1.1855",
"1863": "1.3.1.1863",
"1891": "1.3.1.1891",
"2077": "1.4.0.2077",
"2089": "1.4.1.2089",
"2110": "1.4.2.2110",
"2152": "1.4.3.2152",
"2215": "1.4.4.2215",
"2243": "1.4.5.2243",
"2256": "1.4.5.2256",
"2332": "1.5.0.2332",
"2335": "1.5.1.2335",
"2395": "1.6.0.2395",
"2401": "1.6.1.2401"
}
}

Data after Update & Registry purge (collapsed)

{
"builds": {
"464": "0.23.5.464",
"559": "0.24.2.559",
"642": "0.25.0.642",
"705": "0.90.0.705",
"830": "1.0.0.830",
"842": "1.0.2.842",
"861": "1.0.4.861",
"1024": "1.0.5.1024",
"1028": "1.0.5.1028",
"1172": "1.1.0.1172",
"1174": "1.1.0.1174",
"1180": "1.1.0.1180",
"1183": "1.1.0.1183",
"1196": "1.1.0.1196",
"1203": "1.1.0.1203",
"1209": "1.1.0.1209",
"1215": "1.1.0.1215",
"1224": "1.1.0.1224",
"1228": "1.1.0.1228",
"1230": "1.1.0.1230",
"1250": "1.1.1.1250",
"1260": "1.1.2.1260",
"1289": "1.1.3.1289",
"1473": "1.2.0.1473",
"1479": "1.2.0.1479",
"1485": "1.2.0.1485",
"1486": "1.2.0.1486",
"1489": "1.2.0.1489",
"1494": "1.2.0.1494",
"1499": "1.2.0.1499",
"1500": "1.2.0.1500",
"1509": "1.2.0.1509",
"1517": "1.2.0.1517",
"1520": "1.2.0.1520",
"1523": "1.2.0.1523",
"1532": "1.2.0.1532",
"1539": "1.2.0.1539",
"1540": "1.2.0.1540",
"1546": "1.2.0.1546",
"1548": "1.2.0.1548",
"1553": "1.2.0.1553",
"1563": "1.2.0.1563",
"1564": "1.2.0.1564",
"1569": "1.2.0.1569",
"1574": "1.2.0.1574",
"1576": "1.2.0.1576",
"1583": "1.2.0.1583",
"1584": "1.2.0.1584",
"1586": "1.2.0.1586",
"1604": "1.2.1.1604",
"1622": "1.2.2.1622",
"1727": "1.2.9.1727",
"1730": "1.2.9.1730",
"1737": "1.2.9.1737",
"1738": "1.2.9.1738",
"1743": "1.2.9.1743",
"1750": "1.2.9.1750",
"1758": "1.2.9.1758",
"1764": "1.2.9.1764",
"1773": "1.2.9.1773",
"1781": "1.2.9.1781",
"1790": "1.2.9.1790",
"1796": "1.2.9.1796",
"1800": "1.2.9.1800",
"1804": "1.3.0.1804",
"1836": "1.3.1.1836",
"1847": "1.3.1.1847",
"1855": "1.3.1.1855",
"1863": "1.3.1.1863",
"1891": "1.3.1.1891",
"2077": "1.4.0.2077",
"2089": "1.4.1.2089",
"2110": "1.4.2.2110",
"2152": "1.4.3.2152",
"2215": "1.4.4.2215",
"2243": "1.4.5.2243",
"2256": "1.4.5.2256",
"2332": "1.5.0.2332",
"2335": "1.5.1.2335",
"2395": "1.6.0.2395",
"2401": "1.6.1.2401"
}
}

Edited by @DasSkelett

[DasSkelett]

Those two lists are completely the same, at least Atom tells me so.
Did you spot a difference?

[jupiterbjy]

Checked with Notepad++, same here. To note here - At this time CKAN failed to initialize after update, so registry might not updated at all, sorry couldn't be helped.

[NicLars04]

@jupiterbjy, @DasSkelett
thanks for all 👍 . after succesfully deleting _(1st KSPBuilds = not yet worked; 2nd = working fine)_ all works fine as allways _(in CKAN v.1.25.4)_.

[NicLars04]

@DasSkelett
I dont know if you you wanna talk more about this probleme here with _jupiterbjy_ , so I'd let you decide when to close it. 👍

[jupiterbjy]

This bug is not solved, so it'll stay as bug until fix is out.

[NicLars04]

ok

[DasSkelett]

So a quick summarize where we are:
After updating to CKAN v1.26, it can happen that CKAN stops working when loading the registry (after changing version compatibilities or a simple refresh probably too) without throwing an error.

---

If you downgrade again. you get the System.ArgumentOutOfRangeException. It has something to do with the cache, but I don't see the exact cause yet.
Interesting: if I downgrade, I get

System.ArgumentNullException: Value cannot be null.
Parametername: key
   bei System.ThrowHelper.ThrowArgumentNullException(ExceptionArgument argument)
   bei System.Collections.Generic.Dictionary`2.FindEntry(TKey key)
   bei System.Collections.Generic.Dictionary`2.TryGetValue(TKey key, TValue& value)
   bei CKAN.Registry.LatestAvailableWithProvides(String module, KspVersionCriteria ksp_version, RelationshipDescriptor relationship_descriptor)
   bei CKAN.Registry.allDependenciesCompatible(CkanModule mod, KspVersionCriteria ksp_version)
   bei CKAN.Registry.Available(KspVersionCriteria ksp_version)
   bei CKAN.Main._UpdateModsList(Boolean repo_updated, IEnumerable`1 mc)
   bei CKAN.Main.CurrentInstanceUpdated()
   bei CKAN.Main.OnLoad(EventArgs e)
   bei System.Windows.Forms.Form.OnCreateControl()
   bei System.Windows.Forms.Control.CreateControl(Boolean fIgnoreVisible)
   bei System.Windows.Forms.Control.CreateControl()
   bei System.Windows.Forms.Control.WmShowWindow(Message& m)
   bei System.Windows.Forms.Control.WndProc(Message& m)
   bei System.Windows.Forms.Form.WmShowWindow(Message& m)
   bei System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)

which is probably because the newer metadata in the registry can't be read by CKAN and all those mods are returned as NULL, throwing this error. It is not a fatal bug, continuing a refreshing once gets CKAN to work again. Maybe doesn't have anything to do with this issue.

---

If you delete the registry key DownloadCacheDir, CKAN works again.
Maybe even a simple cache purge would suffice?
This, together with the exception points to a cache issue.

---

I wasn't able to recreate the issue yet. Can both of you @NicLars04 and @jupiterbjy send me an exported .ckan of your mods, so I can see if specific mods (suspecting those with spec version 1.26) cause the issue (if they are cached or something)?

[NicLars04]

mods_from_ckan.zip
this should be what you requested.

[jupiterbjy]

installed-portable-2019-04-05_17-42-51.txt

Lastest file right before CKAN failure. Convert .txt to .ckan.
This won't be as helpful as @NicLars04's since I let CKAN download at ramdisk.

Outside of topic, is there no temporary file removal on CKAN? Regardless of download successful or failure, CKAN keeps mod-sized temp files on temp folders and never removes it.

[DasSkelett]

Do you mean the cache? You can limit it in the settings.

[jupiterbjy]

CKAN download mod as tmp file at TEMP folder, and CKAN never moves it to cache folder unless all target mods are downloaded. This is not problematic unless there is crash or TEMP folder is limited.

Either using ramdisk or just system disk is full - this crash CKAN as TEMP folder containing drive get full before completing download. Then already created tmp files need to be manually removed or it stays there forever until reboot.

It would be much helpful if CKAN moved mod to cache folder after finishing each every downloads, so user won't have to worry about their capacity on TEMP folder and install large mods one by one.

[DasSkelett]

I think I know what you mean now.
Please open a new issue for this request, where we can discuss that without spamming this one.

[HebaruSan]

System.ArgumentOutOfRangeException: 인덱스 및 길이는 문자열 내의 위치를 참조해야 합니다.
매개 변수 이름: length (Rough translation: The index and length must refer to the position within the string. Parameter name: length)
   위치: System.String.Substring(Int32 startIndex, Int32 length)
   위치: System.Linq.Lookup`2.Create[TSource](IEnumerable`1 source, Func`2 keySelector, Func`2 elementSelector, IEqualityComparer`1 comparer)
   위치: System.Linq.GroupedEnumerable`3.GetEnumerator()
   위치: System.Linq.Enumerable.ToDictionary[TSource,TKey,TElement](IEnumerable`1 source, Func`2 keySelector, Func`2 elementSelector, IEqualityComparer`1 comparer)
   위치: System.Linq.Enumerable.ToDictionary[TSource,TKey,TElement](IEnumerable`1 source, Func`2 keySelector, Func`2 elementSelector)
   위치: CKAN.NetFileCache.GetCachedFilename(Uri url, Nullable`1 remoteTimestamp)

That exception means that this block is throwing:

https://github.com/KSP-CKAN/CKAN/blob/44d44ea5cd38e2f568d95f2af6d24674e01f7a45/Core/Net/NetFileCache.cs#L175-L180

Since it's complaining about the length, the cause may be a filename shorter than 8 characters in the cache folder. I can reproduce a freeze on load if I create such a file:

Two questions for those affected by this:

1. What path are you using for your CKAN download cache? (If you don't know, it'd be the default)
2. Are there any short filenames in that folder? If so, what are they and can you figure out how they got there?

EDIT: We have one answer to question 1 already:

@jupiterbjy, what else is in that folder?

[jupiterbjy]

@jupiterbjy, what else is in that folder?

Well, that E:\Tempy folder is not real cause of issue. Original folder was on ramdisk, with several temporary system files. So there's high chance to have file with short filename in there, but for now CKAN's temp file is shortest. (tmp38D5.tmp)

I think I know what you mean now.
Please open a new issue for this request, where we can discuss that without spamming this one.

Okay, will create soon.