Ckan: Please cache downloads outside of the game directory

[rafl]

Currently almost all of CKAN's data, including downloaded mods, is stored within the CKAN directory within a KSP installation. As a consequence, mods previously downloaded and cached in one KSP installation's CKAN directory will be re-downloaded when trying to install the same mods into another KSP installation.

Especially with larger packages, such as the high resolution real solar system textures that weight in at something like 2GB, this gets annoying quickly, especially over slow downlinks.

Rather than caching downloads within KSP installations, please cache them in a more global location such as the user's home directory. On linux systems $XDG_CACHE_DIR/ckan would seem like a reasonable place.

[AlexanderDzhoganov]

I think this is so important that I will start working on it immediately. Sorry for the delay, we were busy handling everything 0.90 related.

[hakan42]

I believe this would also affect netkan, and therefore MirrorKAN... Best check before something stomps all over your filesystem

[taraniselsu]

I would also like to see this feature. I sometimes run KSP from a RAM drive and I don't want to make the drive big enough to hold KSP+mods+the entire cache.

Thanks!

[pjf]

I'm fond of the XDG standard as well. It's not just for Linux. Although I've honestly no idea what people in Windows land expect for caching. Is there a standard there at all that we should be aware of?

[RichardLake]

Judging by other programs it should go into either CommonApplicationData or LocalApplicationData depending if it should be per user. The paths to these can be found via Environment.GetFolderPath(Environment.SpecialFolder.{Name of special folder}).

[taraniselsu]

I would like manual control over where the files are put. Please do not hard code it, even to the OS' "expected" location.

[pjf]

@taraniselsu : Do you have a preferred interface through which this may be achieved? If we defaulted to the XDG/OS standard location, but allowed a manual setting via ckan.exe ksp cache /path/to/cache would that be sufficient?

Given that the CKAN cache should be the same for all installs (see @rafl's use-case in the top of this ticket), I'm going to suggest that the cache directory be stored in the Windows/Mono registry, rather than the CKAN one.

[leftler]

@pjf I very much like that idea, the way I would do it (I don't know off the top of my head the best way to do GetCacheDirFromCommandLine() or I would have made this a Pull Request)

private static string GetCacheDir()
{
    string cacheDir = GetCacheDirFromCommandLine();
    if(cacheDir != null)
        return cacheDir;        

    cacheDir = Environment.GetEnvironmentVariable("XDG_CACHE_DIR");
    if(cacheDir == null)
    {
        cacheDir = Environment.GetFolderPath(Environment.SpecialFolder.CommonApplicationData);      
    }

    cacheDir = Path.Combine(cacheDir, "ckhn");      
    return cacheDir;
}

That should resolve to the correct path on all 3 major OSs, first preferring what was passed in to the command line, if nothing was defined it will then use the XDG_CACHE_DIR environment variable if it is assigned or the common application data folder (which resolves to C:\ProgramData on windows and /usr/share on Linux and OSX) and use the ckhn subfolder within it.

[hakan42]

Just to throw in my non-existent knowledge of how mono really defines its paths: neither c:/ProgramData nor /usr/share on *x are writable for non-admin users.

If we go this way, we really really really need to give users a way to define where they want their cache directory to be.

[leftler]

@hakan42 I don't know how *x handles it but on windows C:\ProgramData can have non elevated programs write to it no problems. Just open notepad as a non elevated user and save a text file in the C:\ProgramData folder and then navigate to the folder in explorer and you will see the file was saved. Perhaps you are thinking of C:\Program Files which is restricted to only allowing elevated users write to it (Well non elevated programs can still write to it too if they don't explicitly disable redirection by stating they are Windows Vista or newer compatible in their manifest, if they don't specify it in their manifest writes to C:\Program Files will get transparently redirected to %LocalAppData%\VirtualStore\Program Files to provide backwards compatibility for programs written for XP that assumed that the user would have admin rights.)

[legolegs]

I see rather strange assumptions here. /usr/share is NOT okay to store cache in. Neither as ProgramFiles.

http://standards.freedesktop.org/basedir-spec/basedir-spec-latest.html

$XDG_CACHE_HOME defines the base directory relative to which user specific non-essential data files should be stored. If $XDG_CACHE_HOME is either not set or empty, a default equal to $HOME/.cache should be used.

Also a good note about cross-platform cache ir can be found there: http://www.chromium.org/user-experience/user-data-directory (except that hardcoded "C:")

[leftler]

@legolegs no one here said program files is the correct place c:\Progam Files and c:\ProgramData are VERY diffrent folders with very diffrent purposes. Also XDG_CACHE_HOME is for user specific cache data, the options discussed so far are all non-user specific.

[legolegs]

Also XDG_CACHE_HOME is for user specific cache data, the options discussed so far are all non-user specific.

There is no other place to store cache in XDG standard. Unless skan is going to become a system-wide tool (like apt-get) there is no way to make something like /var/cache/ckan.

[AlexanderDzhoganov]

I'm implementing @leftler 's version for now, though I will continue to follow the discussion here.

[legolegs]

@leftler 's version is just wrong in linux part. Firstly there is no such thing as XDG_CACHE_DIR, see this bug if you don't believe neither me nor XDG standard.

XDG_CACHE_HOME is the only one specified by the standard

Secondly looking at this research there is a bug with Mono itself: on windows CommonApplicationData expands to something like C:\Documents and Settings\All Users\Application Data which is okay, but Mono expands it to /usr/share which is absolute nonsense, it just like C:\windows\, it is not writable and is not supposed to hold any caches.
The proper way should be like this (I am not a C# programmer so I might be mistaken with syntax):

private static string GetCacheDir()
{
    string cacheDir = GetCacheDirFromCommandLine();
    if(cacheDir != null)
        return cacheDir;        

    cacheDir = Environment.GetEnvironmentVariable("XDG_CACHE_HOME");
    if(cacheDir == null)
    {
        cacheDir = Environment.GetFolderPath(Environment.SpecialFolder.CommonApplicationData);      
        if (cacheDir == "/usr/share") //"/usr/share") is read only, this is bug in Mono
            cacheDir == Path.Combine(Environment.SpecialFolder.Personal, ".cache")
    }

    cacheDir = Path.Combine(cacheDir, "ckhn");      
    return cacheDir;
}

[AlexanderDzhoganov]

@legolegs correction noted, thanks!

[Ippo343]

This issue was moved to KSP-CKAN/CKAN-core#10

[pjf]

re-opening after the re-merge

[7ranceaddic7]

Would it be possible to get the cache directory to be user specified? Currently I have a 128GB SSD partitioned for SxS Linux/Windows/Home. I have multiple SATA drives, one which I have specified a directory for my Software Archive.

I don't need the speed of the SSD for CKAN operations and the .../CKAN/downloads/ directory chews up a sizeable portion of the Home partition that could otherwise be put to better use.

[plague006]

@7ranceaddic7 the workaround I use (windows box) is creating a symlink from the cache folder on my SSD to one on my HDD. Overall being able to choose a cache directory would be ideal, but all I can offer is a workaround. =)

[7ranceaddic7]

@plague006 TY! TY! TY! Workaround or not is works perfectly.

[SilverlightPony]

If a good reason exists for one program to use _another_ program's installation directory as a data "warehouse"/cache, I've never encountered it. For something like this (in Windows), IMHO the only reasonable place to store downloaded mods is in a subfolder of the same directory CKAN.exe is in—perhaps with further subfolders denoting KSP version compatibility. e.g.:

• Some_Folder_Somewhere\
• - CKAN.exe
• - CKAN data\
• - - installed-default.ckan
• - - registry.json
• - - GUIConfig.xml
• - - Plugins\
• - - Downloads\
• - - - 1.0.5\
• - - - 1.1.2\
• - - - 1.1.3\

Please, please, please get CKAN's cache out of the KSP installation directory. It has no business being there.

[politas]

I think the version compatibility subfolders are a bad idea. If we want to use our stored hashed to recognise user-downloaded mods, we want to have a simple location for users to place them. Also, given the min/max provisions, you would quickly gain a plethora of confusing locations

. 0.90-1.0.4
. 1.0.1-1.0.2
. 1.0.2
. 1.0.5-1.0.0
. 1.0.5-1.1
. 1.0.5-1.1.2

CKAN would need to ignore this folder structure and we would need to move files around every time metadata changes, which could make things very confusing. I think a single defined location is best.

[SilverlightPony]

Single location is fine, as long as it's not inside a KSP installation directory.

[politas]

Agreed. Tricky to work out a default in a cross-platform app, unfortunately. Especially when both CLI and GUI need to allow a way to check and change it. I would have thought something like C:/Users/username/(Program Data?)/CKAN/downloads for windows, and ~/.local/CKAN/downloads for *x

[SilverlightPony]

I prefer not to rely on system variables like %programdata% if I can help it. Anything that is not user-specific (and cached mod ZIP files _aren't_ user-specific) should go in a subfolder of the program's own installation directory; the user should have means to choose whether user-specific info (in this case, stuff like KSP installation paths and current installed mods, among other bits of info) are stored in the program's installation directory as well, or in %appdata% (or %localappdata%; I'm still not sure what functional difference there is between the two).

[leftler]

@SilverlightPony normally installation directories are read only by non privileged users (administrators/super users), it is not a safe assumption that a installation path will be writeable.

The difference between %appdata% and %localappdata% is when you are in a domain files located in %appdata% will be copied to the domain controller on log off and copied to the computer when you log on, this lets files travel with you when you log off one machine then log on to another. files located in %localappdata% will always stay on the machine it was created on.

[politas]

I think the best thing initially, is to bypass the arguments about the ideal default location and implement a way for users to set the cache directory. Once that's in place, we can think about moving the default from where users currently expect to find it.

[linuxgurugamer]

I would suggest that when this is FINALLY implemented, that it default to the current behaviour. Give the users the ability to do the following:

1. At first time startup, select cache directory
2. In settings give ability to select the directory.

I honestly don't understand why it's taking so long to do something so simple. You already have dialogs to select game location, so pretty much everything you need is already implemented.

While I am not a contributer to this project (yes), I might consider it to get this feature

[DinCahill]

Everyone loves a good bikeshed!

[linuxgurugamer]

Ok, so, are you saying that I responded too fast (I did read the entire thread, and actually opened an issue yesterday on this), or that I should actually do something about it?
Not sure if you were being sarcastic or serious

[DinCahill]

Lol, it's a joke. We're busy discussing minutiae rather than doing anything. @politas has tried to end it, by getting us to build the bikeshed now, and then we can settle on what colour to paint it later.

[linuxgurugamer]

Ok, but still, has there been any action? If not, then maybe tomorrow I'll take a look at setting up a local CKAN dev repo and seeing if I could do it myself

[linuxgurugamer]

So, this is what I'm planning on doing, seems to cover everything listed above:

1. Add line to settings window showing path of cache directory
2. Add 2 buttons to the right of the line to select cache directory, or use the default
3. Add cache directory path to saved settings
4. Update KSP.cs (DownloadCacheDir) to use the selected path

Since I'm new to this code, please feel free to tell me what I missed :-)
But hopefully this will get it working, and specifics can be updated later if needed

[DinCahill]

If anyone's started working on this privately, please say so!

Your suggestion is in line with what @politas said, so if you want to work on it, @linuxgurugamer, go for it :smile:. Then we can review some actual code in a PR.

[politas]

Need to make sure the directory choice is saved in the registry with the set of KSP installs, rather than the CKAN appearance settings that are saved per KSP install (to be honest, I'm not sure why we do that!), but yes, that's about the gist of it. This was on my personal list of coding projects, but behind a couple of other things. Feel free to have a stab at it. Have a quick read of the Developer guidelines in the CKAN wiki; there's some useful stuff in there.

[politas]

1788, #1847 and #960 are all related (or can be related to this) This particular change (Allowing users to select a cache directory) is probably the first step. Second step is to get CKAN to recognise files in the cache directory by their hashes rather than their filenames, at which stage we will no longer care about what we call the files, and we can think about file naming under #1637.

[techman83]

@politas as far as I know it already looks for just the hashes. I recall looking at it when working on the NetKAN code.

[politas]

It might test the cached files against the hash, but it doesn't find them that way. If you rename an item in the cache, it will re-download.