Julia: Store settings and packages in XDG directories instead of ~/.julia

[nalimilan]

Over the last 10 years most Freedesktop environments (GNOME, KDE, XFCE...) and applications around them have moved to store configuration in the XDG config directory (by default ~/.config) instead of creating a hidden directory directly into the home folder. A well-known non-GUI application that does the same is git:
https://github.com/git/git/blob/master/Documentation/RelNotes/1.7.12.txt#L18-23

The Freedesktop specification is here:
http://standards.freedesktop.org/basedir-spec/basedir-spec-latest.html

and a rationale for this specification can be found here:
https://wiki.gnome.org/GnomeGoals/XDGConfigFolders

I think Julia should follow it too. The idea is very simple, you just need to read an environment variable:

$XDG_CONFIG_HOME defines the base directory relative to which user specific configuration files should be stored. If $XDG_CONFIG_HOME is either not set or empty, a default equal to $HOME/.config should be used.
http://standards.freedesktop.org/basedir-spec/basedir-spec-latest.html

Thus, in case $XDG_CONFIG_HOME is not defined, settings would be stored in the default ~/.config/julia.

Packages should likely go to $XDG_DATA_HOME, which defaults to .local/share/ (the equivalent of /usr/share/). One rule of thumb is that configuration is more like text files with limited size and that do not change that much (some people even keep it under a VCS). OTOH $XDG_DATA_HOME is usually conceived as a place to store valuable user data, which needs backup, while packages can easily be downloaded again. After much reading and thinking I still don't know so this probably means it does not really matter.

So the changes would be pretty limited actually. In the documentation, it would be sensible to speak about ~/.config/julia instead of ~/.julia, without mentioning the environment variables trick (users who change the defaults will know).

One of the potential gains of following that XDG spec is that it also defines $XDG_CONFIG_DIRS and $XDG_DATA_DIRS, which could be used by administrators to provide a system-wide package library, and default settings for new users. Julia would read config from there, but only save user config and packages in the user home directory.

[JeffBezanson]

This is a nice and thorough proposal.

[StefanKarpinski]

Yes, this is a very well-presented proposal. However, we're not going to do this. We only _just_ got .julia in the same place on all systems (Windows was the outlier before). It would be crazy to start doing some different idiosyncratic thing on different systems. This standard doesn't even work on all Linuxes, let alone other UNIXes like OS X, FreeBSD and of course Windows. The only thing that is truly universal is putting a bunch of files in a directory in the user's home directory.

[nalimilan]

Thanks for the kind words. ;-)

I understand that changing the path to the config directory is not a nice task. That's why it took ten years and all programs haven't moved yet. But what I'm proposing merely amounts to changing the path from ~/.julia to ~/.config/julia. On 90% of systems it will be that way; only a few UNIX hackers will change the path to XDG_CONFIG_HOME, and _they_ know what they are doing. There's no issue of the standard "working" or not: either the variable is defined, and you use it, or it's not, and you use your default path; in both cases you end up with the same result most of the time: ~/.config. This is my definition of "putting a bunch of files in a directory in the user's home directory". So I don't think this introduces a real inconsistency between OSes.

BTW, it's false that FreeBSD does not support this: see
http://www.freebsd.org/cgi/query-pr.cgi?pr=133825
I also find it unlikely that any Linux distribution fails to support this, as almost all free desktop environments have made the switch for some time. GLib for example provides functions that default to ~/.config if XDG_CONFIG_HOME is not set, and it's used by most GNOME and XFCE apps now.

Julia is still young, so it's not too late to switch. Big and popular cross-platform applications have made or are making the change: Chromium [1], Firefox (although not without debate [2]), VLC[3], LibreOffice, GIMP [4].

Disclaimer: though I'm member of the GNOME Foundation, I have no shares in Freedesktop.org. :-)

1: https://chromiumcodereview.appspot.com/12386077/
2: https://bugzilla.mozilla.org/show_bug.cgi?id=259356#c81
3: https://trac.videolan.org/vlc/ticket/1267
4: https://mail.gnome.org/archives/gimp-developer-list/2012-October/msg00037.html

[StefanKarpinski]

Having read through the spec, I'm not sold on this at all. I've got many issues with it.

1. This spec isn't even finalized and the draft is pretty recent (2012), so it's not like this is a really well-established best practice or anything. This is some stuff that some people made up last year.
2. This is a spec for Linux _desktop_ programs (KDE, Gnome, etc.). Julia does not fit the bill.
3. Even ignoring the non-desktopness of Julia, it's completely unclear that the spec even remotely applies. The spec is about application configs, user data and caches. The installed Julia packages are not any of these things. There is a repo cache directory, and I would maybe consider following the spec for that if the rest applied, but installed packages are not preferences or user data.
4. The package system is explicitly designed so that you will typically _work_ in your .julia directory, creating and developing packages. Moving it to some path like ~/.local/share/julia is a major pain, not to mention being pretty clearly semantically wrong – again, installed packages are neither configuration nor user data.
5. The spec is not designed to be opt-in. If you _don't_ set any environment variables, you're expected to use the long, fiddly locations of things that are scattered all over the place (~/.config/julia, ~/.local/share/julia, /etc/xdg/julia, etc.). If there was some environment variable that allowed people to opt into this – say XDG=true, or something – and this was set by default on standard-conforming systems, then I might consider supporting that, but that's not how the spec works. It assumes everyone has bought into this wholesale, rudely foisting long annoying paths on people who don't give a crap about XDG and just want their Julia stuff in .julia.

Given all these issues and the fact that this causes a fairly major coding headache and fractures the IMO priceless uniformity of our cross-platform behavior, I'm sticking with my original gut reaction, which is to say no.

[StefanKarpinski]

I realize you mentioned that git "conforms" although it isn't a desktop program. Except that it doesn't really conform – git will _look_ in the XDG locations if they exist, but if it needs to _create_ a config file, it still creates ~/.gitconfig.

[perillo]

It is ok for Julia to not follow XDG, however it is a nice thing to allow users to change
the default directories used by Julia from the configuration file, or during julia source compilation.

[ghost]

like go, julia can have a ~/.config/julia/env for storing environment variables. so we can choose which directory to use.

[DilumAluthge]

Quoting Fredrik in https://github.com/JuliaLang/julia/issues/37503#issuecomment-690036262:

You can set JULIA_DEPOT_PATH of you are not happy with ~/.julia.

[StefanKarpinski]

I was kind of harsh in this issue before, but I still think that consistently putting all Julia-related stuff in ~/.julia by default was the right call. The XDG "spec" (if such a tiny, half-baked document can be called a spec) still has all of the issues I called out in 2013. Which is not too surprising since no one has touched it since it was scribbled on the virtual equivalent of the back of a napkin a decade ago.

The "spec" is terribly UNIX-specific and only makes sense on multiuser UNIX systems, which are no longer the common case: these days Linux and macOS dev laptops are the norm and the vast majority of servers are virtualized/containerized single-user systems. This spec really only makes sense on multiuser UNIX systems at research institutions. For the vast majority of users, the complexity of scattering everything around directories that aren't next to each other doesn't gain you anything since those directories are on the same file system on your computer's one and only hard disk. All it does it make it harder to see what's going on and add unnecessary complexity to Julia's Pkg and code-loading logic that works with those directories (which is already too complex).

For those users that do actually work on a multi-user UNIX system where it makes sense to put config, cache and runtime data on different filesystems, it would be simple to create symlinks for the fixed set of subdirectories stored under ~/.julia to the various XDG directories as they feel appropriate. Such a tool could even be open sourced and maintained by people who have need for it! For reference, the directories that are saved under ~/.julia are listed in the glossary entry for "depots" and include:

• artifacts: content-addressed data and binary dependencies
• environments: shared named environments (e.g. v1.0, devtools)
• clones: bare clones of package repositories
• compiled: cached compiled package images (.ji files)
• config: global configuration files (e.g. startup.jl)
• dev: default directory for package development
• logs: log files (e.g. manifest_usage.toml, repl_history.jl)
• packages: installed package versions
• registries: clones of registries (e.g. General)

I'm not entirely sure how to map these onto XDG locations, which I feel is due to lack of clarity of the "spec" since I do understand what these directories are for but the spec doesn't really give clear criteria for what should go where, it just lists a bunch of technical properties of locations by which you're apparently supposed to infer, together with suggestive names, where to put things? I feel like people who want to use the spec can figure this out.

For people who work on a single-user system where all the XDG directories are on the same file system but think that it's distasteful for Julia to clutter up their home directory with it's one ~/.julia directory, it should be sufficient to change JULIA_DEPOT_PATH to point somewhere else.

[jtrakk]

it should be sufficient to change JULIA_DEPOT_PATH to point somewhere else.

Indeed it is nice that we can set a custom path for julia files. Going a bit further would help: currently users can't separate their configuration from the generated like registry clones. For users to be able to apply the spec to their own systems, it would be helpful if the JULIA_DEPOT_PATH could be split up into more than one variable, so I can say JULIA_CONFIG_DIR goes to ~/.config/julia, JULIA_CACHE_DIR goes to ~/.cache/julia, JULIA_DATA_DIR goes to ~/.local/share/julia.

[DilumAluthge]

For those users that do actually work on a multi-user UNIX system where it makes sense to put config, cache and runtime data on different filesystems, it would be simple to create symlinks for the fixed set of subdirectories stored under ~/.julia to the various XDG directories as they feel appropriate. Such a tool could even be open sourced and maintained by people who have need for it!

[DilumAluthge]

It's unlikely that this will be worked on further in Julia itself. A third-party open source tool sounds like the right answer.

[jtrakk]

I missed an important post from @StefanKarpinski in the other thread https://github.com/JuliaLang/julia/issues/10016#issuecomment-370254878:

The "dotfile bloat" part of this has been addressed, so I'm closing. If someone wants to implement the XDG stuff, they're welcome to make a PR.

So it seems it's on those of us who are interested in XDG to write the code but that @StefanKarpinski wouldn't be opposed to it.

[StefanKarpinski]

If someone wants to work on this, it's fine, but there are questions that needs answers. So far the "proposal" here is "Julia should follow the XDG spec". Ok, suppose we were to do that. What, concretely, does that mean?

First and foremost, how do each of the directories I listed in https://github.com/JuliaLang/julia/issues/4630#issuecomment-690349632 map to XDG locations? Someone who has an interest in this needs to spend some time with the XDG spec and classify each of these and how they should relate to all the XDG environment variables. There are a lot of them: XDG_DATA_HOME, XDG_CONFIG_HOME, XDG_CACHE_HOME, XDG_DATA_DIRS, XDG_CONFIG_DIRS, XDG_RUNTIME_DIR. For each subdirectory of ~/.julia, they need to determine where, based on the values of all of these XDG environment variables, should that subdirectory go? This is a very concrete question: there should be an expression for each subdirectory I listed that computes a path location based on the values of XDG variables. Without this, no progress can be made.

The next step would be to evaluate whether it makes sense to put this logic into Base and Pkg or not. My guess is that it is not worth the significant added complexity to both Base and Pkg, but if someone can refactor the Base and Pkg code so that it abstract over the XDG stuff cleanly, then a case could be made for it.

An alternative is to have a utility package that goes through all the paths above and makes sure that ~/.julia/packages, for example, is a symlink to the place that XDG dictates that Julia packages should be stored. If the directory already exists, the tool would first move the contents of that directory to the XDG-specified location and then create the appropriate symlink from ~/.julia/packages to the XDG location. That way ~/.julia always exists and lets you easily see all your Julia-related data, but that data can be stored in XDG-compliant locations. If the XDG variables never change, then this symlink setup only needs to be run once, but if you anticipate changing them often, the tool could be called from ~/.julia/config/startup.jl so that it gets run on each Julia startup. It should probably try to do some kind of advisory locking (see Pidfiles) so that if multiple instances of the tool are run at the same time, they don't try to do this simultaneously.