A path to XDG compliance

[tecosaur]

In a recent Slack conversation, I offered to help answer questions about the XDG Base Directories spec. and the ~/.local/bin directory, as I've previously looked into this extensively over the development of BaseDirs.jl.

There are a number of problems that result from needing to modify shell initialisation scripts to extend PATH on Linux. The conversation on Slack started with one, somebody needing to give juliaup write permission to their shell config in order to successfully install. Others are noted in this comment of mine in a Pkg.jl PR.

@davidanthoff mentioned some of the reservations he had with the idea as a lack of clarity, and a seeming need for uncomfortable heuristics. To this, I produced the following response on how the ~/.local/bin directory could be well-handled, and am copying it here (at David's request) for posterity.

---

Completely get the heuristics making you uncomfortable. With Mac, everything they do define is well-defined, they just leave a bunch of blindspots, and those blind spots end up being a bit like the wild west. Linux is where I'm most familiar by far though. I'm familiar with the issue you mention regarding ~/.local/bin only being listed on PATH when it already exists. This is usually controlled by /etc/profile, which often includes something like this:

PATH=/usr/local/bin:/usr/bin:/bin
if test "$HOME" != "/" ; then
for dir in $HOME/bin/$CPU $HOME/bin $HOME/.local/bin/$CPU $HOME/.local/bin ; do
    test -d $dir && PATH=$dir:$PATH
done
fi

In this case, the way to confirm that ~/.local/bin* will be on the PATH (short of ugly heuristics that rely on reading the content of profile files) would be to:

• If ~/.local/bin does not exist and is not on PATH
• Create ~/.local/bin (this is in accordance with the XDG spec's instructions on what to do when a given XDG_*_HOME does not exist)
• Spawn a new (login*) instance of $SHELL, and fetch its PATH value
• Test to see $HOME/.local/bin is on the PATH from the previous step
• The vast majority of the time, at this point you will have identified that ~/.local/bin is indeed on the PATH.
• In the unlikely event that it isn't, try to add ~/.local/bin to PATH using whatever kudge was already being used for ~/.julia/bin. In this event, all of the concerns previously raised about trying to modify PATH this way still apply, but you'll only be hitting this in rare cases.

*login: Most shells will perform "login mode" startup with the -l flag, and their manual will contain something like this

         The usual  zsh  startup/shutdown  scripts  are  not  executed.   Login  shells  source
*        /etc/profile  followed  by  $HOME/.profile.  If the ENV environment variable is set on
         invocation, $ENV is sourced after the profile scripts.  The value of ENV is  subjected
         to  parameter  expansion,  command substitution, and arithmetic expansion before being
         interpreted as a pathname.  Note that the PRIVILEGED option also affects the execution
         of startup files.

i.e. they'll source /etc/profile.

*~/.local/bin: It would be good here to use ${XDG_BIN_HOME:-$HOME/.local/bin} it's a proposed extension of the XDG spec that's currently used in practice.

---

Another change that juliaup could make to better integrate with the Linux desktop is to use the XDG_DATA_DIR for the downloaded Julia versions, XDG_STATE_HOME for the *.json files, and XDG_RUNTIME_DIR for the juliaup lock file.

The typical way to make this backwards-compatible and be able to use the old mode of behaviour to is via an envvar like JULIAUP_HOME, or even having JUILAUP_STATE and JULIAUP_DATA (I'd argue that the lock file should live in the runtime dir regardless).

---

Hope that helps, and happy to field follow-up questions!

[ghyatzo]

#429 had a similar idea, it was to fix an issue of symlinking the juliaup binary which would then use the wrong parent folder to manage itself, causing it to delete stuff that was not his when uninstalling itself. This issue arose while I was trying to make juliaup compliant with XDG by symlinking stuff around.
Alas, it was closed and abandoned due to the dislike for the need for extra ENV variables.
I admit it was a while ago, and I am not up to date with all the updates so far, maybe it was fixed already, but reading this issue with XDG compliance made me think about it.
If it can be helpful to cherrypick some ideas from that PR all the better!

[StefanKarpinski]

My only ask is that there be an opt-in variable, eg JULIA_XDG=true which controls whether XDG is used or not. If it's unset, everything goes in the standard places. If it's set and true juliaup and Julia can go hog wild with XDG.

[tecosaur]

If Julia itself (not just Juliaup) goes down the route of XDG-compliance (which I'd love to see!), we can follow a well-trodden road for how to go about this. Most applications I'm aware of follow this sort of logic:

1. Look for the legacy directories. If they exist, use them.
2. Otherwise use the XDG-appropriate directories

The most common augmentation here is introducing a APPNAME_DIRNAME variable(s) to control where various files/folders go, in essence abstracting over the XDG and legacy file structure. If set the logic above is skipped and the variables are just used.

As Stefan requests, a JULIA_XDG variable could also be introduced as a less flexible way of allowing specification of what arrangement should be used (skipping the above logic check). Should this approach be taken I'd advocate for an initial opt-in system that transitions to opt-out after the XDG restructuring proves stable.

Whatever combination of the above you chose, full backward compatibility is maintained while enabling XDG compliance moving forward.

[StefanKarpinski]

No, I don't care if XDG-appropriate directories exist. XDG is not a good default. What we do now is a good default: for the typical user who doesn't know or care about XDG, we put all Julia-managed stuff in a single, easy to find, standard location. End of story. For the minority of people who care about XDG, they can set a single environment variable and get XDG behavior. Otherwise we get a situation where people will inevitably be confused about where their Julia files are. "Oh, you're running Julia but you don't have a ~/.julia directory? That's strange. Oh wait, some other program created an XDG directory and then Julia saw that and started automatically putting things there even though you don't know or care about XDG." Absolutely not. My requirement is this: users who are not aware of XDG should never be forced to go look at XDG locations.

[StefanKarpinski]

Should this approach be taken I'd advocate for an initial opt-in system that transitions to opt-out after the XDG restructuring proves stable.

This will never happen: XDG will always be opt-in. The UNIX business of spreading all the files related to a single application across dozens of directories that no one can keep straight is absolute and utter nonsense. I'm fine with allowing people to opt into it if they are masochists or have an actual need for it (this is like five dozen people worldwide), but it won't ever be the default.

To be clear:

• I'm 100% in favor of XDG-support
• But only if it's fully opt-in
• The opt-in can be dead simple, however
• Specifically, a single environment variable that is respected by both juliaup and julia.

[KristofferC]

If it isn't the default it doesn't seem worth doing at all because all julia infrastructure would have to add support for it with probably very few uses (the vast majority uses the defaults no matter what they are).

[ghyatzo]

XDG hardcore compliance is a pain, and in my opinion other than XDG_CONFIG_HOME very hard to generically apply blindly to any project, but a softer version could be obtained by using an ENV var like JULIAUP_HOME to indicate an install directory (similar to how rustup does). If unset nothing changes, if set juliaup will be installed there. This would allow safe symlinking (no reliance on current_exe() whims) which in turn will allow anyone who cares about it to symlink stuff where he best desires, and in turn letting juliaup manage itself without having to worry about having its files spread out across the whole fs. (as well as letting used straight up already existing XDG_???_HOME env vars as their JULIAUP_HOME)

In my experience XDG compliance is helpful mainly to the user so that he knows where to go look for stuff without worrying about the specific installation paths of each program used. If it is then just a matter of quicker access of that specific part of a program, XDG compliance can be easily achieved (with some extra work from the user that wants it) by symlinking. i.e.: $XDG_CACHE_HOME/project/ -> /real/project/path/cache and everybody is happy.

[tecosaur]

I can't say I'm a fan of how this discussion seems to be drifting towards retreading JuliaLang/julia#4630.

[StefanKarpinski]

My position remains identical. XDG is not a good default for the typical Julia user, even on Linux. It would be a good feature to support for anyone who opts in, however.

[kdheepak]

I agree that XDG by default without additional functionality to guide users can be confusing for a typical Julia user or new users that are new to programming. On MacOS, the XDG folder would be ~/Library/Application Support/julia and I'm certain 99% of my coworkers who use Julia and a Mac would have never even heard of such a folder.

That said, I think even the current $HOME/.julia is quite inaccessible. Majority of people I know would have no idea where to look if I said "open your julia config folder" or "open your startup.jl file". Particularly, on Windows I've found that many don't know that C:\Users\their-user-name is their "home" folder. Also, when people don't have hidden files shown in the file explorer, it is a real annoyance for them to open a startup.jl file. I think using XDG by default is better than $HOME/.julia in this regard.

I don't think most users should ever open their config folder, and if it wasn't for showing people how to set up Revise in several tutorials I would never even include it in tutorials.

Maybe julia --version should print out the config folder? That would make even XDG folders accessible.

[davidanthoff]

I don't have strong views on the substance of this at all (hey, I'm a Windows guy, we don't have these problems!), but I am not keen to support two things, so I'm with @KristofferC on this. It just makes maintenance a huge pain, now we need to test double the scenarios and this becomes a permanent "tax" on the Juliaup dev team. If this is really just for aesthetic reasons and doesn't solve any "real" problem, then I think our scarce time is better spent on other features.

Having said that, our PATH modification approach is a real problem, and if there was a way to make that more robust by hooking into existing folders that are on the PATH then that would be great...

But maybe this is just one more reason to try to get Juliaup published into platform package managers as julia, so that things like brew install julia, apt install julia etc all work...

Random other thought, I think whatever we do about the PATH situation should probably be coordinated with @KristofferC's app work.

[tecosaur]

Juliaup can use ~/.local/bin and the runtime-dir for the lockfile entirely independently from what Julia does (~/.julia or XDG).

I have also brought up ~/.local/bin and the PATH headache in Kristoffer's App PR, I believe it currently uses ~/.local/bin but doesn't check XDG_BIN_HOME.

I have a number of thoughts on Julia and XDG, but I cannot muster the motivation to be part of another ungrounded conversation. Suffice it to say, I see value beyond "aesthetics".

[kenahoo]

My position remains identical. XDG is not a good default for the typical Julia user, even on Linux. It would be a good feature to support for anyone who opts in, however.

Regarding "opt-in" - for pretty much any system that has added XDG support, the "opt-in" mechanism is that you create some files in the XDG location and go delete your files in the previous location. Why isn't that sufficient for the Julia use case? Why is another separate (and redundant, and confusing IMO) mechanism necessary?

I did exactly that for git (move ~/.gitconfig to ~/.config/git/config) a couple years ago and I've never had to think about it again. And it's made it a lot easier to manage syncing my dotfiles across multiple machines. If I also had to make sure some other environment variable were set, for every tool I use, that would be quite unpleasant.

it won't ever be the default.

Regarding "default" - I'm not sure that word should apply here, unless I'm missing something (which I very well might be). The config location isn't a matter of "default" vs. "non-default", it's just a search order, right?