Julia: Pkg directory shared among users has several permission problems

[diegozea]

In my computer, using a package gives me this SystemError (denied permission).
I need to run sudo julia in order to use packages... Is there a way to solve this? Can *.ji be created with less restrictions?

julia> using Compat
ERROR: SystemError: opening file /home/dzea/.julia/lib/v0.4/Compat.ji: Permiso denegado
 in open at ./iostream.jl:90
 in open at iostream.jl:102
 in stale_cachefile at loading.jl:383
 in recompile_stale at loading.jl:418
 in _require_from_serialized at loading.jl:83
 in _require_from_serialized at ./loading.jl:109
 in require at ./loading.jl:186

julia> versioninfo()
Julia Version 0.4.0-dev+7044
Commit 706cad1 (2015-08-27 21:32 UTC)
Platform Info:
  System: Linux (x86_64-linux-gnu)
  CPU: Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz
  WORD_SIZE: 64
  BLAS: libopenblas (NO_LAPACK NO_LAPACKE DYNAMIC_ARCH NO_AFFINITY Sandybridge)
  LAPACK: liblapack.so.3
  LIBM: libopenlibm
  LLVM: libLLVM-3.3

[nalimilan]

Maybe you installed the package while running julia under sudo? Can you post the output of ls -l /home/dzea/.julia/lib/v0.4/Compat.ji?

[diegozea]

I can't remember if I installed the package with sudo. The output is:

-rw------- 1 root root 30383 ago 29 00:51 /home/dzea/.julia/lib/v0.4/Compat.ji

All my *.jis have the same mode: -rw------- 1 root root

[yuyichao]

They cannot possibly have root as their owner if it is not created or changed ownership by root. This is a very typical issue with running command under sudo while maintaining the value of HOME.

[jakebolewski]

That looks to be your problem.

[nalimilan]

I case you don't know how to fix it, chown -R dzea: /home/dzea/.julia should do the trick. Let us know if it ever comes back.

[timholy]

I'm reopening this to point out that using a system-wide shared Pkg folder has several issues:

• Even if /usr/local/julia/packages is created with g+w permissions, any new folders get created with default permissions. If user1 adds CoolPackage.jl, user2 can't fix bugs in it even though they are in the same group.
• The *.ji files get written with permissions like these:

user2@diva:/tmp/packages/lib/v0.4$ ls -l
total 484
-rw------- 1 user2 dummy 376701 Aug 30 11:27 ColorTypes.ji
-rw------- 1 user2 dummy  30165 Aug 30 11:27 Compat.ji
-rw------- 1 user2 dummy  82414 Aug 30 11:27 FixedPointNumbers.ji

What that means is that if one user has used a package, it prevents any other user from accessing it. As user1:

julia> using ColorTypes
ERROR: SystemError: opening file /tmp/packages/lib/v0.4/ColorTypes.ji: Permission denied
 in open at ./iostream.jl:90
 in open at iostream.jl:102
 in stale_cachefile at loading.jl:379
 in recompile_stale at loading.jl:414
 in _require_from_serialized at loading.jl:83
 in _require_from_serialized at ./loading.jl:105
 in require at ./loading.jl:182

julia> 

In this case user1 can read the _source_ files, but because the *.ji file is present julia tries (and fails) to load it.

[timholy]

For the benefit of others, here's an easy way to do this kind of testing:

addgroup dummy
adduser --ingroup dummy user1
adduser --ingroup dummy user2

and then start julia with JULIA_PKGDIR=/tmp/packages /path/to/julia.

[nalimilan]

It would probably make sense to respect the permissions of the parent directory (.julia or a subfolder thereof) when installing packages.

[timholy]

That's my sense too.

[stevengj]

The .ji permissions are the result of #12699, because mkstemp+rename uses mode 0600. I guess it could run chmod afterwards.

[adowling2]

We have setup a shared Julia package folder on a shared computer, and have observed a few (possibly related) issues:

• When a package is recompiled, the .ji permissions are set such that other users of the machine cannot access them. We have a script the fixes this via chown and chgrp and runs every 5 minutes via cron as a work around.
• After one user recompiles a package, any other users with get "Recompiling stale cache file" message. Were constantly recompiling packages on a busy multiuser machine.
• The git repos for the packages inevitably become dirty, and each require a "stash" command before Pkg.update() will do anything.

[tkelman]

If you're sharing package installs, you should also share julia itself, or set the precompile cache to be user-specific.

Packages becoming dirty is either the fault of your script or a bug in the package.

[adowling2]

We're sharing the Julia executable too. I'll look into using user-specific percompile caches.

[tkelman]

if the julia executable is the same, then you shouldn't be seeing recompilation unless people are editing the package sources

[adowling2]

People are not editing packages, so this is strange.

Let's take the JuMP package as an example. When I run

$ cd /opt/julia_packages/.julia/v0.4/JuMP
$ git status

I see many files have been modified. On closer inspection

$ git diff ./test/REQUIRE
diff --git a/test/REQUIRE b/test/REQUIRE
old mode 100644
new mode 100755

I see that only the file permissions have changed. I think this is also causing the dirty git directories. What file permissions are appropriate for /.julia/v0.4 and /.julia/lib/v0.4? All of the authorized users are members of a common group. I'm currently changing all of the files in this folder to said group.

Ideally, I'd like all members of the authorized group to be able to

1. Run shared packages
2. Run Pkg.add()
3. Run Pkg.update()

Thanks in advance.

[tkelman]

I'm not sure if git tracks group ownership, but it does track permissions, and you shouldn't need most git-tracked files in packages to be executable.

[adowling2]

Below is the most recent version of our maintenance script:

chgrp ourgroup -R /opt/julia_packages/.julia
### chmod 774 -R /opt/julia_packages/.julia

chgrp ourgroup -R /opt/julia_packages/.julia/.cache
### chmod 774 -R /opt/julia_packages/.julia/.cache

# Added:
chmod 770 -R /opt/julia_packages/.julia/lib/v0.4

Removing the commented out chmod statements fixed the dirty git repo and constant recompiling issues.

The added chmod statement is necessary, as Julia by default compiles the .ji files to only be accessible by the file owner. Is there a way to change this?

We still need to test if permissions issues persist when one user adds or updates packages.

[vtjnash]

The .ji files were mostly intended to be user-private, to avoid the inherent race condition with multiple users trying to update the same file.

[adowling2]

Is there a recommended setup for a shared package directory? Should I specify LOAD_CACHE_PATH for each user to avoid sharing .ji files and associated permissions issues?

[stevengj]

@adowling2, normally you would have a shared package directory _and_ per-user .julia directories, and have only one person in charge of updating the shared package directory. When other users do Pkg.add etcetera it would only affect their personal .julia directories.

[adowling2]

@stevengj Can you recommend instructions on configuring Julia to use both a shared and per-user .julia directory?

[stevengj]

@adowling, I think /usr/local/share/julia/site/v0.4 is already in the default LOAD_PATH for julia installed in /usr/local, so why not just install Julia packages there?

Otherwise, install wherever you want and then have users add that directory to their LOAD_PATH in their .juliarc files.

If you want to install shared cache files, too, you'll need to have users add that directory to their Base.LOAD_CACHE_PATH.

[simonbyrne]

This has changed considerably with the new Pkg. If still an issue, please open at https://github.com/JuliaLang/Pkg.jl

[StefanKarpinski]

Seems like we should consider reopening this since it's still an issue (and not something that was changed or otherwise addressed by Pkg3). My proposal would be:

• Look in all $depot/compiled/ for .ji files that can be used
• But only write new .ji files into $(DEPOT_PATH[1])/complied/

This should be independent of which depot the package files are found in.

[hute37]

-rw------- permission is not correct

it should respect umask setting

• 002 if the best choice for umask, that defaults to -rx-rw-r-- for files and -rwxrwxr-x for dirs

To restrict write access to user only, just do not change the standard _"user==default-group"_ policy

To let folders sharable to a group of different users, one can assign group setgid mode to folders:

sudo chown -R root:users JuliaPro-1.5.2-1
sudo chmod -R a+rX,g+w JuliaPro-1.5.2-1                           # all: scan dirs, group: write

sudo find  JuliaPro-1.5.2-1 -type d -exec chmod  g+s {} +                         # group setgid
sudo find  JuliaPro-1.5.2-1 -type f -perm -o+x -exec chmod -c  a+x {} +    # execute on files

---

related to this SO question:

• What's the difference between “s” and “S” in ls -la?