Julia: Julia 1.0.3 no longer finds packages in system DEPOT_PATH

[jonathonf]

I have installed a number of modules system-wide under /usr/share/julia/environments/v1.0 using a Linux distribution package https://aur.archlinux.org/packages/julia-ijulia/

With Julia 1.0.2 these packages are detected and loaded, e.g.:

$ strace -f -estat julia
...
julia> using IJulia
[pid 30509] stat("/home/jonathon/.julia/environments/v1.0", 0x7ffdf6e551e0) = -1 ENOENT (No such file or directory)
[pid 30509] stat("/usr/local/share/julia/environments/v1.0", 0x7ffdf6e551e0) = -1 ENOENT (No such file or directory)
[pid 30509] stat("/usr/share/julia/environments/v1.0", {st_mode=S_IFDIR|0755, st_size=242, ...}) = 0
[pid 30509] stat("/usr/share/julia/environments/v1.0/JuliaProject.toml", 0x7ffdf6e551e0) = -1 ENOENT (No such file or directory)
[pid 30509] stat("/usr/share/julia/environments/v1.0/Project.toml", 0x7ffdf6e551e0) = -1 ENOENT (No such file or directory)
[pid 30509] stat("/usr/share/julia/environments/v1.0", {st_mode=S_IFDIR|0755, st_size=242, ...}) = 0
[pid 30509] stat("/usr/share/julia/environments/v1.0/JuliaProject.toml", 0x7ffdf6e55340) = -1 ENOENT (No such file or directory)
[pid 30509] stat("/usr/share/julia/environments/v1.0/Project.toml", 0x7ffdf6e55340) = -1 ENOENT (No such file or directory)
[pid 30509] stat("/usr/share/julia/environments/v1.0/IJulia.jl", 0x7ffdf6e55340) = -1 ENOENT (No such file or directory)
[pid 30509] stat("/usr/share/julia/environments/v1.0/IJulia/src/IJulia.jl", {st_mode=S_IFREG|0644, st_size=12461, ...}) = 0
...

Note that it looks under the path /usr/share/julia/environments/v1.0/IJulia/src/IJulia.jl.

With 1.0.3 this fails:

julia> using IJulia
[pid 17444] stat("/home/jonathon/.julia/environments/v1.0", 0x7ffe0d5fde00) = -1 ENOENT (No such file or directory)
[pid 17444] stat("/usr/local/share/julia/environments/v1.0", 0x7ffe0d5fde00) = -1 ENOENT (No such file or directory)
[pid 17444] stat("/usr/share/julia/environments/v1.0", {st_mode=S_IFDIR|0755, st_size=242, ...}) = 0
[pid 17444] stat("/usr/share/julia/environments/v1.0/JuliaProject.toml", 0x7ffe0d5fde00) = -1 ENOENT (No such file or directory)
[pid 17444] stat("/usr/share/julia/environments/v1.0/Project.toml", 0x7ffe0d5fde00) = -1 ENOENT (No such file or directory)
[pid 17444] stat("/home/jonathon/.julia/environments/v1.0/Project.toml", 0x7ffe0d5fdf80) = -1 ENOENT (No such file or directory)
[pid 17444] stat("/home/jonathon/.julia/environments/v1.0/Project.toml", 0x7ffe0d5fdf80) = -1 ENOENT (No such file or directory)
[pid 17444] stat("/usr/share/julia/stdlib/v1.0", {st_mode=S_IFDIR|0755, st_size=460, ...}) = 0
[pid 17444] stat("/usr/share/julia/stdlib/v1.0/JuliaProject.toml", 0x7ffe0d5fdf80) = -1 ENOENT (No such file or directory)
[pid 17444] stat("/usr/share/julia/stdlib/v1.0/Project.toml", 0x7ffe0d5fdf80) = -1 ENOENT (No such file or directory)
[pid 17444] stat("/usr/share/julia/stdlib/v1.0/IJulia.jl", 0x7ffe0d5fdf80) = -1 ENOENT (No such file or directory)
[pid 17444] stat("/usr/share/julia/stdlib/v1.0/IJulia/src/IJulia.jl", 0x7ffe0d5fdf80) = -1 ENOENT (No such file or directory)
[pid 17444] stat("/usr/share/julia/stdlib/v1.0/IJulia.jl/src/IJulia.jl", 0x7ffe0d5fdf80) = -1 ENOENT (No such file or directory)
ERROR: ArgumentError: Package IJulia not found in current path:
- Run `import Pkg; Pkg.add("IJulia")` to install the IJulia package.

Note that it no longer looks in subdirectories under the path /usr/share/julia/environments/v1.0/.

I am currently investigating https://github.com/JuliaLang/julia/commit/eb8a9333b040b8dc86b2ed7a93bb23e70830cf14 and https://github.com/JuliaLang/julia/commit/71748da03b5a8f9a52fee021570a803e33e309a5 as potentially related but I don't really know if they are.

---

Julia distro package: https://www.archlinux.org/packages/community/x86_64/julia/

julia> versioninfo()
Julia Version 1.0.2
Commit d789231e99 (2018-11-08 20:11 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Core(TM) i7-3630QM CPU @ 2.40GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.0 (ORCJIT, ivybridge)

julia> versioninfo()
Julia Version 1.0.3
Commit 099e826241* (2018-12-18 01:34 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Core(TM) i7-3630QM CPU @ 2.40GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.0 (ORCJIT, ivybridge)

---

Mainly for my own reference, full list of changes between 1.0.2 and 1.0.3: https://github.com/JuliaLang/julia/compare/v1.0.2...v1.0.3

[jonathonf]

After a rebuild of Julia reverting https://github.com/JuliaLang/julia/commit/71748da03b5a8f9a52fee021570a803e33e309a5 the previous operation is restored.

So the question then becomes - is this an intended change in functionality or a regression? @fredrikekre

julia> using IJulia
[pid 22698] stat("/home/jonathon/.julia/environments/v1.0", 0x7fff7d144eb0) = -1 ENOENT (No such file or directory)
[pid 22698] stat("/usr/local/share/julia/environments/v1.0", 0x7fff7d144eb0) = -1 ENOENT (No such file or directory)
[pid 22698] stat("/usr/share/julia/environments/v1.0", {st_mode=S_IFDIR|0755, st_size=242, ...}) = 0
[pid 22698] stat("/usr/share/julia/environments/v1.0/JuliaProject.toml", 0x7fff7d144eb0) = -1 ENOENT (No such file or directory)
[pid 22698] stat("/usr/share/julia/environments/v1.0/Project.toml", 0x7fff7d144eb0) = -1 ENOENT (No such file or directory)
[pid 22698] stat("/usr/share/julia/environments/v1.0", {st_mode=S_IFDIR|0755, st_size=242, ...}) = 0
[pid 22698] stat("/usr/share/julia/environments/v1.0/JuliaProject.toml", 0x7fff7d145010) = -1 ENOENT (No such file or directory)
[pid 22698] stat("/usr/share/julia/environments/v1.0/Project.toml", 0x7fff7d145010) = -1 ENOENT (No such file or directory)
[pid 22698] stat("/usr/share/julia/environments/v1.0/IJulia.jl", 0x7fff7d145010) = -1 ENOENT (No such file or directory)
[pid 22698] stat("/usr/share/julia/environments/v1.0/IJulia/src/IJulia.jl", {st_mode=S_IFREG|0644, st_size=12461, ...}) = 0
...

---

Julia Version 1.0.3
Commit 099e826241* (2018-12-18 01:34 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Core(TM) i7-3630QM CPU @ 2.40GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.0 (ORCJIT, ivybridge)

[jonathonf]

Reading https://github.com/JuliaLang/julia/issues/25709#issuecomment-381198310 it looks like /usr/share/julia/environments/v1.0 should be usable as a system-wide package directory so I _think_ this change has broken expected behaviour (or having a system-wide package directory doesn't work properly yet).

Even though it would work, I don't think putting these packages in @stdlib makes much sense as they're not part of stdlib. Maybe there needs to be a "secondary" system directory, e.g. @packages which could live in /usr/share/julia/packages...

[jonathonf]

Nope, it's because I'm installing the packages to the incorrect location. They should be installed under /usr/share/julia/packages but each package needs to be in a specifically-structured subdirectory, e.g. for JSON.jl: /usr/share/julia/packages/JSON/ebvl3

[jonathonf]

Or not - I cleaned my $HOME/.julia again and now julia is not looking in /usr/share/julia/packages any more.

[cdluminate]

I can reproduce this. The system-wide installation works for 1.0.0 but not 1.0.3.

[cdluminate]

Is this really a correct way to install julia packages system-wide?

    install -dm755 "$pkgdir"/usr/share/julia/environments/v1.0/$_pkgname/
    cp -r {src,test,deps} "$pkgdir"/usr/share/julia/environments/v1.0/$_pkgname/

[jonathonf]

From the reading I've done over the past few months, as far as I can tell there is no "correct" way of doing this. There has been a lot of discussion but no actual final outcome (there are open bugs/discussions from other distro maintainers wanting to know how Julia expects to locate system-wide packages).

The above method was a hack which took advantage of the Julia LOAD_PATH search order to avoid putting non-"standard library" packages in the stdlib directory. Previously, stdlib was named site which matches other language's approaches (e.g. /usr/lib/python3.7/site-packages or /usr/lib/perl6/vendor etc.).

As things stand, there doesn't appear to be a good differentiation between a "DEPOT_PATH" and "LOAD_PATH" as patterns in the LOAD_PATH array are checked for each location in the DEPOT_PATH array, but LOAD_PATH appears to expect a structure matching that in the home depot, "$HOME/.julia" with Project.toml etc. It also expects those package directories to have a (seemingly-arbitrary) subdirectory under the main package name (.julia/packages/JSON/ebvl3/ , .../Primes/uaYlp/) where the package files are actually stored.

However, distribution packages will/should be plain package directories (as far as I can tell, running Pkg.add() does not allow for e.g. reproducible builds), and as things stand Julia doesn't appear to load a plain package directory from anywhere other than under @stdlib - which, as I say, isn't the right place for packages which are not part of the standard library.

Hence, from my naïve view of this, it looks as though Julia should look in an additional location, e.g. @vendor, in the same way as @stdlib.

Or - we just stuff everything from a distro package under @stdlib and be done with it.

[fredrikekre]

I am actually not sure what the correct behavior is, @StefanKarpinski?

[StefanKarpinski]

I need to look more carefully at this, but it certainly should not have broken between 1.0.1 and 1.0.3, so we should figure out what changed and revert it in 1.0.4 at least.

[fredrikekre]

I am pretty confident the old behaviour only worked by accident. In particular it requires DEPOT_PATH[1]/environments/v1.0/Project.toml to _not_ exist, otherwise we don't move on to the system depot here: https://github.com/JuliaLang/julia/blob/4d8a968a87d7edd4586ee901dffa4eaa6856b966/base/initdefs.jl#L162 and DEPOT_PATH[2]/environments/v1.0/Project.toml must not either exist, since that would have been returned instead. It must have been a pretty fragile setup since the moment you add a new package the first file would be created and the system packages would not be found anymore.

What made it work, before 71748da, was that the named v#.# environment simply returned the path instead of a Project.toml file, and then (I think) it was treated as a package directory (e.g. a directory where we look for IJulia/src/IJulia.jl to load) by code loading. I don't think this was the intent for the named environments.

If you want system packages I think the best think is to either include the path to a package directory in LOAD_PATH, or perhaps better yet, include a path to a Project.toml, which makes it easier to control the package versions etc.

Edit: Example with a systempackages environment added to the load path:

$ julia --project=systempackages -e 'using Pkg; Pkg.add("Example")'
[...]

$ export JULIA_LOAD_PATH=:systempackages

$ julia -E 'LOAD_PATH'
["@", "@v#.#", "@stdlib", "systempackages"]

$ julia -E 'Base.load_path()'
["/home/fredrik/.julia/environments/v1.0/Project.toml", "/home/fredrik/julia10/usr/share/julia/stdlib/v1.0", "/home/fredrik/systempackages/Project.toml"]

$ julia -E 'using Example; Example.hello("world")'
"Hello, world"

I like this approach since now the systempackages/Project.toml/systempackages/Manifest.toml defines the system packages just like any other environment, and you have full control over the versions etc there.

[StefanKarpinski]

I don't think this was the intent for the named environments.

Correct, that this worked was a pure accident.

[jonathonf]

Unless/until downstream distros patch an extra load path entry into their respective Julia packages, this means the only remaining approach is to shove system-installed package directories under @stdlib. It feels wrong, but it will at least work for now.

[fredrikekre]

Why can't you modify that env variable yourself?

[StefanKarpinski]

Unless/until downstream distros patch an extra load path entry into their respective Julia packages, this means the only remaining approach is to shove system-installed package directories under @stdlib.

First off, 100% do not do this. I also don't understand the problem. You can install system packages under /usr/share/julia/packages. They won't necessarily be in any environment, but they'll be there and if someone uses those versions, no installation is required. For example, anyone using a manifest with those pre-installed versions doesn't need to install anything. If, on the other hand, someone wants to use other versions they can install alternate versions in their home depot without needing to mess with the preinstalled system versions. You can also provide project and manifest files in /usr/share/julia/environments/v1.0 that includes the pre-installed versions, which will be automatically available to the user since @v#.# is in their default load path.

[jonathonf]

OK - I might very well be missing something about how Julia is supposed to work, so please bear with me...

Why can't you modify that env variable yourself?

I can set it for _me_, but if I package for a distro (or other wider deployment) then I can't set it for every other user. I _could_ make every one of my packages provide an /etc/profile.d file which sets the variable but that's a _lot_ of duplication (unless the distro's julia package provided it).

You can install system packages under /usr/share/julia/packages

These don't appear to be detected or used by default.

Here's a Python use-case:

$ sudo apt-get install python3-pystan
$ python3
>>> import pystan
>>>

Note that I can install the module and use it immediately. This is the case for every other language with system packages (e.g. C, Go, Perl 5, Perl 6, PHP, Python, Ruby, Rust, ...,).

So, here's what I expected to be able to do with Julia:

$ sudo apt-get install julia-ijulia
$ julia
julia> using IJulia
julia> notebook()

That is, I should be able to "use" a package which is installed system-wide (and indeed this was possible using my hacky workaround).

The strace output above shows (or appears to show) that julia, by default, looks only in the following directories:

• $HOME/.julia/environments/v1.0
• /usr/local/share/julia/environments/v1.0
• /usr/share/julia/environments/v1.0
• /usr/share/julia/stdlib/v1.0

and only the last one is checked for "package directories".

A package directory is a directory containing the source trees of a set of packages as subdirectories, and forms an implicit environment. If X is a subdirectory of a package directory and X/src/X.jl exists, then the package X is available in the package directory environment and X/src/X.jl is the source file by which it is loaded.

At no point, at least by default with an empty $HOME/.julia, does it look under /usr/share/julia/packages. While there are references to this in the Julia source I can't see how it is activated - I started tracing through from https://github.com/JuliaLang/julia/blob/master/base/loading.jl#L817 but got lost somewhere along the way.

Pkg.add is not, as far as I can tell, suitable for creating system packages for a few reasons, e.g.:

1. It communicates with an external registry
2. The registry content does not include the "real" source code of the module
3. It alters a Project.toml file independently of the system package manager
4. It automatically includes/installs package dependencies

All of these prevent "reproducible builds" whereby the same result is guaranteed when the distro package is built and rebuilt, and can mean the end result may not be consistent on end-user installations. Hence, installing using sources taken directly from the relevant module repositories is vastly preferred.

So, my "issue" can be distilled down to this:

"Where can I put system-wide package directories so they are found and loaded by default?"

Which may in turn be the wrong question. The right question might well be:

"How should a Linux distro package Julia modules?"

[jschueller]

hi @jonathonf, did you find an answer to "How should a Linux distro package Julia modules?"

[jonathonf]

No. As far as I can tell, there isn't one. Did you find one?

[jschueller]

No, why not leave that one open ?

[StefanKarpinski]

You seem to be a bit confused about environments versus where packages are installed. Unlike these other languages, Julia now separates descriptions of sets of packages (environments) from the actual installed packages (contents of ~/.julia/packages, /usr/share/julia/packages). All of the things you're listing above are environment directories, which is not where packages get installed (unless they are package directories, but that's not the way things should generally work, it mostly exsists for legacy reasons).

Here's how you can install IJulia in a shared system directory:

$ sudo julia
julia> DEPOT_PATH
3-element Array{String,1}:
 "/home/root/.julia"
 "/usr/local/share/julia"
 "/usr/share/julia"

julia> deleteat!(DEPOT_PATH, 1:2)
1-element Array{String,1}:
 "/usr/share/julia"

(v1.2) pkg> status
    Status `/usr/share/julia/environments/v1.0/Project.toml`
  (empty environment)

(v1.2) pkg> add IJulia
   Cloning default registries into `/usr/share/julia`
   Cloning registry from "https://github.com/JuliaRegistries/General.git"
     Added registry `General` to `/usr/share/julia/registries/General`
 Resolving package versions...
 Installed Conda ─────────── v1.1.1
 Installed Compat ────────── v1.4.0
 Installed VersionParsing ── v1.1.3
 Installed IJulia ────────── v1.15.1
 Installed ZMQ ───────────── v1.0.0
 Installed MbedTLS ───────── v0.6.6
 Installed BinaryProvider ── v0.5.3
 Installed JSON ──────────── v0.20.0
 Installed SoftGlobalScope ─ v1.0.8
  Updating `/usr/share/julia/environments/v1.0/Project.toml`
  [7073ff75] + IJulia v1.15.1
  Updating `/usr/share/julia/environments/v1.0/Manifest.toml`
  [b99e7846] + BinaryProvider v0.5.3
  [34da2185] + Compat v1.4.0
  [8f4d0f93] + Conda v1.1.1
  [7073ff75] + IJulia v1.15.1
  [682c06a0] + JSON v0.20.0
  [739be429] + MbedTLS v0.6.6
  [b85f4697] + SoftGlobalScope v1.0.8
  [81def892] + VersionParsing v1.1.3
  [c2297ded] + ZMQ v1.0.0
  [2a0f44e3] + Base64
  [ade2ca70] + Dates
  [8bb1440f] + DelimitedFiles
  [8ba89e20] + Distributed
  [7b1f6079] + FileWatching
  [b77e0a4c] + InteractiveUtils
  [76f85450] + LibGit2
  [8f399da3] + Libdl
  [37e2e46d] + LinearAlgebra
  [56ddb016] + Logging
  [d6f4376e] + Markdown
  [a63ad114] + Mmap
  [44cfe95a] + Pkg
  [de0858da] + Printf
  [3fa0cd96] + REPL
  [9a3f8284] + Random
  [ea8e919c] + SHA
  [9e88b42a] + Serialization
  [1a1011a3] + SharedArrays
  [6462fe0b] + Sockets
  [2f01184e] + SparseArrays
  [10745b16] + Statistics
  [8dfed614] + Test
  [cf7118a7] + UUIDs
  [4ec0a83e] + Unicode
  Building Conda ──→ `/usr/share/julia/packages/Conda/uQitS/deps/build.log`
  Building ZMQ ────→ `/usr/share/julia/packages/ZMQ/ABGOx/deps/build.log`
  Building MbedTLS → `/usr/share/julia/packages/MbedTLS/r1Ufc/deps/build.log`
  Building IJulia ─→ `/usr/share/julia/packages/IJulia/3k6Fz/deps/build.log`

After which, /usr/share/julia/environments/v1.0/Project.toml contains this:

[deps]
IJulia = "7073ff75-c697-5162-941a-fcdaad2a7d2a"

And /usr/share/julia/environments/v1.0/Manifest.toml contains a description of the exact versions of IJulia and all of its dependencies. These are accessible to any user on the system. The packages are installed in /usr/share/julia/packages:

shell> ls -l /usr/share/julia/packages
total 0
drwxr-xr-x 3 root wheel 96 Jan  5 13:35 BinaryProvider
drwxr-xr-x 3 root wheel 96 Jan  5 13:35 Compat
drwxr-xr-x 3 root wheel 96 Jan  5 13:35 Conda
drwxr-xr-x 3 root wheel 96 Jan  5 13:35 IJulia
drwxr-xr-x 3 root wheel 96 Jan  5 13:35 JSON
drwxr-xr-x 3 root wheel 96 Jan  5 13:35 MbedTLS
drwxr-xr-x 3 root wheel 96 Jan  5 13:35 SoftGlobalScope
drwxr-xr-x 3 root wheel 96 Jan  5 13:35 VersionParsing
drwxr-xr-x 3 root wheel 96 Jan  5 13:35 ZMQ

Moreover, even if someone isn't using the above environment file but they are using these versions of these packages, then they can still use them.

I understand that this is different than how other languages do things and I'm familiar with how those work. The new Julia approach will take a little getting used to and is different, but by separating the installation of packages from the description of what versions of packages to use, it actually makes it easier to provide sets of pre-installed packages while still allowing users to augment or modify that set of packages. The tooling for sysadmins does require some work, but normal users have been the focus so far on the presumption that sysadmins are more capable and can do a little bit of manual tweaking if necessary.

If you really want things to work the way they do in older Julias, Perl, Ruby, Python, etc. you can still do that by picking a package directory location and arranging for users to get that in their LOAD_PATH.

[jschueller]

Is it possible to install and precompile packages as user in a custom location so as to use the resulting files in a package (like rpm/debian packages) ?
Building as root user is considered as a bad practice.

[StefanKarpinski]

Sure, you can do it as any user you want to. I just did it that way because the Python example used sudo to install packages, so I followed suit, presuming you wanted the system packages installed as root.

[jonathonf]

Here's how you can install IJulia in a shared system directory:

Yes - I understand that you can run sudo julia and install on a single system for multiple users, similar to sudo pip install. However, this approach is not helpful for distribution packages.

As I've tried to point out, using Pkg.add is not (that I can see) an option for distribution packaging. You've just reinforced the fact that Pkg creates and manages a Project.toml which is non-deterministic: its content depends on which packages are installed.

If I install a distribution package for ijulia, how can the package manager alter the contents of the single Project.toml file in a deterministic way? The package either contains the Project.toml file, which then is incorrect when other packages are installed, or Pkg has to be executed as part of the package update process, and that in turn will download extra data from external sites.

From what I understand so far, requiring a single Project.toml file for modules to be located makes distribution packaging impossible.

So, I think this part is key:

If you really want things to work the way they do in older Julias, Perl, Ruby, Python, etc. you can still do that by picking a package directory location and arranging for users to get that in their LOAD_PATH.

This means that downstream distros will have to patch the LOAD_PATH (or add an environment file setting JULIA_LOAD_PATH) in order to have a sensible, deterministic way of installing packages using their own package manager.

---

Probably the easiest approach is for distributions to add something like this to their julia package:

/etc/profile.d/julia-load-path.sh

export JULIA_LOAD_PATH=":/usr/share/julia/vendor"

and then put the plain "package directories" in there.

[StefanKarpinski]

It’s not recommended to install Julia packages with system package managers. In my experience with other languages I've used, I’ve found that, while you can use a system package manager to install some language packages, it doesn’t work very well and the system package manager never has a comprehensive or up to date set of language packages. Worse still, when you've gone down the path of installing some language packages with the system package manager and later realize that you need a package or version of a package that the system package manager doesn't include, if you try to install something with the language's package manager, then you end up with a confused and broken mix of system-managed and language-managed packages. It's not a good situation. Instead, the natural split is to use the system package manager to install the language and use the language’s package manager to install language-level packages. Then you'll never hit a wall where you need to switch from the system package manager to the language package manager.

You can try to make system language packages work but we’re not going to officially support it. We could discuss adding a system-wide package directory if that will make people happier. On the other hand, I’m not sure it’s something we want to encourage since it’s never going to work as well as Julia’s own package manager and will just end up causing users confusion and problems.

[cdluminate]

From what I understand so far, requiring a single Project.toml file for modules to be located makes distribution packaging impossible.

Impossible unless Pkg.jl supports the merger of two Project.toml/Manifest.toml files. In this way distribution maintainers could setup some postinstall hooks to merge the environments file provided by two different packages together.

This means that downstream distros will have to patch the LOAD_PATH (or add an environment file setting JULIA_LOAD_PATH) in order to have a sensible, deterministic way of installing packages using their own package manager.

Tweaking LOAD_PATH is not a good idea, even if it does not require a depot layout and a clone of generic registry. If I remember correctly putting packages to LOAD_PATH would override package UUID and is likely to break some other package. This basically breaks the design of Pkg.jl

We (Debian Julia Team) have made no decision on whether to create .jl packages for Debian exactly due to the problems @jonathonf pointed out.

[jonathonf]

I'm currently investigating whether including a Project.toml with each package directory would work - if the distribution package can bundle something which is identified with the correct UUID+SHA1=$slug then it should be possible to have a system-wide package which satisfies a certain set of requirements while allowing user-installed packages to satisfy a potentially different set of requirements.

So far, it looks like Pkg is what manages package location by $slug, but I'll dig more into this tomorrow.

I've adjusted my AUR packages to include a Project.toml taken from the Registry, and have begun adding full [deps] to see what happens.

I'm pretty confident there's a good solution to this.

[jschueller]

And what about .so binaries that go along .jl files (using cxxwrap for example), should they be placed together with .jl files ? (in that case it should not be in /usr/share per packaging policies), or is there a dedicated directory to put them ? or just /usr/lib ?

[StefanKarpinski]

If you want to install packages, you just need to put the package contents in the correct /usr/share/julia/packages/$name/$slug location. The slug value is computed here:

https://github.com/JuliaLang/julia/blob/a25945a7c0a52a6853e51f/base/loading.jl#L125-L146

Putting packages there will cause them to be installed but not to be part of any particular environment.

Impossible unless Pkg.jl supports the merger of two Project.toml/Manifest.toml files.

Environments in the LOAD_PATH are merged (or stacked): everything that's available somewhere in the LOAD_PATH is available, with earlier entries taking precedence when determining versions.

Tweaking LOAD_PATH is not a good idea, even if it does not require a depot layout and a clone of generic registry. If I remember correctly putting packages to LOAD_PATH would override package UUID and is likely to break some other package. This basically breaks the design of Pkg.jl

I'm confused about this statement. What's the issue? Putting a package directory somewhere and adding it to the end of every user's LOAD_PATH should work fine.

[StefanKarpinski]

My recommendation is this:

• Install packages into /usr/{,local}/share/julia/packages/$name/$slug depending on whether they are architecture agnostic or specific
• Push @distro onto the end of user's default LOAD_PATH, so their default LOAD_PATH would be ["@", "@v#.#", "@stdlib", "@distro"] instead of just the first three.
• Update /usr/share/julia/environments/distro/{Project,Manifest}.toml as you add/rm distro packages

I'm not sure what the best way to do the modifications of the distro project and manifest files is—can you call Julia's package manager for that? What are the constraints on what you can do while installing or uninstalling packages? Perhaps we should open an issue on Pkg.jl about how distros should package Julia packages.

[jonathonf]

After more experimenting today:

• Adding package directories with a Project.toml into an additional LOAD_PATH directory will allow them to be found by default and identified by UUID. This allows using $name to work in the default environment.

• The slug value can also be computed and the installed package linked in under /usr/share/julia/packages/$name/$slug. This allows Pkg.add($name) to read/bring in the pre-installed package with correct versioning.

I have implemented this for https://aur.archlinux.org/cgit/aur.git/tree/PKGBUILD?h=julia-compat (there are potential improvements for programmatically generating the Project.toml at packaging time based on the Git commit hash to provide reproducibility; unlike Debian, Arch packaging allows the download of external source files).

(_slug() function very much based on @cdluminate's existing work - thank you. :wink: )

---

I haven't yet found a way to generate a [deps] section automatically from the available Registry metadata. Using Pkg might be an option if it has internal API functions but otherwise won't work if external connections are needed (e.g. Debian build infrastructure disallows outgoing connections during builds).

I can do this:

julia> Pkg.TOML.parsefile("Deps.toml")
Dict{String,Any} with 13 entries:
  "0.41-0.68" => Dict{String,Any}("SuiteSparse"=>"4607b0f0-06f3-5cda-b6b1-a6196a1729e9")
  "0.37-1.4"  => Dict{String,Any}("Dates"=>"ade2ca70-3891-5945-98fb-dc099432e06a")
  "1.0-1.4"   => Dict{String,Any}("Statistics"=>"10745b16-79ce-11e8-11f9-7d13ad32a3b2")
...

but haven't yet worked out how to check whether version "1.4.0" is within the range specified by the Dict key String.

[cdluminate]

Tweaking LOAD_PATH is not a good idea, even if it does not require a depot layout and a clone of generic registry. If I remember correctly putting packages to LOAD_PATH would override package UUID and is likely to break some other package. This basically breaks the design of Pkg.jl

I'm confused about this statement. What's the issue? Putting a package directory somewhere and adding it to the end of every user's LOAD_PATH should work fine.

I think the confusion comes from my abnormal usage of LOAD_PATH. As shown below, loading a package from LOAD_PATH where there is no proper depot layout, is not likely taking advantage of UUID or slug, and would potentially break something if another package needs a different version of JSON. Such usage has been used in my past attempt to create .jl Debian packages, and it is exactly what I considered "bad".
(That said, by installing files to /usr/share/julia/... and loading this in the same way as below, we can avoid network access when building debian packages, including cloning the registries.)

⛬ > pushfirst!(LOAD_PATH, ".")
4-element Array{String,1}:
 "."      
 "@"      
 "@v#.#"  
 "@stdlib"
⛬ > using JSON
[ Info: Precompiling JSON [top-level]
⛬ > pathof(JSON)
"/home/lumin/git/JSON/src/JSON.jl"

[cdluminate]

My recommendation is this:

* Install packages into `/usr/{,local}/share/julia/packages/$name/$slug` depending on whether they are architecture agnostic or specific

* Push `@distro` onto the end of user's default `LOAD_PATH`, so their default `LOAD_PATH` would be `["@", "@v#.#", "@stdlib", "@distro"]` instead of just the first three.

* Update `/usr/share/julia/environments/distro/{Project,Manifest}.toml` as you add/rm distro packages

Thanks for the pointers, distribution maintainers need them.

I'm not sure what the best way to do the modifications of the distro project and manifest files is—can you call Julia's package manager for that?

Julia's package manager can be called in post-install hook function to update project and manifest files, if possible. This should be definitely feasible if it can be done without without network access.

What are the constraints on what you can do while installing or uninstalling packages?

I think there is no extra constraints for distribution package managers as long as they do the same thing to the .jl package dependency tree as Pkg.jl does.

Perhaps we should open an issue on Pkg.jl about how distros should package Julia packages.

Yes. This issue is not a good place for further discussion on this topic because of the title.

[jonathonf]

Just for completeness here, I've put together a very naïve solution for generating a Project.toml file based on a named Registry metadata commit.

The main [dep] generation code doesn't look amazing (and I'm sure more experienced Julia programmers will laugh at it) but it seems to work:

using Pkg

alldeps = Pkg.TOML.parsefile(\"$srcdir/$pkgname-Deps.toml\")
version = VersionNumber(\"$pkgver\")
majmin  = VersionNumber(\"${pkgver%.*}\")
deps = Dict{String,Any}()

for (key, value) in alldeps
  vers = split(key, \"-\")
  lower = VersionNumber(vers[1])
  if length(vers) == 2
    upper = VersionNumber(vers[2])
    if (majmin  >= lower && majmin  <= upper) ||
       (version >= lower && version <= upper)
      merge!(deps, value)
    end
  elseif length(vers) == 1
    if majmin == lower || version == lower
      merge!(deps, value)
    end
  end
end

Pkg.TOML.print(deps)

(I've updated my julia-compat package to use this function.)

[nalimilan]

Since @StefanKarpinski said it's not recommended, I'm curious why you find it necessary/useful to provide Julia packages using distribution package managers. Being a Julia packager in Fedora myself, I don't plan to provide any RPMs for Julia packages.

[jonathonf]

I need a way of deploying modules to hundreds of Linux nodes for use by thousands of users. Having each user duplicate the installation of common modules in their personal file stores (for whatever language) is incredibly inefficient.

[StefanKarpinski]

That doesn’t explain why you need to use the system package manager to accomplish that. Presumably there is a shared file system and you can use Julia’s package manager to install the shared packages.

[jonathonf]

I didn't know my answer needed to categorically justify why I want to deploy Julia modules via a distribution package. I'll try and come up with something more expansive which hasn't already been covered. However, you can throw something in there about deployment to laptops.

As things stand, it appears that the methodology I've developed over the last two days for my AUR packages works for Arch-based systems so I'm happy with that. A similar approach should be usable by Debian/Ubuntu systems too, so I'm happy with that too.

[StefanKarpinski]

I'm happy to help figure out how to make system package manager Julia packages work; it should, in fact work much better than with classic language packages since Pkg will allow system-installed packages and language-installed packages to be managed separately but used together seamlessly. The questions are mainly about trying to understand the problem you're trying to solve so that we can avoid an XY problem situation.

[cdluminate]

I did further experiment following @StefanKarpinski 's recommendation https://github.com/JuliaLang/julia/issues/30528#issuecomment-451747228

See https://salsa.debian.org/julia-team/julia-json/tree/master/debian
This package installs the following files to system

/usr/share/julia/environments/distro/Manifest.toml
/usr/share/julia/environments/distro/Project.toml
...
/usr/share/julia/packages/JSON/ebvl3/src/JSON.jl
...

The content of Manifest:

[[JSON]]
deps = ["Dates", "Test", "Sockets", "Mmap", "Unicode", "Distributed"]
git-tree-sha1 = "1f7a25b53ec67f5e9422f1f551ee216503f4a0fa"
uuid = "682c06a0-de6a-54ab-a142-c8b1cf79cde6"
version = "0.20.0"

The content of Project:

[deps]
JSON = "682c06a0-de6a-54ab-a142-c8b1cf79cde6"

but I encountered a strange problem when trying to load JSON

~ ❯❯❯ julia --startup-file=no
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.0.3
 _/ |\__'_|_|_|\__'_|  |  Debian ⛬  julia/1.0.3+dfsg-1
|__/                   |

julia> push!(LOAD_PATH, "@distro")
4-element Array{String,1}:
 "@"      
 "@v#.#"  
 "@stdlib"
 "@distro"

julia> using JSON
[ Info: Precompiling JSON [682c06a0-de6a-54ab-a142-c8b1cf79cde6]
ERROR: LoadError: LoadError: MethodError: Cannot `convert` an object of type Nothing to an object of type Union{UUID, Bool}
Closest candidates are:
  convert(::Type{T}, ::T) where T at essentials.jl:154
Stacktrace:
 [1] explicit_manifest_deps_get(::String, ::Base.UUID, ::String) at ./loading.jl:481
 [2] manifest_deps_get(::String, ::Base.PkgId, ::String) at ./loading.jl:310
 [3] identify_package(::Base.PkgId, ::String) at ./loading.jl:203
 [4] identify_package at ./loading.jl:196 [inlined]
 [5] require(::Module, ::Symbol) at ./loading.jl:818
 [6] include at ./boot.jl:317 [inlined]
 [7] include_relative(::Module, ::String) at ./loading.jl:1044
 [8] include at ./sysimg.jl:29 [inlined]
 [9] include(::String) at /usr/share/julia/packages/JSON/ebvl3/src/JSON.jl:3
 [10] top-level scope at none:0
 [11] include at ./boot.jl:317 [inlined]
 [12] include_relative(::Module, ::String) at ./loading.jl:1044
 [13] include(::Module, ::String) at ./sysimg.jl:29
 [14] top-level scope at none:2
 [15] eval at ./boot.jl:319 [inlined]
 [16] eval(::Expr) at ./client.jl:393
 [17] top-level scope at ./none:3
in expression starting at /usr/share/julia/packages/JSON/ebvl3/src/Common.jl:6
in expression starting at /usr/share/julia/packages/JSON/ebvl3/src/JSON.jl:8
ERROR: Failed to precompile JSON [682c06a0-de6a-54ab-a142-c8b1cf79cde6] to /home/lumin/.julia/compiled/v1.0/JSON/uf6oy.ji.
Stacktrace:
 [1] error(::String) at ./error.jl:33
 [2] compilecache(::Base.PkgId, ::String) at ./loading.jl:1203
 [3] _require(::Base.PkgId) at ./loading.jl:960
 [4] require(::Base.PkgId) at ./loading.jl:858
 [5] require(::Module, ::Symbol) at ./loading.jl:853

julia> 

I'm stuck here.

---

I borrowed and modified some scripts from @jonathonf

[cdluminate]

Here's my preliminary idea about how to make such Debian package work:

1. every julia package ships it's own (private) copy of Manifest.toml and Project.toml.
2. During the post-installation/removal process, some helper script would merge all the private {Manifest,Project}.toml files from all installed julia packages, and rewrite the result to /usr/share/julia/environments/distro/{Manifest,Project}.toml.
3. The dependency information can be imported into the resulting .deb package. Then apt/dpkg will be able to manipulate the dependency tree correctly. (I'm not sure whether this step is easy to implement)

Since in this way Julia loads JSON via UUID, I think it would be better to append slug to the package version, i.e. julia-json_0.20.0~ebvl3-1_all.deb.

[jonathonf]

I don't think there's any need to track the slug as part of the package name as it only changes with either 1) the UUID (which is rarely as it's based on package name) or 2) the SHA1 hash (which I think indicates a compatibility change therefore would also include a version number increment).

My method of an extra LOAD_PATH item combined with a per-module Project.toml appears to work fine. It doesn't appear to (currently?) require a Manifest.toml as this isn't referred to during package loading.

In summary:

1. Add a Project.toml to package (allows identification by UUID and identifies [deps])
2. Add a LOAD_PATH entry to env (allows use of plain package directories)
3. Install package under new LOAD_PATH entry (allows using from default environment)
4. Calculate slug and install package under /usr/share/julia/packages in the recommended way (allows Pkg.add() to find and use pre-installed package instead of downloading)

In this way we don't need to add a "distro" environment to the stack - just/simply the plain packages themselves are available, and there's no need for hooks or manipulation of shared files.

My view is that if a user has a set of packages with specific dependencies they would use a separate environment, but distro packagers would maintain compatible versions within the distro repos - I think that works the same with e.g. pip.

However - I'm not sure what the potential side-effects are of having a system-wide package available if users also want to have an "isolated" environment activated. I don't think it should make any difference if the package isn't "used" or "imported"?

---

It may be that points 2 and 3 aren't really needed, because a user can use Pkg.add() and pull in the pre-installed package in the recommended way. (It's just a cognitive shift from apt-get install -> available to apt-get install -> Pkg.add() -> available.)

I think this is what @StefanKarpinski has been saying all along and I completely missed the point... :sweat:

Although, it may come down to whether Julia modules can/might be used during building other distro packages - if so, Pkg.add won't (currently) work in that context (example, use of Rust requires system-wide crates instead of having cargo download them).

---

Also, @cdluminate's parsing solution looks far nicer than my grep | cut. :)

[cdluminate]

My experimental Debian package worked as expected after adding some missing parts to Manifest.

~ ❯❯❯ rm -rf .julia
~ ❯❯❯ julia --startup-file=no
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.0.3
 _/ |\__'_|_|_|\__'_|  |  Debian ⛬  julia/1.0.3+dfsg-1
|__/                   |

julia> push!(LOAD_PATH, "@distro")
4-element Array{String,1}:
 "@"      
 "@v#.#"  
 "@stdlib"
 "@distro"

julia> using JSON
[ Info: Precompiling JSON [682c06a0-de6a-54ab-a142-c8b1cf79cde6]

julia> 

Patching julia interpreter's LOAD_PATH for distro should be quite easy. Once that's done, such
package would become an apt-install julia-json -> using JSON solution. @jonathonf

The next issue is how to merge Manifest and Project TOML files from different packages, for instance julia-json and julia-zmq. And how to generate correct Project/Manifest gracefully instead of manually.

/usr/share/julia/environments/distro/Manifest.toml
/usr/share/julia/environments/distro/Project.toml

---

Here's an untested patch which modifies default LOAD_PATH

diff --git a/base/initdefs.jl b/base/initdefs.jl
index f95c22b4..07c0ed50 100644
--- a/base/initdefs.jl
+++ b/base/initdefs.jl
@@ -70,7 +70,7 @@ end
 # this will inherit an existing JULIA_LOAD_PATH value or if there is none, leave
 # a trailing empty entry in JULIA_LOAD_PATH which will be replaced with defaults.

-const DEFAULT_LOAD_PATH = ["@", "@v#.#", "@stdlib"]
+const DEFAULT_LOAD_PATH = ["@", "@v#.#", "@stdlib", "@distro"]

 """
     LOAD_PATH

[StefanKarpinski]

However - I'm not sure what the potential side-effects are of having a system-wide package available if users also want to have an "isolated" environment activated. I don't think it should make any difference if the package isn't "used" or "imported"?

This is totally safe. Since package versions are identified by tree hash in manifest files, and each (name, uuid + tree) has a unique installation location, if you ask for a particular version of a package, you cannot possibly get anything else (unless someone has installed a different version at the location where that version should exist). This is why I said that having some packages managed by the distro and others managed by the user will work more smoothly with Pkg than with other language package managers: the distro can provide any set of package versions it wants without messing with what the user uses. Note that the system can, if it wants to, provide multiple different versions of the same Julia package, in case some set of users will want one version while a different set may want another one.

It may be that points 2 and 3 aren't really needed, because a user can use Pkg.add() and pull in the pre-installed package in the recommended way.

Yes, it's not really needed. However, it might be nice for it to be in the default LOAD_PATH so that the user can do apt-get install Julia-Primes and immediately be able to do using Primes from inside of a normal Julia prompt. Otherwise they would need to do apt-get install Julia-Primes to get the system package, then pkg> add Primes to pick up the system-installed Primes package and add it to their current environment, and only then be able to do using Primes. If they have a particular version of Primes in an environment that's earlier in the LOAD_PATH, that will completely occlude the one in the @depot environment, so doing this won't mess with any of their projects either.

The one thing that's still kind of a problem for all of this is that we haven't yet taught Julia's resolver to prefer already-available packages when resolving packages so if someone does do apt-get install Julia-Primes and then pkg> add Primes in some environment other than @depot which apt-get is managing, then the resolver may decide to download and install a newer version of Primes that is not already installed. We'll have to work on that when the version resolver gets revamped.

[StefanKarpinski]

The next issue is how to merge Manifest and Project TOML files from different packages, for instance julia-json and julia-zmq. And how to generate correct Project/Manifest gracefully instead of manually.

You should be able to use env = Pkg.Types.EnvCache("/usr/share/julia/environments/distro") to load the old version of an environment and then env.project.deps and env.manifest are dictionaries that you can manipulate:

julia> env.project.deps
Dict{String,Base.UUID} with 2 entries:
  "IJulia"       => UUID("7073ff75-c697-5162-941a-fcdaad2a7d2a")
  "UnicodePlots" => UUID("b8865327-cd53-5732-bb35-84acbb429228")

julia> env.manifest
Dict{Base.UUID,Pkg.Types.PackageEntry} with 40 entries:
  UUID("8bb1440f-4735-579b-a4… => PackageEntry("DelimitedFiles", nothing, nothing, false…
  UUID("7b1f6079-737a-58dc-b8… => PackageEntry("FileWatching", nothing, nothing, false, …
  UUID("81def892-9a0e-5fdd-b1… => PackageEntry("VersionParsing", "1.1.3", nothing, false…
  ⋮
  UUID("4ec0a83e-493e-50e2-b9… => PackageEntry("Unicode", nothing, nothing, false, GitRe…
  UUID("ade2ca70-3891-5945-98… => PackageEntry("Dates", nothing, nothing, false, GitRepo…
  UUID("8f4d0f93-b110-5947-80… => PackageEntry("Conda", "1.1.1", nothing, false, GitRepo…

julia> first(env.manifest)
UUID("8bb1440f-4735-579b-a4ab-409b98df4dab") => PackageEntry("DelimitedFiles", nothing, nothing, false, GitRepo(nothing, nothing, nothing), Dict("Mmap"=>UUID("a63ad114-7e13-5084-954f-fe012c677804")), Dict{String,Any}("deps"=>["Mmap"],"uuid"=>"8bb1440f-4735-579b-a4ab-409b98df4dab"))

If you merge the corresponding dictionaries between two different system packages that should give you the write data structure—and if the system package manager is doing its version selection correctly, then any key collisions should always contain identical data. Then you should be able to write that merged data structure back out to the original location by doing

Pkg.Types.write_env(Pkg.Types.Context(env=env))

We could work on better external APIs for this; so far this is all stuff that's internal to Pkg.

[cdluminate]

You should be able to use env = Pkg.Types.EnvCache("/usr/share/julia/environments/distro") to load the old version of an environment and then env.project.deps and env.manifest are dictionaries that you can manipulate:

julia> env = Pkg.Types.EnvCache("/usr/share/julia/environments/distro")
ERROR: environment is a package directory: /usr/share/julia/environments/distro
Stacktrace:
 [1] pkgerror(::String) at /build/julia-1WzxVp/julia-1.0.3+dfsg/usr/share/julia/stdlib/v1.0/Pkg/src/Types.jl:120

After checking the code I think this is the correct usage:

julia> env = Pkg.Types.EnvCache("@distro").project
Dict{String,Any} with 1 entry:
  "deps" => Dict{String,Any}("JSON"=>"682c06a0-de6a-54ab-a142-c8b1cf79cde6")

If you merge the corresponding dictionaries between two different system packages that should give you the write data structure—and if the system package manager is doing its version selection correctly, then any key collisions should always contain identical data. Then you should be able to write that merged data structure back out to the original location by doing

Neat! Let me try it out.

We could work on better external APIs for this; so far this is all stuff that's internal to Pkg.

I'm thinking of creating a Distrohelper.jl package (or a Pkg submodule with the same name) so that any distribution that wants to create julia packages can take advantage from this unified solution, and won't have to maintain helper code by themselves. @jonathonf Interested? (Although I have no idea how many julia users would like to install julia packages with apt, pacman or alike.)

[StefanKarpinski]

What's in /usr/share/julia/environments/distro when you do this? You may need to pass it the path to the project file instead, i.e. /usr/share/julia/environments/distro/Project.toml.

The more I think about it the more I'm warming to allowing distros to install and manage a layer of Julia packages. In other languages, it's a big mess because the system package manager and the language package manager end up fighting, but the Pkg design makes that not a problem, so why not allow people to install fallback package versions with system package managers?

[cdluminate]

What's in /usr/share/julia/environments/distro when you do this?

~ ❯❯❯ find /usr/share/julia/environments/
/usr/share/julia/environments/
/usr/share/julia/environments/distro
/usr/share/julia/environments/distro/Project.toml
/usr/share/julia/environments/distro/Manifest.toml
~ ❯❯❯ 

You may need to pass it the path to the project file instead, i.e. /usr/share/julia/environments/distro/Project.toml.

All of them work:

Pkg.Types.EnvCache("/usr/share/julia/environments/distro/Package.toml")
Pkg.Types.EnvCache("/usr/share/julia/environments/distro/Manifest.toml")
Pkg.Types.EnvCache("@distro")

The more I think about it the more I'm warming to allowing distros to install and manage a layer of Julia packages.

With your help it's much easier for us downstream to dig into this.

In other languages, it's a big mess because the system package manager and the language package manager end up fighting, but the Pkg design makes that not a problem, so why not allow people to install fallback package versions with system package managers?

Since Julia allows multiple versions of the same package co-exist, I fear of some kind of version hell (just like javascript) would emerge when the ecosystem of jl-deb packages becomes large enough ... e.g. this javascript package has a terrible version hell 6.0.2+20181021git007b08d01eff070+ds+~0.3.1+~4.0.0+~0.3.0+~5.0.0+ds+~1.6.1+ds-2 ...
I don't fully understand the design of Pkg.jl , do we have chance to avoid such pathetic situation?

[StefanKarpinski]

Since Julia allows multiple versions of the same package co-exist, I fear of some kind of version hell (just like javascript) [...] I don't fully understand the design of Pkg.jl , do we have chance to avoid such pathetic situation?

The Julia Pkg situation is very different than JavaScript. In JavaScript (npm really, but on the server it's the same thing), each package gets its own version of each dependency, so there are many different copies of each dependency. In Julia, that's not the case. In a given environment, there is only a single version of any given package. You can have multiple different environments which use different versions. Thus, you can have one project which uses an old version of some package and a different project which uses a newer version of that same package. Both versions are installed at the same time but you never use both at the same time. Distros should do what they do with other languages: resolve a single coherent set of versions for Julia packages.

[cdluminate]

My starter code can be found here https://github.com/cdluminate/DistroHelper.jl

[cdluminate]

@jonathonf Please check my repo https://github.com/cdluminate/DistroHelper.jl , which could help you get rid of all julia code (slug and deps) and shell code (Project.toml and Manifest.toml) in PKGBUILD. My julia-json example is also updated.

---

The code for Project/Manifest merger is still WIP

[jonathonf]

I've updated my julia-ijulia package to use your much nicer helper.

I did add in an extra function to generate a full Project.toml (I'm not convinced about a distro Manifest.toml but then I'm not that deep into Julia); I haven't opened a PR as I don't think you want/need that for Debian.

[cdluminate]

I've updated my julia-ijulia package to use your much nicer helper.

I did add in an extra function to generate a full Project.toml (I'm not convinced about a distro Manifest.toml but then I'm not that deep into Julia); I haven't opened a PR as I don't think you want/need that for Debian.

Feel free to submit a PR. That helper is not specific to Debian.