Julia: Ubuntu or at least debian packages make much more sense than Fedora/RHEL/Centos

[vegabook]

https://julialang.org/downloads/platform.html

Surprising: the words Ubuntu and/or Debian aren't mentioned at all.

Generic Linux installers are problematic because they require the user constantly to remember to check/upgrade/remember where installed etc. I see the default snap/apt on 19.04 is Julia 1.04. Shouldn't there be an apt (and possibly yum or rpm) custom repository for later versions?

[StefanKarpinski]

Please open issues on the relevant distributions. We have no control over Ubuntu/Debian/etc. packaging.

[staticfloat]

To give a little bit more context; we used to provide an Ubuntu PPA and such, but it was a lot of extra work, and getting Julia included in the default distribution packaging was much more work than we wanted to do. Distributions have pressures on them to not duplicate libraries (such as LLVM, libuv, etc...) however since Julia requires heavily patched versions of these libraries there is an inherent tension between the JuliaLang project (which requires functionality from LLVM that precludes using a "system-wide" libLLVM) and distribution packagers (which reject packages that want to bundle 'duplicates' of libraries as they don't want to have 30 different versions of the same library on the system).

Both viewpoints are understandable, and while we have found some distributions that are willing to work with us, in the end, there are too many advantages to generic linux tarballs for us to spend much time and effort doing anything else. Tarballs can be unpacked and run without sudo privileges, they are self-contained and do not interfere with other system packages, and they are so simple as to work on all glibc Linux systems (modulo some kernel and glibc requirements).

For more complex arrangements (Such as deb packages for Debian and Ubuntu) we're happy to help someone else work with the distribution packagers to make it happen, but we don't have the bandwidth to devote time and energy to this.

[ViralBShah]

We do have snap packages that may be worth mentioning.

[vegabook]

@ViralBShah Aha! Snap (or Flatpak) packages address my issue and also yours, namely you can bundle your own dependencies and you can go cross platform easily, and I can get the latest version that _also_ updates itself automatically.

@ViralBShah Where_ is this mythical beast to be found? I see the Snap in the Ubuntu Store, which is great. This should probably be mentioned on your download landing page. Similarly if you have flatpak.

[ViralBShah]

This is the official one https://snapcraft.io/julia

@ararslan @JeffBezanson Is the plan to continue maintaining these?

[ViralBShah]

Snap store needs the new logo.

[ararslan]

Is the plan to continue maintaining these?

Not really, no. We put up the Snap for 1.0 since it's a long term support release, will means there will be minimal input required from maintainers. Snap's versioning system is not workable for us (it's effectively nonexistent), and the Snap release process requires a completely separate, parallel, manual process for each Julia version.

So while Snap in some limited way provides a decent solution in that it allows us to bundle the libraries we need, it is just not designed for installing programming languages.

[vegabook]

Okay - I guess I'm not using snap for Elixir, Erlang, so you assertion that snap is no good for languages must be right because those languages don't do snap either, nor ppa. Obviously there is a threshold of usage at which it does become worthwhile, see golang, and Racket seems to have one too. https://launchpad.net/~plt/+archive/ubuntu/racket.

Also a bit confused as to why you have one for the Red Hat distros but nothing for the Debian stuff.

I'm going to take a wild guess and say that some languages prefer some distros.

Anyway thanks for the feedback.

[ararslan]

Also a bit confused as to why you have one for the Red Hat distros but nothing for the Debian stuff.

Anything but what's listed on the downloads page of the website is a community effort and should not be considered official nor endorsed by the Julia project. (The exception is the Snap, which Jeff and I made at the Snapcraft Summit since we were invited.)

[vegabook]

@ararslan I'm sorry I have to re-comment on this. I'm evaluating a bunch of next-gen data science languages, and a stack which does not update itself with apt-get update on Debian or Ubuntu is at an immediate major disadvantage, because these two distros have easily > 50% market share in Desktop linux (and probably more like 80%). I have Tensorflow via Python, the latest developments in Swift, Clojure, Scala, even exotics like Futhark and indeed Intel's SYCL as potential competitors. If Julia doesn't want to cater for the Debian/Ubuntu people this scores heavily against you. For new users Julia is a "tryout" and nobody wants to have to remember constantly to visit the site to download the latest version and re-insert it into /opt or wherever with all the hassle of symlinks etc etc. Now I understand that you may believe Julia to sell ifself so strongly that it overcomes what you may think is a minor hurdle. but this data science space is not static and there are dozens of competitors.

[StefanKarpinski]

Again, please complain to the distros in question and ask that they include correctly built, up to date Julia packages. If any distro does that (they do not so far), we'd be happy to recommend them, but until they do, we cannot. This is not in our control, it is only something that the distros can do, so please complain loudly and often to them and get other people to complain as well.

Often official distro policies make it impossible for them to ship an unbroken Julia. Many distros, for example, insist that Julia use the distro versions of libraries like LLVM, which may be impossible—depending on how old or new the LLVM version they support is because LLVM makes breaking API changes in every release—or subtly broken, because there are unpatched bugs in the official versions of these libraries that we carry patches for when we compile them. We cannot do anything about these policies, only the distros can. In order to make that happen, you, as a distro user must complain to them, not us and demand they change their policies and ship unbroken Julia packages.

[StefanKarpinski]

If Julia doesn't want to cater for the Debian/Ubuntu people this scores heavily against you.

This is not the situation: we have done everything we can to make it possible to ship Julia properly on any platform. The ball is in the court of the distros: they are the ones shipping broken Julia packages, usually because of their own policies. It is possible to build a working Julia on all of these systems, why can't the distros ship working Julia packages? You should ask them this.

[ViralBShah]

@vegabook Being an ex-debian developer, I have always wanted to see this happen.

I do see Julia 1.0.x in stable and testing and 1.1.x in unstable. They don't seem to have llvm as a Debian dependency, which might have addressed the key issue. Do these not work for you?

https://packages.debian.org/julia

Cc @ginggs

[ViralBShah]

Spoke too soon. I see that libjulia depends on the Debian LLVM. I am not sure if the right version with all the required patches is being shipped.

[cdluminate]

We (Debian dev) are definitely aware of the discrepancy between julia's official binary release and Debian/Ubuntu's binary packages. We are trying to eliminate this gap.

Debian stable (buster)'s Julia 1.0.4 package use a vendored llvm source copy, so its LLVM is correct. I dropped it for the julia package in unstable because our final goal is to get julia's LLVM patches merged into the llvm packages. (for llvm 6.0 this is basically done. I've not started checking llvm-7 or llvm-8 yet.)

Another problem that makes Debian's Julia package nearly unusable is the BLAS/LAPACK ecosystem (e.g. try ] add Arpack) -- since the libopenblas64_.so that julia prefer does not exist in Debian. I've already started updating Debian's whole BLAS/LAPACK ecosystem to support the "64-bit-indexing" feature and Julia won't have to ship the openblas and suitesparse source in the future. But the distro development is a bottom-up thing which cannot be done within one night.

[cdluminate]

@vegabook What change with respect to Debian/Ubuntu's julia package would you like to see? Newer (LTS/non-LTS) julia version in what Debian/Ubuntu release?

[staticfloat]

@cdluminate I just want to say that, as the person that worked on the julia-debian project in years gone by, I really appreciate your efforts here. It takes a lot of work, and it's just not something that we can devote that much time to, unfortunately.

[StefanKarpinski]

Yes, thank you @cdluminate, for doing that work. We really look forward to having fully working Julia packages in Debian and greatly appreciate your hard work towards that.

[ViralBShah]

@cdluminate Thanks for the update. It is great to see the progress and commitment from the Debian Science devs. Please do let us know how we can help.

[vegabook]

@cdluminate I am a spoiled and irritating Ubuntu user used to having the entire Cuda or AMD Rocm stack magically update itself ever time I do "apt update && apt upgrade". Now I hear all the arguments but why is Fedora all good on the repostiory page but Ubuntu is not?

https://julialang.org/downloads/platform.html

Why does every other major data science language find the ability to have an up to date repo capability for Ubuntu but Julia goes on about how the distros are to blame? Surely this is true for all of them?

Don't get me wrong I think Julia is a fantastic project but this seems anachronistic and I'm sure a bunch of Debian/Ubuntu people are looking at that page and going "no Ubuntu/Debian? But Fedora/RHEL all over the shop? Okay you guys are biased buh bye"

[ararslan]

Why does every other major data science language find the ability to have an up to date repo capability for Ubuntu but Julia goes on about how the distros are to blame? Surely this is true for all of them?

Most don't rely heavily on very specific versions and patches for their dependencies.

[ViralBShah]

The only reason Fedora is there is that it is maintained by @nalimilan. The only official binaries are the ones on the downloads page. The platform page binaries do not come with the same guarantees as the official tarballs.

The point about why Julia is special is discussed at length. No other dynamic scientific language in Linux distros has a jit compiler and llvm dependency.

Also what Ubuntu ships is their decision. So why keep asking here? The octave project does not build binaries for Ubuntu - the Debian and Ubuntu developers do. Why do you expect it to be different here?

Based on @cdluminate 's comments we should at least link the debian packages on the platforms page.

[cdluminate]

Why does every other major data science language find the ability to have an up to date repo capability for Ubuntu but Julia goes on about how the distros are to blame? Surely this is true for all of them?

Because Julia has special requirements on its dependencies (e.g. openblas, llvm, etc), and major distributions don't yet have all the missing features. Even worse, sometimes such features will even cause distribution design problem, e.g. do you want 7 duplicated libopenblas in Fedora with 7 different SONAMEs: libopenblasp, libopenblass, libopenblaso, libopenblas64p, libopenblas64s, libopenblas64o, libopenblas64_ ? The Debian family could avoid such phenomenon but the development process is still lagging behind.

So, using some features that distributions do not support yet is not a fault. From upstream's point of view, recommending something broken (some distro packages) would be irresponsible.

Don't get me wrong I think Julia is a fantastic project but this seems anachronistic and I'm sure a bunch of Debian/Ubuntu people are looking at that page and going "no Ubuntu/Debian? But Fedora/RHEL all over the shop? Okay you guys are biased buh bye"

To some extent this is inevitable at this stage.

[cdluminate]

Let's track the Debian package issue here: https://github.com/JuliaLang/julia/issues/33208

[srgk26]

I use Arch but was also just looking for Julia in flatpak repos, so came looking for an issue here. Anyway I also found it annoying that I had to keep a lookout to manually update julia regularly cos package managers don't provide correctly built versions (like Arpack issue), so I would now just automate the update process with a simple bash script like this:

## Check julia versions
CURRENT_JULIA_VERSION="$(julia -v | sed 's/julia version //')"
JULIA_VERSION="$(curl -fsSL https://julialang.org/downloads | grep -oP 'Current stable release: v.{0,8}' | sed -n '{s/^.*release: v//; s/[[:space:]].*//p}')"

## Install new version of julia if versions do not match
if [ "$CURRENT_JULIA_VERSION" != "$JULIA_VERSION" ]
then
    rm -rf $HOME/julia-"$CURRENT_JULIA_VERSION" ## Remove current julia installation, or wherever yours is
    wget https://julialang-s3.julialang.org/bin/linux/x64/${JULIA_VERSION:0:3}/julia-${JULIA_VERSION}-linux-x86_64.tar.gz
    tar -xvzf julia-${JULIA_VERSION}-linux-x86_64.tar.gz && rm julia-${JULIA_VERSION}-linux-x86_64.tar.gz
    ln -s julia-${JULIA_VERSION}/bin/julia $HOME/bin/julia ## Link to wherever you want, or add julia binary to path, or however you want it to be found
fi

I've added a hook for my package manager to execute this script whenever I do a system upgrade, so I've never actually upgraded julia manually so far. You could do the same if you want.

@cdluminate I am a spoiled and irritating Ubuntu user used to having the entire Cuda or AMD Rocm stack magically update itself ever time I do "apt update && apt upgrade". Now I hear all the arguments but why is Fedora all good on the repostiory page but Ubuntu is not?

https://julialang.org/downloads/platform.html

Why does every other major data science language find the ability to have an up to date repo capability for Ubuntu but Julia goes on about how the distros are to blame? Surely this is true for all of them?

Don't get me wrong I think Julia is a fantastic project but this seems anachronistic and I'm sure a bunch of Debian/Ubuntu people are looking at that page and going "no Ubuntu/Debian? But Fedora/RHEL all over the shop? Okay you guys are biased buh bye"