Julia: Dependency tree of stdlibs

[KristofferC]

Below is the dependency graph of our stdlib:

There are a few unfortunate dependencies. One for example is the Test --> Distributed --> LinearAlgebra, which means that you cannot use the Test stdlib without having the LinearAlgebra stdlib.
Perhaps some refactorings to avoid this would be useful. Test uses only myid from Distributed for example.

Figure created with:

cd("/Users/kristoffer/julia/stdlib")
import TOML
using LightGraphs
using TikzGraphs
using TikzPictures

const STDLIB_DIR = "."
const STDLIBS = readdir(STDLIB_DIR)
deps = Dict{String, Int}()
for (i, stdlib) in enumerate(STDLIBS)
    deps[stdlib] = i
end
g = LightGraphs.SimpleGraphs.SimpleDiGraph(length(STDLIBS))

for (i, stdlib) in enumerate(STDLIBS)
    proj = TOML.parsefile(joinpath(STDLIB_DIR, stdlib, "Project.toml"))
    if haskey(proj, "deps")
        for (stdlib_dep, uuid) in proj["deps"]
            add_edge!(g, deps[stdlib], deps[stdlib_dep])
        end
    end
end
t = TikzGraphs.plot(g, Layouts.SimpleNecklace(), STDLIBS)
TikzPictures.save(SVG("graph"), t)  

[JeffBezanson]

Nice, +1 to breaking dependency edges.

Distributed only uses LinearAlgebra in order to call BLAS.set_num_threads(1). That could be replaced with some kind of hook mechanism instead, so all threaded libraries can register callbacks to do the same thing.

Another one I noticed: SharedArrays uses a single sprintf call that could easily be replaced.

[Keno]

I also just noticed we have a cycle: ["REPL", "InteractiveUtils", "Pkg"], Would be nice to break if possible.

[fredrikekre]

Pkg -> REPL -> InteractiveUtils -> LinearAlgebra makes it impossible to use the package manager without BLAS :stuck_out_tongue:

[fredrikekre]

Updated version of the tree

[StefanKarpinski]

Is it actually a DAG now?

[rschwarz]

GraphViz/dot says yes:

(see gist)

Sorry, my analysis applies to the 1.0.2 release, of course, not some branch where refactoring is happening.

[KristofferC]

Nice graph!

[StefanKarpinski]

Yes, very nice graph! This dependency graph is starting to look a bit more reasonable.

[KristofferC]

Test -> Distributed, InteractiveUtils -> LinearAlgebra, Distributed -> LinearAlgebra are probably the worst ones right now.

[fredrikekre]

Fortunately https://github.com/JuliaLang/julia/pull/29978 removed InteractiveUtils -> LinearAlgebra and https://github.com/JuliaLang/julia/pull/30004 removes Distributed -> LinearAlgebra.

[fredrikekre]

Update after #29978 and #30004 :

In particular LinearAlgebra is now pretty isolated to packages that are actually linalg related and e.g. the package manager is not dependent on LinearAlgebra anymore :tada:

[KristofferC]

Very good, this looks quite reasonable.

[StefanKarpinski]

Wow, that's such a nice, sane dependency graph now!

[Sacha0]

Fantastic work! ❤️ 🎉