Julia: Loading CSV makes tab completion in the REPL extremely slow

[KristofferC]

Doing using CSV and then foo("<TAB in the REPL makes the REPL lag for ~10 seconds. Running with --trace-compile and timing the different precompile statement there is one that takes 10(!) seconds to run. MWE is:

julia> using CSV; using REPL; @time precompile(Tuple{typeof(REPL.REPLCompletions.completions), String, Int64})
 10.879856 seconds (25.56 M allocations: 1.508 GiB, 3.41% gc time)

Entering the Pkg REPL also takes like 20 seconds after using CSV.

[laborg]

See also: https://github.com/JuliaData/CSV.jl/issues/537

[KristofferC]

Weird, on my 1.3.1 backport branch built locally it only takes 0.4 seconds... But downloading the binary from the buildbot with the same commit, it takes 30 seconds!

[KristofferC]

Ok, Jameson helped me figured that one out. The binary from the buildbot has asserts on so it is slower (significantly so in this particular case).

[KristofferC]

Ok, this seems fixed on 1.3.1 and master so I guess good to close 🤷‍♂ .

[evanfields]

@KristofferC I'm still seeing this behavior on 1.3.1 (macOS). Is this just an inevitable consequence of asserts always being enabled?

[KristofferC]

I'll try it again

[KristofferC]

Yes, this does indeed happen on 1.3.1 release. Strange, maybe julia asserts are accidentally enabled for releases(!)?

[KristofferC]

And now it seems to happen when building 1.3.1 locally as well so maybe it isn't about asserts. I have no idea what is going on, would be good for someone else to do some testing, perhaps I am just crazy.

[KristofferC]

Still happens in 1.4 RC1.

julia> using CSV; using REPL; @time precompile(Tuple{typeof(REPL.REPLCompletions.completions), String, Int64})
 49.933756 seconds (33.86 M allocations: 2.063 GiB, 2.78% gc time)
true

(different machine so numbers not directly comparable).

[timholy]

Possibly related: https://discourse.julialang.org/t/slowdown-after-package-load/33533

The weirdest part is that using MCMCChains on its own causes the problem, but if I first load all of MCMCChain's dependencies:

julia> using AbstractMCMC, AxisArrays, DataFrames, Distributions, KernelDensity, LinearAlgebra, Random, RecipesBase, Serialization, Showoff, SpecialFunctions, Statistics, StatsBase

julia> using MCMCChains

then it's fine.

[KristofferC]

Yeah, I have also noticed differences depending on order of package loads. The weird thing here is that I had julia builds when the case here didn't exist at all so there seems to be some pathological case that causes this to happen.

[KristofferC]

Some kind of lead? https://github.com/JuliaData/CSV.jl/issues/537#issuecomment-578439163

Can anyone confirm that commenting out this join method:
https://github.com/JuliaData/DataFrames.jl/blob/e8e5cb5f40fbafa46927c19c00270f6ea3caff74/src/abstractdataframe/join.jl#L309-L431
makes the problem pretty much go away?

In addition, the same is true for the join in FilePathsBase (commenting out the AbstractString argument type in the splatted argument also seems to fix it):
https://github.com/rofinn/FilePathsBase.jl/blob/37ce645ca40de031ac7fd7f05274bfdd1f1edd69/src/path.jl#L230-L242

So it seems that the join methods are interacting with eachother?

[JeffBezanson]

This is really fragile. For example adding the commit from #34473 "fixes" it, but it's very unclear why. All the time is in inference.

Just for fun, #34339 speeds this up by ~8%, so it seems to help a bit more in pathological cases than in the average case.

[robsmith11]

For me with the latest nightly, this is fixed on my Skylake Xeon, but on an AArch64 tablet, I'm still seeing a >1 minute delay for the first tab completion after loading CSV.jl. Same versions of Julia and packages on both systems.