Julia: serializing closures is slow on master

[amitmurthy]

function foo(n)
    io=IOBuffer()
    f = x->x
    @time for i in 1:10^n
        serialize(io, f)
        seekstart(io)
        deserialize(io)
    end
end

function bar(n)
    io=IOBuffer()
    @time for i in 1:10^n
        serialize(io, "Hello World")
        seekstart(io)
        deserialize(io)
    end
end

foo(1)
bar(1)
foo(4)
bar(4)

On 0.4

julia> foo(4)
  0.208748 seconds (680.01 k allocations: 35.706 MB, 1.70% gc time)

julia> bar(4)
  0.010271 seconds (100.00 k allocations: 9.003 MB, 24.10% gc time)

On master

julia> foo(4)
  3.409323 seconds (9.96 M allocations: 541.994 MB, 1.60% gc time)

julia> bar(4)
  0.012568 seconds (80.06 k allocations: 9.616 MB, 19.17% gc time)

@andreasnoack, your hunch was correct, this looks like the cause of the slowdown in distributed arrays - https://travis-ci.org/JuliaParallel/DistributedArrays.jl/builds/131800408

[amitmurthy]

cc: @shashi could also explain the slowdown you are seeing in ComputeFramework.

[ViralBShah]

This is important for 0.5.

[StefanKarpinski]

https://github.com/JuliaLang/julia/pull/16695 helps but is kind of a workaround?

[amitmurthy]

I'll leave this issue open as #16774 optimized it only for parallel processing - and still not upto 0.4 levels. Will remove the 0.5 tag though, because, as discussed in #16774, this is the best for now.

[ExpandingMan]

Not completely sure it's related to this, but deserializing large objects (in this case DataFrames) from disk seems to take significantly longer on 0.5 (took me 9 seconds for a 38 MB dataframe). I'm afraid I can't give my 0.4 number because I replaced the 0.4 build, but I'm quite sure it was much faster (maybe 1 s at most?). Serialization, on the other hand, seems fine.

I did some further testing and it seems that this depends strongly on what the stored datatypes are. I had a 300MB dataframe of floats load in 1s, but 38MB of mixed strings, DateTime and Float takes forever (9s).