Julia: Serialization of nested anonymous functions.

Created on 30 Nov 2016  路  14Comments  路  Source: JuliaLang/julia 
julia> c = x->x
(anonymous function)

julia> d = x->c(x)
(anonymous function)

On 0.4:

julia> pmap(d, 1:2)
2-element Array{Any,1}:
 1
 2

On 0.5:

julia> pmap(d, 1:2)
ERROR: On worker 2:
UndefVarError: c not defined
 in #3 at ./REPL[2]:1

Note that pmap(c, 1:2) works.

Making them local results in a different error:

julia> function wrapper()
           c = x->x
           d = x->c()
           pmap(d, 1:2)
       end
wrapper (generic function with 1 method)

julia> wrapper()
ERROR: On worker 2:
MethodError: no method matching (::Base.Serializer.__deserialized_types__.##5#7)()
bug parallel regression
Source
picture amitmurthy

All 14 comments

cc: @JeffBezanson , @vtjnash

amitmurthy picture amitmurthy on 30 Nov 2016

Any idea if this can be fixed?

bdeonovic picture bdeonovic on 11 Dec 2016

I am pretty sure it can be. Waiting on Jeff or Jameson as they are most familiar with the serialization codebase - unless someone else takes a shot it first.

amitmurthy picture amitmurthy on 11 Dec 2016

Still no ETA @JeffBezanson , @vtjnash ?

bdeonovic picture bdeonovic on 12 Jan 2017

Soon. https://github.com/JuliaLang/julia/pull/19594 will fix this.

amitmurthy picture amitmurthy on 12 Jan 2017

Hmmm, spoke a bit too soon. #19594 fixes the issue when the closures are global. It does not address the second case, i.e. local closures.

amitmurthy picture amitmurthy on 12 Jan 2017

My bad. The wrapper example above has an error. It should be 

function wrapper()
           c = x->x
           d = x->c(x)
           pmap(d, 1:2)
       end

In fact it works on 0.5 too.

amitmurthy picture amitmurthy on 13 Jan 2017

How about:

julia> function wrapper(y)
                  c = x->x
                  d = x->c(x)
                  d(y)
       end

julia> pmap(wrapper,1:2)
ERROR: On worker 2:
UndefVarError: #wrapper not defined
 in deserialize_datatype at ./serialize.jl:823
 in handle_deserialize at ./serialize.jl:571
 in deserialize_msg at ./multi.jl:120
 in message_handler_loop at ./multi.jl:1317
 in process_tcp_streams at ./multi.jl:1276
 in #618 at ./event.jl:68
 in #remotecall_fetch#606(::Array{Any,1}, ::Function, ::Function, ::Base.Worker, ::Int64, ::Vararg{Int64,N}) at ./multi.jl:1070
 in remotecall_fetch(::Function, ::Base.Worker, ::Int64, ::Vararg{Int64,N}) at ./multi.jl:1062
 in #remotecall_fetch#609(::Array{Any,1}, ::Function, ::Function, ::Int64, ::Int64, ::Vararg{Int64,N}) at ./multi.jl:1080
 in remotecall_fetch(::Function, ::Int64, ::Int64, ::Vararg{Int64,N}) at ./multi.jl:1080
 in #remotecall_pool#689(::Array{Any,1}, ::Function, ::Function, ::Function, ::WorkerPool, ::Int64, ::Vararg{Int64,N}) at ./workerpool.jl:93
 in remotecall_pool(::Function, ::Function, ::WorkerPool, ::Int64, ::Vararg{Int64,N}) at ./workerpool.jl:91
 in #remotecall_fetch#692(::Array{Any,1}, ::Function, ::Function, ::WorkerPool, ::Int64, ::Vararg{Int64,N}) at ./workerpool.jl:124
 in remotecall_fetch(::Function, ::WorkerPool, ::Int64, ::Vararg{Int64,N}) at ./workerpool.jl:124
 in (::Base.###697#698#700{WorkerPool,#wrapper})(::Array{Any,1}, ::Function, ::Int64, ::Vararg{Int64,N}) at ./workerpool.jl:151
 in (::Base.##697#699)(::Int64, ::Vararg{Int64,N}) at ./workerpool.jl:151
 in macro expansion at ./asyncmap.jl:63 [inlined]
 in (::Base.##755#757{Base.AsyncCollector,Base.AsyncCollectorState})() at ./task.jl:360

...and 1 other exceptions.

 in sync_end() at ./task.jl:311
 in done(::Base.AsyncCollector, ::Base.AsyncCollectorState) at ./asyncmap.jl:124
 in pump_source(::Base.AsyncGenerator, ::Base.AsyncGeneratorState) at ./asyncmap.jl:185
 in next(::Base.AsyncGenerator, ::Base.AsyncGeneratorState) at ./asyncmap.jl:201
 in _collect(::UnitRange{Int64}, ::Base.AsyncGenerator, ::Base.HasEltype, ::Base.SizeUnknown) at ./array.jl:282
 in #pmap#714(::Bool, ::Int64, ::Void, ::Int64, ::Float64, ::Function, ::Void, ::Void, ::Void, ::Base.#pmap, ::WorkerPool, ::#wrapper, ::UnitRange{Int64}) at ./pmap.jl:121
 in pmap(::WorkerPool, ::Function, ::UnitRange{Int64}) at ./pmap.jl:80
 in #pmap#726(::Array{Any,1}, ::Function, ::Function, ::UnitRange{Int64}) at ./pmap.jl:146
 in pmap(::Function, ::UnitRange{Int64}) at ./pmap.jl:146
bdeonovic picture bdeonovic on 13 Jan 2017

That is a different error since wrapper is not defined on the remote worker. Use @everywhere and you should be good to go.

@everywhere function wrapper(y)
                  c = x->x
                  d = x->c(x)
                  d(y)
       end
amitmurthy picture amitmurthy on 13 Jan 2017

I guess what I'm trying to get at here is how does this relate to the error we are having in Mamba (https://github.com/brian-j-smith/Mamba.jl/issues/109)

bdeonovic picture bdeonovic on 13 Jan 2017

I believe it is the issue of global closures not being serialized which #19594 addresses.

amitmurthy picture amitmurthy on 14 Jan 2017
🎉1

Is this failure the local case?

The following works in Julia v0.5.x but is broken in 0.6.0-rc1 and latest 0.7 binaries.

julia> y = (x) -> (() -> x)
(::#1) (generic function with 1 method)

julia> g = y(1)

julia> g()
1

julia> io = IOBuffer()
IOBuffer(data=UInt8[...], readable=true, writable=true, seekable=true, append=fa
lse, size=0, maxsize=Inf, ptr=1, mark=-1)

julia> serialize(io, y)

julia> seekstart(io)
IOBuffer(data=UInt8[...], readable=true, writable=true, seekable=true, append=fa
lse, size=316, maxsize=Inf, ptr=1, mark=-1)

julia> deserialize(io) # crashes

ERROR: type UnionAll has no field name
Stacktrace:
 [1] handle_deserialize(::SerializationState{Base.AbstractIOBuffer{Array{UInt8,1
}}}, ::Int32) at .\serialize.jl:706
 [2] deserialize(::SerializationState{Base.AbstractIOBuffer{Array{UInt8,1}}}) at
 .\serialize.jl:634
 [3] deserialize_expr(::SerializationState{Base.AbstractIOBuffer{Array{UInt8,1}}
}, ::Int64) at .\serialize.jl:881
 [4] handle_deserialize(::SerializationState{Base.AbstractIOBuffer{Array{UInt8,1
}}}, ::Int32) at .\serialize.jl:693
 [5] deserialize(::SerializationState{Base.AbstractIOBuffer{Array{UInt8,1}}}) at
 .\serialize.jl:634
 [6] deserialize_expr(::SerializationState{Base.AbstractIOBuffer{Array{UInt8,1}}
}, ::Int64) at .\serialize.jl:881
 [7] handle_deserialize(::SerializationState{Base.AbstractIOBuffer{Array{UInt8,1
}}}, ::Int32) at .\serialize.jl:693
 [8] deserialize(::SerializationState{Base.AbstractIOBuffer{Array{UInt8,1}}}) at
 .\serialize.jl:634
 [9] deserialize_expr(::SerializationState{Base.AbstractIOBuffer{Array{UInt8,1}}
}, ::Int64) at .\serialize.jl:881
 [10] handle_deserialize(::SerializationState{Base.AbstractIOBuffer{Array{UInt8,
1}}}, ::Int32) at .\serialize.jl:693
 [11] deserialize_array(::SerializationState{Base.AbstractIOBuffer{Array{UInt8,1
}}}) at .\serialize.jl:870
 [12] handle_deserialize(::SerializationState{Base.AbstractIOBuffer{Array{UInt8,
1}}}, ::Int32) at .\serialize.jl:672
 [13] deserialize(::SerializationState{Base.AbstractIOBuffer{Array{UInt8,1}}}, :
:DataType) at .\serialize.jl:1074
 [14] handle_deserialize(::SerializationState{Base.AbstractIOBuffer{Array{UInt8,
1}}}, ::Int32) at .\serialize.jl:722
 [15] deserialize(::SerializationState{Base.AbstractIOBuffer{Array{UInt8,1}}}) a
t .\serialize.jl:634
 [16] deserialize(::SerializationState{Base.AbstractIOBuffer{Array{UInt8,1}}}, :
:Type{Method}) at .\serialize.jl:781
 [17] handle_deserialize(::SerializationState{Base.AbstractIOBuffer{Array{UInt8,
1}}}, ::Int32) at .\serialize.jl:722
 [18] deserialize_array(::SerializationState{Base.AbstractIOBuffer{Array{UInt8,1
}}}) at .\serialize.jl:870
 [19] handle_deserialize(::SerializationState{Base.AbstractIOBuffer{Array{UInt8,
1}}}, ::Int32) at .\serialize.jl:672
 [20] deserialize(::SerializationState{Base.AbstractIOBuffer{Array{UInt8,1}}}) a
t .\serialize.jl:634
 [21] deserialize_typename(::SerializationState{Base.AbstractIOBuffer{Array{UInt
8,1}}}, ::UInt64) at .\serialize.jl:937
 [22] deserialize(::SerializationState{Base.AbstractIOBuffer{Array{UInt8,1}}}, :
:Type{TypeName}) at .\serialize.jl:890
 [23] handle_deserialize(::SerializationState{Base.AbstractIOBuffer{Array{UInt8,
1}}}, ::Int32) at .\serialize.jl:722
 [24] deserialize(::SerializationState{Base.AbstractIOBuffer{Array{UInt8,1}}}) a
t .\serialize.jl:634
 [25] handle_deserialize(::SerializationState{Base.AbstractIOBuffer{Array{UInt8,
1}}}, ::Int32) at .\serialize.jl:678
 [26] deserialize(::SerializationState{Base.AbstractIOBuffer{Array{UInt8,1}}}) a
t .\serialize.jl:634
 [27] handle_deserialize(::SerializationState{Base.AbstractIOBuffer{Array{UInt8,
1}}}, ::Int32) at .\serialize.jl:681
 [28] deserialize(::SerializationState{Base.AbstractIOBuffer{Array{UInt8,1}}}) a
t .\serialize.jl:634
 [29] deserialize(::Base.AbstractIOBuffer{Array{UInt8,1}}) at .\serialize.jl:631

julia> versioninfo()
Julia Version 0.7.0-DEV.67
Commit d214d57* (2017-05-09 15:12 UTC)
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz
  WORD_SIZE: 64
  BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Haswell)
  LAPACK: libopenblas64_
  LIBM: libopenlibm
  LLVM: libLLVM-3.9.1 (ORCJIT, haswell)
odow picture odow on 10 May 2017

Probably introduced in https://github.com/JuliaLang/julia/pull/21514 .

cc: @JeffBezanson

amitmurthy picture amitmurthy on 10 May 2017

The original issue has been closed by #19594 . Will open a new issue for https://github.com/JuliaLang/julia/issues/19456#issuecomment-300331988

amitmurthy picture amitmurthy on 11 May 2017
👍1
Was this page helpful?
0 / 5 - 0 ratings

Related issues

omus picture omus  路  3Comments
helgee picture helgee  路  3Comments
felixrehren picture felixrehren  路  3Comments
arshpreetsingh picture arshpreetsingh  路  3Comments
iamed2 picture iamed2  路  3Comments