Julia: Serialization of large graphs for pmap fails

[jnkh]

Description of bug
When large graphs are sent to worker processes, an error occurs. It is more likely with larger graphs and more workers. Posted this issue at Lightgraphs.jl, and they recommended I bring it up here.

ERROR: LoadError: IOError: write: bad address in system call argument (EFAULT)
Stacktrace:
 [1] (::getfield(Base, Symbol("##696#698")))(::Task) at ./asyncmap.jl:178
 [2] foreach(::getfield(Base, Symbol("##696#698")), ::Array{Any,1}) at ./abstractarray.jl:1866
 [3] maptwice(::Function, ::Channel{Any}, ::Array{Any,1}, ::UnitRange{Int64}) at ./asyncmap.jl:178
 [4] #async_usemap#681 at ./asyncmap.jl:154 [inlined]
 [5] #async_usemap at ./none:0 [inlined]
 [6] #asyncmap#680 at ./asyncmap.jl:81 [inlined]
 [7] #asyncmap at ./none:0 [inlined]
 [8] #pmap#215(::Bool, ::Int64, ::Nothing, ::Array{Any,1}, ::Nothing, ::Function, ::Function, ::WorkerPool, ::UnitRange{Int64}) at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.1/Distributed/src/pmap.jl:126
 [9] pmap(::Function, ::WorkerPool, ::UnitRange{Int64}) at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.1/Distributed/src/pmap.jl:101
 [10] #pmap#225(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::Function, ::UnitRange{Int64}) at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.1/Distributed/src/pmap.jl:156
 [11] pmap(::Function, ::UnitRange{Int64}) at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.1/Distributed/src/pmap.jl:156
 [12] top-level scope at none:0
 [13] include at ./boot.jl:326 [inlined]
 [14] include_relative(::Module, ::String) at ./loading.jl:1038
 [15] include(::Module, ::String) at ./sysimg.jl:29
 [16] exec_options(::Base.JLOptions) at ./client.jl:267
 [17] _start() at ./client.jl:436
in expression starting at /Users/julian/Dropbox/ubuntu_home/Harvard/research/desai/population/src/figure_generation/fig3/min_example.jl:13
┌ Warning: Forcibly interrupting busy workers
│   exception = rmprocs: pids [2, 3, 4, 6, 7, 8, 9] not terminated after 5.0 seconds.
└ @ Distributed /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.1/Distributed/src/cluster.jl:1139
┌ Warning: rmprocs: process 1 not removed
└ @ Distributed /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.1/Distributed/src/cluster.jl:928

Code demonstrating bug

using Distributed
addprocs(8) #error is more likely with more processors
@everywhere using LightGraphs
@everywhere foo(g) = 0

#needs to be a sufficiently large graph for the error to occur
N = 10000
k = 1000
g = Graph(N,Int(N*k/2))

#g = adjacency_matrix(g) #error goes away if this line is uncommented, this returns a sparse matrix representation of the graph.

pmap(_ -> foo(g),1:nprocs()) #this is where the error happens

Version information

Julia Version 1.1.0
Commit 80516ca202 (2019-01-21 21:24 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin14.5.0)
  CPU: Intel(R) Core(TM) i7-6920HQ CPU @ 2.90GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.1 (ORCJIT, skylake)`

Status `~/.julia/environments/v1.1/Project.toml`
  [093fc24a] LightGraphs v1.2.0

[sbromberger]

FYI: I can reproduce this bug but I'm not sure that it's something that LightGraphs can fix.

[dkarrasch]

I had a very similar error message using pmap in an otherwise different context (for instance, I wasn't loading LightGraphs), but with involved code, so I wouldn't know how to communicate the error.

[Msturroc]

I am also having a similar issue (though also in a different context). As jnkh seems to believe this is related to the number of processors I will try to repeat with fewer processors to see if I still have the same issue.

edit: I find it is unrelated to number of processors but instead the scale of the problem

[miguelraz]

FWIW I couldn't reproduce the issue one 1.3.0_rc3.

On Thu, Oct 17, 2019, 06:07 Marc Sturrock [e-mail] wrote:

I am also having a similar issue (though also in a different context). As
jnkh seems to believe this is related to the number of processors I will
try to repeat with fewer processors to see if I still have the same issue.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/JuliaLang/julia/issues/32459?email_source=notifications&email_token=ADDTRNNWWF2AHZIJYAN2A7LQPBBNZA5CNFSM4H4OGMLKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEBPWRSQ#issuecomment-543123658,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADDTRNJZOYHLOIAEFZP5CQLQPBBNZANCNFSM4H4OGMLA
.

[jamblejoe]

I have a similar issue. Same error, data sent to worker processes consists of sparse matrices. Reducing the size of the matrices got rid of the error. Additionally, I encountered that reducing the number of worker processes helps as well. I am using remote machines, so communication definitely goes over TCP/IP.

Retrying the calculations with the ´retry_delays´ parameter did not solve the issue.

Further, sometimes the error was propagated to the toplevel, so the calculation got aborted, sometimes, the error did not get propagated and ´pmap´ did never finish.

Julia Version 1.2.0 Commit c6da87ff4b (2019-08-20 00:03 UTC) Platform Info: OS: Linux (x86_64-pc-linux-gnu) CPU: Intel(R) Xeon(R) CPU E5335 @ 2.00GHz WORD_SIZE: 64 LIBM: libopenlibm LLVM: libLLVM-6.0.1 (ORCJIT, core2)