Agents.jl: Multi-threaded ensemble run is significantly slower than single thread.

[jacobusmmsmit]

Describe the bug
As per the title, running the same code with and without the use of Distributed's @everywhere macro gives very poor multi-threaded performance compared to without using @everywhere. Please let me know if I am using it incorrectly.

Working Example
Full runnable code is found in src/multiple_runs_single.jl and src/multiple_runs_distributed.jl of this repository. (Note that the number of agents and repetitions is reduced in the multithreaded version as my computer consistently ran out of memory when trying to run it)

Code used in multi-threaded run

using Distributed
using BenchmarkTools
addprocs(3)

@everywhere begin
    using DataFrames
    using Agents
    include("evolutionary_model.jl")
    include("helper_functions.jl")

    distributed_model(total_agents, i, j) = evolutionary_model(;
        num_sitters=i,
        num_identifiers=j,
        num_cheaters=total_agents - i - j,
        hatch_utility=4.0,
        egg_cost=0.9,
        eggs_laid=1,
        identify_cost=0.9,
        p_identify=1,
        p_mutation=0.05
    )

    total_agents = 10
    num_reps = 10
    models = [
        evo_model(total_agents, i, j)
            for i in 1:(total_agents - 1)
                for j in 1:total_agents - i - 1
                    for replicates in 1:num_reps
    ]
end


dist_bench = @benchmark ensemblerun!(models, agent_step!, model_step!, 1; parallel=true)

Code for the single threaded version is the same removing all references to Distributed, addprocs(), @everywhere and parallel.

BenchmarkTools output for multi-thread:

BenchmarkTools.Trial: 
  memory estimate:  381.22 MiB
  allocs estimate:  13206798
  --------------
  minimum time:     4.904 s (1.38% GC)
  median time:      4.944 s (1.67% GC)
  mean time:        4.944 s (1.67% GC)
  maximum time:     4.983 s (1.96% GC)
  --------------
  samples:          2
  evals/sample:     1

BenchmarkTools output for single thread:

BenchmarkTools.Trial: 
  memory estimate:  6.08 MiB
  allocs estimate:  96393
  --------------
  minimum time:     14.598 ms (0.00% GC)
  median time:      20.206 ms (0.00% GC)
  mean time:        22.609 ms (5.54% GC)
  maximum time:     64.478 ms (21.25% GC)
  --------------
  samples:          221
  evals/sample:     1

Julia version
1.6.0
Agents.jl version
Agents v4.2.4

[Datseris]

I think we have to setup some proper benchmarks using e.g. one of the predefined examples of the library. Probably the Daisyworld, it has this nice complexity of creating/deleting agents, but doesn't given any performance downsides of using multiple agent types.

[jacobusmmsmit]

I think one of Agents' strengths is how well it is documented. I'd love to see a full demonstration of how to parallelise one of the predefined examples make its way into the examples section, and it would serve well as a demonstrative benchmark. As of right now there's nothing you can just copy/paste (which lazy me would appreciate a lot).

[Libbum]

This may not be a bug per-se. It _could_ simply be that a 14ms trial just really doesn't warrant the overhead that threads cause. A better test case here would be one where each step was taking 30 seconds, a minute, or more to run; then comparing a threaded vs non-threaded cases.

Something to look into for sure, but as @Datseris suggests: we need to make sure we're looking at the right test case.

[Libbum]

I'd love to see a full demonstration of how to parallelise one of the predefined examples make its way into the examples section, and it would serve well as a demonstrative benchmark.

Let's make this one of the targets for any solution we find to this issue then!

[fbanning]

I've not yet encountered this type of slowdown in my distributed models (in fact quite the contrary) so I was a bit curious what happened here and decided to try to locally reproduce the results from your git repo. I attempted to do it interactively from a regular Julia REPL (no startup flags except activation of current working dir as environment) by including the two files (src/multiple_runs_single.jl and src/multiple_runs_distributed.jl). Here are a few comments from my side:

addprocs(3)

In case you didn't know, you can just do addprocs() to add as many workers as the number of cores of your CPU.

Project.toml was missing Distributions and StatsBase.
src/multiple_runs_distributed.jl doesn't activate the project environment for the workers (@everywhere begin using Pkg; Pkg.activate(".") end or something along those lines). This is needed for the workers to be able to find and use the Agents and DataFrames packages.
Both things are easy enough to find and change so that the MWE works as expected. A more general request from my side would be to make sure that the provided MWE works without (major) tweaks to your code.

(Note that the number of agents and repetitions is reduced in the multithreaded version as my computer consistently ran out of memory when trying to run it)

Both total_agents and num_reps are the same in both files. Am I missing something here?

Back to topic: I was able to see similar results from the benchmarks with the distributed version being about 240x as slow.

julia> include("src\\multiple_runs_single.jl")
BenchmarkTools.Trial:
  memory estimate:  6.63 MiB
  allocs estimate:  104001
  --------------
  minimum time:     12.494 ms (0.00% GC)
  median time:      12.875 ms (0.00% GC)
  mean time:        13.879 ms (5.98% GC)
  maximum time:     21.975 ms (30.05% GC)
  --------------
  samples:          360
  evals/sample:     1

julia> include("src\\multiple_runs_distributed.jl")
BenchmarkTools.Trial:
  memory estimate:  402.95 MiB
  allocs estimate:  13857293
  --------------
  minimum time:     3.226 s (1.67% GC)
  median time:      3.307 s (1.57% GC)
  mean time:        3.307 s (1.57% GC)
  maximum time:     3.387 s (1.48% GC)
  --------------
  samples:          2
  evals/sample:     1

Without further examination of the code, I would wager a guess that this is just because of the overhead introduced by Distributed. Increasing the number of steps from 1 to 100 leads to a significant reduction of the gap between the two versions:

julia> include("src\\multiple_runs_single.jl")
BenchmarkTools.Trial:
  memory estimate:  594.00 MiB
  allocs estimate:  9561183
  --------------
  minimum time:     1.185 s (6.22% GC)
  median time:      1.187 s (6.35% GC)
  mean time:        1.200 s (6.30% GC)
  maximum time:     1.233 s (6.77% GC)
  --------------
  samples:          5
  evals/sample:     1

julia> include("src\\multiple_runs_distributed.jl")
BenchmarkTools.Trial:
  memory estimate:  402.95 MiB
  allocs estimate:  13857313
  --------------
  minimum time:     4.067 s (2.18% GC)
  median time:      4.142 s (2.03% GC)
  mean time:        4.142 s (2.03% GC)
  maximum time:     4.218 s (1.90% GC)
  --------------
  samples:          2
  evals/sample:     1

Now the distributed version is only 3.45x as slow as the regular version, uses less memory and also generally less time for GC. The overall time of the two distributed versions (with 1 step and 100 steps) has only increased by ~25%.

I think this gives at least some support to the hypothesis voiced by @Libbum and me that the computations inside the model itself are just too fast to allow for any increases in speed by introducing multiple threads. Ergo I'd advice to only use parallel replicates if each model run on its own takes some significant amount of time (e.g. by having more agents or a more complex model logic or by simply letting the model run for a longer time). Hope this helps. 🙂

[fbanning]

I'd love to see a full demonstration of how to parallelise one of the predefined examples make its way into the examples section

Isn't this what you're asking for? Maybe I misunderstood your inquiry. I'm definitely open to add such a section to other already existing examples in our docs.

[Libbum]

Great work @fbanning. Not enough datapoints to predict when the distributed version would exceed the single threaded version. Out of interest could you give us 200, 500 and 1000 steps? (if you have the time)

[fbanning]

Out of interest could you give us 200, 500 and 1000 steps? (if you have the time)

Sure, I'm currently just working through my vacation email backlog anyways. Got some computational resources to spare:

@ 200 --> ~2.52x

julia> include("src\\multiple_runs_single.jl")
BenchmarkTools.Trial:
  memory estimate:  1.14 GiB
  allocs estimate:  18940083
  --------------
  minimum time:     2.213 s (6.18% GC)
  median time:      2.222 s (6.60% GC)
  mean time:        2.224 s (6.53% GC)
  maximum time:     2.236 s (6.82% GC)
  --------------
  samples:          3
  evals/sample:     1

julia> include("src\\multiple_runs_distributed.jl")
BenchmarkTools.Trial:
  memory estimate:  402.96 MiB
  allocs estimate:  13857684
  --------------
  minimum time:     5.613 s (2.61% GC)
  median time:      5.613 s (2.61% GC)
  mean time:        5.613 s (2.61% GC)
  maximum time:     5.613 s (2.61% GC)
  --------------
  samples:          1
  evals/sample:     1

@ 500 --> ~1.42x

julia> include("src\\multiple_runs_single.jl")
BenchmarkTools.Trial:
  memory estimate:  2.88 GiB
  allocs estimate:  48341783
  --------------
  minimum time:     6.043 s (5.03% GC)
  median time:      6.043 s (5.03% GC)
  mean time:        6.043 s (5.03% GC)
  maximum time:     6.043 s (5.03% GC)
  --------------
  samples:          1
  evals/sample:     1

julia> include("src\\multiple_runs_distributed.jl")
BenchmarkTools.Trial:
  memory estimate:  403.38 MiB
  allocs estimate:  13859259
  --------------
  minimum time:     8.598 s (0.68% GC)
  median time:      8.598 s (0.68% GC)
  mean time:        8.598 s (0.68% GC)
  maximum time:     8.598 s (0.68% GC)
  --------------
  samples:          1
  evals/sample:     1

@ 1000 --> ~1.33x

julia> include("src\\multiple_runs_single.jl")
BenchmarkTools.Trial:
  memory estimate:  5.79 GiB
  allocs estimate:  97057133
  --------------
  minimum time:     11.850 s (4.97% GC)
  median time:      11.850 s (4.97% GC)
  mean time:        11.850 s (4.97% GC)
  maximum time:     11.850 s (4.97% GC)
  --------------
  samples:          1
  evals/sample:     1

julia> include("src\\multiple_runs_distributed.jl")
BenchmarkTools.Trial:
  memory estimate:  402.98 MiB
  allocs estimate:  13858986
  --------------
  minimum time:     15.767 s (0.45% GC)
  median time:      15.767 s (0.45% GC)
  mean time:        15.767 s (0.45% GC)
  maximum time:     15.767 s (0.45% GC)
  --------------
  samples:          1
  evals/sample:     1

Still no "break even". Let's crank up those numbers then.

@ 2000 --> ~1.16x

julia> include("src\\multiple_runs_single.jl")
BenchmarkTools.Trial:
  memory estimate:  11.57 GiB
  allocs estimate:  194034133
  --------------
  minimum time:     24.904 s (4.93% GC)
  median time:      24.904 s (4.93% GC)
  mean time:        24.904 s (4.93% GC)
  maximum time:     24.904 s (4.93% GC)
  --------------
  samples:          1
  evals/sample:     1

julia> include("src\\multiple_runs_distributed.jl")
BenchmarkTools.Trial:
  memory estimate:  402.98 MiB
  allocs estimate:  13859050
  --------------
  minimum time:     29.059 s (0.24% GC)
  median time:      29.059 s (0.24% GC)
  mean time:        29.059 s (0.24% GC)
  maximum time:     29.059 s (0.24% GC)
  --------------
  samples:          1
  evals/sample:     1

@ 5000 --> 1.07x

julia> include("src\\multiple_runs_single.jl")
BenchmarkTools.Trial:
  memory estimate:  28.73 GiB
  allocs estimate:  483151133
  --------------
  minimum time:     67.180 s (4.86% GC)
  median time:      67.180 s (4.86% GC)
  mean time:        67.180 s (4.86% GC)
  maximum time:     67.180 s (4.86% GC)
  --------------
  samples:          1
  evals/sample:     1

julia> include("src\\multiple_runs_distributed.jl")
BenchmarkTools.Trial:
  memory estimate:  402.98 MiB
  allocs estimate:  13859075
  --------------
  minimum time:     73.067 s (0.09% GC)
  median time:      73.067 s (0.09% GC)
  mean time:        73.067 s (0.09% GC)
  maximum time:     73.067 s (0.09% GC)
  --------------
  samples:          1
  evals/sample:     1

While interesting to see this, conducting further tests doesn't make too much sense imho because altering just one specific parameter (steps) provides limited insight by itself. Feel free to do a regression with the given values if your curious. :D With this specific model at hand, it seems that a pretty high number of steps is required to offset the overhead costs of multihreading.

[jacobusmmsmit]

I'd love to see a full demonstration of how to parallelise one of the predefined examples make its way into the examples section

Isn't this what you're asking for? Maybe I misunderstood your inquiry. I'm definitely open to add such a section to other already existing examples in our docs.

I'm talking about _in-model_ parallelisation. I'm aware that the philosophy of Agents is that it does not concern itself with how the user wishes to define their model and doesn't provide any tools to help with this, but I think it would be nice to provide some tips (for example, what we are discussing now regarding how you perhaps shouldn't parallelise models with very simple computations per step) on how to do so (currently the docs link away to the Base Julia docs about generic parallelisation tips), and an example to go along with it. :)

[fbanning]

I'm talking about in-model parallelisation.

Ah ok, that wasn't clear to me. I think this is a very deep topic that would need to cover a whole lot of intricate aspects of parallelisation itself (first thing that comes to mind would be data race hell) and I am very convinced that our explanations on these topics are not going to be any better than those in the official Julia docs.

Fundamentally speaking, ABMs are just sophisticated loops (NB: this may be an unpopular opinion). So I would argue that there's not much that would hold back users from directly translating general knowledge on parallel computing to the ABM world.

[Libbum]

I agree with that last point @fbanning. There are only a subset of models that may do well with this and it can never be general.

@AayushSabharwal and I are actually working on a model that will need in-model parallelization at the moment, so we'll endeavor to add some decent documentation around it when it's ready.

[Datseris]

I think we can close this issue. In-model parallelization is a different topic.

[Datseris]

(currently the docs link away to the Base Julia docs about generic parallelisation tips)

In-model parallelization, when possible, will have nothing to do with Agents.jl and, at least personally, I can't really think of a way of giving more specific, or better tips, than what this page gives. But sure, having a new example model at the docs that does this could be nice.

[Libbum]

Agreed. Will keep this in mind when writing it up.