Agents.jl: Same seed, different result (with different way to collect data)

Created on 28 May 2021 · 8Comments · Source: JuliaDynamics/Agents.jl

Describe the bug
The model result is deterministic, but in two different way. Different types of adata causes the results to be different.

Minimal Working Example
(This is a pluto notebook file, sorry for the extension, Github doesn't allow drag dropping .jl file). This is the example predator-prey model, I modified it so it uses seed and scheduler.

agentsjl-collect-bug.txt

Agents.jl version

pkg> st Agents
     Project .........
      Status ............
  [46ada45e] Agents v4.2.4

Additional infomation

I think this happened because init_agent_dataframe uses two other functions, one of which uses random_agent, which uses shuffle!(model.rng) while the other doesn't.

https://github.com/JuliaDynamics/Agents.jl/blob/c377d812146de06e30a13471b4b04f9076b8988e/src/simulations/collect.jl#L201-L206

https://github.com/JuliaDynamics/Agents.jl/blob/c377d812146de06e30a13471b4b04f9076b8988e/src/simulations/collect.jl#L211-L213

https://github.com/JuliaDynamics/Agents.jl/blob/c377d812146de06e30a13471b4b04f9076b8988e/src/core/model.jl#L190-L212

bug rng

Source

ndgnuh

All 8 comments

The model result is deterministic, but in two different way. Different types of adata causes the results to be different.

I don't understand the problem, can you please elaborate in more detail? Linking a 200-loc text file is not as transparent as you'd might think :D

EDIT: Okay, yes you are right, you have pinpointed the problem.

Datseris on 28 May 2021

😄1

Sorry for not being a mess :P Basically, I have two function which runs the exact same model, with the exact same seed, with different adata

function collect_agg(sched)
    wolf(a) = a.type == :wolf
    sheep(a) = a.type == :sheep
    df = runsim([(wolf, count), (sheep, count)]; scheduler=sched)
    rename(df, [:step, :nwolf, :nsheep])
end

function collect_normal(sched)
    df = runsim([:type]; scheduler=sched)
    df = combine(
        groupby(df, :step),
        :type => (x -> count(x .== :wolf)) => :nwolf,
        :type => (x -> count(x .== :sheep)) => :nsheep,
    )
end

Here's a quick comparison to see what I mean

let f = Figure()
    adf1 = collect_agg(Schedulers.by_id)
    adf2 = collect_normal(Schedulers.by_id)
    a = Axis(f[1,1])
    b = Axis(f[1,2])
    lines!(a, adf1.step, adf1.nwolf)
    lines!(a, adf2.step, adf2.nwolf)
    lines!(b, adf1.step, adf1.nsheep)
    lines!(b, adf2.step, adf2.nsheep)
    f
end

That's another thing I don't understand... random_agent does not do shuffle!(model.rng)...?

This I might be wrong, but according to the code it does trigger the rng some where.

ndgnuh on 28 May 2021

I edited my comments, you got the problem nailed down.

Datseris on 28 May 2021

So, how do you think we should fix this?

We can either put another random_agent(model) before the multi_agent_types!

    if std_headers == 3
        headers[3] = "agent_type"
        random_agent(model)
        multi_agent_types!(types, utypes, model, properties)
    else
        single_agent_types!(types, model, properties)
    end

or replace the random_agent at single_agent_types! with something else

function single_agent_types!(types::Vector{Vector{T} where T}, model::ABM, properties::AbstractArray)
    a = .. # maybe pick the first agent?
    for (i, k) in enumerate(properties)
        current_type = typeof(get_data(a, k, identity))

I don't know much about the package internal, so...

ndgnuh on 28 May 2021

I'll let Tim handle this one, because he was much more involved in the RNG enhancements, he should have the best course of action.

Datseris on 28 May 2021

👍1

Uh oh. It seems that the problem it not just there, I did a dump patch on my fork and went ahead with my job. The data collected by run! seems ok,

but theo video collected abm_video was not the same, all the agents died around step 1800, I don't know if data collecting with adata triggered the model's rng, but I can't not pass adata or mdata to abm_video.

https://user-images.githubusercontent.com/33273948/119984284-6bb25080-bfeb-11eb-8eaf-fa6995ad9e4f.mp4

ndgnuh on 28 May 2021

Interesting bug! I'm pretty sure you're right about your suggested fix: a = .. # maybe pick the first agent?.

Using a random_agent here is definitely not a requirement and the first solution (adding a random_agent to multi_agent_types!) may get unstuck in the future if we need to extend the data collection system.

I'll play around a bit to make sure that's the best way to go to be certain.

Libbum on 28 May 2021

👍1

## More on the video situation

I did something way dumber in my fork, which is copy the rng value in the random_agent. I'll try again after changing to a = .. # maybe pick the first agent?...

I have only reproduced the mismatch on randomly scheduler (I re-run the code and re-rendered the video twice...). I have not yet figured out what's the problem here though.

These is video and the plot from by_id scheduler. (notice the population matches with the imagery from step ~1800 to end in the videos.