Agents.jl: Model (De)Serialization

[AayushSabharwal]

Points to consider:

• What should the target API look like? I think it should just be two functions save and load (names are of course up for debate)
• What format should it be saved in? JLD2, or using Serialization from stdlib (See the example at the bottom)? It might be worth testing if one is significantly faster than the other, or results in smaller files. I also saw BSON.jl, so maybe that's a possibility?
• This ties in to the previous point: Should the serialized file be able to be viewed without having to load it in as a model? Or is the sole purpose of this feature to load and save, and have the file be completely opaque? I'm in favour of the first option, although it would require some additional effort
• In some cases (GraphSpace and OSMSpace) multiple save files may be required. How should this be handled?
• Should the pathfinding extensions be implemented first, in case there's additional data to be saved or a change in the data layout?

I've gone through all the structs that would need to be saved, and mapped it out:

Properties to save:

• ABM.agents # Possibly as a dataframe-esque format, since the dictionary can be reconstructed from that and it may make viewing easier
• ABM.space
• ABM.properties # The structure and contents of this are unknown, so ideally it should load back to the exact same state it was
• ABM.rng # Each RNG struct would have it's own internal state, so it can resume from the same point in the sequence
• ABM.maxid
• GridSpace.metric
• GridSpace.hoods
• GridSpace.hoods_tuple
• GridSpace.pathfinder
• ContinuousSpace.grid
• ContinuousSpace.dims
• ContinuousSpace.spacing
• ContinuousSpace.extent
• GraphSpace.graph # JuliaGraphs/GraphIO.jl exists, but it saves to a separate file. I don't think there's a point in reinventing the wheel, so there's the possibility of having multiple save files to consider
• OSMSpace.m # A quick glance at pszufe/OpenStreetMapX.jl didn't show any functions to save to a .osm file
• AStar.agent_paths # This is a Dict mapping to Linked List. Not difficult to implement, but worth exploring if the linked list state can be preserved or if it will have to be regenerated from an array
• AStar.grid_dims
• AStar.neighborhood # just save Pathfinder.diagonal_movement
• AStar.admissibility
• AStar.walkable
• AStar.cost_metric # HeightMap is recursive

Properties taken from user at deserialization:

• ContinuousSpace.update_vel!
• ABM.scheduler
  What if we save a symbol, and assume the function exists in the environment when it is loaded?

Properties reconstructed from saved data:

• GridSpace.s
• GraphSpace.s
• OSMSpace.s
  Agents store their positions, so these structures can be regenerated at deserialization

A serialization method that @Libbum found. It uses Serialization from the standard library, so there isn't any additional dependency. However, it's completely opaque. At a glance it seems to work. I'm currently testing if it saves the more intricate data listed above

using Agents
using Serialization # stdlib

(model, agent_step!, model_step!) = Models.predator_prey(   
               n_wolves = 40,   
               n_sheep = 60,   
               dims = (25, 25),   
               Δenergy_sheep = 5,   
               Δenergy_wolf = 13,   
               sheep_reproduce = 0.2,   
               wolf_reproduce = 0.1,   
               regrowth_time = 20,   
           )

serialize("model.bin", model)
# Exit julia, restart
using Agents
model = deserialize("model.bin")

[Libbum]

I'm pretty confident both @Datseris and myself would vote against BSON.jl. We've both had trouble in the past with it and development is sporadic. JLD2 or the stdlib option are the only two options that need to be explored I think.

[Libbum]

Should the serialized file be able to be viewed without having to load it in as a model?

This is the text vs binary question I guess. If we can get serialisation to work independent of the output that would be awesome. What I mean by that is similar to what rust does with serde: an interface to serialise your objects, then the output is format agnostic (list of data formats).

If we can't to that, then I worry that an explicit solution will continue to bring up questions like:

Should the pathfinding extensions be implemented first, in case there's additional data to be saved or a change in the data layout?

I don't want us to have to make sure that multiple portions of the code base need to be updated for any new feature we add in the future.

Food for thought, not a hard rule. Let's see where these ideas take us first.

[AayushSabharwal]

This is the text vs binary question I guess.

Not necessarily. JLD2 can be viewed without loading in as a model

I don't want us to have to make sure that multiple portions of the code base need to be updated for any new feature we add in the future.

I agree. This might not be completely unavoidable though, since new features would probably store new data. However, most of the data would be in one of very few formats, such as Dict or Array if not a primitive type/Tuple. If the more complex types like linked lists and the like can be converted to the above, then it would be largely agnostic of features or the internal data layout. If it sees a type, it either saves it or converts it into a format that can be saved. This would only require implementing that conversion, which is possible through custom serialization in JLD2 (worth noting that this is marked as experimental)

[Datseris]

One important think to check is whether Base.Serialization is conserved across versions. If you save something with it at 1.5, can you load it at 1.6? or 1.7?

[Datseris]

From my pesrpective: The more amount of things that can be re-computed instead of stored the better. I saw in the OP a _lot_ of things related to pathfinding, in fact much more than the "vanilla" model. Can we shave off some?

[AayushSabharwal]

neighbourhood is a Vector, but it can be stored as a boolean value instead. This assumes that the user hasn't replaced it with their own custom neighbourhood. If the option for custom neighbourhoods is provided in the pathfinding extensions, then it might create an issue here

agent_paths could be recomputed by instead storing the initial and final positions in the linked list. This, however, assumes a sort of transitivity in the shortest-path relation (determined by the cost metric). The path in the list more likely than not wouldn't include the starting position from where the path was computed. Basically, if the path computed from A to B goes through C, then ideally calculating the path from C to B would result in the same sequence of positions as in the latter part of A to B. Currently I think our cost metrics are transitive, but this may not be the case if user-defined metrics are used.

It might also be worth noting that if there are a lot of agents on a relatively large/complex space, recomputing all paths would lead to significantly longer loading times

grid_dims for GridSpace could just be taken from the space, but in ContinuousSpace it's more than likely the quantisation of the space would be much higher resolution than the underlying GridSpace. As such, I kept it in the list.

[AayushSabharwal]

I've tested Serialization extensively. Most of the parameters are serialized and deserialized properly. The problems I found were:

• I don't think model.rng saves correctly. I tried saving a model, loading it back into another variable, running both for the same number of steps, and comparing agent parameters. They don't match. For models where agents die/are created at runtime, the number of agents varies
• I haven't yet been able to verify if OSMSpace.m is saving and loading correctly, since a simple == test returns false and the underlying structure is too complex to test every field individually

[Datseris]

Perhaps OSMSpace should not be serialized this way but re-recreated given a path to an OSM map that is loaded.

[AayushSabharwal]

I should add, interestingly enough comparing model.rng from both the model instances with == returns true. I don't know why they diverge when the model is run. I'll try again later

[Libbum]

Trying to serialise OSM is agreed a bad idea. The internals there are non-deterministic, so it'd be better to save the map file, agents start and end lat/lon positions (not indexes) and then recalculate on load.

As for model.rng: It could be either an internal we missed, or something in your test case is not passing model.rng to a randomisation function somewhere.

[AayushSabharwal]

rng is indeed saved across sessions, I must have been missing something in my earlier test using sir.jl

One important think to check is whether Base.Serialization is conserved across versions. If you save something with it at 1.5, can you load it at 1.6? or 1.7?

According to the docs, this isn't the case. It's also not guaranteed to work if the system image loaded with Julia is different, so it appears as if this is not an option. I'll now try creating a serialization framework for JLD2 and see the results there.

[Datseris]

I've been thinking that in this GSOC we should incorporate resolving https://github.com/JuliaDynamics/Agents.jl/issues/403 . This ties in with your initial proposal of a text-based format. I think #403 is rather solved with CSV.jl instead of any of the binary savers

[AayushSabharwal]

Sure. I don't think that should be too difficult. I think it would be a straightforward read_from_csv(filename, model, AgentType) function. The AgentType would only be required if the model has multiple agent types. Both these issues could go in their own submodule, since it would make the relevant dependencies optional

Should I work on resolving that issue before this one?

[Datseris]

Yeah let's start an "IO" with perhaps a better name. We can build it with the easy stuff initially. I guess the most popular target scenario for this module would be open street map applications. In this case, how is the agent location saved @Libbum ?

[AayushSabharwal]

Using JLD2 will need an explicit framework built around it, even for testing. I tried save_object and load_object, and from what I understand it has issues with schedulers such as by_property.
The serialization also takes a little longer than Serialization, and produces larger files. Both of these could be dealt with by either using the experimental custom serialization feature of JLD2, or implementing a custom data layout. Of the two, the former is most likely easier to maintain in the long term. It's probably harder to write and test though, especially considering the nested and type-parameterised nature of our data.

[Datseris]

Yeah JLD2 does not do well with closures, however I am not sure whether such function-based objects should be saved. If we don't save agent stepping and model stepping functions, we shouldn't save the scheduler either I feel.

[AayushSabharwal]

I agree, there's no reason to save the scheduler. I didn't really have a choice with save_object so I couldn't avoid it. There's also the case of deciding what to do for any non-serializable type in model.properties

[Libbum]

open street map applications. In this case, how is the agent location saved

Based on issues with OpenStreetMapX, The best way to save an agent location on disk would be to give an explicit latitude and longitude, then convert that to the appropriate value on-load.

[AayushSabharwal]

I managed to get a partial serialization framework up and running with JLD2, using the custom serialization functionality (https://juliaio.github.io/JLD2.jl/dev/customserialization/). It's an idea for how this can be structured, and it works on the couple of tests I've thrown at it.

After includeing the following file in the REPL:
https://gist.github.com/AayushSabharwal/1f86ae456f8b41c56a82d5f32a02abbd

REPL (Click me)

julia> using Random

julia> @agent Ag GridAgent{2} begin
           dat::Int
       end

julia> function initm(n)
           sp = GridSpace((10,10))
           properties = (a=3,b=4.5,c="foo")
           rng = MersenneTwister(42)
           abm = ABM(Ag, sp; properties, rng)
           for i in 1:n
               add_agent!(abm, floor(Int, rand(rng) * 100))
           end
           return abm
       end

julia> m = initm(5)

julia> m.agents
Dict{Int64, Ag} with 5 entries:
  5 => Ag(5, (9, 4), 95)
  4 => Ag(4, (10, 4), 17)
  2 => Ag(2, (8, 10), 45)
  3 => Ag(3, (1, 3), 1)
  1 => Ag(1, (4, 1), 53)

julia> m.space.s
10×10 Matrix{Vector{Int64}}:
 []   []  [3]  []   []  []  []  []  []  []
 []   []  []   []   []  []  []  []  []  []
 []   []  []   []   []  []  []  []  []  []
 [1]  []  []   []   []  []  []  []  []  []
 []   []  []   []   []  []  []  []  []  []
 []   []  []   []   []  []  []  []  []  []
 []   []  []   []   []  []  []  []  []  []
 []   []  []   []   []  []  []  []  []  [2]
 []   []  []   [5]  []  []  []  []  []  []
 []   []  []   [4]  []  []  []  []  []  []

julia> m.rng
MersenneTwister(42, (0, 2256, 0, 5, 1002, 5))

julia> @save "test.jld2" m

julia> m = nothing

julia> @load "test.jld2" m

julia> m.agents
Dict{Int64, Ag} with 5 entries:
  5 => Ag(5, (9, 4), 95)
  4 => Ag(4, (10, 4), 17)
  2 => Ag(2, (8, 10), 45)
  3 => Ag(3, (1, 3), 1)
  1 => Ag(1, (4, 1), 53)

julia> m.space.s
10×10 Matrix{Vector{Int64}}:
 []   []  [3]  []   []  []  []  []  []  []
 []   []  []   []   []  []  []  []  []  []
 []   []  []   []   []  []  []  []  []  []
 [1]  []  []   []   []  []  []  []  []  []
 []   []  []   []   []  []  []  []  []  []
 []   []  []   []   []  []  []  []  []  []
 []   []  []   []   []  []  []  []  []  []
 []   []  []   []   []  []  []  []  []  [2]
 []   []  []   [5]  []  []  []  []  []  []
 []   []  []   [4]  []  []  []  []  []  []

julia> m.rng
MersenneTwister(42, (0, 2256, 0, 5, 1002, 5))

It would require a little maintenance as features are added, but only whenever some data has to be selectively omitted (m.space.s), or put into a serialization friendly format (m.agents). For other structs, JLD2 does it automatically (I didn't have to write anything to serialize Ag).