Osrm-backend: Which files get loaded into ram?

[marcushat]

Hey guys, have a few questions regarding ram usage of osrm-routed that I don't think have been fully addressed yet. I just ran extract and prepare for the north america dataset(currently ~6.6GB) I got files totalling 25GB!.
Here's the file sizes:

When I ran ran osrm-routed though it showed that the system was only using 8.8GB of memory, so obviously they're not all loaded. So that leaves me with a few questions.

1. Is that the expected ram usage for the dataset? (I got no errors so I guess so)
   If so:
2. Will the ram usage increase as the server is used?
3. Which files are loaded into RAM? In another issue someone said that RAM usage seems to be the combined size of the .hsgr and .edges files. Is this true, can someone confirm this?
4. What happens with the rest of the files that are not loaded into ram(.osrm, .fileIndex, .geometry)?
5. Are these other files needed for osrm-routed? If so, why are they not in ram?
6. Is there a correlation between the size of .osm.pbf and the any of the other files?

It seems like most people run extract and prepare on ec2 then run routed on a server with less ram as this seems to be the cheapest way, so I think it'd be helpful if it was explicitly stated in the wiki which files need to be transferred over the wire to the osrm-routed server(25GB vs 8 is a pretty big difference). Ran extract and prepare on r3.4xlarge instance and trying to decide between a 16GB and 32GB box for routed.

Also, I'm interested in the ram and .stxxl usage and timing for other datasets(north america, europe) besides planet. I'd love to see everyone contribute their usage stats. I'm willing to do some tests with different datasets on ec2 later if needed.

And thanks a million to the the osrm team!

[marcushat]

I accidentally deleted my .osrm file while trying to figure this out. I went ahead and replaced it with a empty file and it seems osrm is working as expected. Is .osrm needed for routed or is just to find the other files?

[TheMarex]

You can have a look at the file server/data_structures/internal_datafacade.hpp which loads the necessary files into memory.
The server needs the following files:

.nodes : Original graph nodes (compressed geometry)
.edges : Original graph edges (compressed geometry)
.geometry : Geometry that was removed from the original graph
.hsgr : Contraced edge-expanded graph (nodes and edges)
.ramIndex : Index of the R-Tree to to segment lookups
.fileIndex: Leaves of the R-Tree that are loaded on-demand to memory
.names : Street names and index

All files but .fileIndex are loaded into memory.

Is .osrm needed for routed or is just to find the other files?

Yes, it is only used by osrm-prepare.

[marcushat]

Okay, thanks for the reply. That would explain the memory usage. If .osrm is no longer needed can we please remove it as a parameter to starting osrm-routed, maybe specify the directory where the needed files are located instead? It seems entirely unintuitive to have an intermediate just to find location the others as it implys its being used.
Also can we add to the wiki that ram usage will be sum of those 6 files and that .osrm can really be discarded after prepare? I'm thinking most people use aws so transfering only the nessesary files could save time n money and they can know exactly how much ram is needed for routed.

[TheMarex]

If .osrm is no longer needed can we please remove it as a parameter to starting osrm-routed, maybe specify the directory where the needed files are located instead?

I don't think changing the backward compatibility of the tool chain is a good idea.

Also can we add to the wiki that ram usage will be sum of those 6 files and that .osrm can really be discarded after prepare

Good idea. :+1: Added it here

[marcushat]

I think it's a great idea, especially if it can help avoid ambiguity and confusion. It can most certainly be implemented in a backwards compatable way. Have it accept either .osrm or the directory as a param and have the wiki promote using the directory way.

On May 17, 2015, at 7:16 AM, Patrick Niklaus [email protected] wrote:

If .osrm is no longer needed can we please remove it as a parameter to starting osrm-routed, maybe specify the directory where the needed files are located instead?

I don't think changing the backward compatibility of the tool chain is a good idea.

Also can we add to the wiki that ram usage will be sum of those 6 files and that .osrm can really be discarded after prepare

Good idea. Added it here

—
Reply to this email directly or view it on GitHub.

[marcushat]

...because now theres an inconsistency, wiki says .osrm doesn't need to be conserved but it's still required to start routed.

[daniel-j-h]

We updated the Wiki with the latest files, take a look here:

https://github.com/Project-OSRM/osrm-backend/wiki/Running-OSRM#file-overview

[marcushat]

Yeah, saw this the other day. Thanks.

On Jan 12, 2016, at 8:43 AM, Daniel J. H. [email protected] wrote:

We updated the Wiki with the latest files, take a look here:

https://github.com/Project-OSRM/osrm-backend/wiki/Running-OSRM#file-overview

—
Reply to this email directly or view it on GitHub.

[RikdeBoer]

I was also struggling with disk space and wondering which files to keep before starting osrm-routed.

Are the points made in the above conversation still valid?

Unlike what is stated above, I don't see these points (about which files to keep) reflected anywhere in this Wiki page: https://github.com/Project-OSRM/osrm-backend/wiki/Running-OSRM#file-overview

[RikdeBoer]

When I removed .fileIndex I got in an error when starting osrm-routed, saying it wants that file.

What I did find is correct is the suggestion above that you do not need the .osrm file _data_ for the osrm-routed command, but you do need a _file_ by that name, even when empty.
This worked for me (saving disk space):

rm all.osrm
touch all.osrm
nohup osrm-routed all.osrm &

[TehNrd]

So glad I found this Issue as the wiki links above make no reference to the required files or mention that the .osrm file can be emptied.

After performing the osrm-contract I added the following code to free up some space:

await fsPromises.writeFile('osm.osrm', 'Empty placeholder file, https://github.com/Project-OSRM/osrm-backend/issues/1480');

I'd be up for updating the wiki with this bit of info if there is a process for that.

Also, is there an updated and definitive list of the files required to start OSRM?

[boogerlad]

Through trial and error using the latest osrm-routed on master, the required files for CH are

.osrm (can be empty file)
.osrm.datasource_names
.osrm.ebg_nodes
.osrm.edges
.osrm.fileIndex
.osrm.geometry
.osrm.hsgr
.osrm.icd
.osrm.maneuver_overrides
.osrm.names
.osrm.nbg_nodes
.osrm.properties
.osrm.ramIndex
.osrm.timestamp
.osrm.tld
.osrm.tls
.osrm.turn_duration_penalties
.osrm.turn_weight_penalties

[danpat]

Small note: the .fileIndex file isn't loaded into RAM - it's very large, so it's mmap()-ed.