One: [onert] Use MMapedData or CachedData?

[periannath]

On master, model is loaded as MMapedData. However, model is loaded as CachedData on release 1.8.0 since MMapedData produces performance degradation on some models. #3817

Should we use CachedData on master branch the same as release 1.8.0?

Here is the comparison of MMapedData and CachedData

MMapedData

• Using mmaped model file
• Pros
  
  
  
  • Fast model load time
  
  
  • Low memory usage at model load
  
  
• Cons
  
  
  
  • Overhead in inference
  
  

CachedData

• Allocated memory using malloc
• Pros
  
  
  
  • Fast inference
  
  
• Cons
  
  
  
  • Slow model load time
  
  
  • High memory usage at model load
  
  

/cc @YongseopKim @chunseoklee

[chunseoklee]

IMHO, we should go for cached data since Overhead in inference is too risky.

[periannath]

I've uploaed #4007 to use CachedData instead of MMapedData.

[hyunsik-yoon]

Can I ask a question? I wonder why mmaped data has overhead in inference. Could you explain a bit? Thanks!

[YongseopKim]

IMHO, we should go for cached data since Overhead in inference is too risky.

If so, what about introducing a build option which could choose one btw CachedData and MMapedData? TFLite also supports like that - BUILD_WITH_MMAP.

[periannath]

@hyunsik-yoon During the inference, runtime should read weight values from model file. When using CachedData, all the weight values are always resident on RAM.
However, using MMapedData makes page fault overhead since it should read weight values from disk if it does not exist in page cache. I'm not sure about the number of increment of page fault, but I suspect it cause an increment in inference time.

[periannath]

@YongseopKim Thanks. It looks good to me. I'll upload the following PR after #4007 is merged.

[s-barannikov]

If so, what about introducing a build option which could choose one btw CachedData and MMapedData?

I think it would be better to choose the behavior dynamically (environment variables or such), so that the user can choose what is best for them -- lower memory footprint or faster inference.

[periannath]

Closing this issue since USE_MMAPED_DATA option is added to used MMapedData.