Ncnn: custom memory allocator and pool allocator

[nihui]

• [x] new abstract Allocator class
• [x] new Mat member allocator
• [x] Mat constructor and Mat::create() / Mat::release() allocator-aware
• [x] Mat::clone() and Mat::reshape() allocator-aware
• [x] copy_make_border / copy_cut_border / bilinear_resize allocator-aware
• [x] Mat::from_pixels() family allocator-aware
• [x] new Extractor::set_allocator() api for setting one allocator per thread
• [x] add Allocator parameter to Layer::forward() family
• [x] try best to make all layer implementation allocator-aware
• [x] simple lock-free pool allocator
• [ ] defrag pool allocator memory(POC)

ncnn::Allocator* pa = new ncnn::PoolAllocator;

for (int i=0; i<imagecount; i++)
{
    ncnn::Extractor ex = net.create_extractor();
    ex.set_allocator(pa);
}

// clone all non-free Mat, expensive
delete pa;

[nihui]

separated api for blob allocation and workspace allocation

ncnn::Allocator* upa = new ncnn::UnlockedPoolAllocator;
ncnn::Allocator* pa = new ncnn::PoolAllocator;

for (int i=0; i<imagecount; i++)
{
    ncnn::Extractor ex = net.create_extractor();
    ex.set_blob_allocator(upa);
    ex.set_workspace_allocator(pa);
}

delete upa;
delete pa;

[nihui]

Mat structure is now allocator-aware via an extra allocator parameter with default zero value.
The good-old ncnn::fastMalloc()/ncnn::fastFree() will be used for a null allocator. You could pass a custom allocator to delegate all memory allocation and deallocation.

class Allocator
{
public:
    virtual void* fastMalloc(size_t size) = 0;
    virtual void fastFree(void* ptr) = 0;
};

ncnn has already implemented two simple pooled Allocator class, with mutex lock or without it.

ncnn::PoolAllocator locked_mempool;
ncnn::UnlockedPoolAllocator unlocked_mempool;

the two allocator types in ncnn

1. blob allocator
   used to allocate memory for all named blobs, which you could retrieve by Extractor::extract()
2. workspace allocator
   used to allocate memory for internal temporary use in layer implementation, such as the temp blob after padding in convolution

by default, all Extractor instance use the two allocator in the default option
You can alter them by ncnn::set_default_option()
or you can set them per Extractor by Extractor::set_blob_allocator()/Extractor::set_workspace_allocator()

blob allocator is guaranteed to be called in-order in layer implementation during each Extractor lifecycle
while workspace allocator may be called synchronously

the practical usage

1. one network, one-by-one inference
   shared unlocked blob allocator for all Extractor
   shared locked workspace allocator for all Extractor
2. one network, concurrent inference
   shared unlocked blob allocator for all Extractor in each thread
   shared locked workspace allocator for all Extractor among all threads
3. concurrent multiple networks, one-by-one inference for each network
   shared unlocked blob allocator for all Extractor of each network
   shared locked workspace allocator for all Extractor among all networks (for saving memory)
4. concurrent multiple networks, concurrent inference for each network
   shared unlocked blob allocator for all Extractor of each network in each thread
   shared locked workspace allocator for all Extractor among all networks (for saving memory)