Rawtherapee: Fast bayer demosaicer for flat regions

[heckflosse]

When using dual-demosaic, currently the very slow VNG4 demosaic is used for flat regions of the image.

I just coded a very fast (vectorized and multithreaded by compiler), bilinear bayer demosaicer, which can be used instead of VNG4 for flat regions.
Here's the code, PR will follow later:

////////////////////////////////////////////////////////////////
//
//  Bilinear bayer demosaic, optimized for speed, intended use is for flat regions of dual-demosaic
//
//  copyright (c) 2020  Ingo Weyrich <[email protected]>
//
//
//  code dated: May 09, 2020
//
//  bayer_bilinear_demosaic.cc is free software: you can redistribute it and/or modify
//  it under the terms of the GNU General Public License as published by
//  the Free Software Foundation, either version 3 of the License, or
//  (at your option) any later version.
//
//  This program is distributed in the hope that it will be useful,
//  but WITHOUT ANY WARRANTY; without even the implied warranty of
//  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
//  GNU General Public License for more details.
//
//  You should have received a copy of the GNU General Public License
//  along with this program.  If not, see <https://www.gnu.org/licenses/>.
//
////////////////////////////////////////////////////////////////

#include "rawimagesource.h"
#define BENCHMARK
#include "StopWatch.h"

using namespace rtengine;

void RawImageSource::bayer_bilinear_demosaic()
{
BENCHFUN
#ifdef _OPENMP
    #pragma omp parallel for
#endif
    for (int i = 1; i < H - 1; ++i) {
        float **nonGreen1 = red;
        float **nonGreen2 = blue;
        if (FC(i, 0) == 2 || FC(i, 1) == 2) { // blue row => swap pointers
            std::swap(nonGreen1, nonGreen2);
        }
#if defined(__clang__)
        #pragma clang loop vectorize(assume_safety)
#elif defined(__GNUC__)
        #pragma GCC ivdep
#endif
        for (int j = 2 - (FC(i, 1) & 1); j < W - 2; j += 2) { // always begin with a green pixel
            green[i][j] = rawData[i][j];
            nonGreen1[i][j] = (rawData[i][j - 1] + rawData[i][j + 1]) * 0.5f;
            nonGreen2[i][j] = (rawData[i - 1][j] + rawData[i + 1][j]) * 0.5f;
            green[i][j + 1] = ((rawData[i - 1][j + 1] + rawData[i][j]) + (rawData[i][j + 2] + rawData[i + 1][j + 1])) * 0.25f;
            nonGreen1[i][j + 1] = rawData[i][j + 1];
            nonGreen2[i][j + 1] = ((rawData[i - 1][j] + rawData[i - 1][j + 2]) + (rawData[i + 1][j] + rawData[i + 1][j + 2])) * 0.25f;
        }
    }
    border_interpolate(W, H, 2, rawData, red, green, blue);
}

On a single core of my 8-core machine, this is ~ 20 times faster than VNG4. When running it on all cores of my 8-core machine it's still ~ 5.7 times faster than VNG4

[Thanatomanic]

Hmm, I like the idea, if the differences are hardly noticeable.
However, wouldn't this further expand the already _long_ list of demosaicers to choose from? Or are you going all the way and making a 'pick whichever two demosaicers you want' option 😉

[heckflosse]

@Thanatomanic In first step I will add RCD+bilinear to allow tests. Then we can decide how to proceed.

[heckflosse]

bayer_bilinear branch ready to test

[Thanatomanic]

I gave it a quick spin and I am surprised by how little difference I see between VNG4 and Bilinear. If there is some noise, and you enable capture sharpening, some differences are visible (VNG is somewhat smoother), but they're minimal and certainly not easy to spot at 100% zoom.

The only problem I could find was around straight lines that may get a little 'zippery', but even so, at 100% zoom it was hardly noticeable. As long as you don't overdo it on the Contrast threshold, things look great. 👍

[heckflosse]

@Thanatomanic I can confirm your observations. On low-ISO images there is almost no difference between RCD+VNG4 and RCD+bilinear. On high-ISO images (I tested some ISO 6400 .orf files) I can see a difference even at 100% (where VNG4 gives better results than bilinear for flat regions).

Nevertheless I think, the speed gain of RCD+Bilinear compared to RCD+VNG4 is worth to keep at least this combination in code. I'm not sure about other combinations. RCD is the fastest demosaicer which gives good results for detailed regions. It also works better with Capture Sharpening than Amaze or DCB.
Is it worth to blow up the gui with further dual demosaicers (Amaze+Bilinear, DCB+Blinear)?

In my opinion it's not worth it.

[ff2000]

Is it worth to blow up the gui with further dual demosaicers (Amaze+Bilinear, DCB+Blinear)?

Probably rewrite the GUI. Add a checkbox to the demosaicker combobox "demosaic flat regions with" and add another combobox to let the user select "VNG4 (for better results)" and "bilinear (for faster demosaicking)". If you come up with a new super duper Algo that is both fast and has high IQ, or has other benefits, you can easily add that.

[heckflosse]

Probably rewrite the GUI. Add a checkbox to the demosaicker combobox "demosaic flat regions with" and add another combobox to let the user select "VNG4 (for better results)" and "bilinear (for faster demosaicking)". If you come up with a new super duper Algo that is both fast and has high IQ, or has other benefits, you can easily add that.

That would mean I had to offer amaze+bilinear and rcd+bilinear as options, because you could select that...

[ff2000]

That would mean I had to offer amaze+bilinear and rcd+bilinear as options, because you could select that...

Or you filter (delete or gray out demosaickers that are not supported to be dual demosaicked) the list of "detail" demosaic algos when the user clicks the dual checkbox. If the currently selected main demosaicker doesn't want to be used in dual demosaicking highlight the checkbox and show a tooltip "the currently selected demosaicker isn't supported as Dual Demosaic option." Then either fallback to RCD or whatever and/or ask the user to select a different demosaicker. Of course if you want to offer amaze+vng4 but not amaze+bilinear things will get a little bit tricky and confusing. A "? help" button could explain what's going on and which pairings are supported. You also can disable "flat region" demosaicers that can't be paired with the currently selected main demosaicer.

(I don't want to force you into this. Just wanted to work on a way to enable more Dual Demosaic options without cluttering the list of demosaic algorithms.)

[heckflosse]

Of course if you want to offer amaze+vng4 but not amaze+bilinear things will get a little bit tricky and confusing.

That's what I meant.

Though maybe the easiest solution is to support them too and follow your approach ;-)

[Thanatomanic]

I like your idea @ff2000, but I don't like the idea of a checkbox _and_ another dropdown. Couldn't we simply have two dropdowns named Demosaicer 1 and 2? The second one has the options None, and some others depending on the selected algorithm in the first. The tooltip can explain what the second demosaicer is for.

We would need to maintain a list for all allowed possibilities. That list could of course expand (or shrink) in the future.

[heckflosse]

@Thanatomanic

The second one has the options None, and some others depending on the selected algorithm in the first.

I guess the first dropdown is for the detailed demosaicer and the second dropdown is for the flat demosaicer. Assume you have chosen RCD as first and bilinear as second, Now you change your mind and use Amaze as first demosaicer (Amaze+bilinear currently does not exist). What shall be used as second demosaicer? None or VNG4?

[Waveluke]

I am certainly in favor of having Amaze+Bilinear, as the speedup could certainly manifest itself noticeably when doing raw video processing for low iso footage.

[heckflosse]

@Waveluke

I am certainly in favor of having Amaze+Bilinear, as the speedup could certainly manifest itself noticeably when doing raw video processing for low iso footage.

That's a reasonable argument for me to add Amaze+Bilinear as well :+1:

For your usecase it might be worth to measure and adjust some parameters

Just start RT from console to get the measures there and make some tries changing the Tiles per Thread settings. May give you a speedup...

[IainMF]

I am not a programmer, so I may have misread your code, but have you considered splitting the greens and interpolating them like the red and blue channels, then averaging them?

The advantage of this is a smoother result, less noise and no zipper artifacts. It may help in detecting smooth areas in noisy images as well.

[heckflosse]

@IainMF Sorry for late reply. My intention was to get a very fast replacemant for the slow VNG4 for flat regions of dual-demosaic. Bilinear output is very close the VNG4, while your proposed method is much smoother.
Left VNG, middle bilinear, right your method

[heckflosse]

I forgot the mention that the last commit reduced memory usage of bilinear dual-demosaicers by width * height * 8 bytes, and that I can make the same optimization for xtrans dual-demosaicers.

Shall I integrate the optimization for xtrans dual-demosaicers into this branch?

[Floessie]

@heckflosse

Shall I integrate the optimization for xtrans dual-demosaicers into this branch?

I think it's better off in dev...

[heckflosse]

I forgot the write up the latest measurements.

AMD FX8350 (8 cores at 4 Ghz) on a 45 MP raw from Nikon D850:

RCD+VNG4 : 2575 ms
RCD+Bilinear: 975 ms

[Thanatomanic]

I guess the first dropdown is for the detailed demosaicer and the second dropdown is for the flat demosaicer. Assume you have chosen RCD as first and bilinear as second, Now you change your mind and use Amaze as first demosaicer (Amaze+bilinear currently does not exist). What shall be used as second demosaicer? None or VNG4?

This describes my intentions correctly regarding the dropdowns. I think it would make most sense to have the second dropdown switch to None when you select a primary demosaicer that does not support a secondary one. Maybe it's even better to not _switch_ the second option, but simply _disable_ it.

We can merge, but I think we should still discuss the GUI.

[heckflosse]

@Thanatomanic

We can merge, but I think we should still discuss the GUI.

In fact, we should discuss the GUI before we merge :smiley:

[heckflosse]

@Thanatomanic After my latest changes, now it will be easier, because every primary demosaicer (Amaze, DCB, RCD) supports every secondary demosaicer (VNG4, Bilinear)

[heckflosse]

I will create a pr now. Maybe we come to a good gui solution this way ;-)