Rawtherapee: Support for Pentax pixel shift files

[heckflosse]

Pentax K70 supports pixel shift mode. In this mode 4 shifted frames are recorded and available in the raw file.
How it works
In first step we could allow access to each of the frames in rt. That's easy, as dcraw already supports that. We would need a frame selector somewhere in gui of the raw demosaic tab and pass the value (1,2,3,4) to dcraw.

I will make a new branch with a prototype to allow testing.

In second step we can define a new 'demosaic' method pixel shift which combines the 4 frames to one image.

[iliasg]

May be this will help .. https://github.com/tomtor/dcrawps

[heckflosse]

@iliasg :+1:

[heckflosse]

@Hombre57 Can you help me with the gui part (a combobox somewhere in raw processing tab to select the number of the frame) when I pushed my changes to allow extraction of individual frames to a new branch (already coded and tested)?

Ingo

[heckflosse]

Prepared engine to handle individual frames

[Hombre57]

@heckflosse

Sure ! Should I add the pp3 parameter as well, or will you do it ?

[Hombre57]

Btw, could you look at the files from Fuji S4/S5 bodies, where 2 images are embedded with different sensor sensibility, it should work the same way. I remember having made some test, and the change was rather trivial.

[heckflosse]

@Hombre57 Great, yes, please add pp3 parameter as well. I'll have a look at S4/S5 too

[heckflosse]

@Hombre57 I pushed two bug fixes. Now it's possible to extract 2nd frame for Fuji S5Pro (did not test yet with S4 RAF)

[heckflosse]

@Hombre57 please don't pull pixelshift atm. I have to make a few corrections to get it working correctly for Fuji cams

[heckflosse]

@Hombre57 Now it should work correctly. I had to fix a bug in dcraw.

[heckflosse]

@Hombre57 Thanks for the gui mockup. I just fixed the segfault we talked about in irc. There's still one issue remaining: We have not only to demosaic from scratch after changing the frame number, but also to load the file again (iirc there's no Ev... for this case atm)

[heckflosse]

@Hombre57 Also frame number is not passed to load function resulting in always loading frame 0

[Hombre57]

@heckflosse Will check that tonight.

We have not only to demosaic from scratch after changing the frame number

Isn't this what it's supposed to do ?

but also to load the file again

I guess that the file is closed once the (sub-)image has been loaded, why keeping it open ? and in your case, when to close it ?

[heckflosse]

@Hombre57
The problem is, that the event EvRawImageNum which is triggered when you select an Image number, does not load the raw file again, but it has to.

[heckflosse]

@Hombre57 Please wait with your modifications. I make some modifications first ...

[heckflosse]

@iliasg Ilias, do you have some insight into pixelshift files? I tried with a pixelshift file from K-70. Dcraw reports 4 frames and I can extract 4 frames too. The first 3 frames are different (as it should be) but the 4th frame is identical to the 1st one. This also fits to the size of the file which is about 3 times the size of one normal uncompressed file.

[heckflosse]

@iliasg Please ignore my last comment :)

[heckflosse]

I updated the pixelshift branch.
Now it's possible to view in editor and to process individual frames of raw files which contain more than one frame (i.e. Pentax pixelshift and Pentax hdr).
For Fuji S5 files I kind of disabled it atm because 1st and 2nd shot have different dimensions which is not solved yet and would crash. That does not mean that Fuji S5 is not supported. It's supported the same way as in master branch.

[heckflosse]

I made a hack to test pixelshift implementation. I'll push later today.
Here's a screenshot. Left is amaze, right is pixelshift.
Edit: Here's another screenshot

[heckflosse]

I pushed to pixelshift branch.
Now there is a new demosaic method pixelshift simple available for bayer sensors.
If you select it on a non pixelshift file it falls back to amaze demosaic automatically.
At the moment it completely bypasses raw preprocessing and simply combines the four sub frames without checking for motion. It's only for testing, code is also not clean.

[heckflosse]

I tested with a few other images. While the result I got with the image I used for the screen shots above is good, there are images where the result is bad. Seems like the shift is not always the same. I'll investigate.

Edit: Or the other shots I tested are no tripos shots...

[iliasg]

Ingo, could we have a user controlable way of defining the shift order ?

[heckflosse]

Ilias, yes, that can be done if really necessary. Isn't that order fixed? Currently I'm not sure. Maybe my test images were shot without tripod....

[heckflosse]

@iliasg Can you make a camconst.json section for Pentax K-1 files? Currently the PEF files from that cam have wrong crop.

[iliasg]

@heckflosse Ingo, OK .. it will ready in a few minutes if you are in a hurry (only color matrix and raw crop) or a few hours to also fill proper White Levels ..

[heckflosse]

@iliasg no Hurry, Thank you :+1:

[heckflosse]

I pushed some changes to pixelshift branch. I fixed a bug, implemented parallel decode of frames and cleaned the code a bit. No change to algorithm this time.

[iliasg]

No hurry meaned 5 days :(
https://github.com/Beep6581/RawTherapee/issues/3298#issuecomment-259758936

[heckflosse]

In case you want to follow the current pixelshift development

[Hombre57]

Updated GUI done. Sorry, I've forgot to reference the issue number. See commit b1e7dcb.

• the new Motion Correction combobox needs a better tooltip
• the History messages needs to be set correctly in 'default' language files
• the Fast Export and Partial Paste feature are not updated yet, will be done before the finale merge
• the show/hide enable/disable logic seem quite broken, I'll look at this tomorrow, unless you fix it yourself
• the pixelshift method probably need to be updated to take care of the new enum

[heckflosse]

@Hombre57 Thanks a lot :+1: I pushed the enum related changes

[Desmis]

I read the 2 texts (in Pixls.us and issue 3489)...and download some images.... I am not sure to all well understand.

1) I cannot download first files in Pixls.us - they are expired : IMPG0003.dng, etc.
2) one of the problems is to detect motion...and noise

• The noise does not correspond to a true Gaussian distribution - one of the best noise evaluation is MAD (with wavelet, see Ftblockdn.cc).
  In fact the curve in bell is very deformed, to the left of the average the curve is compact, and right very dilated. This means that using sigma is not very appropriate in this case, normally with 3 sigma we get about 99.5% of the data, in the case of noise it will be much less, especially in the reds.
  Mathematically this results in high KHI2 values.
  Perhaps can we reduce the noise by a Median (at least 7x7 for chroma...).

But the way in which the problem is approached seems to me to be relevant :)

• for motion, our eyes are very sensitive to luminance, which in the case of images close to the raw, is fairly well represented by the green channels

In branch newwavelet for "merge HDR", I use wavelet to merge images, with suppression of the lowest levels on one of the images, and keep them for the other, at the choice of the user.
By adjusting the number of levels difference, the motion problem is solved: nothing, 1, 2, 4 pixels, etc.
But here, it seems to me, that wavelet will be complex, and will cost a lot of processing time and memory

jacques

[Desmis]

Oh I forgot, congratulations for this great work :)

[iliasg]

Jacques, thanks for the review ..

In fact the curve in bell is very deformed, to the left of the average the curve is compact, and right very dilated. This means that using sigma is not very appropriate in this case, normally with 3 sigma we get about 99.5% of the data, in the case of noise it will be much less, especially in the reds.
Mathematically this results in high KHI2 values.

Why "especially on the reds" ?. We work on raw data without WB applied

This bell curve deformation only happens for very low levels and I think it is not significant ..
It comes from ..

• the nature of the photon noise ( Poisson distribution)
• raw-denoise operations (like removing outliers)
• clipping of raw values at Black level (K-1/K70 do not clip at low ISOs clip slightly at high ISOs, K3II clips everywhere)

We have 3 noise sources in our stdev calculation

• Photon noise which follows Poisson Distribution .. Poisson distribution is very-very close to Gaussian for electron counts > 10 .. even for λ=5 it's close .. 5 is very-very dark relative to Full Well of around 50000 electrons fro ISO100
• Read noise which follows Gaussian distribution by nature .. except if raw_denoise and/or clipping occur.
  Read noise is significant only for very low levels for ISO100 a bit more as ISO rises ..
  We already take care of raw_values_clipping at Black Level and possible raw_denoise with our measures of read noise (black frames) by using only the positive values, reconstructing the distribution by mirroring and measure stdev on the reconstructed data.
  We talked with Ingo about clipping at BlackLevel in RT and he says that RT doen't clip below BlackLevel values if they exist (i.e. if they are not already clipped by the camera).
• Pixel Responce Non Uniformity which is again gaussian_like but with fatter tails but it is symmetrical
• The Central Limit Theorem says

when independent random variables are added, their sum tends toward a normal distribution (commonly known as a bell curve) even if the original variables themselves are not normally distributed

So I think we would be fine with supposing a gaussian disribution and relly on stdev as criterion .. if we have a robust estimation of mean.
The real problem is that we have no robust estimation of mean, because for Green differences we only have two measures at each place of the grid and we use their average as mean.
Unterestimating the mean gives us smaller stdev and this drives to overestimation of move i.e. we detect movement (or light change) when in fact the difference between the two greens comes from natural noise. The inverse happens when we overestimate the mean ..

If we could have a mathematical approach for predicting (and couteract accordigly) the mean_error we would be in better shape. You mentioned chi_square .. can we use this (or any other statistic tool) somehow for better mean estimation ?. Or any other method like MAD .. I am not good with statistics so I need help from a math/stats guy on this.
If not, I am inclined to additionally use data from the neighbouring pixels .. either a weighted average or maybe median of 3X3 grid area or median of 3X3 cross

ByTheWay .. we can fine tune the stdev responce parametrically in the current GUI .. increase the read noise slider to decrease the response at the darks, increase PRNU to decrease the responce at highlights

normally with 3 sigma we get about 99.5% of the data

This is the probability of a sample being within -1.5 to + 1.5 stdevs .. we don't use this but the probability of a pair of samples to differ by more than 3 stdevs. I calculated this in excel by using 0.1 stdev wide sectors and calculate the sum of probabilities for our pairs .. it's around 96.5% for sigma = 3.0 stdevs .. So I think the default criterion is fine at 3.0 stdevs .. if the difference of a pair is more than 3.0 stdevs then motion is detected.

[Desmis]

We will not fight like ragpickers, about the problem of noise. :)
I know the different types of noise for having worked deeply the algorithm of Denoise

Although the photon distribution is Gaussian, but when examining an entire image with over exposed areas, under exposed areas (with large amplitudes of dynamics - up to 13IL) and color differences, there are significant variations in total noise.

Each color photon has an energy which depends on the frequency according to Planck's formula. E= h*f (h planck constant, f frequency).
Which brings about a difference of energy which strikes the sensor according to the color.
The relationship between blue (sky) and red (flower) areas can reach very high values, which take into account all the noises due to photons, exposure, etc. on images that appear not very noisy !(eg colorspace_flowers.pef where maxima are between 10 and 300 for MAD)
To check this try "Noise reduction" - "preview" on different areas (zoom about 500% or more)of an image. This ratio can reach a value of 30 (or more) for the maxima.

With 3 standard deviations, if the distribution is normal, exactly 99,865% of the total population is covered, between the beginning of the left part of the curve and the right part with 3 sigma.

I did not look at the code in detail, but it seems important to be able to differentiate the sigma as a function of the real noise (reminder: that on an "normal" image has maxima that vary in a ratio of 1 to 30).
But this is another matter, which involves evaluating the noise in a very small area around the pixel to be processed...
Maybe it is that realizes the current code, but I do not know I only started looking at it since last night.

Go further and use local KHI2 (which is statistical) or MAD emerges from complex problems, which risk making the constants configurable, considerably increase complexity and processing times.
In this case (of complexity) why not use wavelet which are fantastic tools for analyzing and processing noise, and isolating parts of images (1, 2 , 4 pixels)...but it will be very very complex...and time processing ???
jacques

[heckflosse]

Status report:

We are currently working hard on some new methods to reduce the number of false motion detections while increasing the number of true motion detections. As soon as the the stuff is worth to be pushed to pixelshift branch I will do that...

[Desmis]

Happy holidays to everyone :)

[heckflosse]

@Desmis Thank you. Happy holidays to you as well.

[heckflosse]

I pushed a lot of new stuff to pixelshift branch and André already made a win64 build (the last in the list).

[Hombre57]

I don't know what the gazillions of options does and how to use them but still, I managed to have very detailed images with this new feature, so thanks a lot @heckflosse and @iliasg for have done such a great tool.

Can I ask what the "Median" option does ?

[heckflosse]

@Hombre57

I don't know what the gazillions of options does and how to use them

I will write a tutorial soon. Some of the options surely will not be in the final release.

Can I ask what the "Median" option does ?

Isn't the tooltip understandable?

Use median of all frames instead of selected frame for regions with motion.
Removes objects which are at different places in all frames.
Gives motion effect on slow moving (overlapping) objects.

One use case for 'Median' are shots with water (e.g. the waterfall example from dpreview)

[heckflosse]

@Hombre57

Left is without Median using 4th frame for regions with motion and right is using Median:

[Hombre57]

First, I didn't even looked for a tooltip, sorry for that. But even with the tooltip, I couldn't imaging the use case (in fact I don't fully understand the tooltip, but once translated it should be ;) ).

That median feature is in itself a very cool feature ! I guess that someone will ask to add that feature even for non PS images within a week after the Release :)

[Beep6581]

Some feature requests based on latest dev 5.0-r1-gtk3-259-g2228159f:

• The Editor Toolbox now shows a "Sub-image" combo with values 1-4, even when I edit a plain old raw file. It would be nice if this combo was hidden or grayed-out when editing an image without any sub-images, and if the numbers it shows corresponded to the sub-images the image contains.
• The File Browser Toolbox shows all the new Pixel Shift options in the "Sensor with Bayer Matrix > Demosaicing" frame. It would be cleaner and more user-friendly if all of these went into a "Pixel Shift" frame (or some other generic word if these things are useful for more than just Pixel Shift files). Either a sub-frame in "Demosaicing" or a new "Pixel Shift" frame under "Demosaicing".
• We should spell this feature consistently. Pentax calls the technology "Pixel Shift", so if we use that name then it should be "Pixel Shift" everywhere.

[heckflosse]

@Beep6581
The spelling problem can be solved by making the demosaic method names translatable. Currently they are not translatable and the method name is also used as value in pp3 file.
Making them translatable would also allow to translate 'none' method and use 'AMaZE' instead of 'amaze' ;)

I will take that part now.

[Beep6581]

@heckflosse :+1: I can help clean up default - let me know when is a good time.

[heckflosse]

@Beep6581 I'm already working on that and also on the Sub-Image issue. I will let you know when it's ready.

[heckflosse]

@Beep6581 Status report: I made the changes for translation of demosaic methods and also for sub-image selector locally and they work fine.
I want to use this opportunity to also hide xtrans gui-frame for bayer files and bayer gui-frame for xtrans files. What do you think?

[heckflosse]

Status report: Now also hiding xtrans frame when bayer files are opened and hiding bayer frame when xtrans files are opened works fine here. I will push tomorrow.

[heckflosse]

Further update: Now also hiding the whole raw tab when opening a non-raw file works :)

[agriggio]

Disclaimer: haven't tried yet.
But wouldn't be better to gray out the tab instead of hiding it? I would find it clearer...

[Beep6581]

@heckflosse great!

We need to agree as a team on how to handle the Editor tab's toolbox regarding choices which are invalid at a given time - do we hide them or gray them out? Remember that the File Browser's toolbox shows all options.

I've been thinking about this sporadically for some time and I'm of the opinion that it's better to gray them out instead of hiding them. That means that the user won't be surprised by tools and options appearing and disappearing depending on the condition, i.e. tools which depend on type of raw file, tools which depend on raw vs non-raw file, tools which influence other tools.

[agriggio]

@Beep6581 I vote for grey out, FWIW

[heckflosse]

I'm ambivalent about hiding or graying out.
I will push the changes (with hide mode) to psgtk3 branch soon.
Then everybody can test and we can discuss.

Personally for the raw tab I prefer to gray out instead of hide when loading non raw files.
For the sensor specific panels I prefer to hide instead of gray out because most users have either an xtrans cam or a bayer cam. Very few have both types.

[Desmis]

@heckflosse

I agree with your last proposition :)

[heckflosse]

I pushed my changes to psgtk3 branch.

For raw tab I used 'gray out'
For sensor specific panels I used 'hide'

But of course we can change that :)

Here's a link to an xtrans file for testing.

And here you can find a file with 2 frames in case someone wants to test that the sub-image selector now corresponds to the number of sub-images in a file
Hint: For whatever reason this file downloads with extension tif.
You have to change the extension to cr2

[agriggio]

I prefer to hide instead of gray out because most users have either an xtrans cam or a bayer cam. Very few have both types.

I have to admit (for the nth time) that I'm really new to photography, but I had no idea that bayer vs xtrans was as polarizing as vi vs emacs :smile:

More seriously though, I still prefer greying out, but it's not a strong preference.

[heckflosse]

I guess (not tried yet) if we gray out for example the xtrans panel, we have to collapse it first, because once it's grayed out we can not close it anymore and that's not what we want. Or should we let the expander active and only gray out the corresponding frame?

[agriggio]

Or should we let the expander active and only gray out the corresponding frame?

I vote for this, yes.

[Beep6581]

After testing the latest psgtk3 I admit that I like things the way they are, but there are certain benefits to not hiding the bayer/x-trans demosaicing tool entirely:

• Seeing both but with only one usable means it's clear to the user that, as an example, enabling auto CA correction in one will not enable it in the other. When the X-Trans Demosaicing tool is hidden and the Bayer visible, or vice-versa, the user might think the two are the same tool and be confused.
• Seeing the grayed-out sub-frame combo when loading a raw photo with only one frame means users will know that the option is there.
• Consistency requires that we do the same to other tools/widgets based on whether they are usable with the currently loaded image. Currently we're only judging whether hiding demosaicing-related tools makes sense, but by making a decision here we are in fact (if we decide to go with consistency) committing to impose this decision on other tools which may or may not exist yet. That makes the choice quite difficult :)

Cons:

• Graying out doesn't help reduce the UI. Hiding is cleaner.
• Seeing a disabled sub-frame combo means they will have to figure out why it's disabled.

[heckflosse]

Seems neither me nor @agriggio nor @Beep6581 are really sure about the desired behaviour concerning hide/gray out of sensor specific tools.
For that reason I would also like to get opinions from users: @michaelezra and @sguyader : Please comment

[Hombre57]

It's easy to collapse a Tool before graying it out.

However if we look at some widgets in tools, we support both method with the following general rule (not sure it's always the case, but it will give an idea) :

• if a combobox is swapping from one _mode_ to another, the resulting widget displays the according widgets only, i.e. some widgets will be hidden or shown depending on the active mode
• if a checkbox _enable/disable_ something, that thing is made sensitive or unsensitive

We could consider the image type or sensor type as a _mode_, and then show the according widgets only. So with this scheme in mind, depending on the image type, the Raw tab should be hidden for standard images, and the unused XTrans or Bayer subframe too.

Note that Everything has to be shown (nothing hidden!) in the Batch Editor tools. It would be too complex to handle if you select multiple image type.

But that's just my 2 cents.

[heckflosse]

Ok, now we have 6 opinions.

• I prefer to gray out the raw tab for non raw (tiff, jpeg) files and to hide the unused sensor specific panels.

• @Hombre57 prefers to hide both

• @Beep6581 seems to be ambivalent

• @agriggio seems to prefer graying out both

• @sguyader prefers hiding the unused sensor specific panels but is ambivalent about the raw tab

• @Desmis prefers hiding the unused sensor specific panels but didn't comment about the raw tab

[sguyader]

I'm bit ambivalent too regarding the raw tab:
At first I was in favor of hiding the raw tab altogether for kon-raw images, but now I wouldn't mind tinder it simply greyed out.
I'm also in favor of hiding the unused sensor-specific panels.

[heckflosse]

@sguyader Sébastien, thanks for your comment! I updated my post

[agriggio]

After reading other people's opinions, and thinking about this a bit more, I slightly changed my mind :-)
I vote for greying out the RAW tab, but hiding the sensor-specific stuff (and the choice about multiple frames for single-frame RAWs)

[Beep6581]

I walked the dog, I had my morning coffee, I sat on a bench and watched the birds, I thought things over, concluded that what works best in one case does not work best in another, and I decided that as I'm unable to come up with one solution which is ideal in all cases I'm fine with whatever you choose.

[gaaned92]

Sorry to interfere, but I put my one cent.
I should prefer:

• to hide functions or tab that are never used with a kind of image (e.g. raw tab for jpeg image, bayer sensor for xtrans...) in order to avoid to bloat the gui
• to grey functions that are context sensitive (they are active or not depending on other settings)
  André

[heckflosse]

Ok, after reading all the comments again I came to the conclusion that the approach of psgtk3 branch (hiding unused sensor specific panels and graying out raw tab for non-raw images) is at least better than to show them always.
I would like to merge psgtk3 into dev again soon. Any objections?

[Desmis]

I agree

No objections :)
jacques

[heckflosse]

Merged!

[Beep6581]

Saving a partial profile with nothing selected results in the file containing this extra data:

[RAW]
CAAutoStrength=2

[RAW Bayer]
PixelShiftMotion=0
PixelShiftMotionCorrection=5
PixelShiftMotionCorrectionMethod=1
pixelShiftStddevFactorGreen=5
pixelShiftStddevFactorRed=5
pixelShiftStddevFactorBlue=5
PixelShiftEperIso=0
PixelShiftNreadIso=0
PixelShiftPrnu=1
PixelShiftSigma=1
PixelShiftSum=3
PixelShiftRedBlueWeight=0.69999999999999996
PixelShiftShowMotion=false
PixelShiftShowMotionMaskOnly=false
pixelShiftAutomatic=true
pixelShiftNonGreenHorizontal=false
pixelShiftNonGreenVertical=false
pixelShiftHoleFill=true
pixelShiftMedian=false
pixelShiftMedian3=false
pixelShiftGreen=true
pixelShiftBlur=true
pixelShiftSmoothFactor=0.69999999999999996
pixelShiftExp0=false
pixelShiftLmmse=false
pixelShiftEqualBright=false
pixelShiftNonGreenCross=true
pixelShiftNonGreenCross2=false
pixelShiftNonGreenAmaze=false

[heckflosse]

Hmmm :(

[heckflosse]

@Hombre57 I need your help for the Pixel Shift partial profile issue reported by @Beep6581

[Hombre57]

@heckflosse Looking...

[heckflosse]

@Hombre57 Thank you :+1:

[heckflosse]

@Hombre57 Feel free to push to dev. I've no changes on hold currtently.

[Hombre57]

@heckflosse In the future, could you please keep some consistency when naming procparams values ? Some of them has a lower case "P" first letter in the pp3. I've changed 2 first letter capitalization in the code, but can't do that for the pp3 without breaking compatibility, and I don't want to clutter the ProcParams::load for this. I'll be ready to push in few minutes.

[heckflosse]

@Hombre57 Yes, I will do that in future!

[Hombre57]

@heckflosse Patch committed.

[heckflosse]

@Hombre57 :+1: I knew that I can count on you! Thank you very much!!!

[Hombre57]

It's fast when it's easy 😉 I have 3 opened patch but they're more complicated than expected...

[heckflosse]

:) What's easy for you is not easy for me (and vice versa)!

[Beep6581]

but can't do that for the pp3 without breaking compatibility

There is no stable release with PS yet so it's fine to break compatibility in a dev build.

[Hombre57]

@Beep6581 Could we mention in release notes of 5.1 that pp3 for pixelshift are not compatible to 5.0 dev version ? In that case I could clean that up, other better leave it as is.

[Beep6581]

@Hombre57 we never promised to preserve PP3 compatibility across dev builds. In fact it is even explicitly stated in the dev release notes that compatibility will not be preserved (first section, second-last point). So go ahead and break.

May I add, bon anniversaire! :birthday: :tada:

[michaelezra]

It may be worthwhile extending this functionality to use any number of frames from any camera and create a super-resolution output image. Even simplest median averaging would be great! Advanced with suppression of moving artifacts is a "+"!:)

In the UI - user selects multiple images, right/click/super-resolution merge

[Hombre57]

@michaelezra You mean stitching images to create a super-resolution image ? That would be nice, but please open a new issue.