Rawtherapee: Support for Fujifilm X-Pro2

[Chawoosh]

Fuji makes new RAF RAW with X-Pro2 camera, and we need to work with one.
Sample file is there
https://cloud.mail.ru/public/EL6P/m6uPRH1cB

[Beep6581]

Sample raw file (mirror of the above): http://filebin.net/0f5um6e8h3

[Chawoosh]

I can find more samples if it's needed.

[iliasg]

The above sample in fact comes from http://www.focus-numerique.com/test-2870/compact-a-objectifs-interchangeables-fujifilm-x-pro2-test-terrain-18.html
RT/dcraw cannot decode this new type of RAFs .. we have to wait for D.Coffin to reverse engineer the new compression/decompression and give Dcraw support for compressed RAFs ..

[CharlesRT2]

The X-Pro 2 saves uncompressed or compressed raw files according to user choice. I believe the raw file of the snow man photo here is uncompressed:
http://www.dpreview.com/reviews/retro-through-and-through-fujifilm-x-pro2-first-impressions-review/8
The size, about 50 MB, would correspond to an uncompressed file for a 6000 by 4000 pixel image.

Since RT already demosaics the smaller RAFs of previous Fuji X-Trans cameras, it should be easy to support the new, larger file. Either the great RT team can do it, or someone could give Dave Coffin a message about uncompressed files.

[CharlesRT2]

Here is the crash report for the cited file, RT v. 4.2.510 on Windows 7 64-bit:
The instruction at 0x008eed02 referenced memory at 0xbdfca038. The memory could not be read.

[Beep6581]

@CharlesRT2 you will find Dave's contact details on this page:
http://www.cybercom.net/~dcoffin/dcraw/

[CharlesRT2]

Thanks, Beep. I emailed him.

[heckflosse]

6d7e2df solves the crash Issue and now RT opens files from Fujifilm X-Pro2, but right border is still not correct and needs to be be cropped away. Issue stays open until the border Issue is fixed.

Ingo

[heckflosse]

It seems that the left border is cut and at the right border black pixels appear...

[iliasg]

Ingo, after this patch RT reports raw crop 64 0 6032 4032 which is wrong
for a correct rendering we need 4 0 6032 4032 i.e camconst.json item
{ // Quality B
"make_model": "FUJIFILM X-PRO2",
"dcraw_matrix": [ 11434,-4948,-1210,-3746,12042,1903,-666,1479,5235 ], // DNG_v9.4 D65
"raw_crop": [ 4, 0, 6032, 4032 ], // full raw 6160,4032, Usable 6032,4032 - experimental crop
"ranges": { "white": 16100 }
},

this works for RAFs but is wrong for DNGs because of wrong pixel order
RAF as reported in exiftool
2 1 1 0 1 1 - 0 1 1 2 1 1 - 1 2 0 1 0 2 - 0 1 1 2 1 1 - 2 1 1 0 1 1 - 1 0 2 1 2 0

DNG 0 2 1 2 0 1 - 1 1 0 1 1 2 - 1 1 2 1 1 0 - 2 0 1 0 2 1 - 1 1 2 1 1 0 - 1 1 0 1 1 2

Can we change the pixel order for RAFs to the one of DNGs and use 0 0 6032 4032 raw crop ?

[heckflosse]

@iliasg Ilias, with your camconst entry the RAF gets demosaiced correctly because 64 - 4 = 60 is a multiple of 6 whereas for the DNG 4 - 0 = 4 is not a multiple of 6. Currently camconst crop differences have to be a mutliple of 6 for X-Trans files. I'll make a new branch to solve this Issue.

[iliasg]

@heckflosse Ingo, no
The raf is a 6160X4032 matrix (Raw_width X raw_height in dcraw terminology) with 6032X4032 (width X height in dcraw terminology) image area and the 128 columns at the right is black area.
The DNG has already cropped the black area at the right.

Now dcraw (RT) acts as if we should have to centralize this 6032X4032 area and leaves 128/2=64 columns out at the left and 64 columns at the right which is wrong.

Then the pixel order .. look how is the order .. top left first, top right second

The image 6032X4032 is same for RAFs and DNGs
EDIT .. no it's not the same .. the raf is 6033 (0- 6032) X 4032 (0-4031)

The DNG_order is 0 2 1 2 0 1 - 1 1 0 1 1 2 - 1 1 2 1 1 0 - 2 0 1 0 2 1 - 1 1 2 1 1 0 - 1 1 0 1 1 2 and describes fine what we see at the 6X6 top_left area (reading from right to left and top to bottom).

The RAF order (as reported in exiftool, and hopefully dcraw reads the same) is 2 1 1 0 1 1 - 0 1 1 2 1 1 - 1 2 0 1 0 2 - 0 1 1 2 1 1 - 2 1 1 0 1 1 - 1 0 2 1 2 0 and describes what we see at the rightmost 6X6 area (reading 6line columns from right to left). It happens that the same pattern applies if we choose the 4th leftmost column as a start and read 6line columns from left to right :). I think that's easy to calculate the converted order and use the full raw crop 0,0,6032,4032 without cropping differently in RAF vs DNG

[heckflosse]

@iliasg
Ilias, currently the 6x6 matrix gets not recalculated after cropping in camconst. For that reason a change of left border from 0 to 4 or from 64 to 0 gives wrong results, whereas a change from 64 to 4 works. I'll push a new branch in a few minutes which recalculates the matrix after camconst cropping. That solves the problem and works fine here.

[heckflosse]

@iliasg 382b2e5 :)

[iliasg]

OK if we want to rely on camconst.json, but my point is that Dcraw translates wrongly the raw crop and pixel order. I see that on 16Mp x-trans, Dcraw crops to 4th column !! instead of using the full usable raw size.

This should be the reason you had to increase the borders to 7 for x-trans. The wrong 4 columns left border resulted in black area displaying at the right :)

BTW as I am unable to compile the latest RT .. can I apply the patch by hand (dcraw.h and rawimage.cc) on an old version 09062a95076bb3c2b3c63fb053047df5780be226 I use for tests ?.

[heckflosse]

@iliasg I didn't want to fiddle in dcraw code too much :) Anyway, the new possibility to define x-trans crops in camconst.json which are not a multiple of 6 difference to original crop allows to define a 0-crop for 16Mp x-trans too (though I didn't test it)
I don't know, whether the patch will apply on the old code but it's only a few lines of code which you also can copy/paste to the right place in old code.

[sguyader]

I applied the patch to current master branch, it loads X-Pro2 files fine, but I don't know what I should be looking at ;) I don't see black pixels on the borders, are they cropped?
Regarding compressed raw, though, I couldn't find any to test right now. The smaller (32MB) raw file sent in post #2 is actually from an X-E2 camera. I think all the raws from dpreview are uncompressed (they report a size of 48 MB).

[heckflosse]

@sguyader With the patch there is no more black border for X-Pro2 RAF files ;-)

[sguyader]

Ok, that's what I thought initially, but I wasn't sure (I had read your posts diagonally...)
Should I try to find some compressed raw files to test?

[heckflosse]

I don't think that's necessary.

[heckflosse]

I'll merge the PR now to get more tests.

[iliasg]

I think we are fine now with x-pro2 uncompressed support. For compressed mode support we have to wait the updated Dcraw as it needs Dave's majic :).
Regarding camconst.json and the ability to fill gaps/correct errors this patch improved the power of camconst .. I only miss the ability to force a correct raw_crop when dcraw erroneously detects a smaller than the real. As it is now if we define a larger than detected raw_crop RT limits it at the detected values. It's a bit dangerous but would be possibly useful to be able to overcome this limitation.

[LibRaw]

We just opensourced X-Pro2 compressed decoder in LibRaw:

http://www.libraw.org/news/libraw-compressed-fujifilm-raf-support

Feel free to use in RT.

[heckflosse]

@LibRaw Thank you very much :+1: