Rawtherapee: Fujifilm compressed raw support - X-Pro2 & X-T2

[srjacob999]

Ver 4.2.880 doesn't display the raw files from my Fujifilm X-Pro2 properly.

I downloaded the file: RawTherapee_WinVista_64_Gtk3_Release_4.2.880.zip

from: http://rawtherapee.com/downloads

Attached: screenshop of result of reading the raw file, the actual raw file zipped.

DSCF0008.zip

[sguyader]

@srjacob999 I think you used the in-camera lossless compression, as the file size is 25.3MB (it should be ~48MB uncompressed). Can you try to turn off compression?

[srjacob999]

RawTherapee read the raw file fine when I turn off compression.

However, it takes a while to load the uncompressed image to the SD card, and the uncompressed image is HUGE.

Steve

[sguyader]

Hopefully someone will find how to load compressed Fufifilm raw files soon. When we tried raw files from latest cameras or firmwares, we didn't have compressed RAF files to test. Maybe if can upload a couple of different compressed files to filebin.net and paste the link here, it will help devs.

[srjacob999]

How do I do that?

[sguyader]

Just go to www.filebin.net, click on the button to upload files, then drag and drop a couple compressed raw files there, copy the link of that page and pate it here.
And please, avoid to reply from your mail address, or turn off the including of the previous message as it spams the content of your message.

[srjacob999]

Here’s 5 of them for you.

Adobe DNG Converter 9.5 has no problem reading the compressed raw files. ACR 9.5 doesn’t either. Unfortunately, I am using Photoshop CS6, which can only use ACR 9.1.1, which won’t read the raw files (compressed or not compressed). On1 doesn’t seem to have any problems with them either, nor does FastPictureViewer Codec Pack.

http://filebin.net/zd6jl2jrzu

[sguyader]

@srjacob999 thanks for the upload. Maybe we need to wait for dcraw to include support for compressed RAF files to bring it to Rawtherapee. In, the meantime you can convert your files to DNG and process them in RT (I know it's not optimal beacause of extra space taken by the DNGs and time used for conversion).

[srjacob999]

Could you please bring it to their attention.

Thank you.

I have decided that, for the moment, that DNG is probably the way to go. I can process the DNG directly in Photoshop CS6 Adobe Camera Raw.

[srjacob999]

3 Files uploaded.

[sguyader]

I just saw this post on the FujiX DPReview forum, the guys developing libraw seem to have found a way to decode compressed raws from Fuji cameras. I hope this can help bring support for these compressed files to Rawtherapee.

[heckflosse]

@sguyader Sébastien, that's great news! Can you keep me informed about the progress here please?
Ingo

[srjacob999]

[Post edited by Beep6581: truncated whitespace for readability and removed email links]

Thank you. Thank you. Thank you.

I downloaded the Fast Raw Viewer and it displays my raw files perfectly. When do you think it can be incorporated into Raw Therapee?

What is interesting is that Irfanview also can handle the compressed raw from this camera. I’ll sometimes use that to produce jpegs.

What would be REALLY REALLY REALLY nice is to make Raw Therapee into a plug-in for Photoshop. That way, you wouldn’t have to “develop” the raw file into a tiff before bringing into Photshop. I use Photoshop CS6 and refuse to shell out $$ to Adobe to go to Photoshop CC just to handle my raw files. I currently use the Adobe raw-> dng converter, but I don’t like the way it handles the raw.

Could you please keep me informed as to your progress.

Thanks for your help.
Steve

[sguyader]

@heckflosse it looks like the had something like 200 use cases and that in all cases their code handled the files perfectly. They have their code on github, but I don't know how code works, so it would be more efficient, I think, if one of the RT developers could contact the guys from libraw directly: [email protected]

edit: corrected the email address

[sguyader]

I sent an email to the address above, asking them to point me to relevant part of the libraw code. I'll keep you informed.

[srjacob999]

Thanks.

[sguyader]

I got this response from Alex Tutubalin:

Dear Sir,

X-Trans compressed decoder is under testing now (as a part of our RawDigger
and FastRawViewer software).

We hope to publish Libraw branch with X-Trans/compressed decoder in a few
days (5-7 or so). We'll notify you just after publishing

Alex Tutubalin, LibRaw LLC

[iliasg]

Alex's response ..
https://github.com/Beep6581/RawTherapee/issues/3110#issuecomment-218984022

"We just opensourced X-Pro2 compressed decoder in LibRaw:

http://www.libraw.org/news/libraw-compressed-fujifilm-raf-support

Feel free to use in RT."

2016-05-10 17:40 GMT+03:00 Sébastien Guyader [email protected]:

I got this response from Alex Tutubalin:

Dear Sir,

X-Trans compressed decoder is under testing now (as a part of our RawDigger
and FastRawViewer software).

We hope to publish Libraw branch with X-Trans/compressed decoder in a few
days (5-7 or so). We'll notify you just after publishing

Alex Tutubalin, LibRaw LLC

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
https://github.com/Beep6581/RawTherapee/issues/3229#issuecomment-218178564

[sguyader]

@iliasg you beat me on this, Alex sent me an email with the message.

[iliasg]

@srjacob999 @sguyader The sample files' link expired .. please make some compressed RAFs available to test with Dcrawupdate branch

[srjacob999]

Your wish is my command. Download them here:
www.jjworld.com/images/FujifilmX-Pro2_Raw.zip

There are 2 images, each 6000x4000

Steve

[heckflosse]

@srjacob999 @iliasg As expected, updated dcraw still does not open the compressed xtrans files correctly :(

[srjacob999]

Yes, but LibRaw does.

https://github.com/LibRaw/LibRaw

[heckflosse]

@srjacob999 Yes, I know. Integrating libraw xtrans compressed decoding is on my todo list!

[srjacob999]

I just sent Dave Coffin two compressed X-Pro2 raw files so he can get dcraw working.

[iliasg]

http://www.libraw.org/news/libraw-compressed-fujifilm-raf-support
Is RT compatible with LGPL 2.1 and CDDL 1.0 licenses ?.

[srjacob999]

Yes. It appears so.

[sguyader]

Is anyone working on this at the moment? I'm pretty sure the upcoming Fujifilm XT2 will use the same compressed raw files, it would be really good to have RT to support it.

[heckflosse]

@sguyader Not at the moment. But it's on my todo list

[madSec]

Hi, I was wondering when you will find the time to implement the compressed raw support for X-Pro2 to RT. Just to prepare myself for a longer wait ;-)

[madSec]

Ok, Fujifilm X-T2 with same RAW compression just went available, maybe the now quickly growing number of users will be a motivation to raise this issue on your todo list? :-)

[heckflosse]

Yes, I know that the priority of supporting the decoding of compressed XTrans raw file raises. I'll do my very best to solve this (by using libraw code) asap. asap means not before next week because I've to drive my brother's wedding with heckflosse at the weekend :)

[madSec]

Sounds great! I wish you a fun wedding :-)

[virosh]

Hi all!
Is this going to be implemented soon or it will be leaved for next year?

[madSec]

Since even RT5 isn't out yet I wouldn't expect compressed RAW support anytime soon, was expected to be fixed in RT5.1 or so.

[heckflosse]

Somewhere (maybe in forum) I wrote that I will solve this issue after RT5 is out....

[sguyader]

Hi, the upcoming new Fujifilm cameras based on the 24Mpx Xtrans sensor, the X-T20 and the X100F, will also have lossless compression of raw files enabled. This will make 4 Fuji cameras with compressed raw.
It looks like DPreview will soon publish a "comparison of several leading Raw converters for X-Trans cameras", I hope Rawtherapee will be included in the shootout and if compressed RAFs could be supported in the version they will test, that would make good publicity for Rawtherapee in my opinion.

[sandymc]

As an FYI, be aware that the Fuji GFX 50s also has lossless raw compression, but in a slightly different form, as it's not XTrans. The Libraw guys tell me they will be publishing an update to their code that will handle the non-XTrans version, you might want to wait for that.

[heckflosse]

For information: I try to solve this issue (decode Xtrans compressed files) during the next days.

[sguyader]

Hey Ingo, that's great news, coming at the right time... to me. I should be getting my X-T2 within the next few days!
I should then be able to test things, send test files, etc.

[heckflosse]

@sguyader Great. Tests will be very welcome :+1:

[heckflosse]

I pushed a first (dirty, but working) version to the new xtranscompressed branch

Please test :)

[madSec]

That's great news, compiling right now! I will let you know soon how it goes...

[madSec]

Somehow it doesn't work for my compressed X-Pro2 files. I still get the same half white half black image when loding the RAW file. About/Version says "Branch: xtranscompressed, Version: 5.0". Am I doing anything wrong, do I have to activate anything?

[heckflosse]

@madsector You don't have to activate anything. I tested with the only file I have which is a compressed file from X-T2. Can you upload a sample from X-Pro2 to filebin.net and post the link here?

[madSec]

I don't know what I compiled before but I think it wasn't the actual branch. Now I managed to switch to the right branch but get an compilation error. I will upload the Terminal output and an X-Pro2 RAW file in a second.

[madSec]

Terminal output of compilation error
repo-rt_XTRANS_error_v1.txt

[heckflosse]

@madsector About version should say:

Version: 5.0-r1-gtk3-10-gf2bef65
Branch: xtranscompressed

[heckflosse]

@madsector I pushed a fix for the compilation error

[madSec]

I uploaded a compressed X-Pro2 RAW file: https://filebin.net/qbx89y7c1calhc7y

My installed dcraw version is v9.21 by the way, seems outdated. Or does RT come with it's own dcraw?

Edit: Ok, will try again.

[heckflosse]

@madsector RT comes with his own dcraw. Please try the fix I just pushed :)

[madSec]

@heckflosse Compiling right now, is already through the error in previous version.

[heckflosse]

@madsector Your file opens fine here :)

[madSec]

@heckflosse Works fine here as well now. I can open the file, apply a profile and save the JPG.

Thank you so much!!! I was waiting for that one for some time and can now start to work on some older images from RAW :+1:

Liebe Grüße aus England :)

PS: Plz let me know if I should do further tests or if you need any more details about my system config. Running Linux Mint 18 on a 64bit machine.

[heckflosse]

@madsector Just letting me know if it fails on some files would be great. Thanks for testing :+1:

[madSec]

@heckflosse Any time!

[Beep6581]

Screenshot showing RawTherapee successfully open a compressed X-Trans raw file:

[hchargois]

I too can report success with opening and editing a few dozens X-T2 compressed raw files. It's working beautifully. Thanks.

[heckflosse]

@hchargois Thanks a lot for testing and feedback :+1:

[jbedo]

Working for me too :).

[iliasg]

@heckflosse Ingo thanks,

Fuji has released a new model .. ( GFX-50S ) which exports compressed RAFs but in bayer form instead of x-trans
find 2 samples in https://drive.google.com/drive/folders/0B0NqktTgc54sVHNGN0NVTWVLOUk?usp=sharing

I think it should be the same compression so could belong to this issue ..

[heckflosse]

@iliasg Ilias, as expected it does not work with the files from GFX-50S

[sandymc]

FYI, the libraw folks have now released a revision to their code that will work with GFX:

https://github.com/LibRaw/LibRaw/commit/39f31c8f378a6db0623b258e36bd1fb600af98e6

[heckflosse]

@sandymc Thanks for the pointer :+1:

[heckflosse]

I would like to continue as follows:

1. Push the current code to dev to get more tests and allow xtrans users to use compressed files
2. Integrate the new libraw code for GFX-50S compressed files
3. Review the code together with @Floessie
4. Push the new code

Any objections?

Ingo

[madSec]

Sounds good I think. Could you let us know please when the current code is added to normal dev branch? Till then I will stay with the xtranscompressed branch.

[heckflosse]

I just merged xtranscompressed into dev.

[heckflosse]

I just pushed decode support for compressed GFX-50S files to xtranscompressed branch.
@iliasg It seems the crop for GFX-50S is wrong. Can you take a look?

[heckflosse]

@sguyader @gaaned92 Can you make new windows builds from dev branch to get more tests for xtrans compressed files?

[iliasg]

@heckflosse

It seems the crop for GFX-50S is wrong. Can you take a look?

camconst.json item

    { // Quality C, color data guessed to be same with X-A3
        "make_model": "FUJIFILM GFX 50S",
        "dcraw_matrix": [ 12407,-5222,-1086,-2971,11116,2120,-294,1029,5284 ], // copy from X-A3 DNGv9.8 D65
        "raw_crop": [ 0, 0, 8280, 6208 ], // full raw 9216X6210 - usefull 8280x6208
      //  "raw_crop": [ 6, 6, 8264, 6200 ], // fuji official JPEG 8256X6192 10,11,9,8 - experimental crop to match with official
        "ranges": { "white": 16300 }
    },

[heckflosse]

@iliasg Thank you!

[gaaned92]

Windows64 build dev_5.0-r1-gtk3-20-g24ea9960 uploaded at https://drive.google.com/open?id=0B2q9OrgyDEfPS2FpdDAtMVI1RG8

[srjacob999]

Doesn’t open my Fujifilm X-Pro2 Compressed raw. I get a screen with ½ black, ½ white. No image.

[heckflosse]

@srjacob999 I just tested the build from @gaaned92
Windows64 build dev_5.0-r1-gtk3-20-g24ea9960 opens a compressed X-Pro2 file fine here.

[heckflosse]

@srjacob999 Can you upload a sample file to filebin.net and post the link here?

[srjacob999]

Uploaded https://filebin.net/pdnkoyfgou3zm2z8/_DSF0005.RAF

[heckflosse]

@srjacob999 works fine here

[heckflosse]

@gaaned92 Thanks for making the build :+1:

[srjacob999]

Never got that version. Running RawTherapee_5.0-r1-gtk2_WinVista_64.

Where is this one?

[heckflosse]

@srjacob999

Where is this one?

Here

[sguyader]

Here's a version built with Gtk+ 3.18: https://filebin.net/exoqih33dsopv02q
(contains and installs debug and gdb executables)

[heckflosse]

@sguyader Thanks for making the build :+1:

[sguyader]

I just posted the link to DPReview, to get more users to try it.
Thanks for making it real, @heckflosse !

By the way, is the support for GFX 50S already included?

[heckflosse]

@sguyader

By the way, is the support for GFX 50S already included?

It's included in xtranscompressed branch. Will go into dev after code review

[srjacob999]

This version worked fine.

[heckflosse]

@srjacob999

This version worked fine.

I hope it still works fine. As well as the build from @gaaned92
And of course: you're welcome.....

[ff2000]

One issue I found: Old thumbnails (that show the half black/white picture) won't get updated. I can open and edit the compressed files just fine. Other files I opened and edited before compression support change their thumbs just fine.

[heckflosse]

@ff2000
Select the thumb(s), right click Cache/Clear from cache - full. Then open the image(s). That should build a new correct thumb.

[haklein]

is there a mac build somewhere to test?

[sguyader]

@heckflosse Ingo I just got my X-T2. I confirm that loading a compressed raw in RT works without any flaw. Great job.

[heckflosse]

@sguyader Sébastien, thanks for the feedback :)

[virosh]

I can also confirm that it works on without issues.

[heckflosse]

@sguyader @gaaned92 Because GFX 50S will be available soon, I merged the support for GFX 50S compressed files to dev.

Can you please make new windows builds? Thank you :+1:

[gaaned92]

RawTherapee_dev_5.0-r1-gtk3-82-ga42f5f4_WinVista_64 uploaded at https://drive.google.com/open?id=0B2q9OrgyDEfPS2FpdDAtMVI1RG8

[heckflosse]

@gaaned92 Thank you :+1:

[sguyader]

A gtk+ 3.18 build (RawTherapee_5.0-r1-gtk3-82-ga42f5f4_WinVista_64_gtk3.18) can found here: https://filebin.net/gn1g1bns1f5j2s9m

It's ahead of @gaaned92 's and includes the latest TooWaBlue theme version.

Are the Gtk+ 3.18 builds still needed?

[heckflosse]

@sguyader Sébastien, thank you for the build. To be honest, I don't know whether Gtk+ 3.18 builds are still needed as my own builds are also Gtk+ 3.18 ;)

[Queatz]

I'm running into this with the GFX100. Build: 20190910_1736 - is the solution to shoot uncompressed?

[Thanatomanic]

@Queatz The GFX100 in 16 bit mode gives problems, as you see in #5574

Otherwise I have just tested all Fujifilm compressed files on raw.pixls.us and haven't found any problems anymore. Closing this issue.