Darktable: Olympus E-M1X camera support

[Anthony1357924680]

Is your feature request related to a problem? Please describe.
I am receiving a "Failed to Read White Balance" error when trying to open a RAW file from the Olympus E-M1X Camera. I suspect that it's because this is a newer camera and is not yet in your supported cameras list.

Describe the solution you'd like
I would like support added.

Describe alternatives you've considered
I have considered using Olympus Workspace and RawTherapee, which will both open these files, but I prefer Darktable over either of these alternatives.

Additional context
I have uploaded a sample RAW file to pixls.us. Your camera support page says to attach "enough sample files", but I don't know what samples you are looking for. The page at pixls.us says that series of images are not wanted. If you need additional samples, please reply letting me know what is actually needed. I would like to help, but it's very confusing as to what is actually needed.

[BlueStilton]

Second that. I just got my E-M1X and I cannot edit its RAW (.ORF) files, using DarkTable 2.6.1, with the same error. Resembles Bug #12575 in RedMine.

Example is too large to upload here :- so it's at the URL below:

https://ye.olde.dev/M1X_RAW_FAILS_WHITE_BAL.ZIP

I'll go browse pull reqs but since this is 'Open' I'm guessing I'll either have to assist or wait?

[LebedevRI]

Sorry, got sidetracked, will look into backlog tomorrow.

[BlueStilton]

Uploaded ORF to raw.pixls.us as well. I was tracing through syscalls just before the error, noticed there was logging to stderr, captured that:

RawSpeed:Unable to find camera in database: 'OLYMPUS CORPORATION' 'E-M1X' ''
Please consider providing samples on https://raw.pixls.us/, thanks!

[rawspeed] (20190405_0026.ORF) bool rawspeed::RawDecoder::checkCameraSupported(const rawspeed::CameraMetaData*, const string&, const string&, const string&), 6line 168: Camera 'OLYMPUS CORPORATION' 'E-M1X', mode '' not supported, and not allowed to guess. Sorry.

[temperature] failed to read camera white balance information from 20190405_0026.ORF'! [temperature]OLYMPUS CORPORATION E-M1X' color matrix not found for image
[temperature] failed to read camera white balance information from `20190405_0026.ORF'!

Anything I can do to help create the necessary tables/code.. I'm a C dev first and a photographer second, so please tell me what you need (me to do or give you).

[BlueStilton]

Looking at the comments in common/colormatrices.c (and having Bahi from https://shootraw.co.uk/ explain what those were about) it would appear that I would have to carefully shoot a colour chart under direct sunlight with as few reflected sources as possible, so I ordered what appears to be a favourite in the comments: "Wolf Faust IT8". But where to go once I have those photos?

[Anthony1357924680]

It's been a long time since I've done anything in C so I can't contribute much there, but I found a temporary workaround.

Install exiftool and run the command:
For 1 File:
exiftool -Model="E-M1MarkII" FILENAME.ORF
For all in the folder:
exiftool -Model="E-M1MarkII" .

Darktable then thinks that the file is from a different camera and opens it without an error. Of course, the exported file also has the wrong EXIF data, so you need to run the command on the exported files and set it back to "E-M1X". I'm sure that there's probably some data that are not properly reading into the software, but at least it makes it usable for an amateur like myself. (This idea is not mine - I think I had used the same trick when I got the E-M5II years ago, but I'm too lazy to try to find who the original poster was)

[LebedevRI]

Looking at the comments in common/colormatrices.c (and having Bahi from https://shootraw.co.uk/ explain what those were about) it would appear that I would have to carefully shoot a colour chart under direct sunlight with as few reflected sources as possible, so I ordered what appears to be a favourite in
the comments: "Wolf Faust IT8".

Mainly the color matrix from Adobe DNG Converter is needed (i.e. take the raw, convert to DNG).
That is where the trouble begins. DNG Converter 11.1 is the last one that appears to work under wine,
and this camera is only supported by 11.2 / 11.2.1; So this issue is stuck for the time being,
unless someone with windows can help convert the raw to dng (same with https://redmine.darktable.org/issues/12619)

But where to go once I have those photos?

Not an answer to the question at hand, but i have a lurking suspicion that it would be a good idea to to fork https://raw.pixls.us into a special charts.pixls.us, for color chart pics.

[BlueStilton]

Okay, I had a Windows 10 QEMU image lying around from my reverse-engineering days ;-)

I got DNG Converter for Windows and converted the raw you linked to, above:

Olympus - E-M1X - 16bit (4_3).dng.zip

(zip because github won't take .dng).

I'll do the same for the file in https://redmine.darktable.org/issues/12619, and any others I can help out with, just let me know.

[LebedevRI]

Okay, I had a Windows 10 QEMU image lying around from my reverse-engineering days ;-)

I got DNG Converter for Windows and converted the raw you linked to, above:

Olympus - E-M1X - 16bit (4_3).dng.zip

(zip because github won't take .dng).

Great, that will do, thank you!

I'll do the same for the file in https://redmine.darktable.org/issues/12619, and any others I can help out with, just let me know.

Yes, please!

[LebedevRI]

I'll do the same for the file in https://redmine.darktable.org/issues/12619, and any others I can help out with, just let me know.

Yes, please!

(it's https://raw.pixls.us/getfile.php/3016/nice/Sony%20-%20ILCE-6400%20-%2014bit%20compressed%20(3:2).ARW)

[BlueStilton]

(it's https://raw.pixls.us/getfile.php/3016/nice/Sony%20-%20ILCE-6400%20-%2014bit%20compressed%20(3:2).ARW)

Yeah I'd guessed form the 'Sony' '6400' and exact same file size, but I asked to check to make sure.

[BlueStilton]

Oh well those color charts I ordered will come in handy at some point ;-)

I'll be building darktable from sources now, so I'll be ready to test when the time comes.

I'm curious as to how that conversion to DNG fits your dev workflow in adding camera profiles? Is it easier to parse? Better documented? (Only if you can link me to some README or something, not that important just curious).

[BlueStilton]

Install exiftool and run the command:
For 1 File:
exiftool -Model="E-M1MarkII" FILENAME.ORF
For all in the folder:
exiftool -Model="E-M1MarkII" .

Thanks! That may actually save me for now.

I'd prefer for actual support, though, but if darktable turns out hard to build from sources on my Gentoo system, I may go with your workaround for a while.

[LebedevRI]

Oh well those color charts I ordered will come in handy at some point ;-)

I'll be building darktable from sources now, so I'll be ready to test when the time comes.

I'm curious as to how that conversion to DNG fits your dev workflow in adding camera profiles?

Well, you need color matrix. And DNG has to have one.
So that workarounds* the fact that no one here knows how to produce these color matrixes.
(not really workarounds, it's almost a cargo-cult, by default these adobe matrices are used, so unless / until something earth-shattering happens, with certain exceptions, it may be the way things work..)

[LebedevRI]

if darktable turns out hard to build from sources on my Gentoo system

Hard to build from sources on gentoo? That is a joke right? :)
https://gitweb.gentoo.org/repo/gentoo.git/tree/media-gfx/darktable/darktable-9999.ebuild
(not sure if that works or bitrot, not using gentoo for a while now..)

I may go with your workaround for a while.

Be warned any and all modify operations on raws are likely lossy (as in may irrecoverably destroy data),
so don't do that without having a backup.

[BlueStilton]

Well, you need color matrix. And DNG has to have one.
So that workarounds* the fact that no one here knows how to produce these color matrixes.
(not really workarounds, it's almost a cargo-cult, by default these adobe matrices are used, so unless / until something earth-shattering happens, with certain exceptions, it may be the way things work..)

I'd almost start a PhD in psychology/anthropology to study the intellectual interactions at the edges of entrenched commercial orgs <-> savvy users <-> FOSS Developers... It can get extremely weird but I know exactly how that feels, having more code to my name based on reverse-engineering than on actual specs..

This will change when Open-Source hardware becomes common.. For now we're really 'between worlds'.

Should I rename my profile to 'John Frum'? :-)

[BlueStilton]

if darktable turns out hard to build from sources on my Gentoo system
Hard to build from sources on _gentoo_? That is a joke right? :)
https://gitweb.gentoo.org/repo/gentoo.git/tree/media-gfx/darktable/darktable-9999.ebuild
(not sure if that works or bitrot, not using gentoo for a while now..)

My bad in terms of wording. Obviously, the releases are always built from sources here, and that has been flawless to far. I think I even built it on my wife's OSX at some point, but always from a release. So I mean I've never, so far, built DarkTable from 'HEAD'. It also shows my age ; I was looking at the list of build deps, not at the already available ebuild that will probably just do it all for me.

Also, I've been using photography as a counterweight to more hardline technical stuff, and so I feel more like a user when I type CTRL-P dark return, than a developer, for now.

Be warned any and all modify operations on raws are likely lossy (as in may irrecoverably destroy data),
so don't do that without having a backup.

Check. Thanks. I'll probably check the changes on the head, and work with backups there, but depending on how much I shoot with the new camera in the meantime, I may just attempt a private little backport to 2.6.2 to keep it safe for general photography, in view of my general distractedness :-)

Thanks for the update!

[BlueStilton]

Gentoo ebuild for darktable-9999 ran without a hitch.
About reports "2.7.0+893~g4abc44417-dirty"
My raw M1X photos open in the DarkRoom for editing, as expected.
Great Joy! Thank you!

[BlueStilton]

private backport to 2.6.2 in operation. I'll go back to HEAD once I finish a larger project here and have time to send in detailed reports.