Rawtherapee: Adopt libexiv2 or ExifTool for metadata handling

Created on 5 Apr 2017 · 23Comments · Source: Beep6581/RawTherapee

Metadata handling in RawTherapee is far from optimal. I rip out and process chunks of metadata from ExifTool and manually merge them into the rtexif/* files. This is tedious and error prone, and in all the years that RT has been open-source I'm the only person to have done that. This issue is to adopt libexiv2 or ExifTool to outsource our metadata handling needs. Although this will initially take considerable effort, in the long run it should result in less work when it comes to having up to date camera and lens info, and adding support for reading and writing metadata to various file formats.

Robin of Exiv2 was willing to help in some capacity with libexiv2 adoption. *3

Furthermore, libiptcdata which we're using has been dead since 2009. IPTC is being phased out in favor of XMP. I believe both libexiv2 and ExifTool handle IPTC and XMP, solving that problem as well.

References:
http://owl.phy.queensu.ca/~phil/cpp_exiftool/
http://www.exiv2.org/getting-started.html#lib
https://github.com/Beep6581/RawTherapee/issues/3581#issuecomment-270509717
https://github.com/Beep6581/RawTherapee/issues/1314#issuecomment-276317545
https://github.com/Beep6581/RawTherapee/issues/2307#issuecomment-172172408

metadata enhancement

Source

Beep6581

👍3

Most helpful comment

I can open a PR that replaces rtexif with exiv2. The basics should work, but I haven't tested it much and won't be able to maintain it and/or perform the needed cleanups for integration into dev (if any).
Therefore, I'm basically asking if someone would be willing to "adopt" the branch and do the remaining work to get this merged.

If there is interest, please let me know.

agriggio on 6 May 2019

❤4

All 23 comments

Found this just today : http://dev.exiv2.org/projects/exiv2/wiki/How_does_Exiv2_compare_to_Exiftool

Hombre57 on 27 Jun 2018

If there is interest, please let me know.

agriggio on 6 May 2019

❤4

@agriggio Is there a branch we can have a look at?

Floessie on 7 May 2019

@Floessie I have a branch I can push, but as I said I won't be able to work on it. My question was like "should I push it anyway?".

If your answer is "yes", I'll push later today :-)

agriggio on 7 May 2019

Yes!

Beep6581 on 7 May 2019

Yes, of course! You don't have to create a PR, though...

Floessie on 7 May 2019

Pushed branch metadata-exiv2.

A couple of comments:

some lenses have different names, this is just the way it is with exiv2 vs rtexif
exif tags editing is limited only to editing the following tags:

User Comment
Artist
Copyright
Image Description

No other tag can be modified, added, removed
the metadata are global for each image, there's no more per-frame metadata
(I don't think exiv2 supports this)

agriggio on 7 May 2019

On a fresh clone I get this when trying to build metadata-exiv2 branch on Windows/msys2:

Z:/H2/rt56_exiv2/rtengine/imagedata.cc: In function 'Exiv2::Image::AutoPtr rtengine::open_exiv2(const Glib::ustring&)':
Z:/H2/rt56_exiv2/rtengine/imagedata.cc:45:33: error: no matching function for call to 'std::__cxx11::basic_string<wchar_t>::basic_string(const short unsigned int* const&)'
     const std::wstring wfname(ws);
                                 ^
In file included from C:/msys64/mingw64/include/c++/8.3.0/string:52,
                 from C:/msys64/mingw64/include/c++/8.3.0/stdexcept:39,
                 from C:/msys64/mingw64/include/c++/8.3.0/array:39,
                 from C:/msys64/mingw64/include/c++/8.3.0/tuple:39,
                 from C:/msys64/mingw64/include/c++/8.3.0/functional:54,
                 from Z:/H2/rt56_exiv2/rtengine/imagedata.cc:19:
C:/msys64/mingw64/include/c++/8.3.0/bits/basic_string.h:614:9: note: candidate: 'template<class _InputIterator, class> std::__cxx11::basic_string<_CharT, _Traits, _Alloc>::basic_string(_InputIterator, _InputIterator, const _Alloc&)'
         basic_string(_InputIterator __beg, _InputIterator __end,
         ^~~~~~~~~~~~
C:/msys64/mingw64/include/c++/8.3.0/bits/basic_string.h:614:9: note:   template argument deduction/substitution failed:
Z:/H2/rt56_exiv2/rtengine/imagedata.cc:45:33: note:   candidate expects 3 arguments, 1 provided
     const std::wstring wfname(ws);
                                 ^
In file included from C:/msys64/mingw64/include/c++/8.3.0/string:52,
                 from C:/msys64/mingw64/include/c++/8.3.0/stdexcept:39,
                 from C:/msys64/mingw64/include/c++/8.3.0/array:39,
                 from C:/msys64/mingw64/include/c++/8.3.0/tuple:39,
                 from C:/msys64/mingw64/include/c++/8.3.0/functional:54,
                 from Z:/H2/rt56_exiv2/rtengine/imagedata.cc:19:
C:/msys64/mingw64/include/c++/8.3.0/bits/basic_string.h:576:7: note: candidate: 'std::__cxx11::basic_string<_CharT, _Traits, _Alloc>::basic_string(std::__cxx11::basic_string<_CharT, _Traits, _Alloc>&&, const _Alloc&) [with _CharT = wchar_t; _Traits = std::char_traits<wchar_t>; _Alloc = std::allocator<wchar_t>]'
       basic_string(basic_string&& __str, const _Alloc& __a)
       ^~~~~~~~~~~~
C:/msys64/mingw64/include/c++/8.3.0/bits/basic_string.h:576:7: note:   candidate expects 2 arguments, 1 provided
C:/msys64/mingw64/include/c++/8.3.0/bits/basic_string.h:572:7: note: candidate: 'std::__cxx11::basic_string<_CharT, _Traits, _Alloc>::basic_string(const std::__cxx11::basic_string<_CharT, _Traits, _Alloc>&, const _Alloc&) [with _CharT = wchar_t; _Traits = std::char_traits<wchar_t>; _Alloc = std::allocator<wchar_t>]'
       basic_string(const basic_string& __str, const _Alloc& __a)
       ^~~~~~~~~~~~
C:/msys64/mingw64/include/c++/8.3.0/bits/basic_string.h:572:7: note:   candidate expects 2 arguments, 1 provided
C:/msys64/mingw64/include/c++/8.3.0/bits/basic_string.h:568:7: note: candidate: 'std::__cxx11::basic_string<_CharT, _Traits, _Alloc>::basic_string(std::initializer_list<_Tp>, const _Alloc&) [with _CharT = wchar_t; _Traits = std::char_traits<wchar_t>; _Alloc = std::allocator<wchar_t>]'
       basic_string(initializer_list<_CharT> __l, const _Alloc& __a = _Alloc())
       ^~~~~~~~~~~~
C:/msys64/mingw64/include/c++/8.3.0/bits/basic_string.h:568:7: note:   no known conversion for argument 1 from 'const short unsigned int* const' to 'std::initializer_list<wchar_t>'
C:/msys64/mingw64/include/c++/8.3.0/bits/basic_string.h:541:7: note: candidate: 'std::__cxx11::basic_string<_CharT, _Traits, _Alloc>::basic_string(std::__cxx11::basic_string<_CharT, _Traits, _Alloc>&&) [with _CharT = wchar_t; _Traits = std::char_traits<wchar_t>; _Alloc = std::allocator<wchar_t>]'
       basic_string(basic_string&& __str) noexcept
       ^~~~~~~~~~~~
C:/msys64/mingw64/include/c++/8.3.0/bits/basic_string.h:541:7: note:   no known conversion for argument 1 from 'const short unsigned int* const' to 'std::__cxx11::basic_string<wchar_t>&&'
C:/msys64/mingw64/include/c++/8.3.0/bits/basic_string.h:529:7: note: candidate: 'std::__cxx11::basic_string<_CharT, _Traits, _Alloc>::basic_string(std::__cxx11::basic_string<_CharT, _Traits, _Alloc>::size_type, _CharT, const _Alloc&) [with _CharT = wchar_t; _Traits = std::char_traits<wchar_t>; _Alloc = std::allocator<wchar_t>; std::__cxx11::basic_string<_CharT, _Traits, _Alloc>::size_type = long long unsigned int]'
       basic_string(size_type __n, _CharT __c, const _Alloc& __a = _Alloc())
       ^~~~~~~~~~~~
C:/msys64/mingw64/include/c++/8.3.0/bits/basic_string.h:529:7: note:   candidate expects 3 arguments, 1 provided
C:/msys64/mingw64/include/c++/8.3.0/bits/basic_string.h:514:7: note: candidate: 'std::__cxx11::basic_string<_CharT, _Traits, _Alloc>::basic_string(const _CharT*, const _Alloc&) [with _CharT = wchar_t; _Traits = std::char_traits<wchar_t>; _Alloc = std::allocator<wchar_t>]' <near match>
       basic_string(const _CharT* __s, const _Alloc& __a = _Alloc())
       ^~~~~~~~~~~~
C:/msys64/mingw64/include/c++/8.3.0/bits/basic_string.h:514:7: note:   conversion of argument 1 would be ill-formed:
Z:/H2/rt56_exiv2/rtengine/imagedata.cc:45:33: error: invalid conversion from 'const short unsigned int*' to 'const wchar_t*' [-fpermissive]
     const std::wstring wfname(ws);
                                 ^
In file included from C:/msys64/mingw64/include/c++/8.3.0/string:52,
                 from C:/msys64/mingw64/include/c++/8.3.0/stdexcept:39,
                 from C:/msys64/mingw64/include/c++/8.3.0/array:39,
                 from C:/msys64/mingw64/include/c++/8.3.0/tuple:39,
                 from C:/msys64/mingw64/include/c++/8.3.0/functional:54,
                 from Z:/H2/rt56_exiv2/rtengine/imagedata.cc:19:
C:/msys64/mingw64/include/c++/8.3.0/bits/basic_string.h:499:7: note: candidate: 'std::__cxx11::basic_string<_CharT, _Traits, _Alloc>::basic_string(const _CharT*, std::__cxx11::basic_string<_CharT, _Traits, _Alloc>::size_type, const _Alloc&) [with _CharT = wchar_t; _Traits = std::char_traits<wchar_t>; _Alloc = std::allocator<wchar_t>; std::__cxx11::basic_string<_CharT, _Traits, _Alloc>::size_type = long long unsigned int]'
       basic_string(const _CharT* __s, size_type __n,
       ^~~~~~~~~~~~
C:/msys64/mingw64/include/c++/8.3.0/bits/basic_string.h:499:7: note:   candidate expects 3 arguments, 1 provided
C:/msys64/mingw64/include/c++/8.3.0/bits/basic_string.h:481:7: note: candidate: 'std::__cxx11::basic_string<_CharT, _Traits, _Alloc>::basic_string(const std::__cxx11::basic_string<_CharT, _Traits, _Alloc>&, std::__cxx11::basic_string<_CharT, _Traits, _Alloc>::size_type, std::__cxx11::basic_string<_CharT, _Traits, _Alloc>::size_type, const _Alloc&) [with _CharT = wchar_t; _Traits = std::char_traits<wchar_t>; _Alloc = std::allocator<wchar_t>; std::__cxx11::basic_string<_CharT, _Traits, _Alloc>::size_type = long long unsigned int]'
       basic_string(const basic_string& __str, size_type __pos,
       ^~~~~~~~~~~~
C:/msys64/mingw64/include/c++/8.3.0/bits/basic_string.h:481:7: note:   candidate expects 4 arguments, 1 provided
C:/msys64/mingw64/include/c++/8.3.0/bits/basic_string.h:465:7: note: candidate: 'std::__cxx11::basic_string<_CharT, _Traits, _Alloc>::basic_string(const std::__cxx11::basic_string<_CharT, _Traits, _Alloc>&, std::__cxx11::basic_string<_CharT, _Traits, _Alloc>::size_type, std::__cxx11::basic_string<_CharT, _Traits, _Alloc>::size_type) [with _CharT = wchar_t; _Traits = std::char_traits<wchar_t>; _Alloc = std::allocator<wchar_t>; std::__cxx11::basic_string<_CharT, _Traits, _Alloc>::size_type = long long unsigned int]'
       basic_string(const basic_string& __str, size_type __pos,
       ^~~~~~~~~~~~
C:/msys64/mingw64/include/c++/8.3.0/bits/basic_string.h:465:7: note:   candidate expects 3 arguments, 1 provided
C:/msys64/mingw64/include/c++/8.3.0/bits/basic_string.h:450:7: note: candidate: 'std::__cxx11::basic_string<_CharT, _Traits, _Alloc>::basic_string(const std::__cxx11::basic_string<_CharT, _Traits, _Alloc>&, std::__cxx11::basic_string<_CharT, _Traits, _Alloc>::size_type, const _Alloc&) [with _CharT = wchar_t; _Traits = std::char_traits<wchar_t>; _Alloc = std::allocator<wchar_t>; std::__cxx11::basic_string<_CharT, _Traits, _Alloc>::size_type = long long unsigned int]'
       basic_string(const basic_string& __str, size_type __pos,
       ^~~~~~~~~~~~
C:/msys64/mingw64/include/c++/8.3.0/bits/basic_string.h:450:7: note:   candidate expects 3 arguments, 1 provided
C:/msys64/mingw64/include/c++/8.3.0/bits/basic_string.h:437:7: note: candidate: 'std::__cxx11::basic_string<_CharT, _Traits, _Alloc>::basic_string(const std::__cxx11::basic_string<_CharT, _Traits, _Alloc>&) [with _CharT = wchar_t; _Traits = std::char_traits<wchar_t>; _Alloc = std::allocator<wchar_t>]'
       basic_string(const basic_string& __str)
       ^~~~~~~~~~~~
C:/msys64/mingw64/include/c++/8.3.0/bits/basic_string.h:437:7: note:   no known conversion for argument 1 from 'const short unsigned int* const' to 'const std::__cxx11::basic_string<wchar_t>&'
C:/msys64/mingw64/include/c++/8.3.0/bits/basic_string.h:429:7: note: candidate: 'std::__cxx11::basic_string<_CharT, _Traits, _Alloc>::basic_string(const _Alloc&) [with _CharT = wchar_t; _Traits = std::char_traits<wchar_t>; _Alloc = std::allocator<wchar_t>]'
       basic_string(const _Alloc& __a) _GLIBCXX_NOEXCEPT
       ^~~~~~~~~~~~
C:/msys64/mingw64/include/c++/8.3.0/bits/basic_string.h:429:7: note:   no known conversion for argument 1 from 'const short unsigned int* const' to 'const std::allocator<wchar_t>&'
C:/msys64/mingw64/include/c++/8.3.0/bits/basic_string.h:420:7: note: candidate: 'std::__cxx11::basic_string<_CharT, _Traits, _Alloc>::basic_string() [with _CharT = wchar_t; _Traits = std::char_traits<wchar_t>; _Alloc = std::allocator<wchar_t>]'
       basic_string()
       ^~~~~~~~~~~~
C:/msys64/mingw64/include/c++/8.3.0/bits/basic_string.h:420:7: note:   candidate expects 0 arguments, 1 provided
Z:/H2/rt56_exiv2/rtengine/imagedata.cc:46:12: error: invalid conversion from 'const void*' to 'gpointer' {aka 'void*'} [-fpermissive]
     g_free(ws);
            ^~
In file included from C:/msys64/mingw64/include/glib-2.0/glib/glist.h:32,
                 from C:/msys64/mingw64/include/glib-2.0/glib/ghash.h:33,
                 from C:/msys64/mingw64/include/glib-2.0/glib.h:50,
                 from C:/msys64/mingw64/include/glib-2.0/glib/gprintf.h:21,
                 from C:/msys64/mingw64/include/glib-2.0/glib/gstdio.h:22,
                 from Z:/H2/rt56_exiv2/rtengine/imagedata.cc:24:
C:/msys64/mingw64/include/glib-2.0/glib/gmem.h:71:35: note:   initializing argument 1 of 'void g_free(gpointer)'
 void  g_free           (gpointer  mem);
                         ~~~~~~~~~~^~~
make[2]: *** [rtengine/CMakeFiles/rtengine.dir/build.make:583: rtengine/CMakeFiles/rtengine.dir/imagedata.cc.obj] Fehler 1
make[1]: *** [CMakeFiles/Makefile2:128: rtengine/CMakeFiles/rtengine.dir/all] Fehler 2
make: *** [Makefile:130: all] Fehler 2

heckflosse on 23 May 2019

@heckflosse Either you fix it by yourself, or I'll have a look later.

Floessie on 23 May 2019

Compiled successfully and without issues using GCC-7.3.0 and exiv2-0.27.0 in Sabayon.

Beep6581 on 23 May 2019

Maybe @piponazo, @D4N or @clanmills have some time to look at the exiv2 side of things (https://github.com/Beep6581/RawTherapee/compare/metadata-exiv2#diff-09e85bf37284ee79c4e479205a384db3), and if not they now know that something is happening regarding adaption of their library in RawTherapee :)

imsodin on 23 May 2019

@imsodin Stop, stop, stop!

This looks to me like a wrong parameter type to std::wstring and has nothing to do with libexiv2.

I'll have a look later.

Floessie on 23 May 2019

Should be fixed with fa63b2f7a6a2cc949d9c8a072a88eacb57fb353e, though this code isn't compiled on Linux (which I use). Take it with a grain of salt.

Floessie on 23 May 2019

@Floessie Works fine now :+1:

About this comment from @agriggio

the metadata are global for each image, there's no more per-frame metadata
(I don't think exiv2 supports this)

That's a step backwards for the HDR files from some Pentax cameras :frowning_face:

heckflosse on 23 May 2019

@imsodin Stop, stop, stop!

That wasn't specific to that issue, just in general. They have mentioned being interested to help with adoption, I am not quite sure in which context. Anyway it was more meant as a heads-up to them. Sorry if I stepped on any toes in the process.

imsodin on 23 May 2019

Stop? No problem. I'm on vacation. Let me know if you need me to get involved.

clanmills on 23 May 2019

❤1

@clanmills Does exiv2 support per-frame metadata?

heckflosse on 23 May 2019

Multipage formats such as TIFF? No, we don't.

I believe Exiv2 started off life reading JPEGs and now handles about 20 different formats. However, as JPEG is "one image per file", that's the current architecture. Although "multi-page tiff" has been on the wish-list for years, it still remains in the future.

About 5 years ago, I successfully investigated using the libtiff utilities tiffcut to split a tiff into multiple files, edit the metadata in each image and and tiffcp to bundle them together into a single file.

Sorry to disappoint you. Life isn't perfect!

clanmills on 23 May 2019

@imsodin

Sorry if I stepped on any toes in the process.

No worries, all fine. I just had to hurry before the exiv2 people wasted their time on an RT specific problem. :sweat_smile:

Floessie on 23 May 2019

@heckflosse

That's a step backwards for the HDR files from some Pentax cameras :frowning_face:

In which way are the per-frame metadata relevant? I just ask, because a short while ago (in e39726dbf7c88e9bfce2bf338ff32ef3669b52ca) we had to default to frame 0 in rtexif to prevent crashes.

Floessie on 23 May 2019

@Floessie

In which way are the per-frame metadata relevant? I just ask, because a short while ago (in e39726d) we had to default to frame 0 in rtexif to prevent crashes.

In dev the info-dialog for example shows the exposure-time for each frame of a 3-frames Pentax HDR file correctly for each sub-image selection in raw demosaic tab, while in this branch it always shows the exposure-time from first frame

heckflosse on 23 May 2019

Just wanted to add that this branch seems to also fix RT losing timezone metadata when exporting images. I.e. input RAW has date+time+timezone, and output JPG has date+time, no timezone.

erjiang on 10 Mar 2020

That sounds correct. The Exif specification doesn't support TZ. The DateTime format is a 20 byte string: YYYY:MM:DD HH:MM:SS0. Manufacturers often store TZ in the MakerNotes.

$ exiv2 -pa --grep time/i http://clanmills.com/Stonehenge.jpg
Exif.Image.DateTime              Ascii      20  2015:07:16 20:25:28
Exif.Photo.ExposureTime          Rational    1  1/400 s
Exif.Photo.DateTimeOriginal      Ascii      20  2015:07:16 15:38:54
Exif.Photo.DateTimeDigitized     Ascii      20  2015:07:16 15:38:54
Exif.NikonWt.Timezone            SShort      1  UTC +00:00    <---- from the MakerNotes
Exif.Photo.SubSecTime            Ascii       3  00
Exif.Photo.SubSecTimeOriginal    Ascii       3  00
Exif.Photo.SubSecTimeDigitized   Ascii       3  00
Exif.GPSInfo.GPSTimeStamp        Rational    3  14:38:55.9
$

clanmills on 11 Mar 2020

Was this page helpful?

0 / 5 - 0 ratings

Related issues

scrolling with the mouse wheel in the file browser is disabled when using filters on colour labels and change folder.

LordPhoenixfr · 4Comments

Preview updates incorrectly when using perspective correction

Lawrence37 · 3Comments

TooWaBlue: Warning in console

heckflosse · 5Comments

Local Adjustments: Inconsistency in Preview ΔE

jade-nl · 4Comments

Raw ca correction 'Avoid colour shift' sometimes produces a colour shift

heckflosse · 5Comments