Rawtherapee: Update documentation for compilation under macOS

[rfranke]

First of all: thank you for the great software and comprehensive documentation!

The compilation of the current dev branch was straightforward under Linux Ubuntu 18.04. I just ran the documented sudo apt install ... and picked the build commands out of the supplied build-rawtherapee shell script.

The compilation under macOS Catalina 10.15.2 was more tricky. This might be due to my ignorance on that system, but possibly an updated documentation would help as well.

It wasn't clear to me that Homebrew and MacPorts are alternatives. Some packages are only listed for Homebrew, in particular cmake and libomp. This is why I installed both and then wasted some hours treating conflicts with duplicated installations of things like expat and gtk3. The documentation should make clear that one should decide for either Homebrew or MacPorts.

I finally deinstalled Homebrew and added missing packages to MacPorts. Then things turned out considerably simpler than documented:

• MacPorts rejected to install clang-3.9. I tried clang-8.0 and clang-9.0 instead. Both failed with linking a simple program during cmake. I did nothing in the end, i.e. just used the system compiler coming with Xcode -- Apple clang version 11.0.0 (clang-1100.0.33.16). The use of a custom compiler might be documented in a separate advanced section to make things simpler to start with.
• I had to manually add omp locations for cmake with MacPorts -- as documented for Homebrew. Interestingly I did not seem to need this for Homebrew. However, an omp test failed at the end of cmake -- will do speed comparisons with the released RawTherapee 3.7 and then follow up on this one eventually.
• I skipped the libiconv patch. It would have applied to version 1.15, whereas MacPorts installed 1.16. Everything seems fine (not more sporadic crashes during startup than with the released RawTherapee 3.7).
• Section "Distribution Bundle Framework Library Versions in 5.5-dev" appears outdated.
• Last section "macOS 10.15": I did not copy the result to /Applications; just started it from the command line in the build directory instead. Then the Terminal cares about access rights to removable media. I do also need to start the released version 5.7 under /Applications/RawTherapee/Contents/MacOS/rawtherapee from the command line for access rights to memory cards.

[heckflosse]

@rfranke

However, an omp test failed at the end of cmake -- will do speed comparisons with the released RawTherapee

Without omp RT will be slow. Imho not worth to make a non omp build for machines with multiple cores...

[rfranke]

I did some comparisons with the released version 5.7 today. Went back to branch release-5.7 as the processing seems to have changed in dev. Setting
-DWITH_LTO="ON"
results in the same speed even without omp (about 4s to open a photo in the editor :-( ). Maybe something went wrong with the release build -- or with Catalina? See also #5552. This is why I'm hoping for #1678 now.
Btw: did not get any other than Xcode compiler to work. Hombrew llvm and MacPorts clang both fail to link a simple program on my 10.15.2 system.

[heckflosse]

@rfranke

Setting
-DWITH_LTO="ON" results in the same speed even without omp (about 4s to open a photo in the editor :-( ).

Can you post the console output when RT started from terminal with this enabled and opening an e.g. 24 MP raw?

[rfranke]

OK, I had a deeper look now. The measured times reduce as expected with omp. The effect on the overall wall clock time isn't that big though. Interestingly my perception of the effect of LTO was wrong.

Here are the measures using an Intel Mobile Core i5 "Haswell" (I5-4258U) 2.4 GHz with 2 physical cores. The released 5.7 build gives:

CA correcting 5599x3728 image with 2 tiles per thread
CA correction took 728 ms
Demosaicing 5599x3728 image using AMaZE with 2 Tiles per Thread
amaze demosaic took 812 ms

This sums up to about 1.5s. The total wall clock time between clicking on a tile and seeing the readily rendered photo is 3-4s (towards 3) though. Meaning about 2s other processing.

My own build of branch release-5.7 with Apple clang version 11.0.0 (no omp support) and -DWITH_LTO="ON" gives:

CA correcting 5599x3728 image with 2 tiles per thread
CA correction took 1433 ms
Demosaicing 5599x3728 image using AMaZE with 2 Tiles per Thread
amaze demosaic took 1562 ms

The total wallclock time increases to 4-5s. This reflects the loss of omp, but the other processing time of about 2s remains the same, so that the total loss of performance is not that big.

Unfortunately LTO increases the time for linking rawtherapee considerably. This is why I'm turning it off for experimental builds. An own build with -DWITH_LTO="OFF" gives:

CA correcting 5599x3728 image with 2 tiles per thread
CA correction took 1471 ms
Demosaicing 5599x3728 image using AMaZE with 2 Tiles per Thread
amaze demosaic took 1607 ms

Only minimal increase of measured times. Also the total wall clock time only increases slightly -- more towards 5s now.

[heckflosse]

@rfranke What kind of raw files did you use for your test?

[rfranke]

I'm processing NEF files from a D500 (file size about 24 MB). Do you think that 3s for opening a file in the editor are as expected?
This timing is less important when preselecting photos in the file browser (I'm shooting RAW+JPEG and only need to edit few photos). 100% zoom would be nice though, see #5591.

Meanwhile my own build works with omp using MacPorts clang-mp-9.0. It achieves the same times as the released version, if not a tad faster. Found https://trac.macports.org/ticket/58333#comment:8

sudo port -v uninstall ld64
sudo port -v install ld64 +ld64_xcode

Moreover, MacPorts fftw-3-single does not seem to contain libfftw3f_omp. The cmake test and the build work with libfftw3f_threads though.

[heckflosse]

I'm processing NEF files from a D500 (file size about 24 MB). Do you think that 3s for opening a file in the editor are as expected?

Are you using SETM or METM? If you use METM opening a new editor also takes a while.
Time to open a file in editor also depends on the profile you use. If you use for example 'Auto-matched...' the file will be decoded twice.

[rfranke]

SETM for the reported times, i.e. clicked on thumbnails in the Filmstrip. The Auto-Matched Tone Curve was the killer feature that convinced me to start with RawTherapee :-)

[stdbull]

@rfranke since you are building and running with Mac OS Catalina, with respect to #5552 how does the RT UI behave for you, eg when moving sliders, moving curves, etc?

[rfranke]

Generally fine. The released binaries for version 5.7 and my own build behave alike. If I try to move a slider very fast, then I notice a lag of some milliseconds while the editor updates the photo. Only scrolling with gestures on the track pad or with an Apple mouse is too sensitive -- must use the scrollbars instead.
My only real problem is that RT doesn't exploit the resolution of the monitors when viewing photos, see #5591.

[stdbull]

Interesting. If you drag a curve (eg RGB), does the point on the curve
follow the mouse closely and smoothly, or does it lag? Are you using a
separate (non Apple) mouse when working with a MacBook or other Apple
machine rather than the trackpad (or magic mouse)? From your description it
sounds like the issue I've been observing could be purely due to the use of
the trackpad (I've not used a discrete and non apple mouse). Goes off to
try and find one...

On Wed, 1 Jan 2020, 20:42 rfranke, [e-mail] wrote:

Generally fine. The released binaries for version 5.7 and my own build
behave alike. If I try to move a slider very fast, then I notice a lag of
some milliseconds while the editor updates the photo. Only scrolling with
gestures on the track pad or with an Apple mouse is too sensitive -- must
use the scrollbars instead.
My only real problem is that RT doesn't exploit the resolution of the
monitors when viewing photos, see #5591
https://github.com/Beep6581/RawTherapee/issues/5591.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/Beep6581/RawTherapee/issues/5590?email_source=notifications&email_token=ACM6NAKF2LYYKYTOGGJ2SIDQ3T527A5CNFSM4KBTPUZ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEH5MGJI#issuecomment-570082085,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ACM6NAOWKC4GTHRMMV4KK2DQ3T527ANCNFSM4KBTPUZQ
.

[rfranke]

Yes, when dragging points on curves or sliders, they follow the mouse closely and smoothly. Just scrollbars sometimes seem to hang -- but this I'm used to from the GNOME desktop as well.
I used the built-in touchpad and an Apple Magic Mouse so far. Interestingly: scrolling works nicely too (like under Linux) with a regular PC mouse and scroll wheel. The Magic Mouse works better for other programs, like Web browsers. It seems to have a higher amplification of slow scroll actions.

[Benitoite]

So I noticed in dev the default package manager has been changed to Homebrew from MacPorts.
This PR retains that new default but also allows for /opt

5598

[rfranke]

I had started open minded without previous knowledge of either Homebrew or MacPorts. After having tried both, the latter convinces me more (regular installer instead of copy/paste of hacky command line, use of sudo, unique names, less PATH juggling). Appreciating your pull request!

[Benitoite]

Thanks @rfranke for having a look. Same experience here. I’d used apt before on Linux and find macports to be as simple. Homebrew is a slightly different paradigm but is supported by some autobuilders. PS I updated the pr after discovering a typo that occurred.

[Beep6581]

Hey, what is the status?

[rfranke]

The build process works nicely with MacPorts -- much simpler imho. Everything works, including also omp. RT runs from the build directory. Unfortunately make macosx_bundle generates a bundle with .dylib conficts. #5598 does not help either for me.

[Beep6581]

Does our documentation in RawPedia reflect the most recent approach? i.e. can we close this issue?

[Benitoite]

I noticed the script uses 'cp' in places instead of 'ditto' causing version errors. PR will be forthcoming.

On Jan 24, 2020, at 10:41 PM, rfranke [e-mail] wrote:


The build process works nicely with MacPorts -- much simpler imho. Everything works, including also omp. RT runs from the build directory. Unfortunately make macosx_bundle generates a bundle with .dylib conficts. #5598 does not help either for me.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or unsubscribe.

[rfranke]

@Benitoite: sounds promising!
@Beep6581: the documentation should still be updated.

• better separate the descriptions of Homebrew and MacPorts build from each other -- I think one should not mix both as I did first, but decide for one
• sudo port install: update clang-3.9 to clang-9.0 and add libomp
• libiconv patch does not appear needed anymore (maybe shift to an appendix)
• outdated versions in section: Distribution Bundle Framework Library Versions in 5.5-dev
• maybe add cmake argument -DCMAKE_CXX_FLAGS="-Wno-deprecated-register"

[Benitoite]

Did some updates of https://rawpedia.rawtherapee.com/MacOS

• Added -Wno-pass-failed -Wno-deprecated-register (thanks to @heckflosse et al. the build is quiet except loop vecorizations that go away with no-pass-fail. The non vectorizing loops still show during the LTO, which is not bad because LTO takes a few minutes).
• Outdated library version section removed outright.
• Updated libiconv patch to 1.16. (note: I still build the lib with it but haven't gone back and looked into the original issue for a test):

cd ~/libiconv* && LIBRARY_PATH="/opt/local/lib" LD=ld CC=clang CXX=clang++ CFLAGS="-arch x86_64 -mmacosx-version-min=10.9 -I/opt/local/include" LDFLAGS="-arch x86_64 -mmacosx-version-min=10.9 -L/opt/local/lib" CXXFLAGS="-arch x86_64 -mmacosx-version-min=10.9" ./configure --prefix=/opt/local --disable-static && LIBRARY_PATH="/opt/local/lib" LD=ld CC=clang CXX=clang++ CFLAGS="-arch x86_64 -mmacosx-version-min=10.9 -I/opt/local/include" LDFLAGS="-arch x86_64 -mmacosx-version-min=10.9 -L/opt/local/lib" CXXFLAGS="-arch x86_64 -mmacosx-version-min=10.9" ./configure --prefix=/opt/local --disable-static 'CFLAGS=-arch x86_64 -mmacosx-version-min=10.9 ' 'LDFLAGS=-arch x86_64 -mmacosx-version-min=10.9 ' CXXFLAGS="-arch x86_64 -mmacosx-version-min=10.9" && curl https://raw.githubusercontent.com/Beep6581/RawTherapee/dev/tools/osx/libiconv_1.16_rt.patch -o libiconv_1.16_rt.patch && patch -p1 < libiconv_1.16_rt.patch && LIBRARY_PATH="/opt/local/lib" LD=ld CC=clang CXX=clang++ CFLAGS="-arch x86_64 -mmacosx-version-min=10.9 -I/opt/local/include" LDFLAGS="-arch x86_64 -mmacosx-version-min=10.9 -L/opt/local/lib" CXXFLAGS="-arch x86_64 -mmacosx-version-min=10.9" make -j8 && sudo make install && libtool --finish /opt/local/lib

• Added warning box at beginning of MacPorts section.

[rfranke]

Just tested the updated documentation. Much better now!

Following things might be changed:

• section MacPorts: add libomp to sudo port install
• section MacPorts: sudo port install ld64 +ld64_xcode with two l
• section libiconv patch: don't need this anymore
• section Compile RawTherapee: clang-mp-9.0 and clang++-mp-9.0 to get the compiler installed with MacPorts, as opposed to /usr/bin/clang
• section Compile RawTherapee: add -DLOCAL_PREFIX:PATH="/opt" \ to cmake arguments (needed by macosx_bundle.sh)
• section Compile RawTherapee: can't -DWITH_LTO="OFF" \ be deleted?

RawTherapee compiles and runs from the build directory.
If I add LOCAL_PREFIX (see above) and place the resulting RawTherapee.app in /Applications, then it almost runs as well. Unfortunately still get:

(rawtherapee-bin:31346): Gtk-WARNING **: 16:26:36.528: Could not load a pixbuf from /org/gtk/libgtk/theme/Adwaita/assets/check-symbolic.svg.
This may indicate that pixbuf loaders or the mime database could not be found.
**
Gtk:ERROR:gtkiconhelper.c:494:ensure_surface_for_gicon: assertion failed (error == NULL): Failed to load /Applications/RawTherapee.app/Contents/Resources/share/icons/Adwaita/16x16/status/image-missing.png: Unrecognized image file format (gdk-pixbuf-error-quark, 3)
Abort trap: 6

The file image-missing.png is there though:

[Benitoite]

Great, @rfranke thanks for testing. The issue is not really with any missing image. It is with the particular gdkpixbuf2 loader module. Perhaps due to compilation options, link options, or install_name timing during the bundling. However I noticed that compiling gdkpixbuf2 with the loaders built-in did make the problem disappear for me.

[Benitoite]

LTO = off was placed into the instructions to let people know they can turn it on if they wish.

[Benitoite]

Also, port clang++-mp-9.0 includes clang-mp-9.0
Updates made.

[Pandagrapher]

@Benitoite
Hello,
I tried to follow the MacOS compile process in RawPedia with MacPorts (MacOS v10.15.4) but I coped with several issues:

• To install dependencies: clang++-mp-9.0 package doesn't exist but clang-9.0 does. cmake should have also been added to work.
• To compile: cMake option -DCMAKE_OSX_DEPLOYMENT_TARGET=10.15 is missing otherwise compilation instantly failed.

Otherwise, I still get the following error while linking (it's newlocallabbranch):

[ 80%] Building CXX object rtgui/CMakeFiles/rth.dir/pathutils.cc.o
Undefined symbols for architecture x86_64:
  "_fftwf_cleanup_threads", referenced from:
      rtengine::ImProcFunctions::retinex_pde(float*, float*, int, int, float, float, float*, int, int, int) in librtengine.a(iplocallab.cc.o)
      rtengine::ImProcFunctions::exposure_pde(float*, float*, float*, int, int, float, float) in librtengine.a(iplocallab.cc.o)
      rtengine::ImProcFunctions::fftw_convol_blur(float*, float*, int, int, float, int, int) in librtengine.a(iplocallab.cc.o)
  "_fftwf_init_threads", referenced from:
      rtengine::ImProcFunctions::retinex_pde(float*, float*, int, int, float, float, float*, int, int, int) in librtengine.a(iplocallab.cc.o)
      rtengine::ImProcFunctions::exposure_pde(float*, float*, float*, int, int, float, float) in librtengine.a(iplocallab.cc.o)
      rtengine::ImProcFunctions::fftw_convol_blur(float*, float*, int, int, float, int, int) in librtengine.a(iplocallab.cc.o)
  "_fftwf_plan_with_nthreads", referenced from:
      rtengine::ImProcFunctions::retinex_pde(float*, float*, int, int, float, float, float*, int, int, int) in librtengine.a(iplocallab.cc.o)
      rtengine::ImProcFunctions::exposure_pde(float*, float*, float*, int, int, float, float) in librtengine.a(iplocallab.cc.o)
      rtengine::ImProcFunctions::fftw_convol_blur(float*, float*, int, int, float, int, int) in librtengine.a(iplocallab.cc.o)
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [rtgui/rawtherapee-cli] Error 1
make[1]: *** [rtgui/CMakeFiles/rth-cli.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....

Do you know what I have missed?
Thanks in advance for your answer,
Pierre

[Pandagrapher]

@Benitoite
Finally found the culprit, with MacPorts, fftw-3-single package shall be installed with variant +openmp to contain the missing functions.

Pierre

[Benitoite]

Many thanks @Pandagrapher for that.
Added +openmp to the variants.conf section in the wiki.

[rfranke]

Encountering the problem with missing -DCMAKE_OSX_DEPLOYMENT_TARGET=10.15 as well. Somehow the automatic detection does not work anymore.