Rawtherapee: Continuous integration of Win64 builds

[aferrero2707]

This thread is to discuss the status and improvement for the continuous integration of windows builds.

The current build scripts can be found here. The packages are cross-compiled with mingw-w64, and most of the dependencies are taken from the openSUSE build service. The build environment is based on a custom Docker container that allows to cache most of the dependencies. Currently Travis CI is used as the build server, but in principle any CI service that supports Docker containers could be used.

Experimental packages can be downloaded from here. The download page provides both a portable version of RT (in the form of a .zip archive that can be extracted anywhere), and an installer created with InnoSetup.

The current version has two known problems:

• the windows decorations are not the native ones, and so look a bit "strange"
• there are still some un-necessary files in the packages

If you can, please test the most recent versions and report issues here. The packages will be updated on a daily basis from the main development branches.

Thanks!

[gaaned92]

@TooWaBoo @Hombre57
I tested the RawTherapee_dev-5.4-1133-g5e0cbd4c_WinVista_64.exe from @aferrero2707

I notice that the window decoration is not the same as my builds.
You will find on pixls mail a screenshot. https://pixls-discuss.s3.dualstack.us-east-1.amazonaws.com/original/3X/4/4/449b20e2236f3d7af9b1688d4163d52a0618f638.jpeg
Could you identify and fix the issue? GTK3 3.22.26 is used in automatic builds.

[TooWaBoo]

@gaaned92
The native window has changed with GTK+ 3.22.24 ( https://github.com/Beep6581/RawTherapee/issues/4141 ) and there is no way to change it 'cause it's automatic.

Are you shure it's 3.22.26?

[aferrero2707]

@TooWaBoo
I am using 3.22.26 for my builds... what could be wrong then? Native windows OK with 3.22.24 and not with 3.22.26?

[TooWaBoo]

What's your Gtkmm-version?

[aferrero2707]

I am currently using this version: https://ftp.gnome.org/pub/gnome/sources/gtkmm/3.22/gtkmm-3.22.2.tar.xz

I see there is 3.22.3 now, should I update?

[TooWaBoo]

I would say YES.

[TooWaBoo]

You can update GTK+ to 3.22.30 too. That's what we are using for Windows.

[Beep6581]

I'm on 3.22.30 and experience no (new) problems.

[aferrero2707]

I have prepared new packages compiled with GTKMM 3.22.3 (GTK+ is still at 3.22.26).
I will only be able to test them on Windows 10 in a couple of days, but maybe some of you could already have a look and see if the native windows decorations issue still persists?
Here are the links:

• installer: https://github.com/aferrero2707/rt-win64/releases/download/continuous/RawTherapee_dev-5.4-1269-g1bfe0341_WinVista_64.exe
• portable package: https://github.com/aferrero2707/rt-win64/releases/download/continuous/RawTherapee_dev-5.4-1269-g1bfe0341.zip

[aferrero2707]

I've been researching a bit more about CSD on Windows. It seems that MSYS2 has a custom patch that disables CSD by default on Windows, while the official GTK3 sources have CSD enabled by default, see discussion here.

This might explain the difference between the packages from @gaaned92 and myself (I am not using MSYS2, but the OpenSUSE mingw packages instead).

[gaaned92]

@aferrero2707
sadly, the problem persists.
The discussion you linked identifies the reason. I wonder if there is a way to patch the mingw package?

[aferrero2707]

@gaaned92
In fact I mentioned something wrong... for GTK3 I am compiling the sources, not using the OpenSUSE package (the provided version is too old).

Anyhow, I think that the best solution would be to force GTK_CSD=0 in the RT initialisation, if the environment variable is not set by the user. This would provide a default behaviour that is independent of the GTK3 package being used.

What do you think?

I can try to prepare a patched RT package to see if the idea works in practice...

[gaaned92]

@aferrero2707
perhaps a stupid question : isn't it possible to use the compiled MSYS2 package?
I am in favor of applying the same patch as MSYS2. Before they applied this patch there were complaints of windows users as GTK3 was breaking the snap feature.

[aferrero2707]

@gaaned92
before investigating the use of the MSYS2 package (it's an option I am seriously considering), I have applied a simple patch to the RT sources that mimic the same behaviour: it defaults to GTK_CSD=0 if the variable is not set.

The patch is here, and the new package here or here.

I just tested it on my Win10 box, and the windows decorations are now the native ones for me. Could you check as well?

[gaaned92]

@aferrero2707
Yes that's ok now :thumbsup:
The remaining problems ( unnecessary files, missing icones) don't prevent to use the build.

[aferrero2707]

@gaaned92
Great! At this point I will set up the mechanism for daily builds. For the moment the output packages will be stored in my github repository, waiting for green light from the RT developers to eventually move them together with the AppImages.
Ping @Beep6581 @heckflosse @Hombre57 @agriggio about this...

[aferrero2707]

@gaaned92 @agriggio @Hombre57 @heckflosse
I have just prepared a first cross-compiled win64 build that uses the MSYS2 packages: https://filebin.net/lwj1smg0ad8cw3dh

The package still needs some cleaning for un-necessary stuff, but the main thing at the moment is to check its functionality.

As a reminder, the previous builds were based on mingw64 packages provided by OpenSUSE, but they showed some issues (like missing symbols in Windows 7). The new build instead uses the official MSYS2 packages, and should therefore be equivalent to @gaaned92's packages.

The main difference is that in my case the builds are cross-compiled under Linux, and the whole build environment is based on a Manjaro Docker image that can be run almost everywhere (Travis CI or other CI systems, as well as individual users). In other words, anybody with a running Docker installation should be able to compile the windows version, obtaining a package that is exactly equivalent to the automated builds. The MSYS2 packages are freshly installed each time the build process is run, so they are guaranteed to be up-to-date (excepted for the few packages that are forced to a given version, like jpeg-turbo).

I'd be very much interested in your feedback...

[Hombre57]

@aferrero2707 Thanks for the packages !

I ran it on Win7/64-bits without install (only unzipping, which should work) and started rawtherapee.exe through the Windows Explorer and from an MSYS64 console : RT opens but w/o any language string, i.e. I only see the identifier everywhere.

It seem that the language file has not been loaded at all. When starting RT in verbose mode, I can read this, among lots of messages of course :

Processing file D:\Jean-Christophe\Téléchargements\RawTherapee-dev-5.4-763-g1874259b\.\profiles\Faded\Faded Amber 1.pp3...OK

Fine.

.Could not open camera constants file "D:\Jean-Christophe\Téléchargements\RawTherapee-dev-5.4-763-g1874259b\.\camconst.json": No such file or directory

The base path (D:\Jean-Christophe\Téléchargements\RawTherapee-dev-5.4-763-g1874259b) is exactly the same and it exist (it's the unzipped folder) but still, it can't load camconst.json, that exist too. I guess there must be a similar issue with the language file ?

[Hombre57]

@aferrero2707 @Beep6581 @heckflosse @gaaned92 Wow ! In fact, the culprit is special chars in the base install directory ! If I move the folder to the drive's root (D:\RawTherapee-dev-5.4-763-g1874259b) then it works fine... and my builds react the same ! I never had this test case before, that's for sure something that should be told to the user, if bug unfixed till final release.

@gaaned92 Do you confirm the bug ?

[Hombre57]

@aferrero2707 So I confirm that your version work fine here, but is very slow, probably due to the debug version (rawtherapee.exe alone: 242Mb !). Could you provide release builds (+ eventually a debug exe) ?

[gaaned92]

@Hombre57 bug confirmed also for my builds.

@aferrero2707 version works fine on W10. @heckflosse could you test on W7?

A few icone missing on buttons. could you create in.\share\GTK3 a file settings.ini containing :

[Settings] gtk-button-images=1

[Beep6581]

That's:

[Settings]
gtk-button-images=1

[Thanatomanic]

See here https://github.com/Beep6581/RawTherapee/issues/4784#event-2028538399
@aferrero2707 and @gaaned92: no more need to downgrade the gtk3 and gtkmm3 packages with an update MSYS2 environment!

[aferrero2707]

New test package, with up-to-date GTK3 packages: https://filebin.net/1m0d8j78otky62nv

I had no time yet to test it in detail...

EDIT: @gaaned92 @TooWaBoo @Hombre57 @agriggio now that 5.5 is out, would you have some time to test this new package, and give me some feedback?

Thanks!

[gaaned92]

@aferrero2707
My remarks

• It runs ok on W10 and the issue on W7 is corrected.

• relwithdebinfo build vs release build: on my pc there is unnoticeable difference in loading the exe. perhaps it is different on slow pc.(see @Hombre57 post above)

• useless dlls.

libatomic-1.dll
libjson-glib-1.0-0.dll
libcairo-script-interpreter-2.dll
liblzo2-2.dll
libcharset-1.dll
libminizip-1.dll
libfftw3-3.dll
libmpdec-2.dll
libfftw3l-3.dll
libp11-kit-0.dll
libfreeglut.dll
libpcre16-0.dll
libgailutil-3-0.dll
libpcre32-0.dll
libgettextlib-0-19-8-1.dll
libpcrecpp-0.dll
libgettextpo-0.dll
libpcreposix-0.dll
libgettextsrc-0-19-8-1.dll
libpython3.7m.dll
libglibmm_generate_extra_defs-2.4-1.dll
libquadmath-0.dll
libgmp-10.dll
libreadline7.dll
libgmpxx-4.dll
libsqlite3-0.dll
libgthread-2.0-0.dll
libssl-1_1-x64.dll
libgtkreftestprivate-0.dll
libssp-0.dll
libharfbuzz-gobject-0.dll
libtasn1-6.dll
libharfbuzz-icu-0.dll
libtermcap-0.dll
libharfbuzz-subset-0.dll
libturbojpeg-0.dll
libhistory7.dll
tcl86.dll
libjasper-4.dll
tk86.dll

• useless dirs in share/

gettext-0.19.8
gir-1.0
graphite2
installed-tests
libthai
p11-kit
pki
readline
tabset
terminfo
thumbnailers
vala

• in share/icons/adwaita you need only:

scalable\actions
scalable\devices
scalable\mimetypes
scalable\places
scalable\status
index.theme
cursors\plus.cur
cursors\sb_h_double_arrow.cur
cursors\sb_left_arrow.cur
cursors\sb_right_arrow.cur
cursors\sb_v_double_arrow.cur

• I have also share\glib-2.0\schemas containing gschemas.compiled

• missing settings.ini (see 3 posts above)

• I see that you also have a Licenses dir: do we have to ship it with builds?

and kudos for this implementation

[aferrero2707]

@gaaned92
Thanks for checking!

relwithdebinfo build vs release build: on my pc there is unnoticeable difference in loading the exe. perhaps it is different on slow pc.(see @Hombre57 post above)

This, and the fact that the incompatibility with Win7 is solved, are the most important aspects for the moment.
At this point I will clean the contents of the package and compile in Release mode to reduce the size of the executable. There is also something wrong with the way I am generating the settings.ini, which should be present in the package.

More news after the Christmas break...

[aferrero2707]

@gaaned92
I have uploaded the new build scripts to GitHub, and created a new package that (hopefully) follows all your recommendations. The build scripts are here (see those with MSYS2 in their names) and the new package here.