Darktable: wrong side of picture with "denoise profile" module with CPU path

Created on 4 Nov 2019 · 49Comments · Source: darktable-org/darktable

Issue with right side of processed picture, when "lens correction" module is activated, and only when processed with CPU.

Update 1: This is same issue, as described in #3136, #3249.
Update 2: Quick workaround - increase host memory limit value from default (1500) to some bigger value (3000 and more).

Examples of buggy pictures (processed with CPU):

Examples of same pictures (processed with OpenCL):

You may find full set of these samples (raws + processed jpegs) here:
https://github.com/d2r2/darktable_bug_lens_correction_2019.11.05

Darktable compiled from master branch:

[ddyakov@archlinuxdd2 bin]$ ./darktable --version
this is darktable 3.0.0rc0+10~g742bfcc55-dirty
copyright (c) 2009-2019 johannes hanika
[email protected]

compile options:
  bit depth is 64 bit
  normal build
  SSE2 optimized codepath enabled
  OpenMP support enabled
  OpenCL support enabled
  Lua support enabled, API version 5.0.2
  Colord support enabled
  gPhoto2 support enabled
  GraphicsMagick support enabled
  OpenEXR support enabled

I have a feeling, that issue appeared in 2 weeks or so, because I'm using development Darktable branch for long time, and found this issue in last week only.

I will try to localize commit, which causes this problem in next day.

bugfix

Source

d2r2

Most helpful comment

ok. It works. No margin.

blitzgneisserin on 5 Nov 2019

🎉2 👍1

All 49 comments

Related to #3249 and #3136 issues. @rawfiner and @TurboGit : maybe this could help to resolve this margin problem. From what I see that is common on these issues is the use of CPU (without OpenCL) and a module related to image transformation size/geometry (lens correction or rotate and crop modules) and, apparently denoise profile (@d2r2 : I don't download your images but could you confirm if denoise profile is activate or not on these images ?).

Nilvus on 4 Nov 2019

I have downloaded the first one and I have no lens recognized. In fact the lens module does nothing and in the image information I don't have any lens detected ?

TurboGit on 4 Nov 2019

@Nilvus, good point! You right, when I switch off denoise profile (color+ligthness), then it produce good images with CPU also.

d2r2 on 4 Nov 2019

Good, and what graphic card do you have?

TurboGit on 4 Nov 2019

Also, can you check that setting host memory limit to something bigger than default 1500 (say 4000) fixes the issue?

TurboGit on 4 Nov 2019

@TurboGit, I have an explanation for this: https://github.com/lensfun/lensfun project for some reason doesn't commit support fix for relatively new Lumix LX100M2 camera (while Lumix LX100 is supported).
You can made this fix by adding this xml part to "~/.local/share/lensfun/updates/version_1/compact-panasonic.xml":

<lensdatabase version="1">
    <camera>
        <maker>Panasonic</maker>
        <model>DC-LX100M2</model>
        <!-- Same lens as LX100 -->
        <mount>panasonicDMCLX100</mount>
        <cropfactor>2.21</cropfactor>
    </camera>
</lensdatabase>

d2r2 on 4 Nov 2019

@TurboGit, I'm using my HP ProBook 450 with discrete video card integrated in Intel Core i7 (no separate graphics card). So I always have OpenCL option disabled, since it work very slow compared to CPU processing.

d2r2 on 4 Nov 2019

@TurboGit, host memory limit increase works! Even setting to 2000 eliminate the issue with these specific samples (even without jpeg downsampling in size). Thanks for workaround!

d2r2 on 4 Nov 2019

I am also seeing this issue (and #3249 / #3136), however, I do have a dedicated graphics card which supports OpenCL. I can post my system information later today and am happy to lend a hand testing.

noahwc on 4 Nov 2019

@noahwc, so you have this issue despite dedicated graphics card? Do you have OpenCL option (checkbox) activated in "core options" settings?

d2r2 on 5 Nov 2019

Hi @d2r2 that's correct. With OpenCL enabled in core options and scheduling set to "very fast GPU" I still get the issue:

IMG_6525_01

I also see the issue with OpenCL disabled. I have not changed any other options within darktable related to resource usage.

System info:
darktable version: 3.0.0rc0-git6.3540d428d (from OBS)
OS: Ubuntu 19.10
CPU: Intel Pentium G4560
RAM: 8GB
Graphics card: GTX 1050
Graphics drivers: Nvidia 435

noahwc on 5 Nov 2019

Also thanks @TurboGit for the workaround. It worked for me with OpenCL enabled.

noahwc on 5 Nov 2019

👍1

Off topic.
@TurboGit, thanks to lensfun committer, who finally merge the Lumix LX100M2 lens support, so I hope soon everywhere it will be enough to run lensfun-update-data to get support of LX100M2 camera without extra manipulation.

d2r2 on 5 Nov 2019

@TurboGit, I have an explanation for this: https://github.com/lensfun/lensfun project for some reason doesn't commit support fix for relatively new Lumix LX100M2 camera (while Lumix LX100 is supported).

The explanation for this is that I don't think anyone really thinks of themselves as the maintainer for that project right now... I really prefer to only commit profiles that I can test myself, but that one seems well tested, so it's in. :-)

junkyardsparkle on 5 Nov 2019

👍1

At this stage I'm pretty sure it is a tiling issue.

But I really don't understand why I don't reproduce with CPU pixelpath even when using 500 as host memory limit. So there is some more probably. Indeed all people having the issue have an i915 graphic card, but I don't see how this could change anything.

All idea welcome !

TurboGit on 5 Nov 2019

Is this the same bug that I reported?

blitzgneisserin on 5 Nov 2019

@blitzgneisserin, based on the reasons (modules activated) that cause this bug, I guess this is same issue.

d2r2 on 5 Nov 2019

@TurboGit, here more detailed information on graphic card where I have this bug:

[ddyakov@archlinuxdd2 ~]$ sudo lspci -v -s 00:02.0
00:02.0 VGA compatible controller: Intel Corporation Skylake GT2 [HD Graphics 520] (rev 07) (prog-if 00 [VGA controller])
    DeviceName: Onboard IGD
    Subsystem: Hewlett-Packard Company Skylake GT2 [HD Graphics 520]
    Flags: bus master, fast devsel, latency 0, IRQ 129
    Memory at f0000000 (64-bit, non-prefetchable) [size=16M]
    Memory at e0000000 (64-bit, prefetchable) [size=256M]
    I/O ports at 4000 [size=64]
    [virtual] Expansion ROM at 000c0000 [disabled] [size=128K]
    Capabilities: [40] Vendor Specific Information: Len=0c <?>
    Capabilities: [70] Express Root Complex Integrated Endpoint, MSI 00
    Capabilities: [ac] MSI: Enable+ Count=1/1 Maskable- 64bit-
    Capabilities: [d0] Power Management version 2
    Capabilities: [100] Process Address Space ID (PASID)
    Capabilities: [200] Address Translation Service (ATS)
    Capabilities: [300] Page Request Interface (PRI)
    Kernel driver in use: i915
    Kernel modules: i915

d2r2 on 5 Nov 2019

Just to be clear, I think it is a tiling issue with "denoise profile". When tile are in use (because host memory limit is too low) the output is wrong. When the host memory limit is high enough and make it possible to avoid using tiles so the process is done as a single chunk the output is correct.

Again, I don't see how the graphic card could be in the loop.

Maybe the processor or the compiler version ?

Please all people with the problem can you list:

the processor
the compiler used GCC ? clang ?
the compiler version

Thanks.

TurboGit on 5 Nov 2019

@TurboGit maybe darktable uses tiling also with the GPU depending on the GPU memory?
Anyway, I'll try to make a PR today, as there is a mistake in the tiling code of wavelets. We will see if it solves the issue (hopefully!) :-)

rawfiner on 5 Nov 2019

@d2r2 I have the exact same graphics card. How did you make opencl work with darktable? Here darktable simply discards my video card.

Sorry for this off-topic comment but I do not see how it is possible to write someone a private message.

blitzgneisserin on 5 Nov 2019

@rawfiner : no there is no tiling with OpenCL.

TurboGit on 5 Nov 2019

@rawfiner : no there is no tiling with OpenCL.

What makes you say that?

parafin on 5 Nov 2019

@parafin : reading the code... but double checking I see that I missed process_tiling_cl(). so there is also tiling on GPU indeed. that's really a part I don't know well, sorry for the noise.

TurboGit on 5 Nov 2019

This may be a separate issue entirely to this issue, however, the following issue appeared at the same time.

When enabling lens correction on some images certain portions of the image become black. Note I do have OpenCL enabled. The state of profiled denoise (enabled or disabled) does not seem to affect this issue.

The output is also affected but not in the same way as it is displayed in darktable (notice the thin black border).

Again not sure if this is a separate issue or just another symptom of the issue we're having with profiled denoise.

noahwc on 5 Nov 2019

@blitzgneisserin, I don't remember exactly, but to make OpenCL available on notebook with Intel graphic card, I had installed somehow Intel opencl drivers. Since I have Arch linux, probably it was package intel-compute-runtime...
In real life I never activate OpenCL option, because it significantly freeze notebook CPU, so I use it only for test purpose.

d2r2 on 5 Nov 2019

@d2r2 thanks

blitzgneisserin on 5 Nov 2019

@TurboGit Why was this report marked as bug and not mine?

blitzgneisserin on 5 Nov 2019

@blitzgneisserin : serious? both are the same right, better to discuss the issue in a single thread. but I should have marked yours as duplicate.

TurboGit on 5 Nov 2019

@TurboGit Well I found this bug first! I mean I do not really care, it was just a question. I am just glad that it will be fixed.

blitzgneisserin on 5 Nov 2019

@TurboGit, here my list:
1) CPU:

[ddyakov@archlinuxdd2 .config]$ cat /proc/cpuinfo
processor   : 0
vendor_id   : GenuineIntel
cpu family  : 6
model       : 78
model name  : Intel(R) Core(TM) i7-6500U CPU @ 2.50GHz
stepping    : 3
microcode   : 0xcc
cpu MHz     : 500.009
cache size  : 4096 KB
physical id : 0
siblings    : 4
core id     : 0
cpu cores   : 2
apicid      : 0
initial apicid  : 0
fpu     : yes
fpu_exception   : yes
cpuid level : 22
wp      : yes
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp md_clear flush_l1d
bugs        : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs
bogomips    : 5202.65
clflush size    : 64
cache_alignment : 64
address sizes   : 39 bits physical, 48 bits virtual
power management:

processor   : 1
vendor_id   : GenuineIntel
cpu family  : 6
model       : 78
model name  : Intel(R) Core(TM) i7-6500U CPU @ 2.50GHz
stepping    : 3
microcode   : 0xcc
cpu MHz     : 500.005
cache size  : 4096 KB
physical id : 0
siblings    : 4
core id     : 1
cpu cores   : 2
apicid      : 2
initial apicid  : 2
fpu     : yes
fpu_exception   : yes
cpuid level : 22
wp      : yes
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp md_clear flush_l1d
bugs        : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs
bogomips    : 5202.65
clflush size    : 64
cache_alignment : 64
address sizes   : 39 bits physical, 48 bits virtual
power management:

processor   : 2
vendor_id   : GenuineIntel
cpu family  : 6
model       : 78
model name  : Intel(R) Core(TM) i7-6500U CPU @ 2.50GHz
stepping    : 3
microcode   : 0xcc
cpu MHz     : 500.005
cache size  : 4096 KB
physical id : 0
siblings    : 4
core id     : 0
cpu cores   : 2
apicid      : 1
initial apicid  : 1
fpu     : yes
fpu_exception   : yes
cpuid level : 22
wp      : yes
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp md_clear flush_l1d
bugs        : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs
bogomips    : 5202.65
clflush size    : 64
cache_alignment : 64
address sizes   : 39 bits physical, 48 bits virtual
power management:

processor   : 3
vendor_id   : GenuineIntel
cpu family  : 6
model       : 78
model name  : Intel(R) Core(TM) i7-6500U CPU @ 2.50GHz
stepping    : 3
microcode   : 0xcc
cpu MHz     : 500.004
cache size  : 4096 KB
physical id : 0
siblings    : 4
core id     : 1
cpu cores   : 2
apicid      : 3
initial apicid  : 3
fpu     : yes
fpu_exception   : yes
cpuid level : 22
wp      : yes
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp md_clear flush_l1d
bugs        : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs
bogomips    : 5202.65
clflush size    : 64
cache_alignment : 64
address sizes   : 39 bits physical, 48 bits virtual
power management:

2) gcc:

[ddyakov@archlinuxdd2 .config]$ gcc --version
gcc (GCC) 9.2.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

3) clang:

[ddyakov@archlinuxdd2 .config]$ clang --version
clang version 9.0.0 (tags/RELEASE_900/final)
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

4) Graphic card info:

[ddyakov@archlinuxdd2 darktable-master]$ sudo lspci -v -s 00:02.0
00:02.0 VGA compatible controller: Intel Corporation Skylake GT2 [HD Graphics 520] (rev 07) (prog-if 00 [VGA controller])
    DeviceName: Onboard IGD
    Subsystem: Hewlett-Packard Company Skylake GT2 [HD Graphics 520]
    Flags: bus master, fast devsel, latency 0, IRQ 129
    Memory at f0000000 (64-bit, non-prefetchable) [size=16M]
    Memory at e0000000 (64-bit, prefetchable) [size=256M]
    I/O ports at 4000 [size=64]
    [virtual] Expansion ROM at 000c0000 [disabled] [size=128K]
    Capabilities: [40] Vendor Specific Information: Len=0c <?>
    Capabilities: [70] Express Root Complex Integrated Endpoint, MSI 00
    Capabilities: [ac] MSI: Enable+ Count=1/1 Maskable- 64bit-
    Capabilities: [d0] Power Management version 2
    Capabilities: [100] Process Address Space ID (PASID)
    Capabilities: [200] Address Translation Service (ATS)
    Capabilities: [300] Page Request Interface (PRI)
    Kernel driver in use: i915
    Kernel modules: i915

5) Extra info from compilation output:

-- The CXX compiler identification is GNU 9.2.0
-- The C compiler identification is GNU 9.2.0
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Found OpenMP_C: -fopenmp (found suitable version "4.5", minimum required is "4.0") 
-- Found OpenMP_CXX: -fopenmp (found suitable version "4.5", minimum required is "4.0") 
-- Found OpenMP: TRUE (found suitable version "4.5", minimum required is "4.0")  
-- Performing Test HAVE_OMP_FIRSTPRIVATE_WITH_CONST
-- Performing Test HAVE_OMP_FIRSTPRIVATE_WITH_CONST - Success
-- Performing Test IS_SUPPORTED_PLATFORM
-- Performing Test IS_SUPPORTED_PLATFORM - Success
-- Is the target platform supported: 1
-- Check if the system is big endian
-- Searching 16 bit integer
-- Looking for sys/types.h
-- Looking for sys/types.h - found
-- Looking for stdint.h
-- Looking for stdint.h - found
-- Looking for stddef.h
-- Looking for stddef.h - found
-- Check size of unsigned short
-- Check size of unsigned short - done
-- Using unsigned short
-- Check if the system is big endian - little endian
-- Found little endian system. Good.
-- Performing Test _MSSE2
-- Performing Test _MSSE2 - Success
-- Building SSE2-optimized codepaths: ON
-- Performing Test C_COMPILER_UNDERSTANDS_-Wall
-- Performing Test C_COMPILER_UNDERSTANDS_-Wall - Success
-- Performing Test CXX_COMPILER_UNDERSTANDS_-Wall
-- Performing Test CXX_COMPILER_UNDERSTANDS_-Wall - Success
-- Performing Test C_COMPILER_UNDERSTANDS_-Wformat
-- Performing Test C_COMPILER_UNDERSTANDS_-Wformat - Success
-- Performing Test CXX_COMPILER_UNDERSTANDS_-Wformat
-- Performing Test CXX_COMPILER_UNDERSTANDS_-Wformat - Success
-- Performing Test C_COMPILER_UNDERSTANDS_-Wformat-security
-- Performing Test C_COMPILER_UNDERSTANDS_-Wformat-security - Success
-- Performing Test CXX_COMPILER_UNDERSTANDS_-Wformat-security
-- Performing Test CXX_COMPILER_UNDERSTANDS_-Wformat-security - Success
-- Performing Test C_COMPILER_UNDERSTANDS_-Wshadow
-- Performing Test C_COMPILER_UNDERSTANDS_-Wshadow - Success
-- Performing Test CXX_COMPILER_UNDERSTANDS_-Wshadow
-- Performing Test CXX_COMPILER_UNDERSTANDS_-Wshadow - Success
-- Performing Test C_COMPILER_UNDERSTANDS_-Wtype-limits
-- Performing Test C_COMPILER_UNDERSTANDS_-Wtype-limits - Success
-- Performing Test CXX_COMPILER_UNDERSTANDS_-Wtype-limits
-- Performing Test CXX_COMPILER_UNDERSTANDS_-Wtype-limits - Success
-- Performing Test C_COMPILER_UNDERSTANDS_-Wvla
-- Performing Test C_COMPILER_UNDERSTANDS_-Wvla - Success
-- Performing Test CXX_COMPILER_UNDERSTANDS_-Wvla
-- Performing Test CXX_COMPILER_UNDERSTANDS_-Wvla - Success
-- Performing Test C_COMPILER_UNDERSTANDS_-Wold-style-declaration
-- Performing Test C_COMPILER_UNDERSTANDS_-Wold-style-declaration - Success
-- Performing Test CXX_COMPILER_UNDERSTANDS_-Wold-style-declaration
-- Performing Test CXX_COMPILER_UNDERSTANDS_-Wold-style-declaration - Failed
-- Performing Test C_COMPILER_UNDERSTANDS_-Wthread-safety
-- Performing Test C_COMPILER_UNDERSTANDS_-Wthread-safety - Failed
-- Performing Test CXX_COMPILER_UNDERSTANDS_-Wthread-safety
-- Performing Test CXX_COMPILER_UNDERSTANDS_-Wthread-safety - Failed
-- Performing Test C_COMPILER_UNDERSTANDS_-Wno-unknown-pragmas
-- Performing Test C_COMPILER_UNDERSTANDS_-Wno-unknown-pragmas - Success
-- Performing Test CXX_COMPILER_UNDERSTANDS_-Wno-unknown-pragmas
-- Performing Test CXX_COMPILER_UNDERSTANDS_-Wno-unknown-pragmas - Success
-- Performing Test C_COMPILER_UNDERSTANDS_-Wno-error=varargs
-- Performing Test C_COMPILER_UNDERSTANDS_-Wno-error=varargs - Success
-- Performing Test CXX_COMPILER_UNDERSTANDS_-Wno-error=varargs
-- Performing Test CXX_COMPILER_UNDERSTANDS_-Wno-error=varargs - Success
-- Performing Test C_COMPILER_UNDERSTANDS_-Wno-format-truncation
-- Performing Test C_COMPILER_UNDERSTANDS_-Wno-format-truncation - Success
-- Performing Test CXX_COMPILER_UNDERSTANDS_-Wno-format-truncation
-- Performing Test CXX_COMPILER_UNDERSTANDS_-Wno-format-truncation - Success
-- Performing Test C_COMPILER_UNDERSTANDS_-Wno-error=address-of-packed-member
-- Performing Test C_COMPILER_UNDERSTANDS_-Wno-error=address-of-packed-member - Success
-- Performing Test CXX_COMPILER_UNDERSTANDS_-Wno-error=address-of-packed-member
-- Performing Test CXX_COMPILER_UNDERSTANDS_-Wno-error=address-of-packed-member - Success
-- Performing Test C_COMPILER_UNDERSTANDS_-Wframe-larger-than=32768
-- Performing Test C_COMPILER_UNDERSTANDS_-Wframe-larger-than=32768 - Success
-- Performing Test CXX_COMPILER_UNDERSTANDS_-Wframe-larger-than=32768
-- Performing Test CXX_COMPILER_UNDERSTANDS_-Wframe-larger-than=32768 - Success
-- Performing Test C_COMPILER_UNDERSTANDS_-Wstack-usage=32768
-- Performing Test C_COMPILER_UNDERSTANDS_-Wstack-usage=32768 - Success
-- Performing Test CXX_COMPILER_UNDERSTANDS_-Wstack-usage=32768
-- Performing Test CXX_COMPILER_UNDERSTANDS_-Wstack-usage=32768 - Success
-- Performing Test C_COMPILER_UNDERSTANDS_-Wlarger-than=524288
-- Performing Test C_COMPILER_UNDERSTANDS_-Wlarger-than=524288 - Success
-- Performing Test CXX_COMPILER_UNDERSTANDS_-Wlarger-than=524288
-- Performing Test CXX_COMPILER_UNDERSTANDS_-Wlarger-than=524288 - Success
-- Looking for external programs
-- Found perl
-- Found intltool-merge
-- Found desktop-file-validate
-- Found LLVM 9.0.0
-- Found clang compiler - /usr/bin/clang-9
-- Found clang opencl-c.h header in /usr/lib/clang/9.0.0/include
-- Will be able to test-compile OpenCL programs. Nice.
-- Found jsonschema
-- Found xsltproc
-- Found xmllint
-- All external programs found
-- Found Gettext: /usr/bin/msgmerge (found version "0.20.1") 
-- Found msgfmt to convert .po file. Translation enabled
-- Performing Test RAWSPEED_CXX_DEBUG_INFO_FOR_AUTO_IS_SUPPORTED
-- Performing Test RAWSPEED_CXX_DEBUG_INFO_FOR_AUTO_IS_SUPPORTED - Success
-- Checking for -march=native support
-- Performing Test MARCHNATIVE
-- Performing Test MARCHNATIVE - Success
-- Checking for -march=native support - works
-- Checking for -std=c++14 support
-- Performing Test COMPILER_SUPPORTS_CXX14
-- Performing Test COMPILER_SUPPORTS_CXX14 - Success
-- Checking for -std=c++14 support - works
-- Performing Test RAWSPEED_HAVE_CXX_FLAG_WALL
-- Performing Test RAWSPEED_HAVE_CXX_FLAG_WALL - Success
-- Performing Test RAWSPEED_HAVE_CXX_FLAG_WEXTRA
-- Performing Test RAWSPEED_HAVE_CXX_FLAG_WEXTRA - Success
-- Performing Test RAWSPEED_HAVE_CXX_FLAG_WCAST_QUAL
-- Performing Test RAWSPEED_HAVE_CXX_FLAG_WCAST_QUAL - Success
-- Performing Test RAWSPEED_HAVE_CXX_FLAG_WEXTRA_SEMI
-- Performing Test RAWSPEED_HAVE_CXX_FLAG_WEXTRA_SEMI - Success
-- Performing Test RAWSPEED_HAVE_CXX_FLAG_WFORMAT_2
-- Performing Test RAWSPEED_HAVE_CXX_FLAG_WFORMAT_2 - Success
-- Performing Test RAWSPEED_HAVE_CXX_FLAG_WMISSING_PROTOTYPES
-- Performing Test RAWSPEED_HAVE_CXX_FLAG_WMISSING_PROTOTYPES - Failed
-- Performing Test RAWSPEED_HAVE_CXX_FLAG_WOLD_STYLE_CASTS
-- Performing Test RAWSPEED_HAVE_CXX_FLAG_WOLD_STYLE_CASTS - Failed
-- Performing Test RAWSPEED_HAVE_CXX_FLAG_WPOINTER_ARITH
-- Performing Test RAWSPEED_HAVE_CXX_FLAG_WPOINTER_ARITH - Success
-- Performing Test RAWSPEED_HAVE_CXX_FLAG_WSTRICT_PROTOTYPES
-- Performing Test RAWSPEED_HAVE_CXX_FLAG_WSTRICT_PROTOTYPES - Failed
-- Performing Test RAWSPEED_HAVE_CXX_FLAG_WTRADITIONAL
-- Performing Test RAWSPEED_HAVE_CXX_FLAG_WTRADITIONAL - Failed
-- Performing Test RAWSPEED_HAVE_CXX_FLAG_WVLA
-- Performing Test RAWSPEED_HAVE_CXX_FLAG_WVLA - Success
-- Performing Test RAWSPEED_HAVE_CXX_FLAG_WMISSING_FORMAT_ATTRIBUTE
-- Performing Test RAWSPEED_HAVE_CXX_FLAG_WMISSING_FORMAT_ATTRIBUTE - Success
-- Performing Test RAWSPEED_HAVE_CXX_FLAG_WSUGGEST_ATTRIBUTE_FORMAT
-- Performing Test RAWSPEED_HAVE_CXX_FLAG_WSUGGEST_ATTRIBUTE_FORMAT - Success
-- Performing Test RAWSPEED_HAVE_CXX_FLAG_WNO_UNUSED_PARAMETER
-- Performing Test RAWSPEED_HAVE_CXX_FLAG_WNO_UNUSED_PARAMETER - Success
-- Performing Test RAWSPEED_HAVE_CXX_FLAG_WSTACK_USAGE_4096
-- Performing Test RAWSPEED_HAVE_CXX_FLAG_WSTACK_USAGE_4096 - Success
-- Performing Test RAWSPEED_HAVE_CXX_FLAG_WFRAME_LARGER_THAN_4096
-- Performing Test RAWSPEED_HAVE_CXX_FLAG_WFRAME_LARGER_THAN_4096 - Success
-- Performing Test RAWSPEED_HAVE_CXX_FLAG_WLARGER_THAN_32768
-- Performing Test RAWSPEED_HAVE_CXX_FLAG_WLARGER_THAN_32768 - Success
-- Looking for OpenMP
-- Found OpenMP_C: -fopenmp (found suitable version "4.5", minimum required is "4.0") 
-- Found OpenMP_CXX: -fopenmp (found suitable version "4.5", minimum required is "4.0") 
-- Looking for OpenMP - found (system)
-- Looking for pugixml
-- Found Pugixml 1.10
-- Looking for pugixml - found (system)
-- Looking for JPEG
-- Found JPEG: /usr/lib/libjpeg.so (found version "80") 
-- Looking for JPEG - found
-- Looking for jpeg_mem_src
-- Looking for jpeg_mem_src - found
-- Looking for ZLIB
-- Found ZLIB: /usr/lib/libz.so (found version "1.2.11") 
-- Looking for C++ include zlib.h
-- Looking for C++ include zlib.h - found
-- Check size of uLongf
-- Check size of uLongf - done
-- Looking for Z_OK
-- Looking for Z_OK - found
-- Checking prototype uncompress for HAVE_ZLIB_UNCOMPRESS_PROTOTYPE - True
-- Checking prototype zError for HAVE_ZLIB_ZERROR_PROTOTYPE - True
-- Looking for uncompress
-- Looking for uncompress - found
-- Looking for zError
-- Looking for zError - found
-- Looking for ZLIB - found (system)
-- Looking for malloc
-- Looking for malloc - found
-- Looking for free
-- Looking for free - found
-- Looking for posix_memalign
-- Looking for posix_memalign - found
-- Performing Test HAVE_CXX_THREAD_LOCAL
-- Performing Test HAVE_CXX_THREAD_LOCAL - Success
-- Found XMLLINT: /usr/bin/xmllint  
-- The following features have been enabled:

 * OpenMP-based threading, used for parallelization of the library
 * XML reading, used for loading of data/cameras.xml
 * Lossy JPEG decoding, used for DNG Lossy JPEG compression decoding
 * ZLIB decoding, used for DNG Deflate compression decoding

-- The following OPTIONAL packages have been found:

 * LLVM (required version >= 9)
 * Gettext
 * XMLLINT, command line XML tool, <http://xmlsoft.org/>
   Used for validation of data/cameras.xml

-- The following RECOMMENDED packages have been found:

 * JPEG, free library for handling the JPEG image data format, implements a JPEG codec
   Used for decoding DNG Lossy JPEG compression
 * ZLIB, software library used for data compression
   Used for decoding DNG Deflate compression

-- The following REQUIRED packages have been found:

 * OpenMP (required version >= 4.0), Open Multi-Processing, <https://www.openmp.org/>
   Used for parallelization of the library
 * Pugixml (required version >= 1.2), Light-weight, simple and fast XML parser, <http://pugixml.org/>
   Used for loading of data/cameras.xml

-- Performing Test COMPILER_SUPPORTS_C99
-- Performing Test COMPILER_SUPPORTS_C99 - Success
-- Performing Test COMPILER_SUPPORTS_C11
-- Performing Test COMPILER_SUPPORTS_C11 - Success
-- Checking for -std=c++14 support
-- Checking for -std=c++14 support - works
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE  
-- Found Glib 2.62.2
-- Found PkgConfig: /usr/local/bin/pkg-config (found version "0.29.2") 
-- Checking for module 'gtk+-3.0'
--   Found gtk+-3.0, version 3.24.12
-- Found GTK3: /usr/include/gtk-3.0;/usr/include/at-spi2-atk/2.0;/usr/include/at-spi-2.0;/usr/include/dbus-1.0;/usr/lib/dbus-1.0/include;/usr/include/gtk-3.0;/usr/include/gio-unix-2.0;/usr/include/cairo;/usr/include/pango-1.0;/usr/include/fribidi;/usr/include/atk-1.0;/usr/include/cairo;/usr/include/pixman-1;/usr/include/freetype2;/usr/include/harfbuzz;/usr/include/libpng16;/usr/include/gdk-pixbuf-2.0;/usr/include/libmount;/usr/include/blkid;/usr/lib/libffi-3.2.1/include;/usr/include/glib-2.0;/usr/lib/glib-2.0/include (Required is at least version "3.22") 
-- Found LibXml2: /usr/lib/libxml2.so (found suitable version "2.9.9", minimum required is "2.6") 
-- Checking for one of the modules 'libsoup-2.4;libsoup2'
-- Checking for one of the modules 'libsoup-2.2;libsoup2'
-- Found GPHOTO2: /usr/lib/libgphoto2.so  
-- Found OPENEXR: /usr/lib/libIlmImf.so;/usr/lib/libImath.so;/usr/lib/libIex.so;/usr/lib/libIlmThread.so  
-- Found WEBP: /usr/lib/libwebp.so (Required is at least version "0.3.0") 
-- Found LENSFUN: /usr/lib/liblensfun.so  
-- Found GIO 
-- Found GThread: /usr/lib/libgthread-2.0.so  
-- Found GModule: /usr/lib/libgmodule-2.0.so  
-- Found PANGOCAIRO: /usr/lib/libpangocairo-1.0.so  
-- Found RSVG2: /usr/lib/librsvg-2.so  
-- Found LibXml2: /usr/lib/libxml2.so (found version "2.9.9") 
-- Found Sqlite3 3.30.1
-- Found CURL: /usr/lib/libcurl.so (found version "7.66.0")  
-- Found PNG: /usr/lib/libpng.so (found version "1.6.37") 
-- Found TIFF: /usr/lib/libtiff.so (found version "4.0.10") 
-- Found LCMS2: /usr/lib/liblcms2.so  
-- Found JsonGlib 
-- Found EXIV2: /usr/lib/libexiv2.so  
-- Found FLICKCURL: /usr/lib/libflickcurl.so  
-- Found OpenJPEG 
-- Checking for module 'iso-codes'
--   Found iso-codes, version 4.3
-- Found LIBSECRET: /usr/include/libsecret-1;/usr/include/libmount;/usr/include/blkid;/usr/lib/libffi-3.2.1/include;/usr/include/glib-2.0;/usr/lib/glib-2.0/include  
-- Found GraphicsMagick 
-- Performing Test HAVE_BUILTIN_CPU_SUPPORTS
-- Performing Test HAVE_BUILTIN_CPU_SUPPORTS - Success
-- Does the compiler support __builtin_cpu_supports(): 1
-- Performing Test HAVE_TLS
-- Performing Test HAVE_TLS - Success
-- Checking for -march=native support
-- Checking for one of the modules 'lua53;lua5.3;lua-5.3;lua'
-- Lua support: Enabled
-- Found Pugixml 1.10
-- Map mode: enabled
-- Found Cups: /usr/lib/libcups.so (found version "2.3.0") 
-- Print mode: enabled
-- Found CURL: /usr/lib/libcurl.so (found suitable version "7.66.0", minimum required is "7.56")  
-- Found recent CURL version to build piwigo.
-- Found X11: /usr/include   
-- Looking for XOpenDisplay in /usr/lib/libX11.so;/usr/lib/libXext.so
-- Looking for XOpenDisplay in /usr/lib/libX11.so;/usr/lib/libXext.so - found
-- Looking for gethostbyname
-- Looking for gethostbyname - found
-- Looking for connect
-- Looking for connect - found
-- Looking for remove
-- Looking for remove - found
-- Looking for shmat
-- Looking for shmat - found
-- Looking for IceConnectionNumber in ICE
-- Looking for IceConnectionNumber in ICE - found
-- building darktable-cmstest with colord support. nice.
-- 
-- Found Java: /usr/bin/java (found version "1.8.0.232") found components:  Runtime 
-- No saxon XSLT processor and/or no docbook saxon extension library found.
Missing fop
Xslt processor saxon not found. HTML usermanual will be built with xsltproc;
expect usermanual with inconsistent screenshot and image dimensions.
Can NOT build usermanual and Lua API documentation
-- Configuring done
-- Generating done

d2r2 on 5 Nov 2019

@blitzgneisserin, sorry, I didn’t notice that such problem already registered by you. Maybe because my first suspicion fells on module "lens correction". In any case equally important that we both provide information about issue, because this produce good attention to the problem and give more chances that issue will be handled with corresponding priority.

d2r2 on 5 Nov 2019

Please test if you still have the bug with #3294

rawfiner on 5 Nov 2019

@rawfiner, my samples (3 raw files) test results on CPU path:
1) host memory limit=1500. Produced correct pictures.
2) host memory limit=1000. Produced correct pictures (current master branch just crash with stack output).

I can run this test on big bunch of pictures, if you'd like, but it takes few hours.

@blitzgneisserin, @hellbuster, it would be good to test from your side as well, to be sure that fix works not only in my case.

d2r2 on 5 Nov 2019

I already tested, crash at start

blitzgneisserin on 5 Nov 2019

@blitzgneisserin, did you correctly fetch and checkout pull request with commands, before build?:

git fetch origin pull/3294/head:rawfiner-try-fixing-tiling
git checkout rawfiner-try-fixing-tiling

d2r2 on 5 Nov 2019

ok, I'll try again

blitzgneisserin on 5 Nov 2019

I don't know what I am doing wrong:

anna@mx:~
$ git clone https://github.com/rawfiner/darktable.git
Cloning into 'darktable'...
remote: Enumerating objects: 23, done.
remote: Counting objects: 100% (23/23), done.
remote: Compressing objects: 100% (16/16), done.
remote: Total 174304 (delta 11), reused 15 (delta 7), pack-reused 174281
Receiving objects: 100% (174304/174304), 344.78 MiB | 937.00 KiB/s, done.
Resolving deltas: 100% (136227/136227), done.
anna@mx:~
$ cd $HOME/darktable
anna@mx:~/darktable
$ git submodule init
Submodule 'src/external/rawspeed' (https://github.com/darktable-org/rawspeed.git) registered for path 'src/external/rawspeed'
anna@mx:~/darktable
$ git submodule update
Cloning into '/home/anna/darktable/src/external/rawspeed'...
Submodule path 'src/external/rawspeed': checked out '71f0bfd3a486f63d4cae9159f170a00351a2f677'
anna@mx:~/darktable
$ git fetch origin pull/3294/head:rawfiner-try-fixing-tiling
fatal: Couldn't find remote ref pull/3294/head

blitzgneisserin on 5 Nov 2019

Try this:

git checkout -b rawfiner-rawfiner-try-fixing-tiling master
git pull https://github.com/rawfiner/darktable.git rawfiner-try-fixing-tiling

TurboGit on 5 Nov 2019

I don't know. Apparently I'm too stupid for this.

anna@mx:~/darktable
$ git pull https://github.com/rawfiner/darktable.git rawfiner-try-fixing-tiling
error: You have not concluded your merge (MERGE_HEAD exists).
hint: Please, commit your changes before merging.
fatal: Exiting because of unfinished merge.

blitzgneisserin on 5 Nov 2019

Try this before:

git reset --hard

TurboGit on 5 Nov 2019

@blitzgneisserin, start from empty folder, with the sequence, where you are downloading main darktable master branch, then switch to pull request provided by rawfiner.
In this case the command sequence will looks like:

git clone https://github.com/darktable-org/darktable
cd darktable
git submodule init
git submodule update
git fetch origin pull/3294/head:rawfiner-try-fixing-tiling
git checkout rawfiner-try-fixing-tiling

If everything ok, then your

git branch

, should output:

  master
* rawfiner-try-fixing-tiling

You can try it, if previous advice doesn't work...

d2r2 on 5 Nov 2019

ok. It works. No margin.

blitzgneisserin on 5 Nov 2019

🎉2 👍1

Fixed now.

TurboGit on 5 Nov 2019

Fixed now.

TurboGit on 5 Nov 2019

Fixed now.

TurboGit on 5 Nov 2019

Fixed now.

TurboGit on 5 Nov 2019

The output is also affected but not in the same way as it is displayed in darktable (notice the thin black border).

@noahwc The second image (thin border) is what you'll see after applying lens correction if the scaling factor isn't correctly adjust (should normally happen automatically, but there's a slider for it in the module). I have no idea what would cause the large blocks in the preview, though.