Filament: Headless SwapChain crashes on macOS Catalina (with OpenGL backend)

[yongtang]

Describe the bug

I was experimenting with headless SwapChain on macOS (Catalina 10.15.1 + Xcode 11.2.1), and noticed that OpenGL backend crashes with:

zsh: illegal hardware instruction 

Notice if I switch to Metal backend (see demo.cc) then this issue is resolved, but there will be another issue of :

-[MTLTextureDescriptorInternal validateWithDevice:]:1249:
 failed assertion `MTLTextureDescriptor has invalid pixelFormat (255).'
zsh: abort      bazel-bin/demo

To Reproduce

I have created a small repo to reproduce the issue:
https://github.com/yongtang/demo

The demo.cc above is based on the code in filament's test: filament/test/filament_rendering_test.cpp

1) Download and install Bazel 1.2.0 on macOS Catalina.
2) Run the following to build the binary
sh bazel build -s --verbose_failures //...
3) Run the following to trigger the crash:
sh % bazel-bin/demo Filament library loaded. FEngine (64 bits) created at 0x10aee2000 (threading is enabled) FEngine resolved backend: OpenGL zsh: illegal hardware instruction bazel-bin/demo

4) Switch to Metal backend (see demo.cc and change the line in Engine::create()), a different issue:
```
% bazel build -s --verbose_failures //...
% bazel-bin/demo

-[MTLTextureDescriptorInternal validateWithDevice:]:1249:
failed assertion `MTLTextureDescriptor has invalid pixelFormat (255).'
zsh: abort bazel-bin/demo
```

I tried to debug further on OpenGL issue, it points to [NSOpenGLContext setView:].

Not very familiar with this area so have not been able to find out the reason for the crash yet.

% lldb bazel-bin/demo
(lldb) target create "bazel-bin/demo"
Current executable set to 'bazel-bin/demo' (x86_64).
(lldb) run
Process 45050 launched: '/Users/ytang/tmp/demo/bazel-bin/demo' (x86_64)
Filament library loaded.
FEngine (64 bits) created at 0x101800000 (threading is enabled)
FEngine resolved backend: OpenGL
Process 45050 stopped
* thread #2, name = 'FEngine::loop', stop reason = EXC_BAD_INSTRUCTION (code=EXC_I386_INVOP, subcode=0x0)
    frame #0: 0x00007fff2af66cbc AppKit`-[NSOpenGLContext setView:] + 229
AppKit`-[NSOpenGLContext setView:]:
->  0x7fff2af66cbc <+229>: ud2    
    0x7fff2af66cbe <+231>: movq   %rax, %r14
    0x7fff2af66cc1 <+234>: jmp    0x7fff2af66cd6            ; <+255>
    0x7fff2af66cc3 <+236>: movq   %rax, %rdi
Target 0: (demo) stopped.
(lldb) 

Desktop (please complete the following information):

• OS: macOS Catalina (Xcode latest)
• GPU: n/a
• Backend: OpenGL

Additional context

I did a search, wondering if the issue described in https://github.com/powervr-graphics/Native_SDK/issues/46 might be similar to OpenGL backend's illegal hardware instruction issue?

[prideout]

Possibly related to #1839

[yongtang]

The `MTLTextureDescriptor has invalid pixelFormat (255).' might be related to:
https://github.com/cocos2d/cocos2d-x/issues/19765

Looks like depth24Stencil8PixelFormatSupported is needed to check if the format is supported or not. Haven't' been able to find out where the format is getting triggered though.

[bejado]

@yongtang can you give more info on the system you're running this on? Seems you're right, according to Metal Feature Set Tables that format might not be universally supported:

Query the depth24Stencil8PixelFormatSupported property of MTLDevice to determine support.

Edit: I'm able to repro the crash now if I force use of the integrated Intel GPU.

[yongtang]

@bejado Thanks for the help and quick fix! 👍 ❤️

[bejado]

👍 sure thing. The illegal hardware instruction crash should be fixed by #1959.