As a test for the vertical placement of the ThumbnailPanel I opened way to many windows to see what happens. The ThumbnailPanel expands out of the visibleFrame. There will be some solution required for this use case.
Clarification for AltTab
The UI is fine as long as the thumbnails fit the screen:

Then if more windows are open, the UI is clipped at the bottom:

Note that you can still select the off-screen thumbnails.
Hyperswitch behaves the same

Note that even though the window has its bottom corners rounded, more thumbnails are indeed clipped away at the bottom. You can still blindly select then with a few tabs, as for AltTab.
macOS command-tab shrinks icons


macOS Mission Control shrinks thumbnails


Windows 10 alt-tab clips but let's user cycle to off-screen thumbnails


WindowsSwitcher clips but let's user cycle to off-screen thumbnails

Hard clip: AltTab, HyperSwitch
I can see 3 ways to implement a solution:
Right now I prefer downscaling as it's the easiest to implement, and I think it's the best UX
@lwouis Nice evaluation/comparison of the solutions in other applications. Fully agree here, the downscale has the best UX since it does not require any additional non-native navigation ideas to scroll or paginate and one can see at once his pile of windows.
馃憤 for downscaling!
Most helpful comment
Clarification for AltTab
The UI is fine as long as the thumbnails fit the screen:
Then if more windows are open, the UI is clipped at the bottom:
Note that you can still select the off-screen thumbnails.
Hyperswitch behaves the same
Note that even though the window has its bottom corners rounded, more thumbnails are indeed clipped away at the bottom. You can still blindly select then with a few tabs, as for AltTab.
macOS command-tab shrinks icons
macOS Mission Control shrinks thumbnails
Windows 10 alt-tab clips but let's user cycle to off-screen thumbnails
WindowsSwitcher clips but let's user cycle to off-screen thumbnails