This happens since verison 0.4.1 of Alacritty, rearranging takes like 3 seconds after closing window.

Sounds like something weird going on with Alacritty and is not something that can be fixed from my side. I'd recommend opening an issue there.
@xseman Hello, Same story here. Do you figure out this issue finally? Thanks.
@koekeishiya Dear developer, I opened an issue at Alacritty (https://github.com/alacritty/alacritty/issues/3333) there.
@yanzhang0219 I checked yabai logs, but no luck there.. then I rolled back to previous version of alacritty (0.4.0), because it was unusable with yabai.
@xseman Okay. I am now running the latest version of Alacritty and this issue is very dominant. I enabled the window border in yabai, and then when I closed an Alacritty window, that window was destroyed, but its border was still there for around 3-4 seconds and other windows were not rearranged.
I will switch to kitty terminal emulator to see whether it has a better performance with yabai. On the example configuration file, I saw kitty is the default terminal emulator, so probably it is recommended.
If you're really keen on solving this and have some time to spare, you can always run a git-bisect on Alacritty. You already have known versions for good and bad, so finding the responsible commit in Alacritty shouldn't be too hard.
Update:
I deleted the Alacritty that is installed via homebrew. Instead, I clone the source code and build it manually. Then this window rearranging issue seems disappeared (at lease getting better and relived).