Wlroots: RDP backend rfx (RemoteFX) mode crashes when rendering same screen twice

[m01]

Summary

When using the rdp backend and connecting with xfreerdp ... --rfx, wlroots crashes in backend/rdp/output.c on this line:

    RFX_RECT *rfx_rect;
    int nrects;
    pixman_box32_t *rects =
        pixman_region32_rectangles(damage, &nrects);
    rfx_rect = realloc(context->rfx_rects, nrects * sizeof(*rfx_rect));  // << here

Using gdb, it looks like this is because damage contains x=0, y=0, width=0 and height=0, probably because the screen didn't change. This means nrects is also 0 and we request a realloc of size 0.

Repro & logs

terminal 1:

% WLR_RDP_TLS_CERT_PATH=$KEY_DIR/tls.crt WLR_RDP_TLS_KEY_PATH=$KEY_DIR/tls.key WLR_BACKENDS=rdp XDG_RUNTIME_DIR=/tmp sway
...
[12:46:06:011] [2174:2174] [INFO][com.freerdp.core.connection] - Client Security: NLA:1 TLS:1 RDP:0
[12:46:06:011] [2174:2174] [INFO][com.freerdp.core.connection] - Server Security: NLA:0 TLS:1 RDP:1
[12:46:06:011] [2174:2174] [INFO][com.freerdp.core.connection] - Negotiated Security: NLA:0 TLS:1 RDP:0
[12:46:06:198] [2174:2174] [INFO][com.freerdp.core.connection] - Accepted client: ...
[12:46:06:198] [2174:2174] [INFO][com.freerdp.core.connection] - Accepted channels:
[12:46:06:198] [2174:2174] [INFO][com.freerdp.core.connection] -  cliprdr
[12:46:06:198] [2174:2174] [INFO][com.freerdp.core.gcc] - Active rdp encryption level: NONE
[12:46:06:198] [2174:2174] [INFO][com.freerdp.core.gcc] - Selected rdp encryption method: NONE
2019-05-05 12:46:06 - [swaybg/main.c:300] Found config * for output RDP-1 (RDP RDP )
...
2019-05-05 12:46:06 - [backend/rdp/output.c:113] RDP swap buffers failed: could not realloc rects
double free or corruption (fasttop)
(EE) failed to read Wayland events: Broken pipe
2019-05-05 12:46:06 - [sway/common/ipc-client.c:90] Unable to receive IPC response
zsh: abort (core dumped)  WLR_RDP_TLS_CERT_PATH=$KEY_DIR/tls.crt WLR_RDP_TLS_KEY_PATH=$KEY_DIR/tls.key

(note about log: I added an extra line for logging to backend/rdp/output.c, which is why the line number above is off by one. The extra logging doesn't show when running sway because I didn't change the log level)

terminal 2:

% xfreerdp -v 127.0.0.1 --rfx --bpp 32 --size 800x600

Workaround

use --nsc instead of --rfx

xfreerdp -v 127.0.0.1 --nsc --bpp 32 --size 800x600

System info

OS: Arch Linux
wlroots-git: 6ae6b5dbb6c4f33f3c8ba12cd889100376d1edd8
sway-git: 51c077798cf06ebd5e27271fb5e276c181f8a077

[emersion]

I've come to the same conclusion. Thanks for investigating.

Would you be willing to send a fix?

[m01]

If the fix is just to return true if there aren't any rects to send (and thus not call rfx_compose_message), then I can file a PR for that. That seems to fix this crash. I don't understand the RDP protocol or freeRDP well enough to know whether this is the correct fix though.

[emersion]

This sounds like a good solution to me.