Tdesktop: UI freezes on fdatasync

[rnz]

Steps to reproduce

About ui freezes

1. run strace with ignore some operations

sudo strace -e 'trace=!recvmsg,poll,writev,stat,futex' -p $(pgrep Telegram) 2>&1 | egrep -v 'read\(5|write\(5'

1. type some text in message input box
2. catch some freezes on fdatasync:

getpid()                                = 69543
getpid()                                = 69543
access("/home/rnz/.TelegramDesktop/tdata/D877F783D5D3EF8C/E66391AC0BDD1C96s", F_OK) = 0
access("/home/rnz/.TelegramDesktop/tdata/D877F783D5D3EF8C/E66391AC0BDD1C96s", W_OK) = 0
lstat("/home/rnz/.TelegramDesktop/tdata/D877F783D5D3EF8C/E66391AC0BDD1C96s", {st_mode=S_IFREG|0664, st_size=364, ...}) = 0
openat(AT_FDCWD, "/home/rnz/.TelegramDesktop/tdata/D877F783D5D3EF8C", O_RDWR|O_CLOEXEC|O_TMPFILE, 0600) = 41
access("/home/rnz/.TelegramDesktop/tdata/D877F783D5D3EF8C/E66391AC0BDD1C96s", R_OK) = 0
access("/home/rnz/.TelegramDesktop/tdata/D877F783D5D3EF8C/E66391AC0BDD1C96s", W_OK) = 0
access("/home/rnz/.TelegramDesktop/tdata/D877F783D5D3EF8C/E66391AC0BDD1C96s", X_OK) = -1 EACCES (Permission denied)
fchmod(41, 0664)                        = 0
fstat(41, {st_mode=S_IFREG|0664, st_size=0, ...}) = 0
write(41, "TDF$o\210\36\0\0\0\0\360_\230r\7\27@K\v\356\0t\221\240\262\36\334\203-\227\326"..., 268) = 268
lseek(41, 0, SEEK_SET)                  = 0
fdatasync(41)                           = 0
linkat(AT_FDCWD, "/proc/self/fd/41", AT_FDCWD, "/home/rnz/.TelegramDesktop/tdata/D877F783D5D3EF8C/E66391AC0BDD1C96s", AT_SYMLINK_FOLLOW) = -1 EEXIST (File exists)
linkat(AT_FDCWD, "/proc/self/fd/41", AT_FDCWD, "/home/rnz/.TelegramDesktop/tdata/D877F783D5D3EF8C/E66391AC0BDD1C96s.cvzUPK", AT_SYMLINK_FOLLOW) = 0
close(41)                               = 0
rename("/home/rnz/.TelegramDesktop/tdata/D877F783D5D3EF8C/E66391AC0BDD1C96s.cvzUPK", "/home/rnz/.TelegramDesktop/tdata/D877F783D5D3EF8C/E66391AC0BDD1C96s") = 0
unlink("/home/rnz/.TelegramDesktop/tdata/D877F783D5D3EF8C/E66391AC0BDD1C960") = -1 ENOENT (No such file or directory)
unlink("/home/rnz/.TelegramDesktop/tdata/D877F783D5D3EF8C/E66391AC0BDD1C961") = -1 ENOENT (No such file or directory)

Expected behaviour

No UI freezes on background io operations

Actual behaviour

Freezes.

Configuration

Operating system:
$ lsb_release -a
LSB Version: core-11.1.0ubuntu2-noarch:printing-11.1.0ubuntu2-noarch:security-11.1.0ubuntu2-noarch
Distributor ID: Ubuntu
Description: Ubuntu 20.04 LTS
Release: 20.04
Codename: focal

Version of Telegram Desktop:
2.1.7

Installation source (Linux Only) - the official website / GitHub releases / flatpak / snap / distribution package:
GitHub releases

Used theme:
Night

Additional info:
My /home placed on zfs pool (two hdd in mirror with zlog on nvme ssd) and zfs before version 0.8 didn't support O_DIRECT and operation fdatasync didn't wait disk io. But now zfs support O_DIRECT and operations such as fdatasync now have write time penalties. And for example now when zfs run scrub Telegram is almost completly unusable by freezes because too often call sync operations and not exec this operations in background and have impact on the user interface.

I think fdatasync doesn't need for Telegram - linux fs drivers, vfs and also zfs write stack already have own tunable strategy for sync. But if Telegram is need fsyncdata - that operation must be run in background without impact UI

Also Telegram have too many operations stat("/etc/localtime", ...) on mouse hover action on groups:

...
stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1535, ...}) = 0
stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1535, ...}) = 0
stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1535, ...}) = 0
...

I think this is overhead too

[ilya-fedin]

Also Telegram have too many operations stat("/etc/localtime", ...) on mouse hover action on groups:

Looks like Qt parses this file on each time request and this can't be solved in tdesktop:
https://github.com/qt/qtbase/blob/dev/src/corelib/time/qtimezoneprivate_tz.cpp#L1124

[ilya-fedin]

1. type some text in message input box
2. catch some freezes on fdatasync:

Probably it's due to recent switching to QSaveFile to avoid logout on power outage, and it is called when saving drafts :thinking:
https://github.com/telegramdesktop/tdesktop/commit/2ad48f18f2c4f32f7299e05ad30b74d7e920f71b

[rnz]

Also Telegram have too many operations stat("/etc/localtime", ...) on mouse hover action on groups:

Looks like Qt parses this file on each time request and this can't be solved in tdesktop:
https://github.com/qt/qtbase/blob/dev/src/corelib/time/qtimezoneprivate_tz.cpp#L1124

Why tdesktop need a time request when hovering over the mouse cursor?

[ilya-fedin]

Why tdesktop need a time request when hovering over the mouse cursor?

:thinking: probably, for tooltip or to update the time on the item

[rnz]

@ilya-fedin I don't saw any actions by hovering (no hints, no time updates) in tdesktop

And I think this is wrong(or old style) call: if (open(save, 's')) {
May be need use save.open() with https://doc.qt.io/qt-5/qiodevice.html#OpenModeFlag-enum ?

[ilya-fedin]

And I think this is wrong(or old style) call: if (open(save, 's')) {

No, it's FileWriteDescriptor::open

[ilya-fedin]

I don't saw any actions by hovering (no hints, no time updates) in tdesktop

You can't see any difference since it's the same post probably

[cypa]

my disk io is slow and freezes are crazy long while I type new message ...
"QSaveFile ... and it is called when saving drafts" - is it synchronous disk io on typing new message? crazy! из пушки по воробьям! how do I disable drafts at all?

[rnz]

This is so stupid bug - when running some io operations on filesystem (such as scrubbing on zfs) - Telegram is complete unusable.

[cypa]

friend, put dir like this
.local/share/TelegramDesktop/tdata/D877F783D5D3EF8C
to tmpfs into RAM
that's a makeshift but it works, no lags

[rnz]

friend, put dir like this
.local/share/TelegramDesktop/tdata/D877F783D5D3EF8C
to tmpfs into RAM
that's a makeshift but it works, no lags

Thanks but this workaround is ugly.

I think behaviour - when changes in input box is put to disk - must be configurable (off, custom period) and of course next need to be fix ui bug with freezes.

[GaurangTandon]

@cypa Thanks, I tried this trick, it did entirely get rid of the freezing. Awesome! However, now my telegram desktop resets every time I restart my pc. Which means I have to login again, enter otp, and enter 2fa password every time, and it resets all my settings too :( This is expected since I loaded Telegram Desktop into RAM, but still :(

[jonathonf]

Another workaround is to use eatmydata/libeatmydata, e.g. eatmydata telegram-desktop or LD_PRELOAD=/usr/lib/libeatmydata.so telegram-desktop

It "is designed to transparently disable fsync and friends" - the lack of write safety won't be a concern if you're already putting this data in volatile storage.

I've updated my desktop shortcuts already... :grin:

[rnz]

Another workaround is to use eatmydata/libeatmydata, e.g. eatmydata telegram-desktop or LD_PRELOAD=/usr/lib/libeatmydata.so telegram-desktop

It "is designed to transparently disable fsync and friends" - the lack of write safety won't be a concern if you're already putting this data in volatile storage.

I've updated my desktop shortcuts already...

This workaroud unfortunately not working...

[jonathonf]

Not working in what way, @rnz ?

You can't get the preload to work, or it doesn't fix the issue completely?

[rnz]

Not working in what way, @rnz ?
You can't get the preload to work, or it doesn't fix the issue completely?

It doesn't fix the issue - freezes on input still present when telegram save text from input field

[Dimensional]

This is rather annoying. Telegram sometimes flat out crashes when it hangs due to this, though that hasn't happened for a few weeks. I disabled Spellcheck and it still hangs when saving the draft. Is it trying to write to an actual file or save the draft to a cache in memory? Since I'm running on HDD, the freezing would be very prominent with the seek time.

[rnz]

@ilya-fedin hi man! Is there any progress with fix this bug?

[ilya-fedin]

I don't know, ask @john-preston

[rnz]

@john-preston same question?

[john-preston]

@rnz I have a patch that moves all fdatasync-s to the background thread, I think I'll add it to the next beta release.