hledger-ui --watch not working on GNU/Linux

[simonmichael]

As reported on reddit, hledger-ui --watch is not updating as it should with latest hledger-ui 1.10 on GNU/Linux (seen on Arch and Ubuntu). It works for me on macos. Status on windows, and of hledger-ui 1.9, is unknown.

[lestephane]

It is working for me with hledger-ui 1.15, hledger-ui 1.15.2 on Ubuntu 16.04, and 244 input files, on a gocryptfs mount where filesystem watching (inotify) does not even work in Intellij. So I'm surprised.

HOWEVER, hledger-ui quickly OOMs after steadily increasing its memory usage over a period of a few minutes. I have 8 gigs of RAM, and before it died the process was beyond 4G RSS.

When I don't use --watch it hardly takes 11M RSS.

[simonmichael]

Thanks for the report @lestephane ! I usually see CPU usage creeping up over time, but not this. It would be great to fix this, would you be able to help narrow it down ? Eg compare it on a normal filesystem, with fewer files, etc. (244 files is cool, how do you have that many ?)

[lestephane]

I have many files because for certain accounts I use one journal per day (.timeclock files, pricing for forex pricing pairs). And I use hledger-flow which a) generates quite a few includes b) encourages the 1 1 file / account / month paradigm. So if I have, say, 30 accounts, I'll end up with 30 more files each month. I'll try to isolate the memory behaviour, and file a separate issue if i uncover something.

A small manual test confirms that watching does not work for

• manual modification of one single top-level journal
• manual modification of one inner journal included in one top-level journal
• manual moving away and back of one inner journal included in one top-level journal

So I'm still stumped.

Are you familiar with the linux OS mechanism used by haskell to spot these file changes, maybe I can trace filesystem events, because --watch _does_ work when I run on my many files. But the filesystem events may be different than the average user. The average user expects --watch to see manual edits, while I have many journal files being regenerated entirely by the hledger-flow import command.

[simonmichael]

Thanks for the info. Two quick ideas:

• hledger-ui --watch uses http://hackage.haskell.org/package/fsnotify - check their issue tracker ?
• when troubleshooting, run hledger-ui --watch --debug and watch for "fsnotify" events on the console

[simonmichael]

--watch does work when I run on my many files.

I’m confused.. it works some times ?

[lestephane]

--watch does work when I run on my many files.
I’m confused.. it works some times ?

yup. I'll try with --debug and report the second i get the chance

[lestephane]

@simonmichael I did a hledger-flow import test just now on a subset of my journals with --debug, and sure enough, the heldger-ui display is updated correctly to reflect the new changes, with the following entries in the debug log:

fsnotify using polling ?: False
fsnotify:   "Modified \"scrubbed-2019-05-ALL.journal\" 2020-01-05 10:10:36.251574089 UTC False"
fsnotify:   "Modified \"scrubbed-2019-07-ALL.journal\" 2020-01-05 10:10:36.267817201 UTC False"
fsnotify:   "Modified \"scrubbed-2019-07-ALL.journal\" 2020-01-05 10:10:36.27652122 UTC False"
fsnotify:   "Modified \"scrubbed-2019-05-ALL.journal\" 2020-01-05 10:10:36.310400645 UTC False"
fsnotify:   "Modified \"scrubbed-2019-02-ALL.journal\" 2020-01-05 10:10:36.460791757 UTC False"
fsnotify:   "Modified \"scrubbed-2019-06-ALL.journal\" 2020-01-05 10:10:36.460831429 UTC False"
fsnotify:   "Modified \"scrubbed-2019-02-ALL.journal\" 2020-01-05 10:10:36.460885219 UTC False"
fsnotify:   "Modified \"scrubbed-2019-11-ALL.journal\" 2020-01-05 10:10:36.475032724 UTC False"
fsnotify:   "Modified \"scrubbed-2019-09-ALL.journal\" 2020-01-05 10:10:36.570672565 UTC False"
fsnotify:   "Modified \"scrubbed-2019-04-ALL.journal\" 2020-01-05 10:10:36.599616423 UTC False"
fsnotify:   "Modified \"scrubbed-2019-10-ALL.journal\" 2020-01-05 10:10:36.606001212 UTC False"
fsnotify:   "Modified \"scrubbed-2019-03-ALL.journal\" 2020-01-05 10:10:36.613915173 UTC False"
fsnotify:   "Modified \"scrubbed-2019-03-ALL.journal\" 2020-01-05 10:10:36.61924464 UTC False"
fsnotify:   "Modified \"scrubbed-2019-08-ALL.journal\" 2020-01-05 10:10:36.672623841 UTC False"
fsnotify:   "Modified \"scrubbed-2019-01-ALL.journal\" 2020-01-05 10:10:36.684106843 UTC False"
fsnotify:   "Modified \"scrubbed-2019-01-ALL.journal\" 2020-01-05 10:10:36.689217989 UTC False"

If I then revert one of my change it works as well, and the log says:

fsnotify:   "Modified \"scrubbed-2019-09-ALL.journal\" 2020-01-05 10:45:24.190807616 UTC False"

I'm still on a gocryptfs mount, which is a FUSE filesystem which seems not to (or to only partially support) fsnotify. I'm generally confused about fsnotify vs inotify, and reading this makes my head spin, so I don't think I can add much to this, just that it does work for me. Not much help I know.

[lestephane]

Could this max_user_watches parameter explain why it works for me? It was added by an app I'm using that syncs my google drive:

$ cat /etc/sysctl.conf
...
# added by Insync
fs.inotify.max_user_watches=1048576

Could anyone else try it and see if it helps with --watch?

(Note: I also had to increase fs.file-max but only because I have more journal files than the average user and was hitting the limit there as well)