Lektor: Use 'watchgod' module instead of 'watchdog'

[Endogen]

I'm just starting to use Lektor but i noticed that it uses the watchdog module and one of the first comments that i saw at the quickstart tutorial was:

When I run lektor server I get OSError: inotify watch limit reached

I ran into that myself and thus i know what it is and that it's related to watchdog but a lot of beginners probably don't know. Instead of using separate approaches for each OS to watch for file-changes, i think that using the watchgod module could be a better and simpler solution.

Maybe it's a good idea to at least check the module out and see if it fits better than the overly complex watchdog implementation.

[yagebu]

Yes, that error sounds like it is related to file watching. Does it crash the server or does it just show this error message and work as expected?

I understand that the "one size fits all" approach of watchgod sounds appealing, but I don't think that it would be a good choice for Lektor. Watching for file-changes is inherently platform-specific (by using inotify, fsevents, etc., which watchdog does) or inefficient (by using polling, which watchgod and which is also offered by watchdog). Also, after having looked rather dead for a while, the watchdog repo has gotten more active recently and there's frequent releases, so I think that's also not an argument against watchdog anymore.

[Endogen]

That's fine with me and understandable. But that error definitely lead to broken functionality for me so for personal projects I switched to watchgod.

[ls0h]

Hi! I have same problem with Lektor. It is just empty project, created with lektor quickstart.

lektor server
. . .
  File "/usr/lib/python3/dist-packages/watchdog/observers/inotify_c.py", line 415, in _raise_error
    raise OSError("inotify instance limit reached")
OSError: inotify instance limit reached

I tried to increase /proc/sys/fs/inotify/max_user_watches to 1048576. But nothing changed.
Tested with Lektor 3.1.3 from Ubuntu 20.10 repo, and 3.2.0 from pip. Is here any workaround available? The server has crashed and totally unusable.

[runfalk]

I'm a bit curious. How big is your project? Would you mind running these commands and sharing their output?

lsof | grep inotify | wc -l
find content/ -type f | wc -l
find content/ -type f -name "*.lr" | wc -l
find content/ -type d | wc -l
du -hs content/

I want to get an idea of for which kinds of projects this happens.

As a side note, Lektor 3.1.3 is very old (though we haven't done any changes to the watchdog since then that I know of).

[ls0h]

My project was just created with lektor quickstart and there is nothing more in it.

$ lsof | grep inotify | wc -l
lsof: WARNING: can't stat() tracefs file system /sys/kernel/debug/tracing
      Output information may be incomplete.
7742

$ find content/ -type f | wc -l
3



md5-038d2069f22706920750e7edc8b6bc9b



$ find content/ -type f -name "*.lr" | wc -l
3



md5-038d2069f22706920750e7edc8b6bc9b



$ find content/ -type d | wc -l
3



md5-038d2069f22706920750e7edc8b6bc9b



$ du -hs content/
24K content/

Lektor 3.1.3 is very old

Same bug with 3.2.0.

[dairiki]

@ls0h Something else running on your system is causing the inotify limits to be exceeded. Note that there are three sysfs limits which can be adjusted. Maybe try adjusting fs.inotify.max_user_instances?

Here are a few references:

• https://transang.me/enospc-inotify-in-ubuntu/
• https://man7.org/linux/man-pages/man7/inotify.7.html
• https://unix.stackexchange.com/questions/13751/kernel-inotify-watch-limit-reached

[dairiki]

Hitting the inotify limits is a system problem, not a Lektor problem — sometimes even tail -f won't run in this case — but it would be good if Lektor would degrade gracefully when this happens.

The watchdog distribution provides several different types of "Observers" that can be used to watch for file changes. On Linux, the default observer is InotifyObserver. There are other observers which are used on different OSes, but watchdog also provides a PollingObserver, which — while not as efficient as the other observers — should work anywhere.

From looking at the Lektor code, it appears that Lektor does not catch exceptions when creating the default (InotifyObserver on Linux) observer, so if that fails the server just craps out.

We should catch errors when creating the default observer, report them with a big warning, and then fall back to using PollingObserver.