Watchman: Error: Watchman error: too many pending cache jobs

[ACoolmanBigHealth]

Problem

When I create my react-native bundle - the watchman crawl initially fails with "too many pending cache jobs", which seems to slow down the bundling process:

$ react-native start --reset-cache

Loading dependency graph...jest-haste-map: Watchman crawl failed. Retrying once with node crawler.

Usually this happens when watchman isn't running. Create an empty .watchmanconfig file in your project's root folder or initialize a git or hg repository in your project.

Error: Watchman error: too many pending cache jobs. Make sure watchman is running for this project. See https://facebook.github.io/watchman/docs/troubleshooting.html

Also, these may be related:

• I can get rid of the error by excluding node_modules from the watch config
• Without the error, and the bundler starts in 20 seconds
• With the error, the craw takes 4 seconds, but the bundler starts in 4 minutes

Background

Pre-bundle scripts

I run these before the bundler, to attempt to get more stability:

watchman watch-del-all
watchman shutdown-server
sudo sysctl -w kern.maxfiles=5242880
sudo sysctl -w kern.maxfilesperproc=524288

Watchman config file

I set my .watchconfig file to be extra permissable:

{
  "ignore_dirs": [],
  "fsevents_latency": 0.5,
  "fsevents_try_resync": true
}

Watchman log

The crawl does complete, within a few seconds according to the full log

$ tail -f /usr/local/var/run/watchman/coolman-state/log

22:08:24,193: [client=0x7ff4a9686d98:stm=0x7ff4a9b2de00:pid=0] send_error_response: too many pending cache jobs
22:08:24,340: [listener] Watchman 4.9.0 <no build info set> starting up on coolman
22:08:24,341: [listener] path /Users/coolman/project/react-native-app is on filesystem type apfs
22:08:24,342: [listener] root /Users/coolman/project/react-native-app using watcher mechanism fsevents (auto was requested)
22:08:24,344: [listener] file limit is 2560 kern.maxfilesperproc=524288
22:08:24,344: [listener] raised file limit to 524288
22:08:24,345: [listener] launchd: "sock" wasn't present in Sockets
22:08:28,805: [io 0x7f87ba000818 /Users/coolman/project/react-native-app] PERF: {"ru_nvcsw": 1460, "ru_nsignals": 0, "ru_msgrcv": 0, "ru_msgsnd": 0, "ru_inblock": 0, "ru_majflt": 0, "ru_nswap": 0, "ru_minflt": 28103, "ru_ixrss": 0, "ru_maxrss": 115109888, "system_time": 2.9349970000000001, "user_time": 1.327971, "elapsed_time": 4.4614079999999996, "pid": 29414, "ru_idrss": 0, "meta": {"root": {"watcher": "fsevents", "ticks": 2, "number": 1, "case_sensitive": false, "recrawl_count": 0, "path": "/Users/coolman/project/react-native-app"}}, "version": "4.9.0", "ru_oublock": 0, "ru_nivcsw": 2171, "start_time": 1564952904.3444469, "description": "full-crawl"}
22:08:28,805: [io 0x7f87ba000818 /Users/coolman/project/react-native-app] crawl complete

[wez]

Sounds like you need to tune the metadata cache size(s) for your workload; that error occurs when watchman is attempting to insert more cache entries than are configured.

You can specify larger values in your .watchmanconfig file:

• content_hash_max_items is related to the content.sha1hex metadata and defaults to 131072. A related option is content_hash_max_warm_per_settle which defaults to 1024 and which causes up to this many items to be assessed on each settle.
• symlink_target_max_items defaults to 32768. This cache is used for readlink in the case that queries ask for the symlink target. This is less likely to be your issue.

I would suggest running find /Users/coolman/project/react-native-app | wc -l to get an upper bound on the number of files currently in your project and use that to size these caches; if you have more files than those defaults then you should raise those cache settings accordingly.

I do not recommend changing the fsevents_latency parameter from the default; can you share more context on what led you to change it?

[maxkomarychev]

I started noticing the same issue.
I ran find . -iname '*.js' | wc -l in my project and got number 160k+ files.
I updated watchman config with this:

{
    "fsevents_latency": 0.01,
    "content_hash_max_items": 999999
}

based on answer in this thread and this answer

and it seems to be workin ok now.
(don't forget to do watchman shutdown-server)

@wez are there any problem that setting

    "content_hash_max_items": 999999

can potentially cause in this situation?

[wez]

The content_hash_max_items setting controls the upper bound on the number of elements in the cache. It doesn't pre-allocate memory, so over-shooting with the value won't hurt, but you do need to have enough memory available to support the value that you select in order to avoid swapping.

The amount of memory required per entry for that cache is approximately length(path) + 32, where path is relative to the watched root.

[ACoolmanBigHealth]

works for me :+1: