Hugo: watch with lots of files dies

[philips]

$ find . -type f | wc -l
    8149

29 pages created
0 tags index created
1 topics index created
in 37 ms
Watching for changes in /Users/philips/trees/ifup-hugo/content
Web Server is available at http://localhost:1313
Press ctrl+c to stop
panic: listen tcp :1313: too many open files

goroutine 27 [running]:
runtime.panic(0x3d2a20, 0x210b68240)
    /usr/local/go/src/pkg/runtime/panic.c:266 +0xb6
github.com/spf13/hugo/commands.serve(0x521)
    /home/parallels/.gopathBuild/src/github.com/spf13/hugo/commands/server.go:69 +0x3fa
created by github.com/spf13/hugo/commands.NewWatcher
    /home/parallels/.gopathBuild/src/github.com/spf13/hugo/commands/hugo.go:194 +0x28f

goroutine 1 [semacquire]:
sync.runtime_Semacquire(0x210ad7cd8)
    /usr/local/go/src/pkg/runtime/sema.goc:199 +0x30
sync.(*WaitGroup).Wait(0x210b92180)
    /usr/local/go/src/pkg/sync/waitgroup.go:127 +0x14b
github.com/spf13/hugo/commands.NewWatcher(0x521, 0x0, 0x0)
    /home/parallels/.gopathBuild/src/github.com/spf13/hugo/commands/hugo.go:197 +0x29f
github.com/spf13/hugo/commands.server(0x8bc520, 0x210aa4e40, 0x0, 0x3)
    /home/parallels/.gopathBuild/src/github.com/spf13/hugo/commands/server.go:53 +0x24a
github.com/spf13/cobra.(*Command).execute(0x8bc520, 0x210a35020, 0x3, 0x3, 0x0, ...)
    /home/parallels/.gopathBuild/src/github.com/spf13/cobra/command.go:273 +0x116
github.com/spf13/cobra.(*Command).findAndExecute(0x8bc440, 0x210a35010, 0x4, 0x4, 0x0, ...)
    /home/parallels/.gopathBuild/src/github.com/spf13/cobra/command.go:259 +0xa8
github.com/spf13/cobra.(*Command).Execute(0x8bc440, 0x0, 0x0)
    /home/parallels/.gopathBuild/src/github.com/spf13/cobra/command.go:308 +0x66e
github.com/spf13/hugo/commands.Execute()
    /home/parallels/.gopathBuild/src/github.com/spf13/hugo/commands/hugo.go:52 +0x2f
main.main()
    /home/parallels/.gopathBuild/src/github.com/spf13/hugo/main.go:21 +0x1a

goroutine 24 [chan receive]:
github.com/howeyc/fsnotify.(*Watcher).purgeEvents(0x210b508f0)
    /home/parallels/.gopathBuild/src/github.com/howeyc/fsnotify/fsnotify.go:21 +0x41
created by github.com/howeyc/fsnotify.NewWatcher
    /home/parallels/.gopathBuild/src/github.com/howeyc/fsnotify/fsnotify_bsd.go:103 +0x2fd

goroutine 23 [syscall]:
syscall.Syscall6(0x16b, 0x2f, 0x0, 0x0, 0x210b01000, ...)
    /usr/local/go/src/pkg/syscall/asm_darwin_amd64.s:41 +0x5
syscall.kevent(0x2f, 0x0, 0x0, 0x210b01000, 0xa, ...)
    /usr/local/go/src/pkg/syscall/zsyscall_darwin_amd64.go:199 +0x85
syscall.Kevent(0x2f, 0x0, 0x0, 0x0, 0x210b01000, ...)
    /usr/local/go/src/pkg/syscall/syscall_bsd.go:431 +0x9d
github.com/howeyc/fsnotify.(*Watcher).readEvents(0x210b508f0)
    /home/parallels/.gopathBuild/src/github.com/howeyc/fsnotify/fsnotify_bsd.go:328 +0x31b
created by github.com/howeyc/fsnotify.NewWatcher
    /home/parallels/.gopathBuild/src/github.com/howeyc/fsnotify/fsnotify_bsd.go:102 +0x2e6

goroutine 25 [select]:
github.com/spf13/hugo/commands.func·006()
    /home/parallels/.gopathBuild/src/github.com/spf13/hugo/commands/hugo.go:172 +0x23c
created by github.com/spf13/hugo/commands.NewWatcher
    /home/parallels/.gopathBuild/src/github.com/spf13/hugo/commands/hugo.go:185 +0x1de

goroutine 26 [finalizer wait]:
runtime.park(0xc600, 0x8c4268, 0x8c0d68)
    /usr/local/go/src/pkg/runtime/proc.c:1342 +0x66
runfinq()
    /usr/local/go/src/pkg/runtime/mgc0.c:2276 +0x84
runtime.goexit()
    /usr/local/go/src/pkg/runtime/proc.c:1394

[philips]

Based on the FAQ comment here it seems nothing can be done: https://github.com/howeyc/fsnotify

Perhaps the best solution it to add a "watch mask"? I have lots of static files that will hardly ever change.

[xaprb]

I think a reasonable workaround is to set ulimit -n.

On Mac OSX in particular, -n has a ridiculously small default on my machine.

$ ulimit -n
256
$ ulimit -n 10240
$ ulimit -n
10240

Then it doesn't crash for me. However, I don't think it should panic, either. Panicking is really bad behavior for a program such as this. It should be printing an error and continuing to try to run, or if that's unrecoverable, exiting with a nonzero code, but not panicking.

[xaprb]

I disagree that nothing can be done. I haven't tried it, but I would be surprised if you couldn't make a sysctl to change ulimit, even if it needed to be done via cgo.

[xaprb]

Yeah, with a cursory search I think this is the right place to do it: http://golang.org/pkg/syscall/#Setrlimit

[spf13]

Thanks for digging into this. Care to make a PR?

Steve Francia
spf13.com
@spf13

On Jan 14, 2014, at 7:53 PM, Baron Schwartz [email protected] wrote:

Yeah, with a cursory search I think this is the right place to do it: http://golang.org/pkg/syscall/#Setrlimit

—
Reply to this email directly or view it on GitHub.

[philips]

Thanks!
On Feb 1, 2014 9:51 AM, "Steve Francia" [email protected] wrote:

Closed #168 https://github.com/spf13/hugo/issues/168 via 3e87d7ahttps://github.com/spf13/hugo/commit/3e87d7a86e5dcd244dd713c8712a26f51f02be87
.

Reply to this email directly or view it on GitHubhttps://github.com/spf13/hugo/issues/168
.

[chibicode]

@spf13 @xaprb I'm still seeing this issue...

$ hugo version                                                                                                                                                                                           
Hugo Static Site Generator v0.14-DEV BuildDate: 2015-02-22T01:09:28-07:00

$ find . -type f | wc -l
10567

$ hugo server --theme=hugo-theme-shiori --watch 
...
Serving pages from /Users/shu/code/HugoBasicExample/public
Web Server is available at http://localhost:1313/
Press Ctrl+C to stop
ERROR: 2015/03/08 Error: listen tcp :1313: too many open files in system

By the way, my theme (--theme=hugo-theme-shiori) is using gulp to handle assets, and as a result, needs pull in a lot of node packages inside its node_modules directory. That's why there's a bloat.

One way to avoid this would be if --watch option had a way to ignore certain directories. Just for comparison, Jekyll has exclude option. I couldn't find anything comparable in Configuring Hugo page.

[bep]

@chibicode Are you REALLY synching node_modules? Why? You don't need an ignore feature to fix your problem. Just use Gulp to build just what you need into /static.

[chibicode]

@bep the problem is that my node_modules currently lives under my theme directory (node_modules contains all of the gulp plugins I'm using)

And while I develop my theme, I use HugoBasicExample repo to test it, like this:

And when I try to run HugoBasicExample above with --watch flag, it'll die b/c of all the files inside node_modules.

Anyway, I'm pretty sure there's a workaround, but I can't find anything simple that meets the following requirements:

1. Lets me use tons of gulp plugins (even though node_modules dir will be .gitignore'ed, each plugin needs to live in the theme's directory during development)
2. Lets me develop the theme while having --watch on.

I'm not saying I'd like this feature implemented. I just wish it existed.

[anthonyfok]

Hi @chibicode,

Just double checking:

1. Are you running Hugo on Mac OS X?
2. Does it work if you were to run ulimit -n 65536 _before_ you run hugo with --watch on?

[anthonyfok]

Hi @chibicode, more questions for you:

1. If you are running OS X, which version?
2. If 10.9 (Maverick) or 10.10 (Yosemite), could you please read this http://unix.stackexchange.com/questions/108174/how-to-persist-ulimit-settings-in-osx-mavericks and see if this is indeed what you observe, i.e. ulimit -n 16384 is the highest you can go?
3. If so, could you please start another Terminal and try

$ ulimit -n 999999 $ ulimit -n

and paste the output here?

1. What does hugo check ulimit show on your system? (I just discovered this command, only available on OS X! See hugo/commands/limit_darwin.go.)

Many thanks!

P.S. Strangely, I cannot reproduce this problem on my Debian GNU/Linux at all, not even after I did this: ulimit -n 512 even when I have 16000+ files in my themes/hugo-theme-shiori/ directory. Does --watch work in different ways on different OSes?

P.P.S. Edited this post to change hugo ulimit to hugo check ulimit

[anthonyfok]

Looking through git log:

1. 2014-02-01: @spf13 fixed this issue with commit 3e87d7a86e5dcd244dd713c8712a26f51f02be87

```
commit 3e87d7a86e5dcd244dd713c8712a26f51f02be87
Author: spf13 [e-mail]
Date: Sat Feb 1 12:51:11 2014 -0500

Automatically increase the process ulimit to maximum available. fixes #168.
```

Tries to set the maxfile to 999999 whenever --watch is invoked.

1. 2014-02-06: @spf13 limits this patch to OS X, because on Ubuntu Linux hugo server -w triggers "Error Setting rLimit operation not permitted", see #194.

```
commit 75c260fa1ca4277cf1ef455cfd897bd315ab92a7
Author: spf13 [e-mail]
Date: Thu Feb 6 00:12:05 2014 -0500

Only change rLimit on OSX (where it is needed). Fixed #194

commit 11ca84f8cb19f01b7977b549e63e0ce0d126054e
```

1. 2014-02-17: @mattn fixed the build on Windows (#196, #199) See 8ebb85f1f70baabefa8c21cc21bffbef2a187129
2. There are some more refactoring, and hugo check ulimit and tweakLimit() now resides in commands/limit_darwin.go and commands/limit_others.go.

The code in commands/limit_darwin.go looks alright to me. Hmm...

[anthonyfok]

Hi @chibicode,

The following appear to be one of the most comprehensive info on max open files limit on Mac OS X. Please take a look:

• http://docs.basho.com/riak/latest/ops/tuning/open-files-limit/#Mac-OS-X

In particular, OS X Yosemite (10.10), which came out in October 2014, eight months after the aforementioned fixes, has made changes to how this limit is set.

You may also try this for a quick fix:

$ sudo sysctl -w kern.maxfiles=65536
$ sudo sysctl -w kern.maxfilesperproc=65536
$ ulimit -n 65536 65536    

Please let us know how it works! Thanks!

[anthonyfok]

Hi @chibicode,

I am being adventurous and decided to take up your suggestion, and committed b9b70fb6b085e5141078bee80f342d2a9fdc2f5e to skip node_modules/ as well as bower_components and .git/ directories. Even though Hugo tries to raise the relevant ulimit on OS X whenever --watch is used, having tens of thousands of files inside node_modules can easily exceed the default hard limit (?) of say 16384 files. So, I though I will go ahead and try it out. Afterall, it is still early in the v0.14 release cycle, and I am sure @bep will help me clean up the mess in case I goofed up. :wink:

It turns out many of the mysteries can be explained by reading about fsnotify, the nifty library maintained by @nathany et al. that allows Hugo to watch for file changes using the same API across all platforms. Its FAQ at https://github.com/go-fsnotify/fsnotify/wiki/FAQ has the following important information:

How many files can be watched at once?

There are OS-specific limits as to how many watches can be created:

• Linux: /proc/sys/fs/inotify/max_user_watches contains the limit, reaching this limit results in a "no space left on device" error.
• BSD / OSX: sysctl variables "kern.maxfiles" and "kern.maxfilesperproc", reaching these limits results in a "too many open files" error.

Also, according to https://github.com/go-fsnotify/fsnotify/, fsnotify uses kqueue on OS X (and BSD too!), but inotify on Linux, which explains why lowering the ulimit maxfiles does not affect Linux at all! The planned use of FSEvents on OS X will do away with the silly maxfiles limitation and should hopefully make life easier for Hugo users on OS X.

Last but not least... should we keep the discussion here? Or should we start a new issue and continue from there? Afterall, this bug _was_ already fixed in February/March 2014 last year. ;-)

Cheers,
Anthony

[chibicode]

Hi @anthonyfok, wow, thank you so much for such a thorough response. I did try ulimit -n but yes, I couldn't get it to go past 10240. I tried your commands but the last one returned limit: setrlimit failed: invalid argument.

$ hugo check ulimit
Current rLimit: {4864 9223372036854775807}
Attempting to increase limit
rLimit after change: {10240 10240}

$ sudo sysctl -w kern.maxfiles=65536
kern.maxfiles: 12288 -> 65536

$ sudo sysctl -w kern.maxfilesperproc=65536
kern.maxfiles: 65536 -> 65536

$ ulimit -n 65536 65536
limit: setrlimit failed: invalid argument

Also thanks for https://github.com/spf13/hugo/commit/b9b70fb6b085e5141078bee80f342d2a9fdc2f5e - although this looks like a huge hack and I now feel bad for requesting this.

Sorry I'm on a hurry, but really appreciated your response.

[nathany]

Hugo Static Site Generator v0.15-DEV BuildDate: 2015-10-15T10:07:58-06:00

I'm seeing a number of errors like this on OS X:

[info] Error while watching new path public/tags/golang/: open public/tags/golang: too many open files

I'm not sure why it's watching the public directory.

[spf13]

That's very strange. We haven't seen these kind of errors in a while. It shouldn't be watching public ever.

[bep]

@nathany are you _sure_ this isn't _something else_ running on your computer? Because this one is way above my pay grade to understand ...

find $GOPATH -name "*.go" | xargs grep -i "while watching"
/Users/bep/go/src/github.com/spf13/hugo/hugolib/page_test.go:       t.Fatalf("Empty files should not trigger an error. Should be able to touch a file while watching without erroring out.")

[nathany]

You're probably right, as I also have reflex running in the same Terminal to watch sass files for sassc. I'll have to double check where the error came from.

UPDATE: Yah, that was Reflex. Sorry for the false alarm.

[hypertexthero]

Am also getting Error: listen tcp 127.0.0.1:1313: socket: too many open files after running hugo server --watch despite attempting to increase ulimit (ulimit -n 999999). This is happening on a blog converted from Jekyll with 2300+ posts. When I delete the folder with lots of posts, it works.

It may be due to Apple's System Integrity Protection utility 'csrutil' ignoring any attempts to increase ulimits. This is on OS X Yosemite 10.10.5.

I don't want to disable csrutil, so I guess I'll have to have to wait for FSEvents to be implemented in Hugo's OS X version before I can use it, unless there is another way?

[anthonyfok]

Could Hugo use this?

FSEvents for Go (OS X) - Clean room implementation
https://github.com/fsnotify/fsevents

[nathany]

@anthonyfok I don't see why not. Though perhaps fsnotify will use fsevents, or a new replacement for fsnotify.

[anthonyfok]

@nathany:

Though perhaps fsnotify will use fsevents

That would be awesome! :-D

[hypertexthero]

Regarding my May 14 comment, I found this command (sudo launchctl limit maxfiles 1000000 1000000) that lets me increase the maximum number of open file descriptors for the current session, and then hugo server --watch works!

[nathany]

Thanks @hypertexthero. It's probably worth adding that command to the fsnotify Wiki https://github.com/fsnotify/fsnotify/wiki or even a blog post on https://fsnotify.org/ (which uses Hugo, btw:-).