Website: Command `make serve` in development env may fail due to `too many open files`

Created on 1 Aug 2020 · 16Comments · Source: kubernetes/website

This is a Bug Report

Problem:
Command make serve fails on my laptop (come with macOS Catalina version 10.15.6 (19G73)) due to too many open files after running command make serve.

Built in 16332 ms
Watching for changes in /path/to/website/{archetypes,assets,content,data,i18n,layouts,static,themes}
Watching for config changes in /path/to/website/config.toml, /path/to/website/themes/docsy/config.toml, /path/to/website/go.mod
Environment: "development"
Serving pages from memory
Running in Fast Render Mode. For full rebuilds on change: hugo server --disableFastRender
Web Server is available at http://localhost:1313/ (bind address 127.0.0.1)
Press Ctrl+C to stop
ERROR 2020/08/01 19:09:18 Error: listen tcp 127.0.0.1:1313: socket: too many open files
make: *** [serve] Error 1

I've tried to change the maxfiles limit to 10240 by command sudo launchctl limit maxfiles 10240 unlimited, rerunning make serve got

Error: listen tcp 127.0.0.1:1313: socket: too many open files in system

Seems 10240 exceeds the kernel ulimit. Then I changed my ulimit to 640000/1280000 according to https://superuser.com/a/1171028 , still got no help.

After append options --watch=false to hugo server --buildFuture command of target serve in Makefile, it works, but without hot reloading.

For someone who works on macOS Catalina, the following link may help:
https://discourse.gohugo.io/t/error-hugo-0-52-too-many-open-files/17653
https://superuser.com/a/1171028

It would be better if this issue could be well-documented, for example what's the good value of ulimit should I set.

Proposed Solution:

Provide two targets. make serve-live with option --watch=true to provide hot reloading, while make serve with option --watch=false not.
Well-documented guidelines to let new contributors know this issue, better if along with possible solutions.

Page to Update:
N/A

kinbug

Source

blueabysm

All 16 comments

Maybe PR #22748 is related and #22515 tries to handle this issue?

shuuji3 on 1 Aug 2020

Maybe PR #22748 is related and #22515 tries to handle this issue?

Great. Partial serve would be a good solution to this issue. This one could be closed, thank you!

blueabysm on 1 Aug 2020

👍1

Nice to hear that! 😊

shuuji3 on 1 Aug 2020

Seems this issue needs to be reopened. I've tried Partial serve ，but make partial-serve LANG=en got a lot of errors such as

ERROR 2020/08/02 20:17:33 [en] "docs/reference/glossary/code-contributor.md": "sig" is not a valid glossary term_id, see ./docs/reference/glossary/* for a full list
ERROR 2020/08/02 20:17:33 [en] "docs/reference/glossary/daemonset.md": "node" is not a valid glossary term_id, see ./docs/reference/glossary/* for a full list
ERROR 2020/08/02 20:17:33 [en] "docs/reference/glossary/developer.md": "platform-developer" is not a valid glossary term_id, see ./docs/reference/glossary/* for a full list
ERROR 2020/08/02 20:17:33 [en] "docs/reference/glossary/csi.md": "storage-class" is not a valid glossary term_id, see ./docs/reference/glossary/* for a full list

Error: Error building site: "/path/to/website/content/de/docs/tutorials/hello-minikube.md:41:1": failed to render shortcode "codenew": failed to process shortcode: "/path/to/website/layouts/shortcodes/codenew.html:14:16": execute of template failed: template: shortcodes/codenew.html:14:16: executing "shortcodes/codenew.html" at <readFile $filename>: error calling readFile: file "content/en/examples/minikube/server.js" does not exist
make: *** [partial-serve] Error 255

Different language arguments leads to different error messages.

blueabysm on 3 Aug 2020

/kind bug

sftim on 3 Aug 2020

Hi @blueabysm have you try doing make container-partial-serve LANG=en ?

irvifa on 11 Aug 2020

Hi @blueabysm have you try doing make container-partial-serve LANG=en ?

Hi @irvifa . I've tried make container-partial-serve LANG=en, but got

➜  website git:(master) ✗ docker login
Authenticating with existing credentials...
Login Succeeded
➜  website git:(master) ✗ make container-partial-serve LANG=en
docker run --rm --interactive --tty --volume /path/to/website:/src --mount type=tmpfs,destination=/src/resources,tmpfs-mode=0755 -p 1313:1313 kubernetes-hugo:v0.74.3-477fda0143e3 hugo server --buildFuture --bind 0.0.0.0 --contentDir content/en
Unable to find image 'kubernetes-hugo:v0.74.3-477fda0143e3' locally
docker: Error response from daemon: pull access denied for kubernetes-hugo, repository does not exist or may require 'docker login': denied: requested access to the resource is denied.
See 'docker run --help'.
make: *** [container-partial-serve] Error 125

Before doing, I've pulled from this repo to ensure that the main branch of my fork points to the same as the upstream.

blueabysm on 12 Aug 2020

Seems like the Dockerimage doesn’t exist. You haven’t build that on your local yet. I need to investigate the one without Docker though.

irvifa on 12 Aug 2020

Seems like the Dockerimage doesn’t exist. You haven’t build that on your local yet. I need to investigate the one without Docker though.

Appreciated for your work! 😃

blueabysm on 12 Aug 2020

@blueabysm can you take a look on https://gist.github.com/tombigel/d503800a282fcadbee14b537735d202c#gistcomment-3288161 This works for both of Catalina and Mojave

irvifa on 17 Aug 2020

👍1

@irvifa Thanks for your reply. It works!

According to https://gist.github.com/tombigel/d503800a282fcadbee14b537735d202c#file-load-sh, what I missed is limit.maxproc.plist file.

BTW, I'll use this new Github account mainly from now on. The previous account @blueabysm will still receive email notifications.

How would you think if I write this down into https://github.com/kubernetes/website/blob/master/README.md?

rayw000 on 17 Aug 2020

Okay I'll write it down in my next PR. I'm working on it right now.

irvifa on 17 Aug 2020

👍1

@rayw000 I put it on https://github.com/kubernetes/website/pull/23207

irvifa on 17 Aug 2020

@rayw000 I put it on https://github.com/kubernetes/website/pull/23207

Haha you've done exactly what I meant to do. Thanks a lot! 😄

rayw000 on 17 Aug 2020

/close
via https://github.com/kubernetes/website/pull/23207

irvifa on 18 Aug 2020

@irvifa: Closing this issue.

In response to this:

/close
via https://github.com/kubernetes/website/pull/23207

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.