Peertube: Crash with error EMFILE: too many open files, uv_resident_set_memory

[Nutomic]

• PeerTube version or URL: PeerTube v1.0.0-beta.11, https://peertube.social

• NodeJS version: v8.11.4

• Server log if useful (Gist/Pastebin...):
  https://gist.github.com/Nutomic/cd33c192360300ac27630eeb90c8e7c0

Looks like it is this error: https://stackoverflow.com/q/8965606

As a workaround, I've added the following lines to /etc/security/limits.conf:

* soft nofile 100000
* hard nofile 100000

[rigelk]

We might want to debug this, as this is often synonym with opening too many sockets (not just regular file descriptors). Otherwise we could also use https://github.com/jprichardson/node-fs-extra to replace the fs module which doesn't know how to deal with EMFILE errors.

[Chocobozzz]

Weird, @Nutomic could you # lsof -p peertube_pid to see if this is peertube that opens too many files?

[Nutomic]

Okay turns out it wasn't the fault of Peertube itself. Right now I have 8862 open files, Peertube only had 183. So I pieced together this command:

$ sudo lsof | awk '{print $1}' | uniq -c | sort -n
      1 COMMAND
      3 acpi_ther
      3 ata_sff
      3 charger_m
      3 cpuhp/0
      3 cpuhp/1
      3 crypto
      3 devfreq_w
      3 ecryptfs-
      3 edac-poll
      3 ext4-rsv-
      3 ib-comp-w
      3 ib_mcast
      3 ib_nl_sa_
      3 ipv6_addr
      3 iscsi_eh
      3 jbd2/sda1
      3 kauditd
      3 kblockd
      3 kcompactd
      3 kdevtmpfs
      3 khugepage
      3 khungtask
      3 kintegrit
      3 ksmd
      3 ksoftirqd
      3 ksoftirqd
      3 kstrp
      3 kswapd0
      3 kthreadd
      3 kthrotld
      3 kworker/0
      3 kworker/0
      3 kworker/0
      3 kworker/0
      3 kworker/1
      3 kworker/1
      3 kworker/1
      3 kworker/1
      3 kworker/u
      3 md
      3 migration
      3 migration
      3 mm_percpu
      3 netns
      3 oom_reape
      3 raid5wq
      3 rcu_bh
      3 rcu_sched
      3 rcu_tasks
      3 rdma_cm
      3 scsi_eh_0
      3 scsi_eh_1
      3 scsi_eh_2
      3 scsi_tmf_
      3 scsi_tmf_
      3 scsi_tmf_
      3 watchdog/
      3 watchdog/
      3 watchdogd
      3 writeback
      6 kworker/u
      8 sh
      9 uniq
      9 vnstatd
     10 sort
     17 awk
     17 evicter
     18 lvmetad
     19 dhclient
     19 qemu-ga
     20 atd
     20 top
     21 npm
     22 cron
     23 fish
     24 gmain
     24 irqbalanc
     29 lsof
     30 agetty
     31 ssh
     34 pickup
     34 qmgr
     37 systemd-u
     38 in:imklog
     38 in:imuxso
     38 rs:main
     38 rsyslogd
     39 accounts-
     39 gdbus
     39 gmain
     39 sshd
     39 sudo
     41 dbus-daem
     43 gdbus
     43 gmain
     43 polkitd
     45 tlsmgr
     46 sd-resolv
     46 systemd-t
     47 gmain
     47 networkd-
     48 iscsid
     48 systemd-r
     51 systemd-l
     58 (sd-pam
     58 systemd
     59 sshd
     63 sshd
     69 systemd-j
     84 npm
    105 node
    114 systemd
    127 master
    129 postgres
    132 snapd
    168 lxcfs
    172 redis-ser
    183 peertube
    192 sshfs
    228 nginx
    498 postgres
   1552 ffmpeg
   1647 node
   1836 catfs

I was only using catfs for testing, it wasn't used by Peertube. But it might still be a good idea to recommend higher file limits for production servers.

Edit: Keep in mind that I restarted the server after I got the error. These values might have looked very different when the error occured.

[Chocobozzz]

Okay so closing, feel free to reopen if you have this bug again (with the lsof output :) )

[Nutomic]

I'm keeping the higher file limit to be safe, so I probably won't.