Couchdb: CouchDB keeps Crashing

[dpdornseifer]

Hi, a CouchDB 2.0 instance keeps crashing right now. We have about 140 different dbs on it.

Current Behavior

CouchDB shows different errors in the log output. mochiweb error occurs all the time in different fashions.

[notice] 2017-08-23T16:46:18.372962Z nonode@nohost <0.27331.7> 4d214bcd07 localhost:5984 127.0.0.1 undefined GET / 200 ok 0
[info] 2017-08-23T16:46:19.578817Z nonode@nohost <0.204.0> -------- db shards/40000000-5fffffff/_global_changes.1499819725 died with reason {{badmatch,{error,emfile}},[{couch_file,init,1,[{file,"src/couch_file.erl"},{line,381}]},{gen_server,init_it,6,[{file,"gen_server.er
l"},{line,328}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}

[error] 2017-08-23T16:46:19.578936Z nonode@nohost <0.27401.7> -------- CRASH REPORT Process  (<0.27401.7>) with 1 neighbors exited with reason: no match of right hand value {error,emfile} at couch_file:init/1(line:381) <= gen_server:init_it/6(line:328) <= proc_lib:init_p_
do_apply/3(line:240) at gen_server:init_it/6(line:352) <= proc_lib:init_p_do_apply/3(line:240); initial_call: {couch_file,init,['Argument__1']}, ancestors: [<0.27400.7>], messages: [], links: [#Port<0.14183>,<0.27400.7>], dictionary: [], trap_exit: true, status: running,
heap_size: 610, stack_size: 27, reductions: 628

[notice] 2017-08-23T16:46:23.445746Z nonode@nohost <0.27433.7> bf1f1c93fe localhost:5984 127.0.0.1 undefined GET / 200 ok 0

[error] 2017-08-23T16:46:23.545821Z nonode@nohost <0.27434.7> -------- application: mochiweb, "Accept failed error", "{error,emfile}"

[error] 2017-08-23T16:46:23.545971Z nonode@nohost <0.27434.7> -------- CRASH REPORT Process  (<0.27434.7>) with 0 neighbors exited with reason: {error,accept_failed} at mochiweb_acceptor:init/4(line:73) <= proc_lib:init_p_do_apply/3(line:240); initial_call: {mochiweb_acce
ptor,init,['Argument__1','Argument__2',...]}, ancestors: [chttpd,chttpd_sup,<0.293.0>], messages: [], links: [<0.296.0>], dictionary: [], trap_exit: false, status: running, heap_size: 376, stack_size: 27, reductions: 267

[error] 2017-08-23T16:46:23.546004Z nonode@nohost <0.296.0> -------- {mochiweb_socket_server,341,{acceptor_error,{error,accept_failed}}}

[info] 2017-08-23T16:46:29.132354Z nonode@nohost <0.7.0> -------- Application couch_log started on node nonode@nohost
[info] 2017-08-23T16:46:29.138413Z nonode@nohost <0.7.0> -------- Application folsom started on node nonode@nohost
[info] 2017-08-23T16:46:29.204890Z nonode@nohost <0.7.0> -------- Application couch_stats started on node nonode@nohost
[info] 2017-08-23T16:46:29.205042Z nonode@nohost <0.7.0> -------- Application khash started on node nonode@nohost
[info] 2017-08-23T16:46:29.217673Z nonode@nohost <0.7.0> -------- Application couch_event started on node nonode@nohost
[info] 2017-08-23T16:46:29.224604Z nonode@nohost <0.7.0> -------- Application ibrowse started on node nonode@nohost
[info] 2017-08-23T16:46:29.229435Z nonode@nohost <0.7.0> -------- Application ioq started on node nonode@nohost
[info] 2017-08-23T16:46:29.229567Z nonode@nohost <0.7.0> -------- Application mochiweb started on node nonode@nohost
[info] 2017-08-23T16:46:29.229719Z nonode@nohost <0.7.0> -------- Application oauth started on node nonode@nohost
[info] 2017-08-23T16:46:29.235625Z nonode@nohost <0.198.0> -------- Apache CouchDB 2.0.0 is starting.
[info] 2017-08-23T16:46:29.235793Z nonode@nohost <0.199.0> -------- Starting couch_sup
[info] 2017-08-23T16:46:29.366000Z nonode@nohost <0.198.0> -------- Apache CouchDB has started. Time to relax.
[info] 2017-08-23T16:46:29.366334Z nonode@nohost <0.198.0> -------- Apache CouchDB has started on http://127.0.0.1:5986/

Context

CouchDB is running inside a docker container. Health check is defined to restart the service as soon as it cannot 'ping' port 5984 any more.

Your Environment

CouchDB 2.0
Ubuntu 16.04.2 LTS
4.9.32-15.41.amzn1.x86_64 kernel
Running inside a docker container

[wohali]

See your {error,emfile} in there? This means your ulimit for file handles isn't high enough, and/or your system-limit for file handles isn't large enough, either.

Not a CouchDB bug. Up your container's max file handle limit (ulimit -n as root before launching the process, not sure what's required in Docker) or the OS's limit (sysctl fs.file-max) and try again.

[devios1]

To others struggling with this issue, I found this answer on SO to be the solution: https://stackoverflow.com/a/41122156/238948

[wohali]

@devios1 And this is in our documentation: http://docs.couchdb.org/en/2.1.1/maintenance/performance.html#system-resource-limits