Couchdb: CouchDB 3-node cluster (Windows) - multiple erlang errors

Created on 18 Dec 2017  路  18Comments  路  Source: apache/couchdb

Im receiving multiple erlang errors in my CouchDB 2.1.1 cluster (3 nodes), see errors and node configuration below:

3 nodes (10.0.7.4 - 10.0.7.6), Azure application gateway is used as load balancer.
Why do these errors appear? system resources of the nodes are far from overload.

I would be thankful for any help - thanks in advance.

rexi_server: from: [email protected](<0.14976.568>) mfa: fabric_rpc:changes/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,256}]},{rexi,stream_last,2,[{file,"src/rexi.erl"},{line,224}]},{fabric_rpc,changes,4,[{file,"src/fabric_rpc.erl"},{line,86}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,139}]}]

rexi_server: from: [email protected](<13540.24597.655>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,256}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,204}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,308}]},{couch_mrview,finish_fold,2,[{file,"src/couch_mrview.erl"},{line,642}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,139}]}]

rexi_server: from: [email protected](<13540.5991.623>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,256}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,204}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,308}]},{couch_mrview,map_fold,3,[{file,"src/couch_mrview.erl"},{line,511}]},{couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,222}]},{couch_db,enum_docs,5,[{file,"src/couch_db.erl"},{line,1450}]},{couch_mrview,all_docs_fold,4,[{file,"src/couch_mrview.erl"},{line,425}]}]

req_err(3206982071) unknown_error : normal [<<"mochiweb_request:recv/3 L180">>,<<"mochiweb_request:stream_unchunked_body/4 L540">>,<<"mochiweb_request:recv_body/2 L214">>,<<"chttpd:body/1 L636">>,<<"chttpd:json_body/1 L649">>,<<"chttpd:json_body_obj/1 L657">>,<<"chttpd_db:db_req/2 L386">>,<<"chttpd:process_request/1 L295">>]

** System running to use fully qualified hostnames ** ** Hostname localhost is illegal **

COMPACTION-ERRORS
Supervisor couch_secondary_services had child compaction_daemon started with couch_compaction_daemon:start_link() at <0.18509.478> exit with reason {compaction_loop_died,{timeout,{gen_server,call,[couch_server,get_server]}}} in context child_terminated 

 CRASH REPORT Process couch_compaction_daemon (<0.18509.478>) with 0 neighbors exited with reason: {compaction_loop_died,{timeout,{gen_server,call,[couch_server,get_server]}}} at gen_server:terminate/7(line:826) <= proc_lib:init_p_do_apply/3(line:240); initial_call: {couch_compaction_daemon,init,['Argument__1']}, ancestors: [couch_secondary_services,couch_sup,<0.200.0>], messages: [], links: [<0.12665.492>], dictionary: [], trap_exit: true, status: running, heap_size: 987, stack_size: 27, reductions: 3173

gen_server couch_compaction_daemon terminated with reason: {compaction_loop_died,{timeout,{gen_server,call,[couch_server,get_server]}}} last msg: {'EXIT',<0.23195.476>,{timeout,{gen_server,call,[couch_server,get_server]}}} state: {state,<0.23195.476>,[]}

Error in process <0.16890.22> on node '[email protected]' with exit value: {{rexi_DOWN,{'[email protected]',noproc}},[{mem3_rpc,rexi_call,2,[{file,"src/mem3_rpc.erl"},{line,269}]},{mem3_rep,calculate_start_seq,1,[{file,"src/mem3_rep.erl"},{line,194}]},{mem3_rep,repl,2,[{file,"src/mem3_rep.erl"},{line,175}]},{mem3_rep,go,1,[{file,"src/mem3_rep.erl"},{line,81}]},{mem3_sync,'-start_push_replication/1-fun-0-',2,[{file,"src/mem3_sync.erl"},{line,208}]}]}

`#vm.args

-name [email protected]
-setcookie monster
-kernel error_logger silent
-sasl sasl_error_logger false
+K true
+A 16
+Bd -noinput
+Q 134217727`

`local.ini
[fabric]
request_timeout = infinity

[couchdb]
max_dbs_open = 10000
os_process_timeout = 20000
uuid =

[chttpd]
port = 5984
bind_address = 0.0.0.0

[httpd]
socket_options = [{recbuf, 262144}, {sndbuf, 262144}, {nodelay, true}]
enable_cors = true

[couch_httpd_auth]
secret =

[daemons]
compaction_daemon={couch_compaction_daemon, start_link, []}

[compactions]
_default = [{db_fragmentation, "50%"}, {view_fragmentation, "50%"}, {from, "23:00"}, {to, "04:00"}]

[compaction_daemon]
check_interval = 300
min_file_size = 100000

[vendor]
name = COUCHCLUSTERNODE0X

[admins]
adminuser =

[cors]
methods = GET, PUT, POST, HEAD, DELETE
headers = accept, authorization, content-type, origin, referer
origins = *
credentials = true

[query_server_config]
os_process_limit = 2000
os_process_soft_limit = 1000
`

windows
Source
picture harald-d

All 18 comments

I am facing same issue.

style95 picture style95 on 19 Dec 2017
👍1

If you could post your configuration too it may help the devs identifying the problem :)

harald-d picture harald-d on 19 Dec 2017

I just used default configurations in docker image, apache/couchdb:2.1.

style95 picture style95 on 19 Dec 2017
👍1

I have same issue.
Installed couchdb two node cluster from docker latest image.
I have this issue when replicated 1500 databases (25Gb total size) from other bigcouch cluster.

sergey-safarov picture sergey-safarov on 15 Jan 2018

My guess is this the same as issue #796. I'll leave this open for a bit, but I expect it's the same problem.

You should know that there are reports of performance issues with CouchDB running inside of Docker, and especially on Windows as a host platform. To be absolutely sure, you may want to try running this on similar Azure machines running Linux natively under Hyper-V instead of inside a Windows host, or if possible, directly on Linux without Docker at all.

wohali picture wohali on 17 Jan 2018

Actually I am facing same issue on CentOS with apache/docker image.

style95 picture style95 on 17 Jan 2018

So it sounds like possibly a virtualisation issue.

@kocolosk has been spending more time with Docker as of late, perhaps he has an idea?

wohali picture wohali on 18 Jan 2018

I just want to provide more information for this issue, and I run my tests on physical machine.

style95 picture style95 on 18 Jan 2018
👍1

I started one node couchdb docker instance and got similar error. Looks like issue not related to cluster config

[[email protected] tmp]# cat import.log | grep -v "Request to create"| head
褟薪胁 18 13:27:57 node1.docker.rcsnet.ru sh[28967]: [error] 2018-01-18T13:27:57.441419Z [email protected] <0.14020.0> -------- rexi_server: from: [email protected](<0.14018.0>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,256}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,204}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,308}]},{couch_mrview,finish_fold,2,[{file,"src/couch_mrview.erl"},{line,642}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,139}]}]
褟薪胁 18 13:27:57 node1.docker.rcsnet.ru sh[28967]: [error] 2018-01-18T13:27:57.441429Z [email protected] <0.14026.0> -------- rexi_server: from: [email protected](<0.14018.0>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,256}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,204}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,308}]},{couch_mrview,finish_fold,2,[{file,"src/couch_mrview.erl"},{line,642}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,139}]}]
褟薪胁 18 13:27:57 node1.docker.rcsnet.ru sh[28967]: [error] 2018-01-18T13:27:57.441546Z [email protected] <0.14022.0> -------- rexi_server: from: [email protected](<0.14018.0>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,256}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,204}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,308}]},{couch_mrview,map_fold,3,[{file,"src/couch_mrview.erl"},{line,511}]},{couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848}]},{couch_btree,stream_kp_node,8,[{file,"src/couch_btree.erl"},{line,819}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,222}]},{couch_db,enum_docs,5,[{file,"src/couch_db.erl"},{line,1450}]}]
褟薪胁 18 13:27:57 node1.docker.rcsnet.ru sh[28967]: [error] 2018-01-18T13:27:57.442220Z [email protected] <0.14028.0> -------- rexi_server: from: [email protected](<0.14018.0>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,256}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,204}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,308}]},{couch_mrview,finish_fold,2,[{file,"src/couch_mrview.erl"},{line,642}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,139}]}]
褟薪胁 18 13:27:57 node1.docker.rcsnet.ru sh[28967]: [error] 2018-01-18T13:27:57.442334Z [email protected] <0.14025.0> -------- rexi_server: from: [email protected](<0.14018.0>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,256}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,204}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,308}]},{couch_mrview,finish_fold,2,[{file,"src/couch_mrview.erl"},{line,642}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,139}]}]
褟薪胁 18 13:27:57 node1.docker.rcsnet.ru sh[28967]: [error] 2018-01-18T13:27:57.442524Z [email protected] <0.14027.0> -------- rexi_server: from: [email protected](<0.14018.0>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,256}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,204}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,308}]},{couch_mrview,finish_fold,2,[{file,"src/couch_mrview.erl"},{line,642}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,139}]}]
褟薪胁 18 13:27:57 node1.docker.rcsnet.ru sh[28967]: [error] 2018-01-18T13:27:57.442602Z [email protected] <0.14023.0> -------- rexi_server: from: [email protected](<0.14018.0>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,256}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,204}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,308}]},{couch_mrview,finish_fold,2,[{file,"src/couch_mrview.erl"},{line,642}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,139}]}]
褟薪胁 18 13:27:57 node1.docker.rcsnet.ru sh[28967]: [error] 2018-01-18T13:27:57.442760Z [email protected] <0.14021.0> -------- rexi_server: from: [email protected](<0.14018.0>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,256}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,204}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,308}]},{couch_mrview,map_fold,3,[{file,"src/couch_mrview.erl"},{line,511}]},{couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848}]},{couch_btree,stream_kp_node,8,[{file,"src/couch_btree.erl"},{line,819}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,222}]},{couch_db,enum_docs,5,[{file,"src/couch_db.erl"},{line,1450}]}]
褟薪胁 18 13:28:40 node1.docker.rcsnet.ru sh[28967]: [error] 2018-01-18T13:28:40.922262Z [email protected] <0.26046.0> -------- rexi_server: from: [email protected](<0.26035.0>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,256}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,204}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,308}]},{couch_mrview,finish_fold,2,[{file,"src/couch_mrview.erl"},{line,642}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,139}]}]
褟薪胁 18 13:28:40 node1.docker.rcsnet.ru sh[28967]: [error] 2018-01-18T13:28:40.922386Z [email protected] <0.26048.0> -------- rexi_server: from: [email protected](<0.26035.0>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,256}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,204}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,308}]},{couch_mrview,map_fold,3,[{file,"src/couch_mrview.erl"},{line,511}]},{couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,222}]},{couch_db,enum_docs,5,[{file,"src/couch_db.erl"},{line,1450}]},{couch_mrview,all_docs_fold,4,[{file,"src/couch_mrview.erl"},{line,425}]}]

Next test on bare metal server on CentOS 7 from rpm package

sergey-safarov picture sergey-safarov on 18 Jan 2018

I installed CouchDB from couchdb-2.1.1-2.el7.centos.x86_64.rpm

[[email protected] tmp]# grep error /var/log/couchdb/couchdb.log | grep rexi_server | head
[error] 2018-01-18T18:57:41.932949Z [email protected] <0.12764.0> -------- rexi_server: from: [email protected](<0.12752.0>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,256}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,204}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,308}]},{couch_mrview,finish_fold,2,[{file,"src/couch_mrview.erl"},{line,642}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,139}]}]
[error] 2018-01-18T18:57:41.933954Z [email protected] <0.12760.0> -------- rexi_server: from: [email protected](<0.12752.0>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,256}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,204}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,308}]},{couch_mrview,finish_fold,2,[{file,"src/couch_mrview.erl"},{line,642}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,139}]}]
[error] 2018-01-18T18:57:41.933996Z [email protected] <0.12765.0> -------- rexi_server: from: [email protected](<0.12752.0>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,256}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,204}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,308}]},{couch_mrview,finish_fold,2,[{file,"src/couch_mrview.erl"},{line,642}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,139}]}]
[error] 2018-01-18T18:57:41.934089Z [email protected] <0.12762.0> -------- rexi_server: from: [email protected](<0.12752.0>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,256}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,204}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,308}]},{couch_mrview,finish_fold,2,[{file,"src/couch_mrview.erl"},{line,642}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,139}]}]
[error] 2018-01-18T18:57:41.934061Z [email protected] <0.12763.0> -------- rexi_server: from: [email protected](<0.12752.0>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,256}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,204}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,308}]},{couch_mrview,finish_fold,2,[{file,"src/couch_mrview.erl"},{line,642}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,139}]}]
[error] 2018-01-18T18:57:41.934294Z [email protected] <0.12766.0> -------- rexi_server: from: [email protected](<0.12752.0>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,256}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,204}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,308}]},{couch_mrview,finish_fold,2,[{file,"src/couch_mrview.erl"},{line,642}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,139}]}]
[error] 2018-01-18T18:57:41.934416Z [email protected] <0.12759.0> -------- rexi_server: from: [email protected](<0.12752.0>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,256}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,204}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,308}]},{couch_mrview,finish_fold,2,[{file,"src/couch_mrview.erl"},{line,642}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,139}]}]
[error] 2018-01-18T18:57:41.935568Z [email protected] <0.12761.0> -------- rexi_server: from: [email protected](<0.12752.0>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,256}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,204}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,308}]},{couch_mrview,map_fold,3,[{file,"src/couch_mrview.erl"},{line,511}]},{couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848}]},{couch_btree,stream_kp_node,8,[{file,"src/couch_btree.erl"},{line,819}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,222}]},{couch_db,enum_docs,5,[{file,"src/couch_db.erl"},{line,1450}]}]
[error] 2018-01-18T18:58:25.372933Z [email protected] <0.23503.0> -------- rexi_server: from: [email protected](<0.23501.0>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,256}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,204}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,308}]},{couch_mrview,finish_fold,2,[{file,"src/couch_mrview.erl"},{line,642}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,139}]}]
[error] 2018-01-18T18:58:25.373010Z [email protected] <0.23509.0> -------- rexi_server: from: [email protected](<0.23501.0>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,256}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,204}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,308}]},{couch_mrview,map_fold,3,[{file,"src/couch_mrview.erl"},{line,511}]},{couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,222}]},{couch_db,enum_docs,5,[{file,"src/couch_db.erl"},{line,1450}]},{couch_mrview,all_docs_fold,4,[{file,"src/couch_mrview.erl"},{line,425}]}]

Other types of errors
File operation error

[[email protected] tmp]# grep error /var/log/couchdb/couchdb.log | grep "File operation error" | head
[error] 2018-01-18T18:56:02.373572Z [email protected] <0.12.0> -------- File operation error: emfile. Target: /opt/couchdb/bin/../lib/stdlib-2.8/ebin/erl_posix_msg.beam. Function: get_file. Process: code_server.
[error] 2018-01-18T18:56:02.377095Z [email protected] <0.12.0> -------- File operation error: emfile. Target: erl_posix_msg.beam. Function: get_file. Process: code_server.
[error] 2018-01-18T18:56:02.378314Z [email protected] <0.12.0> -------- File operation error: emfile. Target: /opt/couchdb/bin/../lib/stdlib-2.8/ebin/erl_posix_msg.beam. Function: get_file. Process: code_server.
[error] 2018-01-18T18:56:02.381081Z [email protected] <0.12.0> -------- File operation error: emfile. Target: erl_posix_msg.beam. Function: get_file. Process: code_server.
[error] 2018-01-18T18:56:02.381934Z [email protected] <0.12.0> -------- File operation error: emfile. Target: /opt/couchdb/bin/../lib/stdlib-2.8/ebin/erl_posix_msg.beam. Function: get_file. Process: code_server.
[error] 2018-01-18T18:56:02.384896Z [email protected] <0.12.0> -------- File operation error: emfile. Target: erl_posix_msg.beam. Function: get_file. Process: code_server.
[error] 2018-01-18T18:56:02.385662Z [email protected] <0.12.0> -------- File operation error: emfile. Target: /opt/couchdb/bin/../lib/stdlib-2.8/ebin/erl_posix_msg.beam. Function: get_file. Process: code_server.
[error] 2018-01-18T18:56:02.388574Z [email protected] <0.12.0> -------- File operation error: emfile. Target: erl_posix_msg.beam. Function: get_file. Process: code_server.
[error] 2018-01-18T18:56:02.389582Z [email protected] <0.12.0> -------- File operation error: emfile. Target: /opt/couchdb/bin/../lib/stdlib-2.8/ebin/erl_posix_msg.beam. Function: get_file. Process: code_server.
[error] 2018-01-18T18:56:02.392229Z [email protected] <0.12.0> -------- File operation error: emfile. Target: erl_posix_msg.beam. Function: get_file. Process: code_server.

died with reason

[[email protected] tmp]# grep error /var/log/couchdb/couchdb.log | grep "died with reason" | head
[info] 2018-01-18T18:56:11.273221Z [email protected] <0.207.0> -------- db shards/80000000-9fffffff/account/39/6e/a348ae21457aa78854638ca364e8.1516301771 died with reason {{badmatch,{error,{{badmatch,{error,emfile}},[{couch_file,sync,1,[{file,"src/couch_file.erl"},{line,216}]},{couch_db_updater,sync_header,2,[{file,"src/couch_db_updater.erl"},{line,1008}]},{couch_db_updater,init,1,[{file,"src/couch_db_updater.erl"},{line,67}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,328}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}}},[{couch_db,start_link,3,[{file,"src/couch_db.erl"},{line,136}]},{couch_server,'-open_async/5-fun-0-',5,[{file,"src/couch_server.erl"},{line,310}]}]}
[info] 2018-01-18T18:56:11.273522Z [email protected] <0.207.0> -------- db shards/a0000000-bfffffff/account/39/6e/a348ae21457aa78854638ca364e8.1516301771 died with reason {{badmatch,{error,{{badmatch,{error,emfile}},[{couch_file,sync,1,[{file,"src/couch_file.erl"},{line,216}]},{couch_db_updater,sync_header,2,[{file,"src/couch_db_updater.erl"},{line,1008}]},{couch_db_updater,init,1,[{file,"src/couch_db_updater.erl"},{line,67}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,328}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}}},[{couch_db,start_link,3,[{file,"src/couch_db.erl"},{line,136}]},{couch_server,'-open_async/5-fun-0-',5,[{file,"src/couch_server.erl"},{line,310}]}]}
[info] 2018-01-18T18:56:11.273854Z [email protected] <0.207.0> -------- db shards/60000000-7fffffff/account/39/6e/a348ae21457aa78854638ca364e8.1516301771 died with reason {{badmatch,{error,{{badmatch,{error,emfile}},[{couch_file,sync,1,[{file,"src/couch_file.erl"},{line,216}]},{couch_db_updater,sync_header,2,[{file,"src/couch_db_updater.erl"},{line,1008}]},{couch_db_updater,init,1,[{file,"src/couch_db_updater.erl"},{line,67}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,328}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}}},[{couch_db,start_link,3,[{file,"src/couch_db.erl"},{line,136}]},{couch_server,'-open_async/5-fun-0-',5,[{file,"src/couch_server.erl"},{line,310}]}]}
[info] 2018-01-18T18:56:11.532489Z [email protected] <0.207.0> -------- db shards/80000000-9fffffff/account/39/6e/a348ae21457aa78854638ca364e8.1516301771 died with reason {{badmatch,{error,emfile}},[{couch_file,sync,1,[{file,"src/couch_file.erl"},{line,216}]},{couch_db_updater,sync_header,2,[{file,"src/couch_db_updater.erl"},{line,1008}]},{couch_db_updater,init,1,[{file,"src/couch_db_updater.erl"},{line,67}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,328}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}
[info] 2018-01-18T18:57:05.184325Z [email protected] <0.207.0> -------- db shards/80000000-9fffffff/account/42/f7/db16e5f11dc86e01cbaa653cf6e3.1516301823 died with reason {{badmatch,{error,emfile}},[{couch_file,sync,1,[{file,"src/couch_file.erl"},{line,216}]},{couch_db_updater,sync_header,2,[{file,"src/couch_db_updater.erl"},{line,1008}]},{couch_db_updater,handle_info,2,[{file,"src/couch_db_updater.erl"},{line,338}]},{gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,615}]},{gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,681}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}
[info] 2018-01-18T18:57:05.184600Z [email protected] <0.207.0> -------- db shards/e0000000-ffffffff/account/42/f7/db16e5f11dc86e01cbaa653cf6e3.1516301823 died with reason {{badmatch,{error,emfile}},[{couch_file,sync,1,[{file,"src/couch_file.erl"},{line,216}]},{couch_db_updater,sync_header,2,[{file,"src/couch_db_updater.erl"},{line,1008}]},{couch_db_updater,handle_info,2,[{file,"src/couch_db_updater.erl"},{line,338}]},{gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,615}]},{gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,681}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}
[info] 2018-01-18T18:57:05.184843Z [email protected] <0.207.0> -------- db shards/60000000-7fffffff/account/42/f7/db16e5f11dc86e01cbaa653cf6e3.1516301823 died with reason {{badmatch,{error,emfile}},[{couch_file,sync,1,[{file,"src/couch_file.erl"},{line,216}]},{couch_db_updater,sync_header,2,[{file,"src/couch_db_updater.erl"},{line,1008}]},{couch_db_updater,handle_info,2,[{file,"src/couch_db_updater.erl"},{line,338}]},{gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,615}]},{gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,681}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}
[info] 2018-01-18T19:01:13.389304Z [email protected] <0.207.0> -------- db shards/80000000-9fffffff/system_media.1516301568 died with reason {{badmatch,{error,emfile}},[{couch_file,init,1,[{file,"src/couch_file.erl"},{line,398}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,328}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}
[info] 2018-01-18T19:01:13.390153Z [email protected] <0.207.0> -------- db shards/80000000-9fffffff/tasks.1516301582 died with reason {{badmatch,{error,emfile}},[{couch_file,init,1,[{file,"src/couch_file.erl"},{line,398}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,328}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}
[info] 2018-01-18T19:01:13.391205Z [email protected] <0.207.0> -------- db shards/80000000-9fffffff/brand_provisioner.1516301513 died with reason {{badmatch,{error,emfile}},[{couch_file,init,1,[{file,"src/couch_file.erl"},{line,398}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,328}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}

neighbors exited with reason

[[email protected] tmp]# grep error /var/log/couchdb/couchdb.log | grep "neighbors exited with reason" | head
[error] 2018-01-18T18:56:11.273943Z [email protected] <0.1669.2> -------- CRASH REPORT Process  (<0.1669.2>) with 1 neighbors exited with reason: no match of right hand value {error,emfile} at couch_file:sync/1(line:216) <= couch_db_updater:sync_header/2(line:1008) <= couch_db_updater:init/1(line:67) <= gen_server:init_it/6(line:328) <= proc_lib:init_p_do_apply/3(line:240) at gen_server:init_it/6(line:352) <= proc_lib:init_p_do_apply/3(line:240); initial_call: {couch_db_updater,init,['Argument__1']}, ancestors: [<0.1655.2>], messages: [], links: [<0.1655.2>], dictionary: [{io_priority,{db_update,<<"shards/80000000-9fffffff/account/39/6e/a348ae...">>}},...], trap_exit: false, status: running, heap_size: 1598, stack_size: 27, reductions: 2610
[error] 2018-01-18T18:56:11.274595Z [email protected] <0.1671.2> -------- CRASH REPORT Process  (<0.1671.2>) with 0 neighbors exited with reason: no match of right hand value {error,emfile} at couch_file:sync/1(line:216) <= couch_db_updater:sync_header/2(line:1008) <= couch_db_updater:init/1(line:67) <= gen_server:init_it/6(line:328) <= proc_lib:init_p_do_apply/3(line:240) at gen_server:init_it/6(line:352) <= proc_lib:init_p_do_apply/3(line:240); initial_call: {couch_db_updater,init,['Argument__1']}, ancestors: [<0.1659.2>], messages: [], links: [<0.1659.2>], dictionary: [{io_priority,{db_update,<<"shards/60000000-7fffffff/account/39/6e/a348ae...">>}},...], trap_exit: false, status: running, heap_size: 1598, stack_size: 27, reductions: 2610
[error] 2018-01-18T18:56:11.532530Z [email protected] <0.1701.2> -------- CRASH REPORT Process  (<0.1701.2>) with 2 neighbors exited with reason: no match of right hand value {error,emfile} at couch_file:sync/1(line:216) <= couch_db_updater:sync_header/2(line:1008) <= couch_db_updater:init/1(line:67) <= gen_server:init_it/6(line:328) <= proc_lib:init_p_do_apply/3(line:240) at gen_server:init_it/6(line:352) <= proc_lib:init_p_do_apply/3(line:240); initial_call: {couch_db_updater,init,['Argument__1']}, ancestors: [<0.1697.2>], messages: [], links: [<0.1697.2>], dictionary: [{io_priority,{db_update,<<"shards/80000000-9fffffff/account/39/6e/a348ae...">>}},...], trap_exit: false, status: running, heap_size: 1598, stack_size: 27, reductions: 1278
[error] 2018-01-18T18:57:00.102237Z [email protected] <0.4660.2> -------- CRASH REPORT Process  (<0.4660.2>) with 0 neighbors exited with reason: {error,accept_failed} at mochiweb_acceptor:init/4(line:55) <= proc_lib:init_p_do_apply/3(line:240); initial_call: {mochiweb_acceptor,init,['Argument__1','Argument__2',...]}, ancestors: [chttpd,chttpd_sup,<0.328.0>], messages: [], links: [<0.336.0>], dictionary: [], trap_exit: false, status: running, heap_size: 376, stack_size: 27, reductions: 267
[error] 2018-01-18T18:57:05.184346Z [email protected] <0.5458.2> -------- CRASH REPORT Process  (<0.5458.2>) with 0 neighbors exited with reason: no match of right hand value {error,emfile} at couch_file:sync/1(line:216) <= couch_db_updater:sync_header/2(line:1008) <= couch_db_updater:handle_info/2(line:338) <= gen_server:try_dispatch/4(line:615) <= gen_server:handle_msg/5(line:681) <= proc_lib:init_p_do_apply/3(line:240) at gen_server:terminate/7(line:826) <= proc_lib:init_p_do_apply/3(line:240); initial_call: {couch_db_updater,init,['Argument__1']}, ancestors: [<0.5444.2>], messages: [{'DOWN',#Ref<0.0.5.32558>,process,<0.5446.2>,normal}], links: [<0.207.0>], dictionary: [{io_priority,{db_update,<<"shards/80000000-9fffffff/account/42/f7/d...">>}},...], trap_exit: false, status: running, heap_size: 4185, stack_size: 27, reductions: 6779
[error] 2018-01-18T18:57:05.184669Z [email protected] <0.5455.2> -------- CRASH REPORT Process  (<0.5455.2>) with 0 neighbors exited with reason: no match of right hand value {error,emfile} at couch_file:sync/1(line:216) <= couch_db_updater:sync_header/2(line:1008) <= couch_db_updater:handle_info/2(line:338) <= gen_server:try_dispatch/4(line:615) <= gen_server:handle_msg/5(line:681) <= proc_lib:init_p_do_apply/3(line:240) at gen_server:terminate/7(line:826) <= proc_lib:init_p_do_apply/3(line:240); initial_call: {couch_db_updater,init,['Argument__1']}, ancestors: [<0.5439.2>], messages: [{'DOWN',#Ref<0.0.4.60859>,process,<0.5440.2>,normal}], links: [<0.207.0>], dictionary: [{io_priority,{db_update,<<"shards/e0000000-ffffffff/account/42/f7/d...">>}},...], trap_exit: false, status: running, heap_size: 4185, stack_size: 27, reductions: 6544
[error] 2018-01-18T18:57:05.184976Z [email protected] <0.5459.2> -------- CRASH REPORT Process  (<0.5459.2>) with 0 neighbors exited with reason: no match of right hand value {error,emfile} at couch_file:sync/1(line:216) <= couch_db_updater:sync_header/2(line:1008) <= couch_db_updater:handle_info/2(line:338) <= gen_server:try_dispatch/4(line:615) <= gen_server:handle_msg/5(line:681) <= proc_lib:init_p_do_apply/3(line:240) at gen_server:terminate/7(line:826) <= proc_lib:init_p_do_apply/3(line:240); initial_call: {couch_db_updater,init,['Argument__1']}, ancestors: [<0.5447.2>], messages: [{'DOWN',#Ref<0.0.2.68255>,process,<0.5448.2>,normal}], links: [<0.207.0>], dictionary: [{io_priority,{db_update,<<"shards/60000000-7fffffff/account/42/f7/d...">>}},...], trap_exit: false, status: running, heap_size: 4185, stack_size: 27, reductions: 7479
[error] 2018-01-18T19:01:03.968059Z [email protected] <0.27902.3> -------- CRASH REPORT Process  (<0.27902.3>) with 2 neighbors exited with reason: no match of right hand value {error,emfile} at couch_file:init/1(line:398) <= gen_server:init_it/6(line:328) <= proc_lib:init_p_do_apply/3(line:240) at gen_server:init_it/6(line:352) <= proc_lib:init_p_do_apply/3(line:240); initial_call: {couch_file,init,['Argument__1']}, ancestors: [<0.27901.3>,<0.27900.3>], messages: [], links: [#Port<0.20470>,<0.27901.3>], dictionary: [{read_timestamp,{1516,302063,967577}}], trap_exit: false, status: running, heap_size: 610, stack_size: 27, reductions: 768
[error] 2018-01-18T19:01:03.981875Z [email protected] <0.214.0> -------- CRASH REPORT Process couch_index_server (<0.214.0>) with 1 neighbors exited with reason: no match of right hand value {error,emfile} at couch_file:init/1(line:398) <= gen_server:init_it/6(line:328) <= proc_lib:init_p_do_apply/3(line:240) at gen_server:terminate/7(line:826) <= proc_lib:init_p_do_apply/3(line:240); initial_call: {couch_index_server,init,['Argument__1']}, ancestors: [couch_secondary_services,couch_sup,<0.201.0>], messages: [], links: [<0.216.0>,<0.210.0>], dictionary: [], trap_exit: true, status: running, heap_size: 4185, stack_size: 27, reductions: 584992
[error] 2018-01-18T19:01:05.849952Z [email protected] <0.32628.3> -------- CRASH REPORT Process  (<0.32628.3>) with 2 neighbors exited with reason: no match of right hand value {error,emfile} at couch_file:init/1(line:398) <= gen_server:init_it/6(line:328) <= proc_lib:init_p_do_apply/3(line:240) at gen_server:init_it/6(line:352) <= proc_lib:init_p_do_apply/3(line:240); initial_call: {couch_file,init,['Argument__1']}, ancestors: [<0.32627.3>,<0.32626.3>], messages: [], links: [#Port<0.21996>,<0.32627.3>], dictionary: [{read_timestamp,{1516,302065,849433}}], trap_exit: false, status: running, heap_size: 610, stack_size: 27, reductions: 768

mfa: fabric_rpc:all_docs

[[email protected] tmp]# grep error /var/log/couchdb/couchdb.log | grep "mfa: fabric_rpc:all_docs" | head
[error] 2018-01-18T18:57:41.932949Z [email protected] <0.12764.0> -------- rexi_server: from: [email protected](<0.12752.0>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,256}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,204}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,308}]},{couch_mrview,finish_fold,2,[{file,"src/couch_mrview.erl"},{line,642}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,139}]}]
[error] 2018-01-18T18:57:41.933954Z [email protected] <0.12760.0> -------- rexi_server: from: [email protected](<0.12752.0>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,256}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,204}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,308}]},{couch_mrview,finish_fold,2,[{file,"src/couch_mrview.erl"},{line,642}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,139}]}]
[error] 2018-01-18T18:57:41.933996Z [email protected] <0.12765.0> -------- rexi_server: from: [email protected](<0.12752.0>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,256}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,204}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,308}]},{couch_mrview,finish_fold,2,[{file,"src/couch_mrview.erl"},{line,642}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,139}]}]
[error] 2018-01-18T18:57:41.934089Z [email protected] <0.12762.0> -------- rexi_server: from: [email protected](<0.12752.0>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,256}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,204}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,308}]},{couch_mrview,finish_fold,2,[{file,"src/couch_mrview.erl"},{line,642}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,139}]}]
[error] 2018-01-18T18:57:41.934061Z [email protected] <0.12763.0> -------- rexi_server: from: [email protected](<0.12752.0>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,256}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,204}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,308}]},{couch_mrview,finish_fold,2,[{file,"src/couch_mrview.erl"},{line,642}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,139}]}]
[error] 2018-01-18T18:57:41.934294Z [email protected] <0.12766.0> -------- rexi_server: from: [email protected](<0.12752.0>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,256}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,204}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,308}]},{couch_mrview,finish_fold,2,[{file,"src/couch_mrview.erl"},{line,642}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,139}]}]
[error] 2018-01-18T18:57:41.934416Z [email protected] <0.12759.0> -------- rexi_server: from: [email protected](<0.12752.0>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,256}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,204}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,308}]},{couch_mrview,finish_fold,2,[{file,"src/couch_mrview.erl"},{line,642}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,139}]}]
[error] 2018-01-18T18:57:41.935568Z [email protected] <0.12761.0> -------- rexi_server: from: [email protected](<0.12752.0>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,256}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,204}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,308}]},{couch_mrview,map_fold,3,[{file,"src/couch_mrview.erl"},{line,511}]},{couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848}]},{couch_btree,stream_kp_node,8,[{file,"src/couch_btree.erl"},{line,819}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,222}]},{couch_db,enum_docs,5,[{file,"src/couch_db.erl"},{line,1450}]}]
[error] 2018-01-18T18:58:25.372933Z [email protected] <0.23503.0> -------- rexi_server: from: [email protected](<0.23501.0>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,256}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,204}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,308}]},{couch_mrview,finish_fold,2,[{file,"src/couch_mrview.erl"},{line,642}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,139}]}]
[error] 2018-01-18T18:58:25.373010Z [email protected] <0.23509.0> -------- rexi_server: from: [email protected](<0.23501.0>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,256}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,204}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,308}]},{couch_mrview,map_fold,3,[{file,"src/couch_mrview.erl"},{line,511}]},{couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,222}]},{couch_db,enum_docs,5,[{file,"src/couch_db.erl"},{line,1450}]},{couch_mrview,all_docs_fold,4,[{file,"src/couch_mrview.erl"},{line,425}]}]

too many open files

[[email protected] tmp]# grep error /var/log/couchdb/couchdb.log | grep "too many open files" | head
[error] 2018-01-18T19:01:17.795630Z [email protected] <0.18880.4> -------- Could not open file ./data/shards/a0000000-bfffffff/system_media.1516301568.couch: too many open files
[error] 2018-01-18T19:01:17.817397Z [email protected] <0.18939.4> -------- Could not open file ./data/.shards/60000000-7fffffff/account/6e/25/3aeb9cd6754a47b0c921dc184ae5.1516301928_design/mrview/69e3a57e087bac7b9667b7602970e56c.view: too many open files
[error] 2018-01-18T19:01:17.817488Z [email protected] <0.18939.4> -------- Failed to open view file './data/.shards/60000000-7fffffff/account/6e/25/3aeb9cd6754a47b0c921dc184ae5.1516301928_design/mrview/69e3a57e087bac7b9667b7602970e56c.view': too many open files
[error] 2018-01-18T19:01:17.819229Z [email protected] <0.18945.4> -------- Could not open file ./data/.shards/60000000-7fffffff/account/6e/25/3aeb9cd6754a47b0c921dc184ae5.1516301928_design/mrview/665c8a92ba7daaf86a989aee308fbbe5.view: too many open files
[error] 2018-01-18T19:01:17.819320Z [email protected] <0.18945.4> -------- Failed to open view file './data/.shards/60000000-7fffffff/account/6e/25/3aeb9cd6754a47b0c921dc184ae5.1516301928_design/mrview/665c8a92ba7daaf86a989aee308fbbe5.view': too many open files
[error] 2018-01-18T19:01:17.820790Z [email protected] <0.18951.4> -------- Could not open file ./data/.shards/60000000-7fffffff/account/6e/25/3aeb9cd6754a47b0c921dc184ae5.1516301928_design/mrview/3f703b0b07ced5a4c9804a63d7a15af9.view: too many open files
[error] 2018-01-18T19:01:17.820882Z [email protected] <0.18951.4> -------- Failed to open view file './data/.shards/60000000-7fffffff/account/6e/25/3aeb9cd6754a47b0c921dc184ae5.1516301928_design/mrview/3f703b0b07ced5a4c9804a63d7a15af9.view': too many open files
[error] 2018-01-18T19:01:17.822353Z [email protected] <0.18957.4> -------- Could not open file ./data/.shards/60000000-7fffffff/account/6e/25/3aeb9cd6754a47b0c921dc184ae5.1516301928_design/mrview/ae24bf2461ecaaa8d6368a29badc5eb1.view: too many open files
[error] 2018-01-18T19:01:17.822429Z [email protected] <0.18957.4> -------- Failed to open view file './data/.shards/60000000-7fffffff/account/6e/25/3aeb9cd6754a47b0c921dc184ae5.1516301928_design/mrview/ae24bf2461ecaaa8d6368a29badc5eb1.view': too many open files
[error] 2018-01-18T19:01:17.823900Z [email protected] <0.18963.4> -------- Could not open file ./data/.shards/60000000-7fffffff/account/6e/25/3aeb9cd6754a47b0c921dc184ae5.1516301928_design/mrview/f836e93763ffb225e3b1079db74acbbe.view: too many open files

I will configure to allow daemon open 999999 files and make test again

sergey-safarov picture sergey-safarov on 19 Jan 2018

I added string 'LimitNOFILE=999999' to /usr/lib/systemd/system/couchdb.service
Now systemd unit file looks like:

[Unit]
Description=Apache CouchDB
Wants=network-online.target
After=network-online.target

[Service]
LimitNOFILE=999999
RuntimeDirectory=couchdb
User=couchdb
Group=couchdb
ExecStart=/opt/couchdb/bin/couchdb
Restart=always

[Install]
WantedBy=multi-user.target

All errors is resolved except one

[[email protected] ~]# grep error /var/log/couchdb/couchdb.log | grep -v "Request to create N=3 DB but only 1 node" | head
[error] 2018-01-19T06:03:27.362521Z [email protected] <0.16326.0> -------- rexi_server: from: [email protected](<0.16317.0>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,256}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,204}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,308}]},{couch_mrview,finish_fold,2,[{file,"src/couch_mrview.erl"},{line,642}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,139}]}]
[error] 2018-01-19T06:03:27.363595Z [email protected] <0.16318.0> -------- rexi_server: from: [email protected](<0.16317.0>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,256}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,204}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,308}]},{couch_mrview,finish_fold,2,[{file,"src/couch_mrview.erl"},{line,642}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,139}]}]
[error] 2018-01-19T06:03:27.363671Z [email protected] <0.16325.0> -------- rexi_server: from: [email protected](<0.16317.0>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,256}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,204}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,308}]},{couch_mrview,finish_fold,2,[{file,"src/couch_mrview.erl"},{line,642}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,139}]}]
[error] 2018-01-19T06:03:27.363895Z [email protected] <0.16324.0> -------- rexi_server: from: [email protected](<0.16317.0>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,256}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,204}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,308}]},{couch_mrview,finish_fold,2,[{file,"src/couch_mrview.erl"},{line,642}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,139}]}]
[error] 2018-01-19T06:03:27.363911Z [email protected] <0.16323.0> -------- rexi_server: from: [email protected](<0.16317.0>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,256}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,204}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,308}]},{couch_mrview,finish_fold,2,[{file,"src/couch_mrview.erl"},{line,642}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,139}]}]
[error] 2018-01-19T06:03:27.363992Z [email protected] <0.16319.0> -------- rexi_server: from: [email protected](<0.16317.0>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,256}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,204}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,308}]},{couch_mrview,map_fold,3,[{file,"src/couch_mrview.erl"},{line,511}]},{couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848}]},{couch_btree,stream_kp_node,8,[{file,"src/couch_btree.erl"},{line,819}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,222}]},{couch_db,enum_docs,5,[{file,"src/couch_db.erl"},{line,1450}]}]
[error] 2018-01-19T06:03:27.364032Z [email protected] <0.16320.0> -------- rexi_server: from: [email protected](<0.16317.0>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,256}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,204}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,308}]},{couch_mrview,finish_fold,2,[{file,"src/couch_mrview.erl"},{line,642}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,139}]}]
[error] 2018-01-19T06:03:27.364488Z [email protected] <0.16322.0> -------- rexi_server: from: [email protected](<0.16317.0>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,256}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,204}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,308}]},{couch_mrview,finish_fold,2,[{file,"src/couch_mrview.erl"},{line,642}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,139}]}]
[error] 2018-01-19T06:04:05.524625Z [email protected] <0.26308.0> -------- rexi_server: from: [email protected](<0.26296.0>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,256}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,204}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,308}]},{couch_mrview,finish_fold,2,[{file,"src/couch_mrview.erl"},{line,642}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,139}]}]
[error] 2018-01-19T06:04:05.524713Z [email protected] <0.26303.0> -------- rexi_server: from: [email protected](<0.26296.0>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,256}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,204}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,308}]},{couch_mrview,map_fold,3,[{file,"src/couch_mrview.erl"},{line,511}]},{couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,222}]},{couch_db,enum_docs,5,[{file,"src/couch_db.erl"},{line,1450}]},{couch_mrview,all_docs_fold,4,[{file,"src/couch_mrview.erl"},{line,425}]}]

About 11316 such errors when replicated 1500 databases (25Gb total size).

[[email protected] ~]# grep error /var/log/couchdb/couchdb.log | grep -v "Request to create N=3 DB but only 1 node" | wc -l
11316

Could you add more logging about what is document is replicated from other server?

sergey-safarov picture sergey-safarov on 19 Jan 2018

I enabled debug logs. Log for one database with error follows

[error] 2018-01-19T10:20:33.050143Z [email protected] <0.13106.3> 90be7ab7ff Request to create N=3 DB but only 1 node(s)
[notice] 2018-01-19T10:20:33.352923Z [email protected] <0.13106.3> 90be7ab7ff 217.12.247.99:5984 217.12.247.99 undefined PUT /account%2f73%2fc2%2f51b213d0f6c919aa2159079eb55d/ 201 ok 305
[notice] 2018-01-19T10:20:33.355705Z [email protected] <0.13106.3> 57ba46b662 217.12.247.99:5984 217.12.247.99 undefined GET /account%2f73%2fc2%2f51b213d0f6c919aa2159079eb55d/ 200 ok 2
[notice] 2018-01-19T10:20:33.361529Z [email protected] <0.13106.3> 065fc7460e 217.12.247.99:5984 217.12.247.99 undefined GET /account%2f73%2fc2%2f51b213d0f6c919aa2159079eb55d/ 200 ok 2
[notice] 2018-01-19T10:20:33.618505Z [email protected] <0.13106.3> edf1a78b7d 217.12.247.99:5984 217.12.247.99 undefined GET /account%2f73%2fc2%2f51b213d0f6c919aa2159079eb55d/_local/5b44abf4a8cd8c74e0dae4c180c323df 404 ok 1
[notice] 2018-01-19T10:20:33.620543Z [email protected] <0.13106.3> 49d4b34503 217.12.247.99:5984 217.12.247.99 undefined GET /account%2f73%2fc2%2f51b213d0f6c919aa2159079eb55d/_local/de43dfe7e561450208e7a38e73707c3c 404 ok 1
[notice] 2018-01-19T10:20:33.622778Z [email protected] <0.13106.3> c8e128ce33 217.12.247.99:5984 217.12.247.99 undefined GET /account%2f73%2fc2%2f51b213d0f6c919aa2159079eb55d/_local/39ded5f138a77eb000f9403a741b3dd1 404 ok 1
[notice] 2018-01-19T10:20:33.624903Z [email protected] <0.13106.3> 8b5ae30bb6 217.12.247.99:5984 217.12.247.99 undefined GET /account%2f73%2fc2%2f51b213d0f6c919aa2159079eb55d/_local/bdfed711151ad361f55cf094fb27bded 404 ok 1
[notice] 2018-01-19T10:20:33.632078Z [email protected] <0.13108.3> b6acd943ec 217.12.247.99:5984 217.12.247.99 undefined POST /account%2f73%2fc2%2f51b213d0f6c919aa2159079eb55d/_revs_diff 200 ok 1
[notice] 2018-01-19T10:20:33.632547Z [email protected] <0.13109.3> 60c1d3c53a 217.12.247.99:5984 217.12.247.99 undefined POST /account%2f73%2fc2%2f51b213d0f6c919aa2159079eb55d/_revs_diff 200 ok 1
[notice] 2018-01-19T10:20:33.632585Z [email protected] <0.13110.3> 5e2d7f23c1 217.12.247.99:5984 217.12.247.99 undefined POST /account%2f73%2fc2%2f51b213d0f6c919aa2159079eb55d/_revs_diff 200 ok 1
[notice] 2018-01-19T10:20:33.651400Z [email protected] <0.13057.3> -------- Starting replication 5b44abf4a8cd8c74e0dae4c180c323df+create_target (http://217.12.247.210:5984/account%2f73%2fc2%2f51b213d0f6c919aa2159079eb55d/ -> http://217.12.247.99:5984/account%2f73%2fc2%2f51b213d0f6c919aa2159079eb55d/) from _replicate endpoint worker_procesess:4 worker_batch_size:500 session_id:a939d07237a0b1517be4477caebdbaa3
[notice] 2018-01-19T10:20:33.670336Z [email protected] <0.13106.3> eb10cda3e2 217.12.247.99:5984 217.12.247.99 undefined POST /account%2f73%2fc2%2f51b213d0f6c919aa2159079eb55d/_revs_diff 200 ok 40
[notice] 2018-01-19T10:20:33.686133Z [email protected] <0.13109.3> 18c598dfa9 217.12.247.99:5984 217.12.247.99 undefined POST /account%2f73%2fc2%2f51b213d0f6c919aa2159079eb55d/_bulk_docs 201 ok 49
[notice] 2018-01-19T10:20:33.702962Z [email protected] <0.13108.3> d8b6f1fe6c 217.12.247.99:5984 217.12.247.99 undefined POST /account%2f73%2fc2%2f51b213d0f6c919aa2159079eb55d/_bulk_docs 201 ok 44
[notice] 2018-01-19T10:20:33.707057Z [email protected] <0.13110.3> 3eb0fdb75d 217.12.247.99:5984 217.12.247.99 undefined POST /account%2f73%2fc2%2f51b213d0f6c919aa2159079eb55d/_bulk_docs 201 ok 47
[notice] 2018-01-19T10:20:33.729765Z [email protected] <0.13109.3> e64350d9dd 217.12.247.99:5984 217.12.247.99 undefined POST /account%2f73%2fc2%2f51b213d0f6c919aa2159079eb55d/_revs_diff 200 ok 42
[notice] 2018-01-19T10:20:33.759715Z [email protected] <0.13106.3> 98af90e7e4 217.12.247.99:5984 217.12.247.99 undefined POST /account%2f73%2fc2%2f51b213d0f6c919aa2159079eb55d/_bulk_docs 201 ok 46
[notice] 2018-01-19T10:20:33.955858Z [email protected] <0.13109.3> 8e0d9e28d9 217.12.247.99:5984 217.12.247.99 undefined POST /account%2f73%2fc2%2f51b213d0f6c919aa2159079eb55d/_bulk_docs 201 ok 106
[notice] 2018-01-19T10:20:33.957526Z [email protected] <0.13109.3> 2ba1a50443 217.12.247.99:5984 217.12.247.99 undefined POST /account%2f73%2fc2%2f51b213d0f6c919aa2159079eb55d/_ensure_full_commit 201 ok 1
[notice] 2018-01-19T10:20:33.957854Z [email protected] <0.13057.3> -------- recording a checkpoint for `http://217.12.247.210:5984/account%2f73%2fc2%2f51b213d0f6c919aa2159079eb55d/` -> `http://217.12.247.99:5984/account%2f73%2fc2%2f51b213d0f6c919aa2159079eb55d/` at source update_seq [67,<<"g1AAAACreJzLYWBgYM5gTmGQT8pMT84vTc5wyMtPSTXSS8lPzk4t0itKLs5LLdErKs0BKmRKZEgKDQ0NyUoUIloLSENoHgvDylWrVmUlyhCtD6gFqGMVkPoPBFmJolkABcY5ew">>]
[notice] 2018-01-19T10:20:34.083245Z [email protected] <0.13109.3> 7726e23a50 217.12.247.99:5984 217.12.247.99 undefined PUT /account%2f73%2fc2%2f51b213d0f6c919aa2159079eb55d/_local/5b44abf4a8cd8c74e0dae4c180c323df 201 ok 41
[notice] 2018-01-19T10:20:34.084075Z [email protected] <0.396.0> -------- couch_replicator_scheduler: Job {"5b44abf4a8cd8c74e0dae4c180c323df","+create_target"} completed normally
[notice] 2018-01-19T10:20:34.084514Z [email protected] <0.12894.3> 2876ab6c9a 217.12.247.99:5984 217.12.247.99 undefined POST /_replicate 200 ok 5233
[notice] 2018-01-19T10:20:34.151996Z [email protected] <0.396.0> -------- couch_replicator_scheduler: Job {"cb9ca1f8d9afafcc4c96917b3e547561","+create_target"} started as <0.13869.3>
[error] 2018-01-19T10:20:34.354523Z [email protected] <0.23314.0> -------- rexi_server: from: [email protected](<0.23308.0>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,256}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,204}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,308}]},{couch_mrview,map_fold,3,[{file,"src/couch_mrview.erl"},{line,511}]},{couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,222}]},{couch_db,enum_docs,5,[{file,"src/couch_db.erl"},{line,1450}]},{couch_mrview,all_docs_fold,4,[{file,"src/couch_mrview.erl"},{line,425}]}]
[error] 2018-01-19T10:20:34.354598Z [email protected] <0.23309.0> -------- rexi_server: from: [email protected](<0.23308.0>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,256}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,204}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,308}]},{couch_mrview,finish_fold,2,[{file,"src/couch_mrview.erl"},{line,642}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,139}]}]
[error] 2018-01-19T10:20:34.354637Z [email protected] <0.23326.0> -------- rexi_server: from: [email protected](<0.23308.0>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,256}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,204}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,308}]},{couch_mrview,finish_fold,2,[{file,"src/couch_mrview.erl"},{line,642}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,139}]}]
[error] 2018-01-19T10:20:34.355452Z [email protected] <0.23323.0> -------- rexi_server: from: [email protected](<0.23308.0>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,256}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,204}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,308}]},{couch_mrview,map_fold,3,[{file,"src/couch_mrview.erl"},{line,511}]},{couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,222}]},{couch_db,enum_docs,5,[{file,"src/couch_db.erl"},{line,1450}]},{couch_mrview,all_docs_fold,4,[{file,"src/couch_mrview.erl"},{line,425}]}]
[error] 2018-01-19T10:20:34.355480Z [email protected] <0.23318.0> -------- rexi_server: from: [email protected](<0.23308.0>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,256}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,204}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,308}]},{couch_mrview,map_fold,3,[{file,"src/couch_mrview.erl"},{line,511}]},{couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,222}]},{couch_db,enum_docs,5,[{file,"src/couch_db.erl"},{line,1450}]},{couch_mrview,all_docs_fold,4,[{file,"src/couch_mrview.erl"},{line,425}]}]
[error] 2018-01-19T10:20:34.355520Z [email protected] <0.23319.0> -------- rexi_server: from: [email protected](<0.23308.0>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,256}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,204}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,308}]},{couch_mrview,map_fold,3,[{file,"src/couch_mrview.erl"},{line,511}]},{couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,222}]},{couch_db,enum_docs,5,[{file,"src/couch_db.erl"},{line,1450}]},{couch_mrview,all_docs_fold,4,[{file,"src/couch_mrview.erl"},{line,425}]}]
[error] 2018-01-19T10:20:34.355634Z [email protected] <0.23325.0> -------- rexi_server: from: [email protected](<0.23308.0>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,256}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,204}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,308}]},{couch_mrview,map_fold,3,[{file,"src/couch_mrview.erl"},{line,511}]},{couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,222}]},{couch_db,enum_docs,5,[{file,"src/couch_db.erl"},{line,1450}]},{couch_mrview,all_docs_fold,4,[{file,"src/couch_mrview.erl"},{line,425}]}]
[error] 2018-01-19T10:20:34.355927Z [email protected] <0.23311.0> -------- rexi_server: from: [email protected](<0.23308.0>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,256}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,204}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,308}]},{couch_mrview,map_fold,3,[{file,"src/couch_mrview.erl"},{line,511}]},{couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,222}]},{couch_db,enum_docs,5,[{file,"src/couch_db.erl"},{line,1450}]},{couch_mrview,all_docs_fold,4,[{file,"src/couch_mrview.erl"},{line,425}]}]
[error] 2018-01-19T10:20:37.859158Z [email protected] <0.13109.3> ec41f07f3d Request to create N=3 DB but only 1 node(s)

I will add 30 sec delay between starting next database replication.

sergey-safarov picture sergey-safarov on 19 Jan 2018

This log when exist 30 sec delay between database replication

[notice] 2018-01-19T11:22:37.879643Z [email protected] <0.398.0> -------- couch_replicator_scheduler: Job {"f8021a32ba235a68302e165b8594fc3f","+create_target"} started as <0.14235.1>
[error] 2018-01-19T11:22:41.277632Z [email protected] <0.12984.1> 3cb83ef90b Request to create N=3 DB but only 1 node(s)
[notice] 2018-01-19T11:22:41.671914Z [email protected] <0.12984.1> 3cb83ef90b 217.12.247.99:5984 217.12.247.99 undefined PUT /account%2f0d%2fbb%2ffb1874be1b7360713c26588e9b82/ 201 ok 395
[notice] 2018-01-19T11:22:41.674402Z [email protected] <0.12984.1> 31fe3520ef 217.12.247.99:5984 217.12.247.99 undefined GET /account%2f0d%2fbb%2ffb1874be1b7360713c26588e9b82/ 200 ok 2
[notice] 2018-01-19T11:22:41.678557Z [email protected] <0.12984.1> fa64a0f610 217.12.247.99:5984 217.12.247.99 undefined GET /account%2f0d%2fbb%2ffb1874be1b7360713c26588e9b82/ 200 ok 1
[notice] 2018-01-19T11:22:41.847889Z [email protected] <0.12984.1> 93855d1e2a 217.12.247.99:5984 217.12.247.99 undefined GET /account%2f0d%2fbb%2ffb1874be1b7360713c26588e9b82/_local/f8021a32ba235a68302e165b8594fc3f 404 ok 1
[notice] 2018-01-19T11:22:41.849746Z [email protected] <0.12984.1> 23948381f8 217.12.247.99:5984 217.12.247.99 undefined GET /account%2f0d%2fbb%2ffb1874be1b7360713c26588e9b82/_local/322f6d0122cdfe6f2bc50a7a1d517cb4 404 ok 1
[notice] 2018-01-19T11:22:41.851548Z [email protected] <0.12984.1> 733f302e03 217.12.247.99:5984 217.12.247.99 undefined GET /account%2f0d%2fbb%2ffb1874be1b7360713c26588e9b82/_local/fb2de1a5c13bb1b964f91e00608e9548 404 ok 1
[notice] 2018-01-19T11:22:41.853539Z [email protected] <0.12984.1> 700eb93122 217.12.247.99:5984 217.12.247.99 undefined GET /account%2f0d%2fbb%2ffb1874be1b7360713c26588e9b82/_local/30124e4fbfcc51a01ee063fdd33c472e 404 ok 1
[notice] 2018-01-19T11:22:41.856901Z [email protected] <0.14235.1> -------- Starting replication f8021a32ba235a68302e165b8594fc3f+create_target (http://217.12.247.210:5984/account%2f0d%2fbb%2ffb1874be1b7360713c26588e9b82/ -> http://217.12.247.99:5984/account%2f0d%2fbb%2ffb1874be1b7360713c26588e9b82/) from _replicate endpoint worker_procesess:4 worker_batch_size:500 session_id:89fc5fc8dabfbb2c6e65851a59286bb4
[notice] 2018-01-19T11:22:41.898276Z [email protected] <0.12984.1> febe2351dc 217.12.247.99:5984 217.12.247.99 undefined POST /account%2f0d%2fbb%2ffb1874be1b7360713c26588e9b82/_revs_diff 200 ok 40
[notice] 2018-01-19T11:22:41.898332Z [email protected] <0.12974.1> 63c0b3ba2a 217.12.247.99:5984 217.12.247.99 undefined POST /account%2f0d%2fbb%2ffb1874be1b7360713c26588e9b82/_revs_diff 200 ok 39
[notice] 2018-01-19T11:22:41.898361Z [email protected] <0.12982.1> f930be65d3 217.12.247.99:5984 217.12.247.99 undefined POST /account%2f0d%2fbb%2ffb1874be1b7360713c26588e9b82/_revs_diff 200 ok 40
[notice] 2018-01-19T11:22:41.898478Z [email protected] <0.12983.1> 9b9f9c80da 217.12.247.99:5984 217.12.247.99 undefined POST /account%2f0d%2fbb%2ffb1874be1b7360713c26588e9b82/_revs_diff 200 ok 40
[notice] 2018-01-19T11:22:41.946928Z [email protected] <0.12984.1> 7620d7df1b 217.12.247.99:5984 217.12.247.99 undefined POST /account%2f0d%2fbb%2ffb1874be1b7360713c26588e9b82/_bulk_docs 201 ok 44
[notice] 2018-01-19T11:22:41.947011Z [email protected] <0.12983.1> e05ecc5c19 217.12.247.99:5984 217.12.247.99 undefined POST /account%2f0d%2fbb%2ffb1874be1b7360713c26588e9b82/_bulk_docs 201 ok 43
[notice] 2018-01-19T11:22:41.948407Z [email protected] <0.12982.1> 340faf367a 217.12.247.99:5984 217.12.247.99 undefined POST /account%2f0d%2fbb%2ffb1874be1b7360713c26588e9b82/_bulk_docs 201 ok 45
[notice] 2018-01-19T11:22:41.953472Z [email protected] <0.12974.1> 07ad9e1124 217.12.247.99:5984 217.12.247.99 undefined POST /account%2f0d%2fbb%2ffb1874be1b7360713c26588e9b82/_bulk_docs 201 ok 49
[notice] 2018-01-19T11:22:41.990657Z [email protected] <0.12984.1> 1c618702f8 217.12.247.99:5984 217.12.247.99 undefined POST /account%2f0d%2fbb%2ffb1874be1b7360713c26588e9b82/_revs_diff 200 ok 42
[notice] 2018-01-19T11:22:42.215544Z [email protected] <0.12984.1> b572660a41 217.12.247.99:5984 217.12.247.99 undefined POST /account%2f0d%2fbb%2ffb1874be1b7360713c26588e9b82/_bulk_docs 201 ok 100
[notice] 2018-01-19T11:22:42.217546Z [email protected] <0.12984.1> 48806332e3 217.12.247.99:5984 217.12.247.99 undefined POST /account%2f0d%2fbb%2ffb1874be1b7360713c26588e9b82/_ensure_full_commit 201 ok 1
[notice] 2018-01-19T11:22:42.217908Z [email protected] <0.14235.1> -------- recording a checkpoint for `http://217.12.247.210:5984/account%2f0d%2fbb%2ffb1874be1b7360713c26588e9b82/` -> `http://217.12.247.99:5984/account%2f0d%2fbb%2ffb1874be1b7360713c26588e9b82/` at source update_seq [74,<<"g1AAAACreJzLYWBgYM5gTmGQT8pMT84vTc5wyMtPSTXSS8lPzk4t0itKLs5LLdErKs0BKmRKZEgKDQ0NyUqUIFoLSENoHgvDylWrVmUlyhOtD6gFqGMVkPoPBFmJwlkACS45gg">>]
[notice] 2018-01-19T11:22:42.342623Z [email protected] <0.12984.1> 5eba670cad 217.12.247.99:5984 217.12.247.99 undefined PUT /account%2f0d%2fbb%2ffb1874be1b7360713c26588e9b82/_local/f8021a32ba235a68302e165b8594fc3f 201 ok 41
[notice] 2018-01-19T11:22:42.343058Z [email protected] <0.398.0> -------- couch_replicator_scheduler: Job {"f8021a32ba235a68302e165b8594fc3f","+create_target"} completed normally
[notice] 2018-01-19T11:22:42.343341Z [email protected] <0.14217.1> 9354efdd10 217.12.247.99:5984 217.12.247.99 undefined POST /_replicate 200 ok 4466
[error] 2018-01-19T11:23:09.315208Z [email protected] <0.2741.1> -------- rexi_server: from: [email protected](<0.2737.1>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,256}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,204}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,308}]},{couch_mrview,finish_fold,2,[{file,"src/couch_mrview.erl"},{line,642}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,139}]}]
[error] 2018-01-19T11:23:09.315224Z [email protected] <0.2742.1> -------- rexi_server: from: [email protected](<0.2737.1>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,256}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,204}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,308}]},{couch_mrview,finish_fold,2,[{file,"src/couch_mrview.erl"},{line,642}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,139}]}]
[error] 2018-01-19T11:23:09.315259Z [email protected] <0.2740.1> -------- rexi_server: from: [email protected](<0.2737.1>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,256}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,204}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,308}]},{couch_mrview,map_fold,3,[{file,"src/couch_mrview.erl"},{line,511}]},{couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,222}]},{couch_db,enum_docs,5,[{file,"src/couch_db.erl"},{line,1450}]},{couch_mrview,all_docs_fold,4,[{file,"src/couch_mrview.erl"},{line,425}]}]
[error] 2018-01-19T11:23:09.316323Z [email protected] <0.2745.1> -------- rexi_server: from: [email protected](<0.2737.1>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,256}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,204}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,308}]},{couch_mrview,map_fold,3,[{file,"src/couch_mrview.erl"},{line,511}]},{couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,222}]},{couch_db,enum_docs,5,[{file,"src/couch_db.erl"},{line,1450}]},{couch_mrview,all_docs_fold,4,[{file,"src/couch_mrview.erl"},{line,425}]}]
[error] 2018-01-19T11:23:09.316374Z [email protected] <0.2746.1> -------- rexi_server: from: [email protected](<0.2737.1>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,256}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,204}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,308}]},{couch_mrview,map_fold,3,[{file,"src/couch_mrview.erl"},{line,511}]},{couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,222}]},{couch_db,enum_docs,5,[{file,"src/couch_db.erl"},{line,1450}]},{couch_mrview,all_docs_fold,4,[{file,"src/couch_mrview.erl"},{line,425}]}]
[error] 2018-01-19T11:23:09.316501Z [email protected] <0.2744.1> -------- rexi_server: from: [email protected](<0.2737.1>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,256}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,204}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,308}]},{couch_mrview,finish_fold,2,[{file,"src/couch_mrview.erl"},{line,642}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,139}]}]
[error] 2018-01-19T11:23:09.316625Z [email protected] <0.2738.1> -------- rexi_server: from: [email protected](<0.2737.1>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,256}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,204}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,308}]},{couch_mrview,map_fold,3,[{file,"src/couch_mrview.erl"},{line,511}]},{couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,222}]},{couch_db,enum_docs,5,[{file,"src/couch_db.erl"},{line,1450}]},{couch_mrview,all_docs_fold,4,[{file,"src/couch_mrview.erl"},{line,425}]}]
[error] 2018-01-19T11:23:09.317073Z [email protected] <0.2739.1> -------- rexi_server: from: [email protected](<0.2737.1>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,256}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,204}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,308}]},{couch_mrview,map_fold,3,[{file,"src/couch_mrview.erl"},{line,511}]},{couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,222}]},{couch_db,enum_docs,5,[{file,"src/couch_db.erl"},{line,1450}]},{couch_mrview,all_docs_fold,4,[{file,"src/couch_mrview.erl"},{line,425}]}]
[notice] 2018-01-19T11:23:12.394370Z [email protected] <0.398.0> -------- couch_replicator_scheduler: Job {"9bcd18316a671dbd7367c932e7083abc","+create_target"} started as <0.15559.1>
[error] 2018-01-19T11:23:12.465643Z [email protected] <0.12984.1> 9101974cf8 Request to create N=3 DB but only 1 node(s)
[notice] 2018-01-19T11:23:12.786568Z [email protected] <0.12984.1> 9101974cf8 217.12.247.99:5984 217.12.247.99 undefined PUT /account%2f0d%2fc1%2fc97c58fd658391cc05fdc219b0d7/ 201 ok 322
[notice] 2018-01-19T11:23:12.788976Z [email protected] <0.12984.1> 598275b62f 217.12.247.99:5984 217.12.247.99 undefined GET /account%2f0d%2fc1%2fc97c58fd658391cc05fdc219b0d7/ 200 ok 2
[notice] 2018-01-19T11:23:12.793112Z [email protected] <0.12984.1> 546fa3d0da 217.12.247.99:5984 217.12.247.99 undefined GET /account%2f0d%2fc1%2fc97c58fd658391cc05fdc219b0d7/ 200 ok 1
[notice] 2018-01-19T11:23:12.963100Z [email protected] <0.12984.1> 0aabd63bd6 217.12.247.99:5984 217.12.247.99 undefined GET /account%2f0d%2fc1%2fc97c58fd658391cc05fdc219b0d7/_local/9bcd18316a671dbd7367c932e7083abc 404 ok 1

Please look at message timestamp. From 2018-01-19T11:22:37.879643Z to 2018-01-19T11:23:09.317073Z one replication.
After 2018-01-19T11:23:12.394370Z next

Replication is finished at 2018-01-19T11:22:42.343058Z then executed sleep 30 command. At this sleep time couchdb is idle. and we got error messages.
At 2018-01-19T11:23:12.394370Z sleep command is finished adn started next replication.

sergey-safarov picture sergey-safarov on 19 Jan 2018

Some of these errors are expected - couch is noisy in logging.

Are you actually experiencing problems with replication or is this a request for our logging to be less noisy?

wohali picture wohali on 19 Jan 2018

@wohali Hello Joan
rexi_server messages have severity level error and I expects that replication is completed with errors.
How i can check integrity of couchdb server?
May be set severity level for rexi_server messages to notice?

sergey-safarov picture sergey-safarov on 20 Jan 2018

No, it's a valid error, but it doesn't mean that replication completed with errors. It's one that CouchDB recovers from on its own. It is possible something else is wrong with your setup, of course, but that doesn't mean replication failed on its own.

You can determine if the replication completed successfully via the standard API, not our logfile. I encourage use of the /_replicator database.

More information is available here: http://docs.couchdb.org/en/latest/replication/replicator.html#replication-states

I'm closing this for now. If you can show that you are actually having failed replication, or 500 errors via the HTTP API, not just errors in the logfile, then please reopen.

wohali picture wohali on 20 Jan 2018

@wohali I'm also experiencing this remaining rexi issue. I'm not using the replicator but a custom tool which copies batches of 100 documents from a source.

When the below rexi error occurs, the batch fails to provide the copy tool with the needed 201 OK which allows the next batch of 100 documents to be sent. Without the affirmative reply, my document copy tool fails to progress and it's position is lost. 

[error] 2018-04-12T14:32:14.885843Z couchdb@hostname-removed <0.20568.192> -------- rexi_server: from: couchdb@hostname-removed(<0.28033.198>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,256}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,204}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,308}]},{couch_mrview,finish_fold,2,[{file,"src/couch_mrview.erl"},{line,642}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,139}]}]
[error] 2018-04-12T14:32:14.886158Z couchdb@hostname-removed <0.2978.199> -------- rexi_server: from: couchdb@hostname-removed(<0.28033.198>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,256}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,204}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,308}]},{couch_mrview,map_fold,3,[{file,"src/couch_mrview.erl"},{line,511}]},{couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848}]},{couch_btree,stream_kp_node,8,[{file,"src/couch_btree.erl"},{line,819}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,222}]},{couch_db,enum_docs,5,[{file,"src/couch_db.erl"},{line,1450}]}]
[error] 2018-04-12T14:32:14.886183Z couchdb@hostname-removed <0.3631.188> -------- rexi_server: from: couchdb@hostname-removed(<0.28033.198>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,256}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,204}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,308}]},{couch_mrview,finish_fold,2,[{file,"src/couch_mrview.erl"},{line,642}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,139}]}]
[error] 2018-04-12T14:32:14.886279Z couchdb@hostname-removed <0.22947.194> -------- rexi_server: from: couchdb@hostname-removed(<0.28033.198>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,256}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,204}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,308}]},{couch_mrview,finish_fold,2,[{file,"src/couch_mrview.erl"},{line,642}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,139}]}]
[error] 2018-04-12T14:32:14.886336Z couchdb@hostname-removed <0.31484.202> -------- rexi_server: from: couchdb@hostname-removed(<0.28033.198>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,256}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,204}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,308}]},{couch_mrview,map_fold,3,[{file,"src/couch_mrview.erl"},{line,511}]},{couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848}]},{couch_btree,stream_kp_node,8,[{file,"src/couch_btree.erl"},{line,819}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,222}]},{couch_db,enum_docs,5,[{file,"src/couch_db.erl"},{line,1450}]}]
[error] 2018-04-12T14:32:14.886809Z couchdb@hostname-removed <0.30230.195> -------- rexi_server: from: couchdb@hostname-removed(<0.28033.198>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,256}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,204}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,308}]},{couch_mrview,map_fold,3,[{file,"src/couch_mrview.erl"},{line,511}]},{couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848}]},{couch_btree,stream_kp_node,8,[{file,"src/couch_btree.erl"},{line,819}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,222}]},{couch_db,enum_docs,5,[{file,"src/couch_db.erl"},{line,1450}]}]
[error] 2018-04-12T14:32:14.886892Z couchdb@hostname-removed <0.17694.179> -------- rexi_server: from: couchdb@hostname-removed(<0.28033.198>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,256}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,204}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,308}]},{couch_mrview,map_fold,3,[{file,"src/couch_mrview.erl"},{line,511}]},{couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848}]},{couch_btree,stream_kp_node,8,[{file,"src/couch_btree.erl"},{line,819}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,222}]},{couch_db,enum_docs,5,[{file,"src/couch_db.erl"},{line,1450}]}]
[error] 2018-04-12T14:32:14.887968Z couchdb@hostname-removed <0.20630.199> -------- rexi_server: from: couchdb@hostname-removed(<0.28033.198>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,256}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,204}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,308}]},{couch_mrview,map_fold,3,[{file,"src/couch_mrview.erl"},{line,511}]},{couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848}]},{couch_btree,stream_kp_node,8,[{file,"src/couch_btree.erl"},{line,819}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,222}]},{couch_db,enum_docs,5,[{file,"src/couch_db.erl"},{line,1450}]}]
grahamsnz picture grahamsnz on 12 Apr 2018

@grahamsnz Can you please file a new issue? Thanks. You can reference this one in that.

wohali picture wohali on 12 Apr 2018
Was this page helpful?
0 / 5 - 0 ratings

Related issues

maciozo picture maciozo  路  5Comments
mojito317 picture mojito317  路  3Comments
ondra-novak picture ondra-novak  路  4Comments
mtabb13 picture mtabb13  路  5Comments
andrasbacsai picture andrasbacsai  路  5Comments