Couchdb: Very slow replication for databases with attachments

Created on 24 Jan 2018  路  11Comments  路  Source: apache/couchdb

Expected Behavior

Between bigcouch host replication take not more 3-4 hours for 1500 databases (25Gb total size).
I expects replication from couchdb 2.0 host to bigcouch will be similar

Current Behavior

Some databases cannot be replicated for several hours.

Possible Solution

Not knows

Steps to Reproduce (for bugs)

  1. Create 1500 databases about 25Gb total size on couchdb 2.0 hosts;
  2. Start replication to bigcouch host from couchdb, curl command is send to couchdb host;
  3. Check Active Tasks screen on coucudb host.

Context

I want replicate database to other host.

Your Environment

  • Version used: 2.1.1
  • Browser Name and version: Google Chrome Version 63.0.3239.132 (Official Build) (64-bit)
  • Operating System and version (desktop or mobile): CentOS 7 on bare metal server.
  • Link to your project: none
bug waiting on user
Source
picture sergey-safarov
👍1

All 11 comments

Example of _active_tasks output (formated)

[  
   {  
      "node":"[email protected]",
      "pid":"<0.23837.50>",
      "changes_pending":76072,
      "checkpoint_interval":30000,
      "checkpointed_source_seq":"1-g1AAAAF1eJzLYWBg4MhgTmEQTM4vTc5ISXIwNDLXMwBCwxygFFMiQ5L8____sxIZ8ChKUgCSSfaE1TmA1MWD1GUwJzLmAnnsiWmmhhZpydj04DMpAWRSPUEb81iAJEMDkAIqnU-M2gUQtfuJUXsAovY-MWofQNSC3JsFAGwAZt4",
      "continuous":false,
      "database":null,
      "doc_id":null,
      "doc_write_failures":0,
      "docs_read":27,
      "docs_written":27,
      "missing_revisions_found":27,
      "replication_id":"83de18e815b496003e016afaca85d7bc+create_target",
      "revisions_checked":27,
      "source":"http://217.12.247.99:5984/account%2f63%2ff8%2f14795fd5b8c74aa04ee865c52f73-201712/",
      "source_seq":"527-g1AAAAJjeJyd0L0KwjAQAOBgBV19AFFfQJK2adrJOjmIk7s2f6WUqpOzvom-ib6Jvkm9JB2L0BK4wB33cXcVQmhceApN5OUqCyVS4rMlhkcqKA04ErO6rsvC4-sTJEYS6yCmrK3hDyPmEMWqkQ5W8jVnWJGuUmqkYyMtrMRzSuJcdpUyI90aaWolnBCpVdftzkOI6A4fYA-jbawmIhUKzXtpT6e9jLa3WsIopoHqpb2d9jHazt1MRpFUfi_t6zR7t62bLQ5ZjltnK3-ySqGL",
      "started_on":1516774642,
      "target":"http://217.12.247.98:5984/account%2f63%2ff8%2f14795fd5b8c74aa04ee865c52f73-201712/",
      "through_seq":"1-g1AAAAF1eJzLYWBg4MhgTmEQTM4vTc5ISXIwNDLXMwBCwxygFFMiQ5L8____sxIZ8ChKUgCSSfaE1TmA1MWD1GUwJzLmAnnsiWmmhhZpydj04DMpAWRSPUEb81iAJEMDkAIqnU-M2gUQtfuJUXsAovY-MWofQNSC3JsFAGwAZt4",
      "type":"replication",
      "updated_on":1516776894,
      "user":null
   },
   {  
      "node":"[email protected]",
      "pid":"<0.2536.290>",
      "changes_pending":null,
      "checkpoint_interval":30000,
      "checkpointed_source_seq":0,
      "continuous":false,
      "database":null,
      "doc_id":null,
      "doc_write_failures":0,
      "docs_read":4,
      "docs_written":4,
      "missing_revisions_found":4,
      "replication_id":"880f88e653a7900c1e7c43461eff471d+create_target",
      "revisions_checked":4,
      "source":"http://217.12.247.99:5984/account%2fa4%2f87%2f97afcbf4c1782d55613a5f86b0bf-201712/",
      "source_seq":"4-g1AAAAHbeJzLYWBg4MhgTmEQTM4vTc5ISXIwNDLXMwBCwxygFFMiQ5L8____sxIZ8ChKUgCSSfYgdRnMiYy5QB67RXJKinGKITY9-ExyAJkUj2KSobFpslGqGakmJYBMqifo9jwWIMnQAKSASucTo3YBRO1-hAvTzJMtTY2SSHQhxLQDENPuI0xLTTZLM0gjNeQgpj2AmAbycxYAxMp_Nw",
      "started_on":1516776088,
      "target":"http://217.12.247.98:5984/account%2fa4%2f87%2f97afcbf4c1782d55613a5f86b0bf-201712/",
      "through_seq":"4-g1AAAAHbeJzLYWBg4MhgTmEQTM4vTc5ISXIwNDLXMwBCwxygFFMiQ5L8____sxIZ8ChKUgCSSfYgdRnMiYy5QB67RXJKinGKITY9-ExyAJkUj2KSobFpslGqGakmJYBMqifo9jwWIMnQAKSASucTo3YBRO1-hAvTzJMtTY2SSHQhxLQDENPuI0xLTTZLM0gjNeQgpj2AmAbycxYAxMp_Nw",
      "type":"replication",
      "updated_on":1516776089,
      "user":null
   },
   {  
      "node":"[email protected]",
      "pid":"<0.29148.294>",
      "changes_pending":null,
      "checkpoint_interval":30000,
      "checkpointed_source_seq":0,
      "continuous":false,
      "database":null,
      "doc_id":null,
      "doc_write_failures":0,
      "docs_read":4,
      "docs_written":4,
      "missing_revisions_found":4,
      "replication_id":"1aebc296b560ed25c8153cda8f55069e+create_target",
      "revisions_checked":4,
      "source":"http://217.12.247.99:5984/account%2fa4%2f87%2f97afcbf4c1782d55613a5f86b0bf-201710/",
      "source_seq":"4-g1AAAAG5eJzLYWBg4MhgTmEQTM4vTc5ISXIwNDLXMwBCwxygFFMiQ5L8____sxIZ8ChKUgCSSfYgdRnMiUy5QB67pYGhYWKiGTY9-ExyAJkUT9jGBJC6eqiNjGAbzSwtUw1SDEm0MY8FSDI0ACmgYfPx2wpRuwCidj_CZsMkI0NLM1OybD4AMe0-MTY_gKgFhU0WAEpFdiQ",
      "started_on":1516776027,
      "target":"http://217.12.247.98:5984/account%2fa4%2f87%2f97afcbf4c1782d55613a5f86b0bf-201710/",
      "through_seq":"1-g1AAAAF1eJzLYWBg4MhgTmEQTM4vTc5ISXIwNDLXMwBCwxygFFMiQ5L8____sxIZ8ChKUgCSSfYgdRnMiYy5QB67pYGhYWKiGTY9-ExyAJkUT9jGBJC6eoLq8liAJEMDkAIqnU-M2gUQtfuJUXsAovY-MWofQNSC3JsFAMMyZnM",
      "type":"replication",
      "updated_on":1516776028,
      "user":null
   },
   {  
      "node":"[email protected]",
      "pid":"<0.31083.294>",
      "changes_pending":null,
      "checkpoint_interval":30000,
      "checkpointed_source_seq":0,
      "continuous":false,
      "database":null,
      "doc_id":null,
      "doc_write_failures":0,
      "docs_read":4,
      "docs_written":4,
      "missing_revisions_found":4,
      "replication_id":"c125005084bccdfaffc089a2d25cdd43+create_target",
      "revisions_checked":4,
      "source":"http://217.12.247.99:5984/account%2fa4%2f87%2f97afcbf4c1782d55613a5f86b0bf-201711/",
      "source_seq":"4-g1AAAAGXeJzLYWBg4MhgTmEQTM4vTc5ISXIwNDLXMwBCwxygFFMiQ5L8____szKYE5lygQLsSSmWJoYWKdg04DEmSQFIJtmjmGRoamCSaG5AqkkOIJPiQSYlMuBTlwBSV09QXR4LkGRoAFJApfOJUbsAonY_MWoPQNTeJ0btA4hakHuzABv8bmY",
      "started_on":1516776055,
      "target":"http://217.12.247.98:5984/account%2fa4%2f87%2f97afcbf4c1782d55613a5f86b0bf-201711/",
      "through_seq":"1-g1AAAAF1eJzLYWBg4MhgTmEQTM4vTc5ISXIwNDLXMwBCwxygFFMiQ5L8____sxIZ8ChKUgCSSfYgdRnMiYy5QB67oamBSaK5ATY9-ExyAJkUT9jGBJC6eoLq8liAJEMDkAIqnU-M2gUQtfuJUXsAovY-MWofQNSC3JsFAI3AZkI",
      "type":"replication",
      "updated_on":1516776057,
      "user":null
   },
   {  
      "node":"[email protected]",
      "pid":"<0.32724.392>",
      "changes_pending":null,
      "checkpoint_interval":30000,
      "checkpointed_source_seq":0,
      "continuous":false,
      "database":null,
      "doc_id":null,
      "doc_write_failures":0,
      "docs_read":0,
      "docs_written":0,
      "missing_revisions_found":0,
      "replication_id":"7a796b343b45bb41d6cb9a76e6f9e675+create_target",
      "revisions_checked":0,
      "source":"http://217.12.247.99:5984/account%2fc3%2f43%2f646ab95638f945c8ec895263cf2e-201707/",
      "source_seq":0,
      "started_on":1516776898,
      "target":"http://217.12.247.98:5984/account%2fc3%2f43%2f646ab95638f945c8ec895263cf2e-201707/",
      "type":"replication",
      "updated_on":1516776898,
      "user":null
   },
   {  
      "node":"[email protected]",
      "pid":"<0.7273.48>",
      "changes_pending":null,
      "checkpoint_interval":30000,
      "checkpointed_source_seq":0,
      "continuous":false,
      "database":null,
      "doc_id":null,
      "doc_write_failures":0,
      "docs_read":8,
      "docs_written":8,
      "missing_revisions_found":8,
      "replication_id":"260c79c446f46814cdf361b5d02b59ce+create_target",
      "revisions_checked":8,
      "source":"http://217.12.247.99:5984/account%2f63%2ff8%2f14795fd5b8c74aa04ee865c52f73-201711/",
      "source_seq":"1041-g1AAAAJjeJyd0LsNwjAQBmCLIEHLBDABivOyXZEdGABsn60oBKhoKICahhFgE1gEwSbBj5QRUqKT_pPudF9xFUJoXASAJnJ_kAWIHEdkHprClVkNOBLTuq7LIuBqawYjpkKuCGs7-MOImUmxaKSTkwSVOo1oVym30qqRjk4CAlpy6CqtrXRupKuXFERSdJV2Q5PoYprB7la7OY2zBHSGe2kPrz2t9vb_yqiMdT_t5bWP1ZZOIxR4Eqte2tdr7m8bp2nMaJSKtrvyB9O6pI8",
      "started_on":1516774615,
      "target":"http://217.12.247.98:5984/account%2f63%2ff8%2f14795fd5b8c74aa04ee865c52f73-201711/",
      "through_seq":"3-g1AAAAGXeJzLYWBg4MhgTmEQTM4vTc5ISXIwNDLXMwBCwxygFFMiQ5L8____sxIZ8ChKUgCSSfaE1TmA1MUTVpcAUlcPUpfBnMiYC-Sxp6SmGCUnpWDTg8ekPBYgydAApICGzcdvK0TtAoja_cSoPQBRex_kSiawK80tUhJNjFPJcuUDiGmgsMkCAN8Hb1s",
      "type":"replication",
      "updated_on":1516774616,
      "user":null
   }
]

Replication to database account%2f63%2ff8%2f14795fd5b8c74aa04ee865c52f73-201711 is now acrive about 1 hour, but processes only 8 doc.

sergey-safarov picture sergey-safarov on 24 Jan 2018

After two hours

[  
   {  
      "node":"[email protected]",
      "pid":"<0.23837.50>",
      "changes_pending":76072,
      "checkpoint_interval":30000,
      "checkpointed_source_seq":"1-g1AAAAF1eJzLYWBg4MhgTmEQTM4vTc5ISXIwNDLXMwBCwxygFFMiQ5L8____sxIZ8ChKUgCSSfaE1TmA1MWD1GUwJzLmAnnsiWmmhhZpydj04DMpAWRSPUEb81iAJEMDkAIqnU-M2gUQtfuJUXsAovY-MWofQNSC3JsFAGwAZt4",
      "continuous":false,
      "database":null,
      "doc_id":null,
      "doc_write_failures":0,
      "docs_read":878,
      "docs_written":878,
      "missing_revisions_found":878,
      "replication_id":"83de18e815b496003e016afaca85d7bc+create_target",
      "revisions_checked":878,
      "source":"http://217.12.247.99:5984/account%2f63%2ff8%2f14795fd5b8c74aa04ee865c52f73-201712/",
      "source_seq":"878-g1AAAAJjeJzLYWBg4MhgTmEQTM4vTc5ISXIwNDLXMwBCwxygFFMiQ5L8____szKYE3NzgQLsyQapxham5tg04DEmSQFIJtlDTeoEm2SUmmhukGJIqkkOIJPioSb5g01KTDM1tEhLJtWkBJBJ9VCTjMEmGVgaJqemkOq7PBYgydAApICGzQeZVgk2LcksxSQpNZEs0xZATNsPMq0VbJqluamBqXEKWaYdgJh2H2RaPSTMks3MklOMyDLtAcS0_wifWlqYmKcZYHVbFgBFpqLq",
      "started_on":1516774642,
      "target":"http://217.12.247.98:5984/account%2f63%2ff8%2f14795fd5b8c74aa04ee865c52f73-201712/",
      "through_seq":"504-g1AAAAJjeJyd0EsKwjAQANBgBd16AFEvIEnbNC0IFlyqa5c2yaSUUnXlWm-iN9Gb6E1qPl0WoSUwgRnmMTMVQmhceIAm8nKVBYiU-GyJ9SOVLg04ErO6rsvC46uTTowkVkFMWVvDH0bMdRTrRjpYyVecYSBdpdRIx0ZaWInnlMS57CplRro10tRKOCFSQdftzkMd0V1_GnsYbWM1EUEoFO-lPZ32MtrOagmjmAbQS3s77WO0vbuZjCIJfi_t6zR7t62bLQ5ZjltnK3-QTqF0",
      "type":"replication",
      "updated_on":1516781845,
      "user":null
   },
   {  
      "node":"[email protected]",
      "pid":"<0.29148.294>",
      "changes_pending":null,
      "checkpoint_interval":30000,
      "checkpointed_source_seq":0,
      "continuous":false,
      "database":null,
      "doc_id":null,
      "doc_write_failures":0,
      "docs_read":4,
      "docs_written":4,
      "missing_revisions_found":4,
      "replication_id":"1aebc296b560ed25c8153cda8f55069e+create_target",
      "revisions_checked":4,
      "source":"http://217.12.247.99:5984/account%2fa4%2f87%2f97afcbf4c1782d55613a5f86b0bf-201710/",
      "source_seq":"4-g1AAAAG5eJzLYWBg4MhgTmEQTM4vTc5ISXIwNDLXMwBCwxygFFMiQ5L8____sxIZ8ChKUgCSSfYgdRnMiUy5QB67pYGhYWKiGTY9-ExyAJkUT9jGBJC6eqiNjGAbzSwtUw1SDEm0MY8FSDI0ACmgYfPx2wpRuwCidj_CZsMkI0NLM1OybD4AMe0-MTY_gKgFhU0WAEpFdiQ",
      "started_on":1516776027,
      "target":"http://217.12.247.98:5984/account%2fa4%2f87%2f97afcbf4c1782d55613a5f86b0bf-201710/",
      "through_seq":"1-g1AAAAF1eJzLYWBg4MhgTmEQTM4vTc5ISXIwNDLXMwBCwxygFFMiQ5L8____sxIZ8ChKUgCSSfYgdRnMiYy5QB67pYGhYWKiGTY9-ExyAJkUT9jGBJC6eoLq8liAJEMDkAIqnU-M2gUQtfuJUXsAovY-MWofQNSC3JsFAMMyZnM",
      "type":"replication",
      "updated_on":1516776028,
      "user":null
   },
   {  
      "node":"[email protected]",
      "pid":"<0.31083.294>",
      "changes_pending":5214,
      "checkpoint_interval":30000,
      "checkpointed_source_seq":"1-g1AAAAF1eJzLYWBg4MhgTmEQTM4vTc5ISXIwNDLXMwBCwxygFFMiQ5L8____sxIZ8ChKUgCSSfYgdRnMiYy5QB67oamBSaK5ATY9-ExyAJkUT9jGBJC6eoLq8liAJEMDkAIqnU-M2gUQtfuJUXsAovY-MWofQNSC3JsFAI3AZkI",
      "continuous":false,
      "database":null,
      "doc_id":null,
      "doc_write_failures":0,
      "docs_read":4,
      "docs_written":4,
      "missing_revisions_found":4,
      "replication_id":"c125005084bccdfaffc089a2d25cdd43+create_target",
      "revisions_checked":4,
      "source":"http://217.12.247.99:5984/account%2fa4%2f87%2f97afcbf4c1782d55613a5f86b0bf-201711/",
      "source_seq":"4-g1AAAAGXeJzLYWBg4MhgTmEQTM4vTc5ISXIwNDLXMwBCwxygFFMiQ5L8____szKYE5lygQLsSSmWJoYWKdg04DEmSQFIJtmjmGRoamCSaG5AqkkOIJPiQSYlMuBTlwBSV09QXR4LkGRoAFJApfOJUbsAonY_MWoPQNTeJ0btA4hakHuzABv8bmY",
      "started_on":1516776055,
      "target":"http://217.12.247.98:5984/account%2fa4%2f87%2f97afcbf4c1782d55613a5f86b0bf-201711/",
      "through_seq":"1-g1AAAAF1eJzLYWBg4MhgTmEQTM4vTc5ISXIwNDLXMwBCwxygFFMiQ5L8____sxIZ8ChKUgCSSfYgdRnMiYy5QB67oamBSaK5ATY9-ExyAJkUT9jGBJC6eoLq8liAJEMDkAIqnU-M2gUQtfuJUXsAovY-MWofQNSC3JsFAI3AZkI",
      "type":"replication",
      "updated_on":1516779838,
      "user":null
   },
   {  
      "node":"[email protected]",
      "pid":"<0.7273.48>",
      "changes_pending":235248,
      "checkpoint_interval":30000,
      "checkpointed_source_seq":"3-g1AAAAGXeJzLYWBg4MhgTmEQTM4vTc5ISXIwNDLXMwBCwxygFFMiQ5L8____sxIZ8ChKUgCSSfaE1TmA1MUTVpcAUlcPUpfBnMiYC-Sxp6SmGCUnpWDTg8ekPBYgydAApICGzcdvK0TtAoja_cSoPQBRex_kSiawK80tUhJNjFPJcuUDiGmgsMkCAN8Hb1s",
      "continuous":false,
      "database":null,
      "doc_id":null,
      "doc_write_failures":0,
      "docs_read":773,
      "docs_written":773,
      "missing_revisions_found":773,
      "replication_id":"260c79c446f46814cdf361b5d02b59ce+create_target",
      "revisions_checked":773,
      "source":"http://217.12.247.99:5984/account%2f63%2ff8%2f14795fd5b8c74aa04ee865c52f73-201711/",
      "source_seq":"1041-g1AAAAJjeJyd0LsNwjAQBmCLIEHLBDABivOyXZEdGABsn60oBKhoKICahhFgE1gEwSbBj5QRUqKT_pPudF9xFUJoXASAJnJ_kAWIHEdkHprClVkNOBLTuq7LIuBqawYjpkKuCGs7-MOImUmxaKSTkwSVOo1oVym30qqRjk4CAlpy6CqtrXRupKuXFERSdJV2Q5PoYprB7la7OY2zBHSGe2kPrz2t9vb_yqiMdT_t5bWP1ZZOIxR4Eqte2tdr7m8bp2nMaJSKtrvyB9O6pI8",
      "started_on":1516774615,
      "target":"http://217.12.247.98:5984/account%2f63%2ff8%2f14795fd5b8c74aa04ee865c52f73-201711/",
      "through_seq":"769-g1AAAAJjeJyd0EsKwjAQBuBgBd16Aj2BNOkj6crewYXgRjN5EEvVlWu9id5Eb6I3qXl04aIILQP_wAzzLaZGCE1NJNFMnC_CSCgxocvYFq7tasQRzJumqUzEt0c7mBQq5ooWXQd_GFjYhFUrKS8BEzojrK9UOmnXSsZLkkotuOwr7Z10baVDkJQkAvpKp7FNdLPNYnengdd4kUqd40HaI2jPn3_lTCR6mPYK2ttpa69RJnmaqEHaJ2j-bxuvaVwwkkHXXfUFtg-jfw",
      "type":"replication",
      "updated_on":1516781818,
      "user":null
   }
]

Same database. 773 docs written and 235248 pending changes.

sergey-safarov picture sergey-safarov on 24 Jan 2018

Same database after 6 hours, 2542 docs written and 234248 pending changes.

[  
   {  
      "node":"[email protected]",
      "pid":"<0.29148.294>",
      "changes_pending":6843,
      "checkpoint_interval":30000,
      "checkpointed_source_seq":"1-g1AAAAF1eJzLYWBg4MhgTmEQTM4vTc5ISXIwNDLXMwBCwxygFFMiQ5L8____sxIZ8ChKUgCSSfYgdRnMiYy5QB67pYGhYWKiGTY9-ExyAJkUT9jGBJC6eoLq8liAJEMDkAIqnU-M2gUQtfuJUXsAovY-MWofQNSC3JsFAMMyZnM",
      "continuous":false,
      "database":null,
      "doc_id":null,
      "doc_write_failures":0,
      "docs_read":4,
      "docs_written":4,
      "missing_revisions_found":4,
      "replication_id":"1aebc296b560ed25c8153cda8f55069e+create_target",
      "revisions_checked":4,
      "source":"http://217.12.247.99:5984/account%2fa4%2f87%2f97afcbf4c1782d55613a5f86b0bf-201710/",
      "source_seq":"4-g1AAAAG5eJzLYWBg4MhgTmEQTM4vTc5ISXIwNDLXMwBCwxygFFMiQ5L8____sxIZ8ChKUgCSSfYgdRnMiUy5QB67pYGhYWKiGTY9-ExyAJkUT9jGBJC6eqiNjGAbzSwtUw1SDEm0MY8FSDI0ACmgYfPx2wpRuwCidj_CZsMkI0NLM1OybD4AMe0-MTY_gKgFhU0WAEpFdiQ",
      "started_on":1516776027,
      "target":"http://217.12.247.98:5984/account%2fa4%2f87%2f97afcbf4c1782d55613a5f86b0bf-201710/",
      "through_seq":"1-g1AAAAF1eJzLYWBg4MhgTmEQTM4vTc5ISXIwNDLXMwBCwxygFFMiQ5L8____sxIZ8ChKUgCSSfYgdRnMiYy5QB67pYGhYWKiGTY9-ExyAJkUT9jGBJC6eoLq8liAJEMDkAIqnU-M2gUQtfuJUXsAovY-MWofQNSC3JsFAMMyZnM",
      "type":"replication",
      "updated_on":1516794240,
      "user":null
   },
   {  
      "node":"[email protected]",
      "pid":"<0.7273.48>",
      "changes_pending":234248,
      "checkpoint_interval":30000,
      "checkpointed_source_seq":"2041-g1AAAAJveJyd0EsKwjAQANBoBQVXbtzWE0iTfpKs7EEEzZdSqq5c60106S30Bh5Bdx4jpknBTRFaAhOYYR4zUwEAJkUgwUwcjqKQPIcILyP7YGVLQwZ4aIwpi4B9djYxpipiCtO2hj8MX9jIV17iAAymDuNE6BSRrlheY5sfFjpMYqkFk12xbY2dmh2Nl5REgneV9iMbwdl-Frs0o80dyGgidQZ7gVcP3htw7Q-XERHrfuDDg69635vTMJEsiVUv7e01d72n0zSkBKW8ra_8AuJDpH8",
      "continuous":false,
      "database":null,
      "doc_id":null,
      "doc_write_failures":0,
      "docs_read":2542,
      "docs_written":2542,
      "missing_revisions_found":2542,
      "replication_id":"260c79c446f46814cdf361b5d02b59ce+create_target",
      "revisions_checked":2542,
      "source":"http://217.12.247.99:5984/account%2f63%2ff8%2f14795fd5b8c74aa04ee865c52f73-201711/",
      "source_seq":"2541-g1AAAAJ4eJyd0LsNwjAQBmBDkKClpggsgGLnYbsirEET_FQUBaioYRPoGAM2gRWYIDh2UBqElOik_6Q73VdcCQCY5J4EU3E4ilzyFCK8DEzB0qyGDHC_qqoi9zgAg9nOzMZUBUxh-uvmj8TnJvmqxTYW40ToGJGuWFpjWYtlFpNYasFkV2xbY6cWWztMSSR4V2w_MgnOphnv0oALCzIaSZ3AXuDVgfcGvLnfJUSEuh_4cODTgOxtNUwki0LVS3s57ftA34IaUoJi_uu0-AAzHaN5",
      "started_on":1516774615,
      "target":"http://217.12.247.98:5984/account%2f63%2ff8%2f14795fd5b8c74aa04ee865c52f73-201711/",
      "through_seq":"2541-g1AAAAJ4eJyd0LsNwjAQBmBDkKClpggsgGLnYbsirEET_FQUBaioYRPoGAM2gRWYIDh2UBqElOik_6Q73VdcCQCY5J4EU3E4ilzyFCK8DEzB0qyGDHC_qqoi9zgAg9nOzMZUBUxh-uvmj8TnJvmqxTYW40ToGJGuWFpjWYtlFpNYasFkV2xbY6cWWztMSSR4V2w_MgnOphnv0oALCzIaSZ3AXuDVgfcGvLnfJUSEuh_4cODTgOxtNUwki0LVS3s57ftA34IaUoJi_uu0-AAzHaN5",
      "type":"replication",
      "updated_on":1516792740,
      "user":null
   }
]
sergey-safarov picture sergey-safarov on 24 Jan 2018

I guess that issue related to databases that contains docs with attachments.

sergey-safarov picture sergey-safarov on 24 Jan 2018

Thanks for the report, Sergey

You might be encountering this issue: https://github.com/apache/couchdb/issues/745

A few questions to help narrow down the cause:

  • What is your exact version of CouchDB, is it 2.0.0, 2.1.1 or other

  • What is the size distribution of your documents (average, what are the largest ones)

  • What are the attachment sizes, what is their distribution (average, largest)

    • Are the databases which failed to replicate enough the ones with attachments? Do you have any that don't have attachments and those replicated fast enough?

    • If you can attach the contents of these config sections (but sanitize them to not contain database paths, username and other such things): couchdb, replicator, chttpd and cluster. It's easy to get those via the node local port (5986 one) and the _config section:

Here is an example of my config from the replicator section (jq part is the json pretty printer)

curl -s http://<adminuser>:<adminpas>@localhost:15986/_config/replicator | jq '.'
{
  "startup_jitter": "5000",
  "connection_timeout": "30000",
  "max_churn": "20",
  "interval": "60000",
  "retries_per_request": "5",
  "max_jobs": "500",
  "verify_ssl_certificates": "false",
  "http_connections": "20",
  "ssl_certificate_max_depth": "3",
  "socket_options": "[{keepalive, true}, {nodelay, false}]",
  "worker_processes": "4",
  "worker_batch_size": "500"
}
nickva picture nickva on 24 Jan 2018
👍1

@sergey-safarov

See update to related ticket https://github.com/apache/couchdb/issues/745#issuecomment-360251475

Try that fix and see if it changes anything.

Thanks again for your report

nickva picture nickva on 24 Jan 2018

Hello @nickva
I use RPM package http://apache.bintray.com/couchdb-rpm/el7/x86_64/couchdb-2.1.1-2.el7.centos.x86_64.rpm
Average document size with attacmetns - 2,3Gb / 12876 docs = 179 byte per doc
Largest doc - not more 1 Kb without attachments
Largest attachment - 2,4Mb
Average attachment - 800Kb
Think all databases without attachments is replicated normally.
Formated config is attached (used default).

{  
   "native_query_servers":{  
      "query":"{mango_native_proc, start_link, []}"
   },
   "httpd_design_handlers":{  
      "_compact":"{couch_mrview_http, handle_compact_req}",
      "_info":"{couch_mrview_http, handle_info_req}",
      "_list":"{couch_mrview_show, handle_view_list_req}",
      "_rewrite":"{couch_httpd_rewrite, handle_rewrite_req}",
      "_show":"{couch_mrview_show, handle_doc_show_req}",
      "_update":"{couch_mrview_show, handle_doc_update_req}",
      "_view":"{couch_mrview_http, handle_view_req}",
      "_view_changes":"{couch_mrview_http, handle_view_changes_req}"
   },
   "uuids":{  
      "algorithm":"sequential",
      "max_count":"1000"
   },
   "cluster":{  
      "n":"3",
      "q":"8"
   },
   "cors":{  
      "credentials":"false"
   },
   "chttpd":{  
      "backlog":"512",
      "bind_address":"0.0.0.0",
      "docroot":"./share/www",
      "port":"5984",
      "prefer_minimal":"Cache-Control, Content-Length, Content-Range, Content-Type, ETag, Server, Transfer-Encoding, Vary",
      "require_valid_user":"false",
      "socket_options":"[{recbuf, 262144}, {sndbuf, 262144}, {nodelay, true}]"
   },
   "httpd_global_handlers":{  
      "/":"{couch_httpd_misc_handlers, handle_welcome_req, <<\"Welcome\">>}",
      "_active_tasks":"{couch_httpd_misc_handlers, handle_task_status_req}",
      "_all_dbs":"{couch_httpd_misc_handlers, handle_all_dbs_req}",
      "_config":"{couch_httpd_misc_handlers, handle_config_req}",
      "_plugins":"{couch_plugins_httpd, handle_req}",
      "_replicate":"{couch_replicator_httpd, handle_req}",
      "_restart":"{couch_httpd_misc_handlers, handle_restart_req}",
      "_session":"{couch_httpd_auth, handle_session_req}",
      "_stats":"{couch_stats_httpd, handle_stats_req}",
      "_system":"{chttpd_misc, handle_system_req}",
      "_utils":"{couch_httpd_misc_handlers, handle_utils_dir_req, \"./share/www\"}",
      "_uuids":"{couch_httpd_misc_handlers, handle_uuids_req}",
      "favicon.ico":"{couch_httpd_misc_handlers, handle_favicon_req, \"./share/www\"}"
   },
   "attachments":{  
      "compressible_types":"text/*, application/javascript, application/json, application/xml",
      "compression_level":"8"
   },
   "query_server_config":{  
      "os_process_limit":"100",
      "reduce_limit":"true"
   },
   "vendor":{  
      "name":"The Apache Software Foundation"
   },
   "replicator":{  
      "connection_timeout":"30000",
      "http_connections":"20",
      "interval":"60000",
      "max_churn":"20",
      "max_jobs":"500",
      "retries_per_request":"5",
      "socket_options":"[{keepalive, true}, {nodelay, false}]",
      "ssl_certificate_max_depth":"3",
      "startup_jitter":"5000",
      "verify_ssl_certificates":"false",
      "worker_batch_size":"500",
      "worker_processes":"4"
   },
   "ssl":{  
      "port":"6984"
   },
   "log":{  
      "file":"/var/log/couchdb/couchdb.log",
      "level":"info",
      "writer":"file"
   },
   "indexers":{  
      "couch_mrview":"true"
   },
   "view_compaction":{  
      "keyvalue_buffer_size":"2097152"
   },
   "query_servers":{  
      "coffeescript":"./bin/couchjs ./share/server/main-coffee.js",
      "javascript":"./bin/couchjs ./share/server/main.js"
   },
   "features":{  
      "scheduler":"true"
   },
   "daemons":{  
      "auth_cache":"{couch_auth_cache, start_link, []}",
      "compaction_daemon":"{couch_compaction_daemon, start_link, []}",
      "external_manager":"{couch_external_manager, start_link, []}",
      "httpd":"{couch_httpd, start_link, []}",
      "index_server":"{couch_index_server, start_link, []}",
      "os_daemons":"{couch_os_daemons, start_link, []}",
      "query_servers":"{couch_proc_manager, start_link, []}",
      "uuids":"{couch_uuids, start, []}",
      "vhosts":"{couch_httpd_vhost, start_link, []}"
   },
   "couch_peruser":{  
      "delete_dbs":"false",
      "enable":"false"
   },
   "httpd":{  
      "allow_jsonp":"false",
      "authentication_handlers":"{couch_httpd_auth, cookie_authentication_handler}, {couch_httpd_auth, default_authentication_handler}",
      "bind_address":"0.0.0.0",
      "default_handler":"{couch_httpd_db, handle_request}",
      "enable_cors":"false",
      "enable_xframe_options":"false",
      "max_http_request_size":"67108864",
      "port":"5986",
      "secure_rewrites":"true",
      "socket_options":"[{recbuf, 262144}, {sndbuf, 262144}]",
      "vhost_global_handlers":"_utils, _uuids, _session, _users"
   },
   "httpd_db_handlers":{  
      "_all_docs":"{couch_mrview_http, handle_all_docs_req}",
      "_changes":"{couch_httpd_db, handle_db_changes_req}",
      "_compact":"{couch_httpd_db, handle_compact_req}",
      "_design":"{couch_httpd_db, handle_design_req}",
      "_design_docs":"{couch_mrview_http, handle_design_docs_req}",
      "_local_docs":"{couch_mrview_http, handle_local_docs_req}",
      "_temp_view":"{couch_mrview_http, handle_temp_view_req}",
      "_view_cleanup":"{couch_mrview_http, handle_cleanup_req}"
   },
   "database_compaction":{  
      "checkpoint_after":"5242880",
      "doc_buffer_size":"524288"
   },
   "csp":{  
      "enable":"true"
   },
   "couch_httpd_auth":{  
      "allow_persistent_cookies":"false",
      "auth_cache_size":"50",
      "authentication_db":"_users",
      "authentication_redirect":"/_utils/session.html",
      "iterations":"10",
      "require_valid_user":"false",
      "timeout":"600"
   },
   "couchdb":{  
      "attachment_stream_buffer_size":"4096",
      "changes_doc_ids_optimization_threshold":"100",
      "database_dir":"./data",
      "default_security":"admin_local",
      "delayed_commits":"false",
      "file_compression":"snappy",
      "max_dbs_open":"500",
      "os_process_timeout":"5000",
      "uuid":"aa251fc49c416b4b37f8ba8b1eb6f1aa",
      "view_index_dir":"./data"
   },
   "compactions":{  
      "_default":"[{db_fragmentation, \"70%\"}, {view_fragmentation, \"60%\"}]"
   },
   "compaction_daemon":{  
      "check_interval":"300",
      "min_file_size":"131072"
   }
}

Next i will test recommended fix

sergey-safarov picture sergey-safarov on 25 Jan 2018

I increased max_http_request_size up to 10 times - issue is still persist.
I can create VM with installed couchdb master and one database that cannot be replicated.
If you provide your public ssh key, then you can look issue on host.

sergey-safarov picture sergey-safarov on 25 Jan 2018

@sergey-safarov please follow apache/couchdb#745, this is likely a known issue with replication of documents with attachments.

wohali picture wohali on 1 Feb 2018

@sergey-safarov have you tried the latest master to see if this is resolved? Multiple patches have landed since 25 January.

wohali picture wohali on 27 Mar 2018

This page may be key for understanding of slow performance of containerized app
https://sysdig.com/blog/container-isolation-gone-wrong/

sergey-safarov picture sergey-safarov on 21 Nov 2018
👍1
Was this page helpful?
0 / 5 - 0 ratings

Related issues

sploders101 picture sploders101  路  3Comments
wohali picture wohali  路  5Comments
popojargo picture popojargo  路  5Comments
mtabb13 picture mtabb13  路  5Comments
YC picture YC  路  5Comments