Influxdb: V1.1 disk read speed of 20M per second

[zhang8h]

Directions

Hello, please help me look at this log, influxdb disk read at this time the speed of 20M per second, resulting in the server can not respond, can see what is the reason

v1.1 Server

log

[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:11 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" 8a220532-0e19-11e7-b911-000000000000 365112
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:11 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" 8a21ef2b-0e19-11e7-b90f-000000000000 517099
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:11 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" 8a4bc4c9-0e19-11e7-b913-000000000000 294030
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:11 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" 8a4bcbe4-0e19-11e7-b914-000000000000 320487
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:11 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" 8a4c3677-0e19-11e7-b915-000000000000 350234
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:11 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" 8a4c3b3d-0e19-11e7-b916-000000000000 368202
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:11 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" 8a70051c-0e19-11e7-b917-000000000000 159559
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:11 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" 8a700cba-0e19-11e7-b918-000000000000 307441
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:12 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" 8a956025-0e19-11e7-b91b-000000000000 86156
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:11 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" 8a7011cc-0e19-11e7-b919-000000000000 352367
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:11 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" 8a70169b-0e19-11e7-b91a-000000000000 391211
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:12 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" 8a95bf14-0e19-11e7-b91c-000000000000 438277
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:12 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" 8a95c2e8-0e19-11e7-b91d-000000000000 512867
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:12 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" 8ae27573-0e19-11e7-b922-000000000000 27693
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:12 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" 8abb73c3-0e19-11e7-b91e-000000000000 533077
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:12 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" 8abb7b72-0e19-11e7-b91f-000000000000 656143
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:12 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" 8abb7f42-0e19-11e7-b920-000000000000 664527
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:12 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" 8abb833b-0e19-11e7-b921-000000000000 672467
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:12 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" 8af073bd-0e19-11e7-b924-000000000000 351375
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:12 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" 8af07b94-0e19-11e7-b925-000000000000 377759
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:12 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" 8af0808b-0e19-11e7-b926-000000000000 418231
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:13 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" 8b2e1508-0e19-11e7-b92f-000000000000 27488
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:12 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" 8af0884d-0e19-11e7-b927-000000000000 523220
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:12 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" 8ae3f42f-0e19-11e7-b923-000000000000 676980
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:12 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" 8b0ce6ee-0e19-11e7-b928-000000000000 553687
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:13 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" 8b2df569-0e19-11e7-b929-000000000000 347179
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:13 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" 8b2dfce6-0e19-11e7-b92a-000000000000 356198
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:13 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" 8b2e00a5-0e19-11e7-b92b-000000000000 364972
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:13 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" 8b2e0616-0e19-11e7-b92c-000000000000 378446
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:13 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" 8b2e0abb-0e19-11e7-b92d-000000000000 388143
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:13 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" 8b2e0faf-0e19-11e7-b92e-000000000000 395650
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:13 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" 8b702c97-0e19-11e7-b932-000000000000 68970
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:13 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" 8b57a857-0e19-11e7-b930-000000000000 279212
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:13 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" 8b80dba9-0e19-11e7-b933-000000000000 17636
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:13 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" 8b5ac4ce-0e19-11e7-b931-000000000000 275078
[tsm1] 2017/03/21 17:34:13 beginning full compaction of group 0, 2 TSM files
[tsm1] 2017/03/21 17:34:13 compacting full group (0) /home/waterdrop/shared/data/coldcloud/default/105/000000193-000000003.tsm (#0)
[tsm1] 2017/03/21 17:34:13 compacting full group (0) /home/waterdrop/shared/data/coldcloud/default/105/000000194-000000001.tsm (#1)
[tsm1] 2017/03/21 17:34:13 beginning full compaction of group 0, 2 TSM files
[tsm1] 2017/03/21 17:34:13 compacting full group (0) /home/waterdrop/shared/data/coldcloud/default/118/000000223-000000003.tsm (#0)
[tsm1] 2017/03/21 17:34:13 compacting full group (0) /home/waterdrop/shared/data/coldcloud/default/118/000000227-000000003.tsm (#1)
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:13 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" 8b80f77f-0e19-11e7-b934-000000000000 337244
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:13 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" 8bb1196c-0e19-11e7-b935-000000000000 103196
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:13 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" 8bb12424-0e19-11e7-b936-000000000000 148113
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:13 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" 8bb12ad6-0e19-11e7-b937-000000000000 256995
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:14 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" 8bc55e7d-0e19-11e7-b938-000000000000 234787
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:14 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" 8bd0a793-0e19-11e7-b939-000000000000 173811
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:14 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" 8bd10d92-0e19-11e7-b93a-000000000000 184384
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:14 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" 8bfb6bac-0e19-11e7-b93b-000000000000 11214
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:14 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" 8bfd47d7-0e19-11e7-b93c-000000000000 179488
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:14 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" 8c000ff2-0e19-11e7-b93d-000000000000 181560
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:14 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" 8c207b86-0e19-11e7-b93e-000000000000 12989
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:14 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" 8c20b68d-0e19-11e7-b93f-000000000000 38010
[tsm1] 2017/03/21 17:34:14 beginning full compaction of group 0, 2 TSM files
[tsm1] 2017/03/21 17:34:14 compacting full group (0) /home/waterdrop/shared/data/coldcloud/default/150/000000157-000000002.tsm (#0)
[tsm1] 2017/03/21 17:34:14 compacting full group (0) /home/waterdrop/shared/data/coldcloud/default/150/000000158-000000001.tsm (#1)
[tsm1] 2017/03/21 17:34:14 beginning full compaction of group 0, 3 TSM files
[tsm1] 2017/03/21 17:34:14 compacting full group (0) /home/waterdrop/shared/data/coldcloud/default/131/000000137-000000003.tsm (#0)
[tsm1] 2017/03/21 17:34:14 compacting full group (0) /home/waterdrop/shared/data/coldcloud/default/131/000000140-000000002.tsm (#1)
[tsm1] 2017/03/21 17:34:14 compacting full group (0) /home/waterdrop/shared/data/coldcloud/default/131/000000141-000000001.tsm (#2)
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:14 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" 8c4f85e0-0e19-11e7-b941-000000000000 336059
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:14 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" 8c4b750a-0e19-11e7-b940-000000000000 375208
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:15 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" 8c5de2cc-0e19-11e7-b942-000000000000 264790
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:15 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" 8c79ac64-0e19-11e7-b943-000000000000 528827
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:15 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" 8ca00f9d-0e19-11e7-b945-000000000000 562323
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:15 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" 8ce6f347-0e19-11e7-b948-000000000000 208049
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:15 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" 8c9ec749-0e19-11e7-b944-000000000000 688657
[tsm1] 2017/03/21 17:34:16 beginning full compaction of group 0, 2 TSM files
[tsm1] 2017/03/21 17:34:16 compacting full group (0) /home/waterdrop/shared/data/coldcloud/default/206/000000162-000000002.tsm (#0)
[tsm1] 2017/03/21 17:34:16 compacting full group (0) /home/waterdrop/shared/data/coldcloud/default/206/000000163-000000001.tsm (#1)
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:15 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" 8cbace96-0e19-11e7-b946-000000000000 723131
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:16 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" 8d157a04-0e19-11e7-b94b-000000000000 138508
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:15 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" 8cc8d890-0e19-11e7-b947-000000000000 718649
[tsm1] 2017/03/21 17:34:16 beginning full compaction of group 0, 2 TSM files
[tsm1] 2017/03/21 17:34:16 compacting full group (0) /home/waterdrop/shared/data/coldcloud/default/193/000000156-000000002.tsm (#0)
[tsm1] 2017/03/21 17:34:16 compacting full group (0) /home/waterdrop/shared/data/coldcloud/default/193/000000157-000000001.tsm (#1)
[tsm1] 2017/03/21 17:34:16 beginning full compaction of group 0, 2 TSM files
[tsm1] 2017/03/21 17:34:16 compacting full group (0) /home/waterdrop/shared/data/coldcloud/default/205/000000149-000000002.tsm (#0)
[tsm1] 2017/03/21 17:34:16 compacting full group (0) /home/waterdrop/shared/data/coldcloud/default/205/000000150-000000001.tsm (#1)
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:16 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" 8d326707-0e19-11e7-b94c-000000000000 359636
[tsm1] 2017/03/21 17:34:16 beginning full compaction of group 0, 2 TSM files
[tsm1] 2017/03/21 17:34:16 compacting full group (0) /home/waterdrop/shared/data/coldcloud/default/230/000000167-000000002.tsm (#0)
[tsm1] 2017/03/21 17:34:16 compacting full group (0) /home/waterdrop/shared/data/coldcloud/default/230/000000168-000000001.tsm (#1)
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:16 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" 8d502cbf-0e19-11e7-b94d-000000000000 365448
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:16 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" 8cfdb15f-0e19-11e7-b94a-000000000000 1021921
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:16 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" 8cfda048-0e19-11e7-b949-000000000000 1137384
[tsm1] 2017/03/21 17:34:17 beginning full compaction of group 0, 2 TSM files
[tsm1] 2017/03/21 17:34:17 compacting full group (0) /home/waterdrop/shared/data/coldcloud/default/243/000000150-000000002.tsm (#0)
[tsm1] 2017/03/21 17:34:17 compacting full group (0) /home/waterdrop/shared/data/coldcloud/default/243/000000151-000000001.tsm (#1)
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:17 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" 8da87fa4-0e19-11e7-b951-000000000000 256712
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:16 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" 8d8a4096-0e19-11e7-b94f-000000000000 925128
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:18 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" 8e3d5636-0e19-11e7-b95a-000000000000 33637
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:17 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" 8d95808f-0e19-11e7-b950-000000000000 1248236
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:18 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" 8e5a4935-0e19-11e7-b95d-000000000000 151294
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:16 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" 8d60f7e5-0e19-11e7-b94e-000000000000 1934739
[tsm1] 2017/03/21 17:34:18 beginning full compaction of group 0, 2 TSM files
[tsm1] 2017/03/21 17:34:18 compacting full group (0) /home/waterdrop/shared/data/coldcloud/default/256/000000206-000000002.tsm (#0)
[tsm1] 2017/03/21 17:34:18 compacting full group (0) /home/waterdrop/shared/data/coldcloud/default/256/000000207-000000001.tsm (#1)
[tsm1] 2017/03/21 17:34:19 beginning full compaction of group 0, 2 TSM files
[tsm1] 2017/03/21 17:34:19 compacting full group (0) /home/waterdrop/shared/data/coldcloud/default/288/000000189-000000002.tsm (#0)
[tsm1] 2017/03/21 17:34:19 compacting full group (0) /home/waterdrop/shared/data/coldcloud/default/288/000000190-000000001.tsm (#1)
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:17 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" 8dbe072a-0e19-11e7-b952-000000000000 1849691
[tsm1] 2017/03/21 17:34:19 beginning full compaction of group 0, 2 TSM files
[tsm1] 2017/03/21 17:34:19 compacting full group (0) /home/waterdrop/shared/data/coldcloud/default/266/000000166-000000002.tsm (#0)
[tsm1] 2017/03/21 17:34:19 compacting full group (0) /home/waterdrop/shared/data/coldcloud/default/266/000000167-000000001.tsm (#1)
[tsm1] 2017/03/21 17:34:19 beginning full compaction of group 0, 2 TSM files
[tsm1] 2017/03/21 17:34:19 compacting full group (0) /home/waterdrop/shared/data/coldcloud/default/276/000000192-000000002.tsm (#0)
[tsm1] 2017/03/21 17:34:19 compacting full group (0) /home/waterdrop/shared/data/coldcloud/default/276/000000193-000000001.tsm (#1)
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:19 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" 8f286a6a-0e19-11e7-b96b-000000000000 354968
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:17 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" 8df11b08-0e19-11e7-b953-000000000000 2429060
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:20 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" 8f58ec9d-0e19-11e7-b96d-000000000000 201778
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:18 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" 8e33da91-0e19-11e7-b956-000000000000 2251160
[tsm1] 2017/03/21 17:34:20 beginning full compaction of group 0, 2 TSM files
[tsm1] 2017/03/21 17:34:20 compacting full group (0) /home/waterdrop/shared/data/coldcloud/default/298/000000250-000000002.tsm (#0)
[tsm1] 2017/03/21 17:34:20 compacting full group (0) /home/waterdrop/shared/data/coldcloud/default/298/000000251-000000001.tsm (#1)
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:18 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" 8e33efdc-0e19-11e7-b957-000000000000 2800812
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:20 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" 8fd6e078-0e19-11e7-b97a-000000000000 747117
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:18 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" 8e3a2c41-0e19-11e7-b958-000000000000 3770148
[tsm1] 2017/03/21 17:34:21 beginning full compaction of group 0, 2 TSM files
[tsm1] 2017/03/21 17:34:21 compacting full group (0) /home/waterdrop/shared/data/coldcloud/default/309/000000245-000000002.tsm (#0)
[tsm1] 2017/03/21 17:34:21 compacting full group (0) /home/waterdrop/shared/data/coldcloud/default/309/000000246-000000001.tsm (#1)
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:17 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" 8e113c48-0e19-11e7-b954-000000000000 4745580
[tsm1] 2017/03/21 17:34:22 beginning full compaction of group 0, 2 TSM files
[tsm1] 2017/03/21 17:34:22 compacting full group (0) /home/waterdrop/shared/data/coldcloud/default/378/000000168-000000002.tsm (#0)
[tsm1] 2017/03/21 17:34:22 compacting full group (0) /home/waterdrop/shared/data/coldcloud/default/378/000000169-000000001.tsm (#1)
[tsm1] 2017/03/21 17:34:22 beginning full compaction of group 0, 2 TSM files
[tsm1] 2017/03/21 17:34:22 compacting full group (0) /home/waterdrop/shared/data/coldcloud/default/387/000000181-000000002.tsm (#0)
[tsm1] 2017/03/21 17:34:22 compacting full group (0) /home/waterdrop/shared/data/coldcloud/default/387/000000182-000000001.tsm (#1)
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:17 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" 8e11445a-0e19-11e7-b955-000000000000 5160061
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:22 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" 910b8293-0e19-11e7-b987-000000000000 349562
[tsm1] 2017/03/21 17:34:23 beginning full compaction of group 0, 2 TSM files
[tsm1] 2017/03/21 17:34:23 compacting full group (0) /home/waterdrop/shared/data/coldcloud/default/407/000000190-000000002.tsm (#0)
[tsm1] 2017/03/21 17:34:23 compacting full group (0) /home/waterdrop/shared/data/coldcloud/default/407/000000191-000000001.tsm (#1)
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:22 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" 910baefe-0e19-11e7-b98f-000000000000 956987
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:18 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" 8e3d4d38-0e19-11e7-b959-000000000000 5785966
[tsm1] 2017/03/21 17:34:24 beginning full compaction of group 0, 2 TSM files
[tsm1] 2017/03/21 17:34:24 compacting full group (0) /home/waterdrop/shared/data/coldcloud/default/418/000000179-000000002.tsm (#0)
[tsm1] 2017/03/21 17:34:24 compacting full group (0) /home/waterdrop/shared/data/coldcloud/default/418/000000180-000000001.tsm (#1)
[tsm1] 2017/03/21 17:34:24 beginning full compaction of group 0, 2 TSM files
[tsm1] 2017/03/21 17:34:24 compacting full group (0) /home/waterdrop/shared/data/coldcloud/default/397/000000200-000000003.tsm (#0)
[tsm1] 2017/03/21 17:34:24 compacting full group (0) /home/waterdrop/shared/data/coldcloud/default/397/000000201-000000001.tsm (#1)
[tsm1] 2017/03/21 17:34:25 beginning full compaction of group 0, 2 TSM files
[tsm1] 2017/03/21 17:34:25 compacting full group (0) /home/waterdrop/shared/data/coldcloud/default/428/000000178-000000002.tsm (#0)
[tsm1] 2017/03/21 17:34:25 compacting full group (0) /home/waterdrop/shared/data/coldcloud/default/428/000000179-000000001.tsm (#1)
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:18 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" 8e3d5b68-0e19-11e7-b95b-000000000000 7255463
[tsm1] 2017/03/21 17:34:25 beginning full compaction of group 0, 2 TSM files
[tsm1] 2017/03/21 17:34:25 compacting full group (0) /home/waterdrop/shared/data/coldcloud/default/448/000000182-000000002.tsm (#0)
[tsm1] 2017/03/21 17:34:25 compacting full group (0) /home/waterdrop/shared/data/coldcloud/default/448/000000183-000000001.tsm (#1)
[tsm1] 2017/03/21 17:34:26 beginning full compaction of group 0, 2 TSM files
[tsm1] 2017/03/21 17:34:26 compacting full group (0) /home/waterdrop/shared/data/coldcloud/default/459/000000187-000000002.tsm (#0)
[tsm1] 2017/03/21 17:34:26 compacting full group (0) /home/waterdrop/shared/data/coldcloud/default/459/000000188-000000001.tsm (#1)
[tsm1] 2017/03/21 17:34:26 beginning full compaction of group 0, 2 TSM files
[tsm1] 2017/03/21 17:34:26 compacting full group (0) /home/waterdrop/shared/data/coldcloud/default/438/000000176-000000002.tsm (#0)
[tsm1] 2017/03/21 17:34:26 compacting full group (0) /home/waterdrop/shared/data/coldcloud/default/438/000000177-000000001.tsm (#1)
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:18 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" 8e5a3a42-0e19-11e7-b95c-000000000000 8101388
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:18 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" 8e60cd41-0e19-11e7-b95e-000000000000 8333796
[tsm1] 2017/03/21 17:34:26 beginning full compaction of group 0, 2 TSM files
[tsm1] 2017/03/21 17:34:26 compacting full group (0) /home/waterdrop/shared/data/coldcloud/default/469/000000199-000000002.tsm (#0)
[tsm1] 2017/03/21 17:34:26 compacting full group (0) /home/waterdrop/shared/data/coldcloud/default/469/000000200-000000001.tsm (#1)
[tsm1] 2017/03/21 17:34:27 beginning full compaction of group 0, 2 TSM files
[tsm1] 2017/03/21 17:34:27 compacting full group (0) /home/waterdrop/shared/data/coldcloud/default/479/000000198-000000002.tsm (#0)
[tsm1] 2017/03/21 17:34:27 compacting full group (0) /home/waterdrop/shared/data/coldcloud/default/479/000000199-000000001.tsm (#1)
[tsm1] 2017/03/21 17:34:27 beginning full compaction of group 0, 2 TSM files
[tsm1] 2017/03/21 17:34:27 compacting full group (0) /home/waterdrop/shared/data/coldcloud/default/489/000000208-000000002.tsm (#0)
[tsm1] 2017/03/21 17:34:27 compacting full group (0) /home/waterdrop/shared/data/coldcloud/default/489/000000209-000000001.tsm (#1)
[tsm1] 2017/03/21 17:34:27 beginning full compaction of group 0, 2 TSM files
[tsm1] 2017/03/21 17:34:27 compacting full group (0) /home/waterdrop/shared/data/coldcloud/default/498/000000203-000000003.tsm (#0)
[tsm1] 2017/03/21 17:34:27 compacting full group (0) /home/waterdrop/shared/data/coldcloud/default/498/000000204-000000001.tsm (#1)
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:18 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" 8e60d57c-0e19-11e7-b95f-000000000000 9801784
[tsm1] 2017/03/21 17:34:28 beginning full compaction of group 0, 2 TSM files
[tsm1] 2017/03/21 17:34:28 compacting full group (0) /home/waterdrop/shared/data/coldcloud/default/507/000000187-000000002.tsm (#0)
[tsm1] 2017/03/21 17:34:28 compacting full group (0) /home/waterdrop/shared/data/coldcloud/default/507/000000188-000000001.tsm (#1)
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:18 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 500 44 "-" "WaterdropClient" 8e7fe1e9-0e19-11e7-b960-000000000000 11083472
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:18 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 500 44 "-" "WaterdropClient" 8e7fe81a-0e19-11e7-b961-000000000000 11147829
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:18 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 500 44 "-" "WaterdropClient" 8e81c337-0e19-11e7-b962-000000000000 11228313
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:18 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 500 44 "-" "WaterdropClient" 8e9910a7-0e19-11e7-b963-000000000000 11076513
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:18 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 500 44 "-" "WaterdropClient" 8ebafded-0e19-11e7-b964-000000000000 11091250
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:19 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 500 44 "-" "WaterdropClient" 8f286094-0e19-11e7-b96a-000000000000 10468846
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:19 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 500 44 "-" "WaterdropClient" 8f287213-0e19-11e7-b96c-000000000000 10469794
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:19 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 500 44 "-" "WaterdropClient" 8ed3da4a-0e19-11e7-b965-000000000000 11134312
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:19 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 500 44 "-" "WaterdropClient" 8ed3e304-0e19-11e7-b966-000000000000 11134677
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:19 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 500 44 "-" "WaterdropClient" 8ed3e8ae-0e19-11e7-b967-000000000000 11135122
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:19 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 500 44 "-" "WaterdropClient" 8ef6c833-0e19-11e7-b968-000000000000 10907122
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:19 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 500 44 "-" "WaterdropClient" 8f09be7d-0e19-11e7-b969-000000000000 10783810
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:20 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 500 44 "-" "WaterdropClient" 8f73ff94-0e19-11e7-b975-000000000000 10946147
[tsm1] 2017/03/21 17:34:31 beginning full compaction of group 0, 2 TSM files
[tsm1] 2017/03/21 17:34:31 compacting full group (0) /home/waterdrop/shared/data/coldcloud/default/534/000000193-000000002.tsm (#0)
[tsm1] 2017/03/21 17:34:31 compacting full group (0) /home/waterdrop/shared/data/coldcloud/default/534/000000194-000000001.tsm (#1)
[tsm1] 2017/03/21 17:34:31 beginning full compaction of group 0, 2 TSM files
[tsm1] 2017/03/21 17:34:31 compacting full group (0) /home/waterdrop/shared/data/coldcloud/default/516/000000201-000000003.tsm (#0)
[tsm1] 2017/03/21 17:34:31 compacting full group (0) /home/waterdrop/shared/data/coldcloud/default/516/000000202-000000001.tsm (#1)
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:20 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 500 44 "-" "WaterdropClient" 8fc566c8-0e19-11e7-b976-000000000000 10447943
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:20 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 500 44 "-" "WaterdropClient" 8fc57180-0e19-11e7-b977-000000000000 10448705
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:20 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 500 44 "-" "WaterdropClient" 8fc575cc-0e19-11e7-b978-000000000000 10449673
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:20 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 500 44 "-" "WaterdropClient" 8fd6d6d8-0e19-11e7-b979-000000000000 10337004
[tsm1] 2017/03/21 17:34:31 beginning full compaction of group 0, 2 TSM files
[tsm1] 2017/03/21 17:34:31 compacting full group (0) /home/waterdrop/shared/data/coldcloud/default/525/000000209-000000002.tsm (#0)
[tsm1] 2017/03/21 17:34:31 compacting full group (0) /home/waterdrop/shared/data/coldcloud/default/525/000000210-000000001.tsm (#1)
[tsm1] 2017/03/21 17:34:31 beginning full compaction of group 0, 2 TSM files
[tsm1] 2017/03/21 17:34:31 compacting full group (0) /home/waterdrop/shared/data/coldcloud/default/54/000000165-000000004.tsm (#0)
[tsm1] 2017/03/21 17:34:31 compacting full group (0) /home/waterdrop/shared/data/coldcloud/default/54/000000166-000000001.tsm (#1)
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:21 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 500 44 "-" "WaterdropClient" 8ffd3cf8-0e19-11e7-b97b-000000000000 10247253
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:21 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 500 44 "-" "WaterdropClient" 901d503e-0e19-11e7-b97c-000000000000 10604369
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:20 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 500 44 "-" "WaterdropClient" 8f5ea93f-0e19-11e7-b96e-000000000000 12024206
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:20 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 500 44 "-" "WaterdropClient" 8f5eb1aa-0e19-11e7-b96f-000000000000 12025357
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:20 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 500 44 "-" "WaterdropClient" 8f5eb977-0e19-11e7-b970-000000000000 12026414
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:20 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 500 44 "-" "WaterdropClient" 8f5ebebc-0e19-11e7-b971-000000000000 12027578
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:20 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 500 44 "-" "WaterdropClient" 8f5ec470-0e19-11e7-b972-000000000000 12028670
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:20 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 500 44 "-" "WaterdropClient" 8f5eca0e-0e19-11e7-b973-000000000000 12029953
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:20 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 500 44 "-" "WaterdropClient" 8f5ed064-0e19-11e7-b974-000000000000 12032007
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:21 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 500 44 "-" "WaterdropClient" 9047158f-0e19-11e7-b97d-000000000000 11285141
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:21 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 500 44 "-" "WaterdropClient" 9048f01a-0e19-11e7-b97e-000000000000 11274564
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:21 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 500 44 "-" "WaterdropClient" 9048fa91-0e19-11e7-b97f-000000000000 11275961
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:21 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 500 44 "-" "WaterdropClient" 90490c5f-0e19-11e7-b980-000000000000 11276894
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:21 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 500 44 "-" "WaterdropClient" 9049124b-0e19-11e7-b981-000000000000 11278135
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:21 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 500 44 "-" "WaterdropClient" 904917ae-0e19-11e7-b982-000000000000 11279175
[tsm1] 2017/03/21 17:34:32 beginning full compaction of group 0, 2 TSM files
[tsm1] 2017/03/21 17:34:32 beginning full compaction of group 0, 2 TSM files
[tsm1] 2017/03/21 17:34:32 compacting full group (0) /home/waterdrop/shared/data/coldcloud/default/573/000000485-000000002.tsm (#0)
[tsm1] 2017/03/21 17:34:32 compacting full group (0) /home/waterdrop/shared/data/coldcloud/default/573/000000486-000000001.tsm (#1)
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:22 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 500 44 "-" "WaterdropClient" 90cd99e6-0e19-11e7-b983-000000000000 10577989
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:22 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 500 44 "-" "WaterdropClient" 90ef1308-0e19-11e7-b984-000000000000 10379291
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:22 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 500 44 "-" "WaterdropClient" 90ef1cb0-0e19-11e7-b985-000000000000 10380520
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:22 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 500 44 "-" "WaterdropClient" 910b7a1b-0e19-11e7-b986-000000000000 10253825
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:22 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 500 44 "-" "WaterdropClient" 910b88dd-0e19-11e7-b988-000000000000 10255241
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:22 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 500 44 "-" "WaterdropClient" 910b8e70-0e19-11e7-b989-000000000000 10256472
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:22 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 500 44 "-" "WaterdropClient" 910b9402-0e19-11e7-b98a-000000000000 10257501
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:22 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 500 44 "-" "WaterdropClient" 910b9956-0e19-11e7-b98b-000000000000 10258520
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:22 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 500 44 "-" "WaterdropClient" 910b9ede-0e19-11e7-b98c-000000000000 10259953
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:22 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 500 44 "-" "WaterdropClient" 910ba444-0e19-11e7-b98d-000000000000 10261450
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:22 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 500 44 "-" "WaterdropClient" 910ba9f8-0e19-11e7-b98e-000000000000 10263283
[tsm1] 2017/03/21 17:34:33 compacting full group (0) /home/waterdrop/shared/data/coldcloud/default/543/000000246-000000003.tsm (#0)
[tsm1] 2017/03/21 17:34:33 compacting full group (0) /home/waterdrop/shared/data/coldcloud/default/543/000000247-000000001.tsm (#1)
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:23 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 500 44 "-" "WaterdropClient" 9161c901-0e19-11e7-b991-000000000000 10110631
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:23 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 500 44 "-" "WaterdropClient" 913e4759-0e19-11e7-b990-000000000000 10472902
[tsm1] 2017/03/21 17:34:34 beginning full compaction of group 0, 2 TSM files
[tsm1] 2017/03/21 17:34:34 compacting full group (0) /home/waterdrop/shared/data/coldcloud/default/554/000000276-000000002.tsm (#0)
[tsm1] 2017/03/21 17:34:34 compacting full group (0) /home/waterdrop/shared/data/coldcloud/default/554/000000277-000000001.tsm (#1)
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:24 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 500 44 "-" "WaterdropClient" 91f645a7-0e19-11e7-b994-000000000000 11760611
[tsm1] 2017/03/21 17:34:37 beginning full compaction of group 0, 2 TSM files
[tsm1] 2017/03/21 17:34:37 compacting full group (0) /home/waterdrop/shared/data/coldcloud/default/61/000000157-000000003.tsm (#0)
[tsm1] 2017/03/21 17:34:37 compacting full group (0) /home/waterdrop/shared/data/coldcloud/default/61/000000158-000000001.tsm (#1)
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:26 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 500 44 "-" "WaterdropClient" 92f808f4-0e19-11e7-b995-000000000000 11135642
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:24 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 500 44 "-" "WaterdropClient" 91d20623-0e19-11e7-b993-000000000000 13093404
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:23 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 500 44 "-" "WaterdropClient" 91ae27ee-0e19-11e7-b992-000000000000 13493699
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:27 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 500 44 "-" "WaterdropClient" 93889fb8-0e19-11e7-b996-000000000000 10560510
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:27 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 500 44 "-" "WaterdropClient" 939bd141-0e19-11e7-b997-000000000000 10435935
[httpd] 127.0.0.1 - ccqawriter [21/Mar/2017:17:34:37 +0800] "POST /write?consistency=&db=coldcloudqa&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" 99b95340-0e19-11e7-b99a-000000000000 1493496
[tsm1] 2017/03/21 17:34:39 beginning full compaction of group 0, 2 TSM files
[tsm1] 2017/03/21 17:34:39 compacting full group (0) /home/waterdrop/shared/data/coldcloud/default/629/000000427-000000002.tsm (#0)
[tsm1] 2017/03/21 17:34:39 compacting full group (0) /home/waterdrop/shared/data/coldcloud/default/629/000000428-000000001.tsm (#1)
[tsm1] 2017/03/21 17:34:39 beginning full compaction of group 0, 2 TSM files
[tsm1] 2017/03/21 17:34:39 compacting full group (0) /home/waterdrop/shared/data/coldcloud/default/650/000000460-000000003.tsm (#0)
[tsm1] 2017/03/21 17:34:39 compacting full group (0) /home/waterdrop/shared/data/coldcloud/default/650/000000462-000000001.tsm (#1)
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:28 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 500 44 "-" "WaterdropClient" 94799502-0e19-11e7-b998-000000000000 11136238
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:29 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 500 44 "-" "WaterdropClient" 952eb80d-0e19-11e7-b999-000000000000 10001667
[query] 2017/03/21 17:34:49 SELECT * FROM coldcloud."default".c21161635 WHERE time > 1490074475s AND time < 1490088877s
[retention] 2017/03/21 17:34:49 retention policy shard deletion check commencing
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:44 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 500 44 "-" "WaterdropClient" 9e2d732d-0e19-11e7-b99c-000000000000 10449982
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:34:44 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 500 44 "-" "WaterdropClient" 9e075493-0e19-11e7-b99b-000000000000 11531881
[httpd] 10.163.218.131 - coldcloud [21/Mar/2017:17:34:47 +0800] "GET /query?db=coldcloud&epoch=s&q=select+%2A+from+c21161635+where+time+%3E+1490074475s+and+time+%3C+1490088877s HTTP/1.1" 200 1152 "-" "WaterdropClient" 9f8e3872-0e19-11e7-b99d-000000000000 10196538
[httpd] 127.0.0.1 - ccqawriter [21/Mar/2017:17:35:07 +0800] "POST /write?consistency=&db=coldcloudqa&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" ab62237f-0e19-11e7-b99f-000000000000 1419715
[httpd] 127.0.0.1 - ccqawriter [21/Mar/2017:17:35:08 +0800] "POST /write?consistency=&db=coldcloudqa&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" ac2429e6-0e19-11e7-b9a5-000000000000 3462071
[httpd] 127.0.0.1 - ccqawriter [21/Mar/2017:17:35:07 +0800] "POST /write?consistency=&db=coldcloudqa&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" abd377ff-0e19-11e7-b9a1-000000000000 5847263
[httpd] 127.0.0.1 - ccqawriter [21/Mar/2017:17:35:07 +0800] "POST /write?consistency=&db=coldcloudqa&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" abe8108f-0e19-11e7-b9a2-000000000000 8202915
[httpd] 127.0.0.1 - ccqawriter [21/Mar/2017:17:35:07 +0800] "POST /write?consistency=&db=coldcloudqa&precision=s&rp= HTTP/1.1" 500 44 "-" "WaterdropClient" ab622cef-0e19-11e7-b9a0-000000000000 10109600
[httpd] 127.0.0.1 - ccqawriter [21/Mar/2017:17:35:07 +0800] "POST /write?consistency=&db=coldcloudqa&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" abe819a7-0e19-11e7-b9a3-000000000000 9464099
[httpd] 127.0.0.1 - ccwriter [21/Mar/2017:17:35:04 +0800] "POST /write?consistency=&db=coldcloud&precision=s&rp= HTTP/1.1" 500 44 "-" "WaterdropClient" aa01f485-0e19-11e7-b99e-000000000000 12877947
[httpd] 127.0.0.1 - ccqawriter [21/Mar/2017:17:35:07 +0800] "POST /write?consistency=&db=coldcloudqa&precision=s&rp= HTTP/1.1" 204 0 "-" "WaterdropClient" abe820ca-0e19-11e7-b9a4-000000000000 10343436
[query] 2017/03/21 17:35:18 SELECT * FROM coldcloud."default".c21160460 ORDER BY time DESC LIMIT 1

[e-dard]

Can you provide more detail please, as outlined in the Issue template?

[zhang8h]

I see the log found every time the disk read to 20M per second, the log will appear a lot of the following information

[I] 2017-03-22T00:04:37Z beginning full compaction of group 0, 2 TSM files engine=tsm1
[I] 2017-03-22T00:04:37Z compacting full group (0) /home/waterdrop/shared/data/coldcloud/default/407/000000190-000000002.tsm (#0) engine=tsm1
[I] 2017-03-22T00:04:37Z compacting full group (0) /home/waterdrop/shared/data/coldcloud/default/407/000000191-000000001.tsm (#1) engine=tsm1
[I] 2017-03-22T00:04:38Z beginning full compaction of group 0, 2 TSM files engine=tsm1
[I] 2017-03-22T00:04:38Z compacting full group (0) /home/waterdrop/shared/data/coldcloud/default/418/000000179-000000002.tsm (#0) engine=tsm1
[I] 2017-03-22T00:04:38Z compacting full group (0) /home/waterdrop/shared/data/coldcloud/default/418/000000180-000000001.tsm (#1) engine=tsm1

[zhang8h]

Just upgraded to 1.2, but still the problem

[jwilder]

We need more information in order to diagnose what is going on. Can you update the issue description with the instructions listed in our issue template?

Specifically, please collect these and attach your config used for the server:

curl -o block.txt "http://localhost:8086/debug/pprof/block?debug=1" 
curl -o goroutine.txt "http://localhost:8086/debug/pprof/goroutine?debug=1" 
curl -o heap.txt "http://localhost:8086/debug/pprof/heap?debug=1" 
curl -o vars.txt "http://localhost:8086/debug/vars" 
iostat -xd 1 30 > iostat.txt
influx -execute "show shards" > shards.txt
influx -execute "show stats" > stats.txt
influx -execute "show diagnostics" > diagnostics.txt

[zhang8h]

block.txt
diagnostics.txt
goroutine.txt
heap.txt
iostat.txt
shards.txt
stats.txt
vars.txt

[zhang8h]

My server information
system:Ubuntu 14.04 LTS \n \l - 64BIT
cpu:2*CPU - Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz Intel(R) Xeon(R) CPU E5-2630 0 @
ram:4G

[zhang8h]

The log information below keeps repeating everytime when there is a extremely high read speed, usually at 20Mb/s.

[I] 2017-03-22T00:04:37Z beginning full compaction of group 0, 2 TSM files engine=tsm1
[I] 2017-03-22T00:04:37Z compacting full group (0) /home/waterdrop/shared/data/coldcloud/default/407/000000190-000000002.tsm (#0) engine=tsm1
[I] 2017-03-22T00:04:37Z compacting full group (0) /home/waterdrop/shared/data/coldcloud/default/407/000000191-000000001.tsm (#1) engine=tsm1
[I] 2017-03-22T00:04:38Z beginning full compaction of group 0, 2 TSM files engine=tsm1
[I] 2017-03-22T00:04:38Z compacting full group (0) /home/waterdrop/shared/data/coldcloud/default/418/000000179-000000002.tsm (#0) engine=tsm1
[I] 2017-03-22T00:04:38Z compacting full group (0) /home/waterdrop/shared/data/coldcloud/default/418/000000180-000000001.tsm (#1) engine=tsm1

[hpbieker]

zhang8h, the "full compaction" is run from time to time to merge multiple .tsm files into one or more 2 GB .tsm files (the last one will be smaller). As you notice from the log above, it is rewriting shard 407 and 418. If you do a ls -l in those directories you will see that it will write one or more .tsm.tmp files when the log shows "compacting", and it will finally rename the file to just .tsm and remove the old files. When it is done it shows "compacted" in the log. Because compacting will rewrite the whole shard, it will have high read and write IO usage. Furthermore, it will be lots of processing, so the CPU will typically also be high.

However, this should not prevent other queries from running according to the documentation.

[ghost]

I have the same problem ....

When it is compacting it has too high read and write IO usage and this provokes high "CPU iowait" usage what causes that the machine is blocked even when the compaction is done...

[zhang8h]

I have a query which frequently querying the database, the problem disappeared after I removed this query. Is there a possibility that this may related to frequently querying the database?

[jwilder]

TSM files are memory mapped so the OS will swap blocks in and out of RAM as needed. If you are querying across datasets that are not in the page cache, disk IO would be higher.

Can you upgrade to 1.2.2 and see if the write timeouts still occur?

[zhang8h]

I have upgraded the RAM to 8G recently and the problem is gone, could the problem possibly because the RAM is not large enough before?
P.S. my version is already 1.2.2

[bsmarcosj]

If we reduce the "compact-full-write-cold-duration" in the conf file, would be this problem solved since there would be less data to compact?

Which is the default value for that setting?