High disk write load and flood of memcached logs

Couchbase Server
1

Hi Guys,

I’m running a single node instance of Couchbase server version 4.0.0-4051 Community Edition on Ubuntu server 14.04 Trusty, with 32GB of Ram, 12-core CPU and a 7200rpm SATA disk.

I’m new to Couchbase and I wanted to know if this kind of behavior is normal. I’ve noticed a high write rate for the kind of usage pattern that I have and that the memcached logs are flooding with some DCP warning about mapreduce views.

Vmstat:

$ vmstat 1 -w
procs ---------------memory-------------- ---swap-- -----io---- -system-- ------cpu-----
 r  b     swpd     free     buff    cache   si   so    bi    bo   in   cs us sy id wa st
10  0     1292  5984588   163240  8063812    0    0    33  1325   58   56 31 10 56  2  0
 8  0     1292  5987572   163240  8068032    0    0    80 29040 33151 107624 34 10 53  2  0
25  2     1292  5975380   163240  8069952    0    0    20 11804 39066 158195 45 15 38  2  0
 6  1     1292  5953288   163244  8080656    0    0   152 13740 42349 135591 47 14 37  2  0
12  0     1292  5973960   163244  8082256    0    0    96 30160 30307 114487 25  9 63  4  0
 4  0     1292  5983900   163244  8083832    0    0    40 12264 30247 117545 25  8 64  3  0
 3  1     1292  5989692   163244  8085244    0    0    72 40576 30516 106294 23  9 65  4  0
11  0     1292  5968504   163244  8087148    0    0    44 12184 39259 137938 47 15 36  3  0
17  0     1292  5949096   163244  8091008    0    0    36 13488 39302 180319 53 16 28  3  0
 7  0     1292  5932696   163244  8100948    0    0    88 11744 30099 109528 49 14 35  2  0
 4  3     1292  5946340   163244  8100252    0    0    64 11636 28364 100678 34  9 54  3  0
 4  1     1292  5988744   163244  8064152    0    0    40 25140 32058 80722 31  9 58  3  0
11  1     1292  5973316   163244  8067924    0    0    56 12520 40898 147670 45 15 38  2  0
15  0     1292  5974744   163244  8069488    0    0    20 11324 30937 142078 64 13 21  2  0
11  0     1292  5979700   163244  8077252    0    0    60 29624 28353 121686 61 11 25  3  0
 8  0     1292  6011816   163244  8057428    0    0   120 11560 31245 118471 53  9 34  3  0
 2  1     1292  6020152   163244  8058832    0    0   116 28932 32282 120756 30  8 57  4  0
33  1     1292  5997756   163244  8060980    0    0    64 13508 38587 160636 45 15 37  3  0
22  2     1292  5981820   163244  8067688    0    0   300 13588 40300 152140 54 16 27  3  0
14  0     1292  5971716   163244  8077336    0    0    32 13028 26944 111572 50 13 34  3  0
 5  0     1292  5977652   163244  8082708    0    0    92 11392 28576 108490 33  9 53  5  0
 6  0     1292  5986700   163244  8083080    0    0   152 23648 33961 120820 25  9 63  2  0
21  1     1292  5966580   163244  8086404    0    0    72 12840 40261 132412 43 14 39  3  0
11  1     1292  5972724   163244  8081204    0    0    76 18112 41593 153637 51 15 32  2  0
 3  0     1292  5972532   163244  8088292    0    0    44 12952 31868 111139 30 10 57  3  0
 2  1     1292  5980148   163244  8089808    0    0    56 29380 31800 106241 24  9 64  3  0
 3  1     1292  5987848   163244  8091072    0    0    68 12060 31026 122062 23  8 66  4  0
25  2     1292  5960280   163244  8095444    0    0   112 12928 40531 149005 49 16 33  3  0
17  0     1292  5941520   163244  8103904    0    0    16 14480 34590 145383 55 15 28  3  0
12  1     1292  5940472   163244  8108344    0    0    56 13008 30006 117541 43 13 41  3  0
 4  1     1292  5950756   163244  8110276    0    0   132 13536 33571 103573 28 10 58  4  0
12  2     1292  5955388   163244  8111892    0    0    76 17652 31322 101617 22  8 67  3  0
10  0     1292  5945900   163248  8115972    0    0   124 13512 40253 142894 44 15 38  3  0
 9  1     1292  5936196   163248  8118672    0    0    88 13368 46383 131377 49 16 33  3  0
 8  0     1292  5943140   163248  8126240    0    0   124 12764 27824 128100 29  9 59  3  0
 9  0     1292  5952688   163248  8127672    0    0    28 10992 34174 133653 25  9 64  2  0
 5  1     1292  5960892   163248  8124996    0    0   100 39732 35632 108649 25 10 62  4  0
12  2     1292  5931728   163256  8132984    0    0   156 30048 46104 141779 48 16 32  4  0
13  0     1292  5910504   163256  8137700    0    0    76 13076 40630 162826 55 16 26  3  0
16  1     1292  5888084   163256  8147928    0    0    36 11728 30343 120382 50 14 34  2  0
 5  1     1292  5901488   163256  8155316    0    0   112 13392 31974 114691 37 11 47  4  0
 4  0     1292  5914444   163256  8154724    0    0    32 40080 37219 131212 29 11 56  4  0
12  1     1292  5902244   163256  8156880    0    0    80 13264 40681 136653 45 15 35  5  0
 9  1     1292  5899176   163256  8161124    0    0    76 12840 41613 145213 50 16 32  2  0
 5  0     1292  5893980   163256  8168844    0    0     0 12308 29749 120559 31 10 56  4  0
 3  1     1292  5903940   163256  8170008    0    0     0 12292 34493 135681 26  9 62  2  0

Iotop:

$ iotop -a -o
Total DISK READ :      67.42 K/s | Total DISK WRITE :      38.87 M/s
Actual DISK READ:      67.42 K/s | Actual DISK WRITE:      28.39 M/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND
  280 be/3 root          0.00 B      2.61 M  0.00 % 29.38 % [jbd2/dm-0-8]
 1493 be/4 couchbas     68.00 K     34.80 M  0.00 %  8.94 % memcached -C /opt/couchbase/var/lib/couchbase/config/memcached.json
 1492 be/4 couchbas     96.00 K     34.43 M  0.00 %  8.75 % memcached -C /opt/couchbase/var/lib/couchbase/config/memcached.json
 1490 be/4 couchbas    144.00 K     33.83 M  0.00 %  8.70 % memcached -C /opt/couchbase/var/lib/couchbase/config/memcached.json
 1491 be/4 couchbas    112.00 K     45.93 M  0.00 %  8.49 % memcached -C /opt/couchbase/var/lib/couchbase/config/memcached.json
 1486 be/4 couchbas    520.00 K      0.00 B  0.00 %  1.33 % memcached -C /opt/couchbase/var/lib/couchbase/config/memcached.json
 1488 be/4 couchbas    500.00 K      0.00 B  0.00 %  1.22 % memcached -C /opt/couchbase/var/lib/couchbase/config/memcached.json
 1487 be/4 couchbas    432.00 K      0.00 B  0.00 %  0.80 % memcached -C /opt/couchbase/var/lib/couchbase/config/memcached.json
 1489 be/4 couchbas    344.00 K      0.00 B  0.00 %  0.80 % memcached -C /opt/couchbase/var/lib/couchbase/config/memcached.json
 1494 be/4 couchbas      0.00 B      0.00 B  0.00 %  0.20 % memcached -C /opt/couchbase/var/lib/couchbase/config/memcached.json
21926 be/4 root          0.00 B      0.00 B  0.00 %  0.05 % [kworker/u28:1]
 1201 be/4 couchbas      0.00 B     34.61 M  0.00 %  0.04 % beam.smp -A 16 -sbt u -P 327680 -K true -swt low -MMmcs 30 -e102400 ~/couchdb/default.d/geocouch.ini /opt/couchbase/etc/couchdb/local.ini
 1247 be/4 couchbas      0.00 B      0.00 B  0.00 %  0.04 % beam.smp -P 327680 -K true -- -root /opt/couchbase/lib/erlang -progn~b/local.ini -setcookie mjqtrgmvxdvyalyl -name couchdb_ns_1@127.0.0.1
 1246 be/4 couchbas      0.00 B      0.00 B  0.00 %  0.03 % beam.smp -P 327680 -K true -- -root /opt/couchbase/lib/erlang -progn~b/local.ini -setcookie mjqtrgmvxdvyalyl -name couchdb_ns_1@127.0.0.1
 1205 be/4 couchbas      0.00 B     17.25 M  0.00 %  0.03 % beam.smp -A 16 -sbt u -P 327680 -K true -swt low -MMmcs 30 -e102400 ~/couchdb/default.d/geocouch.ini /opt/couchbase/etc/couchdb/local.ini
 1199 be/4 couchbas      0.00 B     17.25 M  0.00 %  0.02 % beam.smp -A 16 -sbt u -P 327680 -K true -swt low -MMmcs 30 -e102400 ~/couchdb/default.d/geocouch.ini /opt/couchbase/etc/couchdb/local.ini
22994 be/4 root          0.00 B      0.00 B  0.00 %  0.02 % [kworker/u25:2]
28761 be/4 couchbas      0.00 B     21.12 M  0.00 %  0.80 % couch_view_group_compactor
 1457 be/4 couchbas      0.00 B     27.01 M  0.00 %  0.01 % memcached -C /opt/couchbase/var/lib/couchbase/config/memcached.json
 1252 be/4 couchbas      0.00 B      2.44 M  0.00 %  0.01 % beam.smp -P 327680 -K true -- -root /opt/couchbase/lib/erlang -progn~b/local.ini -setcookie mjqtrgmvxdvyalyl -name couchdb_ns_1@127.0.0.1
 1249 be/4 couchbas      0.00 B      2.88 M  0.00 %  0.01 % beam.smp -P 327680 -K true -- -root /opt/couchbase/lib/erlang -progn~b/local.ini -setcookie mjqtrgmvxdvyalyl -name couchdb_ns_1@127.0.0.1
 1245 be/4 couchbas      0.00 B   1384.00 K  0.00 %  0.01 % beam.smp -P 327680 -K true -- -root /opt/couchbase/lib/erlang -progn~b/local.ini -setcookie mjqtrgmvxdvyalyl -name couchdb_ns_1@127.0.0.1
 1248 be/4 couchbas      0.00 B     10.88 M  0.00 %  0.01 % beam.smp -P 327680 -K true -- -root /opt/couchbase/lib/erlang -progn~b/local.ini -setcookie mjqtrgmvxdvyalyl -name couchdb_ns_1@127.0.0.1
 1254 be/4 couchbas      0.00 B     10.52 M  0.00 %  0.01 % beam.smp -P 327680 -K true -- -root /opt/couchbase/lib/erlang -progn~b/local.ini -setcookie mjqtrgmvxdvyalyl -name couchdb_ns_1@127.0.0.1
 1253 be/4 couchbas      0.00 B      2.00 M  0.00 %  0.01 % beam.smp -P 327680 -K true -- -root /opt/couchbase/lib/erlang -progn~b/local.ini -setcookie mjqtrgmvxdvyalyl -name couchdb_ns_1@127.0.0.1
 1197 be/4 couchbas      0.00 B     17.25 M  0.00 %  0.00 % beam.smp -A 16 -sbt u -P 327680 -K true -swt low -MMmcs 30 -e102400 ~/couchdb/default.d/geocouch.ini /opt/couchbase/etc/couchdb/local.ini
 1251 be/4 couchbas      0.00 B      0.00 B  0.00 %  0.00 % beam.smp -P 327680 -K true -- -root /opt/couchbase/lib/erlang -progn~b/local.ini -setcookie mjqtrgmvxdvyalyl -name couchdb_ns_1@127.0.0.1
 1198 be/4 couchbas      0.00 B     17.25 M  0.00 %  0.00 % beam.smp -A 16 -sbt u -P 327680 -K true -swt low -MMmcs 30 -e102400 ~/couchdb/default.d/geocouch.ini /opt/couchbase/etc/couchdb/local.ini
 1295 be/4 couchbas      0.00 B      0.00 B  0.00 %  0.00 % godu
 1250 be/4 couchbas      0.00 B   1084.00 K  0.00 %  0.00 % beam.smp -P 327680 -K true -- -root /opt/couchbase/lib/erlang -progn~b/local.ini -setcookie mjqtrgmvxdvyalyl -name couchdb_ns_1@127.0.0.1
 1299 be/4 couchbas      0.00 B      0.00 B  0.00 %  0.00 % godu
 1260 be/4 couchbas      0.00 B      0.00 B  0.00 %  0.00 % beam.smp -P 327680 -K true -- -root /opt/couchbase/lib/erlang -progn~b/local.ini -setcookie mjqtrgmvxdvyalyl -name couchdb_ns_1@127.0.0.1
10811 be/4 couchbas      0.00 B      0.00 B  0.00 %  0.00 % goxdcr -sourceKVAdminPort=8091 -xdcrRestPort=9998 -isEnterprise=false
 1214 be/4 couchbas      0.00 B      0.00 B  0.00 %  0.00 % beam.smp -A 16 -sbt u -P 327680 -K true -swt low -MMmcs 30 -e102400 ~/couchdb/default.d/geocouch.ini /opt/couchbase/etc/couchdb/local.ini
 1449 be/4 couchbas      0.00 B      0.00 B  0.00 %  0.00 % goxdcr -sourceKVAdminPort=8091 -xdcrRestPort=9998 -isEnterprise=false
 1202 be/4 couchbas      0.00 B    604.00 K  0.00 %  0.00 % beam.smp -A 16 -sbt u -P 327680 -K true -swt low -MMmcs 30 -e102400 ~/couchdb/default.d/geocouch.ini /opt/couchbase/etc/couchdb/local.ini
 1155 be/4 couchbas      0.00 B   2000.00 K  0.00 %  0.00 % beam.smp -A 16 -- -root /opt/couchbase/lib/erlang -progname erl -- -~/couchbase-server.cookie-ns-server" -ns_server enable_mlockall false
 1485 be/4 couchbas      0.00 B      0.00 B  0.00 %  0.00 % goxdcr -sourceKVAdminPort=8091 -xdcrRestPort=9998 -isEnterprise=false
 1451 be/4 couchbas      0.00 B      0.00 B  0.00 %  0.00 % goxdcr -sourceKVAdminPort=8091 -xdcrRestPort=9998 -isEnterprise=false
18707 be/4 couchbas      0.00 B      0.00 B  0.00 %  0.00 % goxdcr -sourceKVAdminPort=8091 -xdcrRestPort=9998 -isEnterprise=false
 1896 be/4 couchbas      0.00 B      0.00 B  0.00 %  0.00 % goxdcr -sourceKVAdminPort=8091 -xdcrRestPort=9998 -isEnterprise=false
 1263 be/4 couchbas      0.00 B      0.00 B  0.00 %  0.00 % beam.smp -P 327680 -K true -- -root /opt/couchbase/lib/erlang -progn~b/local.ini -setcookie mjqtrgmvxdvyalyl -name couchdb_ns_1@127.0.0.1
 1216 be/4 couchbas      0.00 B      0.00 B  0.00 %  0.00 % beam.smp -A 16 -sbt u -P 327680 -K true -swt low -MMmcs 30 -e102400 ~/couchdb/default.d/geocouch.ini /opt/couchbase/etc/couchdb/local.ini
 1543 be/4 couchbas      0.00 B      0.00 B  0.00 %  0.00 % goxdcr -sourceKVAdminPort=8091 -xdcrRestPort=9998 -isEnterprise=false
 1219 be/4 couchbas      0.00 B      0.00 B  0.00 %  0.00 % beam.smp -A 16 -sbt u -P 327680 -K true -swt low -MMmcs 30 -e102400 ~/couchdb/default.d/geocouch.ini /opt/couchbase/etc/couchdb/local.ini
 1209 be/4 couchbas      0.00 B      0.00 B  0.00 %  0.00 % beam.smp -A 16 -sbt u -P 327680 -K true -swt low -MMmcs 30 -e102400 ~/couchdb/default.d/geocouch.ini /opt/couchbase/etc/couchdb/local.ini
  639 be/4 syslog        0.00 B      8.00 K  0.00 %  0.00 % rsyslogd [rs:main Q:Reg]
 1156 be/4 couchbas      0.00 B      4.00 K  0.00 %  0.00 % beam.smp -A 16 -- -root /opt/couchbase/lib/erlang -progname erl -- -~/couchbase-server.cookie-ns-server" -ns_server enable_mlockall false
 1160 be/4 couchbas      0.00 B     56.00 K  0.00 %  0.00 % beam.smp -A 16 -- -root /opt/couchbase/lib/erlang -progname erl -- -~/couchbase-server.cookie-ns-server" -ns_server enable_mlockall false
 1191 be/4 couchbas      0.00 B     88.00 K  0.00 %  0.00 % beam.smp -A 16 -sbt u -P 327680 -K true -swt low -MMmcs 30 -e102400 ~/couchdb/default.d/geocouch.ini /opt/couchbase/etc/couchdb/local.ini
 1195 be/4 couchbas      0.00 B     24.00 K  0.00 %  0.00 % beam.smp -A 16 -sbt u -P 327680 -K true -swt low -MMmcs 30 -e102400 ~/couchdb/default.d/geocouch.ini /opt/couchbase/etc/couchdb/local.ini
 1196 be/4 couchbas      0.00 B     32.00 K  0.00 %  0.00 % beam.smp -A 16 -sbt u -P 327680 -K true -swt low -MMmcs 30 -e102400 ~/couchdb/default.d/geocouch.ini /opt/couchbase/etc/couchdb/local.ini

Look at the number of the memcached.log rotations:

$ ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.8*
/opt/couchbase/var/lib/couchbase/logs/memcached.log.829.txt  /opt/couchbase/var/lib/couchbase/logs/memcached.log.837.txt  /opt/couchbase/var/lib/couchbase/logs/memcached.log.845.txt
/opt/couchbase/var/lib/couchbase/logs/memcached.log.830.txt  /opt/couchbase/var/lib/couchbase/logs/memcached.log.838.txt  /opt/couchbase/var/lib/couchbase/logs/memcached.log.846.txt
/opt/couchbase/var/lib/couchbase/logs/memcached.log.831.txt  /opt/couchbase/var/lib/couchbase/logs/memcached.log.839.txt  /opt/couchbase/var/lib/couchbase/logs/memcached.log.847.txt
/opt/couchbase/var/lib/couchbase/logs/memcached.log.832.txt  /opt/couchbase/var/lib/couchbase/logs/memcached.log.840.txt  /opt/couchbase/var/lib/couchbase/logs/memcached.log.848.txt
/opt/couchbase/var/lib/couchbase/logs/memcached.log.833.txt  /opt/couchbase/var/lib/couchbase/logs/memcached.log.841.txt  /opt/couchbase/var/lib/couchbase/logs/memcached.log.849.txt
/opt/couchbase/var/lib/couchbase/logs/memcached.log.834.txt  /opt/couchbase/var/lib/couchbase/logs/memcached.log.842.txt  /opt/couchbase/var/lib/couchbase/logs/memcached.log.850.txt
/opt/couchbase/var/lib/couchbase/logs/memcached.log.835.txt  /opt/couchbase/var/lib/couchbase/logs/memcached.log.843.txt
/opt/couchbase/var/lib/couchbase/logs/memcached.log.836.txt  /opt/couchbase/var/lib/couchbase/logs/memcached.log.844.txt

Most of the content in memcached logs are with these messages:

2016-10-06T12:26:46.232170-03:00 WARNING (Turing) DCP (Producer) eq_dcpq:mapreduce_view: Turing _design/type_cbinfo (prod/main) - (vb 7) Backfill task (39205 to 39208) finished
2016-10-06T12:26:46.232470-03:00 WARNING (Turing) DCP (Producer) eq_dcpq:mapreduce_view: Turing _design/type_cbinfo (prod/main) - (vb 7) Stream closing, 4 items sent from backfill phase, 0 items sent from memory phase, 39208 was last seqno sent, reason: The stream ended due to all items being streamed
2016-10-06T12:26:46.233115-03:00 WARNING (Turing) DCP (Producer) eq_dcpq:mapreduce_view: Turing _design/ddl_messageStatistics_type_serverip_viewtype_timestamp (prod/main) - (vb 4) Sending disk snapshot with start seqno 32430 and end seqno 32432
2016-10-06T12:26:46.233510-03:00 WARNING (Turing) DCP (Producer) eq_dcpq:mapreduce_view: Turing _design/ddl_def_type (prod/main) - (vb 6) Sending disk snapshot with start seqno 65761 and end seqno 65765
2016-10-06T12:26:46.233820-03:00 WARNING (Turing) DCP (Producer) eq_dcpq:mapreduce_view: Turing _design/ddl_messageStatistics_type_serverip_viewtype_timestamp (prod/main) - (vb 4) Backfill complete, 0 items read from disk 2 from memory, last seqno read: 32432
2016-10-06T12:26:46.234031-03:00 WARNING (Turing) DCP (Producer) eq_dcpq:mapreduce_view: Turing _design/ddl_messageStatistics_type_serverip_viewtype_timestamp (prod/main) - (vb 4) Backfill task (32431 to 32431) finished
2016-10-06T12:26:46.234209-03:00 WARNING (Turing) DCP (Producer) eq_dcpq:mapreduce_view: Turing _design/ddl_#primary (prod/main) - (vb 8) Backfill complete, 0 items read from disk 2 from memory, last seqno read: 83741
2016-10-06T12:26:46.235438-03:00 WARNING (Turing) DCP (Producer) eq_dcpq:mapreduce_view: Turing _design/ddl_#primary (prod/main) - (vb 8) Backfill task (83740 to 83741) finished
2016-10-06T12:26:46.234146-03:00 WARNING (Turing) DCP (Producer) eq_dcpq:mapreduce_view: Turing _design/ddl_messageStatistics_type_serverip_viewtype_timestamp (prod/main) - (vb 4) Stream closing, 2 items sent from backfill phase, 0 items sent from memory phase, 32432 was last seqno sent, reason: The stream ended due to all items being streamed
2016-10-06T12:26:46.235404-03:00 WARNING (Turing) DCP (Producer) eq_dcpq:mapreduce_view: Turing _design/type_cbinfo (prod/main) - (vb 8) stream created with start seqno 83730 and end seqno 83741
2016-10-06T12:26:46.236258-03:00 WARNING (Turing) DCP (Producer) eq_dcpq:mapreduce_view: Turing _design/type_cbinfo (prod/main) - (vb 8) Sending disk snapshot with start seqno 83730 and end seqno 83741
2016-10-06T12:26:46.236526-03:00 WARNING (Turing) DCP (Producer) eq_dcpq:mapreduce_view: Turing _design/ddl_def_type (prod/main) - (vb 6) Backfill complete, 0 items read from disk 4 from memory, last seqno read: 65765
2016-10-06T12:26:46.236570-03:00 WARNING (Turing) DCP (Producer) eq_dcpq:mapreduce_view: Turing _design/ddl_def_type (prod/main) - (vb 6) Backfill task (65762 to 65765) finished
2016-10-06T12:26:46.236709-03:00 WARNING (Turing) DCP (Producer) eq_dcpq:mapreduce_view: Turing _design/type_cbinfo (prod/main) - (vb 8) Backfill complete, 0 items read from disk 11 from memory, last seqno read: 83741
2016-10-06T12:26:46.236738-03:00 WARNING (Turing) DCP (Producer) eq_dcpq:mapreduce_view: Turing _design/type_cbinfo (prod/main) - (vb 8) Backfill task (83731 to 83741) finished
2016-10-06T12:26:46.236823-03:00 WARNING (Turing) DCP (Producer) eq_dcpq:mapreduce_view: Turing _design/ddl_def_type (prod/main) - (vb 6) Stream closing, 4 items sent from backfill phase, 0 items sent from memory phase, 65765 was last seqno sent, reason: The stream ended due to all items being streamed
2016-10-06T12:26:46.235785-03:00 WARNING (Turing) DCP (Producer) eq_dcpq:mapreduce_view: Turing _design/ddl_#primary (prod/main) - (vb 8) Stream closing, 2 items sent from backfill phase, 0 items sent from memory phase, 83741 was last seqno sent, reason: The stream ended due to all items being streamed
2016-10-06T12:26:46.237234-03:00 WARNING (Turing) DCP (Producer) eq_dcpq:mapreduce_view: Turing _design/type_cbinfo (prod/main) - (vb 8) Stream closing, 11 items sent from backfill phase, 0 items sent from memory phase, 83741 was last seqno sent, reason: The stream ended due to all items being streamed
2016-10-06T12:26:46.241074-03:00 WARNING (Turing) DCP (Producer) eq_dcpq:mapreduce_view: Turing _design/ddl_#primary (prod/main) - (vb 9) stream created with start seqno 51561 and end seqno 51563
2016-10-06T12:26:46.241058-03:00 WARNING (Turing) DCP (Producer) eq_dcpq:mapreduce_view: Turing _design/ddl_def_type (prod/main) - (vb 7) stream created with start seqno 39207 and end seqno 39208
2016-10-06T12:26:46.241424-03:00 WARNING (Turing) DCP (Producer) eq_dcpq:mapreduce_view: Turing _design/ddl_#primary (prod/main) - (vb 9) Sending disk snapshot with start seqno 51561 and end seqno 51563
2016-10-06T12:26:46.241614-03:00 WARNING (Turing) DCP (Producer) eq_dcpq:mapreduce_view: Turing _design/ddl_def_type (prod/main) - (vb 7) Sending disk snapshot with start seqno 39207 and end seqno 39208
2016-10-06T12:26:46.241790-03:00 WARNING (Turing) DCP (Producer) eq_dcpq:mapreduce_view: Turing _design/ddl_#primary (prod/main) - (vb 9) Backfill complete, 0 items read from disk 2 from memory, last seqno read: 51563
2016-10-06T12:26:46.242004-03:00 WARNING (Turing) DCP (Producer) eq_dcpq:mapreduce_view: Turing _design/ddl_#primary (prod/main) - (vb 9) Backfill task (51562 to 51563) finished
2016-10-06T12:26:46.242068-03:00 WARNING (Turing) DCP (Producer) eq_dcpq:mapreduce_view: Turing _design/ddl_def_type (prod/main) - (vb 7) Backfill complete, 0 items read from disk 1 from memory, last seqno read: 39208
2016-10-06T12:26:46.242089-03:00 WARNING (Turing) DCP (Producer) eq_dcpq:mapreduce_view: Turing _design/ddl_def_type (prod/main) - (vb 7) Backfill task (39208 to 39208) finished
2016-10-06T12:26:46.241973-03:00 WARNING (Turing) DCP (Producer) eq_dcpq:mapreduce_view: Turing _design/type_cbinfo (prod/main) - (vb 9) stream created with start seqno 51558 and end seqno 51563
2016-10-06T12:26:46.242271-03:00 WARNING (Turing) DCP (Producer) eq_dcpq:mapreduce_view: Turing _design/ddl_def_type (prod/main) - (vb 7) Stream closing, 1 items sent from backfill phase, 0 items sent from memory phase, 39208 was last seqno sent, reason: The stream ended due to all items being streamed
2016-10-06T12:26:46.243181-03:00 WARNING (Turing) DCP (Producer) eq_dcpq:mapreduce_view: Turing _design/ddl_messageStatistics_type_serverip_viewtype_timestamp (prod/main) - (vb 5) stream created with start seqno 26193 and end seqno 26194
2016-10-06T12:26:46.243932-03:00 WARNING (Turing) DCP (Producer) eq_dcpq:mapreduce_view: Turing _design/ddl_def_type (prod/main) - (vb 8) stream created with start seqno 83739 and end seqno 83741
2016-10-06T12:26:46.245551-03:00 WARNING (Turing) DCP (Producer) eq_dcpq:mapreduce_view: Turing _design/ddl_#primary (prod/main) - (vb 9) Stream closing, 2 items sent from backfill phase, 0 items sent from memory phase, 51563 was last seqno sent, reason: The stream ended due to all items being streamed
2016-10-06T12:26:46.246307-03:00 WARNING (Turing) DCP (Producer) eq_dcpq:mapreduce_view: Turing _design/ddl_#primary (prod/main) - (vb 10) stream created with start seqno 47079 and end seqno 47082
2016-10-06T12:26:46.248330-03:00 WARNING (Turing) DCP (Producer) eq_dcpq:mapreduce_view: Turing _design/type_cbinfo (prod/main) - (vb 9) Sending disk snapshot with start seqno 51558 and end seqno 51563
2016-10-06T12:26:46.248524-03:00 WARNING (Turing) DCP (Producer) eq_dcpq:mapreduce_view: Turing _design/type_cbinfo (prod/main) - (vb 9) Backfill complete, 0 items read from disk 5 from memory, last seqno read: 51563
2016-10-06T12:26:46.248564-03:00 WARNING (Turing) DCP (Producer) eq_dcpq:mapreduce_view: Turing _design/type_cbinfo (prod/main) - (vb 9) Backfill task (51559 to 51563) finished
2016-10-06T12:26:46.248670-03:00 WARNING (Turing) DCP (Producer) eq_dcpq:mapreduce_view: Turing _design/type_cbinfo (prod/main) - (vb 9) Stream closing, 5 items sent from backfill phase, 0 items sent from memory phase, 51563 was last seqno sent, reason: The stream ended due to all items being streamed
2016-10-06T12:26:46.250007-03:00 WARNING (Turing) DCP (Producer) eq_dcpq:mapreduce_view: Turing _design/type_cbinfo (prod/main) - (vb 10) stream created with start seqno 47072 and end seqno 47082
2016-10-06T12:26:46.248676-03:00 WARNING (Turing) DCP (Producer) eq_dcpq:mapreduce_view: Turing _design/ddl_messageStatistics_type_serverip_viewtype_timestamp (prod/main) - (vb 5) Sending disk snapshot with start seqno 26193 and end seqno 26194
2016-10-06T12:26:46.252506-03:00 WARNING (Turing) DCP (Producer) eq_dcpq:mapreduce_view: Turing _design/ddl_def_type (prod/main) - (vb 8) Sending disk snapshot with start seqno 83739 and end seqno 83741

Are these two events related? Will I have to get an SSD to run this instance?

Thanks in advance!

2

Besides configuring full ejection on bucket settings. Are there any other configuration to alleviate heavy write loads?

3

We removed those extraneous log messages in a more recent build - not sure off the top of my head but either 4.1 or 4.5.

4

Thank you for your reply.

It seems that I only need to focus on optimising my write performance then. Do you have any suggestions?

I’ve tried mounting the partition with noative, barriers=0, reltime, commit=60, but it didnt help much. Swappiness=0 and transparent_hugepages=never are also setted.

Any help is appreciated.
Thank you

5

What are your requirements for "write performance?

Typically people are concerned with ensuring that the persistence queue is stable, and perhaps their persistTo latencies are below some business retirement.

Couchbase by its nature will spend a bunch of time writing to disk, but as long as you’re meeting your requirements I don’t see why disk IO should be a problem.

6

Hello Pedro,
Also, could be interesting monitor the disk I/O to see if any botleneck exits.
By example the command:

7

Hello Pedro,
Also, could be interesting monitor the disk I/O to see if any botleneck exits.
By example the command:
iostat -x 1

Show disk utilization like:
%útil - show the disk rate utilization
avgqu-sz - show the average queue size. (number of process waiting to Access the I/O device)

Also, latency in I/O operations normally need be less than 10ms.

Yo say that use 7200RPM disks. (this disks are slower).
There are also other factors such as if the disk where the buckets are stored are stored on disks in the array, using write back cache setting. This could improve the performance of read / write operations.

Also you can isolate Couchabase Bucket data in disks/filesystems dedicated.
By default the default directory is /var/… , share the operating system activity, or other applications.

Best Regards!
Arturo

8

Hi, guys.

I think we have the same problem with disk load on our environment.
We also use CB community edition v4.0.0-4051 on ubuntu 14.04 (virtual environment on Xen). We have only one node in cluster and have around 300 ops/s on couchbase and our disk is used for 99%.
We get from “iostat -mx 5”:
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 288.40 0.00 185.80 0.00 2.55 28.06 2.88 15.29 0.00 15.29 5.32 98.88
dm-0 0.00 0.00 0.00 474.40 0.00 2.54 10.96 6.69 14.03 0.00 14.03 2.09 98.96
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

Any advice to solve this issue?