I would like to know benchmark for syncgateway, am using syncgateway with below configurations
RAM : 4GB
SyncGW version : 1.2.1
docs : 20 M
channels : 20000
Couchbase : 3 nodes, 2 buckets
using webhooks and would like to know what is the ops/second can syncgw supports?
1 Like
A general rule of thumb ,based on your NIC and memory , SG can do about 5,000 to 10,000 concurrent users per SG node. One thing to consider is going to be write through put. If you write heavy READ/WRITE 10/90 you’ll going to want to use Sync Gateway Accelerator.
Here is an example config.
https://github.com/couchbase/sync_gateway/blob/master/examples/basic_sg_accel_config.json
DOWNLOAD HERE
1 Like
Thanks househippo,
am surely try Sync Gateway Accelerator. could you please what is the significance of Sync Gateway Accelerator.
currently we are testing for 100 ops WRITE and 50 ops READ . we are seeing sync-gateway memory leak issue and also delay in responses.
Sync Gateway Accelerator help scale Sync Gateway horizontally by removing the constraints of a global sequence number management from SG. Instead it uses Couchbase Server’s internal sequence number, called DCP.
This lets you scale SG from 2 - 3 servers to 10s, 100s or 1000s of servers connected to Couchbase Server.
There were some bad memory leaks in the older version of SG but all major one should have been fixed in 1.1
or you could have found a bug.
Response times can be logged like below by changing the SG config.json file
{
…
“logs”:[“HTTP”,“HTTP+”]
…
}
What are you trying to do with webhooks?
What is the number of ops syncgateway with 4GB can supports. i could see some response are more than 200ms. what are the scenarios for syncgateway delays in responses.
i could also see some warning in syncgateway error logs
2016/07/06 21:00:48 go-couchbase: call to ViewCustom(“sync_gateway”, “channels”) in github.com/couchbase/sync_gateway/db.(*DatabaseContext).getChangesInChannelFromView took 2.497996224s
2016/07/06 21:00:48 go-couchbase: call to ViewCustom(“sync_gateway”, “channels”) in github.com/couchbase/sync_gateway/db.(*DatabaseContext).getChangesInChannelFromView took 2.205767554s
2016/07/06 12:01:44 go-couchbase: call to Do(“sdefault.doc”) in github.com/couchbase/go-couchbase.(*Bucket).casNext took 271.958457ms
what is the meaning of these error. is it view call to couchbase is taking more time?
looks like it. How is your Couchbase server setup?
Is Swappiness=0 ?
Did the ulimits for the couchbase user get applied by the startup script?
There is a great blog that talks about the correct OS settings needed on the CB Server Side
Here is the OS settings for sync gateway. Couchbase Capella for Mobile Developers
both swappiness and THP both are set as per recommendations
we have set below in /etc/security/limits.conf
couchbase hard nofile 131072
couchbase soft nofile 131072
The above warnings we are getting when we are processing 100 ops documents and internal view index. not sure whether because of this we are getting delay from syncgateway.
is there any parameter we need tune ? what is ops a syncgateway with 4GB supports for 3 couchbase nodes.
also fetching some documents from couchbase is delay
2016/07/08 08:16:14 go-couchbase: call to Do(“s919591210434_Location.doc”) in github.com/couchbase/go-couchbase.(*Bucket).casNext took 241.42931ms
2016/07/08 08:35:50 go-couchbase: call to Do(“s919591214237_Location.doc”) in github.com/couchbase/go-couchbase.(*Bucket).casNext took 227.700976ms
go to the couchbase server and check the cbstats for response times for the GET()s for the keys.
#/opt/couchbase/bin/cbstats localhost:11210 timings -b {your bucket name} -p {bucket password}
You want to find the cmd_get it will show a histogram of time for key fetches.
Here are the docs Couchbase SDKs
thanks hippo,
i ran the cbstats and below are the results. not able to understand , cmd_get part avg showing in micro seconds.
set_with_meta (96418 total)
1us - 2us : ( 1.82%) 1753 #
2us - 4us : ( 14.08%) 11823 ############
4us - 8us : ( 26.11%) 11599 ############
8us - 16us : ( 33.00%) 6645 #######
16us - 32us : ( 36.25%) 3128 ###
32us - 64us : ( 37.78%) 1476 #
64us - 128us : ( 67.25%) 28419 ##############################
128us - 256us : ( 87.04%) 19083 ####################
256us - 512us : ( 94.05%) 6756 #######
512us - 1ms : ( 96.91%) 2758 ###
1ms - 2ms : ( 98.81%) 1826 #
2ms - 4ms : ( 99.62%) 783
4ms - 8ms : ( 99.93%) 305
8ms - 16ms : ( 99.99%) 54
16ms - 32ms : (100.00%) 9
32ms - 65ms : (100.00%) 1
Avg : ( 134us)
bg_wait (2895991 total)
2us - 4us : ( 0.00%) 11
4us - 8us : ( 0.03%) 997
8us - 16us : ( 29.81%) 862345 ##############################
16us - 32us : ( 79.01%) 1424787 ##################################################
32us - 64us : ( 86.98%) 230729 ########
64us - 128us : ( 91.65%) 135295 ####
128us - 256us : ( 94.45%) 81106 ##
256us - 512us : ( 96.68%) 64569 ##
512us - 1ms : ( 98.52%) 53343 #
1ms - 2ms : ( 99.53%) 29198 #
2ms - 4ms : ( 99.90%) 10611
4ms - 8ms : ( 99.99%) 2722
8ms - 16ms : (100.00%) 262
16ms - 32ms : (100.00%) 14
32ms - 65ms : (100.00%) 2
Avg : ( 57us)
get_stats_cmd (3733180 total)
0 - 1us : ( 0.01%) 408
1us - 2us : ( 0.10%) 3441
2us - 4us : ( 0.17%) 2647
4us - 8us : ( 0.18%) 124
8us - 16us : ( 0.18%) 42
16us - 32us : ( 0.18%) 13
32us - 64us : ( 2.88%) 100944 ##
64us - 128us : ( 15.06%) 454780 ############
128us - 256us : ( 18.64%) 133616 ###
256us - 512us : ( 40.63%) 820864 ######################
512us - 1ms : ( 70.86%) 1128613 ###############################
1ms - 2ms : ( 79.82%) 334371 #########
2ms - 4ms : ( 99.55%) 736367 ####################
4ms - 8ms : ( 99.60%) 1953
8ms - 16ms : ( 99.60%) 202
16ms - 32ms : ( 99.61%) 172
32ms - 65ms : ( 99.62%) 405
65ms - 131ms : ( 99.64%) 795
131ms - 262ms : ( 99.76%) 4402
262ms - 524ms : (100.00%) 8948
524ms - 1s : (100.00%) 73
Avg : ( 1ms)
disk_del (26392395 total)
0 - 1us : ( 72.22%) 19060351 #########################################################################
1us - 2us : ( 98.62%) 6967343 ##########################
2us - 4us : ( 99.61%) 261414 #
4us - 8us : ( 99.84%) 60213
8us - 16us : ( 99.98%) 38568
16us - 32us : (100.00%) 3427
32us - 64us : (100.00%) 391
64us - 128us : (100.00%) 158
128us - 256us : (100.00%) 148
256us - 512us : (100.00%) 98
512us - 1ms : (100.00%) 104
1ms - 2ms : (100.00%) 95
2ms - 4ms : (100.00%) 56
4ms - 8ms : (100.00%) 23
8ms - 16ms : (100.00%) 5
262ms - 524ms : (100.00%) 1
Avg : ( 1us)
del_vb_cmd (1364 total)
8us - 16us : ( 6.52%) 89 #######
16us - 32us : ( 59.82%) 727 #########################################################
32us - 64us : ( 96.85%) 505 #######################################
64us - 128us : (100.00%) 43 ###
Avg : ( 22us)
get_cmd (93018896 total)
0 - 1us : ( 21.84%) 20310947 ######################
1us - 2us : ( 50.51%) 26672977 #############################
2us - 4us : ( 63.74%) 12306365 #############
4us - 8us : ( 94.96%) 29040303 ###############################
8us - 16us : ( 99.16%) 3911452 ####
16us - 32us : ( 99.96%) 738844
32us - 64us : ( 99.99%) 33262
64us - 128us : (100.00%) 2937
128us - 256us : (100.00%) 974
256us - 512us : (100.00%) 455
512us - 1ms : (100.00%) 192
1ms - 2ms : (100.00%) 102
2ms - 4ms : (100.00%) 62
4ms - 8ms : (100.00%) 21
8ms - 16ms : (100.00%) 3
Avg : ( 2us)
storage_age (92635975 total)
0 - 1s : ( 96.74%) 89612382 ######################################################################################################
1s - 2s : ( 99.89%) 2918487 ###
2s - 4s : ( 99.99%) 94525
4s - 7s : (100.00%) 7283
7s - 10s : (100.00%) 549
10s - 16s : (100.00%) 1000
16s - 23s : (100.00%) 657
23s - 34s : (100.00%) 948
34s - 49s : (100.00%) 144
Avg : ( 1s)
disk_vbstate_snapshot (10655 total)
0 - 1us : ( 0.68%) 72
1us - 2us : ( 0.69%) 2
2us - 4us : ( 0.77%) 8
4us - 8us : ( 0.88%) 12
8us - 16us : ( 1.36%) 51
16us - 32us : ( 2.87%) 161 #
32us - 64us : ( 20.63%) 1892 ##################
64us - 128us : ( 21.07%) 47
256us - 512us : ( 21.16%) 10
512us - 1ms : ( 27.10%) 632 ######
1ms - 2ms : ( 28.13%) 110 #
2ms - 4ms : ( 28.56%) 46
4ms - 8ms : ( 28.83%) 29
8ms - 16ms : ( 29.16%) 35
16ms - 32ms : ( 30.38%) 130 #
32ms - 65ms : ( 31.94%) 166 #
65ms - 131ms : ( 75.27%) 4617 #############################################
131ms - 262ms : ( 97.78%) 2398 #######################
262ms - 524ms : ( 99.70%) 205 ##
524ms - 1s : ( 99.96%) 28
8s - 16s : (100.00%) 4
Avg : ( 68ms)
disk_update (66243580 total)
0 - 1us : ( 19.75%) 13084847 ####################
1us - 2us : ( 93.41%) 48794589 ###########################################################################
2us - 4us : ( 99.12%) 3784277 #####
4us - 8us : ( 99.60%) 317353
8us - 16us : ( 99.98%) 249227
16us - 32us : (100.00%) 12119
32us - 64us : (100.00%) 928
64us - 128us : (100.00%) 145
128us - 256us : (100.00%) 49
256us - 512us : (100.00%) 16
512us - 1ms : (100.00%) 13
1ms - 2ms : (100.00%) 9
2ms - 4ms : (100.00%) 6
4ms - 8ms : (100.00%) 1
8ms - 16ms : (100.00%) 1
Avg : ( 1us)
disk_vb_del (682 total)
256us - 512us : ( 0.29%) 2
512us - 1ms : ( 8.80%) 58 #########
1ms - 2ms : ( 92.38%) 570 #########################################################################################
2ms - 4ms : ( 99.27%) 47 #######
4ms - 8ms : ( 99.41%) 1
16ms - 32ms : ( 99.56%) 1
32ms - 65ms : (100.00%) 3
Avg : ( 1ms)
notify_io (104257193 total)
2us - 4us : ( 0.00%) 14
4us - 8us : ( 0.26%) 271270
8us - 16us : ( 39.46%) 40868330 #######################################
16us - 32us : ( 89.92%) 52603925 ###################################################
32us - 64us : ( 98.55%) 9004838 ########
64us - 128us : ( 99.44%) 927416
128us - 256us : ( 99.76%) 331552
256us - 512us : ( 99.89%) 131602
512us - 1ms : ( 99.95%) 64893
1ms - 2ms : ( 99.98%) 33430
2ms - 4ms : ( 99.99%) 13198
4ms - 8ms : (100.00%) 4580
8ms - 16ms : (100.00%) 1711
16ms - 32ms : (100.00%) 314
32ms - 65ms : (100.00%) 74
65ms - 131ms : (100.00%) 17
131ms - 262ms : (100.00%) 25
262ms - 524ms : (100.00%) 3
524ms - 1s : (100.00%) 1
Avg : ( 16us)
set_vb_cmd (2728 total)
1us - 2us : ( 0.44%) 12
2us - 4us : ( 5.57%) 140 #####
4us - 8us : ( 18.15%) 343 #############
8us - 16us : ( 37.10%) 517 ####################
16us - 32us : ( 49.93%) 350 #############
32us - 64us : ( 74.16%) 661 #########################
64us - 128us : ( 93.18%) 519 ####################
128us - 256us : (100.00%) 186 #######
Avg : ( 32us)
disk_commit (82769333 total)
0 - 1s : (100.00%) 82769309 ###########################################################################################################
1s - 2s : (100.00%) 8
2s - 4s : (100.00%) 9
4s - 7s : (100.00%) 7
Avg : ( 1s)
store_cmd (21137480 total)
1us - 2us : ( 0.00%) 117
2us - 4us : ( 0.00%) 148
4us - 8us : ( 0.00%) 51
8us - 16us : ( 0.09%) 18490
16us - 32us : ( 5.39%) 1120702 #####
32us - 64us : ( 74.65%) 14639430 ######################################################################
64us - 128us : ( 97.36%) 4799477 #######################
128us - 256us : ( 98.79%) 303998 #
256us - 512us : ( 99.25%) 96597
512us - 1ms : ( 99.53%) 59980
1ms - 2ms : ( 99.69%) 33561
2ms - 4ms : ( 99.80%) 23662
4ms - 8ms : ( 99.90%) 19400
8ms - 16ms : ( 99.98%) 18518
16ms - 32ms : (100.00%) 3111
32ms - 65ms : (100.00%) 202
65ms - 131ms : (100.00%) 10
131ms - 262ms : (100.00%) 7
262ms - 524ms : (100.00%) 7
524ms - 1s : (100.00%) 12
Avg : ( 60us)
item_alloc_sizes (21137480 total)
64 - 128 : ( 0.00%) 1
128 - 256 : ( 1.12%) 236283 #
256 - 512 : ( 37.24%) 7636370 #####################################
512 - 1KB : ( 51.76%) 3067692 ###############
1KB - 2KB : ( 91.24%) 8346192 #########################################
2KB - 4KB : ( 94.17%) 619204 ###
4KB - 8KB : ( 95.96%) 377390 #
8KB - 16KB : ( 98.89%) 619852 ###
16KB - 32KB : (100.00%) 234496 #
Avg : ( 1KB)
batch_read (2895991 total)
16us - 32us : ( 0.07%) 2098
32us - 64us : ( 13.83%) 398287 ##############
64us - 128us : ( 84.21%) 2038289 ########################################################################
128us - 256us : ( 90.78%) 190436 ######
256us - 512us : ( 95.71%) 142746 #####
512us - 1ms : ( 97.59%) 54428 #
1ms - 2ms : ( 99.00%) 40866 #
2ms - 4ms : ( 99.67%) 19282
4ms - 8ms : ( 99.94%) 7746
8ms - 16ms : ( 99.99%) 1659
16ms - 32ms : (100.00%) 123
32ms - 65ms : (100.00%) 28
65ms - 131ms : (100.00%) 3
Avg : ( 124us)
bg_load (2895991 total)
8us - 16us : ( 0.04%) 1214
16us - 32us : ( 1.50%) 42337 #
32us - 64us : ( 52.51%) 1477055 ####################################################
64us - 128us : ( 87.62%) 1016728 ####################################
128us - 256us : ( 92.17%) 131970 ####
256us - 512us : ( 96.18%) 116167 ####
512us - 1ms : ( 97.84%) 47852 #
1ms - 2ms : ( 99.10%) 36658 #
2ms - 4ms : ( 99.70%) 17299
4ms - 8ms : ( 99.94%) 7079
8ms - 16ms : (100.00%) 1499
16ms - 32ms : (100.00%) 103
32ms - 65ms : (100.00%) 27
65ms - 131ms : (100.00%) 3
Avg : ( 103us)
what is mean by below erros
2016/07/08 08:16:14 go-couchbase: call to Do(“s919591210434_Location.doc”) in github.com/couchbase/go-couchbase.(*Bucket).casNext took 241.42931ms
2016/07/08 08:35:50 go-couchbase: call to Do(“s919591214237_Location.doc”) in github.com/couchbase/go-couchbase.(*Bucket).casNext took 227.700976ms
is delay from syncgateway because of this? from this logs i could understand that doc casNext at couchbase has taken 227ms.
could you please suggest me what is causing delay from syncgateway ?
Looks like Couchbase Server is delivery as expected for K/V. So that means data from CB Server to SG is taking a long time.
Here is link talking about the same problem above. Redirecting to Google Groups
These warning we are getting in syncgateway error logs in every 5-10 minutes. is it syncgateway overloaded to capture the response ? is it connection between syncgateway and couchbase causing issue?
how much requests can syncgateway handle with 4GB ram?
1 Like
There is a pattern so that means some process is causing the issue.
A good rule of thumb for memory is for every connect there is 380KB of memory used to maintain the connection + NIC bandwidth.
@househippo, This is not the pattern . Its random warnings we are getting. as per the documentations, these warnings will be seen if delay of data fetch from couchbase to syncgateway is more then 220ms.
so delay will be from couchbase in getching or updating docs?