Data sync issue: Taking very long time to sync data and writes are also slowed down due to sync gateway

Mobile Sync Gateway
1

Hi, We are using syncgateway2.5 and couchbase 6.0 both community edition. We have total data of 23,00,000 in the database (3 nodes cluster). We enabled syncgateway in one of the node. websync enabled.
We need to sync around 4 lakhs records out the total data for which we added filter. Below is the sync gateway config.

But the data in syncgateway is very slow. In 24hrs only 12k records synced. But the data loaded is 23lakhs. Kindly suggest inorder to improve the data sync equal to the data load

{
“logging”: {
“console”: {
“log_keys”: [""]
}
},
“SSLCert”: “./ssl/cert.pem”,
“SSLKey”: “./ssl/privkey.pem”,
“adminInterface”: “ip1:4985”,
“interface”: “ip1:4984”,
“databases”: {
“bucket”: {
“server”: “http://ip1,ip2,ip3:8091”,
“username”:“bucket”,
“password”:"",
“bucket”:“bucket”,
“rev_cache_size”:30000,
“enable_shared_bucket_access”: true,
“num_index_replicas”:0,
“import_docs”: “continuous”,
“import_filter”: function(doc) { if (doc.type == "Article" || doc.type == "ReferenceData") { return true } return false },
“users”: {
“GUEST”: {“disabled”: false, “admin_channels”: [""] },
“user”:{“password”:“pwd”,“admin_channels”: [“ios”] ,“disabled”:false}
},
“sync”: function(doc,olddoc){ if(doc.type == "Article" || doc.type == "ReferenceData") { access ( ["user"], ["ios"]); channel("ios"); } }
}
}
}

2

Hi,

We expect Sync Gateway imports to be running at around 4k docs per second based on our performance testing on this spec machine:
OS: CentOS 7 CPU: E5-2680 v3 (16 vCPU) Memory: 32GB Disk: Samsung DCT883

So the 12k in 24 hours you’re seeing seems very low, at ~0.14 docs per second.

Are Sync Gateway and a Couchbase Server node running on the same machine? Based on your post it sounds like it is.
We recommend running each service on their own hardware, otherwise they will bottleneck each other.

3

Hi,

Thank you for the suggestion. I will try this and update back.

Thanks,
Jahnavi

4

We set one dedictaed VM for syncgateway still we have same issue. It looks like processor usage is full. We are using 2 core processor. We will try to increase and check. With memory increase we dont see any improvement after removing couchbase database from the VM where syncgateway is installed.
@bbrks
If you have any other suggestion kindly suggest
sync-gateway 111.44% 279.3MiB / 7.62GiB 3.58% 0B / 0B 0B / 0B 20

5

We still have sync issue even after increasing CPU and memory. Daily 10k records are processed. Not even 1percent improvement. Could you help us to identify root cause for performance issue. Is there any mistake in our configuration?

6

@bbrks Can you help us here?
Also we see below warn logs
DCP RollbackEx request - rolling back DCP feed for: vbucketId: 721, rollbackSeq: 0. – base.(*DCPReceiver).RollbackEx() at dcp_feed.go:296
c:#15364 Changes feed: error getting revision body for…(1-5437e1c8bd446c82e116c6d9fe48d8cc): 404 missing – db.(*Database).addDocToChangeEntry() at changes.go:127

while calling admin ui we get this error
400 invalid database name “favicon.ico” – rest.(*handler).writeError() at handler.go:690

Also the syncgateway indexes are shared between two nodes. will that also cause any performance issue? Setting index replica in syncgateway will it help in any performance improvement?

7

It looks like you’ve removed the doc ID from the log message, so can’t really tell the actual cause - There was a bug related to _user documents where this warning would occur, but is benign.

Nothing to worry about. This is just your browser trying to guess where an icon is served from and not finding it!

Increasing index replicas will help share the load on the Couchbase Server side, and also provide high availability when adding/removing server nodes.

I’ve looked at your config, and it all seems OK. It definitely seems like you have some significant performance problems here compared to how fast we’ve performance tested the import process, but I don’t have any spare time to diagnose further.

Our Enterprise offering will give you 24x7x365 access to the Couchbase Technical Support team, and going via that route will be your best bet to get dedicated help on this issue.