I am running Couch base server 4.0 with sync gateway 1.1.1 on 5 nodes community version.
(After this issue started then i have upgraded couchbase from 3.0.1 to 4.0 and sync gateway from 1.1.o to 1.1.1)
Since last week Couchbase sync gateway CPU was suddenly goes high and it never come back down unless restart server or restart sync gateway. It happens randomly on 1 or 2 nodes and rest of the nodes cpu utilization less than 5%
When this thing happens some times sync gateway does not respond anymore and only option to restart the server and start sync gateway.
BTW 172.16.1.76 is our hardware load balance IP as external traffic comes through load balance and then forward that traffic to couchbase sync gateway nodes.
I have tried settings revision limit to 20 and increased “maxFileDescriptors” to 10000 from 5000 default limit and also increased open files limit to “20000” in centos.
Please let me know what other things i can try to fix this issue as this is causing system unstable and going offline quite often. If you need more information please inform me.
The log information looks like a symptom of the problem (dropped client connections), and not the underlying cause.
There were several CPU utilization improvements included in Syng Gateway version 1.2, so you might try updating and see if it addresses your issue. If not, I’d request that you file an issue in the Sync Gateway github repo (https://github.com/couchbase/sync_gateway/issues/new), and attach more detailed logs there.
I am planning on upgrading Sync gateway from version 1.1.1 to 1.2 and would like to know if i upgrade do i need to upgrade couch lite to 1.2 as well on mobile device?
I have taken out one of the sync gateway node out from load balance and then waited until load balance session go down to “0” and then run below command to check sync gate way connections and i still see 5000+ session on sync gate way but they are not on LB.
Hi @Karunakar, we had similar issues, upgrading to 1.2 definitely improved performance. You don’t need to upgrade the client, we are running sync gateway 1.2 and couchbase lite 1.1 in production
@Karunakar One of the enhancements included in 1.2 is improved detection and release of half-closed connections by Sync Gateway. I think that will address the issue you’re describing.
Yes, i am planning on upgrading and it seems like newer version of sync gateway 1.2 start the service automatically and also put logs and config files to different location (/home/sync_gateway) so i am planning on changing location where sync gateway saves logs and would do upgrade tomorrow and also noticed that sync_config file in /etc/init folder and also creating sync gateway user account.
In the mean time i have installed couch mobile 1.2 and tested with sync gateway 1.1.1 and noticed below connections and would like to know is it normal behavior.
Hi @Karunakar, can you provide some more information, like the type of replication you are using. Is it continuous, or one shot? Is it pull or push. Technically push replication is one shot, it sends the data and then closes the connection. The continuous pull replication keeps a connection open with the server, it can open several connections during bulk download, but then releases them. I was playing with this recently and found a bug in our application which was keeping the connection open through a long running service running in the background. You should make sure that your database manager is a singleton for your application and you are not accidentally starting multiple replications on multiple instances of the object.
@karunakar The initial connection count sounds reasonable - Sync Gateway maintains connections with services on the backing Couchbase Server nodes, and if you’ve got a 5 node cluster, that doesn’t sound unreasonable.
As Vlad mentions, the number of active connections triggered per client will depend on the number and type of replications you’ve got set up, as well as things like channel distribution, etc.
