Slow indexing for map function in _design/sync_gateway

Couchbase Server
1

Hi Couchbase team and Couchbase Experts,

We seek your expert advice.
We am using latest version of Couchbase Server 3.0.2 and Couchbase Sync Gateway 1.0.3 (64 bit) in ubuntu server 14

  • There are only 3 buckets with 6 gb ram alloted to main buckets and its shadow, default has 1 gb ram
    Issue : The default map functions installed by sync-gateway take a lot of time to index on avg 1 sec per document.
    -avg doc size is not more then 1 kb

Other things that are happening
-beam.smp run by couchbase consumer more than 100% cpu

  • plus i see several page faults

What i tried:

1 I removed all string manipulation from our sync-gateway code
2 i ensured no empty keys are being indexed
3 moved to SSD

but when i removed my code or when sync maps functions do not have to compute channels or access or principal it takes 1500 docs a sec on to show in the shadow bucket.

My sync function :

-----------------------------start--------------------------
if (doc.type){

if(doc.owner && typeof doc.owner !== ‘undefined’ )

{ access(doc.owner,‘ch-’+doc.owner); channel(‘ch-’+doc.owner); }

if(doc.type==‘hx’){ channel(‘hx’,doc.channels);}
if(doc.type==‘cx’ && doc.dx_s && typeof doc.dx_s !== ‘undefined’ ) {

                if(doc.mx && typeof doc.mx !== 'undefined' && doc.owner typeof doc.owner !== 'undefined' ) 

                { access([doc.mx,doc.owner],'ch-'+doc.dx_s); channel('ch-'+doc.dx_s); } 
                else{ access(doc.owner,'ch-'+doc.dx_s); channel('ch-'+doc.dx_s);}  }
            
            if(doc.type=='qx' && doc.receivermx && typeof doc.receivermx !== 'undefined')
            {  access(doc.receivermx,'ch-'+doc._id); channel('ch-'+doc._id); } 
            
            if(doc.type=='cxdx'&& doc.dxtype )
            {  if( doc.dxtype!='Private' && doc.dxnumber && typeof doc.dxnumber !== 'undefined' ) 
                {    if(doc.mx && && typeof doc.mx !== 'undefined'){ access([doc.mx,doc.owner],'ch-'+doc.dxnumber); } 
                    else {access(doc.owner,'ch-'+doc.dxnumber);}  channel('ch-'+doc.dxnumber); }  } 
            
            if(doc.type=='dx'&& doc.dxtype && doc.mxarr && doc.dxtype!='Private'){   
               access([doc.owner,doc.mxarr],'ch-'+doc._id); channel('ch-'+doc._id); 
            }

} }

-----------------------------end--------------------------

Question

1 can sync gateway handle multiple if loops and check as i have added
2 should i transfer most if the code to mobile device and do minimum checks here
3 if there some mistake in our code.

Please help us.
Thanks in advance
Arjun

2

Hi Arjun,

To your specific questions:

  1. Sync gateway shouldn’t have any problem with the conditional logic you have there. If there was a syntax problem, you’d be seeing a sync function execution error in your Sync Gateway logs.
  2. In general, doing access() grants in the sync function are more computationally expensive than doing channel() assignments. Access() grants require sync_gateway to index and recalculate the channels assigned to the user. It looks like there are a handful of cases where you’re defining one document per channel, and doing access grants to that channel to a single user. Whether you have any alternatives to this approach probably depends on your use case (and whether you’re doing other grants for that doc elsewhere), but you might consider whether you can do something like predefining a channel for the user, and channel the doc to that.

I’m not crystal clear on the performance you’re currently seeing - when you say it’s taking 1 second per doc for indexing - how are you measuring that indexing time?

Thanks,
Adam

3

Hi thanks Adam for taking time to help me out.

1 i am measuring 1 doc per sec via keeping an eye on fill rate and number of items under General Bucket Analytics.

2 by monitoring device and bucket as i know no of docs in mobile device.

It looks like there are a handful of cases where you’re defining one document per channel, and doing access grants to that channel to a single user. Whether you have any alternatives to this approach probably depends on your use case (and whether you’re doing other grants for that doc elsewhere),

3 i am doing other grants to the doc else where

4 I will predefine a channel for the user, and channel the doc to that. - hmm…i can do that via rest request but is it also possible to assign channels via cbl lite (could not find anything supporting it ), idea is to transfer some work to device, as per documentation only filtered pull is possible.

Thanks in advcance
Arjun