2ms or 40ms for random gets

Couchbase Server
1

Help me understand what i’m doing wrong here.
I run a series of gets (single key), from a php script that gets called many times per second on a 3 nodes cluster.

At random times, couchbase is returning in either <2ms or 40-42ms, and rarely 80-85ms. Nothing ever in between, or different values.

how can I get a consistent <2ms ?

2

Do these latencies include the initial bootstrap time, or is this only for the actual get?

3

not sure what you mean by the “initial bootstrap time”… but I measured the time for the try { ->get } catch(){} part.
I couldn’t find any pattern or rule that makes this predictable.

4

How does your script look like? Keep in mind there are several factors which control the timing of retrieval:

  1. Whether the key is active in memory or needs to be fetched from disk
  2. The size of the value in the key
  3. The current network load
  4. Whether an existing TCP connection already exists on the client-side; i.e. if this is the first key being fetched by the client, it may take longer sometimes, if the client needs to open a new connection to the node hosting the key.
5
  1. at this stage, all keys are in memory (we’re just running some tests here).
  2. the doc is quite small, just around 20 variable_name = (int)
  3. not sure the load is relevant, while out of 1000 such request, about 20% get 40ms, and all the rest get <2 miliseconds.
  4. this happens randomly. both when the first fetched key and when the 5th, 10th, etc.

here is the code:

 try{
 shuffle($_COUCHBASE_SERVERS);
 $_COUCHBASE_CONNECTION=implode(',',$_COUCHBASE_SERVERS);
 $_COUCHBASE = new CouchbaseCluster('couchbase://'.$_COUCHBASE_CONNECTION);				
 } catch (CouchbaseException $e){
 _log("couchbase connection not available",0);
 }

    
try{
$_CB_USERS = $_COUCHBASE->openBucket('users');
_log("couchbase bucket opened ",4); 
} catch (CouchbaseException $e){
_log("USERS bucket not available",0);
}


try{
$r=$_CB_USERS->get($obj);
$RESULT=$r->value;
_log("_get: loaded: $obj in "._timer($time)." ms.",4);
} catch(CouchbaseException $e){
_log("_get: FAILED loading: $obj : ".$e->getMessage(),1);
}
6

Aha, so the time includes bootstrapping as well. This largely depends on which node you actually connect to.

When the client connects to the cluster, it will initially connect to only one node. The node it connects to is random (since you’re randomizing the list), and the node that the key is mapped to depends on the node (what the key’s vBucket is).

In the best case scenario, the node which the client bootstraps from is the same node which is the master for the key you query, in this case there is only a single TCP connection to establish.

Each connection to a node involves a negotiation process which is around 2-3 network round trips; an actual get is only 2 network round trips; thus in the case where all you do is connect to a cluster and fetch an item, the overhead of the negotiation is greater than the overhead of actually fetching the item.

There are some features you can use in order to reduce the time it takes to perform the bootstrap process; mainly the configuration cache, which eliminates the bootstrap process in most cases. See http://docs.couchbase.com/developer/c-2.4/options.html and search for config_cache on that page. (This is the documentation for the connection string which is available in PHP and parsed by the C library).