In Couchbase, memory management in the Query service is key to keeping the service efficient and responsive, especially as the service handles an increasing number of queries simultaneously. Without proper memory management, things can go awry – greedy queries can hog memory, and the combined memory usage of multiple concurrent queries can overwhelm the service, leading to degraded performance.

Fortunately, the Query service has several features that allows users to manage the memory usage of queries and the overall service.

This blog will explore these features in detail:

    • Per Request Memory Quota
    • Soft Memory Limit
    • Node-wide Document Memory Quota

Per-request memory quota

A significant portion of memory usage of the Query service comes from transient values, which can include documents or computed values. The memory used by these transient values will be referred to as document memory in the blog.

The Query service receives documents from the Data service as an encoded byte stream. However, the memory used by the value associated with the document can be much larger than the original stream. This is because the Query service decodes the stream into a structure which can be large as it must store all fields, values and any nested objects. The Query service is optimized for performance and not for compactness.

What happens if a resource-intensive query comes along and starts consuming a large amount of document memory? It can end up hogging memory and cause other queries to stall. How do we prevent a greedy query from affecting the execution of other active queries?

This is exactly where the per-request memory quota feature comes in!

Since Couchbase 7.0, the Query service provides a setting called memory quota to limit the maximum amount of document memory that a query request can use at any given time during its execution.

This per-request memory quota works by terminating a query if it exceeds its quota, while allowing all other active queries to continue execution. This ensures that only the greedy query is stopped, preventing it from affecting the performance of the other queries.

The memory quota does not correspond to OS memory use. It only accounts for the document memory usage and not for any memory used in the heap, stack, execution operators, etc.

How does the memory quota work?

The per-request memory quota can be thought of as configuring a document memory pool for a query request. The size of the pool is determined by the value of the query’s memory quota.

When the query requires a document/ value, it allocates the size of the document/ value from this pool. When the value/ document is no longer needed, the allocated size is returned back to the pool for reuse by the request.

At any given moment the total amount of document memory being used by the query request cannot exceed the size of its pool, i.e., its memory quota. If the query tries to use more document memory than what is available in its pool, the request will be terminated and an error returned.

It is important to note that the Query service is highly parallelized and operators can run simultaneously. This means that whether a query exceeds its memory quota can vary between runs. This is because, depending on the specifics of each run, the amount of document memory that is being used ( and hence allocated from its request pool ) can vary, even at the same stage of execution.

How to configure the memory quota

The per request memory quota can be set at a cluster, node and request level.

    • Unit: MiB
    • Default: 0 i.e. there is no limit on how much document memory a request can use

Cluster level

Set the memory quota for every Query node in the cluster with the queryMemoryQuota cluster-level setting. The value at the cluster level is persisted and when set, over-writes the node level setting for every Query node.

    • Learn how to set a cluster-level setting here

Node level

Set the memory quota for a particular Query node with the memory-quota node-level setting. The value set at the node level is the default memory quota for all query requests executed on the node. The node level value is not persisted and is over-written when the cluster level setting is modified.

    • Learn how to set a node-level setting here

Request level

Set the memory quota for a particular query request with the memory_quota parameter. The request level parameter overrides the value of the node-level setting. However, if the node level setting is greater than zero, the request level value is limited by the node level value.

    • Learn how to set a request-level parameter here

Soft memory limit of the query service

Now that we have explored how to limit the document memory usage of a query, you might be wondering, is there a way to limit the memory usage of the Query service?

The Query service has no setting to enforce a hard limit on the memory usage of the service. This is because the programming language used to develop SQL++ does not provide a mechanism to enforce a hard limit on its runtime memory usage. But it does provide a mechanism to adjust the soft memory limit…

Hence in Couchbase 7.6.0 the node quota setting was introduced to adjust the soft memory limit of the Query service!

Since this is a soft limit, there is no guarantee that the memory usage of the Query service will always strictly stay below it or that out-of-memory conditions will not occur.  However, an effort is made to maintain the Query service’s memory usage below this limit by running the garbage collector (GC) more frequently when this limit is crossed or approached closely.

Important note:
If the memory usage stays close to the soft limit, the GC runs aggressively which can cause high CPU utilization.

How to configure the node quota

The node quota can be set at a cluster, node and request level.

    • Unit: MiB
    • Default: 0
    • Minimum: 1

While the minimum value of the node-quota is 1 MiB, please set the node quota to practical values depending on the workloads and the system’s capabilities.

Cluster level

Set the node quota for every Query node in the cluster with the queryNodeQuota cluster-level setting. The value at the cluster level is persisted and when set, over-writes the node level setting for every Query node.

    • Learn how to set a cluster-level setting here

Important Note:

One way of configuring this setting cluster wide is by using the Couchbase Web Console. In the Web Console, this can be configured under the Memory Quota per server node in the Settings page.

This section is specifically for configuring the Query service’s cluster level node quota and must not be confused with setting the cluster-level memory quota setting.

Node level

Set the node quota for a particular Query node with the node-quota node-level setting. The value set at the node level is the default memory quota for all query requests executed on the node. The node level value is not persisted and is over-written when the cluster level setting is modified.

    • Learn how to set a node-level setting here

How to configure the soft memory limit

The soft memory limit of the Query service is set using the value of the node quota. If not set, a default value is calculated.

Node quota

If the node quota is set for a node, this is the soft memory limit. The soft memory limit will be capped at a maximum allowable value which is calculated using these steps:

a) The difference between the total system RAM and 90% of the total system RAM is calculated.

Total System RAM - (0.9 * Total System RAM)

b) If the difference is greater than 8 GiB, the maximum soft memory limit will be:

Total System RAM - 8 GiB

c) If the difference is 8 GiB or less, the maximum soft memory limit will be set to 90% of the total system RAM.

If the node quota exceeds the calculated maximum, then the soft memory limit is silently set to the maximum.

Default

If the node quota setting is not set for a node, a default value is calculated for the soft limit using the following steps:

a) The difference between the total system RAM and 90% of the total system RAM is calculated.

Total System RAM - (0.9 * Total System RAM)

b) If the difference is greater than 8 GiB, the default soft memory limit will be:

Total System RAM - 8 GiB

c) If the difference is 8 GiB or less, the default soft memory limit will be set to 90% of the total system RAM.

Node-wide document memory quota

What if a workload has a query that requires a large amount of memory to execute. Enforcing a per-request memory quota might not be ideal as this query might frequently be terminated for exceeding its quota. How can this query successfully execute while still protecting the Query service from excessive memory usage?

Consider another scenario with multiple queries executing concurrently, each with a per-request memory quota set. In this scenario, the memory usage of the service has become very high. But the document memory use of the queries remains below their respective quotas. So no query is terminated. As a result, the overall memory usage of the Query service remains high, causing problems. How can this be addressed?

Starting in Couchbase 7.6.0, the Query service has a mechanism to limit the cumulative amount of document memory that active queries can use! The introduction of a node-wide document memory quota attempts to address these challenges.

How does node-wide quota work?

The node-wide quota can be thought of as configuring a document memory pool for the entire Query service on a node.

When the node-quota is set, a memory session is created for each request. By default, this session starts with an initial size of 1 MiB. When the node-wide quota is configured, 1 MiB is allocated for every servicer and subtracted from the node-wide pool. This default allocation guarantees that each servicer has at least a minimum amount of reserved space, ensuring that incoming requests can always be serviced.

When a request requires a value/document, the size of the value/document is allocated from its session. If the session does not have enough memory for this allocation, it will grow in minimum increments of 1 MiB to accommodate the allocation request. The additional memory required for this growth is allocated from the node-wide pool.

If an active request’s memory session attempts to grow beyond the available remaining memory in the node-wide pool, the request will be stopped and an error returned.

Once the request no longer needs the value/document, it returns the allocated size back to its session.

The session’s memory (excluding the 1 MiB of the initial servicer reservation) is only returned to the node-wide pool once the request’s execution completes.

At any time, the total size of all memory sessions cannot exceed the size of the node-wide quota.

It is important to understand that this memory session is not to be confused with the per-request pool that is configured when the memory quota is set for a request. The two are not the same.Both a node-wide quota and a per-request memory quota can be configured. Read Configuring both Node-Wide Document Quota and Per-Request Memory Quota section below to understand more.

In this way, the node-wide quota places a limit on the amount of document memory that is being used by all active requests.

The node-wide document quota can only be configured when the node quota setting is explicitly set for a node. The size of this quota is calculated using two Query settings, node quota (explored in an earlier section) and node quota value percent.

How to configure node quota value percent?

The node quota value percent is the percentage of the node quota dedicated to tracked value content memory/ ”document memory” across all active requests.

The node quota value percent can be set at the cluster and the node level.

    • Unit: MiB
    • Default: 67
    • Minimum: 0
    • Maximum: 100

Cluster level

Set the node quota value percentage for every Query node in the cluster with the queryNodeQuotaValPercent cluster-level setting. The value at the cluster level is persisted and when set, over-writes the node level setting for every Query node.

    • Learn how to set a cluster-level setting here

Node level

Set the node quota value percentage for a particular Query node with the node-quota-val-percent node-level setting. The value set at the node level is the default memory quota for all query requests executed on the node. The node level value is not persisted and is over-written when the cluster level setting is modified.

    • Learn how to set a node-level setting here

How to configure the node-wide document memory quota

The size of the node-wide document memory quota is calculated relative to the node-quota. The node-quota must be set for the node-wide document memory quota to be configured.

The size of the node-wide pool is calculated using the following steps:

1. Calculate the percentage of the node quota dedicated to tracking document memory across all active queries using the following formula:

node-quota * node-quota-val-percent / 100

2. Calculate the minimum allowable value for the node-wide document memory quota.

The execution of SQL++ statements is handled by servicers. When a query is to be executed it is assigned to a servicer thread that is responsible for its execution. The Query service is configured with a number of servicers to handle incoming requests. There are two types of servicers, unbounded servicers and plus servicers.

The Query engine reserves 1 MiB of document memory for each servicer. Hence, the default initial value of each request’s memory session is 1 MiB. This means that the baseline document memory usage will be the total number of unbounded and plus servicers, measured in MiB.

Therefore, the size of the node-wide document memory quota must be at least equal to the number of servicers, measured in MiB.

      • Learn more about unbounded servicers here and plus servicers here

Formula 1

Quota reserved for servicers =
(number of unbounded servicers + number of plus servicers) MiB

3. The size of the node-wide document memory quota is calculated using the following formula:

Formula 2

Size of node-wide document memory quota
=
MAX( node-quota * node-quota-val-percent / 100, Quota reserved for servicers)

    • The quota reserved for the servicers is calculated using Formula 1

This is the maximum allowable size of all memory sessions across active requests and includes the initial reservation for each servicer.

Calculating available quota in the pool for document memory growth

The initial reservation for the servicers is deducted from the node-wide document memory quota for the node. Any remaining space in the node-wide memory pool can be used by each active request to grow its document memory usage beyond its initial 1 MiB reservation.

This remaining quota available for document memory growth is calculated using the following formula:

Formula 3

Size of node-wide document memory quota available for memory sessions of active requests to grow =
Size of node-wide document memory quota - Quota reserved for servicers

    • The size of node-wide document memory quota is calculated in Formula 2 
    • The quota reserved for the servicers is calculated using Formula 1 

It is important to set appropriate node-quota and node-quota-val-percent values that are practical and suitable for workloads. An example to illustrate the importance of this:

Example

Consider a Query node with 32 unbounded servicers and 128 plus servicers. The Administrator sets the node-quota to 10 MiB. The node-quota-val-percent is the default value of 67.

Using Formula 2 to calculate the size of the node-wide document memory quota:

Using Formula 3 to calculate the amount of document memory available in the node-wide pool available for requests’ memory growth:

This means that there is no room for document memory growth of requests beyond their 1 MiB initial reservation. In other words, each request is limited to using a maximum of 1 MiB of document memory.

Additionally the node-quota of 10 MiB is very small, garbage collection will likely be forced to run frequently causing high CPU utilization.

Reporting document memory figures

If the memory quota was set for a request or a node-wide document memory pool configured, information about the same will be reported in several SQL++ features which will be explored below. This information is helpful for debugging.

Response output

a) Metrics:

The usedMemory field in the metrics section of the query’s response reports the high water mark (HWM) document memory usage of the query in bytes.

The Query service is highly parallelized, and operators can run simultaneously.  As a result, the usedMemory figures can vary between runs for the same query. This is because, depending on the specifics of each run, the HWM document memory usage can be different.

b) Controls section:
If the controls Query setting is enabled and the memory quota configured for the request, the memoryQuota field in the controls section of the query’s response reports the value of the memory quota set.

    • Learn more about the controls setting here

System keyspaces

system:completed_requests, system:active_requests

In a request’s entry in these system keyspaces:

The usedMemory field is the HWM document memory usage of the query in bytes. The Query service is highly parallelized, and operators can run simultaneously.  As a result, the usedMemory figures can vary between runs for the same query. This is because, depending on the specifics of each run, the HWM document memory usage can be different.

The memoryQuota field is the value of the memory quota set for the request

    • Learn more about system:completed_requests here and system:active_requests here.

Configuring both per-request memory quota and node-wide document quota

As described in the Per Request Memory Quota section, if a request has a memory quota configured, the maximum amount of document memory that it can use at any given time during its execution is limited by the memory quota.

Additionally, as explained in the Node-Wide Document Memory Quota section, when the node-quota and a node-wide document memory quota is configured, each request gets its own “memory session.” Any growth in the size of these sessions is allocated from the node-wide document memory quota.

If a node-wide document memory quota is configured and a request has a memory quota set, the document memory usage of the query request is limited by both quotas.

How would a document/value allocation be performed?

When the request requires a document/ value, the following steps are performed during the allocation process:

  1. Memory Session Allocation:
    • The request first tries to allocate memory for the document from its memory session.
    • If there is enough space in the session, the allocation is successful.
    • If there is insufficient space in the session, the session attempts to grow its size by allocating from the node-wide document memory quota. (i.e. from the “node-wide document memory pool” ).
    • If there is not enough space for the session’s growth in the node-wide pool, the request will be stopped and an error returned.
  1. Request Memory Quota Allocation:
    • If the session allocation is successful, the request will attempt to allocate memory for the document from its memory quota. ( i.e. from its “request memory pool”).
    • If there is sufficient space left in its memory quota, the allocation succeeds and the request proceeds.
    • If there is not enough remaining space in the memory quota, the request will fail and an error will be returned.

Monitoring with system:vitals

The system keyspace system:vitals contains important information about each Query node in the cluster, including information related to memory and CPU usage, garbage collection and much more.  Users can use this system keyspace to monitor the health and vitals of the Query nodes.

There are two ways to access this information:

1. Query the system:vitals keyspace using SQL++.

SELECT * FROM system:vitals;

    1. Accessing the vitals per node using the Query service’s /admin/vitals endpoint.

curl -u $USER:$PASSWORD $QUERY_NODE_URL/admin/vitals

Below is a sample of a record in system:vitals for a Query node.

    • Learn more about the Vitals here and system:vitals here.

How does the query service trigger the garbage collector?

Starting in 7.6.0, the Query service routinely checks if the garbage collector (GC) has run in the last 30 seconds. If it has not, the GC is triggered to run. During this check, the amount of free system memory is also monitored. If the amount of free memory is less than 25%, an attempt is made to return as much memory to the OS as possible.

Run garbage collector on demand

Starting in 7.6.0, the Query service provides a REST endpoint /admin/gc that can be invoked to run the garbage collector. This endpoint can be invoked to trigger a GC run in an attempt to reduce memory utilization.

To force a GC run issue a GET request to the API:

curl -u $USER:$PASSWORD $QUERY_NODE_URL/admin/gc

To force a GC run and attempt to return as much memory to the OS as possible issue a POST request to the API

curl -X POST -u $USER:$PASSWORD $QUERY_NODE_URL/admin/gc

    • Learn more about this endpoint here.

Important Note:
Aggressively running the garbage collector can cause high CPU utilization.

Helpful references

Author

Posted by Dhanya Gowrish, Software Engineer

All Posts

Leave a reply