Admin REST endpoint not returning all the results

Couchbase Server
, ,
1

Hello, I have built a dashboard for checking completed queries using the admin/completed_queries REST API. The reason for not using the N1QL query dashboard is because we are using Couchbase 5.1.1. Community Edition so we don’t have the query monitoring dashboard yet, and also we would like to keep the data in our monitoring stack so we can do some analysis for specific time periods.

To get all queries, I used a http poller that polls the REST API every minute and filter those than happened in the past 1 minute. I have set completed-threshold to be 0 and completed-limit to be 4000 on admin/settings API

However, I noticed we are not getting all queries. I have set up a load testing tool that makes n1ql queries with 10qps. In 1 minutes, it should make about 600 queries, but in the rest admin endpoint, I only get 20 queries per minute consistently. In the Couchbase UI, I can clearly see it making 10 queries per second corresponding to my load testing tool. So why are the queries missing from the admin endpoint?

In my scenario, I am making only a simple "select * from test-bucket". The index for this bucket is only on 1 node in a 30 node cluster. And I am reading the results from that node’s admin/rest endpoint

2

If you set completed-threshold 0, completed-limit to be 4000 and total statements doesn’t exceed it should be there. once exceed limit it overwrite. The settings, completed_requests data will not persist (will be reset on restart query service). You should try latest CB version which also give cluster level (all query nodes) info.

3

Hi @vsr1,

For my implementation, I fire 600 requests per minute yet there is only 20 requests including the latest minute so I don’t think the lost statements are being overwritten. Are there any documentations on explaining how completed_requests work?

I would try latest version of CB but due to organizational constraints we are still using 5.1.1

4

You mentioned 30 node cluster. 600/30=20
Check other query nodes. Admin endpoint on query node may return the current node info only. cc @marcog

If you run the following command it should give all the requests

SELECT * FROM system:completed_requests;

5

Hi @vsr1, thank you. That is a big clue!

Does that mean queries can happen on other buckets without the index? I only have index on “test-bucket” in my first node.

6

Unless you have USE KEYS queries will fail without index.

Run this on any query node it should give all of them
SELECT * FROM system:completed_requests;

Also check https://docs.couchbase.com/server/current/manage/monitor/monitoring-n1ql-query.html#vitals

7

Right - the situation with cbq-engine monitoring is as follows:

  • the system keyspaces (active_requests, completed_requests, prepareds) return entries from all nodes in the enterprise edition and only from the local node in the community editions.
  • the admin REST endopoints only get the information from the local node (they are used to implement multi node access from system keyspaces)
    So if you want to access all the requests from all nodes, you need to access all 30 admin endpoints, or get an enterprise edition and use the system keyspaces
8

@Marco_Greco, @vsr1,

Thank you for your replies. My misconception is that the queries are only sent to the nodes with the index.It turns out that the index is uniformly distributed to ALL nodes. After setting up my monitoring pipeline for all 30 nodes, I managed to get all the queries. Thank you!

9

Index is global secondary index. It present on one of the indexer node and it can service queries executed on any query node.

10

Sorry I meant to say queries are distributed to all nodes. My index is only on one node but it is accessed by all the query services. Thank you for the information!