How to access system:completed_requests via KV service?

[david.nault]

Hi! Is data in the system:completed_requests catalog accessible via KV?

Context: @yeikel wants to stream this info using the Couchbase Kafka source connector for later analytis; the Kafka connector only works with KV documents.

Thanks,
David

[vsr1]

It is not possible. Writing to KV means huge resources. Depends on workload these can run tens of thousands of mutations per sec.

Those are in memory built collection in query service.

[yeikel]

Thanks for explaining

Besides doing some sort of polling(ie: query the table every n time), what other method do I have to take that data out of Couchbase?

[vsr1]

NOTE: These are resource intensive which can impact cluster performance and TCO.

SELECT c.* FROM system:completed_requests s;

CE : It gives node specific , if multiple nodes one must do each node
EE: It gives from all nodes so expensive
each query must go through all memory entries and filter so expensive on any filter
It only keep track that took more than (completed-threshold) 1sec and only keeps (completed-limit) 4000 (evicted randomly) configurable via Configure Queries | Couchbase Docs

Even pooling you may not get uptodate, One can use filter on requestTime to avoid last seen once

One can use endpoints on node specific Manage and Monitor Queries | Couchbase Docs

In future release one can configure dump into file and access via history [MB-62492] - Couchbase Cloud. [DOC-12312] - Couchbase Cloud

[mreiche]

@yeikel

what other method do I have to take that data out of Couchbase?

Observability - Metrics Reporting | Couchbase Docs

metrics option on queries - queryOptions().metrics(true) Query | Couchbase Docs