ViewQuery query causes OutOfMemory (Observable thread does not block)

Java SDK
1

I’m working with Couchbase SDK version 2.4.6 (core 1.4.6) and tried to get a huge ViewResult (~ 600 mil keys) via the asynchronous interface:

bucket.async().query(viewQuery))

I’m mapping the resulting rows into a Flowable of AsyncViewRows, which will be mapped into a Flowable of Strings (which are the resulting keys from the view).

I’m subscribing to that flowable to fetch corresponding Documents from couchbase. After some time I got an OutOfMemory error and figured out, that the background process of the SDK wont be blocked. Seems like all keys from the view will be loaded in memory in background and I’m not able to block that thread.

Code looks like following:

ViewQuery viewQuery = ViewQuery.from(design, view)
.startKey(startKey)
.endKey(endKey)
.stale(stale);

Flowable keys = bucket.async().query(viewQuery))…;

keys.flatMap(async.get(key))…;

I’m able to block the last thread which will fetch the documents, but I didn’t find a way to block the one which fetches the keys.

I want to keep the memory usage as low as possible. How can I reach it? How can I make the backround thread to be blocked?

Thanks!

2

At the moment there is no provision for pushing backpressure down to the response. That really leaves two options… one is process the records as they come in (the improved streaming parser in 2.5.x would use less space) and keep them in process, and the second is to write/read to a local file.

We can look into improving this in the future, tracked in JVMCBC-536.