Query - batch-wise pull the data from Sync Gateway

Mobile Couchbase Lite
1

Hi,

I am using couchbase Lite ReST based support 1.4.1 version and we have huge data to pull during login time. Currently I am facing the issue like UI is busy for updating for certain time.

is there any way to pull the recent data only? So later user can pull rest data…if required. Please suggest how to implement this.

Example: consider I am having of 1 lacs document, i need to pull only the recent 100 documents out of this.

Thanks,

1 Like
2

@jens @hideki please share your thoughts…

3

@jens @hideki do you have any update on this ticket? We would greatly appreciate if we could get some feedback on this sooner.

Br,Arun

4

Hi all,

Following is a just idea, not verified.

  1. Call /{db} to get update_seq
  2. Call /{db}/_changes with since parameter with updated_seq - 100
  3. If Step 2 could not obtain unique 100 document IDs, keep calling step 2 with since parameter another 100 lower number.

Hope this helps you!

5

@hideki thanks for the reply…

https://developer.couchbase.com/documentation/mobile/1.4/references/couchbase-lite/rest-api/index.html#/database/post__db___changes

since parameter is at global users level right? or is it per user level?

Thanks
Nithin

6

@nitz_couchbase
per database

7

This is somewhat fragile, I believe. Normal best practice recommendation is to treat updated_seq as opaque and not rely on it having a particular structure. So you run the (probably small) risk of having this break on future updates.

There are at least a couple other approaches to doing this that will depend on specifics. If you can do backend processing, one way would be to keep a document with the IDs of the most recent updates. Pull that document first, then do a oneshot replication filtered by those document IDs. That could have problems with race conditions, so it would depend on how sensitive you are to that.

I believe there’s a solution in use by at least one other product that bins documents into channels by creation date. So, say, all of October 2017 goes into one channel.

One other idea (which I’ve never tried) would be to have a date stamp, or even a simple “new” vs. “old” flag. Assign documents to two channels based on the stamp/flag. This again I think would need a back end process to flip the flag, or in some other way trigger a reprocessing of the documents. You can just trigger a reprocessing of all documents, but that generally means having sync gateway effectively offline for a period.

1 Like
8

You mean “pull from Sync Gateway”, correct? So this is referring to the first launch of the app.

Currently I am facing the issue like UI is busy for updating for certain time

There’s no need to block the UI while the replicator runs. You can display the current data and re-run your queries periodically when you get notified that the database has changed (by monitoring the changes feed.)

9

@jens requirement is
1)user has say 1000 docs at server in time frame of 6 months.
2)Get only last 100 docs.
3)Or get last 1 month docs.

How feasible to get these documents #2 and # 3.

Thanks
Nithin

10

@jens please share your thoughts…

11

One option to have a server side logic that will periodically move old documents or last N-100 docs from the user’s “active” channel to a history channel.