Q: cb server haven't purge document tomb

Mobile
1

HI,

I set the Tombstone Purge Interval to 0.04 days(1 hours) to pruge the documents from cb server(bucket).
after one day, it seem not work for me. I database still contains a lot of documents with empty body.

then I find those documents ID, manual purge them from couchbase lite and sync gateway.
then I flush the bucket from web console ui, resync with couchbase lite, why the documents with empty body(tomb) still sync to the bucket?

my bucket has 25000 documents, the tomb has 20000 documents. how can I purge those documents tombo?

thanks

angular

2

Sounds like your using Couchbase with xattr (Docs on what xattr here: Extended Attributes | Couchbase Docs). Using xattr lets you edit and delete docs in CouchBase Server directly and still sync to CBL.

In the Mobile world there are “Deletes” and “Purges”:
A “Delete” marking a document to not be viewable & queryable. Plus Marking it with a tombstone(_deleted:true) so that devices offline can be told of delete next time it syncs. With xattr the _deleted:true is hidden in the meta data so the JSON is empty and not searchable or queryable.

On your server side you have set the tombstone purge interval to 1 hour = After you “delete” a document on the server (or sync a deleted:true document from CBL ) the tombstone will only live for 1 hour. So if your device(s) is away for more than an hour it will not know about Delete.

on the device are you Deleting the document or Purging the document? if you delete the document on CBL and don’t have any rules on who can sync deletes in SG’s Sync Function CBL is probably pushing it back to SG.

EX Sync Function rejects writes from non-editors role. In the below an invoice doc with number 12345 goes through SG’s Sync function checks to see if they are an editor

{
“_id”:“invoice:12345”,
“_deleted”:true
}

function(doc,oldDoc){
var a = split(doc._id,“:”)
if(a[0] ===“invoice”){
if(doc._deleted === true){
requireRole(“editor”)
channel(oldDoc.channels)
}
}
}

3

HI @househippo,

thanks for your reply!

yes. I enabled xattr from sync gateway configure.

I know if enable xattr, the sync gateway import documents will apart attrs and document’s body to save.
I inspect the document with empty body, the contents as follows:

the document ID is: MsgGoods.11a9d08a-03eb-4cf4-9203-bf2baa3f0dd6

  1. view the document from web console UI:

image
image1116×92 13.8 KB

  1. get the document from sg:

curl http://localhost:4985/test1/MsgGoods.11a9d08a-03eb-4cf4-9203-bf2baa3f0dd6
{“error”:“not_found”,“reason”:“deleted”}

  1. query the document from web console N1ql:

select meta().xattrs.$document from kitchen use keys “MsgGoods.11a9d08a-03eb-4cf4-9203-bf2baa3f0dd6”

[
{
“$document”: {
“CAS”: “0x15a857f3830a0000”,
“datatype”: [
“json”,
“xattr”
],
“deleted”: false,
“exptime”: 0,
“flags”: 0,
“last_modified”: “1560593974”,
“seqno”: “0x0000000000000001”,
“value_bytes”: 2,
“value_crc32c”: “0x297bd0aa”,
“vbucket_uuid”: “0x00007800e3310616”
}
}
]

the deleted property is false. I use couchbase lite delete method deleted this document, the first two query is right. why the last get $document delete property is false?
I think because this delete of $document property is false, so I setted purge time but not pruged those documents.

how can resolve this issue?

thanks

angular

4

select meta().xattrs._sync
or
#http://{sync-gateway-host}:4985/{db-name}/ _raw/MsgGoods.11a9d08a-03eb-4cf4-9203-bf2baa3f0dd6

will get you the rev history of the document.

In your sync gateway logs I suspect that CBL is pushing it back but with an empty body = {} and with a simple sync function that not catching the empty body.

5

HI @househippo,

Thank you for your prompt reply.

I get the document from sg _raw endpoint:

curl http://localhost:4985/test1/_raw/MsgGoods.11a9d08a-03eb-4cf4-9203-bf2baa3f0dd6

{“_sync”:{“rev”:“2-7ad33dc7a80dc26a4e74e535ffb030d6a1b6371c”,“flags”:1,“sequence”:49,“recent_sequences”:[49],“history”:{“revs”:[“1-e98dc20d95db2dd4d8e7853c4ea9de8d03d01661”,“2-7ad33dc7a80dc26a4e74e535ffb030d6a1b6371c”],“parents”:[-1,0],“deleted”:[1],“channels”:[null,null]},“cas”:“0x00000a83f357a815”,“value_crc32c”:“0x297bd0aa”,“tombstoned_at”:1560593974,“time_saved”:“2019-06-15T10:19:34.27080908Z”}}

this document as to revision:

I get the first revision from sg:

curl http://localhost:4985/test1/MsgGoods.11a9d08a-03eb-4cf4-9203-bf2baa3f0dd6?rev=1-e98dc20d95db2dd4d8e7853c4ea9de8d03d01661
{“error”:“not_found”,“reason”:“missing”}

get the second revision:

curl http://localhost:4985/test1/MsgGoods.11a9d08a-03eb-4cf4-9203-bf2baa3f0dd6?rev=2-7ad33dc7a80dc26a4e74e535ffb030d6a1b6371c
{“_deleted”:true,“_id”:“MsgGoods.11a9d08a-03eb-4cf4-9203-bf2baa3f0dd6”,“_rev”:“2-7ad33dc7a80dc26a4e74e535ffb030d6a1b6371c”}

the first revision already can’t fetch. the second revision _deleted is true.
I also suspect the CBL push those documents with empty body to SG then store in CB SERVER.
my sync gateway function as follows:

function sync(doc, oldDoc) {

if (isRemoved()) {
    return;
}

if (!isDelete()) {
    validateNotEmpty("channelId", doc.channelId);
    validateNotEmpty("className", doc.className);
    if (isCreate()) {
        if (!hasPrefix(doc._id, doc.className + ".")) {
            throw ({ forbidden: "文档ID必须以对应的className作为前缀" });
        }
    }
}

/* Routing */
// Add doc to the user's channel.
if (isDelete()) {
  channel(oldDoc.owner ? oldDoc.owner : oldDoc.channelId);
} else {
  channel(doc.owner ? doc.owner : doc.channelId);
}

first: check the documents is remove from original channel. is removed, return and without any operate.
second: if is a not deleted document, validte the document.
third: if is a deleted document, according to the property of the document incorporate into channel.

please point where I make a mistake? or how can I filtered which the CBLITE document with empty body?

thanks very much

angular

6

Keep make your sync function simple at first(Like below)
&
Print out different things while docs go through your sync function to test that the logic is working.
console.log("output here")

You can use simple HTTP PUT/POST/GET/DELETES to test out your logic via maybe POSTMAN

Like above you can use this to sniff for deletes.
You can give CBL user a role like: “customer” , “users” , … etc.

CREATING ROLES:
https://docs.couchbase.com/sync-gateway/current/admin-rest-api.html#/role/put__db___role__name_

UPDATING USERS ACCESS(giving them roles
https://docs.couchbase.com/sync-gateway/current/admin-rest-api.html#/user/put__db___user__name_)

function(doc,oldDoc){
           var a = split(doc._id,":")
           console.log(a)
           if(a[0] ==="invoice"){
	           if(doc._deleted === true){
	           		console.log("DocID: "+doc._id+" is Deleted")
					requireRole("editor")
					channel(oldDoc.channels)
				}
			}
}
7

HI @househippo，

I seem to understand why occurred those documents with an empty body.
operate as follows:

  1. let android sync data with sg.
  2. delete some documents of couchbase lite from android. now query the deleted document will prompt not exist. then quit the app stop sync.
  3. flush the bucket from the server.
  4. launch the Android app resync data with the sync gateway.
  5. now inspect documents in the bucket, will find a lot of documents with an empty body.

I guess the reason is: :smile:
the step 1-2 is work fine, because the deleted documents synced to the sg and cb server, so the documents has a tomb,
when I flush the bucket(step 3), then resync data from lite to sg (step 4), then lite local database has the deleted documents tomb, those tomb documents sync to sg. but sg haven’t those documents tomb xattrs information, so the sg create those documents as normal documents as empty body.

I don’t know if I guessed it right?

thanks your reply again.

angular

8

HI @househippo,

I haven’t use sg roles, I only use the sg users and channels. because I think the channel is light than the roles
in my application scenario, one customer only has one sg user, then they can create logical app users binding the channel. the sync roles will base on the channels. I feel this method will be more flexible.
I don’t know if I use it like this?

thanks!
angular

9

you don’t have to create roles docs … you can just say this user is a “user” or “editor” and do a requireRole(“editor”) check in the sync function.

PUT :4985/{db-name}/_user/bob

{
"name":"bob",
"admin_channels":["bob"],
"admin_roles":["user"]
}
10

HI @househippo,
thanks for your reply.
I already create sg user, and I validate the document must have the “channel” property.
but I dont kown why when I flush the server bucket, then sync lite local data to server bucket, how to filter those documents with empty body.
I think check the removed document logical in my sf have error.

function sync（doc，oldDoc）{

if (isRemoved()) {
    return;
}

    function isRemoved() {
        return (isDelete() && oldDoc == null);
    }

if the sf detect the document is removed then return. the return will don’t sync the document?

thanks

angular