How to remove all sync information from bucet. Purge does not work ! Sync issues!

Mobile Sync Gateway
1

Hello,

My environment:

Couchbase 6.0.0 CE
Couchbase Sync Gateway 2.1
Couchbase Lite 2.1.x Community (.NET Xamarin)
Mobile phones with Android 8
Ubuntu 18.04 LTS
One node
8 GB RAM
CPU 4 cores

Questions:

  1. I have one bucket with ~50 000 documents, no user activity/workload but I have 15-25% CPU usage. Why and how to decrease?
  2. I have one bucket with 0 documents, no user activity/workload but I have same 15-25% CPU usage. Why and how to decrease?
  3. Sync with mobile and tombstones deletion does not work as documented and expected. I have issues with sync (very long sync time and unnecessary data transfer via network). Does document purge function work? How to delete all sync information from bucket?

My sync problem:

I have added and deleted ~50 000 documents several times. Deleted via Couchbase workbench with nq1 delete query.

I have added ~50 000 new documents. Documents have type and channels properties. At this point I have ~ 150 000 deleted documents and ~50 000 not deleted new documents. ~7000 active documents are in the channel Inventory-GP-ACTIVE. No sync with devices.

My first sync (only one user and one mobile phone) request looks:

first
first.png1405×718 27.6 KB

For the first request for the channel Inventory-GP-ACTIVE I received ~33 000 documents. I know ~ 7000 should be not deleted documents and rest documents are deleted documents. Why I received deleted documents? This is first sync and phones does not know anything about any document.

My second request:

second
second.png1439×878 31.9 KB

Again for the channel Inventory-GP-ACTIVE I received same ~33 000 documents. Why? I received these documents with the first request already. Why sequence/checkpoint was not updated after first sync?

Then I thought I need to purge deleted documents.

I tried to change the couchbase server Auto-Compaction settings but that didn’t help:

q

I tried to purge documents via admin rest api but without any success.

With request:

_changes?channels=Inventory-GP-ACTIVE&filter=sync_gateway/bychannel&include_docs=true&limit=10

Received a lot of documents like this:

“results”: [
{
“seq”: 102830,
“id”: “0000de8f-6fc5-41d6-8689-1d613b7485b9”,
“deleted”: true,
“changes”: [{“rev”: “2-7285628553e96d85a9ff158ba09253f8”}]
},
…

In the log I see errors:

2019-05-26T21:29:22.948+03:00 [INF] HTTP: #013: GET /zzzz_mobile_app_debug/_changes?channels=Inventory-GP-ACTIVE&filter=sync_gateway/bychannel&include_docs=true&limit=10 (as ADMIN)
2019-05-26T21:29:22.948+03:00 [INF] Changes: MultiChangesFeed(channels: {Inventory-GP-ACTIVE}, options: {Since:0 Limit:10 Conflicts:false IncludeDocs:true Wait:false Continuous:false Terminator:0xc42041faa0 HeartbeatMs:0 TimeoutMs:300000 ActiveOnly:false}) …
2019-05-26T21:29:22.949+03:00 [INF] Cache: getCachedChanges(“Inventory-GP-ACTIVE”, 0) --> 10 changes valid from #455330
2019-05-26T21:29:22.949+03:00 [INF] Cache: Querying ‘channels’ view for “Inventory-GP-ACTIVE” (start=#1, end=#455330, limit=10)
2019-05-26T21:29:22.968+03:00 [INF] Cache: Got 10 rows from query for “Inventory-GP-ACTIVE”: #102830 … #102843
2019-05-26T21:29:22.968+03:00 [INF] Cache: GetChangesInChannel(“Inventory-GP-ACTIVE”) --> 10 rows
2019-05-26T21:29:22.969+03:00 [WRN] Changes feed: error getting revision body for “0000de8f-6fc5-41d6-8689-1d613b7485b9” (2-7285628553e96d85a9ff158ba09253f8): key not found – db.(*Database).addDocToChangeEntry() at changes.go:124
2019-05-26T21:29:22.970+03:00 [WRN] Changes feed: error getting revision body for “0005c79f-4a81-4149-9b0a-d1307c27b535” (8-5997d613152f5d0caab9b7ded7e93e59): key not found – db.(*Database).addDocToChangeEntry() at changes.go:124
2019-05-26T21:29:22.971+03:00 [WRN] Changes feed: error getting revision body for “0005f916-f8c4-4015-be69-f8c26766860b” (2-d2bf1ca58eb5b80559e3656d1b89f9ea): key not found – db.(*Database).addDocToChangeEntry() at changes.go:124
2019-05-26T21:29:22.971+03:00 [WRN] Changes feed: error getting revision body for “0008301c-9eb3-4523-826f-5d1d4173f424” (2-1bfb308febd8c3d08d52d888b7e0c459): key not found – db.(*Database).addDocToChangeEntry() at changes.go:124
2019-05-26T21:29:22.972+03:00 [WRN] Changes feed: error getting revision body for “00085739-017c-4c1a-8a49-c05ffd7e64e7” (4-9fbc8cafecb2d37e856c94cd7d7fc9bb): key not found – db.(*Database).addDocToChangeEntry() at changes.go:124
2019-05-26T21:29:22.973+03:00 [WRN] Changes feed: error getting revision body for “0010c1e7-b3a7-4d02-8490-97ed5339ca27” (2-afba3573b4e13b644aaa5c481ac4e3f2): key not found – db.(*Database).addDocToChangeEntry() at changes.go:124
2019-05-26T21:29:22.974+03:00 [WRN] Changes feed: error getting revision body for “0013abca-a526-4a11-b6a4-73afa0a2be94” (2-a1a8325a0b19c2db6ead9682db3ef0a1): key not found – db.(*Database).addDocToChangeEntry() at changes.go:124
2019-05-26T21:29:22.974+03:00 [WRN] Changes feed: error getting revision body for “00140647-2fef-4e5b-b6c9-7a1329792f43” (4-e66c23110043fbbacb5ce52879c34559): key not found – db.(*Database).addDocToChangeEntry() at changes.go:124
2019-05-26T21:29:22.975+03:00 [WRN] Changes feed: error getting revision body for “001944aa-2d38-47e8-9c5c-c810e0419df6” (2-0511e52b31a49aeeb5a08bee8b7db8f8): key not found – db.(*Database).addDocToChangeEntry() at changes.go:124
2019-05-26T21:29:22.976+03:00 [WRN] Changes feed: error getting revision body for “001bfa6d-286f-43d6-8e08-04bf8fee6cac” (2-2348af856d1e9d4dc892683493b7960f): key not found – db.(*Database).addDocToChangeEntry() at changes.go:124
2019-05-26T21:29:22.976+03:00 [INF] Changes: MultiChangesFeed done
2019-05-26T21:29:22.977+03:00 [INF] HTTP+: #013: --> 200 OK (28.8 ms)

Sent purge request:

POST /zzzz_mobile_app_debug/_purge

{
“0000de8f-6fc5-41d6-8689-1d613b7485b9”: [
“*”
]
}

Got error:

2019-05-26T21:36:13.197+03:00 [INF] HTTP: #019: POST /zzzz_mobile_app_debug/_purge (as ADMIN)
2019-05-26T21:36:13.198+03:00 [INF] CRUD: purging document = 0000de8f-6fc5-41d6-8689-1d613b7485b9
2019-05-26T21:36:13.199+03:00 [INF] CRUD: Failed to purge document 0000de8f-6fc5-41d6-8689-1d613b7485b9, err = key not found
2019-05-26T21:36:13.199+03:00 [INF] HTTP+: #019: --> 200 OK (2.6 ms)

Now I have situation where every mobile phone receives unnecessary info about deleted documents with every request.

How to delete/purge documents like this?

“results”: [
{
“seq”: 102830,
“id”: “0000de8f-6fc5-41d6-8689-1d613b7485b9”,
“deleted”: true,
“changes”: [{“rev”: “2-7285628553e96d85a9ff158ba09253f8”}]
},
…

I can reproduce this problem every time. If I delete ~50 000 documents again, my mobile clients will start receiving info about ~200 000 deleted documents with every request and this is not efficient and normal. Couchbase is promising solution but it looks like it has bugs and is unreliable for big scale projects. I need some answers and help, please.

2

When you have enable_shared_bucket_access: true in your Sync Gateway config, N1QL deletes will not be purges, they will be tombstoned for replication to mobile clients. These are necessary during normal use, when clients need to know that a document has been deleted. You can read more about this here: Tombstones | Couchbase Docs

If you need to empty the bucket completely, when developing/testing, you can flush the Couchbase Server bucket: Flush a Bucket | Couchbase Docs

If you need to purge documents on an individual basis, you will need to do this through Sync Gateway, via the _purge endpoint, as you’ve seen. But be aware that mobile clients who have pulled these documents will attempt to push the document back up as a new one.

3

Thank you for the answer. I can read documentation and understand delete and purge concept. After database flush sync looks like works, except, after some time a lot of tombstones appeared (there was no deletion)

My question is how to purge already deleted documents? How to purge tombstones? I am receiving an error:

2019-05-26T21:36:13.199+03:00 [INF] CRUD: Failed to purge document 0000de8f-6fc5-41d6-8689-1d613b7485b9, err = key not found. In workbench I can find 0000de8f-6fc5-41d6-8689-1d613b7485b9 document.