How to detect and handling _bulk_docs 404 error from sync gateway

Let us image that a document completely gets deleted directly from the couchbase server without going through the sync gateway. I understand that this should never happen. However, during development, I seemed to have flushed my bucket directly from the server and it has caused some undesirable affects.

Now, each time my phone replication starts, CBLRestPusher tries to push bulk changes to the server. However, I get the following error output on the client side (xcode terminal output)

2018-07-05 12:55:03.859630+0200 6Degrees[952:246723] WARNING: CBLRestPusher[http://123.123.123.123:4984/db_name/]: _bulk_docs got an error: {
error = “not_found”;
id = “-MikcSvTzcig5MZTa4cg0Ql”;
reason = missing;
status = 404;
} {at -[CBLRestPusher uploadBulkDocs:changes:]_block_invoke:402}

The sync gateway output is

2018-07-04T17:43:43.157Z WARNING: Unrecoverable error attempting to update xattr for key:-MikcSvTzcig5MZTa4cg0Ql cas:0 deleteBody:false error:key not found – base.CouchbaseBucketGoCB.UpdateXattr.func1() at bucket_gocb.go:1029
2018-07-04T17:43:43.158Z BulkDocs: Doc “-MikcSvTzcig5MZTa4cg0Ql” --> 404 missing (key not found)

What I would like to do is catch the error on the client side (the iPhone in my scenario) and handle it. The error is clearly getting down to the phone, as it is printing out the error description. My question is where??? Or is there an event/notification I can register for to handle this 404 rejection error? Essentially, I want to locally purge any document on my phone when the sync gateway says it does not exist.

Let us image that a document completely gets deleted directly from the couchbase server without going through the sync gateway. I understand that this should never happen.

It’s actually legal if you’re using SG 2.0 (and whichever newer version of Server it requires) — the integration has been tightened so you can make changes directly in the bucket and SG will pick them up correctly.

But since you’re not using 2.0, and you’re in development, I would just reset the mobile app too. Once you’ve nuked the server-side bucket you’re in kind of an undefined state, and that SG error does not sound like something that should occur in normal operation.

You are correct. I am using the following configuration

Server: 5.0.1 ubuntu 16.04
Sync Gateway: 1.5.1 x86_64
Couchbase Lite: 1.4.1

Also, you are correct again in that this error “should not” happen after release. However, I have one question. I currently have my “Metadata Purge Interval” set to 1 day. From my understanding, this means that my server will purge all documents (including tombstones) after one day of being marked as “_deleted”. In this scenerio, if a user does not use the app regularly, is it not possible that they will have this 404 error? Eventually, building up a bunch of unknown documents on the client side?

If the above scenario is true, then my original question still stands. How can I detect this 404 not found error? Will I have to modify the couchbase framework source code, or is there an event I can register for. As I mentioned before, the error must be getting down to the client (iPhone) as a debug message is being printed. I just cannot seem to figure out how to correctly listen for this error.

Thank you for your support,
Sean

I’m not sure what’s going on in SG in your situation — that Unrecoverable error attempting to update xattr for key doesn’t sound like an expected thing. As I don’t work on SG, someone else would need to answer that… @adamf?

Has your Sync Gateway node been restarted since the accidental flushed bucket? I don’t know of a case where SG should hit that error for a normal CBL update, but one possibility might be that Sync Gateway assumes the document already exists because it’s resident in Sync Gateway’s revision cache, and then hits that error attempting to update. If that’s the scenario, restarting Sync Gateway should prevent the error.

Yes, the sync gateway gets killed and restarted. These are the steps I take to totally annihilate my database and start fresh.

  1. Kill the sync gateway process: killall sync_gateway
  2. Flush my bucket
  3. Delete the .cblite2 database from each device
  4. Restart sync gateway /opt/couchbase-sync-gateway/bin/sync_gateway ./sync_gateway_config.json &

If I do the above without step 3, I get the 404 error on a bunch of documents.

Before hearing from you two, I thought this made perfect sense. The sync gateway and the server know nothing about the couchbase lite documents. So when the client starts up and tries to push revisions that should have parents already in the server side, the sync gateway returns an error saying, I cannot find that document? It does not exist. So then I am left with dead documents on the client. From what you are saying, it sounds like this should never happen at all.

My goal in this scenario was just to detect the 404 error (on the client side) and purge the document from the client. Problem solved. lol. However, I can’t seem to find where the event is thrown so that I can add logic to purge the document.

Thanks for the additional details. It sounds like you may be hitting https://github.com/couchbase/sync_gateway/issues/3307, which has been fixed in Sync Gateway 2.0.

I think that might be it! I also did not know that Sync Gateway 2.0 was part of the community addition yet! So it might be time to upgrade our configuration!

Do you know where the capability chart exist for sync gateway, couchbase server, and couchbase lite. I remember seeing it somewhere but can no longer find it.

Thanks

I am still getting the following from old deleted documents

2018-09-08 19:58:30.228583+0700 6Degrees[2951:1584053] WARNING: CBLRestPusher[http://138.197.203.204:4984/six_degreez_db/]: _bulk_docs got an error: {
error = “not_found”;
id = “errorLog_1.3.0_user_1534945269005”;
reason = missing;
status = 404;
} {at -[CBLRestPusher uploadBulkDocs:changes:]_block_invoke:402}

For some reason my phone (couchbase lite) is storing these documents and trying to push them to the server.

How do I delete these from the phone? Are they dead rows in a table somewhere? I cannot find where this error is even printing out in the source code.

Thank you,
Sean

You can use the purge method to nuke the documents from the db … but the implication of the error message is that the document doesn’t exist (in which case it’s mysterious why the replicator is trying to push it.)