Binary document keys in sync gateway?

Mobile Sync Gateway
1

Hi! Our business requirements are causing us to migrate all of our data (hundreds of millions of documents) into a different data store. We’ve been working and working to figure out how to continuously write changes from Couchbase into the other data store using home-baked solutions, but ran into problems with each of them. Today we learned more about the sync gateway, and it looks like it could be golden, the exact thing we need. Except one big problem. A good portion of our documents are binary docs, and those don’t appear to show up at all in the _changes SG REST API.

I seem to be able to get the binary document body API just fine if I already know the key, and I’m using the single doc endpoint, but what I really really need is for the key of the binary doc to show up in the change feed at all. Any tips? Is this possible?

We’d really love to avoid either doing a massive migration to try and change our binary docs into base64 JSON strings, or implementing our own DCP client to handle binary docs.

Thanks!

Couchbase Server version 5.1.1
SG version 2.6

Config:

{
  "log": ["*"],
  "databases": {
    "sync": {
      "server": "127.0.0.1",
      "bucket": "Sync",
      "username": "*********",
      "password": "*********",
      "enable_shared_bucket_access": true,
      "import_docs": true,
      "use_views": true,
      "num_index_replicas": 0,

      "users": {
        "GUEST": { "disabled": false, "admin_channels": ["*"] }
      },
      "sync": `function (doc, oldDoc) {
        if (doc.sdk) {
          channel(doc.sdk);
        }
      }`
    }
  }
}
2

Couchbase Mobile, including Sync Gateway, only works with JSON documents. You can have binary “attachments” aka “blobs” in a document, but the document itself must be JSON.

3

Okay, thanks Jens. We ended up solving our problem by using Couchbase’s Java DCP client library, which streams changes to all documents. Thanks!

Link for any interested: https://github.com/couchbase/java-dcp-client