Couchbase Lite to CouchDB filtered replication in Android

Mobile
1

Hi. I have setup a CouchDB 2.0 server and I’m trying to pull filtered documents to the mobile database (v1.3.1).

If I don’t use a filter, all documents are pulled correctly.

If I set a filter in the Replication object (just setting the filter property in the replication object with the format designName/filter) I only get the initial changed event in the event listener but not the rest of the replication events and the documents are not pulled.

In the Android Logcat I can see that it is retrying the replication requests every few seconds but for each request that it tries it receives a time out exception

ChangeTracker{http://serverurl, OneShot, @59b2fe5}: Exception in change tracker java.net.SocketTimeoutException

By monitoring the server logs (its behind an nginx proxy) I can see multiple 499 errors (client request closed) on the POST call to the filtered feed, one for every timeout that I receive above.

If I copy the URL reported in the nginx logs and use it in Postman I can get to the changes feed, filtered to the correct documents, so the filter seems to be working fine.

Any hints on why that might be happening? What else can I look into?

Thanks,

Alejandro

2

I have debugged the app a bit and I have been able to reproduce the timeout by using Postman if I send the exact same request body that the replication process is doing. In fact, adding any type of request body to that call (POST to the changes feed) makes que request to timeout.

If I send a POST with an empty body, or a GET request to the _changes feed, I get the filtered results just fine (passing the parameters in the query string as the replication object also does).

Is this an issue with CouchDB 2.0? Any extra hints?

Thanks,

Alejandro

3

Reading the CouchDB replication protocol spec, it appears that the _changes feed request should be a GET request:

http://docs.couchdb.org/en/2.0.0/replication/protocol.html#listen-to-changes-feed

the Replicator SHOULD read the Source’s Changes Feed by using a GET /{db}/_changes request. This request MUST be made with the following query parameters

There is a POST request to _changes in the spec but it relates to specific document ids (http://docs.couchdb.org/en/2.0.0/api/database/changes.html#post--db-_changes) but the POST request that the CouchbaseLite Android implementation is making seems to be wrong.

Could this be a bug in the Android implementation then?

Thanks,

Alejandro

4

Hi @alejandro.mezcua,

Thank you for reporting this issue and investgating further.

Current implementation of CBL for Android does not meet the /_changes REST API spec of CouchDB v2.0. I filed the ticket to the github repo. We will track this issue with the ticket.

https://github.com/couchbase/couchbase-lite-java-core/issues/1516

Thanks,
Hideki

5

Thanks.

I have build a patched Couchbase Lite library that issues a GET call there instead of a POST and it appears to work, but I don’t know yet if that has any side effects that I’m unaware of at this moment.

Alejandro

6

For what it’s worth others have reported issues with POST calls to CouchDB even on 1.6.1. Wasn’t there a ticket pending in java-core to select whether or not to use POST? I added this for .NET.

However, the bottom line is that we are not bound to what CouchDB does as they are a separate entity and so we may or may not decide to support things like this so buyer be warned.

7

Hi @alejandro.mezcua,

If CouchDB v2.0’s GET /_changes spec is same with v1.6.1 or earlier, forcing to use GET or fallback to GET from POST should be fine.

8

@borrrden Fair enough but then I would consider the way the documentation is written as it states that filtered replication is supported against CouchDB (http://developer.couchbase.com/documentation/mobile/current/guides/couchbase-lite/native-api/replication/index.html#filtered-pull-from-couchdb-pouchdb-or-cloudant)

@hideki I’ll keep testing the patch I made to see if I find any more issues.

Thanks,

Alejandro

9

Oh goody. I didn’t know we had docs specifically dealing with CouchDB…well in any case the fix you used would be the fix we implement most likely.

10

I have exactly the same problem. Is there a published solution?

11

I am having the same problem as well. It looks like the filter function passed via replication.setFilter() is appearing in both the query string and the doc body, but the filter parameter set via replication.setFilterParams() is not. This is a problem with CouchDB, since it doesn’t look at the POST body unless the filter query param is set to _doc_ids or _selector. Actually, there’s also appears to be a bug in CouchDB 2.0.0 where if a POST body is present when doing a POST to _changes, the server times out and returns a 500 code. I’m also looking into that issue.

Is there any reason that the Couchbase Lite replicator uses the POST to _changes rather than using a GET?

There’s an additional issue, in that the documentation for POSTs to _changes differs between CouchDB/Cloudant and Couchbase Sync Gateway. For CouchDB, the body is only used if the fliter query param is set to _doc_ids or _selector, where the Couchbase documentation mentions putting the query params in the body, so this is a divergence in the shared replication protocol.

12

@riebobaz, @alejandro.mezcua, @jfspencer

It seems that you have, at one time or another, expressed an interest in using CBL with CouchDB. It’s my understanding that, going forward, the CBL team will not be maintaining compatibility with CouchDB replication. Our team has also been using CBL for our client development, but are considering other options given the compatibility issues. Are you still using CBL with CouchDB, and do you have plans to continue longer-term? If so what’s your plan for dealing with compatibility?

Thanks!