SG REST Admin API gives misleading response

benjamin_glatzeder · June 12, 2019, 2:55pm

Hi,

the output of a SG REST Admin API function is misleading. Below is a sample Java web app that creates a document via SG REST Admin API function[1]. The document has the contents color=green. Then it’s updated via the same function to color=red.

The output of the update function is:

{“error”:“conflict”,“reason”:“Document revision conflict”}

But the document is updated when checking the document via Couchbase console (web). It also has an updated revision, i.e. 2-xxx…

Why does the output show a revision conflict please? Is this a bug?

Thanks!

[1] https://docs.couchbase.com/sync-gateway/current/admin-rest-api.html#/document/put__db___doc_

    @Override
    protected void doPost(HttpServletRequest req, HttpServletResponse resp) {
        try {
            /* init cluster*/
            CouchbaseCluster cluster = CouchbaseCluster.create("localhost");
            cluster.authenticate("sync_gateway", "my_pass");
            Bucket bucket = cluster.openBucket("my_bucket");
            long docName = System.currentTimeMillis();
            log.info("docName = " + docName);
            String urlEncodedDocName = URLEncoder.encode(String.valueOf(docName), "UTF-8");
            JsonObject content = JsonObject.empty().put("color", "green");
            ProcessBuilder pb = new ProcessBuilder("curl", "-X", "PUT",
                    "http://localhost:4985/my_database/" + urlEncodedDocName + "?new_edits=true", "-H", "accept: application/json",
                    "-H", "Content-Type: application/json", "-d", content.toString());
            pb.start();
            pb.redirectErrorStream(true);
            Process process = pb.start();
            String s = IOUtils.toString(process.getInputStream(), StandardCharsets.UTF_8.name());
            log.info("process = " + s);
            /* Get revision */
            String queryString = "SELECT meta().xattrs._sync.rev FROM my_bucket USE KEYS '" + docName + "';";
            log.info("queryString = " + queryString);
            N1qlQuery query = N1qlQuery.simple(queryString, N1qlParams.build());
            N1qlQueryResult result = bucket.query(query);
            String rev = result.allRows().get(0).value().getString("rev");
            log.info("rev = " + rev);
            /* Update document */
            content = JsonObject.empty().put("color", "red");
            pb = new ProcessBuilder("curl", "-X", "PUT",
                    "http://localhost:4985/my_database/" + urlEncodedDocName + "?new_edits=true&rev=" + rev, "-H",
                    "accept: application/json", "-H", "Content-Type: application/json", "-d", content.toString());
            pb.start();
            pb.redirectErrorStream(true);
            process = pb.start();
            s = IOUtils.toString(process.getInputStream(), StandardCharsets.UTF_8.name());
            log.info("process = " + s);
        } catch (Exception e) {
            e.printStackTrace();
        }
    }

priya.rajagopal · June 17, 2019, 12:34pm

I’d suggest instrumenting your code to log the request and corresponding response and review the log output to see order in which operations happen. If the document is in fact persisted on server then chances are that there is some sort of race condition in your code and in fact your second request is getting through before the first and that’s why you are seeing the updates and the error that you are seeing is for the first request. Check the revision of the document that was persisted.ed
On related note, you are using N1QL query to get the rev. We discourage the direct use of the meta().xattrs from apps - that is intended to be used internally for sync.
The recommended pattern is for the app to first fetch the document that is persisted , get the revId and then use it.

jens · June 17, 2019, 8:31pm

Unless you delete the document (or flush the bucket) every time you run this code, the document from the last run is still going to exist, causing the first PUT to fail. That’s probably what you’re seeing.

Some other minor issues with the code:

Invoking a curl process to send an HTTP request from Java seems really roundabout, since Java already has its own HTTP client APIs.
You don’t need the new_edits=true option or the Accept: header.
The PUT request already returns the document’s revision ID in the response. Don’t make a separate request to get it; that’s prone to race conditions.

benjamin_glatzeder · June 17, 2019, 8:42pm

Hi Priya,

thank you for the explanation and the warning not to use it. My original problem was to update a document via SG Admin REST API and I needed to get the latest revision of an existing document. [1]

I ran the original code twice and further down are the logs. As far as I can tell the document ist created as it returns an id, status ok=true and a revision. The first time I ran the code updating worked fine. Second time there was a conflict. I started the code manually by clicking a button. The doc name is the current time in millis. So it’s not a for-loop which might create a race condition.

docName = 1560799655926
process = % Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0100 92 100 75 100 17 1960 444 --:--:-- --:--:-- --:--:-- 1973
{"id":"1560799655926","ok":true,"rev":"1-34b681f7e2d8fa8e4f4e75e95c14ee88"}
queryString = SELECT meta().xattrs._sync.rev FROM my_bucket USE KEYS '1560799655926';
rev = 1-34b681f7e2d8fa8e4f4e75e95c14ee88
process = % Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0100 90 100 75 100 15 13758 2751 --:--:-- --:--:-- --:--:-- 15000
{"id":"1560799655926","ok":true,"rev":"2-a000951bd8edf462fd063d47af1caa5b"}


docName = 1560799702281
process = % Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0100 92 100 75 100 17 10799 2447 --:--:-- --:--:-- --:--:-- 12500
{"id":"1560799702281","ok":true,"rev":"1-34b681f7e2d8fa8e4f4e75e95c14ee88"}
queryString = SELECT meta().xattrs._sync.rev FROM my_bucket USE KEYS '1560799702281';
rev = 1-34b681f7e2d8fa8e4f4e75e95c14ee88
process = % Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0100 73 100 58 100 15 9566 2474 --:--:-- --:--:-- --:--:-- 11600
{"error":"conflict","reason":"Document revision conflict"}

The new code following @graham.pople’s advice uses the Java SDK to get the latest revision:

/* init cluster*/
        CouchbaseCluster cluster = CouchbaseCluster.create("localhost");
        cluster.authenticate("sync_gateway", "my_pass");
        Bucket bucket = cluster.openBucket("my_bucket");
        /* Create document */
        long docName = System.currentTimeMillis();
        log.info("docName = " + docName);
        String urlEncodedDocName = URLEncoder.encode(String.valueOf(docName), "UTF-8");
        JsonObject content = JsonObject.empty().put("color", "green");
        ProcessBuilder pb = new ProcessBuilder("curl", "-X", "PUT", "http://localhost:4985/my_bucket/" + urlEncodedDocName + "?new_edits=true",
                "-H", "accept: application/json", "-H", "Content-Type: application/json", "-d", content.toString());
        pb.start();
        pb.redirectErrorStream(true);
        Process process = pb.start();
        String s = IOUtils.toString(process.getInputStream(), StandardCharsets.UTF_8.name());
        log.info("process = " + s);
        /* Get revision */
        DocumentFragment<Lookup> result = bucket.lookupIn(String.valueOf(docName)).get("_sync.rev", SubdocOptionsBuilder.builder().xattr(true)).execute();
        String rev = (String) result.content(0);
        log.info("rev = " + rev);
        /* Update document */
        content = JsonObject.empty().put("color", "red");
        pb = new ProcessBuilder("curl", "-X", "PUT", "http://localhost:4985/my_bucket/" + urlEncodedDocName + "?new_edits=true&rev=" + rev,
                "-H", "accept: application/json", "-H", "Content-Type: application/json", "-d", content.toString());
        pb.start();
        pb.redirectErrorStream(true);
        process = pb.start();
        s = IOUtils.toString(process.getInputStream(), StandardCharsets.UTF_8.name());
        log.info("process = " + s);

I tried 3 times and the third time a conflict occured. Here are the logs:

docName = 1560800656201
process = % Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0100 64 100 47 100 17 4034 1459 --:--:-- --:--:-- --:--:-- 4272
{"error":"conflict","reason":"Document exists"}
rev = 1-34b681f7e2d8fa8e4f4e75e95c14ee88
process = % Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0100 90 100 75 100 15 56137 11227 --:--:-- --:--:-- --:--:-- 75000
{"id":"1560800656201","ok":true,"rev":"2-a000951bd8edf462fd063d47af1caa5b"}


docName = 1560800686403
process = % Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0100 92 100 75 100 17 8679 1967 --:--:-- --:--:-- --:--:-- 9375
{"id":"1560800686403","ok":true,"rev":"1-34b681f7e2d8fa8e4f4e75e95c14ee88"}
rev = 1-34b681f7e2d8fa8e4f4e75e95c14ee88
process = % Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0100 90 100 75 100 15 13794 2758 --:--:-- --:--:-- --:--:-- 15000
{"id":"1560800686403","ok":true,"rev":"2-a000951bd8edf462fd063d47af1caa5b"}


docName = 1560800710049
process = % Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0100 64 100 47 100 17 3549 1283 --:--:-- --:--:-- --:--:-- 3615
{"error":"conflict","reason":"Document exists"}
rev = 1-34b681f7e2d8fa8e4f4e75e95c14ee88
process = % Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0100 73 100 58 100 15 9689 2505 --:--:-- --:--:-- --:--:-- 9666100 73 100 58 100 15 9460 2446 --:--:-- --:--:-- --:--:-- 8285
{"error":"conflict","reason":"Document revision conflict"}

Because of the above I believe that there is a bug when updating a document via SG Admin REST API. When you run the code do you also see this conflict? Or how would one update a document correctly with the SG please? Where did I go wrong in the code?

Thanks!

[1] https://www.couchbase.com/forums/t/retrieve-xattrs-sync-rev-via-n1ql-java-sdk-or-admin-rest-api/21759

benjamin_glatzeder · June 17, 2019, 8:52pm

Hi Jens,

thank you for following up as well. My overall goal is to delete an attachment of a document via a web server. I went with the Java SDK since that’s what I know best. But as far as I know deleting (??) an attachment is not possible via the Java SDK. Hence I went with the SG Admin REST API…

Before I keep repeating and probably misspell my goal I wrote it here a couple of days ago and thought this is the solution. Also I was unable to find documentation on how to work with attachments with the Java SDK. I’m happy to read the docs if you have a link! That’d be great!

Thanks!

jens · June 20, 2019, 4:56pm

Benjamin, I answered your question and explained the reason your code is failing. Did you not see that?

benjamin_glatzeder · June 21, 2019, 8:42am

Thanks to @jens I was able to solve the issue.

I also changed the prerequisites:

document exists already, i.e. it was created by CBL client
document is updated via Java http stack

/* init cluster code not included*/
String docName = "my_doc";
String urlEncodedDocName = URLEncoder.encode(docName, "UTF-8");
/* Get revision */
DocumentFragment<Lookup> result = bucket.lookupIn(docName).get("_sync.rev", SubdocOptionsBuilder.builder().xattr(true)).execute();
String rev = (String) result.content(0);
log.info("rev = " + rev);
/* Update document */
HttpClient httpClient = new DefaultHttpClient();
String uri = "http://localhost:4985/my_db/" + urlEncodedDocName + "?rev=" + rev;
log.info("uri = " + uri);
HttpPut httpPut = new HttpPut(uri);
StringEntity jsonEntity;
String payLoad = "{\"color\":\"green\"}";
jsonEntity = new StringEntity(payLoad);
httpPut.setEntity(jsonEntity);
httpPut.setHeader("Content-type", "application/json");
HttpResponse response = httpClient.execute(httpPut);
String s = IOUtils.toString(response.getEntity().getContent(), StandardCharsets.UTF_8);
log.info("response = " + s);

The responses never showed any conflict when testing. However using a curl statement without the accept header and the new_edits=true param there was still a document conflict logged. This happened about 30% of the time or so. Still the document always was updated when checking it, i.e. via Couchbase console in the browser. The code is the same as above but using curl instead of the Java http stack:

/* init cluster code not included*/
String docName = "my_doc";
String urlEncodedDocName = URLEncoder.encode(docName, "UTF-8");
/* Get revision */
DocumentFragment<Lookup> result = bucket.lookupIn(docName).get("_sync.rev", SubdocOptionsBuilder.builder().xattr(true)).execute();
String rev = (String) result.content(0);
log.info("rev = " + rev);
/* Update document */
JsonObject content = JsonObject.empty().put("color", "green");
ProcessBuilder pb = new ProcessBuilder("curl", "-X", "PUT", "http://localhost:4985/my_db/" + urlEncodedDocName +
"?rev=" + rev,
        "-H", "Content-Type: application/json", "-d", content.toString());
pb.start();
pb.redirectErrorStream(true);
Process process = pb.start();
String s = IOUtils.toString(process.getInputStream(), StandardCharsets.UTF_8.name());
log.info("process = " + s);

My conclusion is that this issue is solved but there might still be an issue when using curl. I used curl as it was easy to copy-paste the command when trying it out in the SG API explorer.

Thanks!

jens · June 21, 2019, 4:38pm

DO NOT DO THIS. The _sync property is for internal use only and you should never use or modify it. It doesn’t even exist in newer versions of Couchbase Server (where we’re able to use external metadata storage instead of having to stuff our metadata into the document.)

benjamin_glatzeder · June 21, 2019, 7:42pm

Thank you for the follow-up and the strong warning!

Admittedly I’m a bit confused as @graham.pople suggested this here: Retrieve xattrs._sync.rev via n1ql, Java SDK or admin rest api - #2 by graham.pople

As far as I understand the function uses the metadata which is stored outside of the document itself. I’m on Couchbase 6 and SG 2.x. So now it’s possible to edit documents via Java SDK (and SG REST API) and the changes get synced down to Couchbase lite clients. And as far as I know this was not possible in earlier versions. Not sure about bucket shadowing; I never used it.

Looking forward to some clarifications and a recommended way to get the latest revision of an existing document!

Thanks!

jens · June 21, 2019, 9:14pm

Hm. In the thread you linked to, they’re using xattrs._sync.rev. The xattrs is the external metadata storage I mentioned. This is somewhat safer, but I don’t know if it’s considered stable enough for developers to access directly. I’m also not sure whether the xattrs are updated the instant a document is saved or whether they’re more “eventually consistent”. (@adamf, can you confirm?)

Anyway, the best way to get the revision of a document is to use the SG REST API. Just GET the document and look at the _rev property of the response. Or if you’ve just created/updated the document with a PUT, the revision will also be available in the response.

benjamin_glatzeder · June 21, 2019, 9:32pm

Thank you for pointing this out. I’ll use the SG GET function. Thanks!

Topic		Replies	Views
Sync Function detect document conflicts Sync Gateway	3	1542	October 2, 2017
Document update conflict Couchbase Lite	5	3314	February 13, 2019
Conflict Resolution using Admin REST API – How to get “Common Parent” document Sync Gateway	6	1138	November 13, 2018
Client supplied _rev property in document when updating via Sync Gateway REST API Sync Gateway rest	2	709	December 13, 2022
Document revision issue Couchbase Server	5	3076	November 24, 2015

SG REST Admin API gives misleading response

Related topics