Cbimport not working as expected

Couchbase Server
1

I exported data using cbq command in a json file. When I tried to import the same data on another couchbase cluster, cbimport is throwing below error.

2021-11-27T17:54:27.251+05:30 ERRO: Failed to import document 1: Key generation for document failed, resulting field does not exist – jsondata.(*Source).handleError() at source.go:415
JSON import failed: 0 documents were imported, 1 documents failed to be imported
JSON import failed: Some errors occurred while transferring data, see logs for more details

However the key which I am mentioning in the command is exist in the file itself.

cbimport json -c http://{target-cluster}:8091 -u Administrator -p Administrator -b sample -d file://C:\Users{myusername}\Desktop\temp.json -f list -g %docId%

2

I think you need to ensure the case of the field name matches:

C:\tmp>type example.json
[
    {
        "data": "2021-11-28T07:50:08.332Z",
        "docId": "k1"
    }
]

C:\tmp>cbimport json -c http://localhost:8091 -u Administrator -p password -b default -d file://C:\tmp\example.json -f list -g %docId%
JSON `file://C:\tmp\example.json` imported to `http://localhost:8091` successfully
Documents imported: 1 Documents failed: 0

C:\tmp>type example2.json
[
    {
        "data": "2021-11-28T07:50:08.332Z",
        "docid": "k1"
    }
]

C:\tmp>cbimport json -c http://localhost:8091 -u Administrator -p password -b default -d file://C:\tmp\example2.json -f list -g %docId%
2021-11-28T08:22:49.975+00:00 ERRO: Failed to import document 1: key generation for document failed, resulting field does not exist -- jsondata.(*Source
).handleError() at source.go:421
JSON import failed: 0 documents were imported, 1 documents failed to be imported
JSON import failed: Some errors occurred while transferring data, see logs for more details

(Notice the only difference is “docId” vs “docid” in the JSON source.)

Further you should make sure there isn’t accidental shell expansion when using %…%; typically use ^%…^% to avoid this:

C:\tmp>set docId="Shouldn't be the key"

C:\tmp>cbimport json -c http://localhost:8091 -u Administrator -p password -b default -d file://C:\tmp\example.json -f list -g %docId%
JSON `file://C:\tmp\example.json` imported to `http://localhost:8091` successfully
Documents imported: 1 Documents failed: 0

C:\tmp>cbimport json -c http://localhost:8091 -u Administrator -p password -b default -d file://C:\tmp\example.json -f list -g ^%docId^%
JSON `file://C:\tmp\example.json` imported to `http://localhost:8091` successfully
Documents imported: 1 Documents failed: 0

C:\tmp>cbq -u Administrator -p password -script "select meta().id,* from default"
 Connected to : http://localhost:8091/. Type Ctrl-D or \QUIT to exit.

 Path to history file for the shell : C:\Users\dhaggart\.cbq_history

 select meta().id,* from default
{
    "requestID": "d3b1f7df-2151-4e22-b5b7-7ea49fcaed44",
    "signature": {
        "*": "*",
        "id": "json"
    },
    "results": [
    {
        "default": {
            "data": "2021-11-28T07:50:08.332Z",
            "docId": "k1"
        },
        "id": "Shouldn't be the key"
    },
    {
        "default": {
            "data": "2021-11-28T07:50:08.332Z",
            "docId": "k1"
        },
        "id": "k1"
    }
    ],
    "status": "success",
    "metrics": {
        "elapsedTime": "4.5477ms",
        "executionTime": "4.5477ms",
        "resultCount": 2,
        "resultSize": 279,
        "serviceLoad": 25
    }
}

If it isn’t this, please share the version you’re using.

HTH.

3

Hi,

Yes it worked. However I still have one more query.

I am extracting data from cbq command into a file but it has signature, metrics, commands, results in it. I only want the result in the file. Is there a way to ignore remaining filed the file.

FYI:

  1. I am using \REDIRECT to extract data into a file.
  2. using \set metrics/Signature I was able to ignore these two fields. But still I am seeing the commands, extra-spaces, status fields in it other than result.

Any help would be appreciated.

4

No, you can’t “export” from a statement in cbq; you have to edit the output to produce a cbimport compatible file. You may be able to script this, or use a tool such as “jq” to filter the output.

C:\tmp>cbq -q -u Administrator -p password -s "select now_str() data"|jq-win64 .results
[
  {
    "data": "2021-11-28T12:09:26.946Z"
  }
]

In 7.1 there is “terse” mode which will permit less editing, but it will still require statement and “\redirect off” removal from the output file. It is after all output redirection, not export still – and remember, cbq must still report errors etc. hence the “results” field.

5

Thanks for the detailed explanation.