Cbbackupmgr restoring same gsi index multiple time which is causing an error

zizou · April 23, 2019, 12:47pm

Hi,

I’m using cbbackupmgr on couchabse 5.1 for backup/restore of a cluster of couchbase (3 nodes).
The backup is done, I was trying to restore my data to the same cluster (after purging data/indexes) but the restore is failing .Checking the server indexer log, Im noticing a strange behaviour:
the cluster is trying to restore some gsi indexed multiple times which is causing “index already exists” error.
I have attached the indexer log and we can check for example the server is trying to restore the primary index and the index ss_index twice which is causing the fail.

Any help please, this is an urgent production issue.

[Info] RestoreContext:  Processing index in backup image (bucket_2, #primary, 0).
[Info] RestoreContext:  Processing index in backup image (bucket_2, date_index, 0).
[Info] RestoreContext:  Processing index in backup image (bucket_2, ss_index, 0).
[Info] RestoreContext:  Processing index in backup image (bucket_2, rule_index, 0).
[Info] RestoreContext:  Processing index in backup image (bucket_2, c_date_index, 0).
[Info] RestoreContext:  Processing index in backup image (bucket_2, #primary, 0).
[Info] RestoreContext:  Processing index in backup image (bucket_2, context_index, 0).
[Info] RestoreContext:  Processing index in backup image (bucket_2, audit_index, 0).
[Info] RestoreContext:  Processing index in backup image (bucket_2, rule_index, 0).
[Info] RestoreContext:  Processing index in backup image (bucket_2, ss_index, 0).
[Info] RestoreContext:  Processing index in backup image (bucket_2, date_index, 0).
[Info] RestoreContext:  Processing index in backup image (bucket_2, archive_index, 0).
[Info] RestoreContext:  Processing index in backup image (bucket_2, search_index, 0).
[Info] RestoreContext:  Processing index in backup image (bucket_2, archive_index, 0).
[Info] RestoreContext:  Processing index in backup image (bucket_2, c_date_index, 0).
[Info] RestoreContext:  Processing index in backup image (bucket_2, context_index, 0).
[Info] RestoreContext:  Processing index in backup image (bucket_2, search_index, 0).
[Info] RestoreContext:  Processing index in backup image (bucket_2, audit_index, 0).
[Info] ClustMgr:handleStats &{92 <nil> false}
[Info] RestoreContext:  Index (bucket_2, #primary, 0) does not exist in current cluster.  Make it a restore candidate.
[Info] RestoreContext:  Index (bucket_2, date_index, 0) does not exist in current cluster.  Make it a restore candidate.
[Info] RestoreContext:  Index (bucket_2, ss_index, 0) does not exist in current cluster.  Make it a restore candidate.
[Info] RestoreContext:  Index (bucket_2, rule_index, 0) does not exist in current cluster.  Make it a restore candidate.
[Info] RestoreContext:  Index (bucket_2, c_date_index, 0) does not exist in current cluster.  Make it a restore candidate.
[Info] RestoreContext:  Index (bucket_2, #primary, 0) does not exist in current cluster.  Make it a restore candidate.
[Info] RestoreContext:  Index (bucket_2, context_index, 0) does not exist in current cluster.  Make it a restore candidate.
[Info] RestoreContext:  Index (bucket_2, audit_index, 0) does not exist in current cluster.  Make it a restore candidate.
[Info] RestoreContext:  Index (bucket_2, rule_index, 0) does not exist in current cluster.  Make it a restore candidate.
[Info] RestoreContext:  Index (bucket_2, ss_index, 0) does not exist in current cluster.  Make it a restore candidate.
[Info] RestoreContext:  Index (bucket_2, date_index, 0) does not exist in current cluster.  Make it a restore candidate.
[Info] RestoreContext:  Index (bucket_2, archive_index, 0) does not exist in current cluster.  Make it a restore candidate.
[Info] RestoreContext:  Index (bucket_2, search_index, 0) does not exist in current cluster.  Make it a restore candidate.
[Info] RestoreContext:  Index (bucket_2, archive_index, 0) does not exist in current cluster.  Make it a restore candidate.
[Info] RestoreContext:  Index (bucket_2, c_date_index, 0) does not exist in current cluster.  Make it a restore candidate.
[Info] RestoreContext:  Index (bucket_2, context_index, 0) does not exist in current cluster.  Make it a restore candidate.
[Info] RestoreContext:  Index (bucket_2, search_index, 0) does not exist in current cluster.  Make it a restore candidate.
[Info] RestoreContext:  Index (bucket_2, audit_index, 0) does not exist in current cluster.  Make it a restore candidate.
[Info] RestoreContext:  Index Layout before planning
[Info] 	RestoreContext:  Indexer NodeId node1:8091 IndexerId 080900281ad4da7Xaa5841c813666f5ea
[Info] 	RestoreContext:  Indexer NodeId node2:8091 IndexerId 7b1e39c9f6a40577Xa58bdb0467e40e54
[Info] 	RestoreContext:  Indexer NodeId node3:8091 IndexerId 70cabc3210ea4828Xc25c73da4a24d8e2
[Info] 	RestoreContext:  Indexer NodeId 4abaa0604e5c671f4c71f15c434c8bd8 IndexerId 4abaa0604e5c671f4c71f15c434c8bd8
[Info] 		RestoreContext:  Index (bucket_2, #primary, 0)
[Info] 		RestoreContext:  Index (bucket_2, date_index, 0)
[Info] 		RestoreContext:  Index (bucket_2, ss_index, 0)
[Info] 		RestoreContext:  Index (bucket_2, rule_index, 0)
[Info] 	RestoreContext:  Indexer NodeId c2c5093159fc59X56293b27f207e9f44e IndexerId c2c5093159fc59X56293b27f207e9f44e
[Info] 		RestoreContext:  Index (bucket_2, c_date_index, 0)
[Info] 		RestoreContext:  Index (bucket_2, #primary, 0)
[Info] 		RestoreContext:  Index (bucket_2, context_index, 0)
[Info] 		RestoreContext:  Index (bucket_2, audit_index, 0)
[Info] 		RestoreContext:  Index (bucket_2, rule_index, 0)
[Info] 		RestoreContext:  Index (bucket_2, ss_index, 0)
[Info] 		RestoreContext:  Index (bucket_2, date_index, 0)
[Info] 		RestoreContext:  Index (bucket_2, archive_index, 0)
[Info] 		RestoreContext:  Index (bucket_2, search_index, 0)
[Info] 	RestoreContext:  Indexer NodeId e8d85c10940cd08c917da66ff480c976 IndexerId e8d85c10940cd08c917da66ff480c976
[Info] 		RestoreContext:  Index (bucket_2, archive_index, 0)
[Info] 		RestoreContext:  Index (bucket_2, c_date_index, 0)
[Info] 		RestoreContext:  Index (bucket_2, context_index, 0)
[Info] 		RestoreContext:  Index (bucket_2, search_index, 0)
[Info] 		RestoreContext:  Index (bucket_2, audit_index, 0)
[Info] ************ Indexer Layout *************
[Info] ----------------------------------------
[Info] Memory Quota: 7516192768 (7G)
[Info] CPU Quota: 32
[Info] ----------------------------------------
[Info] Indexer Memory Mean 0 (0)
[Info] Indexer Memory Deviation 0 (0) (0.00%)
[Info] Indexer Memory Utilization 0.0000
[Info] Indexer CPU Mean 0.0000
[Info] Indexer CPU Deviation 0.00 (0.00%)
[Info] Indexer CPU Utilization 0.0000
[Info] Total Index Data (in original layout) 0
[Info] Index Data Moved (after planning) 0 (0.00%)
[Info] No. Index (in original layout) 0
[Info] No. Index Moved (after planning) 0 (0.00%)
[Info] ----------------------------------------
[Info] 
[Info] Indexer serverGroup:Group 1, nodeId:node1:8091, useLiveData:true
[Info] Indexer total memory:0 (0), data:0 (0), overhead:0 (0), cpu:0.0000, number of indexes:5 isDeleted:false isNew:true
[Info] 		------------------------------------------------------------------------------------------------------------------
[Info] 		Index name:archive_index, bucket:bucket_2, defnId:2069453184060899133, instId:1909686809119982976, new/moved:true defer:false ignoreEquivCheck:false
[Info] 		Index total memory:0 (0), data:0 (0), overhead:0 (0), cpu:0.0000
[Info] 		------------------------------------------------------------------------------------------------------------------
[Info] 		Index name:c_date_index, bucket:bucket_2, defnId:11291573438480902931, instId:8193215093999208328, new/moved:true defer:false ignoreEquivCheck:false
[Info] 		Index total memory:0 (0), data:0 (0), overhead:0 (0), cpu:0.0000
[Info] 		------------------------------------------------------------------------------------------------------------------
[Info] 		Index name:context_index, bucket:bucket_2, defnId:12866538974849553918, instId:12128833875134126828, new/moved:true defer:false ignoreEquivCheck:false
[Info] 		Index total memory:0 (0), data:0 (0), overhead:0 (0), cpu:0.0000
[Info] 		------------------------------------------------------------------------------------------------------------------
[Info] 		Index name:search_index, bucket:bucket_2, defnId:13505671716855971152, instId:4419719151511314113, new/moved:true defer:false ignoreEquivCheck:false
[Info] 		Index total memory:0 (0), data:0 (0), overhead:0 (0), cpu:0.0000
[Info] 		------------------------------------------------------------------------------------------------------------------
[Info] 		Index name:audit_index, bucket:bucket_2, defnId:15830718730262099701, instId:14543496332732822243, new/moved:true defer:false ignoreEquivCheck:false
[Info] 		Index total memory:0 (0), data:0 (0), overhead:0 (0), cpu:0.0000
[Info] 
[Info] Indexer serverGroup:Group 1, nodeId:node2:8091, useLiveData:true
[Info] Indexer total memory:0 (0), data:0 (0), overhead:0 (0), cpu:0.0000, number of indexes:9 isDeleted:false isNew:true
[Info] 		------------------------------------------------------------------------------------------------------------------
[Info] 		Index name:c_date_index, bucket:bucket_2, defnId:11291573438480902931, instId:7538599140524132818, new/moved:true defer:false ignoreEquivCheck:false
[Info] 		Index total memory:0 (0), data:0 (0), overhead:0 (0), cpu:0.0000
[Info] 		------------------------------------------------------------------------------------------------------------------
[Info] 		Index name:#primary, bucket:bucket_2, defnId:15727696393653815423, instId:7658393146625706509, new/moved:true defer:false ignoreEquivCheck:false
[Info] 		Index total memory:0 (0), data:0 (0), overhead:0 (0), cpu:0.0000
[Info] 		------------------------------------------------------------------------------------------------------------------
[Info] 		Index name:context_index, bucket:bucket_2, defnId:12866538974849553918, instId:9131996837407901675, new/moved:true defer:false ignoreEquivCheck:false
[Info] 		Index total memory:0 (0), data:0 (0), overhead:0 (0), cpu:0.0000
[Info] 		------------------------------------------------------------------------------------------------------------------
[Info] 		Index name:audit_index, bucket:bucket_2, defnId:15830718730262099701, instId:16897468386531290402, new/moved:true defer:false ignoreEquivCheck:false
[Info] 		Index total memory:0 (0), data:0 (0), overhead:0 (0), cpu:0.0000
[Info] 		------------------------------------------------------------------------------------------------------------------
[Info] 		Index name:rule_index, bucket:bucket_2, defnId:7617837682162938941, instId:6041135326713349948, new/moved:true defer:false ignoreEquivCheck:false
[Info] 		Index total memory:0 (0), data:0 (0), overhead:0 (0), cpu:0.0000
[Info] 		------------------------------------------------------------------------------------------------------------------
[Info] 		Index name:ss_index, bucket:bucket_2, defnId:13286887612614279168, instId:12426977418396683569, new/moved:true defer:false ignoreEquivCheck:false
[Info] 		Index total memory:0 (0), data:0 (0), overhead:0 (0), cpu:0.0000
[Info] 		------------------------------------------------------------------------------------------------------------------
[Info] 		Index name:date_index, bucket:bucket_2, defnId:16655560177260919351, instId:13678697307690311920, new/moved:true defer:false ignoreEquivCheck:false
[Info] 		Index total memory:0 (0), data:0 (0), overhead:0 (0), cpu:0.0000
[Info] 		------------------------------------------------------------------------------------------------------------------
[Info] 		Index name:archive_index, bucket:bucket_2, defnId:2069453184060899133, instId:1102663903838181114, new/moved:true defer:false ignoreEquivCheck:false
[Info] 		Index total memory:0 (0), data:0 (0), overhead:0 (0), cpu:0.0000
[Info] 		------------------------------------------------------------------------------------------------------------------
[Info] 		Index name:search_index, bucket:bucket_2, defnId:13505671716855971152, instId:6113395698417029322, new/moved:true defer:false ignoreEquivCheck:false
[Info] 		Index total memory:0 (0), data:0 (0), overhead:0 (0), cpu:0.0000
[Info] 
[Info] Indexer serverGroup:Group 1, nodeId:node3:8091, useLiveData:true
[Info] Indexer total memory:0 (0), data:0 (0), overhead:0 (0), cpu:0.0000, number of indexes:4 isDeleted:false isNew:true
[Info] 		------------------------------------------------------------------------------------------------------------------
[Info] 		Index name:#primary, bucket:bucket_2, defnId:15727696393653815423, instId:4096245016537395240, new/moved:true defer:false ignoreEquivCheck:false
[Info] 		Index total memory:0 (0), data:0 (0), overhead:0 (0), cpu:0.0000
[Info] 		------------------------------------------------------------------------------------------------------------------
[Info] 		Index name:date_index, bucket:bucket_2, defnId:16655560177260919351, instId:15679718370850184967, new/moved:true defer:false ignoreEquivCheck:false
[Info] 		Index total memory:0 (0), data:0 (0), overhead:0 (0), cpu:0.0000
[Info] 		------------------------------------------------------------------------------------------------------------------
[Info] 		Index name:ss_index, bucket:bucket_2, defnId:13286887612614279168, instId:266880605015013666, new/moved:true defer:false ignoreEquivCheck:false
[Info] 		Index total memory:0 (0), data:0 (0), overhead:0 (0), cpu:0.0000
[Info] 		------------------------------------------------------------------------------------------------------------------
[Info] 		Index name:rule_index, bucket:bucket_2, defnId:7617837682162938941, instId:8872793381245656402, new/moved:true defer:false ignoreEquivCheck:false
[Info] 		Index total memory:0 (0), data:0 (0), overhead:0 (0), cpu:0.0000
[Info] ****************************************
[Info] RestoreContext:  Restoring index (bucket_2, #primary, 0) at indexer node2:8091
[Info] RestoreContext:  Restoring index (bucket_2, date_index, 0) at indexer node2:8091
[Info] RestoreContext:  Restoring index (bucket_2, ss_index, 0) at indexer node2:8091
[Info] RestoreContext:  Restoring index (bucket_2, rule_index, 0) at indexer node2:8091
[Info] RestoreContext:  Restoring index (bucket_2, c_date_index, 0) at indexer node1:8091
[Info] RestoreContext:  Restoring index (bucket_2, #primary, 0) at indexer node2:8091
[Info] RestoreContext:  Restoring index (bucket_2, context_index, 0) at indexer node1:8091
[Info] RestoreContext:  Restoring index (bucket_2, audit_index, 0) at indexer node1:8091
[Info] RestoreContext:  Restoring index (bucket_2, rule_index, 0) at indexer node2:8091
[Info] RestoreContext:  Restoring index (bucket_2, ss_index, 0) at indexer node2:8091
[Info] RestoreContext:  Restoring index (bucket_2, date_index, 0) at indexer node2:8091
[Info] RestoreContext:  Restoring index (bucket_2, archive_index, 0) at indexer node1:8091
[Info] RestoreContext:  Restoring index (bucket_2, search_index, 0) at indexer node1:8091
[Info] RestoreContext:  Restoring index (bucket_2, archive_index, 0) at indexer node1:8091
[Info] RestoreContext:  Restoring index (bucket_2, c_date_index, 0) at indexer node1:8091
[Info] RestoreContext:  Restoring index (bucket_2, context_index, 0) at indexer node1:8091
[Info] RestoreContext:  Restoring index (bucket_2, search_index, 0) at indexer node1:8091
[Info] RestoreContext:  Restoring index (bucket_2, audit_index, 0) at indexer node1:8091
[Info] Refreshing indexer list due to cluster changes or auto-refresh.
[Info] Refreshed Indexer List: [node1:9100 node2:9100 node3:9100]
[Info] switched currmeta from 3 -> 5 force true 
[Info] Refreshing indexer list due to cluster changes or auto-refresh.
[Info] Refreshed Indexer List: [node1:9100 node2:9100 node3:9100]
[Info] switched currmeta from 5 -> 6 force true 
[Info] Refreshing indexer list due to cluster changes or auto-refresh.
[Info] Refreshed Indexer List: [node1:9100 node2:9100 node3:9100]
[Info] switched currmeta from 6 -> 8 force true 
[Info] Refreshing indexer list due to cluster changes or auto-refresh.
[Info] Refreshed Indexer List: [node1:9100 node2:9100 node3:9100]
[Info] switched currmeta from 8 -> 11 force true 
[Info] Refreshing indexer list due to cluster changes or auto-refresh.
[Info] Refreshed Indexer List: [node1:9100 node2:9100 node3:9100]
[Info] switched currmeta from 11 -> 11 force true 
[Info] Refreshing indexer list due to cluster changes or auto-refresh.
[Info] Refreshed Indexer List: [node1:9100 node2:9100 node3:9100]
[Info] switched currmeta from 11 -> 12 force true 
[Info] Refreshing indexer list due to cluster changes or auto-refresh.
[Info] Refreshed Indexer List: [node1:9100 node2:9100 node3:9100]
[Info] switched currmeta from 12 -> 14 force true 
[Error] requestHandler.makeCreateIndexRequest(): create index request fails. Error=Index bucket_2.#primary already exists
iple response.WriteHeader calls
[Error] requestHandler.makeCreateIndexRequest(): create index request fails. Error=Index bucket_2.rule_index already exists
[Error] requestHandler.makeCreateIndexRequest(): create index request fails. Error=Index bucket_2.ss_index already exists

zizou · April 24, 2019, 8:13am

Any help please on this topic?

pvarley · April 24, 2019, 4:57pm

Would it be possible to share the error message from cbbackupmgr?

As a workaround cbbackupmgr restore can be configured to not restore the indexes:

 --disable-gsi-indexes    Disables restoring GSI index definitions

For example:

cbbackupmgr restore -a /backup -r  repo-1 -u Administrator -p password -c localhost  --disable-gsi-indexes

zizou · April 24, 2019, 6:24pm

Thanks @pvarley ,
The error showing on the restore is:

(Rest) POST http://node1:9102/restoreIndexMetadata?bucket=bucket_2 500
e[31mERRO: Internal server error while executing "POST http://node1:9102/restoreIndexMetadata?bucket=bucket_2" check the server logs for more detailse[0me[2m -- plan.(*gsi).execute() at gsi.go:64e[0m

For disabling gsi restoration,yes,I know it but we are building an utility on top of the cbbakcupmanager which is not exposing this parameter and also we need to restore gsi indexes as we have many of them.
I will be grateful if you can help identifying the cause of this issue to be able to resolve it as it is a critical issue for us.

amit.kulkarni · April 25, 2019, 7:16pm

Hi @zizou,

As per the logs you attached:

[Info] RestoreContext:  Processing index in backup image (bucket_2, #primary, 0).
[Info] RestoreContext:  Processing index in backup image (bucket_2, date_index, 0).
[Info] RestoreContext:  Processing index in backup image (bucket_2, ss_index, 0).
[Info] RestoreContext:  Processing index in backup image (bucket_2, rule_index, 0).
[Info] RestoreContext:  Processing index in backup image (bucket_2, c_date_index, 0).
[Info] RestoreContext:  Processing index in backup image (bucket_2, #primary, 0).
[Info] RestoreContext:  Processing index in backup image (bucket_2, context_index, 0).
[Info] RestoreContext:  Processing index in backup image (bucket_2, audit_index, 0).
[Info] RestoreContext:  Processing index in backup image (bucket_2, rule_index, 0).
[Info] RestoreContext:  Processing index in backup image (bucket_2, ss_index, 0).
[Info] RestoreContext:  Processing index in backup image (bucket_2, date_index, 0).
[Info] RestoreContext:  Processing index in backup image (bucket_2, archive_index, 0).
[Info] RestoreContext:  Processing index in backup image (bucket_2, search_index, 0).
[Info] RestoreContext:  Processing index in backup image (bucket_2, archive_index, 0).
[Info] RestoreContext:  Processing index in backup image (bucket_2, c_date_index, 0).
[Info] RestoreContext:  Processing index in backup image (bucket_2, context_index, 0).
[Info] RestoreContext:  Processing index in backup image (bucket_2, search_index, 0).
[Info] RestoreContext:  Processing index in backup image (bucket_2, audit_index, 0).

The 3-tuples in these log messages represent (bucket name, index name, replica id). For each main index, there is one replica index present in the list, but the replica ids for both, main and replica index are zero. In regular scenario, the replica id for replica index should be 1 (not zero). So, when cbbackupmgr tries to restore two indexes with same name, same bucket name and same replica id, indexer returns the “index already exists” error. Due to this reason the restore is failing.

Is there any chance of the backup index metadata being corrupted/modified ?

cbbackupmgr creates index metadata file (gsi.json) in the archive directory (full path of index metadata file inside archive directory is <archive directory>/<repo name>/<backup timestamp>/<bucket name-bucket uuid>). If you can attach this file, I can further analyse this and suggest next steps.

Thanks.

zizou · April 26, 2019, 8:58am

Hi @pvarley
Please find below the content of file gsi.json (I have modified my index conditions , the where condition, to hide some internal data )

{
  "code": "success",
  "result": {
    "metadata": [
      {
        "definitions": [
          {
            "bucket": "bucket_2",
            "bucketUUID": "f0dbbb922bc1f6a126681a2924933721",
            "defnId": 18166560991698769723,
            "exprType": "N1QL",
            "isPrimary": true,
            "name": "#primary",
            "nodes": [
              "4abaa0604e5c671f4c71f15c434c8bd8"
            ],
            "partitionScheme": "SINGLE",
            "using": "plasma"
          },
          {
            "bucket": "bucket_2",
            "bucketUUID": "f0dbbb922bc1f6a126681a2924933721",
            "defnId": 2015963911817448609,
            "desc": [
              false,
              false
            ],
            "exprType": "N1QL",
            "name": "date_index",
            "nodes": [
              "4abaa0604e5c671f4c71f15c434c8bd8"
            ],
            "partitionScheme": "SINGLE",
            "secExprs": [
              "`balanceDate`",
              "`lastValidDate`"
            ],
            "using": "plasma",
            "where": "xxxxxxxxxx"
          },
          {
            "bucket": "bucket_2",
            "bucketUUID": "f0dbbb922bc1f6a126681a2924933721",
            "defnId": 6476249483949568856,
            "desc": [
              false
            ],
            "exprType": "N1QL",
            "name": "ss_index",
            "nodes": [
              "4abaa0604e5c671f4c71f15c434c8bd8"
            ],
            "partitionScheme": "SINGLE",
            "secExprs": [
              "`sourceSystemTradeId`"
            ],
            "using": "plasma",
            "where": "xxxxxxxx"
          },
          {
            "bucket": "bucket_2",
            "bucketUUID": "f0dbbb922bc1f6a126681a2924933721",
            "defnId": 6489124291496061185,
            "desc": [
              false,
              false,
              false,
              false
            ],
            "exprType": "N1QL",
            "name": "rule_index",
            "nodes": [
              "4abaa0604e5c671f4c71f15c434c8bd8"
            ],
            "partitionScheme": "SINGLE",
            "secExprs": [
              "(`event`.`id`)",
              "(`context`.`name`)",
              "`_type`",
              "`_versioned`"
            ],
            "using": "plasma",
            "where": "xxxxxx"
          }
        ],
        "indexerId": "4abaa0604e5c671f4c71f15c434c8bd8",
        "nodeUUID": "4abaa0604e5c671f4c71f15c434c8bd8",
        "storageMode": "plasma",
        "topologies": [
          {
            "bucket": "bucket_2",
            "definitions": [
              {
                "bucket": "bucket_2",
                "defnId": 18166560991698769723,
                "instances": [
                  {
                    "instId": 4096245016537395240,
                    "partitions": [
                      {
                        "keyPartition": {},
                        "singlePartition": {
                          "slices": [
                            {
                              "indexerId": "4abaa0604e5c671f4c71f15c434c8bd8"
                            }
                          ]
                        }
                      }
                    ],
                    "state": 4,
                    "steamId": 1,
                    "storageMode": "plasma"
                  }
                ],
                "name": "#primary"
              },
              {
                "bucket": "bucket_2",
                "defnId": 6476249483949568856,
                "instances": [
                  {
                    "instId": 266880605015013666,
                    "partitions": [
                      {
                        "keyPartition": {},
                        "singlePartition": {
                          "slices": [
                            {
                              "indexerId": "4abaa0604e5c671f4c71f15c434c8bd8"
                            }
                          ]
                        }
                      }
                    ],
                    "state": 4,
                    "steamId": 1,
                    "storageMode": "plasma"
                  }
                ],
                "name": "ss_index"
              },
              {
                "bucket": "bucket_2",
                "defnId": 2015963911817448609,
                "instances": [
                  {
                    "instId": 15679718370850184967,
                    "partitions": [
                      {
                        "keyPartition": {},
                        "singlePartition": {
                          "slices": [
                            {
                              "indexerId": "4abaa0604e5c671f4c71f15c434c8bd8"
                            }
                          ]
                        }
                      }
                    ],
                    "state": 4,
                    "steamId": 1,
                    "storageMode": "plasma"
                  }
                ],
                "name": "date_index"
              },
              {
                "bucket": "bucket_2",
                "defnId": 6489124291496061185,
                "instances": [
                  {
                    "instId": 8872793381245656402,
                    "partitions": [
                      {
                        "keyPartition": {},
                        "singlePartition": {
                          "slices": [
                            {
                              "indexerId": "4abaa0604e5c671f4c71f15c434c8bd8"
                            }
                          ]
                        }
                      }
                    ],
                    "state": 4,
                    "steamId": 1,
                    "storageMode": "plasma"
                  }
                ],
                "name": "rule_index"
              }
            ],
            "version": 19
          }
        ]
      },
      {
        "definitions": [
          {
            "bucket": "bucket_2",
            "bucketUUID": "f0dbbb922bc1f6a126681a2924933721",
            "defnId": 10824946879106943769,
            "desc": [
              false
            ],
            "exprType": "N1QL",
            "name": "c_date_index",
            "nodes": [
              "c2c5093159fc59X56293b27f207e9f44e"
            ],
            "partitionScheme": "SINGLE",
            "secExprs": [
              "`clearingDate`"
            ],
            "using": "plasma",
            "where": "xxxxx"
          },
          {
            "bucket": "bucket_2",
            "bucketUUID": "f0dbbb922bc1f6a126681a2924933721",
            "defnId": 8032106326720113472,
            "exprType": "N1QL",
            "isPrimary": true,
            "name": "#primary",
            "nodes": [
              "c2c5093159fc59X56293b27f207e9f44e"
            ],
            "partitionScheme": "SINGLE",
            "using": "plasma"
          },
          {
            "bucket": "bucket_2",
            "bucketUUID": "f0dbbb922bc1f6a126681a2924933721",
            "defnId": 7442352253566377791,
            "desc": [
              false,
              false,
              false,
              false
            ],
            "exprType": "N1QL",
            "name": "context_index",
            "nodes": [
              "c2c5093159fc59X56293b27f207e9f44e"
            ],
            "partitionScheme": "SINGLE",
            "secExprs": [
              "(`resultEntity`.`id`)",
              "(`context`.`name`)",
              "`_type`",
              "`_versioned`"
            ],
            "using": "plasma",
            "where": "xxxxxxxxxx"
          },
          {
            "bucket": "bucket_2",
            "bucketUUID": "f0dbbb922bc1f6a126681a2924933721",
            "defnId": 14844685067983095254,
            "desc": [
              false,
              false,
              false,
              false
            ],
            "exprType": "N1QL",
            "name": "audit_index",
            "nodes": [
              "c2c5093159fc59X56293b27f207e9f44e"
            ],
            "partitionScheme": "SINGLE",
            "secExprs": [
              "`_type`",
              "`_versioned`",
              "`id`",
              "`created`"
            ],
            "using": "plasma"
          },
          {
            "bucket": "bucket_2",
            "bucketUUID": "f0dbbb922bc1f6a126681a2924933721",
            "defnId": 5987262406230322210,
            "desc": [
              false,
              false,
              false,
              false
            ],
            "exprType": "N1QL",
            "name": "rule_index",
            "nodes": [
              "c2c5093159fc59X56293b27f207e9f44e"
            ],
            "partitionScheme": "SINGLE",
            "secExprs": [
              "(`event`.`id`)",
              "(`context`.`name`)",
              "`_type`",
              "`_versioned`"
            ],
            "using": "plasma",
            "where": "xxxxxxxxx"
          },
          {
            "bucket": "bucket_2",
            "bucketUUID": "f0dbbb922bc1f6a126681a2924933721",
            "defnId": 11766927272443417451,
            "desc": [
              false
            ],
            "exprType": "N1QL",
            "name": "ss_index",
            "nodes": [
              "c2c5093159fc59X56293b27f207e9f44e"
            ],
            "partitionScheme": "SINGLE",
            "secExprs": [
              "`sourceSystemTradeId`"
            ],
            "using": "plasma",
            "where": "xxxxxxx"
          },
          {
            "bucket": "bucket_2",
            "bucketUUID": "f0dbbb922bc1f6a126681a2924933721",
            "defnId": 13029859502669044847,
            "desc": [
              false,
              false
            ],
            "exprType": "N1QL",
            "name": "date_index",
            "nodes": [
              "c2c5093159fc59X56293b27f207e9f44e"
            ],
            "partitionScheme": "SINGLE",
            "secExprs": [
              "`balanceDate`",
              "`lastValidDate`"
            ],
            "using": "plasma",
            "where": "xxxxxxxxxx"
          },
          {
            "bucket": "bucket_2",
            "bucketUUID": "f0dbbb922bc1f6a126681a2924933721",
            "defnId": 16450926905861193996,
            "desc": [
              false,
              false
            ],
            "exprType": "N1QL",
            "name": "archive_index",
            "nodes": [
              "c2c5093159fc59X56293b27f207e9f44e"
            ],
            "partitionScheme": "SINGLE",
            "secExprs": [
              "`_versioned`",
              "`_archived`"
            ],
            "using": "plasma",
            "where": "xxxxxxxxxxxx"
          },
          {
            "bucket": "bucket_2",
            "bucketUUID": "f0dbbb922bc1f6a126681a2924933721",
            "defnId": 8510788496546168299,
            "desc": [
              false,
              false,
              false
            ],
            "exprType": "N1QL",
            "name": "search_index",
            "nodes": [
              "c2c5093159fc59X56293b27f207e9f44e"
            ],
            "partitionScheme": "SINGLE",
            "secExprs": [
              "`_type`",
              "`_versioned`",
              "`id`"
            ],
            "using": "plasma"
          }
        ],
        "indexerId": "c2c5093159fc59X56293b27f207e9f44e",
        "nodeUUID": "c2c5093159fc59X56293b27f207e9f44e",
        "storageMode": "plasma",
        "topologies": [
          {
            "bucket": "bucket_2",
            "definitions": [
              {
                "bucket": "bucket_2",
                "defnId": 8032106326720113472,
                "instances": [
                  {
                    "instId": 7658393146625706509,
                    "partitions": [
                      {
                        "keyPartition": {},
                        "singlePartition": {
                          "slices": [
                            {
                              "indexerId": "c2c5093159fc59X56293b27f207e9f44e"
                            }
                          ]
                        }
                      }
                    ],
                    "state": 4,
                    "steamId": 1,
                    "storageMode": "plasma"
                  }
                ],
                "name": "#primary"
              },
              {
                "bucket": "bucket_2",
                "defnId": 16450926905861193996,
                "instances": [
                  {
                    "instId": 1102663903838181114,
                    "partitions": [
                      {
                        "keyPartition": {},
                        "singlePartition": {
                          "slices": [
                            {
                              "indexerId": "c2c5093159fc59X56293b27f207e9f44e"
                            }
                          ]
                        }
                      }
                    ],
                    "state": 4,
                    "steamId": 1,
                    "storageMode": "plasma"
                  }
                ],
                "name": "archive_index"
              },
              {
                "bucket": "bucket_2",
                "defnId": 10824946879106943769,
                "instances": [
                  {
                    "instId": 7538599140524132818,
                    "partitions": [
                      {
                        "keyPartition": {},
                        "singlePartition": {
                          "slices": [
                            {
                              "indexerId": "c2c5093159fc59X56293b27f207e9f44e"
                            }
                          ]
                        }
                      }
                    ],
                    "state": 4,
                    "steamId": 1,
                    "storageMode": "plasma"
                  }
                ],
                "name": "c_date_index"
              },
              {
                "bucket": "bucket_2",
                "defnId": 11766927272443417451,
                "instances": [
                  {
                    "instId": 12426977418396683569,
                    "partitions": [
                      {
                        "keyPartition": {},
                        "singlePartition": {
                          "slices": [
                            {
                              "indexerId": "c2c5093159fc59X56293b27f207e9f44e"
                            }
                          ]
                        }
                      }
                    ],
                    "state": 4,
                    "steamId": 1,
                    "storageMode": "plasma"
                  }
                ],
                "name": "ss_index"
              },
              {
                "bucket": "bucket_2",
                "defnId": 8510788496546168299,
                "instances": [
                  {
                    "instId": 6113395698417029322,
                    "partitions": [
                      {
                        "keyPartition": {},
                        "singlePartition": {
                          "slices": [
                            {
                              "indexerId": "c2c5093159fc59X56293b27f207e9f44e"
                            }
                          ]
                        }
                      }
                    ],
                    "state": 4,
                    "steamId": 1,
                    "storageMode": "plasma"
                  }
                ],
                "name": "search_index"
              },
              {
                "bucket": "bucket_2",
                "defnId": 13029859502669044847,
                "instances": [
                  {
                    "instId": 13678697307690311920,
                    "partitions": [
                      {
                        "keyPartition": {},
                        "singlePartition": {
                          "slices": [
                            {
                              "indexerId": "c2c5093159fc59X56293b27f207e9f44e"
                            }
                          ]
                        }
                      }
                    ],
                    "state": 4,
                    "steamId": 1,
                    "storageMode": "plasma"
                  }
                ],
                "name": "date_index"
              },
              {
                "bucket": "bucket_2",
                "defnId": 14844685067983095254,
                "instances": [
                  {
                    "instId": 16897468386531290402,
                    "partitions": [
                      {
                        "keyPartition": {},
                        "singlePartition": {
                          "slices": [
                            {
                              "indexerId": "c2c5093159fc59X56293b27f207e9f44e"
                            }
                          ]
                        }
                      }
                    ],
                    "state": 4,
                    "steamId": 1,
                    "storageMode": "plasma"
                  }
                ],
                "name": "audit_index"
              },
              {
                "bucket": "bucket_2",
                "defnId": 5987262406230322210,
                "instances": [
                  {
                    "instId": 6041135326713349948,
                    "partitions": [
                      {
                        "keyPartition": {},
                        "singlePartition": {
                          "slices": [
                            {
                              "indexerId": "c2c5093159fc59X56293b27f207e9f44e"
                            }
                          ]
                        }
                      }
                    ],
                    "state": 4,
                    "steamId": 1,
                    "storageMode": "plasma"
                  }
                ],
                "name": "rule_index"
              },
              {
                "bucket": "bucket_2",
                "defnId": 7442352253566377791,
                "instances": [
                  {
                    "instId": 9131996837407901675,
                    "partitions": [
                      {
                        "keyPartition": {},
                        "singlePartition": {
                          "slices": [
                            {
                              "indexerId": "c2c5093159fc59X56293b27f207e9f44e"
                            }
                          ]
                        }
                      }
                    ],
                    "state": 4,
                    "steamId": 1,
                    "storageMode": "plasma"
                  }
                ],
                "name": "context_index"
              }
            ],
            "version": 44
          }
        ]
      },
      {
        "definitions": [
          {
            "bucket": "bucket_2",
            "bucketUUID": "f0dbbb922bc1f6a126681a2924933721",
            "defnId": 6177948648146310309,
            "desc": [
              false,
              false
            ],
            "exprType": "N1QL",
            "name": "archive_index",
            "nodes": [
              "e8d85c10940cd08c917da66ff480c976"
            ],
            "partitionScheme": "SINGLE",
            "secExprs": [
              "`_versioned`",
              "`_archived`"
            ],
            "using": "plasma",
            "where": "xxxxxxxxxxxxxx"
          },
          {
            "bucket": "bucket_2",
            "bucketUUID": "f0dbbb922bc1f6a126681a2924933721",
            "defnId": 7560228076897210073,
            "desc": [
              false
            ],
            "exprType": "N1QL",
            "name": "c_date_index",
            "nodes": [
              "e8d85c10940cd08c917da66ff480c976"
            ],
            "partitionScheme": "SINGLE",
            "secExprs": [
              "`clearingDate`"
            ],
            "using": "plasma",
            "where": "xxxxxxxxxxx"
          },
          {
            "bucket": "bucket_2",
            "bucketUUID": "f0dbbb922bc1f6a126681a2924933721",
            "defnId": 856993922708612272,
            "desc": [
              false,
              false,
              false,
              false
            ],
            "exprType": "N1QL",
            "name": "context_index",
            "nodes": [
              "e8d85c10940cd08c917da66ff480c976"
            ],
            "partitionScheme": "SINGLE",
            "secExprs": [
              "(`resultEntity`.`id`)",
              "(`context`.`name`)",
              "`_type`",
              "`_versioned`"
            ],
            "using": "plasma",
            "where": "xxxxxxxxxxxxxxxxxx"
          },
          {
            "bucket": "bucket_2",
            "bucketUUID": "f0dbbb922bc1f6a126681a2924933721",
            "defnId": 11667660212473064805,
            "desc": [
              false,
              false,
              false
            ],
            "exprType": "N1QL",
            "name": "search_index",
            "nodes": [
              "e8d85c10940cd08c917da66ff480c976"
            ],
            "partitionScheme": "SINGLE",
            "secExprs": [
              "`_type`",
              "`_versioned`",
              "`id`"
            ],
            "using": "plasma"
          },
          {
            "bucket": "bucket_2",
            "bucketUUID": "f0dbbb922bc1f6a126681a2924933721",
            "defnId": 3855660843423176310,
            "desc": [
              false,
              false,
              false,
              false
            ],
            "exprType": "N1QL",
            "name": "audit_index",
            "nodes": [
              "e8d85c10940cd08c917da66ff480c976"
            ],
            "partitionScheme": "SINGLE",
            "secExprs": [
              "`_type`",
              "`_versioned`",
              "`id`",
              "`created`"
            ],
            "using": "plasma"
          }
        ],
        "indexerId": "e8d85c10940cd08c917da66ff480c976",
        "nodeUUID": "e8d85c10940cd08c917da66ff480c976",
        "storageMode": "plasma",
        "topologies": [
          {
            "bucket": "bucket_2",
            "definitions": [
              {
                "bucket": "bucket_2",
                "defnId": 6177948648146310309,
                "instances": [
                  {
                    "instId": 1909686809119982976,
                    "partitions": [
                      {
                        "keyPartition": {},
                        "singlePartition": {
                          "slices": [
                            {
                              "indexerId": "e8d85c10940cd08c917da66ff480c976"
                            }
                          ]
                        }
                      }
                    ],
                    "state": 4,
                    "steamId": 1,
                    "storageMode": "plasma"
                  }
                ],
                "name": "archive_index"
              },
              {
                "bucket": "bucket_2",
                "defnId": 7560228076897210073,
                "instances": [
                  {
                    "instId": 8193215093999208328,
                    "partitions": [
                      {
                        "keyPartition": {},
                        "singlePartition": {
                          "slices": [
                            {
                              "indexerId": "e8d85c10940cd08c917da66ff480c976"
                            }
                          ]
                        }
                      }
                    ],
                    "state": 4,
                    "steamId": 1,
                    "storageMode": "plasma"
                  }
                ],
                "name": "c_date_index"
              },
              {
                "bucket": "bucket_2",
                "defnId": 11667660212473064805,
                "instances": [
                  {
                    "instId": 4419719151511314113,
                    "partitions": [
                      {
                        "keyPartition": {},
                        "singlePartition": {
                          "slices": [
                            {
                              "indexerId": "e8d85c10940cd08c917da66ff480c976"
                            }
                          ]
                        }
                      }
                    ],
                    "state": 4,
                    "steamId": 1,
                    "storageMode": "plasma"
                  }
                ],
                "name": "search_index"
              },
              {
                "bucket": "bucket_2",
                "defnId": 3855660843423176310,
                "instances": [
                  {
                    "instId": 14543496332732822243,
                    "partitions": [
                      {
                        "keyPartition": {},
                        "singlePartition": {
                          "slices": [
                            {
                              "indexerId": "e8d85c10940cd08c917da66ff480c976"
                            }
                          ]
                        }
                      }
                    ],
                    "state": 4,
                    "steamId": 1,
                    "storageMode": "plasma"
                  }
                ],
                "name": "audit_index"
              },
              {
                "bucket": "bucket_2",
                "defnId": 856993922708612272,
                "instances": [
                  {
                    "instId": 12128833875134126828,
                    "partitions": [
                      {
                        "keyPartition": {},
                        "singlePartition": {
                          "slices": [
                            {
                              "indexerId": "e8d85c10940cd08c917da66ff480c976"
                            }
                          ]
                        }
                      }
                    ],
                    "state": 4,
                    "steamId": 1,
                    "storageMode": "plasma"
                  }
                ],
                "name": "context_index"
              }
            ],
            "version": 24
          }
        ]
      }
    ]
  }
}

amit.kulkarni · April 29, 2019, 9:01am

@zizou

Looking at the gsi.json file, I am not sure how one can end up in this situation. Same combination of (bucket-name, index-name) is not allowed. Only exception being index replica, but in that case, replica id will be explicitly mentioned for replica index AND both main index and replica index will have same defnId.

So, there are duplicated indexes in the backup index metadata.

You can modify the index metadata to fix duplicate indexes in gsi.json, so that index creation can succeed. The simplest way is to rename the duplicate indexes. The gsi.json file is organised in specific way. The “metadata” key is an array of documents, elements in each document belongs to one indexer node. Each of these documents have two keys “definitions” and “topologies”. Every index has its entry in “definitions” and “topologies”.

You will have to change the index name from both places, “definitions” and “topologies”, for only one occurrence of that index. For example, replace context_index with context_index_1 (or some other similar name) only for node “c2c5093159fc59X56293b27f207e9f44e” (or only for defnId 7442352253566377791). You will have to keep the name “context_index” same as before for the other node “e8d85c10940cd08c917da66ff480c976” for defnId 856993922708612272.

This should lead to successful restore.

Note that this will create equivalent indexes in your cluster (context_index and context_index_1 will be equivalent indexes) which will be useful for providing High Availability.

Topic		Replies	Views
Error in restoring backup using cbbackupmgr in multi node setup Couchbase Server	3	2045	March 28, 2018
Restore indexes only with cbbackupmgr Couchbase Server index , backup	4	1859	April 7, 2022
Error : restoreIndexMetadata when restoring with cbbackupmgr Couchbase Server backup	4	1959	June 4, 2020
Create index or Alter replica cannot proceed Couchbase Server n1ql , index	1	1661	November 14, 2020
Issue with indexes when restoring cluster Couchbase Server index , backup	5	2215	August 29, 2019

Cbbackupmgr restoring same gsi index multiple time which is causing an error

Related topics