Is it possible to de-duplicate documents, comparing fields in an array and identify same data in more than a document

Couchbase Server
1

I’m looking to de-duplicate documents, with same data in “fields” array. Example document is given below.

{
“notes”: {
“_class”: “abclass”,
“createdBy”: “xyz”,
“createdDate”: 1521133343372,
“fields”: {
“accountNumber”: “accxyz”,
“comments”: “comments”,
“contactReason”: “reason”,
“ntId”: “aaa”,
“perm”: false,
“source”: “iii”
},
“id”: “ooooo”,
“typeRef”: “uuuuu”
}

We are using 4.6.3 Any quick suggestion would be appreciated, if couchbase is capable to do this at server level or should this be dealt with piece of code to compare.

2

Is object not array. If it is ARRAY you want eliminate duplicates do ARRAY_DISTINCT(fields)

Based on Above document the following query can give array of documents that have same fields.

SELECT  fields, ag  FROM default 
GROUP BY fields 
LETTING ag = ARRAY_AGG(META().id)
 HAVING ARRAY_LENGHTH(av) > 1;
3

Thank you. I was trying to fetch the document Ids with duplicate data in fields array. The above n1ql works on smaller set of data, but not with millions of document. Alternatively, tried to retrieve the docIds using view, without any luck. Ref - https://blog.couchbase.com/want-get-rid-documents-duplicate-content/
That resulted in ‘reduction too large’ errors. Any clue to do this other than from client side {code} to identify the duplicate documents. Final goal is to delete all the duplicates from couchbase.

thank you!

4

Do multiple queries by applying predicate of small ranges.