Couchbase Enterprise Edition 6.0.3 : FTS query not returning the specified fields

Full Text Search
1

Hi,

I’m using Couchbase edition 6.0.3. Currently implementing a search functionality using .NET Api.
I have created a FTS index and it returns the expected document but it returns only the Id but not the field mentioned in the query request (refer below query JSON).

{“ctl”:{“timeout”:75000},“size”:1,“fields”:[“ContentType”],“query”:{“conjuncts”:[{“term”:“brexit”,“prefix_length”:0,“fuzziness”:0,“field”:“Title”},{“min”:0.0,“inclusive_min”:true,“max”:2.0,“inclusive_max”:false,“field”:“ContentType”},{“bool”:true,“field”:“IsLatestVersion”}]}}`

Note: The Field mentioned in the query request(ContentType) is an indexed field too.

  1. Am I missing something? Kindly help.

  2. As I’m not using couchbase 6.5, I’m not able to use SEARCH in N1QL, Is there any other way to use the FTS index in N1QL in couchbase 6.0.3?

2

By default, FTS only returns the document IDs (or keys) as results.
We do however provide a way for the user to extract document fields with an FTS request.
This is via the “fields” setting in the search request - I see that you’ve already included this section in your query.

For this to work - you will need the index definition to support it.
Here’s how to do this …

  • Within the index index definition’s type mapping(s), while you are choosing a field to index - you will need to enable the “store” option for the field. This essentially stores the entire content of the field as is.
  • Here’s how the child field mapping would need to look within your index definition -
"ContentType": {
  "dynamic": false,
  "enabled": true,
  "fields": [
    {
      "analyzer": "en",
      "index": true,
      "name": "city",
      "store": true,
      "type": "text"
    }
  ]
}

Access to FTS indexes from N1QL is only supported from couchbase-server 6.5 onwards.

1 Like
3

@Kannan_Muruganmony as an alternative approach if you want to keep your index smaller, you can also grab the document ID that gets returned and use the .NET SDK API (either full doc or subdoc KV) to fetch the parts you need. You see it is a tradeoff between lazily fetching the parts with a second roundtrip. vs keeping it in your index and making it bigger.

1 Like
4

@daschl Thanks a ton mate. This idea was awesome and saved me a lot of time, highly useful. My index looks less heavy now.

5

Hi,

Thanks for the help earlier. I was able to set up the FTS in QA environment and use .net sdk to get the desired results and finally we are looking to replace ES.

This FTS index will power our CMS dashboard(which is extremely important).
I have two questions stated below:

1)I’m currently looking to deploy to production. So, I wanted to ensure that the FTS index will be always active even when a server goes down(else the CMS dashboard will show no content - which shouldn’t happen anytime).

We have three servers. Could you please suggest which is the best way to handle this.

Do you suggest (a or b) :

a) Creating three same FTS indexes and mapping them to the single Alias which power the dashboard?
example: FTS_Index_1,FTS_Index_2,FTS_Index_3 mapped to Alias ‘Dashboard_Index’

b) Create a single FTS index with index replicas as 2?
example: FTS_Index_1(index replicas: 2), mapped to Alias ‘Dashboard_Index’

If you have any other suggestions, please let me know.

2)I created a FTS index(with 0 index replica), it got built and was displaying indexing progress as 100% but a new document was created and the index the index progress was back again to 60% and then moved to 100%. Why is it happening? I’m losing some documents in the resultset when the second indexing progress happens.

Kindly help. Thank you @abhinav @daschl .

6

Here’re some details, let me know if these answers your questions -

  • You would need at least one KV node in the cluster to access the cluster. Your FTS index will remain active through out the period when some KV nodes are unavailable. Should you experience any data loss with multiple KV nodes and you decide to rebalance the cluster with no way around the data loss - then the data in the search nodes will be rolled back to become consistent with KV.
  • An FTS alias is simply a way to query multiple FTS indexes at the same time.
  • An FTS index can support replicas, but you’ll need enough nodes in the cluster hosting the FTS service to support this. Now if you have a 3 node cluster containing an FTS index with 1 replica and 6 partitions. Each node would host 2 active and 2 replica partitions. If one of the nodes become unreachable - your queries would see partial results. However - the moment you failover the unreachable node, the replicas of the active partitions that went down will be promoted to active status - and queries will immediately start seeing full results again. Upon a rebalance after - partitions will be moved around and re-built in the now 2 node cluster - you’d have 3 active and 3 replica partitions on each node.

Could you answer these questions for me?

  • You mention you have 3 servers, are these hosting KV or FTS only or both the services?
  • As for your 2nd question - whenever you add new documents to KV, the FTS index will index them automatically - reason why you might see indexing progress dip briefly and come back up once all the new documents are received. Now when you say you’re losing some documents in the result set on the second indexing progress - is your query changing or if your KV data is going through updates - this could happen.
1 Like
7

@abhinav Thanks for your reply.

Please find below the answers:

  • Both the services are active. I hope by KV, FTS you mean data and search services? I have attached the active services below.
    services

  • Yes, the search query might change (based on the input search parameters provided by the user). Sorry, I’m new to Couchbase terms. I believe by KV data, you mean the documents? If so, yes, the document data will be continuously updated by users via UI.

However, what should I do to make the FTS index returns a consistent dataset? It might be a major setback if the data returned is not consistent.

I have attached the dashboard screenshot below for reference.

dashboard
dashboard2008×257 21.7 KB

8

Yes by KV I meant the data service and by FTS i mean the search service. Apologies on not using the standard nomenclature :slight_smile:

The Full text search queries return results based on the content indexed and the content indexed will reflect that in the Data nodes. So if the content in the data nodes is continuously updated, the content in the FTS index will also change. There is no way around this.

If you wish your indexed content to remain the same, then I’d recommend pointing it to a couchbase bucket that doesn’t see continuous updates. This couchbase bucket could be a snapshot of the data from another bucket that receives continuous updates.

9

@abhinav Sorry, I think there is a misunderstanding.

I explained my scenario below with an example :

  1. Initial FTS index is created ( Indexing Progress - 100%, doc count - 1000 (Doc_1 to Doc_1000) )

  2. I make a request for the last 5 docs. In the result set, I get Doc_1000 to Doc_995.

  3. Now the user creates a new document (Doc_1001)

  4. The FTS indexing progress moves back to 60% and the doc count goes to 600.

  5. Now, I make a request for the last 5 docs. The result set returned is (Doc_600 to Doc_595), whereas I expect to get Doc_1001 to Doc_996.

Hope I made it clear now :slight_smile: please let me know if it is not.

Note: The second reindexing doesn’t happen all the time but on most times and during that I get the below message in the logs(please refer below)

Service ‘fts’ exited with status 2. Restarting. Messages:
The Go Programming Language(0xc04edb4690, 0xc073a5d000, 0x1000, 0x1000, 0x1460240, 0x72, 0x0)
:1 +0x61
net/http.(*connReader).Read(0xc04edb46f0, 0xc073a5d000, 0x1000, 0x1000, 0x401101, 0xc05eb6bd08, 0x402471)
C:/Users/Administrator/cbdepscache/exploded/amd64/go-1.9.6/go/src/net/http/server.go:753 +0x10c
bufio.(*Reader).fill(0xc07e4da5a0)
C:/Users/Administrator/cbdepscache/exploded/amd64/go-1.9.6/go/src/bufio/bufio.go:97 +0x121
bufio.(*Reader).Peek(0xc07e4da5a0, 0x4, 0xbcc8105769, 0x1460240, 0x0, 0x0, 0x1460240)

10

The FTS process restarting is NOT a good sign.

Could you share the following -

  • what your request looks like…?
  • the FTS index definition.

I will need to take a look into the logs. If you could go to the logs tab on the UI - you will be able to schedule a logs collection, and once the .zip is ready - feel free to share them here.

11

I have scheduled for collecting logs. I will attach the zip file once it is ready.

PFA the three files:

  • Index_Preview.txt : Index preview Json
  • Request_Json.txt : Search Query Json
  • Alert.png : I get this alert some times, I ignored it, but just thought let you know.

Index_Info.zip (37.1 KB)

Please let me know if you require any more info. Thank you very much mate.
Please let me know if you see any more way to optimize the FTS index/ Search query.

12

The index definition and query could use some optimizing but look generally fine.
I’d look into the IP address changing warnings to see why it’s happening and this could possibly cause erratic behavior.

Without the logs I will not be able to tell you why your FTS service is restarting.

13

Thanks for checking the query and definition. The log collection is running. Does it take a long time? It’s been an hour and in the activity section it still says ‘Collecting logs from 1 node 0.0%’

14

No, it should be rather quick, especially that you have just a 1000 items in your couchbase bucket.
You can also run one from the command line. Look up …

/opt/couchbase/bin/cbcollect_info --help
15

Nope, 1000 items were an example. We have 9 buckets and the items displayed below :slight_smile:

bucket_info
bucket_info322×763 13.6 KB

16

Ok seeing that you have a reasonable big deployment of couchbase - I would strongly recommend you to get in touch with couchbase support for your production needs.

With support, we will be able to provide you a better platform for uploading logs and faster turn around times for any questions and concerns. We will also be able to better advise you on sizing, index definition guidelines and possible query optimizations.

17

I checked with our ops team and it seems like we got Couchbase support only for the live boxes.
But these issues are happening in QA boxes. So I could not check with the Couchbase support. However, I’m still stuck with “Service ‘fts’ exited with status 2” error. Shall I provide the logs(zip file) here, so you could help, please? Kindly help. @abhinav