What is the most effective FTS Index Analyzer to use for product codes

Full Text Search
1

Hi Guys,

  I have a question, What is the best and most effective FTS Index Analyzer to use for search  product codes?

My product codes look like this.: 0017 and 743715000179

Also, how can I boost my search result.

2

Hi @lyndondonz,

If your product codes are text fields, then you may use a “keyword” analyser for indexing this. It preserves the field content intact skipping any text analysis.

ref -https://docs.couchbase.com/server/current/fts/fts-using-analyzers.html#pre-constructed-analyzers

With boosting, way you may control the ranking(score) for the matches from a specific sub query.
Few samples below.
ref - https://docs.couchbase.com/server/6.5/fts/query-string-queries.html#boosting

Boost factor is applicable to most of the query types in FTS.
ref -https://docs.couchbase.com/server/6.5/fts/fts-query-types.html

curl -XPOST -H "Content-Type: application/json" -uUname:Passwd  http://host:port/api/index/FTS/query -d '{
 "query" : {
  "disjuncts": [
     {"match_phrase": "query text1", "field": "name", "boost": 5}, 
     {"match_phrase": "query text2", "field": "name", "boost": 20}
    ]
  }}'

Cheers!

3

Hi @sreeks

 Thanks man, do you have any idea how to use boost with NODEJS SDK ?

Thanks.

4

Hi @sreeks

I already implemented the `boost` on my search query it's already part of the search query string.

I have another question and clarification:

How and what analyzer would I use for my product code.
For example, my full product code is like this: 0017 , so if if my search keyword is this: 017 I can still get a result.
My problem here when I use the Analyzer keyword I really need to input the full product code 0017 just to get the result.

Need help thanks.

5

Hi @lyndondonz,

There are couple of ways to do this.

  1. Cheap and quick solution is try wildcard queries in FTS with keyword analyser. But this may end up with scaling issues if there is a whole lot of product codes in the document set satisfying the given query.
{
"query": {
 "wildcard": "*17",
 "field": "productCode"
}}'

ref -https://docs.couchbase.com/server/current/fts/fts-query-types.html#wildcard-query

  1. If you need to do this at scale, then create a custom analyser with an edge-n-gram token filter, And edge n gram filter of min len = 1, max = 4 which starts from back of the token, would index all the substrings like 7, 17, 017, 0017 for the input field value of 0017.

The minimum length of 1 is assumed here because there could be a search for 7 when the product code is 0007.
But the flip side in all these cases is that - a search for product code 7 will return all documents that has product codes ending in 7.

Sharing a sample mapping definition here with a single field productCode indexed for document type “Product”.

"mapping": {
      "analysis": {
        "analyzers": {
          "custom": {
            "char_filters": [
              "asciifolding"
            ],
            "token_filters": [
              "to_lower",
              "edgengram"
            ],
            "tokenizer": "unicode",
            "type": "custom"
          }
        },
        "token_filters": {
          "edgengram": {
            "back": true,
            "max": 4,
            "min": 1,
            "type": "edge_ngram"
          }
        }
      },
      "default_analyzer": "standard",
      "default_datetime_parser": "dateTimeOptional",
      "default_field": "_all",
      "default_mapping": {
        "dynamic": true,
        "enabled": false
      },
      "default_type": "_default",
      "docvalues_dynamic": true,
      "index_dynamic": true,
      "store_dynamic": false,
      "type_field": "_type",
      "types": {
        "Product": {
          "dynamic": false,
          "enabled": true,
          "properties": {
            "price": {
              "dynamic": false,
              "enabled": true,
              "fields": [
                {
                  "analyzer": "custom",
                  "index": true,
                  "name": "price",
                  "type": "text"
                }
              ]
            }
          }
        }
      }
    },

Cheers!,