Creating index on specific document types

Full Text Search
1

Hello,

I have two questions regarding the creation or filtering the FTS results.

  1. I want to create an index in which just documents with a specific field are indexed. For example documents in which type=“something”. Is this possible? If not, is possible to filter the results when I do the search?

  2. I have multiple documents for different languges in which a product name is the same accross multiple languages. Is it possible to group the results by a specific field in the document? All of these documents have an unique field, but can have different product name, so they could be groupped by that filed. Can this be done at the index or search moment?

Thank you,
F

2

I want to create an index in which just documents with a specific field are indexed. For example documents in which type=“something”. Is this possible? If not, is possible to filter the results when I do the search?

Yes, this is supported. You would need to disable the “default” type mapping, and add a type mapping that reflects the “type” of the documents you want to index.

For example, assuming you want to index all JSON documents with the field “type”:“something” in them, you will need to set the “type field” (within the doc_config) to “type” (name of the field) and create a type mapping with the name “something” (field value). You can either leave the “dynamic” setting on - in which case all the content within the documents that match the type expression is indexed or select “only index specified fields” to index select fields from within the filtered documents.

I have multiple documents for different languges in which a product name is the same accross multiple languages. Is it possible to group the results by a specific field in the document? All of these documents have an unique field, but can have different product name, so they could be groupped by that filed. Can this be done at the index or search moment?

There is a sort capability with Couchbase’s search service. Sorting based on the “field” should order results by the field value. Here’s some documentation on it …
https://docs.couchbase.com/server/6.6/fts/fts-sorting.html

You can also leverage N1QL’s GROUP-BY capability by running full text search requests from N1QL (supported 6.5 onwards). Here’s documentation on it …

3

Thanks for the fast answer.

I am not sure I understand this corectly"

So, in this case:

image
image868×213 12.4 KB

if the document is something like:

{
  "product_name": "product 1",
  "country_code": "us"
}

and another document

{
  "product_name": "product 1",
  "country_code": "de"
}

the index will contain just documents with country code: “US”?

4

Hmm, not quite.
Let me attach a screenshot for you, that will index everything from documents whose “country_code” is “us” …

Screen Shot 2021-01-25 at 2.44.09 PM
Screen Shot 2021-01-25 at 2.44.09 PM2370×468 40.7 KB

Here’s how the index definition would look …

{
 "name": "default",
 "type": "fulltext-index",
 "params": {
  "mapping": {
   "types": {
    "us": {
     "enabled": true,
     "dynamic": true
    }
   },
   "default_mapping": {
    "enabled": false,
    "dynamic": true
   },
   "default_type": "_default",
   "default_analyzer": "standard",
   "default_datetime_parser": "dateTimeOptional",
   "default_field": "_all",
   "store_dynamic": false,
   "index_dynamic": true,
   "docvalues_dynamic": false
  },
  "store": {
   "indexType": "scorch",
   "kvStoreName": ""
  },
  "doc_config": {
   "mode": "type_field",
   "type_field": "country_code",
   "docid_prefix_delim": "",
   "docid_regexp": ""
  }
 },
 "sourceType": "couchbase",
 "sourceName": "default",
 "sourceUUID": "",
 "sourceParams": {},
 "planParams": {
  "maxPartitionsPerPIndex": 171,
  "numReplicas": 0,
  "indexPartitions": 6
 },
 "uuid": ""
}
5

Oh, I see… hmm, in this case, is there any option in which I can have two fields?

like country_code = us and product_type = 2

6

So the type mapping is a single string condition expression to index documents that satisfy said condition.

Not sure I follow your question entirely, but here’s an example search: “product_name”:“product_2” for the definition I shared above.

Again this search will only bring you back of documents with country_code:us. You can optionally choose to just index specific fields - what I mean is if product_name is the only field you’re ever searching for, you could just index that field within the “us” type mapping.

7

Sorry for not being clear.

What I need is to be able to index docuents if two conditions are met: like country_code = us and product_type = 1

and then based on this index to be able to search on another field, like product_name = blue

Does this make sense?

Later edit:

I tried to filter out the result doing a query, and somehow it is not working. Not sure what I am doing wrong.

In the Search section of the Couchbase UI if I search for the word blue, I get the result in a few miliseconds.

Now, if I copy the Json Query request and use it in a standard query, the query never return and remains in “Executing Statement”.

The query I am using is:

SELECT prod.*
FROM `app-live` AS prod
WHERE SEARCH(prod, {
  "explain": true,
  "fields": [
    "*"
  ],
  "highlight": {},
  "query": {
    "query": "dark"
  }
})

The JSON Query Request I am getting from the Search section of the UI is:

{
  "explain": true,
  "fields": [
    "*"
  ],
  "highlight": {},
  "query": {
    "query": "dark"
  }
}

Why does my query doesn’t return anything but the FTS is working perfectly?

If I click on advice, I get the message: “Existing indexes are sufficient.”

Thanks,
F

8

Ah I see. The search service will allow for you to set type mappings over a single field only. So you will not be able to set up 2 type mappings from different fields.

Your N1QL query with the search request will need an additional expression because you have a custom type mapping within your search index definition. Check out Example 4 here …

If you can share your index definition, I will be able to recommend the query expectation.
The “query advisor” works for secondary indexes only at the moment and not full text search indexes.

9

here is the index definition:

{
  "type": "fulltext-index",
  "name": "product_name",
  "uuid": "1e5a29f48f7f891e",
  "sourceType": "couchbase",
  "sourceName": "app-live",
  "sourceUUID": "54a8a8a8c2c77f7cd8a8b789fbdb55be",
  "planParams": {
    "maxPartitionsPerPIndex": 171,
    "indexPartitions": 6
  },
  "params": {
    "doc_config": {
      "docid_prefix_delim": "",
      "docid_regexp": "",
      "mode": "type_field",
      "type_field": "sub_type"
    },
    "mapping": {
      "analysis": {},
      "default_analyzer": "standard",
      "default_datetime_parser": "dateTimeOptional",
      "default_field": "_all",
      "default_mapping": {
        "dynamic": true,
        "enabled": false
      },
      "default_type": "_default",
      "docvalues_dynamic": true,
      "index_dynamic": true,
      "store_dynamic": false,
      "type_field": "_type",
      "types": {
        "product: {
          "dynamic": false,
          "enabled": true,
          "properties": {
            "product_a_store": {
              "dynamic": false,
              "enabled": true,
              "properties": {
                "product_name": {
                  "dynamic": false,
                  "enabled": true,
                  "fields": [
                    {
                      "docvalues": true,
                      "include_in_all": true,
                      "include_term_vectors": true,
                      "index": true,
                      "name": "product_name",
                      "type": "text"
                    }
                  ]
                },
                "product_provider_name": {
                  "dynamic": false,
                  "enabled": true,
                  "fields": [
                    {
                      "docvalues": true,
                      "include_in_all": true,
                      "include_term_vectors": true,
                      "index": true,
                      "name": "product_provider_name",
                      "type": "text"
                    }
                  ]
                },
                "product_seller_name": {
                  "dynamic": false,
                  "enabled": true,
                  "fields": [
                    {
                      "docvalues": true,
                      "include_in_all": true,
                      "include_term_vectors": true,
                      "index": true,
                      "name": "product_seller_name",
                      "type": "text"
                    }
                  ]
                },
                "bundle_id": {
                  "dynamic": false,
                  "enabled": true,
                  "fields": [
                    {
                      "docvalues": true,
                      "include_in_all": true,
                      "include_term_vectors": true,
                      "index": true,
                      "name": "bundle_id",
                      "type": "text"
                    }
                  ]
                }
              }
            },
            "product_g": {
              "dynamic": false,
              "enabled": true,
              "properties": {
                "product_name": {
                  "dynamic": false,
                  "enabled": true,
                  "fields": [
                    {
                      "docvalues": true,
                      "include_in_all": true,
                      "include_term_vectors": true,
                      "index": true,
                      "name": "product_name",
                      "type": "text"
                    }
                  ]
                },
                "product_provider_name": {
                  "dynamic": false,
                  "enabled": true,
                  "fields": [
                    {
                      "docvalues": true,
                      "include_in_all": true,
                      "include_term_vectors": true,
                      "index": true,
                      "name": "product_provider_name",
                      "type": "text"
                    }
                  ]
                },
                "product_seller_name": {
                  "dynamic": false,
                  "enabled": true,
                  "fields": [
                    {
                      "docvalues": true,
                      "include_in_all": true,
                      "include_term_vectors": true,
                      "index": true,
                      "name": "product_seller_name",
                      "type": "text"
                    }
                  ]
                },
                "bundle_id": {
                  "dynamic": false,
                  "enabled": true,
                  "fields": [
                    {
                      "docvalues": true,
                      "include_in_all": true,
                      "include_term_vectors": true,
                      "index": true,
                      "name": "bundle_id",
                      "type": "text"
                    }
                  ]
                }
              }
            },
            "country_code": {
              "dynamic": false,
              "enabled": true,
              "fields": [
                {
                  "docvalues": true,
                  "include_in_all": true,
                  "include_term_vectors": true,
                  "index": true,
                  "name": "country_code",
                  "type": "text"
                }
              ]
            }
          }
        }
      }
    },
    "store": {
      "indexType": "scorch"
    }
  },
  "sourceParams": {}
}
10

Here’s a query that should work …

SELECT meta().id
FROM `app-live` AS prod
WHERE sub_type="product"
AND SEARCH(prod, {
  "explain": true,
  "fields": [
    "*"
  ],
  "highlight": {},
  "query": {
    "query": "dark"
  }
})

Note that the type expression sub_type="product" that N1QL queries need for type mapped FTS indexes.

11

ok, it make sense. thank you.

Still I get an empty result even thought that the FTS return 241395 result.
What could be the problem?

12

Could you get me the output you see when you hit “Explain” on your N1QL query?

13

this one?

image
image1985×851 47.7 KB

{
    "#operator": "Sequence",
    "~children": [
        {
            "#operator": "ValueScan",
            "values": "[]"
        },
        {
            "#operator": "Fetch",
            "as": "prod",
            "keyspace": "app-live",
            "namespace": "default"
        },
        {
            "#operator": "Parallel",
            "~child": {
                "#operator": "Sequence",
                "~children": [
                    {
                        "#operator": "Filter",
                        "condition": "((\"sub_type\" = \"product\") and search(`prod`, {\"explain\": true, \"fields\": [\"*\"], \"highlight\": {}, \"query\": {\"query\": \"dark\"}}))"
                    },
                    {
                        "#operator": "InitialProject",
                        "result_terms": [
                            {
                                "expr": "(meta(`prod`).`id`)"
                            }
                        ]
                    },
                    {
                        "#operator": "FinalProject"
                    }
                ]
            }
        }
    ]
}
14

Ok looks like it’s not hitting FTS at all.
Can you try this query …

SELECT meta().id
FROM `app-live` AS a
WHERE "sub_type"="product"
AND SEARCH(a, "dark");

What version of couchbase are you on? You’ll need 6.5+ for this to work.

15

I am getting the same results (empty)

16

What’s your couchbase server version?

17

It is enteprise 6.6.0 build 7909

18

Ah I made a mistake with the type expression syntax - try this …

SELECT meta().id
FROM `app-live` AS a
WHERE sub_type="product"
AND SEARCH(a, "dark");

Note the updated condition expression …
sub_type = "product"

19

yes, indeed it is working now.
why does it matter if it is with " or without ?

Thank you so much for your fast answers, I really appreciate!

20

sub_type="product" looks for “product” within a field name “sub_type” --> This is the type expression syntax.
"sub_type"="product" just compares 2 constants.