Full Text Search (or FTS) is a new feature that is being released as a developer preview in Couchbase 4.5. It is not meant for production, and is not yet supported, but it is a glimpse at things to come.
When offering the ability to search, a simple LIKE '%foo%' with N1QL is not enough. Your users expect a richer search experience: one that is language aware and ranks results by their relevance.
In this blog post, I’m going to show you:
- How to set up a Full Text Search index
- How to execute a Full Text Search query in the Couchbase Console
- How to execute a Full Text Search query with highlighting in ASP.NET
Set up a Full Text Search index
First, you need to make sure that you are using Couchbase 4.5 (beta, at least), that you have access to the Couchbase Console, and that your cluster is running a Full Text service. You can check to see if you have a Full Text service running by clicking “Server Nodes” in Couchbase Console. Services (like Full Text) have to be enabled/disabled when adding a node to the cluster. If you need a refresher, check out my blog post on how to install and setup Couchbase on Windows.
I’m going to setup a full text search index on the travel-sample
In this sample data, there are documents of type “landmark”. Among other things, these documents contain ‘content’ fields. Here’s a snippet from one of the landmark documents. (ID ‘landmark_16079’ if you’re following along):
|
1 2 3 4 5 6 7 |
{ "title": "London/City of London", "name": "20 Fenchurch Street", "content": "The unusual "walkie talkie" profile of this under-construction skyscraper by Rafael Vinoly has seen it grab the headlines.", "type": "landmark" } |
With that structure in mind, let’s create the index. Go to Indexes->Full Text, and click “New Full Text Index”.
I’m going to name the index travel-sample-idx, and put it on the travel-sample bucket. There are a lot of options here, but I’m just going to cover type mappings. Expand “Type Mappings”, and click “Add Type Mapping”. This allows you to restrict the full text search to documents of a certain type (in this case, “landmark”, which is matched to the “type” field). Also check the “only index specified fields” box, otherwise all the fields in the documents will be indexed for search. Hit “ok”.
Next, hover over the ‘landmark’ type mapping, click the + symbol, and click ‘insert child field’. If you want to index search data that’s further down in the hierarchy of your document, then you would need to ‘insert child mapping’ (as many levels as it takes) until you get down to it. In our case, the content field is at the root level, so I don’t need to do that.
For the child field, enter the field name (“content”) in field, leave everything else as-is, and check the “store” checkbox (more on that later), and hit “ok.”
Finally, edit the “default” type mapping and uncheck the “enabled” option. This mapping would enable searching across all documents, but in this example we want to limit to just landmark. Hit “ok”, and “Create Index”.
After you hit “Create Index”, you’ll be shown an overview of the index you just created. It will immediately start indexing documents. If you click the Refresh button, you’ll be shown the count of the documents that have been indexed so far.
It will be 100% when it’s complete.
Execute a Full Text Search query in the Couchbase Console
Before we go into C#/.NET, you can try the full text search right from the console. Go to Indexes->Full Text, and select the full text index that you just created. Enter a search term (if you’re using the travel sample, “inn”, “skyscraper”, or “bed and breakfast” are good terms to try.
When you click search, you’ll be given a page of search results (document ID and a fragment of content) in order of their relevance score. Notice that the search terms are highlighted in the content fragment. Enabling “store” on the child field makes this work. If you don’t enable store, you’ll still get the document results in relevant order, but you won’t get the highlighted content fragment.
Note that the full text search uses Bleve, and allows you to do more complex searches (exact phrases, boolean queries, etc). Try searching "bed and breakfast" or bed -breakfast, for instance, and see what happens.
Full Text Search query with highlighting in ASP.NET
Once you are satisfied with a full text search index that you’ve created, it’s time to take it out of the Couchbase Console and into a real application. I’ve created an example in ASP.NET that demonstrates how to use the Couchbase .NET SDK to perform a full text search. For this example, I’m assuming that you’re comfortable with ASP.NET MVC and with the basics of integrating ASP.NET with Couchbase. If that’s not the case, I recommend that you review the ASP.NET “Hello World” example that I blogged about previously.
I’ve created two actions: Index and Search. The Index action shows a view that allows a user to type in a search string. Here’s the controller and action:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
public class HomeController : Controller { private readonly IBucket _bucket; public HomeController(IBucket bucket) { _bucket = bucket; } public ViewResult Index() { return View(); } } |
And here’s the view in Index.cshtml:
|
1 2 3 4 5 6 7 8 |
<h1>Search</h1> @using (Html.BeginForm("Search", "Home")) { @Html.TextBox("q","") } |
It’s not pretty, but it’s functional. Next, I created a Search action. This action will take in the search query and output the results from Couchbase. The results will look very similar to what we saw in Couchbase Console: document ID, relevance score, and highlighted fragments.
There are three steps to executing a full text search:
- Instantiate some object that implements
ISearchQuery. There are many options; I’m usingMatchQueryfor this example. - Instantiate a SearchParams object to specify search options.
- Use an
IBucketto execute the query+parameters and get results.
In this example, I’ve done all three steps right in the Search action:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
public ActionResult Search(string q) { if (string.IsNullOrEmpty(q)) return View("Index"); var query = new MatchQuery(q); var searchParams = new SearchParams() .Limit(10) .Timeout(TimeSpan.FromMilliseconds(10000)) .Highlighting(HighLightStyle.Html); var results = _bucket.Query("travel-sample-idx", query, searchParams); var searchResultsModel = new SearchResultsModel(q, results); return View(searchResultsModel); } |
Some notes on the SearchParams:
- I’m limiting it to the first 10. There is also a Skip() method that you can use for paging.
- I’m setting a timeout of 10000ms. If I create the index correctly, the results will be returned very quickly, but if I tinker with the index and something goes wrong, I want the search to fail quickly.
- I’m setting the highlight style to HTML. This will put the
tag around the highlighted search terms in the fragment results.
I’ve also created a SearchResultsModel view model. This does the mapping that I need to get the document ID, score, fragments from the search results (as well as the keywords that the user typed in).
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 |
public class SearchResultsModel { public List Results { get; private set; } public string Keywords { get; private set; } public SearchResultsModel(string keywords, ISearchQueryResult results = null) { Keywords = keywords; Results = results == null ? new List() : results.Hits.Select(r => new SearchResultModel(r)).ToList(); } } public class SearchResultModel { public string DocumentId { get; private set; } public double Score { get; private set; } public List Fragments { get; set; } public SearchResultModel(ISearchQueryRow searchQueryRow) { DocumentId = searchQueryRow.Id; Score = searchQueryRow.Score; Fragments = searchQueryRow.Fragments.SelectMany(f => f.Value).ToList(); } } |
Finally, here’s the Search.cshtml view:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 |
@model FullTextSearchExample.Models.SearchResultsModel <h1>Search</h1> @using (Html.BeginForm("Search", "Home")) { @Html.TextBox("q", Model.Keywords) } @if (!Model.Results.Any()) { No results found } else { foreach (var result in Model.Results) { <h2>Document: @result.DocumentId</h2> <h3>Score: @result.Score</h3> foreach (var fragment in result.Fragments) { @Html.Raw(fragment) } } } |
Not a terribly pretty HTML page, but it’s functional. Here’s what it looks like in action:
Conclusion
Remember that this feature is just a developer preview: it’s not supported yet and it’s not recommended for production yet. But full text search is coming to Couchbase. In fact, you can have a say in how the feature is implemented. Check out the Full Text Search RFC on GitHub, and feel free to ask questions or leave feedback.
Still have questions? Need help with anything? Leave a comment, ping me on Twitter, or email me (matthew.groves AT couchbase DOT com).
