What is a Vector

A Vector is an object that represents a real-world item as an array of floating numbers. 

Each item in the real world is represented in Vector format(as an array) and has many dimensions (attributes) associated with the object based on its characteristics. 

For example, if we want to represent Colours in Vector format, we can create an array of attribute values. Example [ “R”,”G”,”B”]

Each colour in an RGB image is represented by three values: the amount of red, green, and blue light present. These values typically range from 0 to 255, indicating the intensity of each colour component.

Pure Red= [ “255”,”0”,”0”]

Where:

    • R represents the intensity of red (in this case, maximum intensity, 255),
    • G represents the intensity of green (in this case, 0, so no green),
    • B represents the intensity of blue (in this case, 0, so no blue).

Similarly, you can represent any colour using this RGB vector format, with values ranging from 0 to 255 for each colour channel.

If we want to find close matches for the colour red, we can find out based on the first attribute value of the colour.

Real world objects can have many other attributes that they have to represent and, therefore, a Vector representing a real world object is represented by a larger array of 512, 1028, 1536 or 2048 attribute values.

What is Vector Search?

Vector search is a method of finding items based on their vector representation. In vector search, each item is represented in multidimensional space where each dimension represents the value of the attribute of the item.

More details can be found at:

Industry-wide Use Cases

Vector search can be used across industries for various use cases, here are a few of them:

    1. Content Generation
    2. Anomaly detection
    3. Hybrid Search
    4. AI powered chatbots.

Vector Search vs. Full Text Search?

Vector search and full-text search are both methods used for searching through collections of data, but they operate in different ways and are suited to different types of data and use cases.

Full text search: is a technique used in information retrieval to search and analyse textual content within documents or databases. Unlike traditional search methods that match exact phrases or keywords, full text search engines analyse the content of documents or records to match search queries based on the meaning and context of the words. 

Comparison area Full text search Vector Search
1 Representation of data Data is represented as documents of text or strings Data is represented as vectors in multidimensional space
2 Matching criteria Exact or fuzzy match Nearest neighbouring match
3 Search Textual search or comparison Contextual search or comparison based on attributes of object.
4 Use case Searching through a document, web page, email content, etc. Searching through audio, video, image, text, etc.

Why Couchbase for Vector Search?

    1. Vector across our products: First in the industry to announce support for all 3 deployments: cloud, on-prem, mobile.
    2. Broad Capabilities: Integrated Cache, Full Text Search, Analytical Search, Time Series, Key-Value, Eventing and other features along with Vector search into single platform.
    3. Ecosystem integration: LangChain and LlamaIndex integration.
    4. Proven Speed and Flexibility: In-memory architecture, flexible json format and powering indexing.

More details can be found in our vector search release announcement. 

Prerequisites

  1. Couchbase Capella or Couchbase Server 7.6 EE 
  2. Have already created a database
  3. Sample data:
  4. Index file: color-index.json

    • color-index.json:
  5. Sample search definition:
    { "fields": ["*"], "query": { "match_none": "" }, "knn": [ { "k": 2, "field": "colorvect_l2", "vector": [ 0, 0, 128 ] } ] }

Steps

Create Sample Data

Open Capella UI, Go to Database, and start importing data using import from browser using data tools:

    • Use sample rgb.json data file provided in prerequisites.
    • Choose option load from browser.
    • Select file: rgb.json
    • Specify new bucket with name: vector-sample
    • Specify new scope with name: color
    • Specify new collection with name: rgb
    • In step 3, preview your data,
      Choose how Capella creates identifiers for each of your documents. Select option as field and specify field: Id as identified as shown in screenshot below.
    • Click Import.

Create Vector Search Index 

In Search Options under Data Tools:

    • Create Search Index

    • Select advanced mode
    • Click on Index Definition on the right side of the UI
    • Select option Import from file
    • Choose file color-index.json specified in prerequisites
    • Specify Index Name as:  color-index.json
    • Choose bucket: vector-sample
    • Scope will be auto-populated as color
    • Click on Create Index

Perform a Vector Search 

Select the Search option in the color-index row (button near far right)

Paste the search text from the prerequisite step into the Search window.

Click on Search to get a result (shows in window below the search text).

 

Conclusion

In this post, we have gone through the basics of what vector search is and how to quickly get started with Vector Search with Couchbase. 

After executing a basic vector search, one can easily combine SQL queries with vector search in Couchbase, helping to consolidate your database stack and avoid writing multiple queries to get a single meaningful result for an application.

Start for free

References

 

Author

Posted by Kishor Deshpande - Solutions Engineer

All Posts

Leave a reply