MongoDB, one of the leading NoSQL databases,
is well known for its fast performance, flexible schema, scalability and
great indexing capabilities. At the core of this fast performance lies MongoDB
indexes, which support efficient execution of queries by avoiding full-collection
scans and hence limiting the number of documents MongoDB searches.
Starting from
version 2.4, MongoDB began with an experimental feature supporting Full-Text Search using Text Indexes. This feature has now
become an integral part of the product (and is no longer an experimental
feature). In this article we are going to explore the full-text search
functionalities of MongoDB right from fundamentals.
If you are new to MongoDB, I recommend that you read the
following articles on Envato Tuts+ that will help you understand the basic concepts
of MongoDB:
The Basics
Before we get into any details, let us look at some background.
Full-text search refers to the technique of searching a full-text database against the search criteria specified by the user.
It is something similar to how we search any content on Google (or in fact any
other search application) by entering certain string keywords/phrases and
getting back the relevant results sorted by their ranking.
Here
are some more scenarios where we would see a full-text search happening:
- Consider
searching your favorite topic on Wiki. When you enter a search text on Wiki,
the search engine brings up results of all the articles related to the keywords/phrase you searched for (even if those keywords were used deep inside
the article). These search results are sorted by relevance based on their
matched score. - As
another example, consider a social networking site where the user can make a
search to find all the posts which contain the keywordcats
in them; or to be more complex, all the posts which have
comments containing the wordcats
.
Before we move on, there are certain general terms related
to full-text search which you should know. These terms are applicable to any
full-text search implementation (and not MongoDB-specific).
Stop Words
Stop words are the irrelevant words that should be filtered
out from a text. For example: a, an, the, is, at, which, etc.
Stemming
Stemming is the process of reducing the words to their stem.
For example: words like standing, stands, stood, etc. have a common base stand.
Scoring
A relative ranking to measure which of the search results is
most relevant.
Alternatives to
Full-Text Search in MongoDB
Before MongoDB came up with the concept of text indexes, we
would either model our data to support keyword searches or use regular expressions for implementing such search
functionalities. However, using any of these approaches had its own limitations:
- Firstly,
none of these approaches supports functionalities like stemming, stop words,
ranking, etc. - Using
keyword searches would require the creation of multi-key indexes, which are not
sufficient compared to full-text. - Using
regular expressions is not efficient from the performance point of view, since
these expressions do not effectively utilize indexes. - In
addition to that, none of these techniques can be used to perform any phrase searches
(like searching for ‘movies released in 2015’) or weighted searches.
Apart from these approaches, for more advanced and complex
search-centric applications, there are alternative solutions like Elastic
Search or SOLR. But
using any of these solutions increases the architectural complexity of the
application, since MongoDB now has to talk to an additional external database.
Note
that MongoDB’s full-text search is not proposed as a complete replacement of search
engine databases like Elastic, SOLR, etc. However, it can be effectively used
for the majority of applications that are built with MongoDB today.
Introducing MongoDB
Text Search
Using MongoDB full-text search, you can define a text index
on any field in the document whose value is a string or an array of strings. When we create a text
index on a field, MongoDB tokenizes and stems the indexed field’s text content,
and sets up the indexes accordingly.
To understand things further, let us now dive into some practical
things. I want you to follow the tutorial with me by trying out the
examples in mongo shell. We will first create some sample data which we will be
using throughout the article, and then we’ll move on to discuss key concepts.
For the purpose of this article, consider a collection messages
which stores documents of the
following structure:
{ "subject":"Joe owns a dog", "content":"Dogs are man's best friend", "likes": 60, "year":2015, "language":"english" }
Let us insert some sample documents using the insert
command to create our test data:
db.messages.insert({"subject":"Joe owns a dog", "content":"Dogs are man's best friend", "likes": 60, "year":2015, "language":"english"}) db.messages.insert({"subject":"Dogs eat cats and dog eats pigeons too", "content":"Cats are not evil", "likes": 30, "year":2015, "language":"english"}) db.messages.insert({"subject":"Cats eat rats", "content":"Rats do not cook food", "likes": 55, "year":2014, "language":"english"}) db.messages.insert({"subject":"Rats eat Joe", "content":"Joe ate a rat", "likes": 75, "year":2014, "language":"english"})
Creating a Text Index
A text index is created quite similar to how we create a
regular index, except that it specifies the text
keyword instead of specifying an ascending/descending order.
Indexing a Single Field
Create a text index on the subject
field of our document using the following query:
db.messages.createIndex({"subject":"text"})
To test this newly created text index on the subject
field, we will search documents using the $text
operator. We will be looking for
all the documents that have the keyword dogs
in their subject
field.
Since we
are running a text search, we are also interested in getting some
statistics about how relevant the resultant documents are. For this purpose, we
will use the { $meta:
expression, which provides information on the processing
"textScore" }
of the $text
operator. We will also sort
the documents by their textScore
using the sort
command. A higher textScore
indicates a more relevant
match.
db.messages.find({$text: {$search: "dogs"}}, {score: {$meta: "toextScore"}}).sort({score:{$meta:"textScore"}})
The above query returns the following documents containing
the keyword dogs
in their subject
field.
{ "_id" : ObjectId("55f4a5d9b592880356441e94"), "subject" : "Dogs eat cats and dog eats pigeons too", "content" : "Cats are not evil", "likes" : 30, "year" : 2015, "language" : "english", "score" : 1 } { "_id" : ObjectId("55f4a5d9b592880356441e93"), "subject" : "Joe owns a dog", "content" : "Dogs are man's best friend", "likes" : 60, "year" : 2015, "language" : "english", "score" : 0.6666666666666666 }
As you can see, the first document has a score of 1 (since
the keyword dog
appears twice in its subject) as opposed to the second document
with a score of 0.66. The query has also sorted the returned documents in descending
order of their score.
One question that might
arise in your mind is that if we are searching for the keyword dogs
, why is the search engine is taking
the keyword dog
(without ‘s’) into
consideration? Remember our discussion on stemming, where any search keywords
are reduced to their base? This is the reason why the keyword dogs
is reduced to dog
.
Indexing Multiple
Fields (Compound Indexing)
More often than not, you will be using text search on
multiple fields of a document. In our example, we will enable compound text
indexing on the subject
and content
fields. Go ahead and execute
the following command in mongo shell:
db.messages.createIndex({"subject":"text","content":"text"})
Did this work? No!! Creating a second text index will give
you an error message saying that a full-text search index already exists. Why is it so? The answer is that text
indexes come with a limitation of only one text index per collection. Hence if
you would like to create another text index, you will have to drop the existing
one and recreate the new one.
db.messages.dropIndex("subject_text") db.messages.createIndex({"subject":"text","content":"text"})
After executing the above index creation queries, try
searching for all documents with keyword cat
.
db.messages.find({$text: {$search: "cat"}}, {score: {$meta: "textScore"}}).sort({score:{$meta:"textScore"}})
The above query would output the following documents:
{ "_id" : ObjectId("55f4af22b592880356441ea4"), "subject" : "Dogs eat cats and dog eats pigeons too", "content" : "Cats are not evil", "likes" : 30, "year" : 2015, "language" : "english", "score" : 1.3333333333333335 } { "_id" : ObjectId("55f4af22b592880356441ea5"), "subject" : "Cats eat rats", "content" : "Rats do not cook food", "likes" : 55, "year" : 2014, "language" : "english", "score" : 0.6666666666666666 }
You can see that the score of the first document, which contains
the keyword cat
in both subject
and content
fields, is higher.
Indexing the Entire
Document (Wildcard Indexing)
In the last example, we put a combined index on the subject
and content
fields. But there can be scenarios where you want any text
content in your documents to be searchable.
For example, consider storing
emails in MongoDB documents. In the case of emails, all the fields, including
Sender, Recipient, Subject and Body, need to be searchable. In such scenarios you
can index all the string fields of your document using the $**
wildcard specifier.
The query would go something like this (make sure you are
deleting the existing index before creating a new one):
db.messages.createIndex({"$**":"text"})
This query would automatically set up text indexes on any
string fields in our documents. To test this out, insert a new document with a
new field location
in it:
db.messages.insert({"subject":"Birds can cook", "content":"Birds do not eat rats", "likes": 12, "year":2013, location: "Chicago", "language":"english"})
Now if you try text searching with keyword chicago
(query below), it will return
the document which we just inserted.
db.messages.find({$text: {$search: "chicago"}}, {score: {$meta: "textScore"}}).sort({score:{$meta:"textScore"}})
A few things I would like to focus on here:
- Observe
that we did not explicitly define an index on thelocation
field after we inserted a new document. This is because we
already have defined a text index on the entire document using the$**
operator. - Wildcard
indexes can be slow at times, especially in scenarios where your data is very
large. For this reason, plan your document indexes (aka wildcard indexes)
wisely, as it can cause a performance hit.
Advanced Searching
Phrase Search
You can search for phrases like “smart birds who love cooking” using text indexes. By default, the
phrase search makes an OR search on
all the specified keywords, i.e. it will look for documents which contains
either the keywords smart
, bird
, love
or cook
.
db.messages.find({$text: {$search: "smart birds who cook"}}, {score: {$meta: "text Score"}}).sort({score:{$meta:"text Score"}})
This query would output the following documents:
{ "_id" : ObjectId("55f5289cb592880356441ead"), "subject" : "Birds can cook", "content" : "Birds do not eat rats", "likes" : 12, "year" : 2013, "location" : "Chicago", "language" : "english", "score" : 2 } { "_id" : ObjectId("55f5289bb592880356441eab"), "subject" : "Cats eat rats", "content" : "Rats do not cook food", "likes" : 55, "year" : 2014, "language" : "english", "score" : 0.6666666666666666 }
In case you would like to perform an exact phrase search
(logical AND), you can do so by
specifying double quotes in the search text.
db.messages.find({$text: {$search: ""cook food""}}, {score: {$meta: "textScore"}}).sort({score:{$meta:"textScore"}})
This query would result in the following document, which
contains the phrase “cook food” together:
{ "_id" : ObjectId("55f5289bb592880356441eab"), "subject" : "Cats eat rats", "content" : "Rats do not cook food", "likes" : 55, "year" : 2014, "language" : "english", "score" : 0.6666666666666666 }
Negation Search
Prefixing a search keyword with –
(minus sign) excludes all the documents that contain the negated
term. For example, try searching for any document which contains the
keyword rat
but does not contain birds
using the following query:
db.messages.find({$text: {$search: "rat -birds"}}, {score: {$meta: "textScore"}}).sort({score:{$meta:"textScore"}})
Looking Behind the Scenes
One important functionality I did not disclose till now is
how you look behind the scenes and see how your search keywords are being stemmed,
stop wording applied, negated, etc. $explain
to the rescue. You can run the explain query by passing true
as its parameter, which will give you detailed stats on the
query execution.
db.messages.find({$text: {$search: "dogs who cats dont eat ate rats "dogs eat" -friends"}}, {score: {$meta: "textScore"}}).sort({score:{$meta:"textScore"}}).explain(true)
If
you look at the queryPlanner
object
returned by the explain command, you will be able to see how MongoDB parsed the
given search string. Observe that it neglected stop words like who
, and stemmed dogs
to dog
.
You can also see the terms which we neglected from our search
and the phrases we used in the parsedTextQuery
section.
"parsedTextQuery" : { "terms" : [ "dog", "cat", "dont", "eat", "ate", "rat", "dog", "eat" ], "negatedTerms" : [ "friend" ], "phrases" : [ "dogs eat" ], "negatedPhrases" : [ ] }
The explain query will be highly useful as we perform more
complex search queries and want to analyze them.
Weighted Text Search
When we have indexes on more than one field in our document,
most of the times one field will be more important (i.e. more weight) than
the other. For example, when you are searching across a blog, the title of the
blog should be of highest weight, followed by the blog content.
The default weight for every indexed field is 1. To assign
relative weights for the indexed fields, you can include the weights
option while using the createIndex
command.
Let’s understand this with an example. If you try searching for the cook
keyword with our current
indexes, it will result in two documents, both of which have the same
score.
db.messages.find({$text: {$search: "cook"}}, {score: {$meta: "textScore"}}).sort({score:{$meta:"textScore"}})
{ "_id" : ObjectId("55f5289cb592880356441ead"), "subject" : "Birds can cook", "content" : "Birds do not eat rats", "likes" : 12, "year" : 2013, "location" : "Chicago", "language" : "english", "score" : 0.6666666666666666 } { "_id" : ObjectId("55f5289bb592880356441eab"), "subject" : "Cats eat rats", "content" : "Rats do not cook food", "likes" : 55, "year" : 2014, "language" : "english", "score" : 0.6666666666666666 }
Now let us modify our indexes to include weights; with the subject
field having a weight of 3
against the content
field having a weight
of 1.
db.messages.createIndex( {"$**": "text"}, {"weights": { subject: 3, content:1 }} )
Try searching for keyword cook
now, and you will see that the document which contains this keyword
in the subject
field has a greater score
(of 2) than the other (which has 0.66).
{ "_id" : ObjectId("55f5289cb592880356441ead"), "subject" : "Birds can cook", "content" : "Birds do not eat rats", "likes" : 12, "year" : 2013, "location" : "Chicago", "language" : "english", "score" : 2 } { "_id" : ObjectId("55f5289bb592880356441eab"), "subject" : "Cats eat rats", "content" : "Rats do not cook food", "likes" : 55, "year" : 2014, "language" : "english", "score" : 0.6666666666666666 }
Partitioning Text
Indexes
As the data stored in your application grows, the size of your text indexes keeps on growing too. With this increase in size of text indexes,
MongoDB has to search against all the indexed entries whenever a text search is
made.
As a technique to keep your text search efficient with growing indexes, you
can limit the number of scanned index entries by using equality conditions with a regular $text
search. A very common
example of this would be searching all the posts made during a certain
year/month, or searching all the posts with a certain category/tag.
If you observe the documents which we are working upon, we
have a year
field in them which we
have not used yet. A common scenario would be to search messages by year, along
with the full-text search that we have been learning about.
For this, we can
create a compound index that specifies an ascending/descending index key on year
followed by a text index on the subject
field. By doing this, we are
doing two important things:
- We are logically partitioning the
entire collection data into sets separated by year. - This would limit the text search to
scan only those documents which fall under a specific year (or call it set).
Drop the indexes that you already have and create a new
compound index on (year
, subject
):
db.messages.createIndex( { "year":1, "subject": "text"} )
Now execute the following query to search all the messages
that were created in 2015 and contain the cats
keyword:
db.messages.find({year: 2015, $text: {$search: "cats"}}, {score: {$meta: "textScore"}}).sort({score:{$meta:"textScore"}})
The query would return only one matched document as expected.
If you explain
this query and look
at the executionStats
, you will find
that totalDocsExamined
for this
query was 1, which confirms that our new index got utilized correctly and
MongoDB had to only scan a single document while safely ignoring all other
documents which did not fall under 2015.
Text Indexes: Benefits
What More Can Text
Indexes Do?
We have come a long way in this article learning about text
indexes. There are many other concepts that you can experiment with text
indexes. But owing to the scope of this article, we will not be able to discuss
them in detail today. Nevertheless, let’s have a brief look at what these
functionalities are:
- Text
indexes provide multi-language support, allowing you to search in different
languages using the$language
operator. MongoDB currently supports around 15
languages, including French, German, Russian, etc. - Text
indexes can be used in aggregation pipeline queries. The match stage in an
aggregate search can specify the use of a full-text search query. - You
can use your regular operators for projections, filters, limits, sorts, etc., while
working with text indexes.
MongoDB Text Indexing
vs. External Search Databases
Keeping in mind the fact that MongoDB full-text search is not
a complete replacement for traditional search engine databases used with
MongoDB, using the native MongoDB functionality is recommended for the
following reasons:
- As
per a recent talk at MongoDB, the current scope of text search works
perfectly fine for a majority of applications (around 80%) that are built using
MongoDB today. - Building
the search capabilities of your application within the same application
database reduces the architectural complexity of the application. - MongoDB
text search works in real time, without any lags or batch updates. The moment
you insert or update a document, the text index entries are updated. - Text
search being integrated into the db kernel functionalities of MongoDB, it is
totally consistent and works well even with sharding and replication. - It
integrates perfectly with your existing Mongo features such as filters,
aggregation, updates, etc.
Text Indexes: Drawbacks
Full-text search being a relatively new feature in MongoDB,
there are certain functionalities which it currently lacks. I would divide them into three categories. Let’s have a look.
Functionalities Missing
From Text Search
- Text
Indexes currently do not have the capability to support pluggable interfaces
like pluggable stemmers, stop words, etc. - They
do not currently support features like searching based on synonyms, similar words, etc. - They
do not store term positions, i.e. the number of words by which the two keywords
are separated. - You
cannot specify the sort order for a sort expression from a text index.
Restrictions in
Existing Functionalities
- A
compound text index cannot include any other type of index, like multi-key
indexes or geo-spatial indexes. Additionally, if your compound text index
includes any index keys before the text index key, all the queries must specify
the equality operators for the preceding keys. - There
are some query-specific limitations. For example, a query can specify only a single$text
expression, you can’t use$text
with$nor
, you can’t use thehint()
command with$text
, using$text
with$or
needs all the clauses in your$or
expression to be indexed, etc.
Performance Downsides
- Text indexes create an overhead while inserting new documents. This in turn hits the insertion throughput.
- Some queries like phrase searches can be relatively slow.
Wrapping Up
Full-text search has always been one of the most demanded
features of MongoDB. In this article, we started with an introduction to what full-text search is, before moving on to the basics of creating text indexes.
We then explored
compound indexing, wildcard indexing, phrase searches and negation searches. Further,
we explored some important concepts like analyzing text indexes, weighted
search, and logically partitioning your indexes. We can expect some major updates to this functionality in the
upcoming releases of MongoDB.
I recommend that you give text-search a try and share your thoughts. If you have already implemented it in your application, kindly share your experience here. Finally, feel free to post your questions, thoughts and
suggestions on this article in the comment section.
Source: Envato Tuts+ CodeNew feed