Diagnosing MongoDB Search Performance with Atlas Metrics

Your MongoDB Atlas metrics show elevated disk I/O, high search CPU, increasing query latency, or a rising search max replication lag. Like when your car’s mysterious “check engine” light comes on, these signals can mean different things. To figure out the issue, you’ll want to start by identifying which process and metric is under pressure.

Scaling the database tier may provide temporary relief, but it does not separate the search and operational workloads. When mongot and mongod share nodes, search can compete with database activity for memory, disk I/O, and CPU. The right response depends on whether the index does not fit in available memory, the workloads contend during traffic spikes, or possibly even both occurring simultaneously.

This post explains how to diagnose slow search performance, decide whether dedicated search nodes are the right fix, and size them correctly if they are.

Why tackling the underlying issue matters

The default response to slow search is often to scale up the database tier. More RAM and CPU can help the database portion of the workload, but they do not create a separate resource budget for search on a co-located deployment.

Two cases can produce similar symptoms, but they call for different responses. With index-size pressure, the Search index is larger than the memory available to mongot, so queries page data from disk even when database activity is low. A larger Search tier with enough memory addresses this problem; isolation by itself does not. With resource contention, the index fits in memory, but search latency rises when database reads, writes, or replication become busy. Separate search nodes address this contention by giving mongot different CPU, memory, and disk resources.

The distinction matters when you select a tier. Dedicated search nodes remove competition with database workloads, but they still need to be sized correctly.

Search nodes explained

MongoDB Search and MongoDB Vector Search use mongot, the search process that maintains and queries search indexes. mongod is the database process that stores application data and handles database operations. In a colocated deployment, both processes run on the same nodes.

Dedicated search nodes run mongot on separate infrastructure from the nodes running mongod. Dedicated search nodes provide workload isolation for search, similar to how Analytics Nodes isolate analytical workloads.

With dedicated search nodes, database nodes handle operational reads and writes while search nodes handle search queries and index work. Each workload gets its own memory budget, CPU capacity, and scaling lifecycle. This applies to both MongoDB Search and MongoDB Vector Search on Atlas.

Why colocated search runs into resource constraints

When mongot and mongod share nodes, memory and CPU are common sources of resource pressure. WiredTiger, the operating system, and mongot share RAM; if the index is larger than the memory available to mongot, queries may read index pages from disk. Search queries and background indexing also compete with database reads, writes, and replication for CPU time, so latency can rise even when the index fits in memory.

Atlas metrics help show which process is under pressure and whether the issue is paging or contention. Look at these signals together rather than treating any one metric as a diagnosis:

  • Search Page Fault, which reports average page faults per second in the Search process

  • Search Process Memory and Search Index Size compared with the node's RAM

  • Search Normalized Process CPU compared with Normalized System CPU

  • Search Process Throttling, which indicates that work is arriving faster than the Search process can admit it

  • Search query latency and Search Max Replication Lag

If database CPU and memory look healthy while Search metrics are under pressure, Search is consuming the resources available to it. That does not identify the root cause by itself. Compare index size with available memory to test for paging; compare latency and Search CPU with database activity to test for contention.

If page faults and latency stay high while the database is quiet, Search needs more memory. If latency only rises when the database gets busy, the two workloads are competing.

How dedicated search nodes separate search and database workloads

Memory isolation and cache performance

On a colocated node, the memory shown in the Atlas tier specification is shared. WiredTiger uses part of it for the database cache, the operating system and runtime need additional memory, and the remainder is available to mongot. The exact amount varies with tier and workload, and there is no fixed co-located cache size for mongot.

Dedicated search nodes give mongot a separate memory budget. MongoDB documentation describes approximately 90% of search node RAM as available for index data, with the remainder used by the JVM and other process needs. MongoDB Search indexes do not need to fit entirely in memory. A single query usually touches only part of the index, and the OS keeps the frequently used pages in RAM. MongoDB Vector Search is different and works best when the index fits in memory.

When there is sufficient memory to accommodate the majority of the index data, sustained page faults should reduce and latency should become more stable. If page faults remain high after migration to search nodes, the Search tier may still be undersized, or the workload may have another bottleneck. Isolation will help, but if this is an index sizing issue at the core, you may need to combine this approach with other techniques. Those techniques could include switching from dynamic mapping to static mapping on specific fields, using alternate tokenization strategies other than nGram for autocomplete fields, leveraging vector quantization for vector search indexes, and other actions needed to reduce the size of the index.

CPU separation and parallel segment search

Search queries and index updates can consume substantial CPU. Separate search nodes prevent database traffic from taking those cores during the same interval. For $search queries, parallel segment search can search multiple Lucene segments concurrently when enabled with the query's concurrent option. For $vectorSearch queries, concurrent segment search is on by default on search nodes. The Search tier's CPU count limits the parallelism available.

MongoDB Search might fall back to single-threaded execution when too many concurrent queries are queued. Treat parallel segment search as additional capacity, not as a guarantee that every query will run in parallel. Monitor query latency, Search Normalized Process CPU, and Search Process Throttling under representative concurrency.

No competition with operational traffic

Search Max Replication Lag means index staleness: How far behind mongot is in applying changes from the mongod oplog to a MongoDB Search or MongoDB Vector Search index. This is different from replica set replication lag, which measures how far secondary mongod members are behind the primary.

Isolation removes one cause of Search Max Replication Lag: Indexing work from mongot competing with mongod for CPU on the same node. It does not remove the indexing work. mongot still reads every change from mongod and applies it to the search index on search nodes.

Isolation does not resolve issues due to write-heavy workloads. Sustained write volumes and the amount of indexing work created by each change can increase index staleness. Factors include the number of indexed fields, autocomplete fields using nGram or edgeGram tokenizers, and embeddedDocuments mappings where one document change can rewrite many Lucene documents. Indexes get updated when their associated documents and fields change; minimally, one write to a document will generate at least one corresponding write to the index(es) (where the indexed field is updated). Schema changes or sharding may be required with dedicated nodes to optimally support the workload.

Replica set replication lag is separate. Search nodes may reduce CPU contention that affects the database, but they do not guarantee lower replica set lag or lower Search Max Replication Lag when write or indexing volume remains high.

Example: From page faults to consistent search performance

Consider a co-located M30 deployment with 8 GB of RAM per node. As an illustration rather than a fixed Atlas allocation, about 2 GB may go to the WiredTiger cache and at least 1 GB to the OS, mongod, connections, and other runtime needs, leaving about 5 GB or less for mongot. If the Search index is 6 GB, sustained Search Page Fault activity is highly likely. The database may show low CPU and healthy memory metrics while Search still reads index data from disk.

Before search nodes

In this example, the co-located deployment is an M30 with 8 GB RAM per node and a 6 GB Search index. Atlas metrics show sustained non-zero Search Page Fault during normal traffic, and search query latency p95 is 650 ms. Validate the estimated 5 GB or less available to mongot against the actual metrics.

After search nodes

Assume the workload moves to an S40 Search tier with 16 GB of RAM per node. Using the documented approximate 90% allocation as a planning estimate gives about 14.4 GB for index data, leaving room for a 6 GB index and query working set. Validate the tier's available memory and observed behavior in Atlas rather than treating the estimate as a guarantee.

After the move, the dedicated S40 Search tier has about 14.4 GB of memory available for the same 6 GB index. In this example, Search Page Fault falls after the index is resident, except during warm-up or workload changes, and search query latency p95 is 120 ms.

Performance improves once Atlas finishes building the index on the search nodes and routes queries to them. During the build, queries continue to use the existing colocated indexes, and index construction adds read load to the database cluster. Plan for that transition instead of expecting a change when the Search Nodes are first deployed.

When to move to dedicated search nodes

Dedicated search nodes are not necessary for every slow query: if an index fits comfortably in available memory and latency targets are met without contention, co-location may remain the simpler choice. The example above shows why a team might move when paging indicates that the Search index does not fit the memory budget. In production, look for a pattern across several signals before choosing a dedicated tier:

  • Sustained non-zero Search Page Fault with an index that approaches or exceeds available Search memory

  • Search Process Throttling remaining above zero, indicating that work is arriving faster than the Search process can admit it

  • High Search Normalized Process CPU or a high Search-to-system CPU ratio during query traffic

  • Search latency that rises when database activity increases

  • Search Max Replication Lag that rises with indexing activity, after ruling out write volume and index-mapping causes

  • Query latency targets that require consistent performance

  • Repeated database scaling to accommodate Search growth

On a colocated node, total disk I/O combines mongod and mongot activity, so disk IOPS alone is a weak signal for Search diagnosis. Start with Search Page Fault to attribute paging to the Search process, and use total disk IOPS as corroborating evidence.

Best practices

These checks make a tier decision more reliable: they help identify whether the bottleneck is memory, CPU, or indexing work, and when to plan the migration. Use the relevant metrics together, and account for the workload and operational requirements before changing deployment architecture.

  • Check Search Process Memory, Search Index Size, and Search Page Fault together when testing for memory pressure.

  • Check Search Normalized Process CPU, Normalized System CPU, latency, and Search Process Throttling together when testing for contention or saturation.

  • Treat Search Max Replication Lag as index staleness, and investigate replica set replication lag separately.

  • Size Search Nodes for both index memory and query or indexing CPU; plan for index growth and concurrency.

  • Confirm prerequisites before migration. Dedicated search nodes require an M10 or higher cluster, so upgrade a Flex cluster before attempting to use them.

  • Expect an index-building period before query routing changes, then monitor the search nodes after the cutover.

What workload isolation enables

Analytics nodes separate analytical workloads from operational database traffic. Dedicated search nodes apply the same workload-isolation model to MongoDB Search and MongoDB Vector Search, allowing each infrastructure tier to be sized for its own workload.

The result is easier diagnosis and more predictable behavior. Page faults can be addressed with sufficient Search memory, CPU contention can be addressed with isolation, and indexing lag can be investigated using write volume and index-mapping details rather than treated as a generic replication problem. Views for search indexes, quantization for vector indexes, sharding, and other techniques can also be used with isolation to further improve performance. Search index partitioning can also extend the capacity of supported deployments beyond a single index or node's limits; consult the current MongoDB documentation for those limits.

Next steps

Use the metrics to identify whether memory pressure, workload contention, or indexing volume is limiting performance. Then use the deployment guidance to assess the appropriate Search tier and plan a migration if dedicated nodes fit the diagnosis.

  • Review MongoDB Search metrics for Search Process Memory, Search Index Size, Search Page Fault, and Search Process Throttling.

  • Identify whether page faults indicate an index-memory problem or whether latency follows database activity and indicates contention.

  • Check Search Max Replication Lag and replica set replication lag as separate metrics with separate investigations.

  • Review tips for sizing and scaling Search Nodes and MongoDB Vector Search production sizing.

  • Account for index build time and the temporary read load before evaluating post-migration performance.

← MongoDBAll Companies