# Elasticsearch max_result_window: The 10,000 Limit, Heap Risks, and search_after The Elasticsearch index.max_result_window setting limits standard pagination to 10,000 documents per query across all shards. Raising this limit to accommodate deep offset queries introduces severe JVM heap memory pressure and risks out-of-memory cluster crashes. Implementing search_after alongside Point in Time (PIT) snapshots provides efficient, stateless cursor pagination without overloading coordinator nodes. Source: https://fast.io/resources/elasticsearch-max-result-window/ Author: [Derek Labian](https://fast.io/authors/derek-labian/) Last reviewed: 2026-10-18 ## What Is the Elasticsearch max_result_window Limit? In Elasticsearch, any search request where the sum of `from` and `size` exceeds 10,000 documents fails immediately with a `Result window is too large` error. This 10,000-hit boundary is governed by the `index.max_result_window` cluster setting as documented in the [Elasticsearch pagination documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/paginate-search-results.html), an intentional safeguard designed to prevent coordinator nodes and shards from collapsing under the memory weight of deep offset pagination. Elasticsearch index.max_result_window is a cluster protection setting that restricts the sum of from and size to 10,000 documents per query to prevent out-of-memory crashes during deep pagination. When client applications submit a search query that crosses this threshold, Elasticsearch terminates the query phase across all shards and returns an HTTP 500 or 400 error payload: ```json { "error": { "root_cause": [ { "type": "query_phase_execution_exception", "reason": "Result window is too large, from + size must be less than or equal to: [10000] but was [10010]. See the scroll api in the documentation for large results." } ], "type": "search_phase_execution_exception", "reason": "all shards failed", "phase": "query", "grouped": true, "failed_shards": [] }, "status": 500 } ``` ### The Mechanics of from and size Pagination Standard pagination in Elasticsearch mirrors traditional SQL offset pagination through two integer parameters: - `from`: The numerical offset indicating how many matching hits to skip (defaulting to 0). - `size`: The maximum number of hits to return in the response payload (defaulting to 10). For shallow pagination, such as displaying the first five pages of product search results in an ecommerce web interface (`from: 0, size: 20` up to `from: 80, size: 20`), offset pagination functions smoothly with negligible latency. The cluster easily collects the top 100 documents across participating shards and formats the JSON response. The operational breakdown begins when client systems attempt to treat Elasticsearch as an export queue or deep analytical cursor. If an API consumer asks for page 501 using `from: 10000, size: 10`, the sum evaluates to 10,010. Because this exceeds the default `index.max_result_window` ceiling of 10,000, the coordinator node rejects the execution. By default, Elasticsearch restricts the sum of from and size parameters to a maximum of 10,000 hits per query window. Understanding why this restriction exists requires examining how distributed shards execute queries across a cluster. ## How Deep Pagination Triggers JVM Heap Memory Pressure Elasticsearch indices are divided into shards distributed across physical or virtual cluster nodes. When an application dispatches a search query with `from` and `size`, the cluster coordinates execution through a two-phase protocol: the query phase and the fetch phase. Search requests usually span multiple shards, where each shard must load its requested hits and the hits for any previous pages into memory. ``` +-----------------------------------------------------------------------------------+ | COORDINATOR NODE | | Receives request: from=9990, size=20 | | Dispatches scatter query to all participating index shards | +------------------------------------+----------------------------------------------+ | +-------------------------+-------------------------+ | | v v +-----------------------+ +-----------------------+ | SHARD 1 | | SHARD 2 | | Evaluates query | | Evaluates query | | Builds local queue | | Builds local queue | | Holds 10,010 hits | | Holds 10,010 hits | | Sorts local entries | | Sorts local entries | +-----------+-----------+ +-----------+-----------+ | | | Transmits 10,010 descriptors | Transmits 10,010 descriptors +-------------------------+-------------------------+ | v +-----------------------------------------------------------------------------------+ | COORDINATOR NODE (GATHER) | | Merges shard streams into global priority queue (5 shards = 50,050 entries) | | Executes global heap sort to identify true top 10,010 hits | | Discards the first 9,990 items from JVM heap memory | | Fetches _source only for the final 20 documents | | Returns 20 hits to client | +-----------------------------------------------------------------------------------+ ``` ### The Scatter Phase at the Shard Level In the query phase, the coordinator node forwards the query to a primary or replica copy of every shard in the target index. An individual shard cannot know how its local documents compare to documents residing on other shards without a global sort. Therefore, to identify the global top `from + size` documents correctly, every individual shard must collect and score its own top `from + size` documents. If an index consists of 5 shards and a client requests `from: 9990, size: 20`: 1. Each of the 5 shards must build an internal Lucene priority queue sized for `10,010` hits. 2. Each shard loads, scores, and sorts 10,010 candidate document descriptors in local memory. 3. Each shard serializes all 10,010 hit descriptors (Lucene doc IDs and sort keys) and transmits them across the internal cluster network to the coordinator node. In this scenario, the 5 shards transfer a combined total of `50,050` document descriptors over the network to fulfill a request for just 20 documents. ### The Gather Phase on the Coordinator Node Once the shard responses arrive, the coordinator node must perform a global merge-sort across all 50,050 entries. It allocates a priority queue in its Java Virtual Machine (JVM) heap to sort the entire merged set. After completing the sort: 1. The coordinator discards the first 9,990 items from memory. 2. It identifies the remaining 20 hits that represent the requested page. 3. It sends individual fetch requests back to the specific shards holding those 20 documents to retrieve the raw `_source` JSON fields. 4. It compiles the final response payload and delivers it to the client. ### Algorithmic Complexity and Heap Exhaustion The computational and memory cost of offset pagination scales as O(S * (from + size)), where S represents the total number of shards queried. Under single-user testing, sorting 50,000 items in memory consumes a few milliseconds. In production environments where dozens of concurrent automated jobs, API clients, or background export scripts request deep pages simultaneously, memory demands multiply rapidly. Multiple concurrent queries at deep offsets across participating shards force coordinator nodes to allocate, sort, and collect millions of ephemeral object references simultaneously. This pattern causes rapid JVM heap memory degradation: 1. **Young Generation Saturation:** The rapid allocation of large object arrays overwhelms the JVM young generation (Eden space), triggering frequent young garbage collection cycles. 2. **Premature Object Tenuring:** Ephemeral priority queues that survive young collections because of active query execution are pushed prematurely into the Old Generation. 3. **Stop-The-World Pauses:** The JVM initiates concurrent mark-sweep or G1GC garbage collection phases to clean old gen heap space. These collection cycles cause multi-second Stop-The-World (STW) pauses. 4. **Node Ping Timeouts and Cluster Partitions:** When garbage collection freezes a data node for several seconds, the node stops responding to cluster heartbeat pings. The master node assumes the data node has crashed, removes it from the cluster state, and starts re-routing shard replicas. 5. **Cascading Outages:** Shard recovery and re-routing flood network interfaces and disk queues on the remaining nodes, causing them to undergo memory exhaustion and crash with `java.lang.OutOfMemoryError: Java heap space`. ## Why Increasing index.max_result_window Is an Anti-Pattern When developers encounter the `Result window is too large` exception, a common first reaction is to search for a configuration toggle. Elasticsearch allows administrators to modify `index.max_result_window` dynamically on a live index using the index settings API: ```http PUT /customer_events/_settings { "index.max_result_window": 50000 } ``` Many community posts, tutorials, and quick answers advise setting this parameter to 50,000 or 100,000 to make the error disappear. Raising this setting in production environments is an operational anti-pattern that creates substantial cluster instability. ### Why Raising the Ceiling Fails Bumping `index.max_result_window` does not optimize query execution. It does not introduce index skipping, nor does it compress memory structures. It simply disables the cluster protection mechanism, permitting client requests to execute queries that consume dangerous amounts of CPU and RAM. Once the window is increased to 50,000, client applications invariably begin issuing requests with offsets near that ceiling. The resulting workload introduces three systemic points of failure: #### 1. Circuit Breaker Trips Elasticsearch incorporates built-in memory circuit breakers to guard against out-of-memory crashes. The primary breaker, configured via `indices.breaker.total.use_real_memory`, tracks actual JVM heap utilization. When a deep query attempts to allocate memory that would push total usage beyond the breaker threshold (a configured share of heap capacity), Elasticsearch terminates the query and returns a `CircuitBreakingException`: ``` [parent] Data too large, data for [] would be [314572800/300mb], which is larger than the limit of [306184192/292mb] ``` When circuit breakers trip, queries fail intermittently across the entire index, affecting both shallow and deep searches. #### 2. Search Thread Pool Saturation Each Elasticsearch node maintains a dedicated `search` thread pool. When coordinator nodes spend hundreds of milliseconds executing sorting routines on hundreds of thousands of shard descriptors, search worker threads remain occupied. Subsequent queries accumulate in the search thread pool queue (defaulting to a capacity of 1,000). When the queue fills up, the node rejects incoming search requests with an HTTP 429 status code and an `EsRejectedExecutionException`: ``` org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution of org.elasticsearch.transport.TransportService ``` #### 3. Segment Cache Thrashing Sorting across deep result sets requires reading field data or Lucene doc values into memory. If the data exceeds available OS page cache budgets, nodes must constantly read segments from physical disk storage. This page cache churn creates severe I/O wait spikes, elevating search latencies across completely unrelated indices hosted on the same data nodes. ### Permissible Exceptions There are rare operational scenarios where a temporary adjustment to `index.max_result_window` is acceptable: - Running a one-time administrative migration on a dedicated, single-shard staging index during an off-peak maintenance window. - Accommodating an internal legacy reporting tool that lacks cursor support, provided the index is small, strictly rate-limited, and not shared with user-facing production services. In all other production environments, `index.max_result_window` should remain at its default value of 10,000. Instead of increasing the window, teams should adopt cursor-based pagination. ## Implementing Deep Pagination with search_after and Point in Time The official and production-tested mechanism for deep pagination in Elasticsearch is `search_after`. Rather than defining a numerical offset from the beginning of the result set, `search_after` uses a cursor composed of the sort values from the last document on the preceding page. Because `search_after` tells Lucene where to resume scanning, shards do not need to calculate or collect prior hits. Each shard only needs to evaluate and return its local top `size` documents. Memory consumption remains constant at O(S * size) whether you are fetching page 1 or page 50,000. As a result, `search_after` is completely exempt from the `index.max_result_window` boundary. ### The Tie-Breaker Requirement To use `search_after` reliably, queries must define an explicit and deterministic sort order. If you sort documents using a non-unique field, such as a creation date or a numeric status code: ```json "sort": [ { "created_at": "desc" } ] ``` Multiple documents will often share identical timestamps. When a page boundary falls across documents with duplicate sort values, Elasticsearch cannot determine which document served as the true cursor. Documents can be skipped or returned twice across consecutive pages. Lucene internal document IDs can also differ between primary and replica shards as background segment merges occur. For deterministic results, always append a unique tie-breaker field to your sort array. The most common tie-breaker is the document `_id` field: ```json "sort": [ { "created_at": "desc" }, { "_id": "asc" } ] ``` ### Freezing State with Point in Time (PIT) In an active production index, documents are continuously inserted, updated, and deleted. In basic `search_after` implementations, index mutations that occur between paginated requests alter the document rankings, causing result drift. To maintain a consistent, stable view across multiple requests, pair `search_after` with the Point in Time (PIT) API. A Point in Time creates a lightweight, read-consistent view of the index state across shards at the moment it is opened. Unlike the deprecated Scroll API, which maintained an active, resource-heavy search execution context on every shard, a PIT preserves the underlying Lucene segment readers without holding open expensive query state. ### Step-by-Step Implementation Workflow #### Step 1: Open a Point in Time Open a Point in Time by targeting your index and specifying a keep-alive duration: ```http POST /customer_events/_pit?keep_alive=2m ``` Elasticsearch responds with a unique PIT ID string: ```json { "id": "46ToAwMDY3VzdG9tZXJfZXZlbnRzFkFBRF9YWFhYWFhYWFhYWFhBQUFBQUEAFjFYYmNfZGVmR2hpSmtsbW5vcHFyc3cAAQAAAAAAAAAAAAAWQUFEX1hYWFhYWFhYWFhYQUFBQUFBABZ1c2VyX2V2ZW50c18wMDAwMDEAAA==" } ``` #### Step 2: Execute the Initial Search Request Execute your initial search request against the global `/_search` endpoint. Instead of targeting the index name in the URL, pass the PIT ID inside the request body. Include your desired `size`, query conditions, and explicit sort criteria: ```http POST /_search { "size": 100, "query": { "match": { "event_type": "account_created" } }, "pit": { "id": "46ToAwMDY3VzdG9tZXJfZXZlbnRzFkFBRF9YWFhYWFhYWFhYWFhBQUFBQUEAFjFYYmNfZGVmR2hpSmtsbW5vcHFyc3cAAQAAAAAAAAAAAAAWQUFEX1hYWFhYWFhYWFhYQUFBQUFBABZ1c2VyX2V2ZW50c18wMDAwMDEAAA==", "keep_alive": "2m" }, "sort": [ { "created_at": "desc" }, { "_id": "asc" } ] } ``` The response includes the first 100 matching hits. Each hit object in the `hits.hits` array contains a `sort` array representing its specific sort values: ```json { "took": 14, "timed_out": false, "hits": { "total": { "value": 150000, "relation": "eq" }, "hits": [ { "_index": "customer_events_000001", "_id": "evt_1001", "_score": null, "_source": { "event_type": "account_created", "created_at": "2026-10-18T04:12:00.000Z" }, "sort": [ 1792296720000, "evt_1001" ] } ] } } ``` #### Step 3: Fetch Subsequent Pages Using search_after To retrieve the next page, extract the `sort` values from the very last document in the `hits.hits` array (for instance, `[1792296720000, "evt_1001"]`). Pass these values into the `search_after` parameter of the next request. Keep the same query, sort order, and PIT ID: ```http POST /_search { "size": 100, "query": { "match": { "event_type": "account_created" } }, "pit": { "id": "46ToAwMDY3VzdG9tZXJfZXZlbnRzFkFBRF9YWFhYWFhYWFhYWFhBQUFBQUEAFjFYYmNfZGVmR2hpSmtsbW5vcHFyc3cAAQAAAAAAAAAAAAAWQUFEX1hYWFhYWFhYWFhYQUFBQUFBABZ1c2VyX2V2ZW50c18wMDAwMDEAAA==", "keep_alive": "2m" }, "sort": [ { "created_at": "desc" }, { "_id": "asc" } ], "search_after": [ 1792296720000, "evt_1001" ] } ``` Continue iterating through pages until the response returns an empty `hits.hits` array, indicating that all matching documents have been retrieved. #### Step 4: Explicitly Close the Point in Time Each open Point in Time holds references to underlying Lucene segments, preventing old segments from being cleaned up after background merges. Once pagination completes, always release cluster resources immediately using the `DELETE /_pit` endpoint: ```http DELETE /_pit { "id": "46ToAwMDY3VzdG9tZXJfZXZlbnRzFkFBRF9YWFhYWFhYWFhYWFhBQUFBQUEAFjFYYmNfZGVmR2hpSmtsbW5vcHFyc3cAAQAAAAAAAAAAAAAWQUFEX1hYWFhYWFhYWFhYQUFBQUFBABZ1c2VyX2V2ZW50c18wMDAwMDEAAA==" } ``` If a client crashes or fails to send the delete call, Elasticsearch automatically frees the PIT once its `keep_alive` timer expires. ### Pagination Strategy Comparison The following matrix compares standard offset pagination against cursor and snapshot patterns: | Pagination Strategy | Optimal Use Case | Bound by max_result_window? | Coordinator Memory Load | Mutation Consistency | Production Status | |---|---|---|---|---|---| | `from` + `size` | Shallow interactive search (pages 1 to 10) | Yes (10,000 documents) | High (O(S * (from + size))) | Vulnerable to page drift | Standard for web search UIs | | `search_after` (Live) | Infinite scrolling on write-heavy event streams | No (Unrestricted) | Constant (O(S * size)) | Susceptible to inserts/deletions | Recommended for feeds and logs | | `search_after` + PIT | Deep exports, batch harvesting, audits | No (Unrestricted) | Constant (O(S * size)) | Fully consistent across requests | Recommended for deep paging | | Scroll API | Large batch data exports (Legacy) | No (Unrestricted) | Heavy (locks shard search context) | Fully consistent across requests | Deprecated for general search | ## Decoupling Search from Agent Ingestion to Prevent Context Exhaustion The challenge of deep search pagination has become acute with the rise of autonomous AI agents and retrieval-augmented generation (RAG) pipelines. When engineering teams build agents to analyze enterprise corpora, audit contract repositories, or generate customer intelligence reports, they often point the agent directly at an internal Elasticsearch cluster. Inevitably, agents encounter the 10,000-document ceiling. An agent evaluating a query like "find all customer escalation logs related to billing anomalies" frequently attempts to harvest thousands of candidate documents by paginating deeply. This approach introduces two critical bottlenecks: 1. **Search Engine Destabilization:** Automated agents loop through deep offsets in seconds, triggering `max_result_window` exceptions or saturating JVM heap memory. 2. **Context Window Degradation:** Feeding thousands of raw search results into an LLM context window exhausts token limits and drives up latency. More problematically, overloading models with massive document sets degrades reasoning accuracy, causing models to miss critical evidence located in the middle of extended contexts. ### Context Ceilings and the Multi-File Corpus Problem This retrieval friction mirrors the ingestion boundaries developers encounter across native AI chat and project interfaces. As documented by Anthropic's [Claude Help Center](https://support.claude.com/en/articles/8241126-upload-files-to-claude), a standard Claude chat session accepts up to 20 files at up to 500MB each, while a Claude Project accepts files up to 30MB each with no fixed file-count cap, bounded only by the model's context window. While project libraries can hold dozens or hundreds of small documents, every document loaded into prompt context consumes token capacity and increases inference latency. Once a reference corpus exceeds the model context window, in-memory attachment breaks down. Rather than trying to force an entire document library into in-context attachments or running brittle deep-offset queries against production datastores, engineering teams decouple storage from prompt assembly. The complete document collection resides in an intelligent workspace. Files enter the workspace through direct upload or automated sync from existing repositories, including Google Drive, Dropbox, Box, and OneDrive. Once Intelligence Mode is turned on, the platform indexes the documents for semantic and lexical search. AI assistants then connect through remote Model Context Protocol (MCP) endpoints, querying the indexed workspace for targeted passages instead of loading unindexed document batches into memory. ### The Decoupled Workspace Pattern High-performing engineering teams avoid turning production search clusters into raw document dumps for AI agents. Instead, they decouple real-time search backends from document collaboration and analysis by adopting intelligent workspaces. In an intelligent workspace, document collections are stored in persistent, organization-owned directories. Rather than forcing an agent to script complex `search_after` loops against production datastores, the files are managed in a shared environment where humans and AI agents collaborate directly. Fastio provides cloud workspaces designed specifically for agentic teams. In Fastio: - Documents are organized in shared workspaces with per-file version history and role-based permissions configured at the organization, workspace, and share levels. - The detailed activity log records uploads, modifications, and administrative updates, providing full operational visibility into what changed and who changed it. - When Intelligence Mode is enabled on a workspace, incoming files (PDFs, spreadsheets, research papers, scanned images, and documentation) are automatically indexed for hybrid search, combining full-text keywords with semantic vector retrieval. ### Connecting AI Agents via MCP Rather than building custom API connectors or exporting massive batches of documents over HTTP, autonomous agents interact with Fastio workspaces through the Model Context Protocol (MCP). Fastio provides a consolidated remote MCP server hosted at `mcp.fast.io` running over Streamable HTTP: - **General Assistants and Claude:** Connect via `https://mcp.fast.io/mcp/tools`. - **Coding Agents and Terminal Runners:** Connect via `https://mcp.fast.io/mcp/code`. When an agent needs reference material to answer a query or complete a task, it does not execute deep offset queries across millions of database rows. Instead, it dispatches an MCP search action to query the indexed workspace. The search engine returns concise, highly relevant excerpts (typically 500 to 1,500 tokens) accompanied by verified document citations. For teams that need structured data from unstructured document sets, Fastio [Metadata Views](/product/document-data-extraction/) transform messy archives into live, queryable tables. Users define schema fields in natural language (Text, Integer, Decimal, Boolean, URL, JSON, Date & Time), and AI populates structured tables across PDFs, spreadsheets, and scanned documents without manual OCR rules. Agents can inspect schemas and query extraction results directly through MCP. Cloud Sync connects Google Drive, Dropbox, Box and OneDrive folders to a workspace, one-way or two-way, on a schedule or on demand. This allows teams to mirror their existing cloud storage libraries into an intelligent workspace without altering existing enterprise workflows. Monthly plans start with a 14-day free trial that requires a credit card. Plans are Starter at `$9.99/mo`, Business at `$49.99/mo`, and Enterprise at `$199.99/mo` on [Fastio pricing](/pricing/). Decoupling search indexing from agent consumption keeps production clusters protected while equipping agents with precise context. ## Frequently asked questions ### How do I fix Result window is too large in Elasticsearch? To fix the Result window is too large error, transition from offset-based from and size pagination to cursor-based pagination using search_after. For consistent snapshots during pagination, combine search_after with the Point in Time (PIT) API. Avoid increasing index.max_result_window in production, as doing so leads to severe JVM heap memory pressure. ### Why is index.max_result_window set to 10,000 by default? Elasticsearch sets index.max_result_window to 10,000 to protect cluster nodes from out-of-memory crashes. In distributed queries using from and size, each shard must load, score, and sort from plus size documents locally before transmitting them to the coordinator node. At deep offsets, this scatter-gather process consumes excessive JVM heap memory and triggers severe garbage collection pauses. ### How do I paginate beyond 10,000 records in Elasticsearch? To paginate beyond 10,000 records, open a Point in Time with `POST /index/_pit?keep_alive=2m`, execute an initial search request specifying a unique sort order, and pass the sort values of the last hit into the search_after field of subsequent requests. When pagination is complete, close the Point in Time using `DELETE /_pit`. ### Why is raising index.max_result_window to 50,000 discouraged? Raising index.max_result_window to 50,000 removes the cluster safety brake without making queries more efficient. Deep offset requests still force shards and coordinator nodes to allocate and sort massive document sets in memory, which quickly trips parent circuit breakers, saturates the search thread pool, and causes node disconnects during long garbage collection pauses. ### What is the difference between search_after and the Elasticsearch Scroll API? The Scroll API holds open a heavy search execution context on every participating shard, freezing resources and degrading cluster performance during high concurrency. The modern search_after parameter is stateless and lightweight, and when paired with a Point in Time (PIT), it provides consistent, stable pagination without holding open active query contexts. ### Why does search_after require a tie-breaker sort field? If documents are sorted solely by non-unique fields like timestamps, multiple documents will share identical sort values. Because Lucene internal document IDs can vary across replica shards due to background segment merges, an explicit unique tie-breaker field like _id ensures deterministic document ordering and prevents skipped or duplicate hits across pages. ### How do AI agents retrieve large corpora without hitting search pagination limits? Instead of executing deep offset pagination against operational database clusters, AI agents store document collections in intelligent workspaces. With Intelligence Mode enabled, documents are automatically indexed for hybrid search, allowing agents to query the repository via the Model Context Protocol (MCP) and retrieve concise passages with citations rather than brute-forcing deep pagination windows. ## Sources - [Elasticsearch Reference: Paginate search results](https://www.elastic.co/guide/en/elasticsearch/reference/current/paginate-search-results.html): Elasticsearch restricts the sum of from and size parameters to a maximum of 10,000 hits per query window by default. - [Claude Help Center: Upload files to Claude](https://support.claude.com/en/articles/8241126-upload-files-to-claude): Claude chat sessions accept up to 20 files at up to 500MB each, while Claude Projects accept files up to 30MB each with no fixed file-count cap, bounded only by the model's context window. ## About Fast.io Fast.io provides shared workspaces where people and AI agents work on the same files, with built-in semantic search and citation-backed chat over what they hold. Agents reach it through a remote MCP server, a REST API at https://api.fast.io/current/, and a command line client published on npm as @vividengine/fastio-cli. MCP setup is at https://mcp.fast.io/docs: Claude and most MCP clients connect to https://mcp.fast.io/mcp/tools, ChatGPT to https://mcp.fast.io/mcp/operations, and coding agents to https://mcp.fast.io/mcp/code.