# How to Paginate SharePoint REST API: skiptoken, next Links, and Agent Retrieval SharePoint REST API pagination retrieves large result sets from lists and libraries in sequential batches using OData __next continuation URLs and skiptoken markers. While endpoints return 100 items by default, unindexed queries trigger the 5,000-item List View Threshold. Traversing raw paginated endpoints across multi-page libraries also consumes significant agent context tokens. Implementing server-driven cursors or syncing to indexed workspaces keeps document retrieval reliable. Source: https://fast.io/resources/sharepoint-rest-api-pagination/ Author: [Tom Langridge](https://fast.io/authors/tom-langridge/) Last reviewed: 2026-10-11 ## How SharePoint REST API Pagination and Continuation Links Function SharePoint REST API pagination is the process of retrieving large result sets from SharePoint lists and libraries in sequential batches using OData __next continuation URLs and skiptoken markers. According to Microsoft Learn documentation on SharePoint Online limits, a list can have up to 30 million items and a library can have up to 30 million files and folders. Despite that high storage ceiling, querying items in production environments is constrained by strict database thresholds and network payload boundaries. When an application queries a SharePoint list endpoint such as `/_api/web/lists/getbytitle('Documents')/items`, the service does not stream the entire collection in a single response. By default, SharePoint REST endpoints return the first 100 items unless an explicit `$top` query parameter overrides that limit. If a caller requests a page size larger than the server allows, or if the target library holds thousands of records, the client must traverse the collection sequentially. Many developers attempting to page through SharePoint lists instinctively reach for the OData `$skip` query parameter, which is standard in OData v4 and third-party REST services. In SharePoint REST API implementations, `$skip` does not function when querying list items. Microsoft documentation confirms that the OData `$skip` query parameter does not work when querying list items. Sending a request like `items?$skip=100&$top=50` either produces an HTTP 400 error or is ignored by the SharePoint query engine, returning the first 50 items repeatedly. To retrieve subsequent pages, SharePoint uses server-driven cursor pagination. Instead of calculating offsets, the SharePoint web front end computes a stateful continuation token that points directly to the next item marker in the database index. The server injects this token into an OData continuation URL named `__next` (in verbose OData format) or `@odata.nextLink` (in minimal metadata format). The client application must extract this URL and request it verbatim to fetch the next slice of data. ### Anatomy of the OData Continuation URL The payload format returned by SharePoint depends on the HTTP `Accept` header supplied in the request. The legacy SharePoint 2013 REST service defaults to verbose JSON, whereas modern applications typically request minimal metadata: ```http GET https://{tenant}.sharepoint.com/sites/{site}/_api/web/lists/getbytitle('Documents')/items?$top=100 Authorization: Bearer {token} Accept: application/json;odata=verbose ``` Under `odata=verbose`, the response wraps the list records inside `d.results` and includes continuation metadata under `d.__next`: ```json { "d": { "results": [ { "__metadata": { "id": "Web/Lists(guid'...')/Items(1)", "uri": "https://tenant.sharepoint.com/sites/site/_api/Web/Lists(guid'...')/Items(1)", "type": "SP.Data.DocumentsItem" }, "Id": 1, "Title": "Quarterly_Report_Q1.pdf" } ], "__next": "https://tenant.sharepoint.com/sites/site/_api/web/lists/getbytitle('Documents')/items?%24skiptoken=Paged%3dTRUE%26p_ID%3d100&%24top=100" } } ``` If your client requests `Accept: application/json;odata=minimalmetadata` or `odata=nometadata`, the payload shape matches OData v4 conventions. The array of records sits inside the top-level `value` property, and the continuation pointer appears as `@odata.nextLink` or `odata.nextLink`: ```json { "value": [ { "Id": 1, "Title": "Quarterly_Report_Q1.pdf" } ], "@odata.nextLink": "https://tenant.sharepoint.com/sites/site/_api/web/lists/getbytitle('Documents')/items?%24skiptoken=Paged%3dTRUE%26p_ID%3d100&%24top=100" } ``` A production ingestion pipeline must inspect both potential keys. If neither `__next` nor `@odata.nextLink` exists in the payload, the client has reached the final page of the collection. ## How to Deconstruct the skiptoken Query Parameter and Avoid Encoding Bugs The continuation link returned by SharePoint encodes pagination state into the `$skiptoken` parameter. While `$skiptoken` looks like an arbitrary string, it is an encoded set of internal query markers derived from the underlying SQL Server index. In its simplest form, when querying items without custom sorting, `$skiptoken` contains two key directives: ```http ?$skiptoken=Paged=TRUE%26p_ID=100&$top=100 ``` The component `Paged=TRUE` instructs the SharePoint query processor to switch from offset evaluation to cursor evaluation. The component `p_ID=100` defines the boundary condition: fetch records whose internal primary key `ID` is strictly greater than 100. A pervasive bug in SharePoint API integrations involves improper URL encoding of the `$skiptoken` value. Notice the `%26` character separating `Paged=TRUE` and `p_ID=100`. In the HTTP query string standard, an unencoded ampersand (`&`) delineates distinct query parameters. If client code reconstructs or parses this token and outputs an unencoded ampersand: ```http ?$skiptoken=Paged=TRUE&p_ID=100&$top=100 ``` The web server parses `p_ID=100` as a standalone, unrecognized query argument rather than part of `$skiptoken`. The SharePoint engine evaluates only `$skiptoken=Paged=TRUE`. Because no item cursor is provided, the query defaults back to item index zero. The request succeeds with HTTP 200, but returns the first page again. Automated scripts that rebuild URLs without encoding `%26` enter an infinite loop, fetching page one thousands of times until rate limits trigger. When your query includes an `$orderby` clause on a field other than `ID` (for example, ordering by `Modified` or `Title`), SharePoint appends secondary cursor attributes: ```http ?$skiptoken=Paged=TRUE%26p_Title=Project_Omega_Specs.docx%26p_ID=450&$top=100&$orderby=Title ``` In multi-column cursors, the query engine evaluates records where the `Title` is greater than or equal to `Project_Omega_Specs.docx`, using `p_ID` to break ties among identical values. Modifying or truncating any portion of this compound token causes queries to skip records or fail with invalid cursor errors. ### Step-by-Step Python Implementation for Paginating Results The most dependable way to paginate SharePoint REST results is to treat the continuation URL as an opaque pointer. Rather than constructing `$skiptoken` strings manually, your code should execute the initial request, harvest records, and follow the exact URL supplied by the server until the continuation key is null. The following Python script implements this pattern using the `requests` library. It supports both verbose and minimal metadata formats, manages authentication headers, and collects all items cleanly: ```python import time from typing import Optional import requests def fetch_all_sharepoint_items( site_url: str, list_title: str, access_token: str, page_size: int = 100 ) -> list[dict]: # Establish base request headers supporting both OData variants headers = { "Authorization": f"Bearer {access_token}", "Accept": "application/json;odata=verbose", "Content-Type": "application/json" } # Initial endpoint targeting list items with an explicit page size endpoint = ( f"{site_url.rstrip('/')}/_api/web/lists/" f"getbytitle('{list_title}')/items?$top={page_size}" ) all_items: list[dict] = [] current_url: Optional[str] = endpoint page_number = 1 while current_url: response = requests.get(current_url, headers=headers, timeout=30) # Handle transient throttling backoff if rate limits are encountered if response.status_code == 429: retry_after = int(response.headers.get("Retry-After", 10)) time.sleep(retry_after) continue response.raise_for_status() payload = response.json() # Extract items based on active OData metadata level if "d" in payload and isinstance(payload["d"], dict): # Verbose OData format records = payload["d"].get("results", []) next_link = payload["d"].get("__next") else: # Minimal or no-metadata OData format records = payload.get("value", []) next_link = payload.get("@odata.nextLink") or payload.get("odata.nextLink") all_items.extend(records) page_number += 1 # Terminate traversal when no next link is returned current_url = next_link return all_items ``` This implementation prevents parameter encoding bugs because it re-submits the exact, server-validated `__next` string. It also avoids memory exhaustion by processing records per page as they arrive. ## Why Unindexed Queries Fail: Overcoming the 5,000 Item List View Threshold The primary obstacle when paginating SharePoint collections is the List View Threshold. In SharePoint Online, Microsoft enforces an unalterable limit of 5,000 items on database queries. This restriction exists to safeguard multi-tenant SQL Server infrastructure from table locks caused by full table scans. Developers often assume that adding `$top=100` protects queries from threshold exceptions. If a list contains 12,000 items and you issue an unindexed filter, the query fails: ```http GET /_api/web/lists/getbytitle('Contracts')/items?$filter=Status eq 'Active'&$top=100 ``` When evaluating this request, SQL Server cannot locate matching records without inspecting every row in the list. The database engine begins a full scan, hits the 5,000-item evaluation ceiling, and halts execution before applying the `$top=100` slice. The API responds with an HTTP 500 error containing `SPQueryThrottledException`: > The attempted operation is prohibited because it exceeds the list view threshold. To make pagination functional on lists exceeding 5,000 items, every column referenced in a `$filter` or `$orderby` clause must have an indexed column created in list settings. When an index exists, SQL Server executes a selective seek rather than a scan, keeping the processed item count below 5,000 and allowing `$skiptoken` pagination to advance normally. ### Recursive Folder Paging and CAML Query State When data pipelines require complex multi-field filtering, folder hierarchy traversal, or operations across large document libraries where standard OData queries fail, developers turn to CAML queries via the `GetItems` endpoint: ```http POST https://{tenant}.sharepoint.com/sites/{site}/_api/web/lists/getbytitle('Documents')/GetItems Authorization: Bearer {token} Accept: application/json;odata=verbose Content-Type: application/json;odata=verbose ``` CAML queries do not use `$skiptoken` in the URL query string. Instead, pagination state is passed inside the JSON request body using `ListItemCollectionPosition`: ```json { "query": { "__metadata": { "type": "SP.CamlQuery" }, "ViewXml": "0200", "ListItemCollectionPosition": { "PagingInfo": "Paged=TRUE&p_ID=200" } } } ``` The `Scope='RecursiveAll'` attribute tells SharePoint to recurse through all folders and subfolders, ignoring directory boundaries to return only file objects (`FSObjType=0`). To paginate CAML queries, the client inspects the response for `d.ListItemCollectionPosition`. If additional items exist, SharePoint returns a `PagingInfo` string: ```json { "d": { "results": [ ... ], "ListItemCollectionPosition": { "PagingInfo": "Paged=TRUE&p_ID=400" } } } ``` The client extracts `d.ListItemCollectionPosition.PagingInfo` and includes it directly inside the `ListItemCollectionPosition` object of the subsequent POST request. When `ListItemCollectionPosition` returns null or is omitted, all matching records across all nested folders have been retrieved. Unlike URL-based `$skiptoken`, the `PagingInfo` in CAML requests uses standard ampersands inside the JSON string literal. ## When Rate Limits Hit: Managing Throttling, HTTP 429 Backoff, and Agent Costs Automated scripts and data ingestion crawlers that loop through paginated endpoints frequently encounter SharePoint Online throttling. Microsoft implements dynamic, resource-based rate limits to protect tenant performance. Throttling is calculated across CPU usage, database concurrent threads, and request volume over rolling time windows. When a client makes rapid, sequential requests across paginated endpoints, SharePoint returns an `HTTP 429 Too Many Requests` or `HTTP 503 Service Unavailable` response. The response includes an advisory HTTP header: ```http HTTP/1.1 429 Too Many Requests Retry-After: 15 X-SharePointHealthScore: 8 ``` The `Retry-After` header specifies the exact number of seconds the calling process must pause before re-submitting the request. If a client ignores this header and retries immediately, SharePoint extends the block period, escalating from brief pauses to hours-long tenant blocks. Resilient pagination loops must inspect response headers for `Retry-After`, implement exponential backoff with randomized jitter, and supply a unique `User-Agent` identifying the application. Using generic HTTP client signatures increases the likelihood of tenant-level throttling. ### Why Direct REST Pagination Degrades Autonomous Agents While paginating through REST endpoints works for traditional background ETL jobs, it creates severe performance bottlenecks for autonomous AI agents and large language model workflows. When an autonomous agent needs to locate a specific clause or document in a 5,000-file SharePoint library, naive tool execution instructs the agent to page through the list. In OData verbose mode, each page of 100 items contains tens of kilobytes of repetitive metadata wrappers, including GUIDs, type descriptors, edit URLs, and deferred navigation URIs. Consider the token overhead when an agent traverses a medium-sized document library: | Traversal Stage | Raw Payload Size | Estimated Context Tokens | Operational Cost | | :--- | :--- | :--- | :--- | | **Single Page (100 items)** | ~85 KB verbose JSON | ~21,000 tokens | Tolerable for a single tool call | | **5 Pages (500 items)** | ~425 KB verbose JSON | ~105,000 tokens | Fills half of a standard context window | | **10 Pages (1,000 items)** | ~850 KB verbose JSON | ~210,000 tokens | Exceeds most agent memory budgets | | **Full Library (5,000 items)** | ~4.2 MB verbose JSON | ~1,050,000 tokens | Impossible in a single context window | Traversing paginated SharePoint REST endpoints across 10+ pages burns thousands of agent context tokens. The agent spends its limited attention span and compute budget parsing JSON formatting, parent pointers, and schema descriptors instead of analyzing document content. Furthermore, each page requires a distinct network round trip. If SharePoint throttles the connection on page 7, the agent session stalls, loses execution state, or errors out. Autonomous agents should not act as raw database cursors; they require pre-indexed search surfaces. ## How to Modernize Agent Retrieval with Intelligent Workspace Synchronization Modern agentic systems solve the pagination bottleneck by decoupling file synchronization from interactive agent queries. In this pattern, files stored in enterprise repositories like SharePoint, OneDrive, Dropbox, or Box are mirrored into an intelligent workspace where they are indexed on arrival. Many organizations already manage critical files in SharePoint document libraries. Connecting an AI agent directly to native SharePoint REST APIs forces the agent to deal with Entra ID authentication renewals, CAML pagination loops, and List View Thresholds. Fast.io provides an alternative architecture. Through Cloud Sync, SharePoint document libraries sync into a Fast.io workspace through the OneDrive connector. Sync operations can run one-way or two-way, on a schedule or on demand. Cloud Sync also supports Google Drive, Box, and Dropbox. Once documents land in a Fast.io workspace, Intelligence Mode auto-indexes the files for hybrid search combining full-text indexing, vector semantic search, and metadata value matching. Autonomous agents do not need to paginate through thousands of raw items or manage `$skiptoken` strings. Instead, the agent queries the workspace through the Model Context Protocol (MCP). Fast.io hosts a remote MCP server over Streamable HTTP at `https://mcp.fast.io/mcp/tools` for Claude and general clients, `https://mcp.fast.io/mcp/operations` for ChatGPT and Codex custom connectors, and `https://mcp.fast.io/mcp/code` for coding agents like Claude Code, Cursor, and Cline. Rather than pulling multi-page JSON payloads, an agent sends a semantic query and receives only the relevant extracted passages, accompanied by exact file citations. For teams handling structured extraction from forms, contracts, and spreadsheets, Fast.io provides [Metadata Views](/product/document-data-extraction/). Metadata Views turn documents into live, queryable databases by extracting typed fields (Text, Integer, Decimal, Boolean, URL, JSON, and Date & Time) without requiring custom OCR rules. Agents can create Views and query structured data directly via MCP. ### MCP Semantic Search vs Sequential REST Crawling By offloading search and extraction to an intelligent workspace, teams eliminate the latency and token waste of client-side REST pagination. The performance advantages of this pattern were evaluated in third-party connector testing. The connector comparison is published at https://fast.io/benchmarks/ and that page is the only place its numbers live. Fastio was measured the fastest and the lowest cost of the providers tested. By shifting document retrieval from sequential API pagination to remote workspace search, agent workflows complete their tasks with minimal network latency and zero context token waste. Getting started with Fast.io is straightforward. Monthly plans start with a 14-day free trial that requires a credit card. Plans are Starter at `$9.99/mo`, Business at `$49.99/mo`, and Enterprise at `$199.99/mo`. Annual billing options are also available. Credits meter AI operations. Storage and seat allocations are included with each plan. Teams can configure remote agent storage by reviewing [Fast.io storage for agents](/storage-for-agents/), checking [Fast.io pricing](/pricing/), and exploring [Fast.io intelligent workspaces](/product/workspaces/). ## Architectural Checklist for SharePoint API Data Pipelines When designing production software that interacts with SharePoint data, selecting the appropriate integration pattern determines system reliability: 1. **Always Use Server-Generated Next Links**: When consuming OData endpoints, treat the `__next` or `@odata.nextLink` property as an opaque URI. Never attempt to manually construct or decompose the `$skiptoken` query parameter unless explicitly required by legacy custom protocols. 2. **Strictly Encode `%26` in Token Cursors**: If your application must assemble `$skiptoken` parameters, ensure internal delimiters use `%26`. A raw ampersand causes the server to ignore item cursors and loop indefinitely on the initial page. 3. **Index Filter and Sort Columns**: Proactively add indexes to all list columns used in `$filter` or `$orderby` clauses. Unindexed queries on lists exceeding 5,000 items fail immediately due to the List View Threshold. 4. **Implement Throttling Handlers**: Inspect response status codes for HTTP 429 and 503. Always parse the `Retry-After` header and execute exponential backoff with jitter to maintain tenant compliance. 5. **Decouple Agent Ingestion from Direct API Traversal**: Avoid letting autonomous AI agents execute raw pagination loops. Sync SharePoint document libraries into indexed workspace storage, enabling agents to retrieve verified information via semantic search rather than burning context tokens on raw JSON metadata. ## Frequently asked questions ### How do I paginate results in SharePoint REST API? Paginate SharePoint REST API results by appending the $top parameter to your initial query to specify batch size, then inspecting the response for a __next continuation URL. Extract that server-provided URL and submit it as your next GET request. Continue this loop until the response omits the __next link. ### What is skiptoken in SharePoint REST API? A skiptoken is a server-generated cursor parameter embedded inside OData __next continuation links. It indicates the exact boundary position of the last retrieved item using internal database identifiers, such as Paged=TRUE and p_ID=100. It directs the query processor to resume retrieval immediately after that record. ### How do I overcome the 5000 item limit in SharePoint REST API? Overcome the 5,000-item List View Threshold by indexing the columns used in your filter and sort operations. When columns are indexed, SharePoint performs an index seek rather than a full table scan. For complex folder structures or advanced queries, use CAML queries via the GetItems endpoint with ListItemCollectionPosition pagination. ### Why does the OData $skip parameter fail on SharePoint lists? SharePoint list item endpoints do not support the OData $skip parameter. Offset-based skipping requires expensive SQL row-skipping operations that violate performance safeguards in multi-tenant environments. SharePoint enforces server-driven cursor pagination using $skiptoken instead. ### How do I paginate CAML queries using the SharePoint REST API? To paginate CAML queries, submit a POST request to the GetItems endpoint with RowLimit configured in your ViewXml. In the response, read the d.ListItemCollectionPosition.PagingInfo string and include it in the ListItemCollectionPosition object of your subsequent POST request until the position property returns null. ### How does SharePoint REST API pagination impact AI agent token usage? Directly traversing paginated REST endpoints forces an AI agent to ingest verbose JSON metadata, schema definitions, and navigation links into its context window for every page. Traversing 10 or more pages burns thousands of tokens on raw formatting. Pre-indexing documents in a workspace allows agents to retrieve targeted passages without pagination overhead. ## Sources - [Microsoft Learn: SharePoint Online limits](https://learn.microsoft.com/en-us/office365/servicedescriptions/sharepoint-online-service-description/sharepoint-online-limits): A SharePoint list or document library can store up to 30 million items and files, but large operations are constrained by query-level thresholds. - [Microsoft Learn: Working with lists and list items with REST](https://learn.microsoft.com/en-us/sharepoint/dev/sp-add-ins/working-with-lists-and-list-items-with-rest): The OData skip query option is unsupported for retrieving SharePoint list items and fails on large item collections. ## About Fast.io Fast.io provides shared workspaces where people and AI agents work on the same files, with built-in semantic search and citation-backed chat over what they hold. Agents reach it through a remote MCP server, a REST API at https://api.fast.io/current/, and a command line client published on npm as @vividengine/fastio-cli. MCP setup is at https://mcp.fast.io/docs: Claude and most MCP clients connect to https://mcp.fast.io/mcp/tools, ChatGPT to https://mcp.fast.io/mcp/operations, and coding agents to https://mcp.fast.io/mcp/code.