How to Paginate SharePoint REST API: skiptoken, next Links, and Agent Retrieval

SharePoint REST API pagination is the process of retrieving large result sets from SharePoint lists and libraries in sequential batches using OData __next continuation URLs and skiptoken markers. According to Microsoft Learn documentation on SharePoint Online limits, a list can have up to 30 million items and a library can have up to 30 million files and folders. Despite that high storage ceiling, querying items in production environments is constrained by strict database thresholds and network payload boundaries.

When an application queries a SharePoint list endpoint such as /_api/web/lists/getbytitle('Documents')/items, the service does not stream the entire collection in a single response. By default, SharePoint REST endpoints return the first 100 items unless an explicit $top query parameter overrides that limit. If a caller requests a page size larger than the server allows, or if the target library holds thousands of records, the client must traverse the collection sequentially.

Many developers attempting to page through SharePoint lists instinctively reach for the OData $skip query parameter, which is standard in OData v4 and third-party REST services. In SharePoint REST API implementations, $skip does not function when querying list items. Microsoft documentation confirms that the OData $skip query parameter does not work when querying list items. Sending a request like items?$skip=100&$top=50 either produces an HTTP 400 error or is ignored by the SharePoint query engine, returning the first 50 items repeatedly.

To retrieve subsequent pages, SharePoint uses server-driven cursor pagination. Instead of calculating offsets, the SharePoint web front end computes a stateful continuation token that points directly to the next item marker in the database index. The server injects this token into an OData continuation URL named __next (in verbose OData format) or @odata.nextLink (in minimal metadata format). The client application must extract this URL and request it verbatim to fetch the next slice of data.

GET https://{tenant}.sharepoint.com/sites/{site}/_api/web/lists/getbytitle('Documents')/items?$top=100
Authorization: Bearer {token}
Accept: application/json;odata=verbose

{
  "d": {
    "results": [
      {
        "__metadata": {
          "id": "Web/Lists(guid'...')/Items(1)",
          "uri": "https://tenant.sharepoint.com/sites/site/_api/Web/Lists(guid'...')/Items(1)",
          "type": "SP.Data.DocumentsItem"
        },
        "Id": 1,
        "Title": "Quarterly_Report_Q1.pdf"
      }
    ],
    "__next": "https://tenant.sharepoint.com/sites/site/_api/web/lists/getbytitle('Documents')/items?%24skiptoken=Paged%3dTRUE%26p_ID%3d100&%24top=100"
  }
}

{
  "value": [
    {
      "Id": 1,
      "Title": "Quarterly_Report_Q1.pdf"
    }
  ],
  "@odata.nextLink": "https://tenant.sharepoint.com/sites/site/_api/web/lists/getbytitle('Documents')/items?%24skiptoken=Paged%3dTRUE%26p_ID%3d100&%24top=100"
}

• SharePoint REST API: Endpoints, Authentication, and Agent RetrievalThe SharePoint REST API provides direct HTTP access to SharePoint document libraries and metadata using OData...
• How to Debug AI Agent File Access IssuesFile access errors are a leading cause of agent failures. These turn automated workflows into broken pipelines. This...
• How to Create Folders with SharePoint API: REST, Graph, and Agent WorkspacesCreating folders across SharePoint environments requires choosing between legacy REST endpoints and modern Microsoft...
• How to List Files with SharePoint API: REST, Graph, and Agent WorkspacesListing files across SharePoint environments requires choosing between legacy REST endpoints and Microsoft Graph drive...
• Azure AI Search SharePoint: How to Index SharePoint Files for Agentic RetrievalIndexing Microsoft SharePoint document libraries with Azure AI Search requires configuring Entra ID service principals,...
• How to List Files with the Dropbox API: Pagination, Cursors, and Agent WorkspacesListing files through the Dropbox API requires managing cursor-based pagination across the /files/list_folder and...

More on this subject: Agent File and Document Workflows (298 guides)

The continuation link returned by SharePoint encodes pagination state into the $skiptoken parameter. While $skiptoken looks like an arbitrary string, it is an encoded set of internal query markers derived from the underlying SQL Server index.

In its simplest form, when querying items without custom sorting, $skiptoken contains two key directives:

?$skiptoken=Paged=TRUE%26p_ID=100&$top=100

The component Paged=TRUE instructs the SharePoint query processor to switch from offset evaluation to cursor evaluation. The component p_ID=100 defines the boundary condition: fetch records whose internal primary key ID is strictly greater than 100.

A pervasive bug in SharePoint API integrations involves improper URL encoding of the $skiptoken value. Notice the %26 character separating Paged=TRUE and p_ID=100. In the HTTP query string standard, an unencoded ampersand (&) delineates distinct query parameters. If client code reconstructs or parses this token and outputs an unencoded ampersand:

?$skiptoken=Paged=TRUE&p_ID=100&$top=100

The web server parses p_ID=100 as a standalone, unrecognized query argument rather than part of $skiptoken. The SharePoint engine evaluates only $skiptoken=Paged=TRUE. Because no item cursor is provided, the query defaults back to item index zero. The request succeeds with HTTP 200, but returns the first page again. Automated scripts that rebuild URLs without encoding %26 enter an infinite loop, fetching page one thousands of times until rate limits trigger.

When your query includes an $orderby clause on a field other than ID (for example, ordering by Modified or Title), SharePoint appends secondary cursor attributes:

?$skiptoken=Paged=TRUE%26p_Title=Project_Omega_Specs.docx%26p_ID=450&$top=100&$orderby=Title

In multi-column cursors, the query engine evaluates records where the Title is greater than or equal to Project_Omega_Specs.docx, using p_ID to break ties among identical values. Modifying or truncating any portion of this compound token causes queries to skip records or fail with invalid cursor errors.

import time
from typing import Optional
import requests

def fetch_all_sharepoint_items(
    site_url: str,
    list_title: str,
    access_token: str,
    page_size: int = 100
) -> list[dict]:
    # Establish base request headers supporting both OData variants
    headers = {
        "Authorization": f"Bearer {access_token}",
        "Accept": "application/json;odata=verbose",
        "Content-Type": "application/json"
    }
    
    # Initial endpoint targeting list items with an explicit page size
    endpoint = (
        f"{site_url.rstrip('/')}/_api/web/lists/"
        f"getbytitle('{list_title}')/items?$top={page_size}"
    )
    
    all_items: list[dict] = []
    current_url: Optional[str] = endpoint
    page_number = 1
    
    while current_url:
        response = requests.get(current_url, headers=headers, timeout=30)
        
        # Handle transient throttling backoff if rate limits are encountered
        if response.status_code == 429:
            retry_after = int(response.headers.get("Retry-After", 10))
            time.sleep(retry_after)
            continue
            
        response.raise_for_status()
        payload = response.json()
        
        # Extract items based on active OData metadata level
        if "d" in payload and isinstance(payload["d"], dict):
            # Verbose OData format
            records = payload["d"].get("results", [])
            next_link = payload["d"].get("__next")
        else:
            # Minimal or no-metadata OData format
            records = payload.get("value", [])
            next_link = payload.get("@odata.nextLink") or payload.get("odata.nextLink")
            
        all_items.extend(records)
        page_number += 1
        
        # Terminate traversal when no next link is returned
        current_url = next_link
        
    return all_items

The primary obstacle when paginating SharePoint collections is the List View Threshold. In SharePoint Online, Microsoft enforces an unalterable limit of 5,000 items on database queries. This restriction exists to safeguard multi-tenant SQL Server infrastructure from table locks caused by full table scans.

Developers often assume that adding $top=100 protects queries from threshold exceptions. If a list contains 12,000 items and you issue an unindexed filter, the query fails:

GET /_api/web/lists/getbytitle('Contracts')/items?$filter=Status eq 'Active'&$top=100

When evaluating this request, SQL Server cannot locate matching records without inspecting every row in the list. The database engine begins a full scan, hits the 5,000-item evaluation ceiling, and halts execution before applying the $top=100 slice. The API responds with an HTTP 500 error containing SPQueryThrottledException:

The attempted operation is prohibited because it exceeds the list view threshold.

To make pagination functional on lists exceeding 5,000 items, every column referenced in a $filter or $orderby clause must have an indexed column created in list settings. When an index exists, SQL Server executes a selective seek rather than a scan, keeping the processed item count below 5,000 and allowing $skiptoken pagination to advance normally.

POST https://{tenant}.sharepoint.com/sites/{site}/_api/web/lists/getbytitle('Documents')/GetItems
Authorization: Bearer {token}
Accept: application/json;odata=verbose
Content-Type: application/json;odata=verbose

{
  "query": {
    "__metadata": { "type": "SP.CamlQuery" },
    "ViewXml": "<View Scope='RecursiveAll'><Query><Where><Eq><FieldRef Name='FSObjType'/><Value Type='Integer'>0</Value></Where></Query><RowLimit>200</RowLimit></View>",
    "ListItemCollectionPosition": {
      "PagingInfo": "Paged=TRUE&p_ID=200"
    }
  }
}

{
  "d": {
    "results": [ ... ],
    "ListItemCollectionPosition": {
      "PagingInfo": "Paged=TRUE&p_ID=400"
    }
  }
}

Eliminate REST Pagination Latency in Agent Workspaces

Connect your SharePoint libraries to a shared workspace with automated indexing, hybrid search, and remote MCP tooling. Starts with a 14-day free trial.

Start 14-Day Trial

Automated scripts and data ingestion crawlers that loop through paginated endpoints frequently encounter SharePoint Online throttling. Microsoft implements dynamic, resource-based rate limits to protect tenant performance. Throttling is calculated across CPU usage, database concurrent threads, and request volume over rolling time windows.

When a client makes rapid, sequential requests across paginated endpoints, SharePoint returns an HTTP 429 Too Many Requests or HTTP 503 Service Unavailable response. The response includes an advisory HTTP header:

HTTP/1.1 429 Too Many Requests
Retry-After: 15
X-SharePointHealthScore: 8

The Retry-After header specifies the exact number of seconds the calling process must pause before re-submitting the request. If a client ignores this header and retries immediately, SharePoint extends the block period, escalating from brief pauses to hours-long tenant blocks.

Resilient pagination loops must inspect response headers for Retry-After, implement exponential backoff with randomized jitter, and supply a unique User-Agent identifying the application. Using generic HTTP client signatures increases the likelihood of tenant-level throttling.

Modern agentic systems solve the pagination bottleneck by decoupling file synchronization from interactive agent queries. In this pattern, files stored in enterprise repositories like SharePoint, OneDrive, Dropbox, or Box are mirrored into an intelligent workspace where they are indexed on arrival.

Many organizations already manage critical files in SharePoint document libraries. Connecting an AI agent directly to native SharePoint REST APIs forces the agent to deal with Entra ID authentication renewals, CAML pagination loops, and List View Thresholds.

Fast.io provides an alternative architecture. Through Cloud Sync, SharePoint document libraries sync into a Fast.io workspace through the OneDrive connector. Sync operations can run one-way or two-way, on a schedule or on demand. Cloud Sync also supports Google Drive, Box, and Dropbox.

Once documents land in a Fast.io workspace, Intelligence Mode auto-indexes the files for hybrid search combining full-text indexing, vector semantic search, and metadata value matching. Autonomous agents do not need to paginate through thousands of raw items or manage $skiptoken strings. Instead, the agent queries the workspace through the Model Context Protocol (MCP).

Fast.io hosts a remote MCP server over Streamable HTTP at https://mcp.fast.io/mcp/tools for Claude and general clients, https://mcp.fast.io/mcp/operations for ChatGPT and Codex custom connectors, and https://mcp.fast.io/mcp/code for coding agents like Claude Code, Cursor, and Cline. Rather than pulling multi-page JSON payloads, an agent sends a semantic query and receives only the relevant extracted passages, accompanied by exact file citations.

For teams handling structured extraction from forms, contracts, and spreadsheets, Fast.io provides Metadata Views. Metadata Views turn documents into live, queryable databases by extracting typed fields (Text, Integer, Decimal, Boolean, URL, JSON, and Date & Time) without requiring custom OCR rules. Agents can create Views and query structured data directly via MCP.

When designing production software that interacts with SharePoint data, selecting the appropriate integration pattern determines system reliability:

1. Always Use Server-Generated Next Links: When consuming OData endpoints, treat the __next or @odata.nextLink property as an opaque URI. Never attempt to manually construct or decompose the $skiptoken query parameter unless explicitly required by legacy custom protocols.
2. Strictly Encode %26 in Token Cursors: If your application must assemble $skiptoken parameters, ensure internal delimiters use %26. A raw ampersand causes the server to ignore item cursors and loop indefinitely on the initial page.
3. Index Filter and Sort Columns: Proactively add indexes to all list columns used in $filter or $orderby clauses. Unindexed queries on lists exceeding 5,000 items fail immediately due to the List View Threshold.
4. Implement Throttling Handlers: Inspect response status codes for HTTP 429 and 503. Always parse the Retry-After header and execute exponential backoff with jitter to maintain tenant compliance.
5. Decouple Agent Ingestion from Direct API Traversal: Avoid letting autonomous AI agents execute raw pagination loops. Sync SharePoint document libraries into indexed workspace storage, enabling agents to retrieve verified information via semantic search rather than burning context tokens on raw JSON metadata.