How to Implement Fastio API Cursor-Based Pagination

Cursor-based pagination is an efficient method for retrieving data incrementally from a remote API or database resource using a unique reference identifier. This identifier, known as a cursor, acts as a bookmark pointing to the exact last item fetched in your previous request. When you make the next request to the server, you pass this cursor back so the system knows where to resume fetching.

This approach differs from traditional offset pagination. Instead of calculating and skipping a specific number of rows from the beginning of the dataset, the database uses the cursor value to locate the exact starting point through an indexed column. As a result, cursor pagination offers constant time complexity. It delivers flat database performance compared to linear scaling for offsets. That difference becomes noticeable when dealing with large datasets where counting and recounting rows degrades application speed.

Modern enterprise systems and high-throughput applications prefer cursors because they provide consistent, flat performance regardless of how deep you navigate into a dataset. They also prevent data duplication or missing records if new files are added or removed by other users while you paginate through the list.

Offset pagination relies on counting rows from the beginning of a dataset for every request made by the client. If you want to load page multiple, the database must scan, retrieve, and discard all the records from the first multiple pages before returning your desired data.

This process becomes expensive as data volume grows over time. Fastio workspaces can scale to millions of files, requiring an efficient retrieval method that does not bog down the server. Using a traditional offset for millions of files causes slow response times, database bottlenecks, and client-side timeouts.

Offset pagination is also fragile when data changes. If a user uploads a new file to the first page while your application is reading the second page, the items shift their positions. You might see the same file twice on your screen, or skip a file entirely. Cursors solve this problem by anchoring to a specific record rather than relying on an arbitrary numerical position.

The Fastio API uses secure, opaque cursors on storage listings so folder walks stay stable as workspaces grow. List a folder with GET https://api.fast.io/current/workspace/{workspace_id}/storage/{parent_id}/list/. Authenticate every call with Authorization: Bearer {api_key}. Keep the trailing slash. Workspace IDs are singular 19-digit profile IDs. Use parent_id=root for the workspace root.

Query parameters are sort_by (name, updated, created, or type; default name), sort_dir (asc or desc; default asc), page_size (100, 250, or 500; default 100), and cursor (the opaque string from the previous response). The JSON response carries pagination.has_more, pagination.next_cursor, and pagination.page_size.

A page can be short while pagination.has_more is true. End the loop on has_more, not on page fullness. When has_more is true, pass pagination.next_cursor as the cursor query parameter on the next request.

Fastio is designed as an intelligent workspace. Agents and humans share the same environment and capabilities. While humans might use the visual interface to browse folders, AI assistants use the MCP tools available via Streamable HTTP and SSE. Named mode includes the storage tool with action list (profile_type set to workspace or share). You can learn more about configuring these tools in the official MCP documentation.

To retrieve every page in a folder, you must build a continuous loop that fetches data until pagination.has_more is false. The core logic is simple: make an initial request, process the returned page, check has_more, and if it is true, pass pagination.next_cursor as cursor and repeat.

Here is a concise Python snippet demonstrating a reliable while loop for the Fastio storage list API.

import requests

def list_folder_pages(workspace_id, api_key, parent_id="root"):
    url = f"https://api.fast.io/current/workspace/{workspace_id}/storage/{parent_id}/list/"
    headers = {"Authorization": f"Bearer {api_key}"}
    params = {
        "sort_by": "name",
        "sort_dir": "asc",
        "page_size": 100,
    }
    pages = []

while True:
        response = requests.get(url, headers=headers, params=params)
        response.raise_for_status()
        data = response.json()
        pages.append(data)

pagination = data.get("pagination", {})
        if not pagination.get("has_more"):
            break
        params["cursor"] = pagination.get("next_cursor")

return pages

This Python script handles the pagination efficiently. It keeps updating the params dictionary with the latest cursor token until pagination.has_more is false. A short page is not a stop signal; only has_more is.

JavaScript and TypeScript developers can implement similar logic using modern asynchronous functions. This pattern is useful when building integrations or backend services that walk a workspace folder by folder.

Here is how you write the cursor loop using standard fetch inside Node.js or TypeScript environments.

async function listFolderPages(workspaceId, apiKey, parentId = 'root') {
  const baseUrl = `https://api.fast.io/current/workspace/${workspaceId}/storage/${parentId}/list/`;
  const pages = [];
  let cursor = null;
  let hasMore = true;

while (hasMore) {
    const url = new URL(baseUrl);
    url.searchParams.set('sort_by', 'name');
    url.searchParams.set('sort_dir', 'asc');
    url.searchParams.set('page_size', '100');
    if (cursor) {
      url.searchParams.set('cursor', cursor);
    }

const response = await fetch(url.toString(), {
      headers: { 'Authorization': `Bearer ${apiKey}` }
    });

if (!response.ok) {
      throw new Error(`API error: ${response.status}`);
    }

const data = await response.json();
    pages.push(data);

const pagination = data.pagination || {};
    if (pagination.has_more) {
      cursor = pagination.next_cursor;
    } else {
      hasMore = false;
    }
  }

return pages;
}

This pattern walks every page in the folder without treating a short page as the end of the list. New uploads that land earlier in the sort order do not shift you off the cursor you already hold.

Give Your AI Agents Persistent Storage

Get generous storage, 19 consolidated tools, and built-in RAG for your AI agents. Built for fast api cursor based pagination tutorial workflows.

Try the MCP Server

When dealing with millions of files, you must account for network stability and API constraints. Implementing exponential backoff and retries is essential for production code. If a page request returns HTTP 429 with error code 1671, back off until the x-ve-limit-expires header, then retry that same cursor rather than restarting the walk from the first page.

You must consider memory management as well. If you append millions of JSON objects to a single array in memory, your script will crash due to out-of-memory errors. For large datasets, you should process each page inside the while loop and stream the output directly to a local disk or another external database instead of holding everything in memory. Most list endpoints also accept output= to select a smaller response shape, for example output=terse.

After a full folder walk, pick up later activity with GET /current/events/search/ or GET /current/activity/poll/{entityId}?wait=95&lastactivity={timestamp}. To pull a remote file into the folder you are listing, use POST /current/web_upload/ with source_url, file_name, profile_id, profile_type (workspace or share), and folder_id, or the MCP upload tool with action web-import.

Fastio serves as an intelligent workspace rather than simple file storage. When you upload files into the system, the platform automatically indexes them for semantic meaning. Ripley, the built-in RAG agent, means you do not need to configure a separate vector database to make your content searchable by AI assistants.

Agents should use the MCP server. Streamable HTTP is https://mcp.fast.io/mcp. With a Bearer header in the client config, use https://mcp.fast.io/mcp/key. Create that key in Settings > Devices & Agents > API Keys, or with POST /current/user/auth/key/. Named mode includes the storage tool with action list (profile_type set to workspace or share).

{"jsonrpc":"2.0","id":1,"method":"tools/call",
 "params":{"name":"storage","arguments":{"action":"list","profile_type":"workspace","profile_id":"1234567890123456789"}}}

With the OpenClaw integration, you can install tools directly via clawhub install dbalve/fast-io with no additional configuration required. Agents can acquire file locks for concurrent multi-agent access, ensuring they never overwrite each other's changes.

You can also use advanced ownership transfer capabilities. A developer agent can create an organization, build custom workspaces, populate them using cursor-paginated folder listings, and then transfer the finished workspace to a human client while retaining limited administrative access.