How to Implement Fast.io API Cursor-Based Pagination

What is Cursor-Based Pagination?: fast api cursor based pagination tutorial

Cursor-based pagination is an efficient method for retrieving data incrementally from a remote API or database resource using a unique reference identifier. This identifier, known as a cursor, acts as a bookmark pointing to the exact last item fetched in your previous request. When you make the next request to the server, you pass this cursor back so the system knows where to resume fetching.

This approach differs from traditional offset pagination. Instead of calculating and skipping a specific number of rows from the beginning of the dataset, the database uses the cursor value to locate the exact starting point through an indexed column. As a result, cursor pagination offers constant time complexity. It delivers flat database performance compared to linear scaling for offsets. That difference becomes noticeable when dealing with large datasets where counting and recounting rows degrades application speed.

Modern enterprise systems and high-throughput applications prefer cursors because they provide consistent, flat performance regardless of how deep you navigate into a dataset. They also prevent data duplication or missing records if new files are added or removed by other users while you paginate through the list.

Why Offset Pagination Fails at Scale

Offset pagination relies on counting rows from the beginning of a dataset for every request made by the client. If you want to load page multiple, the database must scan, retrieve, and discard all the records from the first multiple pages before returning your desired data.

This process becomes expensive as data volume grows over time. Fast.io workspaces can scale to millions of files, requiring an efficient retrieval method that does not bog down the server. Using a traditional offset for millions of files causes slow response times, database bottlenecks, and client-side timeouts.

Offset pagination is also fragile when data changes. If a user uploads a new file to the first page while your application is reading the second page, the items shift their positions. You might see the same file twice on your screen, or skip a file entirely. Cursors solve this problem by anchoring to a specific record rather than relying on an arbitrary numerical position.

How the Fast.io API Implements Cursors

The Fast.io API uses secure, opaque cursors for its list endpoints to ensure maximum performance and stability. When you call an endpoint like the list files API, the JSON response includes a next_cursor field alongside your array of file data.

If next_cursor is present and not null, it means there is more data available for retrieval. You take this string value and include it as a query parameter in your next HTTPS request. You also specify a limit parameter to control how many items the server returns per page, allowing you to balance memory usage with network latency.

Fast.io is designed as an intelligent workspace. Agents and humans share the same environment and capabilities. While humans might use the visual interface to browse folders, AI assistants use the multiple MCP tools available via Streamable HTTP and SSE. Every capability you see in the UI has a corresponding API and MCP tool. This means any pagination logic you build for custom integrations applies directly to your autonomous agent workflows as well. You can learn more about configuring these tools in the official MCP documentation.

Building the Pagination Loop in Python

To retrieve all files in a large workspace, you must build a continuous loop that fetches data until the cursor is exhausted. The core logic is simple: make an initial request, process the returned items, check for a cursor, and if a cursor is returned, repeat the process.

Here is a concise Python snippet demonstrating a reliable while loop for the Fast.io API.

import requests

def list_all_files(workspace_id, api_token):
    url = f"https://api.fast.io/v1/workspaces/{workspace_id}/files"
    headers = {"Authorization": f"Bearer {api_token}"}
    params = {"limit": 100}
    all_files = []
    
    while True:
        response = requests.get(url, headers=headers, params=params)
        response.raise_for_status()
        data = response.json()
        
        ### Safely append the fetched items
        all_files.extend(data.get("items", []))
        
        next_cursor = data.get("next_cursor")
        if not next_cursor:
            ### The cursor is empty, we have reached the end
            break
            
        ### Update the parameters for the next page
        params["cursor"] = next_cursor
        
    return all_files

This Python script handles the pagination efficiently. It keeps updating the params dictionary with the latest cursor token until the API indicates there are no more pages to fetch.

Implementing the Loop in Node.js

JavaScript and TypeScript developers can implement similar logic using modern asynchronous functions. This pattern is useful when building integrations, backend services, or webhooks that need to react to workspace changes.

Here is how you write the cursor loop using standard fetch inside Node.js or TypeScript environments.

async function listAllFiles(workspaceId, apiToken) {
  const baseUrl = `https://api.fast.io/v1/workspaces/${workspaceId}/files`;
  let allFiles = [];
  let cursor = null;
  let hasMore = true;

while (hasMore) {
    const url = new URL(baseUrl);
    url.searchParams.append('limit', '100');
    
    if (cursor) {
      url.searchParams.append('cursor', cursor);
    }

const response = await fetch(url.toString(), {
      headers: { 'Authorization': `Bearer ${apiToken}` }
    });
    
    if (!response.ok) {
      throw new Error(`API error: ${response.status}`);
    }
    
    const data = await response.json();
    allFiles.push(...data.items);
    
    if (data.next_cursor) {
      cursor = data.next_cursor;
    } else {
      hasMore = false;
    }
  }

return allFiles;
}

This pattern guarantees you capture every file without skipping any records, even if the target workspace receives hundreds of new uploads concurrently.

Best Practices for Massive Workspaces

When dealing with millions of files, you must account for network stability and API constraints. Implementing exponential backoff and retries is essential for production code. If a single page request fails due to a temporary network blip or a rate limit, your code should pause and retry that specific cursor request rather than restarting the entire process from the beginning.

You must consider memory management as well. If you append millions of JSON objects to a single array in memory, your script will crash due to out-of-memory errors. For large datasets, you should process the items in batches within the while loop and stream the output directly to a local disk or another external database instead of holding everything in memory.

You can also complement direct pagination with reactive webhooks. Instead of polling the API to find new files and wasting bandwidth, set up webhooks to build reactive workflows. The Fast.io system will notify you when files change. You can then use the Fast.io API to fetch those details or trigger secondary processing. Features like URL Import allow you to pull files directly from external cloud providers like Google Drive or Dropbox without incurring high local I/O overhead.

Integrating with AI Agents

Fast.io serves as an intelligent workspace rather than simple file storage. When you upload files into the system, the platform automatically indexes them for semantic meaning. This Built-in RAG feature means you do not need to configure a separate vector database to make your content searchable by AI assistants.

Agents interact with this data using the same APIs you do. With the OpenClaw integration, you can install tools directly via clawhub install dbalve/fast-io with no additional configuration required. Agents can acquire file locks for concurrent multi-agent access, ensuring they never overwrite each other's changes.

You can also use advanced ownership transfer capabilities. A developer agent can create an organization, build custom workspaces, populate them using cursor-paginated data ingestion, and then transfer the finished workspace to a human client while retaining limited administrative access. The free tier offers a generous multiple of storage and multiple monthly credits with no credit card required, making it easy to start testing these automated workflows.