How to Build Concurrent File Uploads with the Fastio API

Concurrent file uploads with the Fastio API allow developers to dramatically reduce transfer times by parallelizing data streams while respecting dynamic rate limits. Instead of sending files sequentially, your application opens multiple network connections and transmits multiple files simultaneously. This approach bypasses single-stream TCP limits and maximizes available bandwidth.

For AI agents and automated workflows, data ingestion speed directly impacts performance. When an agent needs to process hundreds of documents, waiting for sequential uploads creates a major bottleneck. Parallel file uploads speed up data ingestion for AI agents by saturating the network connection. By using the Fastio API, developers can ensure that documents, media assets, and training data reach the intelligent workspace as quickly as possible.

Fastio is designed specifically to handle high-concurrency workloads from both human users and AI agents. Agent integrations should use the MCP server at https://mcp.fast.io/mcp (or https://mcp.fast.io/mcp/key with a Bearer header). Named mode exposes 19 action-routed tools, including upload (create-session, chunk, finalize, batch, stream-upload, web-import). Headless agents can drive the same upload actions without hand-writing HTTP. Direct REST callers post to https://api.fast.io/current/upload/ with Authorization: Bearer {api_key}.

Helpful references: Fastio Workspaces, Fastio Collaboration, and Fastio AI.

Sequential uploading is the default behavior for most HTTP clients, but it scales poorly for enterprise workloads. When an application uploads files one at a time, it suffers from accumulated network latency. Every file transfer requires a separate TCP handshake, TLS negotiation, and acknowledgment cycle. Over a batch of a thousand files, this overhead adds up to minutes of wasted time.

A single TCP connection also rarely uses the full capacity of a modern broadband link. Operating systems and network hardware apply congestion control algorithms that limit the throughput of individual streams. If a single upload stream experiences packet loss, the entire transfer slows down.

In contrast, a concurrent approach opens a pool of connections. If one connection experiences a delay, the others continue transferring data. This resilience is especially important when importing files from external sources. Agents can hand Fastio a public URL and let the server pull the bytes: POST https://api.fast.io/current/web_upload/ with form fields source_url, file_name, profile_id, profile_type (workspace or share), and folder_id. The MCP equivalent is the upload tool with action web-import. That keeps local I/O off the client and lets you fire several imports at once.

The Fastio architecture treats storage as an intelligent workspace rather than a passive repository. When you upload files using the API, you are not just writing bytes to disk; you are feeding data into a system that automatically indexes and processes the content. Toggle Intelligence Mode on a workspace, and files are auto-indexed, allowing agents to ask questions with citations immediately after the upload completes. Ripley, the built-in RAG agent, can answer those questions from the new files.

Because the backend performs this immediate processing, managing upload concurrency is critical. Small files go in one multipart request to POST https://api.fast.io/current/upload/ (fields name, size, chunk, action=create, instance_id set to the workspace ID, folder_id=root) and return HTTP 201 with result, id, and new_file_id. Large files open a session on that same route (omit chunk), then send up to 3 parallel parts to POST /current/upload/{id}/chunk/?order=N&size=N, finish with POST /current/upload/{id}/complete/, and wait on GET /current/upload/{id}/details/?wait=60 for {session:{status,new_file_id}}. Up to 200 files of 4MB or less can go through POST /current/upload/batch/. The client still paces the pool so it stays within the API's request rates.

A well-architected client uses connection pooling to maintain a fixed number of active HTTP connections. This prevents the client from overwhelming its own local network interface and ensures that it stays within the API's acceptable request rates. By reusing connections for multiple requests, the client also avoids the overhead of repeated TLS handshakes.

The most common mistake developers make when implementing concurrent file uploads is ignoring rate limits. While opening dozens of simultaneous connections might seem like the fast approach, it will inevitably trigger HTTP Too Many Requests errors. A strong implementation requires safely balancing high concurrency with enterprise API rate limiting.

Rate limiting protects the API from abuse and ensures fair usage across all tenants. When an application exceeds the allowed request rate, the Fastio API responds with HTTP 429 and error code 1671. Back off until the time in the x-ve-limit-expires header. If your client ignores that header and continues blasting requests, the connection pool will stall, and the overall upload process will fail.

To solve this, developers must implement a combination of concurrency limits and exponential backoff. A concurrency limit restricts the maximum number of active upload streams. By defining a strict maximum concurrent thread count, the client will only start the next upload when an active stream completes. This creates a steady, manageable flow of data rather than a massive, uncoordinated spike.

To demonstrate how to build concurrent file uploads with the Fastio API, we will use Node.js and the p-limit library to control concurrency. This example posts an array of files to https://api.fast.io/current/upload/ as multipart/form-data and never exceeds the specified concurrency limit.

import pLimit from 'p-limit';
import fs from 'fs';
import path from 'path';

const API = 'https://api.fast.io/current';
const WORKSPACE_ID = process.env.FASTIO_WORKSPACE_ID;
const AUTH = { Authorization: `Bearer ${process.env.FASTIO_API_KEY}` };

// Whole-file uploads share a small pool
const fileLimit = pLimit(5);
// Large files: up to 3 chunks in flight
const chunkLimit = pLimit(3);

async function uploadSmallFile(filePath) {
  const bytes = fs.readFileSync(filePath);
  const form = new FormData();
  form.append('name', path.basename(filePath));
  form.append('size', String(bytes.length));
  form.append('chunk', new Blob([bytes]));
  form.append('action', 'create');
  form.append('instance_id', WORKSPACE_ID);
  form.append('folder_id', 'root');

const response = await fetch(`${API}/upload/`, {
    method: 'POST',
    headers: AUTH,
    body: form,
  });

if (response.status === 429) {
    const retryUntil = response.headers.get('x-ve-limit-expires');
    console.log(`Rate limited (1671). Back off until ${retryUntil}.`);
    throw new Error(`Rate limited until ${retryUntil}`);
  }

if (!response.ok) {
    throw new Error(`Upload failed with status: ${response.status}`);
  }

// 201 { result, id, new_file_id }
  return response.json();
}

async function uploadLargeFile(filePath, slices) {
  const stat = fs.statSync(filePath);
  const start = new FormData();
  start.append('name', path.basename(filePath));
  start.append('size', String(stat.size));
  start.append('action', 'create');
  start.append('instance_id', WORKSPACE_ID);
  start.append('folder_id', 'root');

const created = await fetch(`${API}/upload/`, {
    method: 'POST',
    headers: AUTH,
    body: start,
  });
  const session = await created.json();
  const id = session.id;

await Promise.all(slices.map(({ order, offset, size }) =>
    chunkLimit(async () => {
      const part = new FormData();
      const buf = Buffer.alloc(size);
      const fd = fs.openSync(filePath, 'r');
      fs.readSync(fd, buf, 0, size, offset);
      fs.closeSync(fd);
      part.append('chunk', new Blob([buf]));
      const response = await fetch(
        `${API}/upload/${id}/chunk/?order=${order}&size=${size}`,
        { method: 'POST', headers: AUTH, body: part },
      );
      if (response.status === 429) {
        const retryUntil = response.headers.get('x-ve-limit-expires');
        throw new Error(`Rate limited (1671) until ${retryUntil}`);
      }
      if (!response.ok) {
        throw new Error(`Chunk ${order} failed with status: ${response.status}`);
      }
    }),
  ));

await fetch(`${API}/upload/${id}/complete/`, {
    method: 'POST',
    headers: AUTH,
  });

const details = await fetch(`${API}/upload/${id}/details/?wait=60`, {
    headers: AUTH,
  });
  return details.json();
}

async function processBatch(filePaths) {
  const uploadPromises = filePaths.map((filePath) =>
    fileLimit(() => uploadSmallFile(filePath))
  );
  return Promise.allSettled(uploadPromises);
}

This pattern keeps a controlled connection pool. The p-limit wrapper caps how many files are in transit. For a large file, pass slices as { order, offset, size } entries and keep the chunk pool at 3. If the API returns HTTP 429 with error code 1671, read x-ve-limit-expires and wait until that time before the stream retries. Same-name upload into the same folder overwrites in place and keeps the old content as a recoverable version; the node_id stays stable.

Accelerate Your Agent Workflows

Build parallel upload pipelines with the Fastio REST API and MCP upload tools. Start a 14-day trial and put concurrent file ingestion in front of an intelligent workspace.

Start 14-Day Trial

While the previous example handles basic rate limits, a production-grade system needs solid exponential backoff. Network instability, temporary API congestion, and edge-case errors can cause uploads to fail intermittently. Implementing a thorough retry strategy ensures that your AI agents do not lose data during automated ingestion workflows.

Exponential backoff increases the wait time between retry attempts. The first failure might trigger a multiple-second delay, the second a multiple-second delay, the third a multiple-second delay, and so on. This prevents the "thundering herd" problem, where dozens of failed requests immediately retry at the exact same moment, causing further congestion.

You should also introduce "jitter" to your backoff algorithm. Jitter adds a small amount of randomization to the delay timer. By randomizing the wait times, you ensure that multiple concurrent streams do not synchronize their retries, further smoothing out the load on the API. On HTTP 429 with error code 1671, wait until x-ve-limit-expires before the next attempt.

In complex AI workflows, multiple agents might attempt to interact with the same workspace simultaneously. One agent might be uploading a batch of documents, while another agent attempts to read, summarize, or modify those documents. This concurrency can lead to race conditions and data corruption if not handled properly.

Fastio provides native file locks for concurrent multi-agent access. Before an agent begins modifying a file, it acquires a lock with POST /current/workspace/{workspace_id}/storage/{node_id}/lock/. Keep the lock alive with POST /current/workspace/{workspace_id}/storage/{node_id}/lock/heartbeat/. Release it with DELETE /current/workspace/{workspace_id}/storage/{node_id}/lock/. Other agents attempting to write the locked node wait until the lock is released. Workspace activity for the same files is visible in the audit log at GET /current/events/search/ or by long-polling GET /current/activity/poll/{entityId}?wait=95&lastactivity={timestamp}.

When designing concurrent upload pipelines, lock the target node before a coordinated write. Once the parallel uploads complete and the files are indexed, the uploading agent releases the lock so downstream summarization or analysis agents can begin their work.

Once your AI agents have successfully uploaded and processed the data via concurrent API calls, the final step often involves handing the results back to a human user. Fastio excels at this human-agent collaboration. Agents and humans share the same workspaces, the same tools, and the same intelligence.

Using the API, an agent can create an organization with POST /current/org/create/, add a workspace with POST /current/org/{org_id}/create/workspace/, populate it with parallel uploads to POST /current/upload/, and invite a human teammate with POST /current/workspace/{workspace_id}/members/{email_or_user_id}/. The teammate then works in the same intelligently indexed workspace the agent just filled.

This workflow is particularly powerful for client portals and data rooms. The automated ingestion pipeline handles the heavy lifting of moving gigabytes of data rapidly, while the human user receives a fully populated, intelligently indexed workspace ready for immediate use. Branded Send, Receive, and Exchange shares (POST /current/workspace/{workspace_id}/create/share/) give outside collaborators a portal into that same workspace.

If you are building custom AI agents, you can bypass direct API programming and use the OpenClaw integration. OpenClaw allows agents to interact with Fastio using natural language commands, completely abstracting the complexity of API endpoints and rate limiting.

To get started, run the installation command: clawhub install dbalve/fast-io. This installs an extensive suite of tools with zero configuration required. The OpenClaw skill automatically manages file uploads, downloads, and workspace creation. While direct API access via concurrent scripting offers the highest performance for massive data ingestion, the OpenClaw integration is the fast way to add intelligent file management to any LLM-based workflow.

Whether you use direct API calls or the OpenClaw toolkit, Fastio ensures that your files are immediately available for built-in RAG and semantic search. It is not just commodity storage; it is the coordination layer where agent output becomes team output.