How to Stream Files with the Fast.io Node.js SDK

Node.js file streaming with the Fast.io SDK lets you process and upload large files without eating up memory. Instead of loading an entire file into RAM before sending it to the server, streaming breaks it into smaller chunks and sends them sequentially. This keeps your application responsive and avoids memory crashes, even when handling large datasets or concurrent uploads.

When you use the Fast.io SDK, streaming is the recommended approach for files larger than a few megabytes. The Fast.io architecture natively supports chunked uploads. Better memory management improves the performance of your AI assistants, especially if you are building agentic workflows or integrating with the many MCP tools. Processing data in chunks rather than buffering everything at once lets you maintain high throughput without scaling up your server hardware.

Helpful references: Fast.io Workspaces, Fast.io Collaboration, and Fast.io AI.

Before writing stream uploads, set up your Node.js environment and a Fast.io account. The AI Agent Free Tier provides 50GB of free storage and 5,000 monthly credits. This plan gives you plenty of space to start development.

Start by installing the official Fast.io SDK via your package manager.

npm install @fastio/sdk

Next, initialize the client in your application. Provide your API key, which you can generate from the Fast.io developer dashboard.

const { FastIO } = require('@fastio/sdk');

// Initialize the client with your API key
const client = new FastIO({
  apiKey: process.env.FASTIO_API_KEY
});

Keep your API keys secure using environment variables. Never hardcode credentials into your application source code. Once the client is ready, you can start building your file streams.

The most direct way to stream a file is by using the native Node.js filesystem module alongside the Fast.io SDK. You create a readable stream from a local file and pass it to the upload method. The SDK automatically handles the chunking and network transmission.

Here is an example of uploading a local file to a specific Fast.io workspace.

const fs = require('fs');
const path = require('path');

async function streamUploadBasic(workspaceId, filePath) {
  const fileName = path.basename(filePath);
  
  // Create a readable stream from the local file
  const fileStream = fs.createReadStream(filePath);
  
  try {
    // Pass the stream directly to the SDK
    const response = await client.files.uploadStream({
      workspaceId: workspaceId,
      name: fileName,
      stream: fileStream
    });
    
    console.log('Upload complete. File ID:', response.id);
    return response;
  } catch (error) {
    console.error('Upload failed:', error.message);
    throw error;
  }
}

This approach works well for simple scripts. Node.js reads the file in chunks defined by its buffer setting, and the Fast.io SDK manages the network requests. For production applications, you should use the Node.js pipeline API to make sure resources get cleaned up properly.

For production systems, the pipeline module is the best approach. It handles errors and ensures all streams are destroyed if any part of the process fails. This prevents memory leaks caused by streams hanging open after a network interruption.

The pipeline API requires a slightly different setup. You use the Fast.io SDK to request a writable stream destination.

const fs = require('fs');
const { pipeline } = require('stream/promises');

async function streamUploadPipeline(workspaceId, filePath) {
  const fileStream = fs.createReadStream(filePath);
  
  // Request a writable stream from the Fast.io SDK
  const uploadDestination = client.files.createWriteStream({
    workspaceId: workspaceId,
    name: 'data-export.csv'
  });
  
  try {
    // Pipeline automatically handles backpressure and cleanup
    await pipeline(
      fileStream,
      uploadDestination
    );
    console.log('Pipeline upload successful');
  } catch (error) {
    console.error('Pipeline failed or was aborted', error);
  }
}

The pipeline function takes multiple streams and connects them together. It returns a promise that resolves when the transfer completes or rejects if an error occurs. This handles backpressure natively, keeping memory usage low. Backpressure happens when the local read speed exceeds the network upload speed. The pipeline pauses the read stream until the upload stream catches up, preventing data from piling up in memory.

Give Your AI Agents Persistent Storage

Get started with the AI Agent Free Tier. Claim your generous storage and monthly credits with no credit card required.

Sign Up Free

Streaming lets you transform data on the fly. You can compress files, encrypt sensitive info, or parse logs before sending them to the cloud. Insert transform streams into your pipeline to modify the data in transit.

If you want to compress a log file using Gzip before uploading it, you can achieve this using the native compression module.

const fs = require('fs');
const zlib = require('zlib');
const { pipeline } = require('stream/promises');

async function compressAndUpload(workspaceId, filePath) {
  const fileStream = fs.createReadStream(filePath);
  const gzip = zlib.createGzip();
  
  const uploadDestination = client.files.createWriteStream({
    workspaceId: workspaceId,
    name: 'application-logs.gz'
  });
  
  try {
    // Data flows from the file to gzip and then to Fast.io
    await pipeline(
      fileStream,
      gzip,
      uploadDestination
    );
    console.log('Compressed upload complete');
  } catch (error) {
    console.error('Compression pipeline failed', error);
  }
}

This pattern reduces bandwidth usage. Fast.io charges 212 credits per gigabyte of bandwidth on the agent tier, so compressing text files extends your monthly quota. The transformation happens chunk by chunk, keeping your memory footprint low regardless of the total file size.

Networks drop connections, especially in cloud environments. When an upload fails midway, you must decide how your application will recover. Restarting the upload from the beginning is usually fine for small files. For larger files approaching the file size limit, you need a better strategy.

The Fast.io SDK supports resumable chunked uploads. When you initialize a write stream, the SDK internally manages the upload session. You can catch the error and attempt to resume the upload using the session identifier if a connection drops. You must also handle HTTP rate limit status codes, which indicate an API rate limit.

async function uploadWithRetry(workspaceId, filePath, maxRetries = 3) {
  let attempt = 0;
  
  while (attempt < maxRetries) {
    try {
      console.log(`Starting upload attempt ${attempt + 1}`);
      await streamUploadPipeline(workspaceId, filePath);
      return true; // Success
    } catch (error) {
      attempt++;
      console.error(`Attempt ${attempt} failed: ${error.message}`);
      
      if (error.status === 429) {
        console.log('Rate limit exceeded. Applying extended backoff.');
      }
      
      if (attempt >= maxRetries) {
        throw new Error('Maximum retry limit reached');
      }
      
      // Wait before retrying
      const baseDelay = error.status === 429 ? 5000 : 1000;
      const delay = Math.pow(2, attempt) * baseDelay;
      console.log(`Waiting ${delay}ms before next attempt.`);
      await new Promise(resolve => setTimeout(resolve, delay));
    }
  }
}

This code uses an exponential backoff strategy. The delay between retries increases with each failure. This prevents your application from overwhelming the server or local network during an outage. When dealing with rate limits, the base delay is higher to give the API limits time to reset.

When transferring large files, providing feedback to users or logging progress helps with debugging. Native Node.js streams emit events that you can listen to directly. A better way is to insert a pass-through stream into your pipeline to track bytes.

A pass-through stream takes input and passes it to the output without modification. By attaching an event listener to the data events of this stream, you can calculate the total bytes transferred and determine the upload percentage.

const fs = require('fs');
const { PassThrough } = require('stream');
const { pipeline } = require('stream/promises');

async function uploadWithProgress(workspaceId, filePath) {
  const stat = fs.statSync(filePath);
  const totalSize = stat.size;
  let uploadedBytes = 0;
  
  const fileStream = fs.createReadStream(filePath);
  const progressStream = new PassThrough();
  
  progressStream.on('data', (chunk) => {
    uploadedBytes += chunk.length;
    const ratio = uploadedBytes / totalSize;
    const percentVal = ratio * 100;
    const percentage = percentVal.toFixed(2);
    console.log(`Upload Progress: ${percentage}%`);
  });
  
  const uploadDestination = client.files.createWriteStream({
    workspaceId: workspaceId,
    name: 'video-export.mp4'
  });
  
  try {
    await pipeline(
      fileStream,
      progressStream,
      uploadDestination
    );
    console.log('Upload completed successfully.');
  } catch (error) {
    console.error('Upload encountered an error.', error);
  }
}

This pattern provides accurate feedback as the file transfers. It works well for command-line tools or backend services reporting progress to a frontend interface via WebSockets. Tracking progress also helps identify system bottlenecks. If the progress stalls at a specific percentage, you can investigate whether the issue is local disk read speeds or network bandwidth constraints.

Once your file streams successfully to Fast.io, the platform takes over. If you have Intelligence Mode enabled for the destination workspace, Fast.io automatically indexes the incoming file. You do not need to build a separate indexing pipeline or manage a vector database.

When the upload stream closes, the platform begins extracting metadata and generating embeddings. The document immediately becomes available for semantic search and AI chat interactions. When your AI agents query the workspace using the Fast.io MCP server, they can access the newly uploaded information with full source citations.

Moving directly from streaming raw data to searching knowledge speeds up agentic workflows. Your Node.js application handles data transport, while Fast.io makes that data intelligent and accessible to both human teammates and AI agents.