How to Migrate from Amazon S3 to Fast.io API

Why Migrate from Amazon S3 to Fast.io

Amazon S3 handles object storage well for basic needs. Developers often pair it with vector databases like Pinecone for RAG, relational DBs for metadata, and custom sharing logic. This splits your stack across services, increases costs, and complicates agent workflows.

Fast.io provides object-like storage through workspaces but adds native intelligence. Upload files and they auto-index for semantic search and RAG chat. Agents access 251 MCP tools for full CRUD, locks, webhooks, and ownership transfer.

Benefits include 50GB free storage on the agent tier, no infrastructure management, and unified APIs for storage + AI + collaboration. For agentic teams, this streamlines architecture without custom integrations.

Fast.io vs Amazon S3 Comparison

Fast.io suits agentic workflows where S3 feels like commodity storage. Use S3 for pure archival; Fast.io for active team/agent use.

Migration Prerequisites

Review your S3 bucket size, object count, and access patterns. Ensure files <1GB or plan splitting. Get AWS credentials with read access. Sign up for Fast.io agent account (no credit card).

Install tools:

• AWS CLI v2
• Python 3.10+ with boto3, requests
• jq for JSON processing

Test Fast.io API access with curl after auth.

Step 1: Set Up Fast.io Organization and Workspace

Create agent account via API:

curl -X POST https://api.fast.io/current/user/ \\
  -u email:password \\
  -d agent=true

Create org and workspace:

ORG_ID=$(curl -s -X POST https://api.fast.io/current/org/ \\
  -H "Authorization: Bearer $JWT" \\
  -d "domain=my-migration-org" -d "name=My Org" | jq -r .response.id)

WS_ID=$(curl -s -X POST "https://api.fast.io/current/org/$ORG_ID/workspace/" \\
  -H "Authorization: Bearer $JWT" \\
  -d "folder_name=migrated-bucket" -d "name=Migrated Bucket" | jq -r .response.id)

Enable intelligence for auto-indexing.

Step 2: Inventory S3 Bucket

List objects and metadata:

aws s3 ls s3://your-bucket --recursive --summarize \\
  --page-size 1000 > s3-inventory.json

Parse for total objects, size, last modified. Identify large files >1GB to split or skip.

Step 3: Zero-Downtime Data Transfer

Use presigned S3 URLs + Fast.io web-import for server-side copy (no local bandwidth).

Python script (migrate_s3.py):

import boto3
import requests
import json
import time
from concurrent.futures import ThreadPoolExecutor

s3 = boto3.client('s3')
FASTIO_BASE = 'https://api.fast.io/current'
headers = {'Authorization': f'Bearer {jwt_token}'}
ws_id = 'your_ws_id'
parent_id = 'root'

def presign_and_import(key):
    presign_url = s3.generate_presigned_url('get_object',
        Params={'Bucket': 'your-bucket', 'Key': key},
        ExpiresIn=3600)
    resp = requests.post(f'{FASTIO_BASE}/upload/web-import/',
        headers=headers,
        data={'profile_type': 'workspace', 'profile_id': ws_id,
              'parent_id': parent_id, 'url': presign_url})
    if resp.json().get('result'):
        print(f'Migrated {key}')
    else:
        print(f'Failed {key}: {resp.text}')

# List keys
paginator = s3.get_paginator('list_objects_v2')
keys = []
for page in paginator.paginate(Bucket='your-bucket'):
    keys.extend(obj['Key'] for obj in page.get('Contents', []))

# Batch import (parallel 10)
with ThreadPoolExecutor(max_workers=10) as executor:
    executor.map(presign_and_import, keys)

# Poll import status if needed

Run in batches for PB scale. Monitor with /upload/web-list/. Incremental: import only modified since last sync.

Step 4: Update Code from S3 SDK to Fast.io API

Replace boto3 put_object with Fast.io upload.

S3 example:

s3.put_object(Bucket='bucket', Key='file.txt', Body=data)

Fast.io:

# Init upload
init_resp = requests.post(f'{FASTIO_BASE}/upload/init/',
    headers=headers,
    data={'profile_type': 'workspace', 'profile_id': ws_id,
          'parent_id': 'root', 'name': 'file.txt', 'size': len(data)})
upload_id = init_resp.json()['response']['id']

# Upload chunk (parallel for large)
requests.put(init_resp.json()['response']['upload_url'], data=data)

# Complete
requests.post(f'{FASTIO_BASE}/upload/{upload_id}/complete/', headers=headers)

Use MCP for agents.

Step 5: Verify and Cut Over

Compare object count/size S3 vs Fast.io storage list.

Test reads from Fast.io preview URLs.

Update DNS/CNAME if bucket website, or app config.

Monitor activity logs. Go live with dual-write during transition.

Petabyte-Scale Tips

• Parallelize with 100+ threads
• Use S3 inventory reports for delta sync
• Split >1GB files with ffmpeg or zip
• Throttle requests (Fast.io rate limits)
• Dual-read until confident