Replicate Rate Limits: Concurrency Caps, HTTP 429 Errors, and Agent Storage

Replicate caps prediction creation at 600 requests per minute and general API endpoints at 3,000 requests per minute, while unverified accounts with granted credits operate under a strict ceiling of 1 request per second and 6 requests per minute, according to Replicate's official rate limit documentation. In high-throughput AI agent architectures, rate limits and concurrency caps govern how many parallel inference tasks can execute before model requests fail.

Replicate rate limits govern the number of simultaneous model predictions and API requests an account can execute before returning HTTP 429 Too Many Requests errors.

Understanding these boundaries requires distinguishing between two distinct throttling mechanisms enforced by Replicate: request throughput limits and execution concurrency caps. Request throughput limits restrict the frequency of HTTP calls sent to the Replicate REST API, including prediction creation (POST /v1/predictions) and polling endpoints (GET /v1/predictions/{id}). Execution concurrency caps limit the number of machine learning models running simultaneously on Replicate's GPU clusters.

When developers deploy autonomous agents that generate images, transcribe audio, or run open-source language models, operations fail if the agent confuses API request capacity with hardware execution capacity. A script can submit 10 HTTP requests within a single second without exceeding the 600 requests per minute API limit. However, if that account holds an execution cap of two concurrent predictions, the third prediction either queues behind the running models or gets rejected with an HTTP 429 status code once queue thresholds are crossed.

Replicate Account Tiers and Concurrency Boundaries

New accounts and accounts relying on trial credits start with strict concurrency allocations. As account spending increases and a verified payment method remains on file, Replicate raises concurrency thresholds to support production workloads.

While short bursts above 600 requests per minute are temporarily accommodated by Replicate's gateway, sustained prediction creation exceeding 600 requests per minute triggers API rate limit throttling. Accounts with unpaid balances or depleting credit balances face dynamic throttling designed to prevent accidental overspending.

• Gemini API Rate Limits: Tier Quotas, 429 Handling, and Large-Payload WorkflowsGemini API rate limits govern requests per minute (RPM), tokens per minute (TPM), and requests per day (RPD) across...
• OpenRouter Rate Limits: 20 RPM Caps, 429 Errors, and Agent Storage WorkaroundsOpenRouter restricts free models to 20 requests per minute and 50 to 1,000 requests per day based on credit purchases,...
• Hugging Face Rate Limits: Serverless Inference, Hub APIs, and Large Asset WorkspacesHugging Face rate limits regulate the frequency of calls to the Serverless Inference API and Git LFS repository...
• How to Fix AutoGen Rate Limits: Handling 429 Errors in Multi-Agent WorkflowsAn AutoGen rate limit occurs when multiple autonomous agents in a group chat collectively exceed host model request or...
• Google AI Studio Rate Limits: Free Tier Quotas, TPM, and Handling 429 ErrorsGoogle AI Studio rate limits enforce operational caps across requests per minute, tokens per minute, and daily request...
• Together AI Rate Limits: 1 RPS Free Caps, Tier Quotas, and 429 Error FixesTogether AI rate limits restrict requests per second (RPS), requests per minute (RPM), and tokens per minute (TPM) sent...

More on this subject: Agent Security and Governance (68 guides)

Automated pipelines and multi-step agents encounter two recurring HTTP status codes when interacting with Replicate: HTTP 429 Too Many Requests and HTTP 413 Request Entity Too Large. Each status code signals a distinct failure mode in how client code interacts with Replicate's gateway and worker pools.

The Mechanics of HTTP 429 Throttling

When an API client crosses a rate threshold or exceeds account concurrency boundaries, Replicate returns an HTTP 429 response containing a JSON body:

{
  "detail": "Request was throttled. Your rate limit resets in ~30s."
}

Three primary operational conditions trigger this response:

1. Bursting Beyond Request Thresholds: Sending rapid bursts of requests to POST /v1/predictions faster than Replicate's token bucket algorithm allows. While brief bursts of 10 requests per second are tolerated, sustained prediction creation exceeding 600 requests per minute triggers API rate limit throttling.
2. Concurrency Pool Saturation: Replicate models run on physical hardware (such as Nvidia A100 or H100 GPUs). If an account is capped at 10 concurrent predictions and attempts to dispatch 15 parallel video generation jobs, the excess requests saturate the account queue. When shared hardware pools face high demand, Replicate sheds load by rejecting excess predictions with HTTP 429 errors.
3. Low Credit Balance Defensive Throttling: Replicate automatically enforces stricter rate limits as an account balance approaches zero. This mechanism prevents accounts from sliding into negative balances during high-concurrency tasks. Replicate recommends maintaining a credit balance above $20 with auto-reload configured to prevent defensive throttling.

The Mechanics of HTTP 413 Payload Failures

While HTTP 429 errors stem from timing and volume, HTTP 413 Request Entity Too Large errors stem from payload construction.

When developers use client libraries to run models on audio files, high-resolution source images, or video clips, passing a local file object directly into the input payload causes the client to serialize large binary streams into the HTTP request body:

"""Problematic pattern: triggers HTTP 413 when audio exceeds body limits"""
output = client.run(
    "openai/whisper:cdd978ddf443c64bde530ddab65c28a501bbdbda16d1a9f4116db7a114e975a0",
    input={"audio": open("interview_recording_120mb.mp3", "rb")}
)

Replicate's API gateway rejects large binary payloads before the data ever reaches a model container. The documented architecture pattern requires hosting the input file on external cloud storage and passing a publicly accessible HTTPS URL to the input dictionary:

"""Recommended pattern: passing a public HTTPS URL avoids HTTP 413"""
output = client.run(
    "openai/whisper:cdd978ddf443c64bde530ddab65c28a501bbdbda16d1a9f4116db7a114e975a0",
    input={"audio": "https://workspaces.example.com/audio/interview_recording.mp3"}
)

Staging source assets in cloud workspaces before initiating inference eliminates HTTP 413 errors and keeps API request bodies lightweight.

Autonomous agents executing multi-step workflows must manage outbound requests client-side rather than spraying unthrottled API calls. Combining an in-memory concurrency semaphore with exponential backoff and jitter ensures predictable execution under tight platform caps.

Enforcing Client-Side Concurrency with Semaphores

If an account has an execution cap of 10 concurrent predictions, the client application should never dispatch 20 simultaneous requests. Using an asynchronous semaphore restricts active workers to match the platform concurrency tier:

import asyncio
import os
import random
import time
import httpx

REPLICATE_API_TOKEN = os.environ.get("REPLICATE_API_TOKEN")
CONCURRENCY_LIMIT = 5
semaphore = asyncio.Semaphore(CONCURRENCY_LIMIT)

async def create_prediction_with_retry(
    client: httpx.AsyncClient,
    model_version: str,
    model_input: dict,
    max_retries: int = 5
) -> dict:
    url = "https://api.replicate.com/v1/predictions"
    headers = {
        "Authorization": f"Bearer {REPLICATE_API_TOKEN}",
        "Content-Type": "application/json"
    }
    payload = {
        "version": model_version,
        "input": model_input
    }
    async with semaphore:
        base_delay = 2.0
        for attempt in range(max_retries):
            response = await client.post(url, json=payload, headers=headers)
            if response.status_code == 201:
                return response.json()
            if response.status_code == 429:
                """Add randomized jitter to avoid thundering herd problem"""
                jitter = random.uniform(0.5, 1.5)
                delay = (base_delay * (2 ** attempt)) + jitter
                print(f"Throttled by Replicate (429). Retrying in {delay:.2f} seconds...")
                await asyncio.sleep(delay)
                continue
            """Non-retryable error"""
            response.raise_for_status()
        raise RuntimeError("Exceeded maximum retries for Replicate prediction creation.")

Avoiding Synchronous Connection Exhaustion

Replicate provides a Prefer: wait HTTP header that tells the gateway to hold an HTTP connection open for 60 seconds while the model generates its result. While this synchronous mode is convenient for quick scripts, it creates severe bottlenecks in multi-agent systems:

• Holding connections open ties up local HTTP socket pools and client memory.
• If model generation takes 65 seconds, the HTTP connection times out even though the model is still executing on Replicate's cluster.
• The client re-issues the prediction, duplicating compute costs and consuming a second concurrency slot.

For production agents running parallel tasks, asynchronous prediction creation is the superior pattern. The client initiates the job, receives a prediction identifier, and releases the connection.

Store Replicate Model Outputs in Persistent Workspaces

Give your AI agents a shared, durable workspace where generated media and prediction assets stay versioned, searchable, and accessible via the Fast.io MCP server. Monthly plans start with a 14-day free trial.

Start 14-Day Trial

A common architecture flaw in AI agent pipelines is polling GET /v1/predictions/{id} in a tight loop to inspect completion status. While Replicate allows up to 3,000 requests per minute on non-creation endpoints, polling creates hidden operational traps as workloads scale.

The Hidden Costs of Polling

Consider an autonomous pipeline processing a batch of 30 image upscaling tasks:

1. Each upscaling prediction takes 15 seconds to run on Replicate hardware.
2. An agent polling every two seconds per prediction generates roughly 8 calls per job.
3. Thirty concurrent jobs generate 240 polling requests in 15 seconds, creating high network chatter.
4. When multiple agents or workflow runs execute simultaneously, polling volume rapidly consumes the 3,000 requests per minute budget, starving other critical API calls.

Polling also introduces latency. If a prediction finishes 50 milliseconds after a poll check, the client waits nearly two seconds before discovering the result on the next polling cycle.

Configuring Asynchronous Webhooks

Replicate provides first-class webhook delivery. When submitting POST /v1/predictions, developers can specify a public HTTPS webhook endpoint and filter the exact events that trigger callbacks:

{
  "version": "50d859923762e880bf4b386192a73e01aa86cfa51fbe913e7737b1993c2c27bc",
  "input": {
    "prompt": "Architectural elevation drawing of an intelligent workspace library"
  },
  "webhook": "https://agent-gateway.example.com/api/webhooks/replicate",
  "webhook_events_filter": ["start", "completed"]
}

Filtering events with ["start", "completed"] suppresses intermediate logging events (which Replicate throttles to once every 500 milliseconds) and ensures the webhook fires only when the job initiates and reaches a terminal state (succeeded, failed, or canceled).

Implementing an Idempotent Webhook Receiver

Network partitions can cause Replicate to retry webhook delivery. A production webhook handler must be idempotent, acknowledging incoming events immediately and deduplicating events by prediction identifier:

from fastapi import FastAPI, Request, status, Response

app = FastAPI()
processed_prediction_ids = set()

@app.post("/api/webhooks/replicate")
async def handle_replicate_webhook(request: Request):
    payload = await request.json()
    prediction_id = payload.get("id")
    status_val = payload.get("status")
    """Return 200 OK immediately to satisfy Replicate delivery timeout"""
    if not prediction_id:
        return Response(status_code=status.HTTP_400_BAD_REQUEST)
    if prediction_id in processed_prediction_ids and status_val == "succeeded":
        return Response(status_code=status.HTTP_200_OK)
    if status_val == "succeeded":
        processed_prediction_ids.add(prediction_id)
        output_urls = payload.get("output")
        """Dispatch background persistence task for generated media"""
        print(f"Prediction {prediction_id} succeeded. Output assets: {output_urls}")
    return Response(status_code=status.HTTP_200_OK)

Transitioning from active polling to event-driven webhooks frees up the 3,000 requests per minute API ceiling and provides sub-second notification when generation finishes.

Overcoming Replicate rate limits and 429 errors only solves the ingestion half of the pipeline. The second major architectural pitfall for autonomous agents is data retention.

For predictions created through the API, Replicate automatically deletes all input parameters, output values, output files, and logs after one hour, according to Replicate's official data retention documentation.

Output files generated by models (such as PNG images, MP4 videos, WAV audio files, or JSON text outputs) are hosted on temporary URLs under the replicate.delivery domain. If an AI agent saves the raw URL string into a database or context memory, that link becomes a dead 404 error sixty minutes later.

Evaluating Storage Layers for AI Media Pipelines

Autonomous agents require persistent storage where generated media assets remain durable, auditable, and accessible to human team members:

1. Local Disk Storage: Saving generated files to a local server or container filesystem works for single-node scripts. However, local files cannot be easily shared with other team members, fail when container instances recycle in serverless cloud environments, and offer no search capabilities.
2. Object Storage (AWS S3 or Google Cloud Storage): Object storage provides durable file persistence. However, teams must configure custom IAM credentials, manage expiring pre-signed URLs for human review, and build separate search indexes to locate assets later.
3. Fast.io Intelligent Workspaces: Fast.io provides shared organization-owned workspaces designed for agentic teams. Rather than treating storage as an isolated bucket, a Fast.io workspace combines persistent cloud storage with built-in indexing and multi-agent coordination.

Persisting Model Outputs into Fast.io Workspaces

When an autonomous agent receives an incoming webhook from Replicate, the payload delivers generated media URLs hosted on replicate.delivery. The agent can immediately ingest these assets into a Fast.io workspace before the temporary links expire.

Agents can pull files using URL import directly from external sources without routing heavy binary payloads through local machine memory, or upload files directly through Fast.io's remote Model Context Protocol (MCP) server.

Agents connect over Streamable HTTP to the Fast.io MCP endpoint:

• Coding agents (such as Claude Code, Cursor, Cline, or Devin) connect to https://mcp.fast.io/mcp/code.
• General AI assistants and autonomous orchestrators connect to https://mcp.fast.io/mcp/tools.

When an autonomous agent writes model outputs into a Fast.io workspace, the assets gain several operational capabilities:

• Built-in Intelligence and Search: When Intelligence is enabled on a workspace, files are indexed for hybrid search (full-text, semantic meaning, and metadata values). An agent or human can query the workspace using natural language to retrieve generated media, transcription notes, or research documents with exact source citations.
• Per-File Version History: If an agent re-runs an image refinement or upscaling model on an existing graphic, the workspace keeps every version. Team members can inspect the evolution of assets or restore previous versions at any time.
• Detailed Activity Logging: Fast.io maintains a detailed activity log recording what changed in an organization and who changed it, covering uploads, edits, moves, and deletions across people and agents alike.
• Scoped Team Sharing: Files and folders can be shared securely through branded shares (Send, Receive, or Exchange) with per-recipient access controls and download restrictions, without granting external parties access to the broader workspace.

Monthly plans start with a 14-day free trial that requires a credit card and ends early if trial credits run out. Plans are Starter at $9.99/mo (3 seats, 250 GB storage, 5 workspaces, 100,000 credits a month), Business at $49.99/mo (10 seats, 5 TB storage, 50 workspaces, 600,000 credits a month), and Enterprise at $199.99/mo (30 seats included, 25 TB storage, 200 workspaces, 3,000,000 credits a month). Credits meter AI operations, with overage at $10 per 100,000 credits. Plan storage and seats are fixed allocations included with each subscription tier.

Pairing client-side rate limiters with durable workspace persistence allows engineering teams to run high-concurrency Replicate pipelines without hitting HTTP 429 bottlenecks or losing ephemeral model outputs.