AWS Bedrock Token Limits: Model Output Caps, Converse API Parameters, and Quotas

Input Context Windows versus Output Completion Ceilings

A primary concept in Amazon Bedrock is the operational asymmetry between input context windows and output completion caps. The input context window defines the maximum sequence length a model can evaluate in its attention mechanism. This sequence includes system instructions, user prompts, past conversational turns, tool schemas, and document attachments.

In contrast, the output limit dictates the maximum number of new tokens the model can produce in response to that input. In the Bedrock API, developers specify this ceiling using request parameters such as maxTokens in the unified Converse API, max_tokens for Anthropic Claude, max_gen_len for Meta Llama, or max_new_tokens for Amazon Nova.

Crucially, you cannot trade input capacity for output capacity. An application sending a brief 100-token prompt to Claude 3.5 Sonnet cannot request a 50,000-token completion simply because 199,900 tokens of input headroom remain empty. The model output cap remains fixed at 8,192 tokens regardless of prompt brevity. Understanding this distinction is essential when designing multi-step agent workflows that generate large codebases, synthetic datasets, or comprehensive technical documentation.

How Upfront Token Reservations Affect Concurrent Requests

Amazon Bedrock employs an upfront token reservation mechanism to maintain service stability across multi-tenant GPU clusters. When a client application dispatches a request to the InvokeModel or Converse endpoints, the Bedrock runtime does not wait to see how many tokens the model actually generates. Instead, Bedrock inspects the input prompt length and adds the full value configured in your maximum output token parameter.

Bedrock immediately reserves that combined token total against your account Tokens Per Minute (TPM) quota for the duration of the API call. If your code submits a prompt of 40,000 tokens and sets maxTokens to 8,192, Bedrock temporarily locks 48,192 tokens against your regional allocation.

If several background workers or autonomous agents execute in parallel, these reservations stack quickly. Ten concurrent workers with those parameters lock nearly 500,000 tokens of quota capacity. If your regional quota is 400,000 TPM, subsequent requests fail immediately with an HTTP 429 ThrottlingException, even if the model returns concise 200-token answers. Setting realistic, tailored output caps rather than leaving parameters at their theoretical maximums is necessary for stable production throughput.

Anthropic Claude Model Limits on Bedrock

Anthropic Claude models represent some of the most widely deployed reasoning engines on Amazon Bedrock. Claude 3.5 Sonnet and Claude 3.5 Haiku provide an input context window of 200,000 tokens, with a maximum output limit of 8,192 tokens. Earlier generation models, such as Claude 3 Opus and Claude 3 Haiku, maintain the same 200,000-token input window but enforce a lower maximum completion ceiling of 4,096 tokens.

When invoking Claude models via Bedrock's native InvokeModel API, developers pass parameters inside a JSON body matching Anthropic's Messages schema:

{
  "anthropic_version": "bedrock-2023-05-31",
  "max_tokens": 8192,
  "temperature": 0.7,
  "messages": [
    {
      "role": "user",
      "content": "Generate a detailed system design specification."
    }
  ]
}

If your application specifies a max_tokens value greater than 8,192 for Claude 3.5 Sonnet, the Bedrock runtime returns a ValidationException stating that the parameter exceeds the maximum allowed value. If the model exhausts the allocated max_tokens before completing its generation, the response JSON sets stop_reason to "max_tokens".

Amazon Nova Model Ceilings and Multimodal Context

Amazon Nova models introduce deep multimodal support alongside expanded context windows. Amazon Nova Premier supports an input context window of 1,000,000 tokens, capable of processing extensive documentation archives, complex code repositories, or lengthy video files. Amazon Nova Pro and Amazon Nova Lite offer 300,000-token input windows, while Amazon Nova Micro provides a 128,000-token text-only context window.

Across the entire Nova family (Premier, Pro, Lite, and Micro), Amazon Bedrock standardizes the maximum completion output ceiling at 10,000 tokens. This makes Nova models capable for tasks requiring long-form text generation, extensive tabular synthesis, or detailed software module creation.

In native invocation payloads, Nova models accept inference settings within an inferenceConfig object, using max_new_tokens to define output limits. When called through the Converse API, Nova models map maxTokens to this underlying 10,000-token hardware cap.

Meta Llama and Mistral AI Output Boundaries

Meta Llama models on Bedrock, including Llama 3.1 (8B, 70B, 405B) and Llama 3.3 70B Instruct, provide a 128,000-token input context window. In native Bedrock invocations using InvokeModel, Meta models use the parameter name max_gen_len. The documented range for max_gen_len is 1 to 2,048 tokens, with a default value of 512 tokens.

Because 2,048 tokens corresponds to approximately 1,500 words, developers building applications that generate long documents with Llama models must account for this compact completion boundary. If a response exceeds max_gen_len, the model sets the stop reason to "length".

Mistral AI models, such as Mistral Large, also support a 128,000-token input context window. Unlike Llama, Mistral Large permits a maximum generation ceiling of 8,192 tokens, configured using the max_tokens parameter in native payloads.

Python Implementation with Boto3 and Usage Metrics

The following Python example demonstrates how to invoke a foundation model using the Converse API in boto3, specify maxTokens, and extract token usage metrics from the response.

import boto3
from botocore.exceptions import ClientError

client = boto3.client("bedrock-runtime", region_name="us-east-1")
model_id = "anthropic.claude-3-5-sonnet-20240620-v1:0"
prompt_text = "Provide a comprehensive technical audit checklist for cloud migration."

try:
    response = client.converse(
        modelId=model_id,
        messages=[{"role": "user", "content": [{"text": prompt_text}]}],
        inferenceConfig={"maxTokens": 2048, "temperature": 0.3, "topP": 0.9},
    )
    # extract generated response text
    output_message = response["output"]["message"]["content"][0]["text"]
    stop_reason = response["stopReason"]
    # inspect detailed token consumption
    usage = response["usage"]
    input_tokens = usage["inputTokens"]
    output_tokens = usage["outputTokens"]
    total_tokens = usage["totalTokens"]
    print(f"Generated text: {output_message[:150]}...")
    print(f"Stop Reason: {stop_reason}")
    print(f"Tokens consumed: Input={input_tokens}, Output={output_tokens}, Total={total_tokens}")
    if stop_reason == "max_tokens":
        print("Warning: Response reached maxTokens limit and was truncated.")
except ClientError as e:
    error_code = e.response["Error"]["Code"]
    error_message = e.response["Error"]["Message"]
    print(f"Bedrock invocation failed [{error_code}]: {error_message}")

In this script, response["usage"] returns exact token metrics. Logging these values enables precise unit-cost tracking and allows monitoring systems to detect when generated responses consistently hit the maxTokens ceiling.

Handling Truncation in ConverseStream Responses

When building interactive applications, developers typically implement ConverseStream to stream text deltas to end users with minimal Time-To-First-Token (TTFT). Monitoring token caps in a streaming context requires listening for specific stream events.

During a ConverseStream session, Bedrock emits a sequence of event blocks:

1. messageStart: Indicates message role (assistant).
2. contentBlockDelta: Emits incremental text chunks through event["contentBlockDelta"]["delta"]["text"].
3. messageStop: Emits the terminal status of the message, including stopReason.
4. metadata: Emits the cumulative usage block containing final inputTokens, outputTokens, and latency metrics.

If the model hits the maxTokens cap during streaming, text generation halts immediately. The messageStop event delivers event["messageStop"]["stopReason"] = "max_tokens". Client applications should detect this event and gracefully notify the user or initiate an automated continuation request.

Tokens Per Minute and Requests Per Minute Mechanics

AWS Service Quotas for Bedrock are primarily metered in two dimensions:

• Tokens Per Minute (TPM): The aggregate volume of input tokens submitted plus output tokens generated across all requests in a rolling 60-second window.
• Requests Per Minute (RPM): The total number of API calls initiated within a rolling 60-second window.

Bedrock tracks quotas separately for on-demand regional invocations, cross-region inference profiles, and provisioned throughput. For example, your account might have a default quota of 250 requests per minute and 400,000 tokens per minute for Claude 3.5 Sonnet in us-east-1.

When an application exceeds either threshold, the Bedrock runtime rejects the request with an HTTP 429 status code and a ThrottlingException. This error indicates that your account has saturated its allotted bandwidth on AWS infrastructure. It does not mean the model failed to process the prompt, nor does it indicate that your prompt exceeded the model's context window.

Comparing Failure Modes and Remediation Paths

Understanding the differences between operational bottlenecks simplifies incident triage. The following comparison highlights how inference caps differ from service quotas:

If you receive an HTTP 400 ValidationException stating that your input prompt exceeded the context window, you must reduce the prompt length or switch to a model with a larger context window, such as Amazon Nova Premier. If you receive an HTTP 429 ThrottlingException, you should open the AWS Service Quotas console, search for Amazon Bedrock, locate the quota for Claude 3.5 Sonnet cross-region TPM, and submit an increase request.

Continuation Patterns for Output Truncation

When an application legitimately requires a completion longer than a model's output cap (for example, generating a 15,000-token software specification with Claude 3.5 Sonnet), developers implement programmatic continuation loops.

When a call to Converse returns stopReason == "max_tokens", the application preserves the generated text and immediately initiates a follow-up turn. The application appends the previous assistant completion to the messages array and sends a user message prompting the model to resume:

# continuation turn pattern
messages.append({"role": "assistant", "content": [{"text": partial_output}]})
messages.append({"role": "user", "content": [{"text": "Continue generating precisely from where you stopped."}]})
continuation_response = client.converse(
    modelId=model_id,
    messages=messages,
    inferenceConfig={"maxTokens": 4096}
)

By stringing consecutive turns together, applications assemble long outputs while keeping each individual completion safely within Bedrock parameter ceilings.

Decoupling Large Document Corpora with Fast.io Workspaces

While continuation loops solve output truncation, handling massive reference libraries requires external storage architecture. Rather than streaming large files directly into prompt tokens, production architectures store files in external workspaces and expose them to models via structured retrieval.

Traditional approaches rely on raw object stores like Amazon S3 paired with custom embedding pipelines, vector databases, and chunking scripts. While functional, building and maintaining custom retrieval infrastructure introduces substantial engineering maintenance. Local file storage offers fast local reads but fails when multiple agents or human reviewers require shared, synchronized access.

The large-corpus path on Fast.io provides a purpose-built workspace platform for agentic teams. Teams upload their document collections into Fast.io workspaces, or sync them directly from Google Drive, Dropbox, Box, or OneDrive. When Intelligence Mode is enabled on the workspace, Fast.io indexes incoming files for hybrid search (combining semantic meaning, full-text search, and metadata matching). Learn more about Fast.io AI capabilities for workspace indexing.

Instead of attaching raw documents to Bedrock prompts, an AI assistant or agent connects to the workspace through the remote Model Context Protocol (MCP) server at https://mcp.fast.io/mcp/tools. For implementation details, consult the Fast.io MCP documentation. Using MCP tools, the agent queries the indexed workspace dynamically, retrieving only the exact paragraphs, metrics, or legal clauses necessary to answer the prompt.

This architecture keeps Bedrock input prompts compact, reduces throttling risks, and leaves full reasoning headroom inside the model context window. Workspaces include per-file version history, a detailed activity log tracking human and agent actions, and role-based permissions.

Monthly plans start with a 14-day free trial, which requires a credit card. See pricing for plan options.