Cohere Context Window: Command R Token Limits and Enterprise Search

Cohere's own model documentation lists Command R and Command R+ at a 128,000 token context window, and the newer Command A and Command A Reasoning models at 256,000. The Cohere context window is the maximum sequence length, measured in tokens, that a Cohere model can evaluate during retrieval-augmented generation and tool use. On Command R that window spans 128,000 tokens across both input prompts and generated responses, providing enterprise applications with the capacity to analyze extended documents.

In computational linguistics and transformer architectures, tokens represent word fragments, punctuation, and syntax characters rather than complete words. In standard English prose, one token corresponds to approximately four characters or 0.75 words. However, token density rises when processing structured data formats such as JSON payloads, source code, tabular CSV exports, and markdown tables. Variable names, nested brackets, indentation spaces, and punctuation symbols tokenize individually. A 50-page technical whitepaper, compliance report, or financial filing can consume 30,000 to 45,000 tokens before taking conversation history or agent system instructions into account. In technical specifications, Cohere context length defines the combined boundary for instructions, retrieved context, and completions.

Shared Input and Output Budgets

A frequent architectural misconception is treating the 128,000-token capacity as an input-only buffer. Across Cohere Command models, input prompt tokens and generated completion tokens share the exact same context window envelope. Understanding the overall Cohere token limit is critical when planning multi-step agent interactions, because prompt bloat directly reduces the space available for responses.

Total sequence consumption follows a strict mathematical boundary:

Total Sequence Tokens = Input Prompt Tokens + Output Completion Tokens <= 128,000 tokens

If an application dispatches a prompt containing 125,000 tokens, the model can generate at most 3,000 output tokens before hitting the sequence ceiling, regardless of whether the client request specified a higher output parameter. Preserving adequate headroom within the 128,000-token sequence is necessary to avoid abrupt truncation of generated answers.

Maximum Output Token Caps

In addition to the total context window, Cohere enforces a dedicated output token limit per generation. Both Command R and Command R+ cap completions at 4,000 tokens per API call. When configuring API requests, setting the Cohere max tokens parameter controls response length while remaining subject to this 4,000-token ceiling.

This 4,000-token output limit establishes an important boundary for enterprise workflows:

• Single-Turn Analytical Reports: An agent cannot generate an entire 20-page document or emit 8,000 lines of refactored code in a single turn. Long-form generation requires multi-turn chunking or delegating document composition to dedicated file-writing tools.
• Completion Headroom Collisions: If your prompt consumes 126,000 tokens and you request max_tokens: 4000, the model cannot satisfy the requested completion size. The request will generate only 2,000 tokens before exhausting the 128,000 sequence limit.

Cohere Model Family Specifications

Cohere provides several models tailored for enterprise reasoning, retrieval, and multilingual processing. The table below compares context windows, maximum output limits, and architectural roles across the Cohere model lineup as of September 2026.

Command R and Command R+ remain Cohere's foundation for conversational search and retrieval, and their 128k window provides sufficient space for multi-document comparisons, extended tool call definitions, and multi-turn conversational history. The Command A family raises the ceiling for the workloads that need it, though it also shows that a wider window and a larger output allowance do not move together: Command A pairs a 256,000 token window with only an 8,000 token completion cap.

• Google Gemini Context Window: Token Limits, Architecture, and Handling Large FilesThe Google Gemini context window spans 1,048,576 input tokens and 65,536 output tokens on current Gemini 3 models such...
• Grok Context Window: Token Limits, Architecture, and Large File HandlingThe Grok context window spans from 256,000 tokens on grok-build-0.1 up to 1,000,000 tokens on Grok 4.3 and the Grok...
• Managing the LangGraph Context Window in Multi-Agent WorkflowsThe LangGraph context window represents the aggregate token limit imposed by the underlying LLM on all accumulated...
• GPT-4o Context Window: Token Limits, Architecture, and MCP SearchThe GPT-4o context window is 128,000 tokens, supporting up to 16,384 completion tokens per API request. While 128,000...
• AWS Bedrock Context Window: Token Limits, Model Capacities, and Memory ArchitectureThe AWS Bedrock context window defines the maximum sequence of input and output tokens a hosted foundation model...
• DeepSeek Context Window: Token Limits, Truncation, and Workspace RetrievalThe DeepSeek context window spans 64,000 to 128,000 tokens on DeepSeek-V3 and R1 before requests hit truncation errors....

More on this subject: Agent Memory and Storage (209 guides)

The 128,000 tokens on Command R+ are a shared quota, not an input allowance: the system prompt, the conversation history, the retriever results, and the user query all draw from the same number. Understanding how the Cohere Chat API handles large payloads is essential for maintaining consistent response quality in production environments.

In typical retrieval-augmented generation pipelines, developers pass retrieved document snippets directly to Cohere's Chat endpoint using the documents parameter. The API formats these snippets into the model prompt and returns grounded responses containing fine-grained citations that link generated statements to specific source snippets.

The Silent Truncation Trap

When prompt payloads grow too large, many API providers reject the request with an HTTP 400 or context length validation error. Cohere's Chat API behaves differently by default when handling the documents parameter.

If the combined token count of the system prompt, chat history, user message, and document list exceeds 128,000 tokens, Cohere silently truncates documents from the end of the list. The inference engine discards the lowest-ranked or oldest document chunks until the remaining payload fits inside the 128,000 token limit. The API returns an HTTP 200 status code, and generation proceeds as though all documents were evaluated.

This silent truncation introduces operational risks:

• Missing Evidence: If an enterprise search query retrieves 30 document chunks and the final 10 chunks are dropped, the model cannot synthesize facts contained in the omitted text.
• Unnoticed Hallucinations: Because the API returns a standard completion without an error code, developers assume the model inspected the entire corpus. If the query asks for details present only in the discarded chunks, the model may state that the information is unavailable or generate an unsupported assumption.
• Degraded Citation Quality: Grounded citations depend on the active document set. When documents are truncated silently, citation coverage drops, undermining auditability in regulated industries.

Auditing Truncation via the Response Payload

To prevent silent truncation from corrupting enterprise workflows, engineering teams must inspect response metadata programmatically. Cohere surfaces two critical fields in the chat completion object:

1. response.usage.input_tokens: Reports the exact number of tokens ingested for the prompt. If input_tokens approaches 124,000 tokens, the prompt is hovering near the sequence ceiling.
2. response.search_results: Enumerates the specific document items that the model ingested during the call.

In production code, developers should compare the length of response.search_results against the number of documents passed in the request:

import cohere
import os

co = cohere.ClientV2(api_key=os.environ["COHERE_API_KEY"])

documents = [
    {"id": "doc_1", "text": "Q3 gross margins expanded across cloud services."},
    {"id": "doc_2", "text": "Operating expenses rose due to hardware investments."},
]

response = co.chat(
    model="command-r-plus",
    messages=[{"role": "user", "content": "What were the primary margin drivers in Q3?"}],
    documents=documents,
    max_tokens=2000
)

if hasattr(response, "search_results") and response.search_results is not None:
    ingested_count = len(response.search_results)
    submitted_count = len(documents)
    if ingested_count < submitted_count:
        print(f"Warning: Silent truncation detected. Sent {submitted_count}, processed {ingested_count}.")

print(f"Input tokens: {response.usage.input_tokens}")
print(f"Output tokens: {response.usage.output_tokens}")

Configuring Prompt Truncation Parameters

When accessing Cohere models directly or through cloud platforms like Amazon Bedrock, developers can adjust prompt truncation behavior using the prompt_truncation setting:

• AUTO_PRESERVE_ORDER: The default behavior. When the prompt exceeds model capacity, the engine drops documents or earlier chat turns while attempting to preserve conversation order.
• OFF: Disables automated truncation. If the prompt exceeds 128,000 tokens, the API throws an explicit validation error instead of silently omitting data.

For enterprise applications where omission of facts is unacceptable, setting prompt_truncation="OFF" ensures that context overflow issues are caught and handled by retry logic or query refinement routines.

Faced with a 128,000 token context window, engineering teams often attempt to solve retrieval by passing entire document collections directly into the prompt. Rather than building a selective retrieval architecture, they append complete PDF transcripts, product manuals, and internal wikis to every query.

While a 128,000-token capacity can ingest approximately 95,000 words of text, context stuffing creates severe technical, economic, and operational drawbacks in production deployments.

Attention Degradation and the Lost-in-the-Middle Problem

Transformer architectures evaluate token relationships through self-attention matrices. Although Command R and Command R+ are fine-tuned for long sequences, needle-in-a-haystack evaluations show that retrieval accuracy is not uniform across 128,000 tokens.

Models exhibit highest retrieval precision when key facts appear near the beginning of the prompt (the system instructions) or near the end (the immediate user query). When critical information is buried in the middle of an extensive document dump, attention scores disperse across irrelevant background text. The model becomes more prone to overlooking subtleties, misattributing citations, or selecting superficial keyword matches over definitive statements.

Latency and Prefill Computation Overhead

Inference latency consists of two distinct phases: prefill time (the time required to process the input prompt) and decode time (the time required to generate output tokens autoregressively).

Processing a 100,000-token prompt requires massive matrix multiplications before the first response token can stream. Time-to-first-token (TTFT) latency scales with prompt size. In interactive workplace search tools or autonomous agent loops where an assistant makes several decisions in sequence, waiting multiple seconds for prompt prefill stalls the entire workflow.

Escalating API Token Costs

Context stuffing carries financial penalties. API providers bill for every token processed during prefill and generation.

Without retrieval filtering, multi-user queries re-send entire manuals repeatedly, multiplying token consumption across every turn. When an assistant queries an indexed repository, extracts only the relevant sections, and passes those focused passages to the model, daily token consumption falls substantially, cutting API expenditure while preventing context bloat.

In multi-turn agent conversations, context stuffing compounds rapidly. An agent executing a multi-step task resends accumulated context on every turn. Without active context discipline, multi-turn tool calling burns token budgets quickly.

Unstructured Document Formats

Enterprise knowledge does not exist as clean, pre-tokenized markdown text. It lives inside scanned PDF invoices, complex multi-tab financial spreadsheets, vendor agreements, and slide presentations.

Passing raw text dumps into a 128k context window strips away critical file metadata, layout geometry, tabular relationships, and version history. True enterprise search requires an indexing layer that parses structure, extracts typed data, and provides targeted retrieval.

Scale enterprise search without exceeding model context windows

Store large document collections in persistent Fast.io workspaces, index files automatically, and query relevant passages over MCP. Every organization starts with a 14-day free trial.

Start 14-Day Trial

Production agent architectures avoid raw context stuffing by separating persistent file storage from active model inference. Rather than cramming large archives into Cohere API payloads, organizations store files in persistent cloud workspaces and query relevant excerpts on demand using Model Context Protocol (MCP).

This architecture preserves the 128,000 token context window for reasoning, complex instructions, and multi-step tool use while providing access to document collections of any size.

Persistent Cloud Workspaces for Humans and Agents

Fast.io workspaces provide shared, org-owned environments where autonomous agents and human team members collaborate on the same files, directories, and shares.

Workspaces solve the fundamental limitation of model context windows by establishing a durable repository outside the LLM:

• Centralized File Ingestion: Upload documents directly or import existing corporate repositories from Google Drive, Dropbox, Box, and OneDrive without requiring local disk operations. Cloud sync ships for Dropbox, Box, and OneDrive. Google Drive supports import today, with sync coming soon.
• Granular Permissions: Maintain control over access rights across organization, workspace, folder, and file levels. Agents interact only with the directories they are granted permission to inspect.
• Per-File Version History: Every document keeps a full version history. When agents generate updated analysis files, summaries, or data extracts, team members can review changes and revert modifications at any time.

Automatic Indexing with Intelligence Mode

When files land in a Fast.io workspace, enabling Intelligence Mode activates automatic background indexing for Retrieval-Augmented Generation. The platform indexes documents on arrival, combining full-text keyword matching, semantic vectors, and metadata values into a unified hybrid search engine.

Rather than writing custom chunking scripts, spinning up external vector databases, and managing embedding pipelines, teams enable Intelligence Mode on the workspace. Connected assistants can then execute semantic queries against hundreds of thousands of pages, retrieving the exact paragraph needed for a specific question in milliseconds.

For structured data extraction, teams configure Metadata Views to convert contracts, invoices, and technical forms into queryable spreadsheets with typed schemas (Text, Integer, Decimal, Boolean, URL, JSON, Date & Time). Metadata Views extract structured fields without custom templates or fragile OCR rules, allowing Cohere agents to query structured records alongside semantic text.

Connecting Cohere Agents via Remote MCP

Agents interact with Fast.io workspaces using standard Model Context Protocol configuration. Fast.io exposes a remote MCP server accessible over Streamable HTTP at https://mcp.fast.io/mcp (or https://mcp.fast.io/mcp/key when using Bearer token authentication) alongside legacy SSE transport at https://mcp.fast.io/sse. Complete configuration details are documented in the agent storage guide.

Below is an example MCP configuration block for an autonomous agent or coding assistant:

{
  "mcpServers": {
    "fastio": {
      "url": "https://mcp.fast.io/mcp/key",
      "headers": {
        "Authorization": "Bearer YOUR_FASTIO_API_KEY"
      }
    }
  }
}

Through this consolidated MCP toolset, an agent running Cohere Command models executes targeted searches across workspace files:

1. The user asks a question about a complex corporate merger documented across 200 PDF filings.
2. The Cohere agent invokes the Fast.io storage tool using the search action with the user query.
3. The workspace index performs hybrid search and returns three relevant excerpts totaling 600 tokens.
4. The agent passes only those 600 tokens into Cohere's Chat API, leaving the vast majority of the context window available for reasoning and tool use.
5. Cohere generates an accurate, citation-grounded response without risking silent document truncation or inflating API bills.

Multi-Agent Coordination and Ownership Transfer

For production environments, Fast.io workspaces serve as a shared coordination layer. A fast, cost-effective model like Command R can extract data points, summarize incoming reports, and write structured files to the workspace. A second agent running Command R+ reads those summaries to perform complex multi-step reasoning.

Fast.io also supports ownership transfer. An agent can configure an organization, populate workspaces with indexed files, and establish shares on behalf of a human client, then transfer organizational ownership to the client while retaining administrative access. Every action taken by agents and humans is logged in an append-only audit log, providing visibility into document access and updates.

Every organization starts with a 14-day free trial, which requires a credit card. Paid subscriptions on Fast.io pricing include Starter at $29/mo, Business at $99/mo, and Enterprise at $299/mo.

Building reliable enterprise systems around Cohere Command R models requires combining model-level context discipline with external storage infrastructure. By applying clear token accounting and retrieval strategies, engineering teams maximize answer accuracy while controlling latency and costs.

1. Enforce Pre-Call Token Accounting

Never dispatch dynamic user queries or automated document retrievals without calculating prompt token size beforehand. Token budgets should be allocated explicitly across system components:

• System Prompt and Agent Persona: 500 to 1,500 tokens.
• Tool Definitions and Function Schemas: 1,000 to 3,000 tokens.
• Conversation History: 2,000 to 8,000 tokens (prune or summarize older turns).
• Retrieved Document Excerpts: 2,000 to 6,000 tokens (limit to top 3 to 5 ranked snippets).
• Reserved Output Headroom: 2,000 to 4,000 tokens.

By establishing strict caps on each segment, the total prompt payload stays well below 25,000 tokens, preserving substantial safety margin against unexpected input spikes.

2. Implement Two-Stage Retrieval with Reranking

When querying large enterprise corpuses, avoid passing first-stage vector search results directly to the LLM. Vector embeddings excel at broad semantic recall but can return noisy or tangentially related passages.

Adopt a two-stage retrieval architecture:

1. Stage 1 (Broad Retrieval): Use Fast.io workspace hybrid search to retrieve candidate passages matching the user query.
2. Stage 2 (Precision Reranking): Pass those candidates through Cohere Rerank 3.5. Rerank computes deep cross-attention between the query and each passage, reordering results by true relevance.
3. Prompt Assembly: Select only the top 3 to 5 reranked passages for inclusion in the Cohere Chat call. This reduces prompt token volume while delivering higher answer fidelity than raw context dumps.

3. Prune Multi-Turn Conversation History

In conversational workflows, chat history expands with every turn. If users engage in 15-turn dialogues with code snippets or tabular data, the conversation log quickly consumes tens of thousands of tokens.

Implement active history pruning:

• Sliding Window: Retain only the last 4 to 6 conversation turns in the active prompt.
• Workspace Summarization: When conversation history exceeds a defined threshold, trigger an asynchronous background call with Command R to summarize key decisions and user preferences. Save the running summary to a Collaborative Note or text file in the workspace, and prepend that condensed summary to subsequent turns.

4. Monitor Truncation and Finish Reasons

Continuously audit API responses in application telemetry. Inspect response.finish_reason on every call:

• COMPLETE: The model completed its response naturally.
• MAX_TOKENS: The generation collided with the output limit or the context window ceiling. If MAX_TOKENS appears frequently, increase max_tokens or reduce input prompt size.

Log the difference between submitted document counts and len(response.search_results). Alert engineering teams whenever discrepancies appear, ensuring silent document truncation is identified immediately.

5. Pair Command R and Command R+ by Task Complexity

Optimize compute budgets by dividing workloads between model tiers:

• Use Command R or Command R7B for routine tasks such as classifying support tickets, extracting basic metadata from documents, and generating short summaries.
• Reserve Command R+ for complex tasks requiring multi-step tool use, cross-document reconciliation, and detailed decision-making.

Both models share the same 128,000-token context window and 4,000-token output cap, allowing coordinated execution across model tiers within the same workspace infrastructure.