vLLM max_tokens Guide: API Configuration, KV Cache Tuning, and Out-of-Memory Fixes

A client requesting a 4,096-token completion against an engine configured with --max-model-len 4096 fails immediately if the input prompt contains even a single token. When prompt length plus requested output tokens exceeds the engine sequence boundary, vLLM rejects the payload with an HTTP 400 BadRequestError before scheduling a single forward pass. This failure illustrates the operational distinction between server-level context allocation and client-level output limits.

In vLLM, max_tokens defines the client-requested upper limit on generated output tokens per completion request, bounded by the server-side --max-model-len parameter.

When an application dispatches an inference request to vLLM, the generation loop continues until one of three conditions occurs:

1. The model outputs a designated End-of-Sequence (EOS) token, such as <|eot_id|> in Llama models or <|im_end|> in Qwen models.
2. The output matches a client-specified stop string or stop token ID.
3. The number of generated completion tokens reaches the value assigned to max_tokens.

When generation halts because it hit max_tokens, vLLM marks the response choice with finish_reason: "length". When it halts because the model completed its thought, the engine sets finish_reason: "stop".

The Default 16-Token Truncation Trap

A frequent source of confusion for engineers deploying vLLM is receiving truncated responses that stop mid-word or mid-sentence after only a couple of lines. In older vLLM releases and standard OpenAI text completion conventions, omitting max_tokens from a request defaults to 16 tokens (SamplingParams.max_tokens = 16).

This default originated in early benchmark harnesses designed to measure raw prompt processing speed without generating long outputs. When client code calls /v1/completions or initializes an offline SamplingParams object without specifying max_tokens, vLLM enforces this 16-token ceiling:

from vllm import SamplingParams

params = SamplingParams(temperature=0.7)
assert params.max_tokens == 16

In modern chat completions endpoints (/v1/chat/completions), vLLM permits max_tokens to be omitted or set to None, dynamically allowing generation to fill the remaining sequence capacity up to --max-model-len - prompt_tokens. However, when third-party SDKs, API gateways, or evaluation frameworks send explicit default values, the 16-token ceiling can still trigger unexpected truncations. For reasoning models that produce extended internal reasoning traces, OpenAI-compatible interfaces also recognize max_completion_tokens to govern output length.

Core Differences Across vLLM Sequence Parameters

Guides often treat --max-model-len and max_tokens interchangeably, confusing server VRAM pre-allocation with per-request completion length. The following table differentiates the scope, target, and operational behavior of these parameters:

According to official vLLM documentation on engine arguments, --max-model-len defines the absolute boundary: "Model context length (prompt and output). If unspecified, will be automatically derived from the model config." When set to -1 or auto, the engine selects the largest context length that fits into physical GPU memory. While --max-model-len allocates the global capacity of the engine, max_tokens dictates the generation budget of individual requests.

• vLLM max-model-len: Tuning Context Length, VRAM, and MCP SearchIn vLLM, max-model-len defines the total token sequence length for prompts and outputs, directly governing GPU KV cache...
• LM Studio Context Length: Configuration, VRAM Tuning, and MCP RetrievalLM Studio context length dictates how much local GPU VRAM and system memory are allocated for the attention KV cache...
• Ollama Max Tokens: Setting Generation Limits and Context BudgetsIn Ollama, setting generation limits requires configuring the num_predict parameter, which sets the maximum number of...
• Salesforce Agentforce Limits: Tokens, Context Windows, and Grounding File CapsAgentforce limits establish operational boundaries for autonomous AI across Salesforce, including a 1.5 million hourly...
• Mistral Token Limits: API Parameters, Generation Ceilings, and Workspace ArchitectureA Mistral token limit specifies the maximum number of tokens accepted in an API request payload or emitted during a...
• HTTP 413 Request Entity Too Large: Causes, Server Limits & WorkaroundsAn HTTP 413 request entity too large error occurs when a client payload exceeds buffer limits configured on a web...

More on this subject: AI Agents: General Guides (172 guides)

Operators and developers interact with vLLM through two primary interfaces: the native Python offline engine (vllm.LLM) and the OpenAI-compatible HTTP serving layer (vllm serve). Each interface exposes specific conventions for configuring output token limits.

Python Offline Inference with SamplingParams

When using vLLM for batch inference, evaluation scripts, or data pipelines, generation parameters are configured through SamplingParams:

from vllm import LLM, SamplingParams

llm = LLM(
    model="meta-llama/Llama-3.1-8B-Instruct",
    max_model_len=8192,
    gpu_memory_utilization=0.90,
)

sampling_params = SamplingParams(
    max_tokens=2048,
    min_tokens=32,
    temperature=0.2,
    stop=["<|eot_id|>"],
)

prompts = [
    "Summarize the mechanical tolerances required for optical telescope mirror mounts."
]

outputs = llm.generate(prompts, sampling_params)

for output in outputs:
    print("Generated tokens:", len(output.outputs[0].token_ids))
    print("Finish reason:", output.outputs[0].finish_reason)
    print(output.outputs[0].text)

In this configuration, max_tokens=2048 establishes an upper limit. If the model produces the stop string <|eot_id|> after 450 tokens, generation stops immediately and returns finish_reason: "stop". If the prompt is 7,000 tokens long, vLLM checks 7000 + 2048 = 9048, which exceeds max_model_len=8192. The offline engine raises a ValueError during schedule validation.

OpenAI-Compatible API Serving Layer

In production deployments, vLLM runs as a persistent service exposing an HTTP API. Start the server using vllm serve with an explicit sequence capacity:

vllm serve meta-llama/Llama-3.1-8B-Instruct   --max-model-len 8192   --gpu-memory-utilization 0.92   --port 8000

Client applications connect using the official openai Python SDK or any standard HTTP client:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="vllm-local-credential",
)

response = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[
        {"role": "system", "content": "You are a technical editor."},
        {"role": "user", "content": "Draft an API documentation section for rate limiting."},
    ],
    max_tokens=1024,
    temperature=0.1,
)

print(response.choices[0].message.content)
print("Finish reason:", response.choices[0].finish_reason)

The HTTP 400 Context Ceiling Error

When a client application submits a request where the input prompt tokens plus the requested max_tokens exceeds the server's configured --max-model-len, vLLM rejects the call immediately:

BadRequestError: Error code: 400 - Context length exceeded. Requested tokens exceed maximum sequence length.

This error occurs frequently in microservices architectures where client applications send static default parameters (such as max_tokens=4096) alongside user-submitted documents. If a user pastes a 5,000-token document, the combined request length of 9,096 tokens breaks the 8,192 server boundary, triggering an application error.

Client-Side Dynamic Token Budgeting

To prevent HTTP 400 exceptions from reaching end users, production clients implement dynamic token budgeting. The client counts input prompt tokens before dispatching the request, clamping max_tokens to fit the available context headroom:

from openai import OpenAI

SERVER_MAX_MODEL_LEN = 8192
DESIRED_COMPLETION_TOKENS = 2048
SAFETY_MARGIN = 16

def create_completion_safe(client: OpenAI, model: str, messages: list, estimated_prompt_tokens: int):
    available_headroom = SERVER_MAX_MODEL_LEN - estimated_prompt_tokens - SAFETY_MARGIN
    if available_headroom <= 0:
        raise ValueError("Prompt exceeds server maximum context capacity.")
    effective_max_tokens = min(DESIRED_COMPLETION_TOKENS, available_headroom)
    return client.chat.completions.create(
        model=model,
        messages=messages,
        max_tokens=effective_max_tokens,
        temperature=0.2,
    )

Dynamic clamping prevents requests from violating engine limits while preserving maximum output capacity for shorter prompts.

Understanding why max_tokens impacts hardware stability requires examining how vLLM allocates GPU memory during generation. While --max-model-len determines the global pool size of KV cache blocks initialized at engine startup, max_tokens dictates how long active sequences occupy those blocks during execution.

PagedAttention Mechanics During Generation

In transformer architectures, computing attention requires storing Key and Value vector projections for all preceding tokens in the sequence. To eliminate memory fragmentation, vLLM implements PagedAttention, which partitions the KV cache into fixed-size physical memory blocks (defaulting to 16 tokens per block).

Inference executes in two distinct phases:

1. Prefill Phase: The engine processes the full input prompt in parallel, allocating KV cache blocks to store prompt tokens.
2. Decode Phase: The engine generates output tokens one by one autoregressively. Each newly generated token requires appending its Key and Value states to the sequence's cache.

As decoding advances toward max_tokens, the sequence periodically exhausts its current 16-token block and requests an unallocated block from the GPU pool.

Calculating KV Cache Memory per Token

The physical memory required to store KV cache states depends on the model architecture:

Bytes per Token = 2 * Num_Layers * Num_KV_Heads * Head_Dimension * Element_Size_Bytes

The multiplier of 2 accounts for separate Key and Value tensors. For a model using Grouped Query Attention (GQA) such as Llama 3.1 8B:

• Number of transformer layers: 32
• Number of Key-Value heads: 8
• Head dimension: 128
• Data type: 16-bit floating point (BF16 or FP16), taking 2 bytes per value

Calculating the byte requirement:

Bytes per Token = 2 * 32 * 8 * 128 * 2 = 131,072 bytes (128 KB)

Every single token in the sequence consumes 128 kilobytes of GPU video RAM. When a request generates 2,048 completion tokens, the decode phase alone consumes:

2,048 tokens * 128 KB/token = 262,144 KB = 256 MB

If an API server hosts 32 concurrent requests that each generate 2,048 tokens, the active decode tokens claim 8 gigabytes of GPU VRAM strictly for completion states, on top of initial prompt KV blocks and base model weights.

The Preemption and Latency Cascade

Because memory is finite, setting excessively large max_tokens values across concurrent clients introduces scheduling bottlenecks:

1. Block Depletion: When traffic spikes and many clients generate long responses simultaneously, free GPU memory blocks drop toward zero.
2. Request Preemption: When the scheduler cannot find a free block for a running sequence, it must preempt an active request to prevent an immediate CUDA crash.
3. CPU Swapping or Recomputation: The scheduler either evicts the sequence's KV blocks to system RAM via --swap-space or discards the decode blocks entirely. When memory becomes free again, the preempted sequence must reload its blocks or recompute its prefill from scratch.
4. Tail Latency Spikes: Swapping and recomputation destroy generation throughput, causing Time-to-First-Token (TTFT) and Inter-Token Latency (ITL) to surge.

Restricting client max_tokens to realistic output lengths preserves free blocks in the PagedAttention pool, preventing memory thrashing and keeping server throughput predictable.

Attempting to solve document question-answering by expanding --max-model-len to 64,000 or 128,000 tokens introduces severe operational penalties. While extended sequence models can mathematically accept massive prompts, packing whole files, technical specifications, and corporate manuals into the prompt creates four critical issues:

• KV Cache Exhaustion: Large prompts consume gigabytes of VRAM before the first output token is generated, leaving minimal memory headroom for client max_tokens generation.
• Degraded First-Token Latency: Processing 50,000 prompt tokens requires substantial computation, keeping clients waiting seconds before generation begins.
• Attention Degradation: Long prompts experience retrieval loss, where models struggle to locate factual answers buried in the middle of extensive context blocks.
• Higher Execution Cost: Re-processing static reference documents on every inference call wastes GPU compute cycles.

The sustainable architectural pattern decouples static document storage from the model context window. Instead of loading raw document archives into prompts, teams maintain their corpus in an external retrieval substrate and supply the model with targeted search tools. Review the agent storage guidelines to understand how autonomous systems structure persistent knowledge.

Integrating vLLM with Fastio Intelligent Workspaces via MCP

Fastio provides intelligent workspaces built for collaboration between people and software agents. By storing reference materials in a Fastio workspace and querying them through the Model Context Protocol (MCP), your application provides models with precise, factual context without inflating --max-model-len.

The operational workflow operates as follows:

1. Store Reference Files in Shared Workspaces: Upload technical manuals, research archives, policy documents, and codebases to an organization-owned workspace. Fastio supports direct file uploads and scheduled or on-demand Cloud Sync from Google Drive, Dropbox, Box, or OneDrive.
2. Automatic Hybrid Indexing: When workspace intelligence is enabled, incoming documents are automatically indexed for hybrid search, combining exact keyword matching with semantic vector retrieval. No separate vector database setup, chunking script, or embedding pipeline is required.
3. Connect Assistants via Remote MCP: Fastio provides a consolidated MCP toolset through its remote server at https://mcp.fast.io/mcp/tools using Streamable HTTP. Local and cloud agents connect with an API key, without installing local npm packages or running background bridges. Review the Fastio agent documentation and MCP setup documentation for endpoint parameters.
4. Precision Context Retrieval: When an agent receives a query, it calls the MCP search tool against the workspace. Fastio returns focused, citation-backed excerpts consuming only 500 to 1,500 tokens of prompt context.
5. Fast Local Generation: The local vLLM instance processes the compact prompt within an efficient 8,192 token context window, leaving ample headroom for max_tokens output without GPU memory contention.

{
  "mcpServers": {
    "fastio": {
      "url": "https://mcp.fast.io/mcp/tools",
      "headers": {
        "Authorization": "Bearer YOUR_FASTIO_API_KEY"
      }
    }
  }
}

When teams need structured data from unstructured contracts, financial receipts, or scanned records, Fastio Metadata Views transform documents into live, queryable tables. Users define schema fields in natural language (Text, Integer, Decimal, Boolean, URL, JSON, Date & Time), and the system populates structured tables across PDFs, spreadsheets, and scanned documents without OCR templates. Agents inspect views and extract structured data via MCP.

Every file in a Fastio workspace retains per-file version history. If an agent or human colleague updates a document, earlier versions remain accessible, and the detailed activity log records the modification.

Monthly plans start with a 14-day free trial that requires a credit card. Plans are Starter at $9.99/mo, Business at $49.99/mo, and Enterprise at $199.99/mo on Fastio pricing. Offloading large document corpora to an intelligent workspace keeps inference infrastructure lean, stable, and responsive.

Scale Agent Output Without Exhausting GPU Memory

Keep vLLM tuned for fast token generation while offloading document libraries to a shared Fastio workspace with automatic hybrid search. Query indexed files through the remote MCP server and eliminate context bloat. Starts with a 14-day free trial.

Start 14-Day Trial

When tuning vLLM for production inference, engineers encounter two distinct categories of Out-of-Memory (OOM) failures: initialization crashes and runtime decode exhaustion. Applying the appropriate configuration flags stabilizes memory consumption without sacrificing generation quality.

Initialization OOM: Resolving Cache Block Failures

If vLLM crashes during startup with a message stating there is no available memory for cache blocks:

ValueError: No available memory for the cache blocks. Try increasing gpu_memory_utilization or decreasing max_model_len.

This error indicates that the combined memory required for model weights, CUDA runtime kernels, and the minimum required KV cache blocks for --max-model-len exceeds the configured GPU memory budget.

To fix initialization crashes:

• Reduce --max-model-len: Lower the context length from its theoretical default (such as 131,072) to an operational ceiling like 8,192 or 4,096.
• Adjust --gpu-memory-utilization: The default ratio is 0.90. On a dedicated GPU where no other processes run, raising this value to 0.94 or 0.95 allocates additional physical VRAM to the KV cache pool:

vllm serve meta-llama/Llama-3.1-8B-Instruct --max-model-len 8192 --gpu-memory-utilization 0.95

• Shard Model Across GPUs: For larger models such as 70B parameter architectures, use --tensor-parallel-size to distribute weights and KV heads across multiple devices:

vllm serve meta-llama/Llama-3.1-70B-Instruct --tensor-parallel-size 4 --max-model-len 8192

Runtime OOM: Optimizing Decode Concurrency

Runtime CUDA OOM errors occur during generation when active sequences demand more memory blocks than the GPU can provide. While PagedAttention prevents memory fragmentation, bursts of concurrent requests with large max_tokens can exhaust physical capacity.

Key configuration levers to stabilize runtime generation:

Quantizing the KV Cache to FP8

By default, vLLM stores KV cache blocks in 16-bit precision matching the model weights. Enabling 8-bit floating point quantization via --kv-cache-dtype fp8 cuts the memory footprint per token in half:

vllm serve meta-llama/Llama-3.1-8B-Instruct   --max-model-len 8192   --kv-cache-dtype fp8   --gpu-memory-utilization 0.92

For Llama 3.1 8B, FP8 quantization lowers the KV cache cost from 128 kilobytes per token to 64 kilobytes per token. An 8,192-token sequence consumes 512 megabytes instead of one gigabyte, doubling the concurrent generation capacity of the server.

Capping Concurrency with --max-num-seqs

The --max-num-seqs argument restricts the maximum number of concurrent sequences the scheduler packs into an iteration (defaulting to 256). For workloads with large max_tokens completions, lowering this value to 64 or 128 prevents the scheduler from over-admitting requests that compete for decode blocks:

vllm serve meta-llama/Llama-3.1-8B-Instruct   --max-model-len 8192   --max-num-seqs 64

Regulating Forward-Pass Batching with --max-num-batched-tokens

The --max-num-batched-tokens parameter dictates the maximum number of tokens processed in a single forward pass across both prefill and decode phases (defaulting to 2,048 or derived from context). When chunked prefill is active (--enable-chunked-prefill), long input prompts are broken into chunks matching this token budget, ensuring that ongoing decode iterations are not starved by heavy prompt arrivals.

Production Configuration Matrix

The following matrix provides verified parameter profiles based on target workload characteristics:

Monitoring and Alerting Checklist

Production deployments require monitoring key metrics exposed at vLLM's /metrics Prometheus endpoint:

1. vllm:gpu_cache_usage_factor: The fraction of GPU KV cache blocks currently in use. When this metric consistently exceeds 0.90, the server is nearing capacity. An alert should trigger at 0.95 to signal impending preemption.
2. vllm:num_requests_waiting: The number of requests queued in the scheduler waiting for free memory blocks. Spikes in this metric indicate that active requests with long max_tokens are monopolizing decode slots.
3. vllm:num_preemptions_total: A counter tracking how many running requests were paused or swapped. In a well-tuned system, this counter remains zero. If preemptions occur, reduce --max-num-seqs or lower client max_tokens.
4. vllm:cpu_cache_usage_factor: The fraction of CPU swap memory occupied by evicted blocks. Sustained usage indicates that the GPU KV cache is undersized for current request volume.