# vLLM max_tokens Guide: API Configuration, KV Cache Tuning, and Out-of-Memory Fixes

In vLLM, max_tokens defines the client-requested upper limit on generated output tokens per completion request, bounded by the server-side --max-model-len parameter. Misconfiguring this setting causes abrupt 16-token response truncations or immediate HTTP 400 context boundary errors. Tuning output token limits alongside KV cache quantization and offloading large document archives to external MCP search stabilizes GPU memory under high concurrency.

Source: https://fast.io/resources/vllm-max-tokens/
Author: [Tom Langridge](https://fast.io/authors/tom-langridge/)
Last reviewed: 2026-10-12

## What Is the vLLM max_tokens Parameter?

A client requesting a 4,096-token completion against an engine configured with `--max-model-len 4096` fails immediately if the input prompt contains even a single token. When prompt length plus requested output tokens exceeds the engine sequence boundary, vLLM rejects the payload with an HTTP 400 `BadRequestError` before scheduling a single forward pass. This failure illustrates the operational distinction between server-level context allocation and client-level output limits.

In vLLM, max_tokens defines the client-requested upper limit on generated output tokens per completion request, bounded by the server-side --max-model-len parameter.

When an application dispatches an inference request to vLLM, the generation loop continues until one of three conditions occurs:

1. The model outputs a designated End-of-Sequence (EOS) token, such as `<|eot_id|>` in Llama models or `<|im_end|>` in Qwen models.
2. The output matches a client-specified stop string or stop token ID.
3. The number of generated completion tokens reaches the value assigned to `max_tokens`.

When generation halts because it hit `max_tokens`, vLLM marks the response choice with `finish_reason: "length"`. When it halts because the model completed its thought, the engine sets `finish_reason: "stop"`.

### The Default 16-Token Truncation Trap

A frequent source of confusion for engineers deploying vLLM is receiving truncated responses that stop mid-word or mid-sentence after only a couple of lines. In older vLLM releases and standard OpenAI text completion conventions, omitting `max_tokens` from a request defaults to 16 tokens (`SamplingParams.max_tokens = 16`).

This default originated in early benchmark harnesses designed to measure raw prompt processing speed without generating long outputs. When client code calls `/v1/completions` or initializes an offline `SamplingParams` object without specifying `max_tokens`, vLLM enforces this 16-token ceiling:

```python
from vllm import SamplingParams

params = SamplingParams(temperature=0.7)
assert params.max_tokens == 16
```

In modern chat completions endpoints (`/v1/chat/completions`), vLLM permits `max_tokens` to be omitted or set to `None`, dynamically allowing generation to fill the remaining sequence capacity up to `--max-model-len - prompt_tokens`. However, when third-party SDKs, API gateways, or evaluation frameworks send explicit default values, the 16-token ceiling can still trigger unexpected truncations. For reasoning models that produce extended internal reasoning traces, OpenAI-compatible interfaces also recognize `max_completion_tokens` to govern output length.

### Core Differences Across vLLM Sequence Parameters

Guides often treat `--max-model-len` and `max_tokens` interchangeably, confusing server VRAM pre-allocation with per-request completion length. The following table differentiates the scope, target, and operational behavior of these parameters:

| Parameter | Scope | Target Tokens | Memory Allocation Impact | Behavior When Exceeded |
|---|---|---|---|---|
| `--max-model-len` | Global engine startup flag | Input prompt plus output completion combined | Determines total KV cache block pool pre-allocated in GPU VRAM | Server rejects request immediately with HTTP 400 error |
| `max_tokens` | Per-request sampling parameter | Newly generated completion tokens only | Determines how long active sequences occupy KV cache blocks | Generation stops with `finish_reason: "length"` |
| `--max-num-batched-tokens` | Engine scheduler configuration | Tokens evaluated in a single forward pass | Caps iteration tensor dimensions without changing total cache | Prompts chunked across multiple scheduler iterations |

According to official vLLM documentation on engine arguments, `--max-model-len` defines the absolute boundary: "Model context length (prompt and output). If unspecified, will be automatically derived from the model config." When set to `-1` or `auto`, the engine selects the largest context length that fits into physical GPU memory. While `--max-model-len` allocates the global capacity of the engine, `max_tokens` dictates the generation budget of individual requests.

## Configuring max_tokens Across Python SDK and API Endpoints

Operators and developers interact with vLLM through two primary interfaces: the native Python offline engine (`vllm.LLM`) and the OpenAI-compatible HTTP serving layer (`vllm serve`). Each interface exposes specific conventions for configuring output token limits.

### Python Offline Inference with SamplingParams

When using vLLM for batch inference, evaluation scripts, or data pipelines, generation parameters are configured through `SamplingParams`:

```python
from vllm import LLM, SamplingParams

llm = LLM(
    model="meta-llama/Llama-3.1-8B-Instruct",
    max_model_len=8192,
    gpu_memory_utilization=0.90,
)

sampling_params = SamplingParams(
    max_tokens=2048,
    min_tokens=32,
    temperature=0.2,
    stop=["<|eot_id|>"],
)

prompts = [
    "Summarize the mechanical tolerances required for optical telescope mirror mounts."
]

outputs = llm.generate(prompts, sampling_params)

for output in outputs:
    print("Generated tokens:", len(output.outputs[0].token_ids))
    print("Finish reason:", output.outputs[0].finish_reason)
    print(output.outputs[0].text)
```

In this configuration, `max_tokens=2048` establishes an upper limit. If the model produces the stop string `<|eot_id|>` after 450 tokens, generation stops immediately and returns `finish_reason: "stop"`. If the prompt is 7,000 tokens long, vLLM checks `7000 + 2048 = 9048`, which exceeds `max_model_len=8192`. The offline engine raises a `ValueError` during schedule validation.

### OpenAI-Compatible API Serving Layer

In production deployments, vLLM runs as a persistent service exposing an HTTP API. Start the server using `vllm serve` with an explicit sequence capacity:

```bash
vllm serve meta-llama/Llama-3.1-8B-Instruct   --max-model-len 8192   --gpu-memory-utilization 0.92   --port 8000
```

Client applications connect using the official `openai` Python SDK or any standard HTTP client:

```python
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="vllm-local-credential",
)

response = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[
        {"role": "system", "content": "You are a technical editor."},
        {"role": "user", "content": "Draft an API documentation section for rate limiting."},
    ],
    max_tokens=1024,
    temperature=0.1,
)

print(response.choices[0].message.content)
print("Finish reason:", response.choices[0].finish_reason)
```

### The HTTP 400 Context Ceiling Error

When a client application submits a request where the input prompt tokens plus the requested `max_tokens` exceeds the server's configured `--max-model-len`, vLLM rejects the call immediately:

```text
BadRequestError: Error code: 400 - Context length exceeded. Requested tokens exceed maximum sequence length.
```

This error occurs frequently in microservices architectures where client applications send static default parameters (such as `max_tokens=4096`) alongside user-submitted documents. If a user pastes a 5,000-token document, the combined request length of 9,096 tokens breaks the 8,192 server boundary, triggering an application error.

### Client-Side Dynamic Token Budgeting

To prevent HTTP 400 exceptions from reaching end users, production clients implement dynamic token budgeting. The client counts input prompt tokens before dispatching the request, clamping `max_tokens` to fit the available context headroom:

```python
from openai import OpenAI

SERVER_MAX_MODEL_LEN = 8192
DESIRED_COMPLETION_TOKENS = 2048
SAFETY_MARGIN = 16

def create_completion_safe(client: OpenAI, model: str, messages: list, estimated_prompt_tokens: int):
    available_headroom = SERVER_MAX_MODEL_LEN - estimated_prompt_tokens - SAFETY_MARGIN
    if available_headroom <= 0:
        raise ValueError("Prompt exceeds server maximum context capacity.")
    effective_max_tokens = min(DESIRED_COMPLETION_TOKENS, available_headroom)
    return client.chat.completions.create(
        model=model,
        messages=messages,
        max_tokens=effective_max_tokens,
        temperature=0.2,
    )
```

Dynamic clamping prevents requests from violating engine limits while preserving maximum output capacity for shorter prompts.

## How Output Token Length Affects KV Cache Allocation and Concurrency

Understanding why `max_tokens` impacts hardware stability requires examining how vLLM allocates GPU memory during generation. While `--max-model-len` determines the global pool size of KV cache blocks initialized at engine startup, `max_tokens` dictates how long active sequences occupy those blocks during execution.

### PagedAttention Mechanics During Generation

In transformer architectures, computing attention requires storing Key and Value vector projections for all preceding tokens in the sequence. To eliminate memory fragmentation, vLLM implements PagedAttention, which partitions the KV cache into fixed-size physical memory blocks (defaulting to 16 tokens per block).

Inference executes in two distinct phases:

1. **Prefill Phase:** The engine processes the full input prompt in parallel, allocating KV cache blocks to store prompt tokens.
2. **Decode Phase:** The engine generates output tokens one by one autoregressively. Each newly generated token requires appending its Key and Value states to the sequence's cache.

As decoding advances toward `max_tokens`, the sequence periodically exhausts its current 16-token block and requests an unallocated block from the GPU pool.

### Calculating KV Cache Memory per Token

The physical memory required to store KV cache states depends on the model architecture:

```text
Bytes per Token = 2 * Num_Layers * Num_KV_Heads * Head_Dimension * Element_Size_Bytes
```

The multiplier of 2 accounts for separate Key and Value tensors. For a model using Grouped Query Attention (GQA) such as Llama 3.1 8B:

* Number of transformer layers: 32
* Number of Key-Value heads: 8
* Head dimension: 128
* Data type: 16-bit floating point (BF16 or FP16), taking 2 bytes per value

Calculating the byte requirement:

```text
Bytes per Token = 2 * 32 * 8 * 128 * 2 = 131,072 bytes (128 KB)
```

Every single token in the sequence consumes 128 kilobytes of GPU video RAM. When a request generates 2,048 completion tokens, the decode phase alone consumes:

```text
2,048 tokens * 128 KB/token = 262,144 KB = 256 MB
```

If an API server hosts 32 concurrent requests that each generate 2,048 tokens, the active decode tokens claim 8 gigabytes of GPU VRAM strictly for completion states, on top of initial prompt KV blocks and base model weights.

| Context and Output Setting | Total Tokens | KV Cache per Sequence (BF16) | VRAM for 16 Concurrent Sequences | VRAM for 64 Concurrent Sequences |
|---|---|---|---|---|
| Prompt 1,024 / Output 512 | 1,536 tokens | 192 MB | 3.0 GB | 12.0 GB |
| Prompt 2,048 / Output 1,024 | 3,072 tokens | 384 MB | 6.0 GB | 24.0 GB |
| Prompt 4,096 / Output 2,048 | 6,144 tokens | 768 MB | 12.0 GB | 48.0 GB |
| Prompt 6,144 / Output 2,048 | 8,192 tokens | 1,024 MB (1 GB) | 16.0 GB | 64.0 GB |
| Prompt 16,384 / Output 4,096 | 20,480 tokens | 2,560 MB (2.5 GB) | 40.0 GB | 160.0 GB |

### The Preemption and Latency Cascade

Because memory is finite, setting excessively large `max_tokens` values across concurrent clients introduces scheduling bottlenecks:

1. **Block Depletion:** When traffic spikes and many clients generate long responses simultaneously, free GPU memory blocks drop toward zero.
2. **Request Preemption:** When the scheduler cannot find a free block for a running sequence, it must preempt an active request to prevent an immediate CUDA crash.
3. **CPU Swapping or Recomputation:** The scheduler either evicts the sequence's KV blocks to system RAM via `--swap-space` or discards the decode blocks entirely. When memory becomes free again, the preempted sequence must reload its blocks or recompute its prefill from scratch.
4. **Tail Latency Spikes:** Swapping and recomputation destroy generation throughput, causing Time-to-First-Token (TTFT) and Inter-Token Latency (ITL) to surge.

Restricting client `max_tokens` to realistic output lengths preserves free blocks in the PagedAttention pool, preventing memory thrashing and keeping server throughput predictable.

## Why Offload Large Document Corpora to MCP Search Workspaces

Attempting to solve document question-answering by expanding `--max-model-len` to 64,000 or 128,000 tokens introduces severe operational penalties. While extended sequence models can mathematically accept massive prompts, packing whole files, technical specifications, and corporate manuals into the prompt creates four critical issues:

* **KV Cache Exhaustion:** Large prompts consume gigabytes of VRAM before the first output token is generated, leaving minimal memory headroom for client `max_tokens` generation.
* **Degraded First-Token Latency:** Processing 50,000 prompt tokens requires substantial computation, keeping clients waiting seconds before generation begins.
* **Attention Degradation:** Long prompts experience retrieval loss, where models struggle to locate factual answers buried in the middle of extensive context blocks.
* **Higher Execution Cost:** Re-processing static reference documents on every inference call wastes GPU compute cycles.

The sustainable architectural pattern decouples static document storage from the model context window. Instead of loading raw document archives into prompts, teams maintain their corpus in an external retrieval substrate and supply the model with targeted search tools. Review the [agent storage guidelines](/storage-for-agents/) to understand how autonomous systems structure persistent knowledge.

### Integrating vLLM with Fastio Intelligent Workspaces via MCP

Fastio provides [intelligent workspaces](/product/workspaces/) built for collaboration between people and software agents. By storing reference materials in a Fastio workspace and querying them through the Model Context Protocol (MCP), your application provides models with precise, factual context without inflating `--max-model-len`.

The operational workflow operates as follows:

1. **Store Reference Files in Shared Workspaces:** Upload technical manuals, research archives, policy documents, and codebases to an organization-owned workspace. Fastio supports direct file uploads and scheduled or on-demand Cloud Sync from Google Drive, Dropbox, Box, or OneDrive.
2. **Automatic Hybrid Indexing:** When [workspace intelligence](/product/ai/) is enabled, incoming documents are automatically indexed for hybrid search, combining exact keyword matching with semantic vector retrieval. No separate vector database setup, chunking script, or embedding pipeline is required.
3. **Connect Assistants via Remote MCP:** Fastio provides a consolidated MCP toolset through its remote server at `https://mcp.fast.io/mcp/tools` using Streamable HTTP. Local and cloud agents connect with an API key, without installing local npm packages or running background bridges. Review the [Fastio agent documentation](/storage-for-agents/) and [MCP setup documentation](https://mcp.fast.io/docs) for endpoint parameters.
4. **Precision Context Retrieval:** When an agent receives a query, it calls the MCP search tool against the workspace. Fastio returns focused, citation-backed excerpts consuming only 500 to 1,500 tokens of prompt context.
5. **Fast Local Generation:** The local vLLM instance processes the compact prompt within an efficient 8,192 token context window, leaving ample headroom for `max_tokens` output without GPU memory contention.

```json
{
  "mcpServers": {
    "fastio": {
      "url": "https://mcp.fast.io/mcp/tools",
      "headers": {
        "Authorization": "Bearer YOUR_FASTIO_API_KEY"
      }
    }
  }
}
```

When teams need structured data from unstructured contracts, financial receipts, or scanned records, Fastio [Metadata Views](/product/document-data-extraction/) transform documents into live, queryable tables. Users define schema fields in natural language (Text, Integer, Decimal, Boolean, URL, JSON, Date & Time), and the system populates structured tables across PDFs, spreadsheets, and scanned documents without OCR templates. Agents inspect views and extract structured data via MCP.

Every file in a Fastio workspace retains per-file version history. If an agent or human colleague updates a document, earlier versions remain accessible, and the detailed activity log records the modification.

Monthly plans start with a 14-day free trial that requires a credit card. Plans are Starter at `$9.99/mo`, Business at `$49.99/mo`, and Enterprise at `$199.99/mo` on [Fastio pricing](/pricing/). Offloading large document corpora to an intelligent workspace keeps inference infrastructure lean, stable, and responsive.

## Diagnosing Out-of-Memory Errors and Tuning vLLM Serving Parameters

When tuning vLLM for production inference, engineers encounter two distinct categories of Out-of-Memory (OOM) failures: initialization crashes and runtime decode exhaustion. Applying the appropriate configuration flags stabilizes memory consumption without sacrificing generation quality.

### Initialization OOM: Resolving Cache Block Failures

If vLLM crashes during startup with a message stating there is no available memory for cache blocks:

```text
ValueError: No available memory for the cache blocks. Try increasing gpu_memory_utilization or decreasing max_model_len.
```

This error indicates that the combined memory required for model weights, CUDA runtime kernels, and the minimum required KV cache blocks for `--max-model-len` exceeds the configured GPU memory budget.

To fix initialization crashes:

* **Reduce `--max-model-len`:** Lower the context length from its theoretical default (such as 131,072) to an operational ceiling like 8,192 or 4,096.
* **Adjust `--gpu-memory-utilization`:** The default ratio is 0.90. On a dedicated GPU where no other processes run, raising this value to 0.94 or 0.95 allocates additional physical VRAM to the KV cache pool:

```bash
vllm serve meta-llama/Llama-3.1-8B-Instruct --max-model-len 8192 --gpu-memory-utilization 0.95
```

* **Shard Model Across GPUs:** For larger models such as 70B parameter architectures, use `--tensor-parallel-size` to distribute weights and KV heads across multiple devices:

```bash
vllm serve meta-llama/Llama-3.1-70B-Instruct --tensor-parallel-size 4 --max-model-len 8192
```

### Runtime OOM: Optimizing Decode Concurrency

Runtime CUDA OOM errors occur during generation when active sequences demand more memory blocks than the GPU can provide. While PagedAttention prevents memory fragmentation, bursts of concurrent requests with large `max_tokens` can exhaust physical capacity.

Key configuration levers to stabilize runtime generation:

#### Quantizing the KV Cache to FP8

By default, vLLM stores KV cache blocks in 16-bit precision matching the model weights. Enabling 8-bit floating point quantization via `--kv-cache-dtype fp8` cuts the memory footprint per token in half:

```bash
vllm serve meta-llama/Llama-3.1-8B-Instruct   --max-model-len 8192   --kv-cache-dtype fp8   --gpu-memory-utilization 0.92
```

For Llama 3.1 8B, FP8 quantization lowers the KV cache cost from 128 kilobytes per token to 64 kilobytes per token. An 8,192-token sequence consumes 512 megabytes instead of one gigabyte, doubling the concurrent generation capacity of the server.

#### Capping Concurrency with --max-num-seqs

The `--max-num-seqs` argument restricts the maximum number of concurrent sequences the scheduler packs into an iteration (defaulting to 256). For workloads with large `max_tokens` completions, lowering this value to 64 or 128 prevents the scheduler from over-admitting requests that compete for decode blocks:

```bash
vllm serve meta-llama/Llama-3.1-8B-Instruct   --max-model-len 8192   --max-num-seqs 64
```

#### Regulating Forward-Pass Batching with --max-num-batched-tokens

The `--max-num-batched-tokens` parameter dictates the maximum number of tokens processed in a single forward pass across both prefill and decode phases (defaulting to 2,048 or derived from context). When chunked prefill is active (`--enable-chunked-prefill`), long input prompts are broken into chunks matching this token budget, ensuring that ongoing decode iterations are not starved by heavy prompt arrivals.

### Production Configuration Matrix

The following matrix provides verified parameter profiles based on target workload characteristics:

| Workload Profile | Recommended `--max-model-len` | Typical Client `max_tokens` | `--kv-cache-dtype` | `--max-num-seqs` | Target Hardware |
|---|---|---|---|---|---|
| High-Concurrency Chat Gateway | 4096 tokens | 1024 tokens | fp8 | 128 | 1x NVIDIA RTX 4090 (24 GB) or L40S (48 GB) |
| Long-Form Agent Generation | 8192 tokens | 4096 tokens | fp8 | 32 | 1x NVIDIA A100 (80 GB) |
| Document QA with MCP Search | 8192 tokens | 2048 tokens | auto | 64 | 1x NVIDIA RTX 4090 (24 GB) or A10G (24 GB) |
| Multi-GPU Extended Synthesis | 16384 tokens | 4096 tokens | fp8 | 64 | 4x NVIDIA A100 (80 GB) or H100 (80 GB) |

### Monitoring and Alerting Checklist

Production deployments require monitoring key metrics exposed at vLLM's `/metrics` Prometheus endpoint:

1. **`vllm:gpu_cache_usage_factor`:** The fraction of GPU KV cache blocks currently in use. When this metric consistently exceeds 0.90, the server is nearing capacity. An alert should trigger at 0.95 to signal impending preemption.
2. **`vllm:num_requests_waiting`:** The number of requests queued in the scheduler waiting for free memory blocks. Spikes in this metric indicate that active requests with long `max_tokens` are monopolizing decode slots.
3. **`vllm:num_preemptions_total`:** A counter tracking how many running requests were paused or swapped. In a well-tuned system, this counter remains zero. If preemptions occur, reduce `--max-num-seqs` or lower client `max_tokens`.
4. **`vllm:cpu_cache_usage_factor`:** The fraction of CPU swap memory occupied by evicted blocks. Sustained usage indicates that the GPU KV cache is undersized for current request volume.

## Frequently asked questions

### What is max_tokens in vLLM?

In vLLM, max_tokens is a per-request generation parameter that defines the maximum number of output tokens the model can generate for a given prompt. It operates as a stopping criterion, halting generation when reached and returning a finish reason of length. Unlike the engine-level max-model-len parameter that sets total sequence capacity and dictates GPU memory allocation, max_tokens governs only the newly generated completion.

### How do you set max output tokens in vLLM?

You set max output tokens in vLLM either by configuring SamplingParams(max_tokens=N) when using the Python offline engine, or by passing max_tokens (or max_completion_tokens) in the JSON request body when calling the OpenAI-compatible /v1/chat/completions HTTP endpoint. Setting this parameter ensures that responses do not exceed the desired length or violate the server's context boundaries.

### Why does vLLM return truncated responses with default settings?

vLLM returns truncated responses when max_tokens is omitted because standard text completion sampling configurations default to generating a maximum of 16 tokens. This convention stops generation after 16 tokens with finish_reason length. To prevent premature cutoffs, client requests must explicitly specify max_tokens with a value appropriate for the task, such as 1024 or 2048.

### What happens when prompt tokens plus max_tokens exceeds max-model-len in vLLM?

When the sum of input prompt tokens and requested max_tokens exceeds the server's configured max-model-len, vLLM rejects the request during upfront validation with an HTTP 400 BadRequestError. The server returns an error message specifying the model's context ceiling and the requested token sum, preventing the engine from queueing a sequence that cannot fit in memory.

### What is the difference between max_tokens and max-model-len in vLLM?

The max-model-len parameter is a server startup argument that defines the total context window for prompt tokens and completion tokens combined, directly dictating how much GPU memory is reserved for the KV cache block pool. In contrast, max_tokens is a client request parameter that limits only the generated output tokens for a single API call.

### How does KV cache quantization affect max_tokens and concurrency in vLLM?

Enabling FP8 KV cache quantization with --kv-cache-dtype fp8 cuts the memory consumed per token in half, from 128 kilobytes to 64 kilobytes for models like Llama 3.1 8B. By reducing per-token memory usage, quantization allows the GPU to store twice as many generated tokens in its PagedAttention pool, doubling concurrent sequence capacity for long max_tokens generations.

### How does external MCP search help manage context limits in vLLM?

External MCP search decouples reference file storage from the model context window. Instead of loading entire document libraries into the prompt and inflating max-model-len, the corpus is stored in an intelligent workspace. An assistant queries the workspace via MCP to retrieve only concise, high-relevance passages of 500 to 1,500 tokens, leaving ample headroom for max_tokens generation without risking memory exhaustion.

## Sources

- [vLLM Documentation: Engine Arguments](https://docs.vllm.ai/en/stable/configuration/engine_args/): In vLLM, the max-model-len argument sets the total context length for input prompts and output generations, defaulting to the model configuration.
- [vLLM Documentation: Engine Arguments](https://docs.vllm.ai/en/stable/configuration/engine_args/): The max-model-len parameter accepts auto or -1 to select the largest sequence length that fits in available GPU memory.

## About Fast.io

Fast.io provides shared workspaces where people and AI agents work on the same files, with built-in semantic search and citation-backed chat over what they hold. Agents reach it through a remote MCP server, a REST API at https://api.fast.io/current/, and a command line client published on npm as @vividengine/fastio-cli. MCP setup is at https://mcp.fast.io/docs: Claude and most MCP clients connect to https://mcp.fast.io/mcp/tools, ChatGPT to https://mcp.fast.io/mcp/operations, and coding agents to https://mcp.fast.io/mcp/code.