Ollama Max Tokens: Setting Generation Limits and Context Budgets

When an Ollama model cuts off generation in the middle of a sentence, developers almost always reach for num_ctx, expanding the context window to 32,000 or 64,000 tokens. That change balloons GPU memory allocation and slows inference, but it does nothing to prevent truncated outputs when the generation ceiling remains at its default. In Ollama, output length and input memory are governed by two entirely separate parameters: num_predict caps the generation budget, while num_ctx allocates the total memory window for input and output combined.

In Ollama, max tokens refers to the num_predict parameter, which governs the maximum number of tokens generated in a response (defaulting to 128 in some templates), distinct from the context window (num_ctx). Understanding how these two boundaries interact is essential for building stable local AI pipelines.

The Token Arithmetic of an Inference Turn

Every interaction with an Ollama model operates within a strict mathematical budget:

total_tokens = prompt_tokens + generated_tokens <= num_ctx

The prompt_tokens count includes your system prompt, conversation history, injected context, and formatting wrappers like ChatML tags. The generated_tokens count represents the tokens the model predicts during its response.

If you set num_predict to 2048, but your prompt consumes 3500 tokens inside a 4096-token num_ctx buffer, the model will not generate 2048 tokens. It will abruptly halt after 596 tokens because it hits the hard ceiling of num_ctx. Conversely, if you allocate 32,768 tokens to num_ctx but leave num_predict set to 128, the model will halt after 128 tokens even though tens of thousands of tokens remain free in the context window.

Key-Value Cache Allocation Versus Generation Stopping Rules

The fundamental engineering distinction between num_ctx and num_predict lies in how memory is allocated:

1. num_ctx is an upfront memory reservation. When Ollama loads a model or initializes a session, it allocates memory for the Key-Value (KV) cache based directly on num_ctx. The KV cache stores intermediate attention states across all transformer layers and attention heads so previous tokens do not need recalculation. Inflating num_ctx immediately consumes video memory (VRAM) before a single output token is generated. Details on memory scaling are covered in the Ollama context length documentation.
2. num_predict is a runtime stopping condition. Setting num_predict does not pre-allocate memory. It functions as an instruction counter for the autoregressive generation loop. Each forward pass generates one token; when the count reaches num_predict, Ollama breaks the generation loop and emits a completion flag. The parameter syntax is defined in the Ollama Modelfile reference.

Conflating these parameters leads developers to waste VRAM trying to fix generation truncation, or to starve model outputs while maintaining an unnecessarily massive context buffer.

• Salesforce Agentforce Limits: Tokens, Context Windows, and Grounding File CapsAgentforce limits establish operational boundaries for autonomous AI across Salesforce, including a 1.5 million hourly...
• Custom GPT Token Limit: Instruction Limits, Context Windows, and RetrievalThe Custom GPT token limit refers to the 8,000-character constraint on configuration instructions and the dynamic...
• vLLM max_tokens Guide: API Configuration, KV Cache Tuning, and Out-of-Memory FixesIn vLLM, max_tokens defines the client-requested upper limit on generated output tokens per completion request, bounded...
• HTTP 413 Request Entity Too Large: Causes, Server Limits & WorkaroundsAn HTTP 413 request entity too large error occurs when a client payload exceeds buffer limits configured on a web...
• ChatGPT Deep Research Limits: Plan Quotas, Reset Rules, and Workspace SolutionsChatGPT Deep Research limits restrict autonomous research tasks through plan-based quotas, rolling 30-day reset cycles,...
• Llama.cpp Context Size: How to Set -c, Calculate KV Cache, and Optimize VRAMIn llama.cpp, context size is the maximum sequence length in tokens allocated for prompt and response processing,...

More on this subject: AI Agents: General Guides (172 guides)

Controlling generation limits in Ollama can be handled at three levels: persistently in a custom Modelfile, dynamically per request via the HTTP API, or interactively during a terminal session.

Setting num_predict in a Modelfile

To create a model with a permanent generation limit, define PARAMETER num_predict in your Modelfile alongside your context window budget.

FROM llama3.2:latest
PARAMETER num_ctx 8192
PARAMETER num_predict 4096
PARAMETER temperature 0.7
PARAMETER repeat_penalty 1.1
SYSTEM You are an expert code generator that writes thorough, fully documented implementations.

Build and test your custom model using the Ollama CLI:

ollama create llama3.2-coder -f ./Modelfile
ollama run llama3.2-coder

Every request sent to llama3.2-coder will now inherit the 4096-token prediction limit unless explicitly overridden at runtime.

Overriding num_predict in API Payloads

When calling Ollama through its REST API, parameters must be placed inside the options object. Placing num_predict at the root of the JSON payload causes Ollama to silently ignore it and revert to the model default.

The following example demonstrates an explicit generation limit sent to the /api/generate endpoint:

curl http://localhost:11434/api/generate -d '{
  "model": "llama3.2",
  "prompt": "Write a complete Python implementation of a B-tree index.",
  "stream": false,
  "options": {
    "num_predict": 2048,
    "num_ctx": 8192,
    "temperature": 0.2
  }
}'

The same syntax applies to the chat completion endpoint (/api/chat):

curl http://localhost:11434/api/chat -d '{
  "model": "llama3.2",
  "messages": [
    {
      "role": "user",
      "content": "Draft an in-depth technical report on database sharding architectures."
    }
  ],
  "stream": false,
  "options": {
    "num_predict": 3000,
    "num_ctx": 8192
  }
}'

Interactive Adjustment in the Ollama CLI

If you are using the interactive terminal interface via ollama run, you can inspect and modify generation parameters on the fly without rebuilding the model:

>>> /set parameter num_predict 2048
Set parameter 'num_predict' to '2048'
>>> /show parameters
num_ctx                4096
num_predict            2048
temperature            0.8

Understanding Special Values: -1 and -2

Ollama supports two special negative integers for num_predict that alter the termination logic:

• num_predict = -1 (Infinite Generation): The engine imposes no artificial cap on output tokens. The model generates continuously until it produces an end-of-sequence token (such as <|eot_id|> or <|end_of_text|>), encounters a configured stop sequence, or fills the remaining capacity of num_ctx. While -1 is the default in Ollama's base Modelfile specification, running autonomous agents with -1 carries risk: if a model enters a repetitive loop, it will run until context exhaustion, consuming GPU compute.
• num_predict = -2 (Fill Context): This setting instructs the engine to generate until the available space in num_ctx is completely exhausted. If your prompt evaluates to 1200 tokens inside an 8192-token context window, setting num_predict = -2 directs the model to generate the remaining 6992 output tokens. Use this mode for exhaustive long-form document drafting where you want the maximum possible output your hardware allocation can accommodate.

When a response ends unexpectedly, you need to verify why generation halted before changing configuration flags. Ollama returns explicit metadata in its completion response that diagnoses the exact cause of termination.

Evaluating done_reason and Token Counters

In non-streaming API requests, or in the final JSON object of a streamed response, Ollama includes performance metrics and status flags:

{
  "model": "llama3.2",
  "created_at": "2026-09-30T10:15:30.123456Z",
  "response": "...truncated code output...",
  "done": true,
  "done_reason": "length",
  "total_duration": 4829104820,
  "load_duration": 1294820,
  "prompt_eval_count": 1840,
  "prompt_eval_duration": 349102000,
  "eval_count": 128,
  "eval_duration": 4478708000
}

Examine three specific fields in this payload:

1. done_reason:
   • "stop": The model finished cleanly by emitting its natural end-of-sequence token or hitting an explicit stop sequence.
   • "length": Generation halted because an execution limit was reached.
2. eval_count: The exact number of tokens generated in this response.
3. prompt_eval_count: The number of tokens ingested from your input prompt and system instructions.

If done_reason is "length" and eval_count matches your configured num_predict (or 128), your output hit the prediction ceiling. Increasing num_predict will immediately resolve the cutoff.

If done_reason is "length" and eval_count + prompt_eval_count equals num_ctx, generation stopped because the context window ran out of space. In this scenario, increasing num_predict will have no effect until you also increase num_ctx or reduce prompt size.

You can also calculate generation throughput by dividing eval_count by eval_duration converted to seconds:

tokens_per_second = eval_count / (eval_duration / 1,000,000,000)

In the example above, 128 / (4,478,708,000 / 1,000,000,000) = 28.58 tokens per second.

Inspecting Active Context and Offloading with ollama ps

To verify how much context Ollama allocated in active memory, run ollama ps while a model is loaded:

ollama ps

The terminal displays a live process table:

NAME             ID              SIZE      PROCESSOR    CONTEXT    UNTIL
llama3.2:latest  a849d4b6eb59    4.2 GB    GPU          8192       4 minutes from now

Pay careful attention to two columns:

• CONTEXT: Displays the active num_ctx allocated for the running instance. If this displays 4096 when you expected 16384, your custom context setting was not applied.
• PROCESSOR: Displays hardware allocation. If it reports pure GPU processing, the entire model and its KV cache reside in VRAM. If it reports mixed execution across GPU and CPU or CPU-only processing, memory pressure forced Ollama to offload layers to system RAM, causing generation speed to drop significantly.

Premature Termination via Stop Sequences

Another subtle cause of truncated responses is stop sequence interference. If a model definition or client wrapper injects an aggressive stop sequence, generation will abort with done_reason: "stop" even though the output appears unfinished.

Inspect the model template and parameters using:

ollama show --modelfile llama3.2

Check the PARAMETER stop lines. If a stop token like "User:" or "assistant" matches text the model generated in code samples or markdown explanations, Ollama terminates generation immediately.

Scaling context length is not a free operation. While increasing num_predict consumes minimal additional memory during execution, expanding num_ctx imposes heavy hardware demands.

Hardware Scaling and KV Cache Overhead

Transformer models maintain key and value vectors for every token in the active sequence across all attention layers. The memory required for this KV cache can be calculated using model architecture specifications:

KV_Cache_Memory = 2 * n_layers * n_heads * d_head * n_ctx * precision_bytes

Official Ollama guidance states that workloads requiring large context, including agents and coding tools, should configure at least 64,000 tokens. However, the memory footprint of the KV cache expands rapidly as sequence length grows:

When physical video memory is exhausted, Ollama offloads attention layers to system memory. Because system RAM bandwidth over PCIe is much slower than unified GPU memory, generation throughput collapses from dozens of tokens per second to single-digit speeds.

The Attention Dilution Problem

Expanding the context window introduces cognitive degradation alongside memory overhead. When you stuff complete documentation repositories, raw log files, and entire source trees into a 64,000-token prompt, the model must distribute attention weights across an enormous sequence.

Open-weight models often struggle to retrieve precise instructions buried in the middle of long contexts. Furthermore, processing an oversized prompt requires a substantial prefill phase: the GPU must calculate attention across tens of thousands of tokens before outputting its first predicted token. On local hardware, this prefill delay can pause execution for twenty to forty seconds.

The External Storage Pattern for Agentic Context

For autonomous coding agents, stuffing entire file collections into local context is inefficient. A more sustainable architecture keeps the local Ollama context window lean (for example, 4,096 to 8,192 tokens) and offloads document storage and indexing to an external intelligent workspace.

In this architecture:

1. Large files, codebases, API specifications, and reference manuals reside in a persistent Fast.io workspace rather than local GPU memory.
2. Workspace Intelligence automatically indexes the documents for hybrid search (combining full-text keyword search and semantic vector search) via Fast.io AI capabilities.
3. The agent connects to the remote Fast.io MCP server over Streamable HTTP at https://mcp.fast.io/mcp/code (signing in with OAuth in the browser for interactive clients, or sending an Authorization: Bearer <api key> header on the connection for headless code).
4. When a user asks a complex question, the agent executes an MCP search call against the workspace, retrieves only the three or four most relevant paragraphs with citations, and injects those precise excerpts into Ollama's prompt.

This pattern maintains a prompt token count near 1500 tokens. The local GPU runs at maximum throughput entirely inside VRAM, prefill latency remains sub-second, and the remaining context budget is preserved for num_predict generation. Developers managing large corpora can explore dedicated storage for agents and review Fast.io pricing for organization plans.

Stop stuffing oversized documents into local model context

Connect your local agents and Ollama instances to a Fast.io workspace. Index large file collections for semantic search over MCP, keeping your local context lean and inference fast. Monthly plans start with a 14-day free trial, requiring a credit card.

Start 14-Day Trial

When tuning Ollama token limits in production environments, several recurring edge cases emerge across different frameworks and API integrations.

1. Responses Truncate After Exactly 128 Tokens

This is the most common issue encountered by developers integrating Ollama with third-party orchestration libraries (such as LangChain, LiteLLM, or custom HTTP clients).

Root Cause: While the native Ollama engine defaults to -1 in bare Modelfiles, many external SDKs and wrapper libraries set an explicit client-side default of max_tokens = 128 or num_predict = 128. If you do not pass an explicit value in your client configuration, the library sends 128 in the API payload, overriding the server's configuration.

Resolution: Explicitly define num_predict (or max_tokens, depending on SDK nomenclature) in your client initialization code:

import json
import urllib.request

payload = {
    "model": "llama3.2",
    "prompt": "Provide a comprehensive architectural overview of microservices.",
    "stream": False,
    "options": {
        "num_predict": 2048,
        "num_ctx": 4096
    }
}

req = urllib.request.Request(
    "http://localhost:11434/api/generate",
    data=json.dumps(payload).encode("utf-8"),
    headers={"Content-Type": "application/json"}
)

with urllib.request.urlopen(req) as resp:
    data = json.loads(resp.read().decode("utf-8"))
    print(f"Generated tokens: {data.get('eval_count')}")
    print(f"Stop reason: {data.get('done_reason')}")

2. Output Repeats the Same Sentence Until Reaching Max Tokens

When a model enters a degenerate repetition loop, it continues emitting duplicate phrases until it hits num_predict.

Root Cause: Insufficient repetition penalization or deterministic sampling settings (temperature = 0).

Resolution: Tune repeat_penalty and repeat_last_n in your Modelfile or API payload:

• repeat_penalty: Set between 1.1 and 1.2 to discourage duplicate phrasing.
• repeat_last_n: Controls how many tokens back the model checks for repetition (default is 64; set to 128 or 256 for long-form outputs).

PARAMETER repeat_penalty 1.15
PARAMETER repeat_last_n 128

3. Server Crashes with CUDA Out of Memory on Prompt Submission

You submit a prompt, and the Ollama server process crashes immediately before generating any text.

Root Cause: The context window num_ctx is set beyond available GPU memory, and Ollama's memory allocator fails during the prompt prefill phase.

Resolution: Lower num_ctx in increments (such as moving from 32,768 to 16,384, or from 16,384 to 8,192). Check ollama ps to verify memory headroom. If your workflow requires large contexts on limited hardware, consider using a more aggressively quantized model (such as Q4_K_M or Q4_0) to free up VRAM for the KV cache.

4. Parameter Passed in API Is Silently Ignored

You send "num_predict": 4096 in your POST request, but the model still cuts off at the previous default.

Root Cause: Passing parameter flags as top-level JSON keys rather than inside the "options" object.

Resolution: Ensure that all generation parameters (num_predict, num_ctx, temperature, top_p, seed) reside strictly inside the options sub-dictionary. Ollama ignores unrecognized top-level keys without raising an HTTP 400 error.