# Llama.cpp Context Size: How to Set -c, Calculate KV Cache, and Optimize VRAM

In llama.cpp, context size is the maximum sequence length in tokens allocated for prompt and response processing, controlled at runtime via the -c or --ctx-size flag. Setting this parameter allocates the Key-Value (KV) cache in VRAM, which scales linearly with token capacity and can cause out-of-memory errors on consumer GPUs. Optimizing context memory through KV cache quantization, Flash Attention, and external retrieval workspaces ensures stable inference without crashing.

Source: https://fast.io/resources/llama-cpp-context-size/
Author: [Derek Labian](https://fast.io/authors/derek-labian/)
Last reviewed: 2026-10-04

## How Llama.cpp Allocates Context Window Memory and Defaults

Setting the context size in llama.cpp is not just a configuration parameter for how many tokens a local model can read; it directly dictates how much GPU memory the engine reserves upfront for its Key-Value cache before generating a single word. When developers attempt to feed entire codebases or lengthy technical documents into a local model without calculating KV cache growth, the process either crashes immediately with a CUDA out-of-memory exception or silently pushes execution into system RAM, slowing inference to a crawl.

In llama.cpp, context size is the maximum sequence length (in tokens) allocated for prompt and response processing, controlled at runtime via the -c or --ctx-size flag.

In the underlying C++ implementation, this configuration is mapped to the `n_ctx` parameter inside `llama_context_params`. When you launch a model, llama.cpp allocates a dedicated Key-Value (KV) cache tensor in memory to hold the intermediate attention keys and values for every position in the sequence. Because this allocation occurs during initialization, memory consumption jumps immediately upon model loading, even before you submit a prompt.

Understanding default behaviors requires distinguishing between legacy releases and modern builds of the software:

* **Legacy Default Context:** In older versions of llama.cpp, the default context window was hardcoded to `512` tokens, and later adjusted to `2048` tokens in subsequent iterations. Running models without an explicit flag frequently truncated inputs that exceeded these legacy limits.
* **Modern Default Behavior:** In current releases, the command-line flag `-c` defaults to `0`. When set to `0`, llama.cpp automatically parses the GGUF model file metadata and extracts the native context length specified by the model creator, such as `8,192` tokens for Llama 3 8B or `32,768` tokens for Mistral 7B.
* **Manual Overrides:** Specifying a positive integer (such as `-c 8192` or `-c 16384`) overrides the metadata default and forces llama.cpp to allocate a context buffer of precisely that sequence length.

A frequent source of configuration confusion is the distinction between `-c` (context size) and `-n` (prediction limit). The `-c` or `--ctx-size` argument sets the absolute sequence boundary for both prompt processing and text generation combined. In contrast, `-n` or `--n-predict` defines the maximum number of new tokens the model is permitted to generate in response to a prompt. By default, `-n` is set to `-1`, representing infinity, meaning generation continues until the model emits an end-of-sequence token or the cumulative tokens exhaust the allocated `-c` context window.

Execution memory also depends on batch sizing parameters. The logical batch size (`-b` or `--batch-size`, defaulting to `2048`) controls how many tokens can be evaluated in parallel during prompt ingestion. The physical micro-batch size (`-ub` or `--ubatch-size`, defaulting to `512`) dictates the maximum chunk size processed in a single compute kernel pass to stay within hardware compute buffers.

This architectural reality creates a strict boundary for local models. On hosted frontier platforms like Claude, file storage limits operate under different constraints. Anthropic documentation notes that Claude chats accept up to 20 files at up to 500MB each, while Claude Projects accept files up to 30MB each with unlimited file count, provided the total content must fit within Claude's context window. On local hardware, however, developers cannot rely on massive cloud memory pools. Allocating context sizes of `32,768` or `131,072` tokens on consumer GPUs demands careful memory budgeting to avoid sudden process crashes.

## How to Set Context Size with CLI Flags and Server Arguments

Configuring context size in llama.cpp can be handled through command-line flags, environment variables, or server startup arguments depending on whether you run interactive sessions or host an API service.

The primary executable for terminal chat and single-prompt evaluation is `llama-cli` (formerly known as `main` in older repository distributions). To load a model with an explicit context window of `8,192` tokens, pass the `-c` flag alongside your model path:

```bash
./llama-cli -m models/Meta-Llama-3-8B-Instruct-Q4_K_M.gguf -c 8192 -p "Summarize the architectural differences between GQA and MHA."
```

If you manage llama.cpp deployments inside containerized environments or automated scripts where editing command arguments is impractical, set the `LLAMA_ARG_CTX_SIZE` environment variable. The engine checks this variable during startup:

```bash
export LLAMA_ARG_CTX_SIZE=8192
./llama-cli -m models/Meta-Llama-3-8B-Instruct-Q4_K_M.gguf
```

When deploying `llama-server` to provide an OpenAI-compatible HTTP endpoint for local applications or coding agents, apply the same argument:

```bash
./llama-server -m models/Meta-Llama-3-8B-Instruct-Q4_K_M.gguf -c 16384 --port 8080 --host 0.0.0.0
```

### Inspecting GGUF Metadata for Native Context Limits

Every GGUF model file embeds architectural metadata written during weight conversion. Setting a context size higher than the model's native training ceiling without frequency adjustments leads to degraded outputs, repetitive hallucinations, or broken syntax.

You can inspect the native context ceiling using `llama-gguf-dump` or by reading the model header in Python:

```bash
./llama-gguf-dump models/Meta-Llama-3-8B-Instruct-Q4_K_M.gguf | grep "context_length"
```

Look for the key `llama.context_length` or `general.architecture`.context_length. For example, Llama 3 models declare a native context length of `8,192` tokens. Setting `-c 16384` without context extension parameters forces the positional embeddings beyond their trained range, causing severe perplexity spikes.

### Extending Context Capacity with RoPE Scaling

When you must process sequences longer than a model's native training limit, llama.cpp provides Rotary Position Embedding (RoPE) scaling. RoPE scaling mathematically compresses token positions into the frequency domain the model was trained on.

The engine supports linear scaling and Yet another RoPE extensioN (YaRN):

```bash
./llama-cli -m models/Meta-Llama-3-8B-Instruct-Q4_K_M.gguf -c 16384 --rope-scaling linear --rope-scale 2.0
./llama-cli -m models/Meta-Llama-3-8B-Instruct-Q4_K_M.gguf -c 16384 --rope-scaling yarn --rope-scale 2.0 --yarn-orig-ctx 8192
```

Linear scaling evenly divides positional frequencies by the scaling factor, allowing basic comprehension across longer sequences. YaRN selectively scales high and low frequency dimensions, preserving local attention precision while extending long-range document reasoning.

## How to Calculate KV Cache VRAM Consumption and Memory Scaling

The primary reason context expansion triggers out-of-memory errors is the memory footprint of the Key-Value cache. During text generation, self-attention requires comparing every new token against the key and value states of all prior tokens. To avoid recomputing these states on every single step, llama.cpp caches them in memory.

Understanding this consumption requires calculating the exact byte requirements for your model architecture. The mathematical formula for total KV cache memory in bytes is:

```text
KV_Bytes = 2 * n_layers * n_head_kv * d_head * n_ctx * bytes_per_element
```

Each variable represents a specific architectural parameter:

* **2:** Multiplier accounting for two distinct tensors: the Key cache and the Value cache.
* **`n_layers`:** Number of transformer layers in the model (for example, `32` layers in Llama 3 8B).
* **`n_head_kv`:** Number of Key-Value attention heads. In older architectures using Multi-Head Attention (MHA), this matched the query head count. Modern architectures use Grouped-Query Attention (GQA), where multiple query heads share a single KV head. Llama 3 8B features `32` query heads but only `8` KV heads, reducing cache memory by a factor of four.
* **`d_head`:** Head dimension, calculated as hidden size divided by query heads (typically `128` bytes per head).
* **`n_ctx`:** The total sequence length in tokens configured via `-c`.
* **`bytes_per_element`:** Data type precision. Standard 16-bit floating point (FP16 or BF16) requires `2` bytes per element. Quantized 8-bit cache (Q8_0) requires `1` byte per element, while 4-bit cache (Q4_0) requires approximately `0.5` bytes per element.

For Llama 3 8B operating at standard FP16 precision, the memory required per single token is:

```text
2 * 32 layers * 8 KV heads * 128 head_dim * 2 bytes = 131,072 bytes (128 KB per token)
```

At `2,048` tokens, the KV cache consumes approximately `256 MB`. At `8,192` tokens, it consumes approximately `1 GB`. If you expand the context window to `32,768` tokens, the cache demands `4 GB`. At `131,072` tokens, the cache alone consumes `16 GB` of VRAM.

The table below illustrates how the KV cache scales across context sizes for an 8B model (such as Llama 3 8B Q4_K_M, with static model weights of approximately `4.9 GB`) and the resulting hardware requirements:

| Context Window (`-c`) | KV Cache FP16 | KV Cache Q8_0 | KV Cache Q4_0 | Total VRAM (FP16 + Weights) | Hardware Viability |
| :--- | :--- | :--- | :--- | :--- | :--- |
| 2,048 tokens | 256 MB | 128 MB | 64 MB | ~5.4 GB | 8 GB GPU (RTX 3060, RTX 4060) |
| 4,096 tokens | 512 MB | 256 MB | 128 MB | ~5.7 GB | 8 GB GPU |
| 8,192 tokens | 1.0 GB | 512 MB | 256 MB | ~6.2 GB | 8 GB / 12 GB GPU |
| 16,384 tokens | 2.0 GB | 1.0 GB | 512 MB | ~7.3 GB | 12 GB GPU (RTX 3060 12GB, RTX 4070) |
| 32,768 tokens | 4.0 GB | 2.0 GB | 1.0 GB | ~9.4 GB | 12 GB / 16 GB GPU |
| 65,536 tokens | 8.0 GB | 4.0 GB | 2.0 GB | ~13.6 GB | 16 GB GPU (RTX 4080) |
| 131,072 tokens | 16.0 GB | 8.0 GB | 4.0 GB | ~21.9 GB | 24 GB GPU (RTX 3090, RTX 4090) |

Notice how memory requirements diverge as context expands. While doubling context length creates a linear increase in KV cache memory, self-attention computation during prompt evaluation scales quadratically with sequence length unless optimized kernels are applied.

## How to Optimize Context VRAM with Flash Attention and Cache Quantization

When running large contexts on local hardware, you can apply three optimization strategies to compress the KV cache, lower compute buffer spikes, and prevent out-of-memory crashes.

### 1. Enabling Flash Attention

During standard attention computation, the engine creates an intermediate attention matrix of size `n_ctx * n_ctx`. For an `8,192` token context, this matrix contains millions of floating point values; for a `32,768` token context, it expands into billions of elements. This intermediate matrix creates large temporary memory spikes during prompt ingestion.

Flash Attention resolves this problem by computing attention tiled in GPU on-chip SRAM without materializing the full matrix in main VRAM. In modern llama.cpp, Flash Attention is enabled using `-fa` or `--flash-attn`:

```bash
./llama-cli -m models/Meta-Llama-3-8B-Instruct-Q4_K_M.gguf -c 16384 -fa on
```

Flash Attention prevents out-of-memory spikes during prompt evaluation, reduces peak memory consumption, and speeds up token ingestion across long prompts.

### 2. Quantizing the Key-Value Cache

By default, llama.cpp allocates the KV cache in 16-bit floating point precision (`f16`). You can compress this cache by specifying quantized types for Keys (`-ctk`) and Values (`-ctv`).

The allowed quantization formats include `q8_0`, `q4_0`, `q4_1`, `q5_0`, and `q5_1`:

* **`q8_0` (8-bit quantization):** Halves KV cache memory footprint from `2` bytes to `1` byte per element. Output quality remains virtually indistinguishable from FP16, with minimal perplexity difference.
* **`q4_0` (4-bit quantization):** Reduces KV cache footprint to approximately `0.5` bytes per element, cutting memory consumption to one quarter of standard FP16. This allows a `32,768` token context window on an 8B model to occupy only `1.0 GB` of VRAM instead of `4.0 GB`.

To launch a model with an 8-bit quantized KV cache and Flash Attention:

```bash
./llama-cli -m models/Meta-Llama-3-8B-Instruct-Q4_K_M.gguf -c 32768 -fa on -ctk q8_0 -ctv q8_0
```

Combining Flash Attention with 8-bit or 4-bit KV quantization allows mid-range GPUs with `12 GB` or `16 GB` of VRAM to comfortably execute sequences of `32,768` or `65,536` tokens that would otherwise trigger immediate CUDA crashes.

### 3. Automatic Device Memory Fitting

If you are unsure how many model layers can fit into GPU VRAM alongside your desired context window, use the automatic fitting flag `-fit`:

```bash
./llama-cli -m models/Meta-Llama-3-8B-Instruct-Q4_K_M.gguf -c 16384 -fit on --fit-target 1024
```

The `--fit-target` parameter reserves a safety margin (in megabytes) on the device to absorb transient compute buffers. If the full model and context cannot fit inside physical VRAM, llama.cpp automatically adjusts the number of GPU-offloaded layers (`-ngl`), retaining the rest in system RAM.

## Why Decoupling Document Storage from Inference Context Prevents Memory Crashes

Expanding local context windows to `32,768` or `131,072` tokens introduces fundamental engineering tradeoffs that software optimizations cannot entirely erase.

First, processing large context payloads creates severe prompt evaluation latency. Ingesting a 30,000-token prompt on a consumer GPU takes significant time before the first response token is generated. Second, transformer models suffer from attention attenuation, commonly called the lost-in-the-middle phenomenon. When reference documents are dumped into a single prompt, models frequently overlook critical details buried in middle tokens while prioritizing instructions at the very beginning and end.

Third, local context is completely ephemeral. When a chat session concludes or the terminal closes, the loaded context vanishes. If multiple agents or human teammates need to consult the same reference documentation, each instance must independently re-read and process the entire corpus.

Production agentic workflows avoid this bottleneck by decoupling persistent storage from local inference context. Instead of stuffing multi-megabyte technical manuals, code repositories, or customer databases directly into `-c`, teams store reference documents in an external, persistent intelligent workspace.

In this architecture, document collections reside in [Fast.io Storage for Agents](/storage-for-agents/). Fast.io provides shared org-owned workspaces with per-file version history, granular permissions, and a detailed activity log. Workspaces can be populated through direct upload or synchronized from Google Drive, Dropbox, Box, or OneDrive, letting engineering teams aggregate project files into one centralized location.

When Intelligence Mode is enabled on a workspace, files are indexed automatically upon arrival for hybrid search, combining full-text lexical indexing with semantic vector retrieval.

Local AI agents, terminal tools, and IDE extensions connect to the workspace through the Model Context Protocol (MCP) using Streamable HTTP at `https://mcp.fast.io/mcp/tools`. To configure MCP in an agent environment, add the server definition to your configuration file:

```json
{
  "mcpServers": {
    "fastio-workspace": {
      "url": "https://mcp.fast.io/mcp/tools",
      "headers": {
        "Authorization": "Bearer YOUR_FASTIO_API_KEY"
      }
    }
  }
}
```

Instead of forcing llama.cpp to maintain a memory-heavy `65,536` token context window, the agent queries the Fast.io workspace using MCP search actions. The intelligent workspace performs hybrid retrieval across indexed documents and returns only the concise, highly relevant excerpts needed to solve the task.

The agent feeds that targeted 500-token snippet into llama.cpp running with a lean, fast `-c 4096` context window. This architecture delivers three operational advantages:

* **Eliminates VRAM Crashes:** The local GPU reserves only `512 MB` for KV cache rather than `8 GB`, leaving ample memory for running higher-parameter models.
* **Maximizes Generation Speed:** Without tens of thousands of prompt tokens to evaluate, time-to-first-token drops dramatically.
* **Centralizes Shared Team Context:** When documentation updates or new code commits arrive, the workspace indexes them once. Both local llama.cpp agents and cloud-hosted assistants query the same versioned source of truth.

Organizations get started with a 14-day free trial, which requires a credit card. Subscription plans scale across Starter (`$9.99/mo` with `3` seats and `250 GB` storage), Business (`$49.99/mo` with `10` seats and `5 TB` storage), and Enterprise (`$199.99/mo` with `30` seats and `25 TB` storage) on the [pricing page](/pricing/).

## Frequently asked questions

### How do I increase context size in llama.cpp?

You can increase context size in llama.cpp by passing the -c or --ctx-size flag followed by your desired token count, such as ./llama-cli -m model.gguf -c 8192. You can also define the LLAMA_ARG_CTX_SIZE environment variable in your shell or container configuration.

### What is the default context size for llama.cpp?

In modern versions of llama.cpp, the default context size is 0, which instructs the runtime to load the native training context length stored in the GGUF model metadata. In legacy releases, llama.cpp defaulted to 512 or 2048 tokens unless explicitly overridden.

### Why does increasing context size in llama.cpp cause out of memory errors?

Increasing context size causes out of memory errors because llama.cpp pre-allocates memory for the Key-Value (KV) cache upon initialization. The KV cache scales linearly with sequence length and model layer depth, rapidly exhausting available GPU VRAM.

### How much VRAM does the llama.cpp KV cache require?

The KV cache footprint depends on model layers, KV attention heads, head dimension, and precision. For an 8B model like Llama 3 with Grouped-Query Attention, 16-bit FP16 cache requires 128 KB per token, amounting to 1 GB at 8k context and 4 GB at 32k context.

### What is the difference between -c and -n in llama.cpp?

The -c parameter sets the total context window capacity including input prompt tokens and generated output tokens. The -n parameter controls the maximum number of new tokens the model generates in response, defaulting to -1 for unlimited generation until context exhaustion or EOS.

### How does KV cache quantization reduce VRAM consumption?

KV cache quantization compresses stored attention keys and values from 16-bit floats to 8-bit (q8_0) or 4-bit (q4_0) integers using the -ctk and -ctv flags. Setting -ctk q8_0 -ctv q8_0 cuts KV cache VRAM by half, while q4_0 cuts it to one quarter of standard FP16.

### When should you use an external retrieval workspace instead of expanding context?

You should use an external retrieval workspace when reference documents exceed 10,000 tokens, when multiple agents require access to the same document corpus, or when local GPU VRAM cannot support large context windows without sacrificing model size or generation speed.

## Sources

- [llama.cpp CLI Documentation](https://github.com/ggml-org/llama.cpp/blob/master/tools/cli/README.md): In llama.cpp, the -c or --ctx-size argument sets the size of the prompt context, defaulting to 0 where the context size is loaded from the model metadata.
- [Claude Help Center: Upload files to Claude](https://support.claude.com/en/articles/8241126-upload-files-to-claude): In Claude Projects, file uploads accept an unlimited number of files up to 30MB each, but the cumulative content must fit within the model context window.

## About Fast.io

Fast.io provides shared workspaces where people and AI agents work on the same files, with built-in semantic search and citation-backed chat over what they hold. Agents reach it through a remote MCP server, a REST API at https://api.fast.io/current/, and a command line client published on npm as @vividengine/fastio-cli. MCP setup is at https://mcp.fast.io/docs: Claude and most MCP clients connect to https://mcp.fast.io/mcp/tools, ChatGPT to https://mcp.fast.io/mcp/operations, and coding agents to https://mcp.fast.io/mcp/code.