Claude Tokenizer: How Token Counting Works for Claude Models

Anthropic's token counting documentation reports that the tokenizer introduced with Claude Opus 4.7 produces roughly 30% more tokens than earlier models for identical text. A system prompt consuming 10,000 tokens on Claude Sonnet 4 requires approximately 13,000 tokens on Claude Fable 5. Developers who skip re-measuring their prompts after a model migration will overshoot context windows and misestimate API costs.

The Claude tokenizer splits text into tokens, the smallest units the model reads and generates. Unlike words or characters, tokens follow statistical patterns learned from training data. Common English words like "the" or "hello" typically map to a single token, while uncommon words or technical terms get broken into several tokens. Punctuation, whitespace, and special characters each consume their own tokens too.

For English prose, Claude averages roughly 1.3 to 1.5 tokens per word on models using the newer tokenizer (Opus 4.7 and later), and about 1.0 to 1.3 tokens per word on older models. That translates to approximately 555,000 words per million tokens on newer models and 750,000 words per million tokens on older ones. These ratios shift for code, JSON, non-Latin scripts, and structured data, which tend to produce more tokens per character.

Tokens matter because every Claude API call is billed by token count, both for input (what you send) and output (what the model generates). Token counts also determine whether your prompt fits within a model's context window, how close you are to rate limits, and whether prompt caching will save you money. Whether you're building a chatbot, running Claude Cowork sessions, or processing documents through the Claude API, understanding how tokenization works is the first step toward controlling your costs.

Anthropic provides a dedicated API endpoint for pre-flight token measurement. Send a POST request to /v1/messages/count_tokens with the same payload structure you would use for the Messages API, and the response tells you exactly how many input tokens that request would consume.

Here is a Python example:

import anthropic

client = anthropic.Anthropic()

response = client.messages.count_tokens(
    model="claude-sonnet-4-6",
    system="You are a helpful assistant.",
    messages=[
        {"role": "user", "content": "Explain quantum computing in simple terms."}
    ],
)

print(f"Input tokens: {response.input_tokens}")

The equivalent curl command:

curl https://api.anthropic.com/v1/messages/count_tokens \
    --header "x-api-key: $ANTHROPIC_API_KEY" \
    --header "content-type: application/json" \
    --header "anthropic-version: 2023-06-01" \
    --data '{
      "model": "claude-sonnet-4-6",
      "system": "You are a helpful assistant.",
      "messages": [{
        "role": "user",
        "content": "Explain quantum computing."
      }]
    }'

The endpoint returns a simple JSON response: {"input_tokens": 18}.

This endpoint supports everything the Messages API accepts: system prompts, tool definitions, images (as base64), PDFs, and extended thinking blocks. For tool use, the count includes the internal system prompt that Claude adds to enable tool calling, which adds 290 to 804 tokens depending on the model and tool choice mode. For extended thinking, thinking blocks from previous assistant turns are ignored and do not count toward input tokens.

One important detail: the token count should be treated as an estimate. In some cases, the actual number of input tokens used when creating a message may differ by a small amount. Token counts may include tokens added automatically by Anthropic for internal optimizations, but you are not billed for system-added tokens.

The endpoint is free to use, with separate rate limits from the Messages API:

• Start tier: 2,000 requests per minute
• Build tier: 4,000 requests per minute
• Scale tier: 8,000 requests per minute

Since these limits are independent from your message creation limits, calling count_tokens before every API request will not eat into your production capacity. The model parameter determines which tokenizer is used for counting, so you can compare how many tokens a prompt would consume across models by counting the same request with different model values.

Anthropic introduced a new tokenizer with Claude Opus 4.7, and every model released since uses it. This creates a clear dividing line across the model lineup.

Models using the new tokenizer:

• Claude Fable 5
• Claude Mythos 5
• Claude Opus 4.8
• Claude Opus 4.7

Models using the older tokenizer:

• Claude Opus 4.6 and earlier
• Claude Sonnet 4.6 and earlier
• Claude Haiku 4.5 and earlier

The new tokenizer produces roughly 30% more tokens from the same text, with increases up to 35% depending on content type. Structured data like JSON and code tend to see larger increases than natural language prose.

Here is what that looks like in practice. A 100,000-word novel requires approximately 173,000 tokens on the new tokenizer versus 133,000 tokens on the older one. A 20,000-word research paper jumps from roughly 27,000 tokens to 35,000 tokens. These differences compound across multi-turn conversations where the entire message history is re-sent with each request.

For developers migrating from an older model to Fable 5 or Opus 4.8, three things need recalibration.

Token budgets

Existing limits and thresholds should be multiplied by approximately 1.3 to account for the increased token count. A context window check that truncates at 90% of the limit will trigger earlier than expected with the new tokenizer.

Cost estimates

The same prompt costs roughly 30% more in input tokens on models using the new tokenizer. Do not reuse token counts measured on older models to forecast Fable 5 costs. Run your representative prompts through count_tokens with model: "claude-fable-5" to get accurate numbers.

Cache warming

Cached token sequences from older models do not transfer to newer models because the token IDs differ between tokenizers. Plan for a cold-cache period when you switch models, and expect temporarily higher costs until your cache stabilizes.

To measure the exact difference for your workload, count the same request twice: once with your current model and once with the target model. Compare the input_tokens values to get your specific inflation ratio.

Give your AI agents a workspace with built-in intelligence

Fast.io provides shared workspaces where agents and humans collaborate on files. Intelligence Mode auto-indexes documents for semantic search and chat, so your agents can query workspace content without re-processing entire files each request. Starts with a 14-day free trial.

Start 14-Day Trial

Once you know a request's token count, calculating cost is straightforward. Multiply input tokens by the model's per-token input rate and output tokens by the output rate.

Current Claude API pricing per million tokens (MTok):

• Claude Fable 5: $10 input, $50 output
• Claude Mythos 5: $10 input, $50 output
• Claude Opus 4.8: $5 input, $25 output
• Claude Sonnet 4.6: $3 input, $15 output
• Claude Haiku 4.5: $1 input, $5 output

Suppose your application sends prompts that average 5,000 input tokens and receives 1,000 output tokens per response. On Claude Sonnet 4.6, the cost per request is:

(5,000 / 1,000,000 x $3) + (1,000 / 1,000,000 x $15) = $0.015 + $0.015 = $0.03 per request.

The same application on Claude Fable 5, accounting for the 30% tokenizer increase on input, would consume approximately 6,500 input tokens:

(6,500 / 1,000,000 x $10) + (1,000 / 1,000,000 x $50) = $0.065 + $0.05 = $0.115 per request.

That is nearly 4x the cost, combining the higher per-token price with the tokenizer inflation.

Two features can reduce these costs.

Prompt caching

Prompt caching stores repeated prompt prefixes so subsequent requests read from cache at 10% of the input price. After the initial cache write (which costs 1.25x the base input rate for a 5-minute cache), every cache hit saves 90% on those tokens. For applications with large, repeated system prompts or document contexts, caching often pays for itself within two or three requests.

Batch processing

The Batch API gives a 50% discount on both input and output tokens in exchange for asynchronous processing. Fable 5 batch pricing drops to $5 input and $25 output per MTok, matching standard Opus 4.8 pricing. For workloads that do not need real-time responses, like content generation pipelines, document analysis, or data extraction, batch processing is the simplest way to cut costs. These discounts stack: a batch request that hits prompt cache pays 50% of the already-discounted cache read rate.

Counting tokens is most valuable when you act on the numbers. Here are concrete techniques for trimming token usage without sacrificing output quality.

Measure before you iterate

Run your current prompt through count_tokens, then make a change and measure again. Small edits can have outsized effects. Removing a verbose system prompt preamble that seemed like 50 words might save 200+ tokens once tool definitions and formatting overhead are accounted for.

Trim tool definitions

Every tool you register adds tokens to every request, even if the model never calls it. The tool use system prompt alone adds 290 to 804 tokens depending on the model and tool choice mode. If your application defines ten tools but a particular conversation only needs two, consider dynamically filtering the tool list per request.

Use caching strategically

Place stable content (system prompts, reference documents, few-shot examples) at the beginning of your messages, where prompt caching can reach it. Variable content like the user's current question goes at the end. This maximizes cache hit rates and reduces the tokens you pay full price for.

Choose the right model for the task

Not every request needs Fable 5 or Opus 4.8. For classification, extraction, or routing tasks, Haiku 4.5 at $1/MTok input delivers fast results at a fraction of the cost. Use count_tokens to verify that your prompt fits within Haiku's context window before routing to it.

Pre-process documents before sending

For AI agent workflows that process files from shared workspaces, extracting relevant sections before sending them to the API reduces token counts dramatically. Instead of pushing a full 50-page document through the API (which might consume over 100,000 tokens), extract the specific pages or sections the agent needs. Platforms like Fast.io with built-in Intelligence Mode can pre-index and search documents semantically, letting agents query for specific information through the Fast.io MCP server rather than processing entire files through the Claude API.

Monitor token usage over time

Track your average tokens per request and cost per conversation. Spikes often indicate prompt drift, where instructions gradually expand as edge cases are patched. Periodic audits of your prompt templates keep token counts under control.