Kafka Message Size Limits: Broker Caps, Configuration, and the Claim-Check Pattern

Apache Kafka restricts incoming payloads to a default message size limit of 1,048,576 bytes (1 MB) governed by the broker configuration parameter message.max.bytes, according to the official Apache Kafka documentation. When producers publish batches larger than this ceiling without coordinated configuration across the messaging pipeline, Kafka halts processing and raises a RecordTooLargeException.

The Kafka message size limit is the maximum payload size, governed by the broker configuration parameter message.max.bytes (defaulting to 1,048,576 bytes or 1 MB), that a producer can publish to a topic partition.

Understanding this constraint requires examining Kafka's core architecture. Kafka is a distributed, append-only commit log optimized for sequential disk I/O, network streaming, and operating system page cache memory efficiency. Small, uniform records allow Kafka brokers to maximize network packet utilization, minimize TCP fragmentation, and maintain predictable millisecond latency across millions of events per second.

Record Batches and Framing Overhead

In Apache Kafka, messages are rarely transmitted or written as isolated units. Starting with message format version 2 (introduced in Apache Kafka 0.11), individual records are grouped into a RecordBatch before transmission.

Each record batch carries structural framing metadata:

• Base Offset (8 bytes): Initial 64-bit integer offset of the first record in the batch.
• Batch Length (4 bytes): Total byte length of the record batch.
• Partition Leader Epoch (4 bytes): State tracking to prevent log divergence during failovers.
• Magic Byte (1 byte): Message format version identifier.
• CRC32C Checksum (4 bytes): Checksum covering the batch payload.
• Attributes (2 bytes): Bit flags indicating compression codec and transactional markers.
• Last Offset Delta (4 bytes): Delta between base offset and last record offset.
• Timestamps (16 bytes): 64-bit integers tracking event generation and ingestion times.
• Producer ID, Epoch, Sequence (14 bytes): Identifiers supporting idempotent delivery semantics.

Because this batch header accounts for 61 bytes of fixed framing metadata, plus record headers, the raw payload capacity of an individual message is slightly smaller than the full batch envelope. In newer distributions including Apache Kafka 3.x and 4.x, the broker configuration message.max.bytes defaults to 1,048,588 bytes. This specific figure accounts for exactly 1,048,576 bytes (1 MiB) of application data plus 12 bytes of internal record batch framing overhead.

Default Limits Across the Message Lifecycle

This default threshold is not merely a broker setting. Apache Kafka enforces a chain of coordinated defaults across producers, brokers, follower replicas, and consumers:

When any link in this chain is misconfigured, message delivery fails. If a producer attempts to transmit an oversized record while the broker enforces standard limits, the broker drops the request. If the broker accepts an oversized record batch because an administrator increased message.max.bytes, but replica.fetch.max.bytes remains at the default limit, follower replicas fail to replicate the partition, creating an immediate data availability crisis.

• Amazon SNS Message Size Limit: 256 KB Payload Caps, Protocols, and Large PayloadsThe Amazon SNS message size limit is a default quota of 256 KB (262,144 bytes) across message bodies and attributes....
• RabbitMQ Message Size Limit: Handling Large Payloads and Agent WorkflowsThe RabbitMQ message size limit defaults to 16 MiB in RabbitMQ version 4 and later, and 128 MiB in versions 3.8 through...
• DynamoDB Item Size Limit: The 400 KB Cap, ValidationException, and Storage PatternsThe DynamoDB item size limit is a strict maximum of 400 KB per item, calculated by summing the byte lengths of all...
• Can ChatGPT Edit Google Docs? Native Limits vs. Workspace SyncChatGPT cannot natively edit, overwrite, or modify live Google Docs through its built-in Google Drive connector. While...
• How to Map OneDrive as a Network Drive: WebDAV Setup, Business Limits, and WorkspacesMapping OneDrive as a network drive mounts cloud storage to a local drive letter using WebDAV URLs, bypassing local...
• How to Connect Dify AI Agents to Box Cloud StorageConnecting Dify to Box allows autonomous AI agents to query enterprise documents as an active knowledge base. While...

More on this subject: AI Agents: General Guides (172 guides)

Many tutorials suggest that handling large messages in Kafka is as simple as updating message.max.bytes=10485760 in server.properties and restarting the cluster. In production environments, modifying a single configuration in isolation creates immediate operational failures. Raising message size limits requires synchronized coordination across broker settings, topic overrides, producer buffers, and consumer fetch parameters.

The Four Interdependent Configuration Parameters

To permit larger message batches safely, system administrators must align four distinct parameters across the pipeline:

1. Broker Batch Ceiling (message.max.bytes): The broker-side ceiling defining the largest record batch allowed by Kafka after compression. If an incoming produce request exceeds this value, the broker rejects the batch with error code 10 (MESSAGE_TOO_LARGE).
2. Producer Request Ceiling (max.request.size): The client-side ceiling defining the maximum size of a request in bytes. This acts as a client-side gatekeeper, preventing the producer from spending network I/O on requests that the broker will refuse.
3. Broker Replication Fetch Size (replica.fetch.max.bytes): The maximum bytes follower replicas fetch from partition leaders during replication cycles. This value must equal or exceed message.max.bytes. If follower replicas cannot fetch large batches, the partition enters an Under-Replicated Partition (URP) state.
4. Consumer Fetch Quotas (max.partition.fetch.bytes and fetch.max.bytes): The consumer-side memory limits per partition and per fetch request. While modern Kafka consumers can fetch a batch that exceeds max.partition.fetch.bytes if it is the first batch in the partition, keeping this setting aligned prevents buffer truncation and fetch delays.

Setting Topic-Level Overrides with kafka-configs.sh

Rather than exposing every topic on a shared cluster to large message allocations, apply overrides selectively to specific topics that require expanded capacity. Modifying broker-wide message.max.bytes in server.properties requires a rolling restart of every broker in the cluster. Topic-level overrides take effect dynamically without service interruption.

Use kafka-configs.sh to apply a 10 MB limit (10,485,760 bytes) to a topic named document-events:

kafka-configs.sh --bootstrap-server localhost:9092 \
  --alter --entity-type topics --entity-name document-events \
  --add-config max.message.bytes=10485760

Verify that the dynamic topic configuration was applied:

kafka-configs.sh --bootstrap-server localhost:9092 \
  --describe --entity-type topics --entity-name document-events

When setting max.message.bytes on a topic, update replica.fetch.max.bytes on brokers if the topic limit exceeds the cluster-wide replication setting. Follower brokers must be capable of replicating whatever batch size the topic accepts.

Producer and Consumer Client Configurations

Once broker and topic parameters are aligned, update producing and consuming client applications.

In Java producer applications, configure max.request.size, buffer.memory, and batch.size:

bootstrap.servers=localhost:9092
key.serializer=org.apache.kafka.common.serialization.StringSerializer
value.serializer=org.apache.kafka.common.serialization.ByteArraySerializer
max.request.size=10485760
buffer.memory=67108864
batch.size=131072

In consumer applications, configure fetch.max.bytes and max.partition.fetch.bytes:

bootstrap.servers=localhost:9092
group.id=document-processor-group
key.deserializer=org.apache.kafka.common.serialization.StringDeserializer
value.deserializer=org.apache.kafka.common.serialization.ByteArrayDeserializer
max.partition.fetch.bytes=10485760
fetch.max.bytes=52428800

Enabling Wire-Level Payload Compression

Before raising numeric byte limits, enable wire-level compression on producers.

Apache Kafka applies message.max.bytes and max.message.bytes checks after compression is applied. If a producer generates a verbose JSON document, enabling LZ4 or Zstandard (zstd) compression substantially reduces serialized wire size. The compressed record batch easily fits beneath the default broker ceiling without requiring broker reboots or configuration changes.

Configure compression in producer properties:

compression.type=zstd

Supported codecs include lz4, zstd, snappy, and gzip. For high-throughput stream pipelines, lz4 delivers the lowest CPU overhead, while zstd provides superior compression ratios for structured text, JSON, and XML payloads.

While adjusting configuration keys allows Kafka to transmit oversized records, treating an event log as a bulk file storage system introduces severe infrastructure degradation across four distinct failure modes.

1. JVM Garbage Collection Pauses and Humongous Allocations

Kafka brokers execute inside the Java Virtual Machine (JVM). Most production Kafka deployments rely on the Garbage-First (G1) collector, which partitions heap memory into uniform regions.

When an allocation exceeds half of the active G1 region size, G1 classifies it as a humongous object, allocating it directly into contiguous regions within the old generation.

When producers publish multi-megabyte messages:

• The broker must allocate multiple contiguous memory regions to store incoming byte buffers.
• If contiguous free regions are unavailable due to heap fragmentation, G1 initiates concurrent marking or falls back to a Full GC pause.
• Stop-the-world GC pauses freeze broker threads for several seconds.
• If a pause exceeds cluster heartbeat timeouts, the cluster metadata controller marks the broker dead.
• The cluster initiates partition leader re-elections across hundreds of partitions simultaneously, causing connection churn and latency spikes.

2. Operating System Page Cache Thrashing

Kafka achieves high throughput by avoiding application-level memory caching. Instead of holding cached messages in JVM heap, Kafka relies on the Linux kernel page cache combined with the sendfile system call for zero-copy network transmission.

When a producer writes a message, data lands in page cache RAM before flushing to physical disk. When consumers read recent records, the Linux kernel serves data directly from page cache RAM, bypassing disk reads entirely.

Large messages destroy page cache efficiency:

• An oversized message consumes the same page cache space as thousands of standard one-kilobyte messages.
• High-volume ingestion of large payloads evicts hot index files and active metadata from RAM.
• Consumers reading other topics on the same broker suffer cache misses, forcing physical disk head movements and random disk reads.
• Disk I/O utilization saturates, degrading throughput for all applications sharing the cluster.

3. Consumer Lag and Rebalance Cascades

Kafka consumers read messages in batches. When a consumer receives an oversized payload, deserializing and processing the record consumes substantially more time and CPU cycles than standard events.

The Kafka consumer coordinator monitors consumer health through max.poll.interval.ms (defaulting to 300,000 milliseconds, or 5 minutes). If a consumer spends more time processing a massive payload than max.poll.interval.ms allows before calling poll() again, the coordinator concludes that the consumer thread has hung or crashed.

The coordinator evicts the consumer from the group and triggers a rebalance:

[ Consumer 1 ]  ──► (Processing massive payload takes 6 minutes) ──► Fails poll deadline
       │
[ Broker Coordinator ]  ──► Marks Consumer 1 dead ──► Triggers Group Rebalance
       │
[ Consumer 2 ]  ──► Assigned Partition ──► Fetches SAME payload ──► Hangs ──► Loop Repeats

This cycle creates a poison pill rebalance storm. Every consumer that receives the large message times out, gets evicted, and passes the problematic record to the next worker, halting partition processing indefinitely.

4. Follower Replication Saturation and Under-Replicated Partitions

Kafka partition durability depends on follower brokers continuously fetching log segments from the partition leader governed by replica.fetch.max.bytes.

When partition leaders ingest massive message batches, follower replicas must fetch and write these payloads across the inter-broker network. Replication traffic saturates network interfaces, delaying acknowledgement of smaller messages on other partitions. If replica.fetch.max.bytes is smaller than the leader's batch size, replication halts completely, creating Under-Replicated Partitions and risking data loss if the leader broker fails.

Store Agent Payloads in Fast.io Workspaces

Connect event-driven AI agents to persistent cloud workspaces instead of passing heavy files across message brokers. Query indexed documents via remote MCP. Monthly plans start with a 14-day free trial.

Start 14-Day Trial

The most common symptom of misaligned message size thresholds is org.apache.kafka.common.errors.RecordTooLargeException. Resolving this error requires identifying whether the rejection occurred client-side inside the producer application or server-side on the broker.

Distinguishing Client-Side from Server-Side Rejection

Although the exception class name is identical, the error originates in two distinct layers of the Kafka architecture:

Scenario A: Client-Side Producer Rejection

When a record's serialized byte array exceeds the producer's configured max.request.size, the Kafka producer library rejects the record locally. The network client never transmits data over the TCP socket. The client-side stack trace shows immediate failure during serialization:

org.apache.kafka.common.errors.RecordTooLargeException: The message is 15482910 bytes when serialized which is larger than the maximum request size you have configured with the max.request.size configuration.
	at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:932)
	at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:862)

In this scenario, changing broker settings has zero effect because the broker never received the request. The solution requires either increasing the producer's max.request.size or reducing payload dimensions.

Scenario B: Server-Side Broker Rejection

When the producer's max.request.size is large enough to allow transmission, but the broker detects that the batch exceeds message.max.bytes (or topic-level max.message.bytes), the broker rejects the request. The broker returns protocol error code 10 (MESSAGE_TOO_LARGE). The producer client parses this network response and converts it into a RecordTooLargeException:

java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.RecordTooLargeException: The server has a configurable maximum message size which was exceeded.
	at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.valueOrError(FutureRecordMetadata.java:115)
	at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:64)

In this scenario, the producer attempted transmission, but the topic or broker rejected the batch.

Diagnostic Verification Checklist for Kafka Teams

When investigating message size exceptions in production, execute this diagnostic workflow:

Resilient Producer Error Handling Pattern

Production applications should intercept oversized payloads defensively rather than allowing unhandled exceptions to crash ingestion services:

import org.apache.kafka.clients.producer.*;
import org.apache.kafka.common.errors.RecordTooLargeException;
import java.util.Properties;

public class ResilientKafkaPublisher {
    private final KafkaProducer<String, byte[]> producer;
    private final int maxRequestSize;
//
    public ResilientKafkaPublisher(Properties props, int maxRequestSize) {
        this.maxRequestSize = maxRequestSize;
        props.put(ProducerConfig.MAX_REQUEST_SIZE_CONFIG, maxRequestSize);
        this.producer = new KafkaProducer<>(props);
    }
//
    public void publishRecord(String topic, String key, byte[] payload) {
        if (payload.length > maxRequestSize) {
            offloadToClaimCheckStorage(topic, key, payload);
            return;
        }
        ProducerRecord<String, byte[]> record = new ProducerRecord<>(topic, key, payload);
        producer.send(record, (metadata, exception) -> {
            if (exception instanceof RecordTooLargeException) {
                offloadToClaimCheckStorage(topic, key, payload);
            }
        });
    }
//
    private void offloadToClaimCheckStorage(String topic, String key, byte[] payload) {
        // Upload payload to external storage and publish metadata reference to Kafka
    }
}

Rather than forcing large files through an event commit log and risking cluster instability, software architects use the Claim-Check pattern. Documented in standard Enterprise Integration Patterns, the claim-check pattern decouples payload data persistence from event notification streaming.

The concept mirrors checking luggage at an airport: passengers hand bulky bags to handlers, board with a compact claim ticket, and retrieve luggage at their destination.

Architectural Workflow

In an event-driven messaging architecture, the claim-check pattern operates across five distinct phases:

[ Producer Client ]
        │
        ├── 1. Upload Heavy Payload to Storage ──► [ External Persistent Storage ]
        │                                                  │
        │   ◄── 2. Return Reference (URI, Hash, Metadata) ─┘
        │
        └── 3. Publish Compact Event ────────────► [ Kafka Topic Partition ]
                                                           │
[ Consumer Client ]                                        │
        │                                                  │
        ├── 4. Read Compact Event ◄────────────────────────┘
        │
        └── 5. Retrieve Heavy Payload On Demand ─► [ External Persistent Storage ]

1. Payload Ingestion: The producer generates a large data asset, such as a PDF document, video file, multi-agent conversation history, or analytical model weights.
2. External Storage Offloading: Instead of serializing raw bytes into a Kafka record, the producer uploads the asset directly to an external persistent storage layer using standard REST or chunked upload APIs.
3. Claim-Check Generation: The storage layer returns a unique resource pointer containing the file URI, storage location, file size, cryptographic checksum, and extraction metadata.
4. Lightweight Kafka Event Publication: The producer constructs a compact JSON or Avro event payload containing the claim-check metadata. The producer writes this compact event to the Kafka topic partition.
5. Consumer Retrieval On Demand: The consumer reads the event from Kafka instantly. Because the message is tiny, consumption introduces zero consumer lag. The consumer inspects the metadata and downloads the full file from persistent storage only when ready for heavy processing.

Claim-Check Schema Specification

A structured claim-check message separates routing headers from data payload pointers:

{
  "eventId": "evt-749102-a89c",
  "eventType": "document.transcription.completed",
  "timestamp": "2026-10-08T05:30:00Z",
  "topic": "document-events",
  "claimCheck": {
    "storageProvider": "cloud-workspace",
    "workspaceId": "ws-engineering-artifacts",
    "fileId": "quarterly-financial-analysis.pdf",
    "fileSizeBytes": 45192800,
    "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "mimeType": "application/pdf"
  },
  "summary": {
    "pageCount": 142,
    "department": "Finance",
    "classification": "Confidential"
  }
}

Downstream applications that only require routing signals, status tracking, or metrics can process the event using the summary attributes without downloading the full file from storage.

Architectural Tradeoffs: Direct Kafka vs. Chunking vs. Claim-Check

Engineering teams evaluate three primary patterns when handling message payloads:

Client-side chunking requires slicing a file into multiple sequential Kafka records tagged with sequence numbers and total counts. Consumers must buffer chunks in memory until the entire file arrives. If a single chunk is lost, delayed, or delivered out of order due to partition rebalancing, reassembly fails, leaking consumer heap memory. For enterprise stability, the claim-check pattern is the industry standard.

When autonomous AI agents coordinate across event-driven systems, passing large files directly through messaging channels creates severe architectural friction. AI assistants face context window limitations when processing enterprise corpora: while Claude Projects accepts individual files up to the documented project cap without a fixed file-count limit, the practical ceiling on a project is Claude's context window. Saturating model context windows or streaming megabyte-scale files across Kafka topics stalls agent swarms.

Teams frequently turn to commodity object stores such as Amazon S3, Google Cloud Storage, or Azure Blob Storage to implement the claim-check pattern. While these services store raw bytes, they lack native document intelligence, semantic search, and collaborative access layers needed by multi-agent workflows.

Intelligent Cloud Workspaces for Multi-Agent Pipelines

Fast.io provides persistent, organization-owned cloud workspaces built for human and AI agent collaboration:

• Chunked Workspace Uploads: Agents upload large assets of any size using chunked uploads directly to workspace storage, completely bypassing Kafka message limits. Maximum upload sizes scale across plan tiers.
• Automated Indexing with Intelligence Mode: When files land in a workspace, Intelligence Mode indexes documents for hybrid search (combining full-text search, semantic vector retrieval, and search-by-metadata-value). AI agents query relevant document excerpts on demand, receiving citation-backed answers that consume minimal context tokens.
• Remote Model Context Protocol (MCP) Server: Agents connect to Fast.io workspaces through a consolidated MCP toolset over Streamable HTTP. Connect Claude apps and general clients at https://mcp.fast.io/mcp/tools, ChatGPT and Codex through the Fastio plugin (or custom servers at https://mcp.fast.io/mcp/operations), and coding agents at https://mcp.fast.io/mcp/code. Through MCP actions, agents inspect directories, search files, and extract passages across persistent workspaces. For complete developer setup guides, see Storage for AI Agents.
• Structured Document Extraction with Metadata Views: Rather than writing custom parsing scripts, teams configure Metadata Views to convert unstructured files into typed, queryable spreadsheet schemas that agents inspect via MCP.
• Concurrency Control with Advisory File Locks: In multi-agent swarms, multiple workers coordinate over shared assets. Fast.io provides advisory per-file leases acquired, heartbeated, and released through MCP storage actions (lock-acquire and lock-release on storage_manage, lock-status on storage), avoiding conflicts while per-file version history preserves every iteration.
• Real-Time Workspace Activity Monitoring: To keep agents synchronized without pushing heavy file contents across messaging brokers, Fast.io provides a workspace activity feed accessible via WebSocket or long-poll subscription (/current/activity/poll/{entity_id}). Agents subscribe to activity events to learn when files are added or modified, retrieving document content on demand through MCP.
• Collaborative Notes and Branded Portals: Teams co-edit documentation in real time using Collaborative Notes, while finished deliverables are shared via branded Send, Receive, or Exchange portals that support password protection, download restrictions, and access expiration.
• Multi-Cloud Sync and Import: Workspaces connect external repositories. Cloud Sync maintains scheduled or on-demand one-way or two-way synchronization with Google Drive, Dropbox, Box, and OneDrive (including SharePoint libraries via the OneDrive connector).

Fast.io operates on a paid subscription model. Monthly plans start with a 14-day free trial, which requires a credit card. Paid subscriptions include Starter, Business, and Enterprise plans, providing team seats, dedicated storage, and credits for workspace intelligence:

By coupling lightweight Kafka event signaling with persistent, indexed cloud workspaces, developers build agent workflows that remain responsive, avoid broker memory exhaustion, and scale across massive document corpora.