DynamoDB Item Size Limit: The 400 KB Cap, ValidationException, and Storage Patterns

Amazon DynamoDB enforces a strict maximum item size limit of 400 KB (409,600 bytes) across every table, partition key, and sort key configuration. This ceiling is an unchangeable architectural constraint defined within AWS Service Quotas. Unlike provisioned read and write capacity units or account table counts, the 400 KB quota cannot be adjusted by submitting an AWS Support request. Every item written to a DynamoDB table through PutItem, UpdateItem, or BatchWriteItem must remain within this boundary.

How DynamoDB Calculates Item Size in Bytes

The 400 KB threshold does not evaluate solely the raw values stored in an item. DynamoDB calculates total item size by summing the binary UTF-8 lengths of all attribute names and attribute values, plus a fixed per-item overhead.

Total Item Size = 100 Bytes Overhead + Sum(Length(Attribute Name)) + Sum(Length(Attribute Value))

The calculation rules across DynamoDB data types operate as follows:

• Strings: Evaluated using UTF-8 binary encoding. The byte consumption equals the UTF-8 byte length of the attribute name plus the UTF-8 byte length of the string value.
• Numbers: Stored in variable-length format with a precision ceiling of 38 significant digits. Leading and trailing zeroes are stripped. The size equals the attribute name length in bytes, plus approximately 1 byte for every two significant digits, plus 1 additional byte for sign and exponent.
• Binary: Transmitted over the wire in base64 format, but DynamoDB calculates size using raw unencoded bytes. The size equals the attribute name length in bytes plus the raw binary byte length.
• Booleans and Nulls: The size equals the attribute name length in bytes plus 1 byte for the type marker.
• Lists and Maps: Document types incur 1 byte of structural overhead for the container, 1 byte of overhead per nested element, plus the recursive size of all contained attribute names and values.
• Sets (String, Number, Binary): Evaluated as the attribute name length in bytes, plus 1 byte of overhead per element, plus the byte length of each element in the set.

Because attribute names count toward the 400 KB cap, verbose JSON keys directly consume available storage. An item containing fifty attributes whose keys each span thirty characters consumes 1,500 bytes on attribute names alone before storing any business values.

Capacity Unit Consumption on Large Items

Writing items near the 400 KB limit carries steep throughput and billing consequences. DynamoDB meters throughput in discrete capacity chunks:

• Write Capacity Units (WCU): One WCU covers one standard write per second for items within 1 KB of size. Items larger than 1 KB are rounded up to the next 1 KB increment.
• Read Capacity Units (RCU): One RCU covers one strongly consistent read per second (or two eventually consistent reads) for items within 4 KB of size. Items larger than 4 KB are rounded up to the next 4 KB increment.

The following table illustrates the throughput consumption across representative payload sizes:

Writing a single 400 KB item exhausts 400 WCUs in standard write mode and 800 WCUs in transactional mode. In high-concurrency environments, writes of this magnitude quickly deplete partition throughput limits (capped at 1,000 WCU and 3,000 RCU per physical partition), triggering provisioned throughput exceeded errors.

• Amazon SNS Message Size Limit: 256 KB Payload Caps, Protocols, and Large PayloadsThe Amazon SNS message size limit is a default quota of 256 KB (262,144 bytes) across message bodies and attributes....
• OneDrive Agent: Connecting AI Agents to OneDrive File StorageAn OneDrive agent connects autonomous AI models to cloud file storage, allowing systems like Claude and Cursor to...
• Kafka Message Size Limits: Broker Caps, Configuration, and the Claim-Check PatternApache Kafka enforces a default message size limit of 1,048,576 bytes (1 MB) across broker, producer, and consumer...
• Custom GPT Token Limit: Instruction Limits, Context Windows, and RetrievalThe Custom GPT token limit refers to the 8,000-character constraint on configuration instructions and the dynamic...
• How to Connect Dify AI Agents to Box Cloud StorageConnecting Dify to Box allows autonomous AI agents to query enterprise documents as an active knowledge base. While...
• How to Mount OneDrive with Rclone: VFS Cache, Throttling, and Agent StorageMounting Microsoft OneDrive through Rclone exposes cloud files as a local filesystem via FUSE, but standard mount...

More on this subject: AI Agents: General Guides (172 guides)

When an application issues a write request that pushes an item past 400 KB, DynamoDB rejects the operation synchronously. The database engine does not truncate data, does not spill excess bytes to disk, and does not accept the write with a warning.

The ValidationException Error Response

The DynamoDB service returns an HTTP 400 Bad Request error with a ValidationException code. The error payload indicates size limit violation:

{
  "__type": "com.amazon.coral.validate#ValidationException",
  "message": "Item size has exceeded the maximum allowed size of 400KB"
}

In the AWS SDK for Python (boto3), this failure surfaces as a botocore.exceptions.ClientError:

import boto3
from botocore.exceptions import ClientError

dynamodb = boto3.resource("dynamodb", region_name="us-east-1")
table = dynamodb.Table("user-session-state")

oversized_item = {
    "session_id": "sess_89412",
    "large_payload": "x" * (401 * 1024)
}

try:
    table.put_item(Item=oversized_item)
except ClientError as error:
    if error.response["Error"]["Code"] == "ValidationException":
        print("Write rejected: Item exceeded the 400 KB cap.")
    else:
        raise

The write operation is atomic: no attributes are saved, and the prior version of the item remains unmodified.

Secondary Indexes and Collection Limits

The 400 KB limit directly influences index structures:

• Local Secondary Indexes (LSIs): For tables configured with one or more LSIs, the 400 KB item limit applies to the sum of base table attributes and all projected index attributes. In addition, DynamoDB enforces a strict size ceiling on each item collection (all items sharing the same partition key). Writing repeated large items under a single partition key with LSIs will rapidly exhaust the local secondary index partition collection size ceiling.
• Global Secondary Indexes (GSIs): Because base table write requests exceeding 400 KB fail immediately, oversized mutations never enter the asynchronous GSI replication stream. However, if a valid 350 KB item projects attributes into three separate GSIs, each GSI write consumes corresponding write capacity units on its own storage partition.

Transactional Failure Modes

When using TransactWriteItems, DynamoDB groups multiple write actions into a single coordinated transaction subject to an aggregate transaction payload ceiling. If any single item in that transaction exceeds the 400 KB item cap, DynamoDB cancels the entire transaction.

The API returns a TransactionCanceledException containing a CancellationReasons array. The specific entry for the failing item shows code ValidationError, and none of the other writes land in their target tables.

Monitoring Item Sizes with ReturnConsumedCapacity

To catch items nearing the 400 KB threshold before they cause production outages, pass ReturnConsumedCapacity='TOTAL' on write requests:

response = table.put_item(
    Item=item_payload,
    ReturnConsumedCapacity="TOTAL"
)

consumed_wcu = response["ConsumedCapacity"]["CapacityUnits"]
if consumed_wcu > 300:
    logger.warning(
        "Item approaching DynamoDB size limit: %d WCU consumed",
        consumed_wcu
    )

Because 1 WCU equals 1 KB of written data, a consumed capacity value of 300 indicates the item has reached 300 KB, providing an operational signal to initiate data offloading.

Engineers encountering the 400 KB cap often attempt client-side workarounds to keep data inside DynamoDB. While these techniques solve immediate constraints for small text records, they introduce distinct structural limitations.

Strategy 1: In-Memory Attribute Compression

One common approach compresses large string attributes using GZIP, Zstandard, or Deflate, storing the compressed output in a Binary (B) attribute:

import gzip
import json

def prepare_compressed_attribute(data_dict: dict) -> bytes:
    raw_json = json.dumps(data_dict).encode("utf-8")
    return gzip.compress(raw_json, compresslevel=6)

compressed_bytes = prepare_compressed_attribute(rich_event_payload)
table.put_item(
    Item={
        "entity_id": "order_48291",
        "payload_data": compressed_bytes
    }
)

The Tradeoffs:

• Loss of Query and Filter Capabilities: DynamoDB cannot inspect, parse, or filter inside compressed binary blobs. You cannot use FilterExpression or KeyConditionExpression against any fields within the compressed payload.
• Client CPU Overhead: Every producer must compress before writing, and every consumer must decompress before reading, increasing request latency and CPU utilization across serverless functions.
• Incompressible Formats: Pre-compressed data, including images (JPEG, PNG), documents (PDF, DOCX), and modern binary serialization formats, experiences minimal reduction under GZIP. If the compressed binary still exceeds 400 KB, the write still fails.

Strategy 2: Vertical Partitioning and Chunking

Another pattern splits large records across multiple distinct items sharing a partition key and indexed by sequential sort keys:

PK: USER#10482    SK: METADATA        (User profile, status, timestamps)
PK: USER#10482    SK: CHUNK#0001      (Bytes 0 to 350,000)
PK: USER#10482    SK: CHUNK#0002      (Bytes 350,001 to 700,000)
PK: USER#10482    SK: CHUNK#0003      (Bytes 700,001 to 1,050,000)

The Tradeoffs:

• Reassembly Complexity: Application code must manage splitting byte boundaries, tracking chunk counts, and reassembling streams on retrieval.
• Multiplied I/O Costs: Reading an oversized object split across four chunks requires a Query operation returning four items, multiplying capacity unit consumption across separate network frames.
• Inconsistent Reads: Without using TransactWriteItems, updating a chunked record is not atomic. If a process fails midway through updating four chunks, readers encounter torn writes and corrupted states.
• Transaction Boundaries: Transactions enforce strict item count and aggregate payload caps, meaning chunking cannot scale to large files or multi-megabyte datasets.

The Modern AI Agent Bottleneck

These legacy workarounds break down when applications incorporate autonomous AI agents and language models. Modern AI agents do not merely exchange short key-value pairs; they produce complex multi-step reasoning traces, comprehensive code repository diffs, tabular analytical datasets, and detailed audit reports.

Attempting to store rich agent execution logs and research artifacts inside DynamoDB items creates constant operational friction. Context windows and document corpora require a purpose-built storage layer rather than relational or NoSQL database records.

Production engineering teams resolve database size constraints by implementing the Claim-Check Pattern. Instead of embedding large data payloads directly into DynamoDB items, the application writes the complete artifact to an external workspace and stores only a lightweight reference pointer in the database.

Producer / Agent ---> (1) Upload Raw Artifact -------> Persistent Workspace
       |                                                       |
       +--------> (2) Save Pointer Item (<1 KB) --> DynamoDB   |
                                                       |       |
Consumer / Agent <--- (3) Read Pointer Metadata <------+       |
       |                                                       |
       +--------> (4) Stream or Search Excerpts <---------------+

This pattern keeps DynamoDB items compact and within 1 KB, consuming minimal capacity units while providing fast indexing. DynamoDB serves as a fast, indexed catalog of entity metadata, while external workspaces handle file persistence, indexing, and multi-user access.

Platform Attachment Limits and Large Document Realities

Decoupling is essential when integrating autonomous agents like Claude, Codex, or Cursor into backend workflows. AI platforms enforce strict file handling boundaries. As documented by the Claude Help Center, Claude chat sessions accept up to 20 files at up to 500 MB per file, while Claude Projects accept files up to 30 MB each with no fixed file-count cap, bounded only by the model's context window.

When agents generate research summaries or process document libraries, passing entire file contents through database records or prompt context exhausts model context windows and triggers DynamoDB ValidationException errors.

The Intelligent Workspace Substrate on Fast.io

Instead of storing raw files in unindexed object buckets, teams connect their data pipelines to org-owned Fast.io workspaces. Fast.io provides an intelligent workspace substrate where files are organized, versioned, and indexed for search on arrival.

• Cloud Sync and Import: Import files once, or connect shared folders using Cloud Sync for Google Drive, Dropbox, Box, and OneDrive (operating on a configured schedule or on demand, never continuous or real-time).
• Intelligence Mode: Once files land in a workspace, Intelligence Mode auto-indexes content across full-text, semantic vectors, and metadata values. AI agents do not need to download multi-megabyte files to answer questions; they search indexed passages directly.
• Metadata Views: For structured document pipelines, Metadata Views extract typed schemas (Text, Integer, Decimal, Boolean, URL, JSON, Date & Time) without OCR templates or manual parsing rules.
• Granular Collaboration Controls: Fast.io provides per-file version history, a detailed activity log, Collaborative Notes for real-time co-editing between people and agents, and secure ownership transfer from agents to human managers.
• Workspace Events Feed: Services subscribe to workspace activity over WebSocket or long-polling (GET /current/activity/poll/{entity_id}?wait=95&lastactivity={timestamp}) to react to file changes without webhooks.

Connecting AI Agents via the Model Context Protocol

AI agents interact with Fast.io workspaces through the remote Model Context Protocol (MCP) server over Streamable HTTP:

• Use https://mcp.fast.io/mcp/tools for Claude applications, Manus, and general MCP clients.
• Use https://mcp.fast.io/mcp/operations for ChatGPT and Codex (installing the Fastio plugin is preferred).
• Use https://mcp.fast.io/mcp/code for coding agents including Claude Code, Cursor, and Gemini CLI.

Setup documentation is available at https://mcp.fast.io/docs, with tool surface specifications at https://mcp.fast.io/skill.md.

When an agent receives a DynamoDB claim check pointing to a workspace document, it queries the workspace through the consolidated MCP storage read tool using the search action:

{
  "method": "tools/call",
  "params": {
    "name": "storage",
    "arguments": {
      "action": "search",
      "search": "unreconciled liabilities in division Q3 balance sheet",
      "files_scope": ["financial_audit_q3_report.pdf"]
    }
  }
}

The workspace searches indexed content and returns precise paragraphs with source citations. An operation that would exceed DynamoDB's 400 KB limit if stored directly consumes minimal prompt context and 1 WCU of database capacity.

Offload Large Payloads to Intelligent Workspaces

Store multi-gigabyte agent artifacts, tool outputs, and document sets beyond DynamoDB's 400 KB limit. Connect AI agents over MCP, index files automatically for hybrid search, and maintain an immutable audit trail. Monthly plans start with a 14-day free trial.

Start 14-Day Trial

The following implementation demonstrates how to build an automated claim-check pattern in Python. The service inspects payload size, writes compact records directly to DynamoDB, and offloads items larger than 50 KB to an external workspace before saving a pointer record.

The Claim-Check Python Service

import hashlib
import json
import logging
import boto3
from botocore.exceptions import ClientError

logger = logging.getLogger("storage.claim_check")
logger.setLevel(logging.INFO)

INLINE_BYTE_THRESHOLD = 50 * 1024

def measure_payload_size(data: dict) -> tuple:
    serialized = json.dumps(data, separators=(",", ":")).encode("utf-8")
    return serialized, len(serialized)

def offload_to_workspace(entity_id: str, payload_bytes: bytes, filename: str) -> dict:
    sha256_hash = hashlib.sha256(payload_bytes).hexdigest()
    byte_size = len(payload_bytes)
    workspace_uri = f"fastio://workspaces/ws_prod_core/artifacts/{entity_id}/{filename}"
    logger.info("Payload size %d bytes exceeds threshold. Offloading to %s", byte_size, workspace_uri)
    return {
        "storage_type": "claim_check",
        "file_uri": workspace_uri,
        "byte_size": byte_size,
        "sha256": sha256_hash,
        "filename": filename,
        "mime_type": "application/json"
    }

def put_record(table, entity_id: str, record_type: str, payload: dict, filename: str = "payload.json") -> dict:
    payload_bytes, byte_size = measure_payload_size(payload)
    if byte_size <= INLINE_BYTE_THRESHOLD:
        item = {
            "pk": f"ENTITY#{entity_id}",
            "sk": f"TYPE#{record_type}",
            "storage_type": "inline",
            "byte_size": byte_size,
            "data": payload
        }
    else:
        pointer = offload_to_workspace(entity_id, payload_bytes, filename)
        item = {
            "pk": f"ENTITY#{entity_id}",
            "sk": f"TYPE#{record_type}",
            **pointer
        }
    try:
        table.put_item(Item=item, ReturnConsumedCapacity="TOTAL")
        return item
    except ClientError as err:
        if err.response["Error"]["Code"] == "ValidationException":
            logger.error("DynamoDB rejected write: item exceeds 400 KB limit")
        raise

def get_record(table, entity_id: str, record_type: str) -> dict:
    response = table.get_item(Key={"pk": f"ENTITY#{entity_id}", "sk": f"TYPE#{record_type}"})
    item = response.get("Item")
    if not item:
        return None
    if item.get("storage_type") == "claim_check":
        logger.info("Resolving claim check reference: %s", item["file_uri"])
        return {
            "entity_id": entity_id,
            "record_type": record_type,
            "claim_check": item,
            "resolved_data": None
        }
    return item.get("data")

Key Design Principles for Production Claim Checks

1. Set Defensive Thresholds Below 400 KB: Do not wait until an item reaches 399 KB to offload. Setting a threshold between 20 KB and 50 KB prevents capacity unit spikes and avoids hitting partition limits under heavy traffic.
2. Store Immutability Hashes: Always record a SHA-256 or SHA-512 checksum in the DynamoDB claim-check record. When a downstream consumer downloads the file, it verifies the payload against the hash to confirm data integrity.
3. Standardize the Envelope: Use a uniform dictionary schema for claim-check pointers across all microservices, ensuring that automated consumers can detect whether an item requires hydration.
4. Handle Poison References: If a claim check points to a workspace file that was deleted or quarantined, the consumer should log a structured warning and route the event to a dead-letter queue rather than retrying indefinitely.

Architectural choices depend on payload dimensions, access frequency, query requirements, and team collaboration needs. The following matrix compares the five primary patterns for handling large payloads in DynamoDB:

Decision Framework: When to Use Each Approach

Evaluate your architectural requirements against these guidelines:

• Use Direct DynamoDB Items: When payloads remain consistently below 10 KB, contain structured business entities, and require high-frequency attribute filtering or conditional updates.
• Use Attribute Compression: When storing repetitive text or legacy JSON logs between 100 KB and 300 KB that do not require server-side filtering, and your infrastructure can tolerate client-side CPU overhead.
• Use Vertical Partitioning: When an entity consists of multiple discrete, independently accessed collections (such as user comments or daily timesheets) that naturally map to distinct sort keys under a single partition key.
• Use Raw S3 Object Pointers: When storing static, unstructured binary files (such as bulk database backups or archived machine data) that require long-term cold storage without interactive search.
• Use Intelligent Workspaces: When building AI agent workflows, automated research pipelines, or collaborative business systems where multi-megabyte documents, code artifacts, and tool outputs must remain searchable, auditable, and accessible to both autonomous agents and human teams.