SFTP File Size Limit: Protocol Constraints, Packet Buffers, and Large Transfers

64-Bit Integer Offsets Versus Legacy 32-Bit FTP Ceilings

A frequent source of confusion across technical documentation is conflating SFTP with classic FTP or FTPS. Classic FTP, originally specified in RFC 959, predated widespread 64-bit computing. Early FTP client and server implementations relied on standard 32-bit signed integers to track file pointers and stream sizes.

A signed 32-bit integer tops out at 2^31 - 1 bytes, creating an arbitrary 2 GB transfer protocol threshold. When legacy FTP daemons attempted to transfer a file exceeding this 2 GB boundary, the internal file offset counter wrapped into negative numbers, causing immediate transfer aborts, file corruption, or silent truncations. Even after FTP extensions added Large File Support (LFS) to handle 64-bit offsets, older scripts and appliances remained bottlenecked by 32-bit boundaries.

SFTP is not an extension of FTP; it is an independent subsystem operating inside an encrypted Secure Shell (SSH) transport session. Because SFTP version 3 defined its offset fields as 64-bit integers from the start, it never inherited the classic 2 GB barrier that plagued older transfer tools.

Protocol Packets Versus File Payloads

To understand how SFTP transfers massive files without a file-level cap, you must separate the overall file payload from individual protocol packets. SFTP does not transmit a file as a single unbroken stream. Instead, the client divides the file into smaller discrete data chunks, encapsulating each chunk inside an individual request packet.

While the file size offset is a 64-bit integer, individual packet lengths are governed by 32-bit fields. The specification dictates that the wire packet length field indicates the total size of the message following the field. However, implementations do not send gigabyte-sized packets. Instead, servers specify baseline expectations for maximum packet sizes to prevent buffer overflows and memory exhaustion on the host machine.

A large file transfer consists of sequential or pipelined packet requests, each carrying a payload chunk (typically between 32 KB and 256 KB) directed at an advancing 64-bit byte offset. If a transfer fails mid-stream, the cause is never that the file exceeded the protocol envelope; the failure points directly to intermediate network interruptions, daemon buffer mismatches, or target storage restrictions.

Operating System Resource Quotas and Ulimit Constraints

Beyond physical filesystem formatting, the operating system account assigned to the SFTP connection may be subject to administrative resource restrictions. In multi-tenant environments, systems administrators configure process limits using ulimit or PAM configuration files.

The maximum file size created by a user process is governed by the fsize resource limit:

# check the maximum file size limit for the current shell session
ulimit -f

# check soft and hard file size boundaries
ulimit -Sa
ulimit -Ha

If the user account running the SFTP subsystem has fsize set to a specific block quota, the Linux kernel sends a SIGXFSZ signal to the server process the moment a write crosses that boundary. If the daemon does not catch the signal, the process exits abruptly, severing the SSH connection without providing a helpful error message to the client.

Similarly, user and group disk quotas monitor available storage blocks on the target mount. When a user quota is exhausted during a transfer, the write operation halts with SSH_FX_FAILURE, leaving a partial, corrupted file on disk.

Cloud Gateway Limits: AWS Transfer Family and Azure SFTP

Organizations frequently replace self-hosted SFTP servers with managed cloud SFTP endpoints, such as AWS Transfer Family or Azure Blob Storage SFTP. While these services eliminate host server maintenance, they enforce application-level file size and upload mechanics that supersede the SFTP protocol.

In AWS Transfer Family, files uploaded via SFTP are converted into Amazon S3 objects, subject to S3 single-object ceilings. Furthermore, because standard SFTP clients perform sequential writes at byte offsets rather than S3 multi-part API calls, AWS Transfer Family buffers writes into multi-part uploads in the background. If a transfer stream becomes inconsistent or client timeouts occur, the background multi-part upload fails to complete.

Similarly, Azure Blob Storage SFTP translates incoming SFTP data into Azure Block Blobs. Depending on the configured block size, Azure limits maximum block blob capacity based on its 50,000 block threshold per blob. Cloud infrastructure also introduces request throttling, API concurrency limits, and idle session disconnects that terminate transfers that run for multiple hours.

Comparison Table of SFTP Limits Across Environments

The following comparison table summarizes theoretical protocol specifications, implementation thresholds, and physical storage limits:

This architectural comparison demonstrates that resolving large file transfer problems requires auditing the host filesystem, user quotas, and network configuration rather than attempting to alter protocol-level settings.

The OpenSSH 256KB Packet Ceiling and limits@openssh.com

Historically, the disparity between the client's conservative 32 KB chunk default and the server's 256 KB buffer capacity resulted in suboptimal transfer speeds, especially over high-bandwidth, high-latency network routes. To complete a massive file transfer using conservative 32 KB chunks, the client must execute hundreds of thousands of round-trip request and acknowledgment cycles.

To overcome this latency barrier without breaking protocol stability, OpenSSH 8.6 introduced a proprietary protocol extension named limits@openssh.com. During the initial connection handshake, an SFTP client supporting this extension sends an extended request querying limits@openssh.com. The server returns an extended reply declaring its exact operational boundaries:

uint64     max-packet-length
uint64     max-read-length
uint64     max-write-length
uint64     max-open-handles

In standard OpenSSH server builds, max-packet-length is reported as 262,144 bytes (256 KB), max-read-length as 262,144 bytes, and max-write-length as 261,120 bytes (SFTP_MAX_MSG_LENGTH - 1024, reserving space for headers). When an OpenSSH client connects to an OpenSSH 8.6+ server, it automatically scales its internal transfer chunks up from 32 KB to the maximum supported write length. This dynamic negotiation substantially reduces the total number of round trips, dramatically increasing throughput on large file transfers.

Tuning Client Buffers and Concurrency for High-Throughput Transfers

If you are running automated transfer scripts or command-line operations with older OpenSSH client versions, or connecting to third-party SFTP appliances that do not advertise limits@openssh.com, you can tune buffer sizes manually.

The OpenBSD sftp(1) client provides the -B flag to define buffer capacity. As documented in the OpenBSD manual pages, the default buffer size is 32,768 bytes. Specifying a larger buffer size reduces the number of round trips required to transmit the file, provided the target server accepts the packet dimensions:

# transfer a large archive with a 256KB buffer chunk size
sftp -B 262144 user@remote-server.com

In addition to buffer chunk size, transfer throughput over high-latency networks depends heavily on request pipelining. Because SFTP operates over TCP, waiting for an acknowledgment packet after every single chunk leaves the network link idle for the duration of the round-trip time (RTT). The OpenSSH sftp client manages this through concurrent in-flight requests, configured with the -R parameter:

# configure 64 concurrent outstanding request packets
sftp -B 262144 -R 64 user@remote-server.com

With multiple requests pipelined concurrently, the client keeps megabytes of data in flight simultaneously. This saturates available network bandwidth and ensures that multi-gigabyte transfers complete rapidly before encountering transient network interruptions.

Preventing Connection Drops with Keepalive Directives

To prevent stateful firewalls and NAT routers from tearing down long-running large file transfers, administrators must configure SSH keepalive packets. These null packets ensure that data traverses the encrypted tunnel at regular intervals, keeping state tables active even during disk write stalls.

On the client side, configure keepalive parameters in ~/.ssh/config:

Host *
    ServerAliveInterval 30
    ServerAliveCountMax 10
    TCPKeepAlive yes

• ServerAliveInterval 30: Instructs the SSH client to send an encrypted heartbeat probe through the tunnel if 30 seconds pass without any data packets.
• ServerAliveCountMax 10: Tells the client to keep attempting probes until ten consecutive heartbeats go unacknowledged before declaring the session dead.

On the host SFTP server, configure equivalent settings in /etc/ssh/sshd_config:

ClientAliveInterval 30
ClientAliveCountMax 10
TCPKeepAlive yes

Another subtle cause of mid-transfer aborts on massive files is SSH session rekeying. By default, OpenSSH initiates a cryptographic rekey after transferring a specific data threshold or after one hour. On high-throughput connections, frequent rekeying can introduce micro-stalls that cause sensitive client software to time out. You can adjust the rekey threshold in sshd_config:

# rekey after 10G data or 1h to maintain cipher freshness
RekeyLimit 10G 1h

Resuming Partial Uploads and Verifying File Integrity

When a multi-gigabyte SFTP transfer aborts despite keepalive tuning, re-uploading the entire file from byte zero consumes unnecessary bandwidth and delays data pipelines. The standard OpenSSH sftp interactive client includes native commands to resume broken transfers:

# resume a broken download from the remote server
sftp> reget remote_archive.tar.gz

# resume a broken upload to the remote server
sftp> reput local_archive.tar.gz

The reget and reput commands inspect the existing partial file size on the destination, seek to that exact 64-bit byte offset, and resume streaming only the remaining payload.

However, automated shell scripts often require non-interactive resumption. When building headless data migration pipelines, pairing SSH with rsync provides reliable, resumable file synchronization with partial progress retention:

# transfer large files over SSH with automatic resumption and progress tracking
rsync -avzP --partial --inplace -e ssh /data/backups/ user@remote-server.com:/storage/backups/

End-to-End File Integrity Verification

A significant limitation of SFTP version 3 is the absence of native end-to-end cryptographic checksumming. Unlike modern object storage APIs that validate MD5 or SHA-256 hashes as part of the upload transaction, SFTP v3 acknowledges that packets were received, but provides no built-in mechanism to confirm that the file on disk matches the source file bit-for-bit.

To ensure data integrity after transferring massive files over SFTP, always execute post-transfer checksum validation:

# generate local checksum
sha256sum local_archive.tar.gz > local.sha256

# generate remote checksum over SSH and compare
ssh user@remote-server.com "sha256sum /storage/local_archive.tar.gz" > remote.sha256
diff -u local.sha256 remote.sha256

Confirming checksum equality guarantees that intermediate packet retries, disk write stalls, or unexpected socket drops did not corrupt the file payload.

Connecting AI Agents to Large Corpora via Fast.io MCP

Fast.io integrates directly with modern AI agents through the Model Context Protocol (MCP) using Streamable HTTP. Instead of scripting custom SFTP connections or managing file transfer daemons, developers connect agents to Fast.io workspaces using designated client endpoints:

• https://mcp.fast.io/mcp/tools for Claude desktop, mobile, web, Claude Cowork, and general agent clients.
• https://mcp.fast.io/mcp/code for coding agents operating in IDEs and terminals, including Claude Code, Cursor, Cline, and Gemini CLI.
• https://mcp.fast.io/mcp/operations for custom ChatGPT and Codex configurations (with the Fastio plugin being the preferred setup in ChatGPT and Codex).

Refer to the Fast.io MCP documentation for client-specific configuration guides.

Through the consolidated MCP toolset, agents authenticate securely via OAuth or scoped Bearer API keys. Agents can list workspace contents, execute structured metadata extraction using Metadata Views, retrieve targeted file chunks, and write outputs back to persistent workspaces with full version history, avoiding the operational overhead of managing physical SFTP servers.

The following table outlines available organization plans and capacities:

Monthly plans start with a 14-day free trial that requires a credit card, providing a production-ready foundation for high-capacity file collaboration. Review available options on the pricing page or explore storage for agents.

Architectural Comparison: SFTP Versus Intelligent Workspaces

The differences between maintaining traditional SFTP server endpoints and deploying intelligent agent workspaces illustrate why engineering teams are updating their file architecture:

• Transfer Mechanics: SFTP relies on sequential packet chunks over a single SSH session that is sensitive to network latency and firewall timeouts. Fast.io uses chunked multi-part cloud uploads designed for high-capacity transfers.
• Data Usability: Files uploaded to an SFTP server remain dormant bytes on a disk partition. Fast.io auto-indexes uploaded files, allowing humans and AI assistants to search, summarize, and cross-reference documents immediately.
• Collaboration and Delivery: Sharing an SFTP folder requires provisioning SSH user accounts and directory permissions. Fast.io provides branded shares (Send, Receive, and Exchange) that can be durable or expiring, with per-recipient access and the ability to disable file downloads without consuming extra member seats.
• Ownership Transfer: Autonomous agents can establish workspaces, ingest project files, configure shares, and transfer ownership to human stakeholders while retaining administrative access.