Git File Size Limit: Protocol Thresholds, Push Blocks, and AI Datasets

Pre-Receive Hooks and the Quarantine Directory

Hosting services enforce limits at the boundary where local commits meet centralized infrastructure. When you run git push, the local Git client sends a negotiation request to the remote server, packs the missing objects into a thin packfile, and streams it to the remote daemon over SSH or HTTPS.

Before the remote server updates its branch references, it passes the incoming packfile through a quarantine environment and executes server-side pre-receive hooks. These hooks inspect the metadata and uncompressed size of every incoming blob.

If any blob in the packfile exceeds the hosting platform threshold, the hook exits with a non-zero code. This rejection aborts the entire push transaction atomically, preventing the large object from becoming part of the permanent remote repository history.

File Size Thresholds Across Major Git Hosting Remotes

Every major hosting provider defines specific thresholds for individual files and aggregate repository sizes. Understanding these limits prevents unexpected push blocks during automated deployments:

On GitHub, pushing a file between 50 MiB and 100 MiB succeeds but prints a warning urging you to remove the file or migrate it to Git LFS. Pushing any file exceeding 100 MiB terminates with an immediate pre-receive error on GitHub. GitLab enforces a 100 MiB per-file limit across its Free tier projects on GitLab.com. Bitbucket Cloud permits individual large files if network buffers allow, but locks the entire repository into read-only mode once the total repository size threshold is reached.

On self-hosted Git servers, administrators can configure the core threshold using git config core.bigFileThreshold 512M. This setting instructs Git to stream objects larger than 512 MiB directly to disk without attempting in-memory delta compression, avoiding memory allocation panics on the server.

Interpreting GitHub and GitLab Rejection Messages

On GitHub, attempting to push an object larger than 100 MiB returns error code GH001. The terminal output identifies the offending branch, file path, exact file size, and the remote limit:

Enumerating objects: 5, done.
Counting objects: 100% (5/5), done.
Delta compression using up to 10 threads
Compressing objects: 100% (4/4), done.
Writing objects: 100% (5/5), 142.30 MiB | 18.50 MiB/s, done.
Total 5 (delta 1), reused 0 (delta 0), pack-reused 0
remote: error: GH001: Large files detected. You may want to investigate Git Large File Storage - https://git-lfs.github.com.
remote: error: Trace: 4b98c397c02b37c093a8d9b1e7f60321a4f89d91782e
remote: error: See https://docs.github.com/articles/about-large-files-on-github/ for more information.
remote: error: File data/training_corpus.parquet is 142.30 MB; this exceeds GitHub's file size limit of 100.00 MB
To https://github.com/organization/project.git
 ! [remote rejected] main -> main (pre-receive hook declined)
error: failed to push some refs to 'https://github.com/organization/project.git'

In contrast, GitLab often reports the error using the raw Git object SHA-1 hash rather than the file path:

remote: GitLab: You are attempting to check in one or more files which exceed the 100MiB limit:
remote: - 257cc5642cb1a054f08cc83f2d943e56fd3ebe99 (142 MiB)
remote: 
remote: Please refer to https://docs.gitlab.com/user/free_user_limit/ for further information.
To https://gitlab.com/organization/project.git
 ! [remote rejected] main -> main (pre-receive hook declined)
error: failed to push some refs to 'https://gitlab.com/organization/project.git'

When GitLab or a custom Git hook provides only a 40-character object SHA, locate the corresponding file path in your local repository using git rev-list:

git rev-list --objects --all | grep 257cc5642cb1a054f08cc83f2d943e56fd3ebe99

This command inspects all reachable objects across all local branches and tags, matching the hash against Git's tree listings to print the path.

Auditing Large Blobs in Packfiles and HTTP Buffer Failures

If a repository has accumulated multiple large commits over time, audit the entire packfile index to rank objects by size. This inspection reveals files that should be stripped before pushing to a remote host.

Run git verify-pack on your pack index files and pipe the output to sort by object size:

git verify-pack -v .git/objects/pack/*.idx | sort -k 3 -g | tail -n 10

The third column of the output displays the uncompressed size in bytes, while the first column shows the object SHA. Once you identify the top offending hashes, convert each hash to its corresponding commit and file name:

git rev-list --objects --all | grep <object-sha>

Another failure mode during large pushes is transport buffer exhaustion over HTTPS. The error appears as:

error: RPC failed; HTTP 413 curl 22 The requested URL returned error: 413 Request Entity Too Large
fatal: the remote end hung up unexpectedly

This error does not indicate a Git hook rejection. Instead, an intermediate reverse proxy (such as Nginx, Cloudflare, or an API gateway) dropped the HTTP POST request because the body exceeded the proxy payload buffer. While increasing the buffer configuration resolves transient transport timeouts on self-hosted instances, it cannot bypass GitHub push blocks where GitHub blocks files larger than 100 MiB via pre-receive hooks.

Fixing the Most Recent Unpushed Commit with Amend

If you added the large file in your most recent commit and have not yet pushed to the remote server, you can remove the object by amending the commit.

Follow these four steps to resolve the push rejection:

1. Unstage the large file from the Git index while keeping the physical file on your disk:

git rm --cached data/training_corpus.parquet

2. Amend the previous commit to record the snapshot without the large file:

git commit --amend -CHEAD

The -CHEAD flag reuses the commit message and authorship metadata from the previous commit without opening an interactive editor prompt.

3. Add the file path or extension to .gitignore so Git ignores it during subsequent staging operations:

echo "data/training_corpus.parquet" >> .gitignore
git add .gitignore
git commit -m "Ignore large training corpus dataset"

4. Push the cleaned commit history to the remote repository:

git push origin main

Because the amended commit replaced the problematic commit entirely, the oversized blob is no longer part of the branch tip, and the remote pre-receive hook accepts the push.

Cleaning Recent History with Interactive Rebase

If you made several commits after introducing the large file, amending HEAD is insufficient. The large file remains trapped in an earlier commit on your local branch.

If the commits exist only on your local workstation and have not been shared with teammates, use an interactive rebase:

git rebase -i HEAD~5

In the interactive rebase list displayed in your editor, locate the commit that introduced the large file and change the command word from pick to edit. Save and close the editor.

Git pauses execution at the specified commit. Remove the large file from the index, amend the commit, and continue the rebase:

git rm --cached path/to/large_file.ext
git commit --amend --no-edit
git rebase --continue

Git replays subsequent commits on top of the modified snapshot without the large file. If later commits modified or referenced that file, resolve merge conflicts before continuing.

Purging Large Files Across All Branches with Git Filter-Repo

When a large file has been merged into multiple branches or committed far back in history, interactive rebase becomes impractical. In these scenarios, use git filter-repo, the official Python-based tool recommended by the Git core project to replace the deprecated git filter-branch and external tools like BFG Repo-Cleaner.

Before running repository-wide history rewriting, create a fresh backup clone of your repository in an independent directory:

git clone --mirror git@github.com:organization/project.git project-backup.git

Next, install git-filter-repo using your package manager and execute the cleanup operation in your primary working repository:

# remove a specific file path from all commits, branches, and tags
git filter-repo --path data/training_corpus.parquet --invert-paths

# alternatively, strip all objects larger than the 100 MiB limit that GitHub blocks
git filter-repo --strip-blobs-bigger-than 100M

The --invert-paths flag instructs git filter-repo to keep everything except the specified path. The --strip-blobs-bigger-than 100M flag automatically identifies and removes any historical blob larger than the 100 MiB limit that GitHub blocks, replacing it with an empty file marker.

After git filter-repo completes, expire local reference logs and run garbage collection:

git reflog expire --expire=now --all
git gc --prune=now --aggressive

Rewriting history alters commit SHA hashes across all affected branches. Coordinate with collaborators before force-pushing rewritten branches:

git push origin --force --all
git push origin --force --tags

Collaborators must create fresh clones of the sanitized repository to avoid reintroducing pruned blobs.

How LFS Pointer Files and Filter Drivers Operate

An LFS pointer file contains three lines of plain text that uniquely identify the file content:

version https://git-lfs.github.com/spec/v1
oid sha256:7b5d84a712f5e31c8901b2a9e3d4891b2c45e6f7a8b9c0d1e2f3a4b5c6d7e8f9
size 149258320

The oid represents the cryptographic SHA-256 checksum of the uncompressed binary, while size records the exact byte count. Because the pointer file is only approximately 130 bytes, Git commits, branches, and merges execute rapidly without inflating repository packfiles.

Git LFS coordinates file movements between your local disk and remote storage using Git filter drivers configured in .gitattributes:

• Clean Filter: When you run git add, the Git LFS clean filter computes the SHA-256 hash of the binary file, moves the payload into the local .git/lfs/objects/ store, and places the 130-byte pointer file in the staging index.
• Smudge Filter: When you run git checkout or git switch, the smudge filter reads the pointer file from the commit tree, looks up the corresponding binary in .git/lfs/objects/ (or downloads it from the remote LFS server if missing), and restores the full binary file in your working directory.

Configuring Git LFS Tracking Rules and Staging Files

To set up Git LFS in a repository, install the extension binary and configure tracking rules:

# initialize Git LFS hooks in your global or local environment
git lfs install

# track specific file extensions or directory paths
git lfs track "datasets/*.parquet"
git lfs track "*.onnx"

# ensure the tracking configuration is staged and committed
git add .gitattributes
git commit -m "Configure Git LFS tracking rules"

# stage and commit the large file normally
git add datasets/training_data.parquet
git commit -m "Add training data via Git LFS"

# push both standard Git commits and LFS binary objects to the remote
git push origin main

When you run git push, the Git LFS pre-push hook intercepts the operation. It uploads the large binary payloads directly to the remote LFS HTTP storage endpoint before Git pushes the commit pointers to the Git remote.

Operational Bottlenecks and Quota Caps in Git LFS

While Git LFS solves the push block where GitHub blocks files larger than 100 MiB, it introduces operational and economic tradeoffs that frequently disrupt development teams:

• Storage and Bandwidth Quotas: Remote hosting providers bill for Git LFS storage and bandwidth separately from code repositories, enforcing monthly quotas that automated CI/CD pipelines can exhaust quickly.
• Missing Dependency Failures: If a developer, CI runner, or automated agent clones a repository without the git-lfs binary, Git downloads the 130-byte text pointer instead of the binary file. Build tools and pipelines fail when attempting to parse pointer text as binary data.
• Pointer Desynchronization: Committing a large file before configuring .gitattributes writes a raw blob into Git history. Adding tracking rules later does not convert historical commits retroactively, leaving hybrid repositories with both packfile bloat and LFS overhead.
• Inefficient for Agent Context: Autonomous AI agents rarely need an entire multi-gigabyte dataset to inspect specific rows or answer questions. Git LFS requires pulling the entire binary file to local disk before an agent can read it, wasting local storage and bandwidth.

The Core Division: Version Control Versus Asset Storage

Code repositories and data storage serve two completely different operational requirements:

• Git for Code: Git excels at tracking short, human-authored text files, versioning logic changes, managing feature branches, and conducting pull request reviews. GitHub recommends repositories remain under 1 GB, allowing agents and developers to clone and branch in seconds.
• Workspaces for Context Assets: Benchmark corpora, fine-tuning datasets, SQLite databases, PDF technical manuals, media assets, and generated artifacts belong in an external, persistent storage layer. These files require fast access, high-capacity uploads, workspace and share permissions, and automated search capabilities without burdening the Git commit graph.

Connecting Coding Agents to Fastio via the Model Context Protocol

Fastio provides shared, organization-owned cloud workspaces designed for human teams and AI agents. Instead of forcing heavy binary assets through Git packfiles or wrestling with Git LFS bandwidth limits, development teams connect their agents directly to Fastio workspaces.

Workspaces support high-capacity file transfers, accommodating individual files up to plan limits (25 GB on Starter, 50 GB on Business, and 100 GB on Enterprise). Development teams store training sets, document archives, and project assets in workspaces where both humans and agents can interact with them.

Agents connect to Fastio through the Model Context Protocol (MCP) using Streamable HTTP. Depending on the agent runtime, configure the remote MCP endpoint:

• Coding Agents (Claude Code, Cursor, Devin, Copilot CLI, Cline): Connect to https://mcp.fast.io/mcp/code. This endpoint exposes tools optimized for coding workflows, including search, execute, execute_manage, upload, and upload_manage.
• Claude Applications and General Agents: Connect to https://mcp.fast.io/mcp/tools. This endpoint provides action-based tooling split between inspection tools (storage, workspace, upload, share) and management tools (storage_manage, workspace_manage, upload_manage, share_manage).
• ChatGPT and Codex: Connect via the Fastio plugin in the plugin directory, or configure custom servers using https://mcp.fast.io/mcp/operations.

When connecting Claude Code, add the remote endpoint in your terminal:

claude mcp add --transport http fast-io https://mcp.fast.io/mcp/code

Sign in via the browser OAuth prompt. Detailed configuration steps and tool documentation are available at Fastio MCP setup and Fastio MCP skill guide.

For developers integrating headless agents or custom Python pipelines, authentication uses a scoped API key passed via the Authorization: Bearer <api_key> header on the HTTP connection.

Intelligent Workspaces, Automated RAG, and Concurrency Controls

Storing datasets in an intelligent workspace transforms passive storage into an active retrieval layer:

• Automated RAG Indexing: Enabling Intelligence Mode on a Fastio workspace automatically indexes incoming documents, markdown files, PDFs, and data dumps upon arrival. Rather than stuffing massive corpora into LLM context windows or pulling them across a Git clone, an agent queries the workspace using hybrid search (combining full-text, semantic similarity, and metadata filtering). The agent retrieves relevant passages with source citations.
• Structured Document Extraction: Through Metadata Views, teams extract typed schema fields (Text, Integer, Decimal, Boolean, JSON, Date & Time) across unstructured files like PDFs, scanned sheets, and technical specs without writing custom OCR parsers or ingestion pipelines. Agents query extraction results directly over MCP.
• Concurrency and Version Safety: Fastio provides per-file version history, recording every file edit and upload so previous revisions remain accessible. For multi-agent workflows, agents acquire advisory leases using the lock-acquire action on the storage_manage tool to coordinate writers before updating shared files.
• Detailed Activity Auditing: Every organization maintains a detailed activity log of changes, tracking file uploads, revisions, deletions, and permission changes across human and agent collaborators.
• Multi-Cloud Integration: With Cloud Sync, teams connect existing folders from Google Drive, Dropbox, Box, and OneDrive to a Fastio workspace. Sync operates one-way or two-way, running on a scheduled interval or triggered on demand, eliminating the need to move existing cloud storage architectures.
• Predictable Billing: Monthly plans start with a 14-day free trial that requires a credit card, providing a production-ready foundation for high-capacity file collaboration. Review available options on the pricing page or explore storage for agents.