Virtual File System in Python: In-Memory VFS, Limits, and Agent Sandboxes

When autonomous agents parse documents, analyze code, or run evaluation sandboxes, local file storage quickly runs into hard memory and application boundaries. As documented by the Anthropic Help Center, Claude chat sessions accept up to 20 files at up to 500 MB per file, while Claude Projects accept files up to 30 MB each with no fixed file-count cap, bounded only by the model's context window. Crossing these thresholds or attempting to buffer gigabytes of unindexed files directly inside runtime environments causes immediate ingestion failures and process memory exhaustion.

A Python virtual file system is an in-memory or abstracted file storage layer created using Python libraries like PyFilesystem2 or fsspec that mimics a physical disk filesystem within the Python process runtime.

Under standard execution, Python interacts with storage through built-in primitives like open(), the os module, and pathlib.Path. These abstractions issue system calls directly to the host kernel, which resolves path lookups across physical disk controllers, SSD blocks, or network mounts. In contrast, a virtual file system (VFS) intercepts path resolution and byte streaming within user space. It translates standard file operations into operations on in-memory buffers, compressed archives, or remote object stores without writing temporary artifacts to host disk drives.

Standard Library Primitives Versus Dedicated VFS Frameworks

Python developers frequently attempt to build ad-hoc virtual storage using standard library modules. While these modules handle single-file buffering, they fall short of providing a true filesystem:

• io.BytesIO and io.StringIO: Provide in-memory byte and string streams that implement Python's io.IOBase interface. They work well for feeding a single PDF buffer to a parser or capturing CSV output from a script. However, they lack directory structures, path resolution, hierarchy traversal, file metadata, and namespace isolation.
• tempfile.TemporaryDirectory: Creates a physical directory on the host filesystem (typically under /tmp or the directory defined by the TMPDIR environment variable). While it provides a complete directory tree and standard POSIX semantics, it touches physical storage or host tmpfs, introduces disk cleanup overhead, and exposes the host filesystem to untrusted file writes from autonomous agents.

When applications require complete directory trees, recursive search, metadata inspection, and sandboxed isolation entirely in RAM, dedicated VFS libraries provide the necessary abstraction layer.

The two primary VFS libraries in Python are PyFilesystem2 (fs) and Filesystem Spec (fsspec). PyFilesystem2 treats filesystems as unified object-oriented instances, offering backends such as MemoryFS, TempFS, and ZipFS. fsspec provides a URL-style protocol interface (memory://, file://, s3://) designed for data tools like Pandas, Dask, and DuckDB.

The following comparison table highlights how Python virtual filesystem implementations handle storage media, persistence, scoping, concurrency, and operational limits:

• FastAPI Upload File Size Limits: Memory Spooling, Middleware, and Direct StorageFastAPI imposes no native file size limit, but Starlette spools multipart uploads in memory up to `1,048,576 bytes`...
• How to Use the LangChain File System for Persistent DataThe LangChain file system integration enables AI agents to read, write, and organize files on local disks or cloud...
• How to Download Files from Box via API: Endpoints, Tokens & LimitsDownloading files through the Box REST API requires requesting the GET /files/{file_id}/content endpoint and following...
• Dropbox API Download File: Endpoints, Rate Limits & Agent WorkflowsThe Dropbox API download file endpoint (/2/files/download) streams raw binary files by passing parameters through HTTP...
• Google AI Studio File Upload Limits: File API Constraints and Large CorporaGoogle AI Studio and the Gemini File API enforce a 2 GB per-file upload limit and a 20 GB total project storage cap,...
• Notion File Upload Limits: Size Caps by Plan and Large-File WorkaroundsNotion limits individual file uploads to 5MB on the Free plan and provides unlimited file uploads with a 5GB maximum...

More on this subject: Agent File and Document Workflows (298 guides)

Creating an in-memory virtual file system in Python requires choosing between PyFilesystem2's object-oriented API and fsspec's URL-driven protocol architecture. Each approach handles namespace scoping and memory allocation differently.

Using PyFilesystem2 MemoryFS

PyFilesystem2 represents every filesystem as an instance of fs.base.FS. To create an in-memory filesystem, instantiate fs.memoryfs.MemoryFS directly or open it using the mem:// resource opener.

Every MemoryFS instance allocates its own independent virtual root in RAM:

from fs.memoryfs import MemoryFS

with MemoryFS() as mem_fs:
    mem_fs.makedirs("workspace/reports")
    mem_fs.makedirs("workspace/raw_data")
    mem_fs.writetext("workspace/reports/summary.md", "Agent Execution Report: All tasks completed.")
    mem_fs.writebytes("workspace/raw_data/output.bin", b"\x00\x01\x02\x03\x04")
    if mem_fs.exists("workspace/reports/summary.md"):
        info = mem_fs.getinfo("workspace/reports/summary.md", namespaces=["basic", "details"])
        print(f"File size: {info.size} bytes")
        print(f"Created: {info.created}")
    content = mem_fs.readtext("workspace/reports/summary.md")
    print(content)
    for path in mem_fs.walk.files():
        print(f"Virtual file: {path}")

When the with block exits, mem_fs.close() runs automatically. It discards the root directory structure and releases the underlying byte buffers for Python garbage collection.

Using fsspec MemoryFileSystem

The fsspec library approaches virtual storage through protocol strings and filesystem registries. Its in-memory implementation is MemoryFileSystem, reachable through the memory:// protocol prefix:

import fsspec

fs = fsspec.filesystem("memory")
with fs.open("memory://agent_output/results.json", "w") as f:
    f.write('{"status": "success", "records_processed": 1420}')

file_list = fs.ls("memory://agent_output")
print(f"Files in memory: {file_list}")

with fs.open("memory://agent_output/results.json", "r") as f:
    payload = f.read()
    print(f"Read payload: {payload}")

Isolating Virtual Namespaces for Multi-Agent Sandboxes

A critical architectural difference between PyFilesystem2 and fsspec lies in state sharing. PyFilesystem2 isolates each MemoryFS() object by default. Two separate calls to MemoryFS() create two distinct, unlinked virtual filesystems in memory.

In contrast, fsspec.filesystem("memory") uses a shared global dictionary by default (global_store=True). If Agent A and Agent B run in different threads within the same Python process and both write to memory://scratch/output.txt, Agent B will silently overwrite Agent A's file.

To prevent data corruption in multi-agent environments, configure fsspec with explicit isolation:

from fsspec.implementations.memory import MemoryFileSystem

isolated_fs_a = MemoryFileSystem(global_store=False, skip_instance_cache=True)
isolated_fs_b = MemoryFileSystem(global_store=False, skip_instance_cache=True)

isolated_fs_a.pipe_text("scratch/task.txt", "Data for Agent A")
isolated_fs_b.pipe_text("scratch/task.txt", "Data for Agent B")

print(isolated_fs_a.cat_text("scratch/task.txt"))
print(isolated_fs_b.cat_text("scratch/task.txt"))

Setting global_store=False and skip_instance_cache=True ensures each agent sandbox operates in its own memory partition.

While in-memory virtual filesystems eliminate disk latency, they introduce constraints that do not exist on physical storage devices. Understanding these limits prevents unexpected process crashes during long-running agent workflows.

Python in-memory VFS instances are strictly capped by available process heap memory and default OS file descriptor limits (typically 1024 or 4096 on Linux/macOS). PyFilesystem2 memory filesystem (MemoryFS) stores entire file buffers in RAM, creating memory exhaustion risks during large dataset processing.

Process Heap Exhaustion and Memory Retention

When an agent writes files into MemoryFS or MemoryFileSystem, the bytes are stored directly as Python objects on the process heap. If an autonomous agent extracts a large multi-gigabyte archive, generates high-dimensional vector embeddings, or parses dense log dumps into an in-memory VFS, the Python runtime allocates that memory from the operating system.

Python's internal memory manager (pymalloc) handles allocations smaller than 512 bytes in dedicated memory arenas, while larger allocations go directly to the system malloc. Even after an application deletes a file inside MemoryFS using fs.remove(), the underlying bytes are only freed within Python's heap. The host operating system does not necessarily reclaim the resident set size (RSS) memory immediately. If consecutive agent runs process large files without terminating the process, memory fragmentation leads to Out-Of-Memory (OOM) termination by the host operating system.

OS File Descriptor Ceilings and Open Handle Leaks

Even though files in a virtual filesystem exist in RAM, opening a virtual file often creates Python stream wrappers, buffered reader objects, and OS handles if the implementation bridges to temporary disk buffers.

On Linux and macOS, the operating system enforces a strict ceiling on the number of open file descriptors an individual process may hold simultaneously. Check this limit in your shell using ulimit -n:

• Linux Default: Typically 1024 open descriptors for standard user processes, or 4096 on modern distributions managed by systemd.
• macOS Default: Typically 256 or 1024 open descriptors depending on shell profile settings.

When multi-threaded agents open virtual files in tight loops without using context managers, file handles remain active until the garbage collector runs. If active handles cross the ulimit -n ceiling, Python raises an OSError with Errno 24 indicating too many open files.

The following diagnostic pattern tracks process memory and file descriptor allocation in Python:

import gc
import os
import resource

def inspect_runtime_limits():
    soft_fd, hard_fd = resource.getrlimit(resource.RLIMIT_NOFILE)
    usage = resource.getrusage(resource.RUSAGE_SELF)
    max_rss_mb = usage.ru_maxrss / (1024 if os.uname().sysname == "Linux" else 1024 * 1024)
    print(f"OS File Descriptor Limits: soft={soft_fd}, hard={hard_fd}")
    print(f"Peak Resident Memory: {max_rss_mb:.2f} MB")

inspect_runtime_limits()

Concurrency Hazards in Multi-Agent Execution

In multi-agent systems where parallel worker threads or coroutines share a virtual filesystem, concurrency bugs emerge quickly:

1. Compound Check-Then-Act Race Conditions: PyFilesystem2 provides thread synchronization for individual primitive calls using an internal reentrant lock (threading.RLock). However, compound sequences like checking if a file exists before writing are not atomic. Two agent threads checking the same path simultaneously will both evaluate existence checks to False, leading to unexpected overwrites.
2. fsspec Global Namespace Collisions: Because fsspec shares state across threads by default, asynchronous workers sharing a memory:// protocol without path prefixes will overwrite each other's intermediate state.
3. Missing Advisory Locks: Standard Python in-memory filesystems lack cross-agent advisory locking. When one agent reads an analytical dataset while another is updating it, the reader observes half-written records or corrupted JSON structures.

Bridge Python Agent Sandboxes to Intelligent Workspaces

Connect Claude, Cursor, and Python agents to Fastio workspaces over Streamable HTTP MCP. Store multi-gigabyte outputs, index documents for hybrid search, and preserve complete version history across sessions. Monthly plans start with a 14-day free trial.

Start 14-Day Trial

In-memory virtual filesystems are effective for scratchpad computation, fast unit testing, and isolated code evaluation. However, production agent pipelines require a durable persistence layer. When an agent process completes its run, crashes, or scales down in a container cluster, all data stored in MemoryFS disappears immediately.

Furthermore, in-memory sandboxes cannot be searched across historical agent runs, audited by engineering leads, or shared with human team members. To build production agent systems, engineering teams bridge ephemeral in-memory sandboxes to persistent cloud workspaces.

Connecting Ephemeral Sandboxes to Fastio Workspaces

Fastio provides shared org-owned workspaces that serve as the persistent substrate for agentic teams. Rather than managing physical disk volumes or complex object store lifecycles, agents commit completed deliverables directly from Python memory into persistent workspaces.

Fastio incorporates key capabilities designed for human-agent collaboration:

• Per-File Version History: Every write to a workspace creates a distinct version. If an agent updates an executive brief or refines a dataset, prior versions remain intact and restorable. Concurrent agent writes both land safely in version history.
• Role-Based Permissions: Permissions are configured per organization, workspace, and share through member roles (never per folder or per file). This design ensures security boundaries remain predictable across multi-agent fleets.
• Detailed Activity Log: Fastio maintains a detailed activity log that records changes across the organization, including uploads, edits, moves, deletions, membership updates, and share creation. It does not record read-only events like views or downloads.
• Intelligence Mode and Hybrid Search: Enabling Intelligence Mode on a workspace automatically indexes files for hybrid search (combining full-text search, semantic search, and metadata filtering). Agents search indexed documents across the workspace without loading entire corpora into context prompts.
• Metadata Views: Teams turn unstructured documents into live, queryable databases using Metadata Views. AI automatically designs a typed schema (Text, Integer, Decimal, Boolean, URL, JSON, Date & Time), extracts structured values, and populates a filterable spreadsheet view that agents query programmatically.
• Cloud Sync: Cloud Sync connects Google Drive, Dropbox, Box and OneDrive folders to a workspace, one-way or two-way, on a schedule or on demand. Agents read synced assets and file deliverables back into the same shared workspace.
• E-Signature: Workspaces support sending documents for signature with sequential or parallel signers, optional one-time passcodes, a tamper-evident audit certificate, and automatic filing of the executed copy back into workspace storage.

Exporting from In-Memory VFS to Fastio via MCP

Autonomous agents interact with Fastio using the Model Context Protocol (MCP) over Streamable HTTP. Depending on your agent client, connect to the dedicated endpoint:

• Coding agents (Claude Code, Cursor, Gemini CLI, Devin, Cline): Connect to https://mcp.fast.io/mcp/code.
• ChatGPT and Codex custom servers: Connect to https://mcp.fast.io/mcp/operations.
• General agents and chat apps (Claude Desktop, Claude apps, custom agent runtimes): Connect to https://mcp.fast.io/mcp/tools.

The following Python example shows how an agent finishes processing files in an in-memory MemoryFS sandbox and streams the resulting deliverable directly to a Fastio workspace using an HTTP client:

import os
from fs.memoryfs import MemoryFS
import httpx

with MemoryFS() as sandbox:
    sandbox.makedirs("analysis")
    sandbox.writetext("analysis/market_summary.md", "Autonomous Agent Analysis: Key growth drivers identified.")
    file_content = sandbox.readbytes("analysis/market_summary.md")

FASTIO_API_KEY = os.environ["FASTIO_API_KEY"]
WORKSPACE_ID = "ws_agent_production_01"

mcp_payload = {
    "jsonrpc": "2.0",
    "id": 1,
    "method": "tools/call",
    "params": {
        "name": "upload_manage",
        "arguments": {
            "action": "upload",
            "workspace_id": WORKSPACE_ID,
            "name": "market_summary.md",
            "content": file_content.decode("utf-8")
        }
    }
}

headers = {
    "Authorization": f"Bearer {FASTIO_API_KEY}",
    "Content-Type": "application/json",
    "Accept": "application/json, text/event-stream"
}

response = httpx.post(
    "https://mcp.fast.io/mcp/tools",
    json=mcp_payload,
    headers=headers,
    timeout=30.0
)

print(f"MCP Response Status: {response.status_code}")

Managing Concurrent Writes with Advisory File Locks

When multiple Python agents collaborate in the same workspace, they coordinate writes using advisory file locks. Agents acquire, heartbeat, and release locks through MCP actions:

• lock-acquire and lock-release on the storage_manage tool.
• lock-status on the storage tool.

A file lock represents an advisory lease. It signals to other agents and team members that a specific agent is updating the file. The lock expires automatically unless heartbeated by the holder. Importantly, advisory locks do not block read access or prevent overwrites: if two agents write simultaneously, both versions are captured safely in Fastio's per-file version history.

Monthly plans start with a 14-day free trial that requires a credit card. Plan tiers include Starter at $9.99/mo (3 seats, 250 GB storage, 5 workspaces, 100,000 credits a month), Business at $49.99/mo (10 seats, 5 TB storage, 50 workspaces, 600,000 credits a month), and Enterprise at $199.99/mo (30 seats included, 25 TB storage, 200 workspaces, 3,000,000 credits a month). Maximum upload sizes are 25 GB on Starter, 50 GB on Business, and 100 GB on Enterprise.

Engineering teams deploying file-aware Python agents must align storage architectures with operational requirements. Different stages of an agentic workflow require different balances of speed, isolation, and persistence.

Three Core Storage Patterns

1. Pure In-Memory Scratchpad (Evaluation & Code Sandboxes):
   • Mechanism: Agent instantiates MemoryFS or fsspec.filesystem("memory", global_store=False) for each isolated execution cycle.
   • Best For: Evaluating untrusted Python code, running syntax linters, parsing abstract syntax trees (ASTs), or transforming temporary JSON files.
   • Tradeoff: Zero disk I/O latency, but data is ephemeral and heap-bounded.
2. Tiered Spilling with MemoryFS and TempFS (Local Big Data Processing):
   • Mechanism: The agent monitors file buffer sizes. Small files stay in MemoryFS. When a payload crosses a defined memory threshold, the agent routes the stream to fs.tempfs.TempFS.
   • Best For: Extracting large multi-gigabyte TAR or ZIP archives, transcode pipelines, and batch CSV processing.
   • Tradeoff: Preserves heap memory, but incurs disk I/O and requires cleanup handling.
3. Hybrid VFS Sandbox with MCP Cloud Persistence (Production Agent Workflows):
   • Mechanism: The agent performs fast iterative operations inside an isolated Python VFS. Upon completing validation, it commits deliverables to Fastio workspaces via MCP.
   • Best For: Autonomous research agents, automated legal matter preparation, financial modeling, and collaborative human-agent workflows.
   • Tradeoff: Combines in-memory execution speed with durable, indexed, human-accessible persistence.

The following matrix evaluates each architectural pattern across operational dimensions:

Engineering Checklist for Python Agent Storage

Before deploying autonomous file-handling agents to production, verify these operational checkpoints:

• Enforce Namespace Isolation: Confirm fsspec memory filesystems use global_store=False and skip_instance_cache=True to prevent cross-agent data leaks.
• Use Context Managers: Wrap all MemoryFS and virtual file streams in with blocks to ensure prompt closure and avoid file descriptor leaks.
• Track Memory Thresholds: Monitor process resident set size (RSS) during batch file processing to prevent Linux OOM terminations.
• Decouple Scratch from Deliverables: Use in-memory VFS exclusively for scratch computation; stream finished artifacts to persistent workspaces via MCP.
• Coordinate Concurrent Writes: Acquire advisory file locks on shared files and rely on workspace version history to handle concurrent multi-agent updates.