# Virtual File System in Python: In-Memory VFS, Limits, and Agent Sandboxes

A Python virtual file system creates an in-memory or abstracted storage layer within the process runtime, but heap boundaries and operating system file descriptor limits create severe ceilings during large data tasks. Autonomous agents require isolated sandboxes for ephemeral execution alongside persistent, indexed cloud workspaces to store deliverables and avoid memory exhaustion.

Source: https://fast.io/resources/virtual-file-system-python/
Author: [Derek Labian](https://fast.io/authors/derek-labian/)
Last reviewed: 2026-10-14

## What Is a Python Virtual File System?

When autonomous agents parse documents, analyze code, or run evaluation sandboxes, local file storage quickly runs into hard memory and application boundaries. As documented by the [Anthropic Help Center](https://support.claude.com/en/articles/8241126-upload-files-to-claude), Claude chat sessions accept up to 20 files at up to 500 MB per file, while Claude Projects accept files up to 30 MB each with no fixed file-count cap, bounded only by the model's context window. Crossing these thresholds or attempting to buffer gigabytes of unindexed files directly inside runtime environments causes immediate ingestion failures and process memory exhaustion.

A Python virtual file system is an in-memory or abstracted file storage layer created using Python libraries like PyFilesystem2 or fsspec that mimics a physical disk filesystem within the Python process runtime.

Under standard execution, Python interacts with storage through built-in primitives like `open()`, the `os` module, and `pathlib.Path`. These abstractions issue system calls directly to the host kernel, which resolves path lookups across physical disk controllers, SSD blocks, or network mounts. In contrast, a virtual file system (VFS) intercepts path resolution and byte streaming within user space. It translates standard file operations into operations on in-memory buffers, compressed archives, or remote object stores without writing temporary artifacts to host disk drives.

### Standard Library Primitives Versus Dedicated VFS Frameworks

Python developers frequently attempt to build ad-hoc virtual storage using standard library modules. While these modules handle single-file buffering, they fall short of providing a true filesystem:

- **`io.BytesIO` and `io.StringIO`:** Provide in-memory byte and string streams that implement Python's `io.IOBase` interface. They work well for feeding a single PDF buffer to a parser or capturing CSV output from a script. However, they lack directory structures, path resolution, hierarchy traversal, file metadata, and namespace isolation.
- **`tempfile.TemporaryDirectory`:** Creates a physical directory on the host filesystem (typically under `/tmp` or the directory defined by the `TMPDIR` environment variable). While it provides a complete directory tree and standard POSIX semantics, it touches physical storage or host tmpfs, introduces disk cleanup overhead, and exposes the host filesystem to untrusted file writes from autonomous agents.

When applications require complete directory trees, recursive search, metadata inspection, and sandboxed isolation entirely in RAM, dedicated VFS libraries provide the necessary abstraction layer.

The two primary VFS libraries in Python are PyFilesystem2 (`fs`) and Filesystem Spec (`fsspec`). PyFilesystem2 treats filesystems as unified object-oriented instances, offering backends such as `MemoryFS`, `TempFS`, and `ZipFS`. `fsspec` provides a URL-style protocol interface (`memory://`, `file://`, `s3://`) designed for data tools like Pandas, Dask, and DuckDB.

The following comparison table highlights how Python virtual filesystem implementations handle storage media, persistence, scoping, concurrency, and operational limits:

| VFS Implementation | Storage Medium | State Persistence | Default Scope | Concurrency Model | Primary Hard Ceiling |
| --- | --- | --- | --- | --- | --- |
| PyFilesystem2 (`MemoryFS`) | Process Heap RAM | Ephemeral (cleared on exit) | Instance-isolated | Internal threading lock per instance | Process heap memory and OS file descriptors |
| fsspec (`MemoryFileSystem`) | Process Heap RAM | Ephemeral (cleared on exit) | Global dictionary across process | Global registry lock | Process heap memory and reference leaks |
| PyFilesystem2 (`TempFS`) | OS Temporary Disk / tmpfs | Ephemeral (cleared on close) | System filesystem path | File locking via `fcntl` or `flock` | Disk storage capacity and inode quotas |
| PyFilesystem2 (`ZipFS`) | Compressed archive in RAM or Disk | Persistent if written to disk | Single archive container | Thread-safe reads; single writer | Maximum ZIP file size (ZIP64: 16 EiB) |
| Standard `io.BytesIO` | Process Heap RAM | Ephemeral (cleared on scope exit) | Object reference | Thread-safe stream operations | Process heap memory; no directory tree |

## How to Create an In-Memory File System in Python

Creating an in-memory virtual file system in Python requires choosing between PyFilesystem2's object-oriented API and fsspec's URL-driven protocol architecture. Each approach handles namespace scoping and memory allocation differently.

### Using PyFilesystem2 MemoryFS

PyFilesystem2 represents every filesystem as an instance of `fs.base.FS`. To create an in-memory filesystem, instantiate `fs.memoryfs.MemoryFS` directly or open it using the `mem://` resource opener.

Every `MemoryFS` instance allocates its own independent virtual root in RAM:

```python
from fs.memoryfs import MemoryFS

with MemoryFS() as mem_fs:
    mem_fs.makedirs("workspace/reports")
    mem_fs.makedirs("workspace/raw_data")
    mem_fs.writetext("workspace/reports/summary.md", "Agent Execution Report: All tasks completed.")
    mem_fs.writebytes("workspace/raw_data/output.bin", b"\x00\x01\x02\x03\x04")
    if mem_fs.exists("workspace/reports/summary.md"):
        info = mem_fs.getinfo("workspace/reports/summary.md", namespaces=["basic", "details"])
        print(f"File size: {info.size} bytes")
        print(f"Created: {info.created}")
    content = mem_fs.readtext("workspace/reports/summary.md")
    print(content)
    for path in mem_fs.walk.files():
        print(f"Virtual file: {path}")
```

When the `with` block exits, `mem_fs.close()` runs automatically. It discards the root directory structure and releases the underlying byte buffers for Python garbage collection.

### Using fsspec MemoryFileSystem

The `fsspec` library approaches virtual storage through protocol strings and filesystem registries. Its in-memory implementation is `MemoryFileSystem`, reachable through the `memory://` protocol prefix:

```python
import fsspec

fs = fsspec.filesystem("memory")
with fs.open("memory://agent_output/results.json", "w") as f:
    f.write('{"status": "success", "records_processed": 1420}')

file_list = fs.ls("memory://agent_output")
print(f"Files in memory: {file_list}")

with fs.open("memory://agent_output/results.json", "r") as f:
    payload = f.read()
    print(f"Read payload: {payload}")
```

### Isolating Virtual Namespaces for Multi-Agent Sandboxes

A critical architectural difference between PyFilesystem2 and fsspec lies in state sharing. PyFilesystem2 isolates each `MemoryFS()` object by default. Two separate calls to `MemoryFS()` create two distinct, unlinked virtual filesystems in memory.

In contrast, `fsspec.filesystem("memory")` uses a shared global dictionary by default (`global_store=True`). If Agent A and Agent B run in different threads within the same Python process and both write to `memory://scratch/output.txt`, Agent B will silently overwrite Agent A's file.

To prevent data corruption in multi-agent environments, configure `fsspec` with explicit isolation:

```python
from fsspec.implementations.memory import MemoryFileSystem

isolated_fs_a = MemoryFileSystem(global_store=False, skip_instance_cache=True)
isolated_fs_b = MemoryFileSystem(global_store=False, skip_instance_cache=True)

isolated_fs_a.pipe_text("scratch/task.txt", "Data for Agent A")
isolated_fs_b.pipe_text("scratch/task.txt", "Data for Agent B")

print(isolated_fs_a.cat_text("scratch/task.txt"))
print(isolated_fs_b.cat_text("scratch/task.txt"))
```

Setting `global_store=False` and `skip_instance_cache=True` ensures each agent sandbox operates in its own memory partition.

## Memory Leaks, Concurrency Hazards, and OS Limits in Python VFS

While in-memory virtual filesystems eliminate disk latency, they introduce constraints that do not exist on physical storage devices. Understanding these limits prevents unexpected process crashes during long-running agent workflows.

Python in-memory VFS instances are strictly capped by available process heap memory and default OS file descriptor limits (typically 1024 or 4096 on Linux/macOS). PyFilesystem2 memory filesystem (MemoryFS) stores entire file buffers in RAM, creating memory exhaustion risks during large dataset processing.

### Process Heap Exhaustion and Memory Retention

When an agent writes files into `MemoryFS` or `MemoryFileSystem`, the bytes are stored directly as Python objects on the process heap. If an autonomous agent extracts a large multi-gigabyte archive, generates high-dimensional vector embeddings, or parses dense log dumps into an in-memory VFS, the Python runtime allocates that memory from the operating system.

Python's internal memory manager (`pymalloc`) handles allocations smaller than 512 bytes in dedicated memory arenas, while larger allocations go directly to the system `malloc`. Even after an application deletes a file inside `MemoryFS` using `fs.remove()`, the underlying bytes are only freed within Python's heap. The host operating system does not necessarily reclaim the resident set size (RSS) memory immediately. If consecutive agent runs process large files without terminating the process, memory fragmentation leads to Out-Of-Memory (OOM) termination by the host operating system.

### OS File Descriptor Ceilings and Open Handle Leaks

Even though files in a virtual filesystem exist in RAM, opening a virtual file often creates Python stream wrappers, buffered reader objects, and OS handles if the implementation bridges to temporary disk buffers.

On Linux and macOS, the operating system enforces a strict ceiling on the number of open file descriptors an individual process may hold simultaneously. Check this limit in your shell using `ulimit -n`:

- **Linux Default:** Typically 1024 open descriptors for standard user processes, or 4096 on modern distributions managed by systemd.
- **macOS Default:** Typically 256 or 1024 open descriptors depending on shell profile settings.

When multi-threaded agents open virtual files in tight loops without using context managers, file handles remain active until the garbage collector runs. If active handles cross the `ulimit -n` ceiling, Python raises an OSError with Errno 24 indicating too many open files.

The following diagnostic pattern tracks process memory and file descriptor allocation in Python:

```python
import gc
import os
import resource

def inspect_runtime_limits():
    soft_fd, hard_fd = resource.getrlimit(resource.RLIMIT_NOFILE)
    usage = resource.getrusage(resource.RUSAGE_SELF)
    max_rss_mb = usage.ru_maxrss / (1024 if os.uname().sysname == "Linux" else 1024 * 1024)
    print(f"OS File Descriptor Limits: soft={soft_fd}, hard={hard_fd}")
    print(f"Peak Resident Memory: {max_rss_mb:.2f} MB")

inspect_runtime_limits()
```

### Concurrency Hazards in Multi-Agent Execution

In multi-agent systems where parallel worker threads or coroutines share a virtual filesystem, concurrency bugs emerge quickly:

1. **Compound Check-Then-Act Race Conditions:** PyFilesystem2 provides thread synchronization for individual primitive calls using an internal reentrant lock (`threading.RLock`). However, compound sequences like checking if a file exists before writing are not atomic. Two agent threads checking the same path simultaneously will both evaluate existence checks to `False`, leading to unexpected overwrites.
2. **fsspec Global Namespace Collisions:** Because `fsspec` shares state across threads by default, asynchronous workers sharing a `memory://` protocol without path prefixes will overwrite each other's intermediate state.
3. **Missing Advisory Locks:** Standard Python in-memory filesystems lack cross-agent advisory locking. When one agent reads an analytical dataset while another is updating it, the reader observes half-written records or corrupted JSON structures.

## Bridging In-Memory Agent Sandboxes to Persistent Workspaces

In-memory virtual filesystems are effective for scratchpad computation, fast unit testing, and isolated code evaluation. However, production agent pipelines require a durable persistence layer. When an agent process completes its run, crashes, or scales down in a container cluster, all data stored in `MemoryFS` disappears immediately.

Furthermore, in-memory sandboxes cannot be searched across historical agent runs, audited by engineering leads, or shared with human team members. To build production agent systems, engineering teams bridge ephemeral in-memory sandboxes to persistent cloud workspaces.

### Connecting Ephemeral Sandboxes to Fastio Workspaces

Fastio provides shared org-owned workspaces that serve as the persistent substrate for agentic teams. Rather than managing physical disk volumes or complex object store lifecycles, agents commit completed deliverables directly from Python memory into persistent workspaces.

Fastio incorporates key capabilities designed for human-agent collaboration:

- **Per-File Version History:** Every write to a workspace creates a distinct version. If an agent updates an executive brief or refines a dataset, prior versions remain intact and restorable. Concurrent agent writes both land safely in version history.
- **Role-Based Permissions:** Permissions are configured per organization, workspace, and share through member roles (never per folder or per file). This design ensures security boundaries remain predictable across multi-agent fleets.
- **Detailed Activity Log:** Fastio maintains a detailed activity log that records changes across the organization, including uploads, edits, moves, deletions, membership updates, and share creation. It does not record read-only events like views or downloads.
- **Intelligence Mode and Hybrid Search:** Enabling Intelligence Mode on a workspace automatically indexes files for hybrid search (combining full-text search, semantic search, and metadata filtering). Agents search indexed documents across the workspace without loading entire corpora into context prompts.
- **Metadata Views:** Teams turn unstructured documents into live, queryable databases using [Metadata Views](/product/document-data-extraction/). AI automatically designs a typed schema (Text, Integer, Decimal, Boolean, URL, JSON, Date & Time), extracts structured values, and populates a filterable spreadsheet view that agents query programmatically.
- **Cloud Sync:** Cloud Sync connects Google Drive, Dropbox, Box and OneDrive folders to a workspace, one-way or two-way, on a schedule or on demand. Agents read synced assets and file deliverables back into the same shared workspace.
- **E-Signature:** Workspaces support sending documents for signature with sequential or parallel signers, optional one-time passcodes, a tamper-evident audit certificate, and automatic filing of the executed copy back into workspace storage.

### Exporting from In-Memory VFS to Fastio via MCP

Autonomous agents interact with Fastio using the Model Context Protocol (MCP) over Streamable HTTP. Depending on your agent client, connect to the dedicated endpoint:

- Coding agents (Claude Code, Cursor, Gemini CLI, Devin, Cline): Connect to `https://mcp.fast.io/mcp/code`.
- ChatGPT and Codex custom servers: Connect to `https://mcp.fast.io/mcp/operations`.
- General agents and chat apps (Claude Desktop, Claude apps, custom agent runtimes): Connect to `https://mcp.fast.io/mcp/tools`.

The following Python example shows how an agent finishes processing files in an in-memory `MemoryFS` sandbox and streams the resulting deliverable directly to a Fastio workspace using an HTTP client:

```python
import os
from fs.memoryfs import MemoryFS
import httpx

with MemoryFS() as sandbox:
    sandbox.makedirs("analysis")
    sandbox.writetext("analysis/market_summary.md", "Autonomous Agent Analysis: Key growth drivers identified.")
    file_content = sandbox.readbytes("analysis/market_summary.md")

FASTIO_API_KEY = os.environ["FASTIO_API_KEY"]
WORKSPACE_ID = "ws_agent_production_01"

mcp_payload = {
    "jsonrpc": "2.0",
    "id": 1,
    "method": "tools/call",
    "params": {
        "name": "upload_manage",
        "arguments": {
            "action": "upload",
            "workspace_id": WORKSPACE_ID,
            "name": "market_summary.md",
            "content": file_content.decode("utf-8")
        }
    }
}

headers = {
    "Authorization": f"Bearer {FASTIO_API_KEY}",
    "Content-Type": "application/json",
    "Accept": "application/json, text/event-stream"
}

response = httpx.post(
    "https://mcp.fast.io/mcp/tools",
    json=mcp_payload,
    headers=headers,
    timeout=30.0
)

print(f"MCP Response Status: {response.status_code}")
```

### Managing Concurrent Writes with Advisory File Locks

When multiple Python agents collaborate in the same workspace, they coordinate writes using advisory file locks. Agents acquire, heartbeat, and release locks through MCP actions:

- `lock-acquire` and `lock-release` on the `storage_manage` tool.
- `lock-status` on the `storage` tool.

A file lock represents an advisory lease. It signals to other agents and team members that a specific agent is updating the file. The lock expires automatically unless heartbeated by the holder. Importantly, advisory locks do not block read access or prevent overwrites: if two agents write simultaneously, both versions are captured safely in Fastio's per-file version history.

Monthly plans start with a 14-day free trial that requires a credit card. Plan tiers include Starter at `$9.99`/mo (3 seats, `250 GB` storage, 5 workspaces, `100,000` credits a month), Business at `$49.99`/mo (10 seats, `5 TB` storage, 50 workspaces, `600,000` credits a month), and Enterprise at `$199.99`/mo (30 seats included, `25 TB` storage, 200 workspaces, `3,000,000` credits a month). Maximum upload sizes are `25 GB` on Starter, `50 GB` on Business, and `100 GB` on Enterprise.

## Architectural Patterns for Python AI Agent File Sandboxes

Engineering teams deploying file-aware Python agents must align storage architectures with operational requirements. Different stages of an agentic workflow require different balances of speed, isolation, and persistence.

### Three Core Storage Patterns

1. **Pure In-Memory Scratchpad (Evaluation & Code Sandboxes):**
   - **Mechanism:** Agent instantiates `MemoryFS` or `fsspec.filesystem("memory", global_store=False)` for each isolated execution cycle.
   - **Best For:** Evaluating untrusted Python code, running syntax linters, parsing abstract syntax trees (ASTs), or transforming temporary JSON files.
   - **Tradeoff:** Zero disk I/O latency, but data is ephemeral and heap-bounded.
2. **Tiered Spilling with MemoryFS and TempFS (Local Big Data Processing):**
   - **Mechanism:** The agent monitors file buffer sizes. Small files stay in `MemoryFS`. When a payload crosses a defined memory threshold, the agent routes the stream to `fs.tempfs.TempFS`.
   - **Best For:** Extracting large multi-gigabyte TAR or ZIP archives, transcode pipelines, and batch CSV processing.
   - **Tradeoff:** Preserves heap memory, but incurs disk I/O and requires cleanup handling.
3. **Hybrid VFS Sandbox with MCP Cloud Persistence (Production Agent Workflows):**
   - **Mechanism:** The agent performs fast iterative operations inside an isolated Python VFS. Upon completing validation, it commits deliverables to Fastio workspaces via MCP.
   - **Best For:** Autonomous research agents, automated legal matter preparation, financial modeling, and collaborative human-agent workflows.
   - **Tradeoff:** Combines in-memory execution speed with durable, indexed, human-accessible persistence.

The following matrix evaluates each architectural pattern across operational dimensions:

| Architecture Pattern | Execution Speed | Memory Overhead | Persistence & Recovery | Multi-Agent Coordination | Best Fit Use Case |
| --- | --- | --- | --- | --- | --- |
| Pure In-Memory Scratchpad | Instantaneous (RAM speeds) | High (consumes heap memory) | None (lost on crash or exit) | None (process-isolated) | Code linting, unit testing, AST evaluation |
| Tiered Memory-to-Disk Spilling | Fast (RAM with disk spill) | Moderate (bounded by threshold) | Partial (managed by OS tmpfs) | File locks via OS primitives | High-capacity archive extraction, batch CSVs |
| Hybrid VFS with MCP Workspaces | Instantaneous scratch, fast export | Low (heap freed after upload) | Complete (durable version history) | Advisory file locks and activity feed | Autonomous agents, team deliverables, RAG pipelines |

### Engineering Checklist for Python Agent Storage

Before deploying autonomous file-handling agents to production, verify these operational checkpoints:

- [ ] **Enforce Namespace Isolation:** Confirm `fsspec` memory filesystems use `global_store=False` and `skip_instance_cache=True` to prevent cross-agent data leaks.
- [ ] **Use Context Managers:** Wrap all `MemoryFS` and virtual file streams in `with` blocks to ensure prompt closure and avoid file descriptor leaks.
- [ ] **Track Memory Thresholds:** Monitor process resident set size (RSS) during batch file processing to prevent Linux OOM terminations.
- [ ] **Decouple Scratch from Deliverables:** Use in-memory VFS exclusively for scratch computation; stream finished artifacts to persistent workspaces via MCP.
- [ ] **Coordinate Concurrent Writes:** Acquire advisory file locks on shared files and rely on workspace version history to handle concurrent multi-agent updates.

## Frequently asked questions

### How do I create a virtual file system in Python?

You can create a virtual file system in Python using libraries like PyFilesystem2 or fsspec. With PyFilesystem2, instantiate `fs.memoryfs.MemoryFS()` to create an isolated, in-memory directory tree where you can create directories, write files, and inspect metadata without touching physical disks. With fsspec, call `fsspec.filesystem('memory')` to read and write files using URL-style paths such as `memory://filename.txt`.

### What are the limitations of PyFilesystem MemoryFS?

PyFilesystem2 MemoryFS stores all directory trees and file contents directly in process heap memory. Its primary limitations include memory exhaustion when handling multi-gigabyte datasets, lack of persistence across process restarts, and susceptibility to file descriptor exhaustion if virtual streams are left unclosed. Additionally, compound operations in MemoryFS are not atomic across multi-threaded applications without external synchronization.

### How can Python AI agents safely handle large files without filling RAM?

Python AI agents can safely process large files by adopting a tiered storage strategy. Keep small files and metadata in memory, but stream high-capacity payloads through `fs.tempfs.TempFS` or direct chunked transfers. For long-term storage and retrieval, agents should upload files to an indexed cloud workspace such as Fastio over MCP, retrieving only the required passages through semantic search rather than loading entire corpora into memory.

### What is the difference between fsspec MemoryFileSystem and PyFilesystem2 MemoryFS?

The primary difference lies in namespace scoping and API philosophy. PyFilesystem2 MemoryFS isolates each instance by default, providing an object-oriented filesystem interface. fsspec MemoryFileSystem uses a URL-driven protocol interface (`memory://`) and shares a global dictionary across the entire Python process by default unless explicitly initialized with `global_store=False`.

### Can multiple agent threads share the same in-memory file system safely?

Individual read and write calls in PyFilesystem2 MemoryFS are thread-safe due to internal reentrant locks. However, compound check-then-act sequences (such as checking if a file exists before writing) remain prone to race conditions. In fsspec, sharing the global memory store without distinct path prefixes can cause parallel threads to overwrite each other's data.

### How do you persist files from a Python in-memory VFS to cloud storage?

To persist files from an in-memory VFS, extract the file buffer using `.readbytes()` or a stream handle and transmit it to an external storage service or cloud workspace. Autonomous agents can use Fastio's remote Model Context Protocol (MCP) server over Streamable HTTP to upload artifacts directly into shared, versioned workspaces accessible to human team members.

## Sources

- [Claude Help Center: Upload files to Claude](https://support.claude.com/en/articles/8241126-upload-files-to-claude): Claude chat sessions accept up to 20 files at up to 500 MB per file, while Claude Projects accept files up to 30 MB each with no fixed file-count cap, bounded only by the model's context window.

## About Fast.io

Fast.io provides shared workspaces where people and AI agents work on the same files, with built-in semantic search and citation-backed chat over what they hold. Agents reach it through a remote MCP server, a REST API at https://api.fast.io/current/, and a command line client published on npm as @vividengine/fastio-cli. MCP setup is at https://mcp.fast.io/docs: Claude and most MCP clients connect to https://mcp.fast.io/mcp/tools, ChatGPT to https://mcp.fast.io/mcp/operations, and coding agents to https://mcp.fast.io/mcp/code.