How to Deploy Hermes Agent on Modal Serverless GPUs

Nous Research Hermes Agent supports seven terminal backends: local, Docker, SSH, Daytona, Singularity, Vercel Sandbox, and Modal. Each backend determines where the agent's shell commands execute. Modal stands out for GPU-intensive workloads because it provides serverless compute that hibernates between sessions and bills per second of actual usage.

Most agent deployment guides default to Docker containers on a VPS or a manually provisioned cloud instance. That works, but it means paying for idle time whenever the agent isn't processing requests. For intermittent workloads like AI agents that handle bursts of activity followed by quiet periods, serverless GPU compute eliminates that waste.

Modal's infrastructure is built in Rust, which enables container cold starts measured in seconds rather than minutes. Combined with GPU fallback chains (where Modal tries your preferred GPU, then falls back to alternatives if unavailable), you get reliable availability without over-provisioning.

The practical difference: a Hermes Agent running on a $5/hour H100 that's active 2 hours per day costs roughly $10/day on Modal. The same workload on a reserved instance would cost $120/day whether the agent runs or not.

Helpful references: Fastio Workspaces, Fastio Collaboration, and Fastio AI.

Before configuring Modal as your Hermes Agent backend, you need three things in place: a working Hermes Agent installation, a Modal account with API credentials, and Python 3.11 or later.

Install Modal CLI

pip install modal
modal setup

The modal setup command opens your browser for OAuth authentication and stores credentials locally. Once authenticated, verify the connection:

modal token list

Configure Hermes Agent

Set Modal as the terminal backend in ~/.hermes/config.yaml:

terminal:
  backend: modal
  container_cpu: 1
  container_memory: 5120
  container_disk: 51200
  container_persistent: true
  modal_image: "nikolaik/python-nodejs:python3.11-nodejs20"

The container_persistent: true flag tells Hermes to snapshot the sandbox filesystem on cleanup and restore it during the next session. This preserves installed packages, downloaded models, and generated files across sessions. Snapshots are tracked in ~/.hermes/modal_snapshots.json and preserve filesystem state (not live processes or background jobs).

Authentication Options

Modal accepts credentials via environment variables or a config file:

Environment variables:

export MODAL_TOKEN_ID=your_token_id
export MODAL_TOKEN_SECRET=your_token_secret

Config file (~/.modal.toml): Modal's standard TOML credentials format, created automatically by modal setup.

Verify the full configuration with hermes doctor, which checks all backend connections and reports any missing dependencies.

Modal offers eight GPU tiers, each suited to different workload profiles. Choosing the right GPU depends on your model size, inference latency requirements, and budget.

Available GPUs

Specifying GPUs in Modal Functions

Modal uses Python decorators for GPU assignment:

@app.function(gpu="A100")           # Single A100-40GB
@app.function(gpu="A100-80GB")      # Specific memory variant
@app.function(gpu="H100:4")         # Multi-GPU (up to 8)
@app.function(gpu="any")            # Whatever's available fast

Fallback Chains

GPU availability fluctuates. Fallback chains let you specify preferences in priority order:

@app.function(gpu=["H100", "A100-80GB", "L40S"])

Modal tries H100 first. If none are available, it falls back to A100-80GB, then L40S. This prevents deployment failures during high-demand periods while letting you target your preferred hardware when it's available.

For Hermes Agent specifically, a 7B parameter model fits comfortably on an L4 or A10G. A 70B model needs A100-80GB or H100. If you're running inference only (not training), the L40S offers the best cost-to-performance ratio for models in the 13B-34B range.

Persist Hermes Agent outputs across Modal sessions

generous storage workspace with MCP-native access. Your agent writes files on Modal GPUs, Fastio stores them permanently for search, sharing, and human review. No credit card required.

Start 14-Day Trial

The biggest cost driver for GPU workloads is re-downloading large model weights on every cold start. Modal Volumes solve this by providing persistent remote filesystems that survive container restarts.

Creating a Persistent Volume

import modal

volume = modal.Volume.from_name("model-cache", create_if_missing=True)

@app.function(
    gpu="A100",
    volumes={"/models": volume}
)
def run_inference(prompt):
    ### Models persist in /models across invocations
    pass

After writing to a volume, call volume.commit() to persist changes. Without the explicit commit, data written during that container's lifetime will be lost.

Hermes Agent Filesystem Persistence

With container_persistent: true in your Hermes config, the agent snapshots the entire sandbox filesystem between sessions. This means:

• Installed pip packages persist (no re-installation on wake)
• Downloaded model weights stay cached
• Generated artifacts survive across sessions
• Configuration changes you make inside the container carry forward

The tradeoff is snapshot size. A container with a 30GB model cached will take longer to restore than a clean container. For large models, prefer Modal Volumes over full-container persistence since volumes mount instantly regardless of size.

Cold Start Optimization Even with persistent storage, GPU containers take a few seconds to initialize. For latency-sensitive deployments:

@app.function(
    gpu="A100",
    container_idle_timeout=300,
    allow_concurrent_inputs=10
)

The container_idle_timeout=300 keeps your container warm for 5 minutes after the last request, avoiding cold starts during bursty workloads. allow_concurrent_inputs=10 lets a single container handle multiple simultaneous requests, reducing the need to spin up new instances.

For model loading specifically, use Modal's @modal.enter() lifecycle hook to preload weights during container initialization rather than on the first request.

Modal provides three deployment commands, each suited to a different stage of your workflow.

Development: modal run

modal run script.py

Executes the script once and terminates. Use this for testing your GPU configuration, running one-off inference jobs, or validating that model weights load correctly. Hermes Agent uses this mode for individual tasks that don't need a persistent endpoint.

Iteration: modal serve

modal serve script.py

Starts a development server with live reloading. Edit your script locally and Modal redeploys automatically. This is ideal for iterating on prompt templates, testing different model configurations, or debugging inference pipelines. The endpoint stays active until you stop it.

Production: modal deploy

modal deploy script.py

Creates a persistent cloud deployment with a stable URL. The deployment scales automatically based on traffic, handles zero-downtime updates, and survives your local machine shutting down. For Hermes Agent gateway deployments (where the agent serves requests from Telegram, Discord, Slack, or other messaging platforms), this is the mode you want.

Web Endpoints

Modal supports exposing functions as HTTP endpoints:

@modal.fastapi_endpoint()
def handle_request(prompt: str):
    return {"response": agent.process(prompt)}

This pairs well with Hermes Agent's gateway mode, where external services send messages to your agent via HTTP and receive responses asynchronously.

Comparing Modal to Other Backends

Docker gives you full control but requires managing infrastructure. SSH connects to existing servers but means paying for idle time. Daytona offers serverless persistence similar to Modal but without GPU access. Modal is the clear choice when your agent needs GPU compute for model inference, batch processing, or training, and you want that compute to cost nothing when idle.

Running Hermes Agent on Modal in production requires thinking about reliability, cost management, and file persistence beyond what a single session needs.

Recommended Configuration

@app.function(
    gpu=["H100", "A100-80GB", "L40S"],
    memory=32768,
    cpu=4,
    timeout=3600,
    container_idle_timeout=120,
    retries=3,
    concurrency_limit=10
)

This configuration targets H100 with fallbacks, allocates 32GB RAM alongside the GPU, sets a 1-hour timeout for long-running agent tasks, retries failed executions up to 3 times, and limits concurrency to control costs.

Dynamic Batching for Throughput

When your agent processes multiple requests, Modal's batching decorator groups inputs automatically:

@modal.batched(max_batch_size=32, wait_ms=100)
def batch_process(prompts: list[str]):
    return model.generate(prompts)

This waits up to 100ms to collect requests, then processes them in a single GPU batch. For inference workloads, batching can improve throughput by 3-5x compared to sequential processing.

Parallel Processing with .map()

For bulk operations (processing a corpus, generating embeddings for many documents), Modal's .map() distributes work across containers:

results = list(process_document.map(documents))

Each document gets its own container, scaling horizontally without manual orchestration.

Persistent File Storage with Fastio

Modal's persistence works well for model weights and container state, but agent-generated outputs need a more durable home. Files that Hermes Agent creates during sessions (reports, analysis results, generated code, processed datasets) benefit from a workspace that persists independently of any compute backend.

Fastio provides generous storage with MCP-native access, meaning your agent can read from and write to shared workspaces through the same tool interface it uses for everything else. The workflow looks like this: Hermes runs inference on Modal GPUs, generates output files, then writes them to a Fastio workspace where humans can review, comment, and share them.

This separation matters because Modal containers are ephemeral by design. Even with persistent volumes, you don't want your only copy of important outputs living inside a serverless compute platform. Fastio workspaces give those files a permanent address that's accessible from any agent session, any backend, and any team member's browser.

Intelligence Mode auto-indexes uploaded files for semantic search, so outputs from past agent sessions become queryable context for future ones. No separate vector database required.

Cost Management

Modal's free tier includes $30/month in credits, enough for roughly 7-8 hours of H100 time or 27 hours of A10G time. For production workloads beyond that:

• Use container_idle_timeout aggressively (60-120 seconds for most agents)
• Prefer L40S over A100 for inference-only workloads (better cost per token)
• Use fallback chains to land on whatever GPU is cheapest and available
• Monitor usage in Modal's dashboard and set spending alerts