Hermes 3: Complete Guide to Nous Research's Open-Source LLM

Hermes 3 is an open-source language model family created by Nous Research, released in August 2024. The models are full-parameter fine-tunes of Meta's Llama 3.1 base weights, available in three sizes: 8B, 70B, and 405B parameters. Nous Research trained them on a dataset of approximately 390 million tokens, primarily synthetically generated, to produce models that reliably follow instructions, generate structured output, and call external tools.

The 405B variant was the first publicly available full-parameter fine-tune of Llama 3.1 405B. That matters because most open-source fine-tunes at that scale use parameter-efficient methods like LoRA, which limit how far the model's behavior can diverge from the base. Nous Research trained the full 405B weight set, which gave them more control over the model's instruction-following behavior and tool-use reliability.

At a practical level, Hermes 3 sits between a raw base model and a commercial API. You get Llama 3.1's language capabilities (128K context window, multilingual support, strong reasoning) plus a fine-tune that makes the model actually useful in pipelines: it follows system prompts precisely, generates valid JSON on demand, and calls functions using a structured XML format. All weights are open and downloadable from HuggingFace under the Llama 3 license.

The technical report (arXiv:2408.11857) documents how Hermes 3 achieves state-of-the-art performance among open-weight models on benchmarks including ARC, BoolQ, HellaSwag, IFEval, and Winogrande, matching or exceeding Meta's own Llama 3.1 Instruct fine-tunes on several of those evaluations.

Hermes 3 comes in three sizes. Each shares the same ChatML prompt format, the same function-calling protocol, and the same training methodology. The differences are capacity, hardware requirements, and inference cost.

Hermes 3 8B

The 8B model runs on a single consumer GPU with 16GB of VRAM when quantized to 4-bit precision, or on a machine with 24GB VRAM at full BF16. This is the entry point for local development: fast inference, low resource requirements, and good enough quality for prototyping agent workflows. At Q4_K_M quantization, the GGUF file is about 4.9 GB. It handles function calling and structured output reliably, though it lacks the reasoning depth of the larger variants on complex multi-step tasks.

Best for: Local development, testing agent pipelines, single-GPU deployments, and applications where latency matters more than peak quality.

Hermes 3 70B

The 70B model needs roughly 40 GB of VRAM at 4-bit quantization, which puts it in reach of dual-GPU consumer setups or a single A100/H100. It offers a significant jump in reasoning quality, multi-turn coherence, and the ability to handle nuanced instructions. For teams running agents in production, 70B is often the sweet spot: capable enough for complex tool chaining, affordable enough to self-host.

Best for: Production agent deployments, complex reasoning tasks, teams with access to server-grade GPUs or cloud instances.

Hermes 3 405B

The flagship. Running the full 405B model requires multiple high-end GPUs (typically 4-8 A100s or H100s). Nous Research used Neural Magic's FP8 quantization to cut memory requirements roughly in half, enabling single-node deployment on an 8-GPU machine. The 405B model delivers the strongest reasoning, the most reliable function calling, and the best performance on academic benchmarks. Lambda AI hosted the initial public deployment on their 1-Click Cluster infrastructure.

Best for: Maximum quality workloads, research, and organizations with GPU cluster access. Also available through API providers like DeepInfra and OpenRouter for teams that prefer not to self-host.

Size Selection Cheat Sheet

Pick 8B when you want fast iteration on a laptop or single GPU. Pick 70B when you need production-grade quality and can allocate server hardware. Pick 405B when quality is the priority and cost is secondary, or when you can access it through an API provider.

What makes Hermes 3 different from running raw Llama 3.1 Instruct? Two things: the training data and the alignment philosophy.

Nous Research trained Hermes 3 in two phases. The first phase was supervised fine-tuning (SFT) on approximately 390 million tokens of primarily synthetic data. The second phase was Direct Preference Optimization (DPO), a technique that trains the model to prefer higher-quality responses without the complexity of a separate reward model.

The training data breaks down by domain:

• General instructions: 60.6%
• Domain expert data: 12.8%
• Mathematics: 6.7%
• Roleplaying and creative writing: 6.1%
• Coding and software development: 4.5%
• Tool use, agentic reasoning, and RAG: 4.3%

That 4.3% dedicated to tool use is why Hermes 3 handles function calling so much better than the base Llama 3.1 model. The training data explicitly includes examples of structured tool invocation, goal-oriented action planning, and retrieval-augmented workflows.

Neutral Alignment

Hermes 3 takes a different approach to safety than most commercial models. Nous Research calls it "neutral alignment." The model follows the system prompt exactly and adapts to whatever worldview the prompt establishes, rather than applying built-in content restrictions. As the team puts it, guardrails belong at the system level, not baked into the model weights.

In practice, this means Hermes 3 is highly steerable. You define the boundaries in your system prompt, and the model respects them. If your system prompt says "respond only in JSON," it will. If your system prompt establishes a specific persona, it will maintain that persona consistently across long conversations. This steerability is one of the main reasons developers prefer Hermes 3 over the standard Llama 3.1 Instruct fine-tune for agent applications, where you need the model to follow a precise behavioral specification without second-guessing the instructions.

Persist your Hermes 3 agent output across sessions

Free 50 GB workspace with auto-indexing, semantic search, and MCP access. Connect your Hermes 3 agents to persistent storage where files are immediately searchable and shareable. No credit card, no expiration.

Start 14-Day Trial

Hermes 3 runs on every major local inference stack. The model weights are available in full BF16 on HuggingFace and in GGUF quantizations for use with llama.cpp-based tools. Here are the main deployment options.

Ollama Ollama is the simplest path for local deployment. Pull a community-maintained Hermes 3 model and start serving:

ollama pull finalend/hermes-3-llama-3.1:8b-q4_K_M
ollama serve

One important detail: Ollama defaults to a 4,096-token context window, which is too small for agent workflows. Set it explicitly:

OLLAMA_CONTEXT_LENGTH=32768 ollama serve

The OpenAI-compatible API does not accept context length from the client, so the model will silently truncate if you skip this step.

vLLM

For production throughput on GPU infrastructure, vLLM provides an OpenAI-compatible API server with efficient batching and paged attention:

pip install vllm
vllm serve "NousResearch/Hermes-3-Llama-3.1-8B" \
  --port 8000 \
  --enable-auto-tool-choice \
  --tool-call-parser hermes

The --tool-call-parser hermes flag tells vLLM to parse the XML tool_call format natively, so you get structured tool calls in the OpenAI-compatible response format without post-processing.

For the 70B model with tensor parallelism across two GPUs:

vllm serve "NousResearch/Hermes-3-Llama-3.1-70B" \
  --port 8000 \
  --max-model-len 65536 \
  --tensor-parallel-size 2 \
  --enable-auto-tool-choice \
  --tool-call-parser hermes

llama.cpp and LM Studio Download a GGUF quantization from the NousResearch GGUF repository on HuggingFace. The 8B model has 47 quantization variants. For most users, Q4_K_M (about 4.9 GB) offers a good balance between quality and memory usage. Q8_0 (about 8.5 GB) preserves near-original quality if you have the VRAM.

Run with llama-server:

llama-server -m Hermes-3-Llama-3.1-8B-Q4_K_M.gguf \
  --ctx-size 32768 \
  --port 8080

LM Studio provides a GUI for the same workflow: download the model, load it, and start the local server. Both expose an OpenAI-compatible API.

HuggingFace Transformers

For direct Python integration, load the model with the Transformers library:

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained(
    "NousResearch/Hermes-3-Llama-3.1-8B"
)
model = AutoModelForCausalLM.from_pretrained(
    "NousResearch/Hermes-3-Llama-3.1-8B",
    torch_dtype="auto",
    device_map="auto"
)

messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "What is the capital of France?"}
]
inputs = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt"
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Add load_in_4bit=True and use_flash_attention_2=True for memory-efficient inference on consumer hardware.

Quantization Quick Reference (8B Model)

• Q2_K (3.2 GB): Lowest quality. Use for memory-constrained testing only.
• Q4_K_M (4.9 GB): Good quality. Recommended for general local development.
• Q5_K_M (5.7 GB): Better quality with minimal size increase. Good for quality-sensitive applications.
• Q8_0 (8.5 GB): Near-original quality. Use when you have VRAM to spare.
• BF16 (16 GB): Original precision. Requires a GPU with 24+ GB VRAM.

For the 70B model, multiply these sizes by roughly 8.75x. A Q4_K_M quantization of the 70B weighs about 40 GB.

Hermes 3 uses a structured format for function calling built on the ChatML prompt template. You define available tools in JSON schema format inside <tools> XML tags in the system prompt. When the model decides a function call is needed, it emits a <tool_call> block:

<tool_call>
{"name": "get_weather", "arguments": {"city": "Tokyo"}}
</tool_call>

Your application parses the XML, executes the real function, and passes the result back as a <tool_response> message. The model then incorporates the result into its natural language reply.

This format works reliably across all three model sizes. The 8B model handles single tool calls well. The 70B and 405B models are better at multi-step tool chaining, where the model plans a sequence of calls to accomplish a complex goal.

Hermes Agent and Building Agents with Hermes 3

The function-calling fine-tune makes Hermes 3 a practical foundation for autonomous agents. Nous Research built Hermes Agent, an open-source agent framework that demonstrates this: persistent memory, skill creation, scheduled automations, and messaging gateway integrations for Telegram, Discord, Slack, WhatsApp, Signal, and email.

Hermes Agent uses Hermes models by default but supports any OpenAI-compatible endpoint. The framework expects at least 64,000 tokens of context for multi-step tool-calling workflows.

For developers building custom agent systems, the key advantage of Hermes 3 over the base Llama 3.1 Instruct is reliability. The 4.3% of training data dedicated to tool use and agentic reasoning means the model produces valid tool calls consistently, follows multi-step plans without losing track of the goal, and handles the back-and-forth of tool invocation and result processing cleanly.

JSON Mode and Structured Output

Beyond tool calling, Hermes 3 generates schema-compliant JSON on demand. Add a JSON schema to the system prompt:

You are a helpful assistant that answers in JSON.
Here's the json schema you must adhere to:
<schema>
{"type": "object", "properties": {"name": {"type": "string"}, "age": {"type": "integer"}}}
</schema>

The model constrains its output to match the schema. This is useful for data extraction pipelines, structured API responses, and any workflow where you need predictable output format without post-processing regex.

Running Hermes 3 locally solves the inference problem, but agents need more than a language model. They need somewhere to store files, share results, and hand off work to humans. Local filesystems work for prototyping, but they break down when agents run across sessions, on different machines, or need to deliver output to non-technical users.

The common approaches each have tradeoffs. S3 or similar object storage gives you durability but no search, no permissions granularity below the bucket level, and no built-in way for a client to browse results. Google Drive and Dropbox work for human sharing but lack APIs designed for agent workflows. Building a custom file management layer takes time away from the actual agent logic.

Fastio is built for this use case. It provides shared workspaces where agents write files and humans read them, with a free tier that includes 50 GB of storage, included credits, and 5 workspaces, with no credit card required. The MCP server exposes workspace, storage, AI, and workflow operations through Streamable HTTP, so your Hermes 3 agent can upload results, create shares, and trigger notifications through standard tool calls.

A typical setup looks like this: Hermes 3 runs locally (or on a server) handling inference and tool calling. When the agent produces output, like a generated report, extracted data, or processed file, it writes that output to a Fastio workspace through the API or MCP server. The workspace auto-indexes the file for semantic search through Intelligence Mode, making it immediately queryable. A human teammate opens the same workspace in their browser, reviews the output, and downloads or shares it.

The ownership transfer feature handles a common agent deployment pattern. An agent creates an organization, builds workspaces with structured content, sets up branded shares, then transfers ownership to a human. The human gets full control. The agent retains admin access for ongoing updates. This works well for agencies or consultancies where an AI pipeline builds deliverables that a client owns.

For multi-agent setups where several Hermes instances work on related tasks, Fastio's file locks prevent conflicts when agents write to shared workspaces concurrently. Webhooks notify downstream agents or human reviewers when files change, enabling reactive workflows without polling.

What makes this approach practical is that the workspace is intelligent by default. Upload a document and it is automatically indexed for semantic search. Ask questions about workspace contents through the AI chat interface and get answers with citations pointing to specific files. Your Hermes 3 agent's output becomes immediately searchable and shareable the moment it lands in the workspace.