Hermes 3 Function Calling: Tool Use Setup and Implementation

Hermes 3 is an open-source LLM family from Nous Research, available in 8B, 70B, and 405B parameter sizes, built on the Llama 3.1 base. It was trained specifically on function calling tasks using the ChatML prompt format, which means it understands a structured protocol for receiving tool definitions, generating calls, and processing results.

The core mechanic: you define available functions in JSON schema format inside <tools> XML tags in the system prompt. When the model decides it needs to call a function, it emits a <tool_call> XML block containing a JSON object with the function name and arguments. Your application parses that block, executes the real function, and feeds the result back as a <tool_response> message. The model then incorporates that result into its next response.

Here is the basic format the model produces:

<tool_call>
{"name": "get_weather", "arguments": {"city": "San Francisco"}}
</tool_call>

And the response your application sends back:

<tool_response>
{"name": "get_weather", "content": {"temp": 62, "condition": "foggy"}}
</tool_response>

This XML-wrapping approach differs from the OpenAI and Anthropic function calling formats. Where OpenAI uses a dedicated tool_calls field in the API response JSON, Hermes 3 embeds the call directly in the generated text. That makes it backend-agnostic: the same format works whether you run the model through Transformers, Ollama, vLLM, or any other inference engine that supports ChatML. The full reference implementation is available in the Hermes Function Calling repository.

The system prompt is where you register available tools. Hermes 3 expects function definitions in OpenAI-compatible JSON schema format, wrapped in <tools> tags. The prompt also includes a Pydantic schema that tells the model exactly how to structure each call.

Here is the full system prompt template:

<|im_start|>system
You are a function calling AI model. You are provided with
function signatures within <tools></tools> XML tags. You may
call one or more functions to assist with the user query.
Don't make assumptions about what values to plug into functions.

Here are the available tools:
<tools>
[{"type": "function", "function": {"name": "get_stock_price",
"description": "Get the current stock price",
"parameters": {"type": "object", "properties": {"symbol":
{"type": "string", "description": "The stock ticker symbol"}},
"required": ["symbol"]}}}]
</tools>

Use the following pydantic model json schema for each tool call
you will make:
{"properties": {"arguments": {"title": "Arguments",
"type": "object"}, "name": {"title": "Name", "type": "string"}},
"required": ["arguments", "name"], "title": "FunctionCall",
"type": "object"}

For each function call return a json object with function name
and arguments within <tool_call></tool_call> XML tags.
<|im_end|>

A few things to note about this structure. The tool definitions use the exact same JSON schema format as the OpenAI API, so if you already have tool definitions for GPT-4 or Claude, you can reuse them directly. The Pydantic FunctionCall schema enforces that every call includes both a name string and an arguments object. And the entire prompt uses ChatML tokens (<|im_start|>, <|im_end|>) for turn boundaries.

When defining tools programmatically in Python, the Nous Research reference implementation uses a @tool decorator pattern:

from langchain_core.utils.function_calling import convert_to_openai_tool

@tool
def get_stock_price(symbol: str) -> dict:
    """Get the current price for a stock ticker.

Args:
        symbol: The stock ticker symbol (e.g., AAPL, TSLA)

Returns:
        dict with price and currency fields
    """
    ### implementation here
    return {"price": 150.25, "currency": "USD"}

tools = [convert_to_openai_tool(get_stock_price)]

Type hints and docstrings matter here. The convert_to_openai_tool function reads them to generate the JSON schema that goes into the system prompt.

Persist your Hermes 3 agent outputs across sessions

Free 50GB workspace with MCP-native access. Your function calling agent writes files, a human reviews them through the web UI. No credit card, no trial expiration.

Start 14-Day Trial

Hermes 3 function calling works across multiple inference backends. Each has different setup requirements and trade-offs.

Ollama

Ollama is the fast way to get Hermes 3 running locally. Pull the model and start serving:

ollama pull hermes3
ollama serve

Then call it through the OpenAI-compatible API:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")

response = client.chat.completions.create(
    model="hermes3",
    messages=[
        {"role": "system", "content": system_prompt_with_tools},
        {"role": "user", "content": "What is AAPL trading at?"}
    ],
    temperature=0.0
)

One critical Ollama gotcha: it defaults to low context lengths (often 4,096 tokens with limited VRAM). For function calling, you need room for tool definitions, conversation history, and results. Set at least 16k context via an environment variable or a Modelfile. The OpenAI-compatible API does not accept context length from the client, so you must configure it server-side.

The 8B model needs roughly 8GB RAM and 6GB VRAM to run the Q4_K_M quantization at full GPU speed.

vLLM

For production deployments, vLLM provides higher throughput with automatic batching. The key flags for function calling are --enable-auto-tool-choice and --tool-call-parser hermes:

vllm serve NousResearch/Hermes-3-Llama-3.1-8B \
  --port 8000 \
  --enable-auto-tool-choice \
  --tool-call-parser hermes

The hermes parser tells vLLM to look for <tool_call> XML tags in the generated text and extract them into structured tool call objects in the API response. This means downstream code can use the standard OpenAI tool_calls response field instead of parsing XML manually.

Transformers (Direct)

For full control over the generation pipeline, load the model directly with HuggingFace Transformers:

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

tokenizer = AutoTokenizer.from_pretrained(
    "NousResearch/Hermes-3-Llama-3.1-8B",
    trust_remote_code=True
)
model = AutoModelForCausalLM.from_pretrained(
    "NousResearch/Hermes-3-Llama-3.1-8B",
    torch_dtype=torch.float16,
    device_map="auto"
)

messages = [
    {"role": "system", "content": system_prompt_with_tools},
    {"role": "user", "content": "Get the price for TSLA"}
]

input_ids = tokenizer.apply_chat_template(
    messages, return_tensors="pt", add_generation_prompt=True
).to(model.device)

output = model.generate(
    input_ids, max_new_tokens=512, temperature=0.0, do_sample=False
)
response = tokenizer.decode(
    output[0][input_ids.shape[-1]:], skip_special_tokens=False
)

Set skip_special_tokens=False when decoding so you can see the <tool_call> tags in the raw output. Your parsing code needs those tags to extract the function call.

Once the model generates a response, you need to extract any tool calls, execute them, and feed results back. Here is a minimal agent loop:

import json
import re

def extract_tool_calls(text):
    pattern = r"<tool_call>\s*(\{.*?\})\s*</tool_call>"
    matches = re.findall(pattern, text, re.DOTALL)
    return [json.loads(m) for m in matches]

def run_agent(messages, tools_map, max_depth=5):
    for depth in range(max_depth):
        response = generate(messages)  # your inference call
        tool_calls = extract_tool_calls(response)

if not tool_calls:
            return response  # final answer, no more calls

messages.append({"role": "assistant", "content": response})

for call in tool_calls:
            func = tools_map[call["name"]]
            result = func(**call["arguments"])
            messages.append({
                "role": "tool",
                "content": f'<tool_response>
{{"name": "{call["name"]}", "content": {json.dumps(result)}}}
</tool_response>'
            })

return response

The loop runs until either the model produces a response with no <tool_call> tags (meaning it has enough information to answer) or the depth limit is reached. The max_depth parameter prevents infinite loops where the model keeps calling functions without converging on an answer. Five iterations is a reasonable default for most use cases.

Hermes 3 supports calling multiple functions in a single response. The model can emit several <tool_call> blocks in one turn, and the regex-based extractor above handles that naturally since re.findall returns all matches. Feed all results back before the next generation step.

Temperature matters for reliability. Set temperature=0.0 for all tool-calling inference steps. Any temperature above zero introduces sampling variance that can corrupt the JSON structure inside the tool call. A missing brace or unquoted key will break json.loads silently. Save higher temperatures (0.3 to 0.7) for the final human-readable response after all tool calls are resolved.

Beyond basic function calling, Hermes 3 was trained on several structured reasoning patterns that improve reliability in complex, multi-step tasks. The model recognizes special XML tokens like <SCRATCHPAD>, <REASONING>, <INNER_MONOLOGUE>, and <PLAN> for organizing its thought process before acting.

The scratch pad pattern is the most useful for function calling. You can instruct the model to plan before executing by including guidance in the system prompt:

Before making any function calls, use a <scratch_pad> to plan:
- Goal: Restate what the user needs
- Actions: List the function calls you will make
- Observation: After receiving results, summarize what you learned
- Reflection: Decide if more calls are needed or if you can answer

When the model follows this pattern, a typical response looks like:

<scratch_pad>
Goal: The user wants current stock prices for both AAPL and TSLA.
Actions: Call get_stock_price twice, once for each symbol.
</scratch_pad>

<tool_call>
{"name": "get_stock_price", "arguments": {"symbol": "AAPL"}}
</tool_call>
<tool_call>
{"name": "get_stock_price", "arguments": {"symbol": "TSLA"}}
</tool_call>

This is a form of Goal Oriented Action Planning (GOAP), a technique borrowed from game AI where an agent decomposes a high-level goal into a sequence of concrete actions. In game development, GOAP helps NPCs plan multi-step behaviors. In LLM function calling, it serves a similar purpose: the model states its goal, identifies what functions it needs, and then executes them in order.

The practical benefit is debugging. When a function calling chain produces wrong results, the scratch pad shows you exactly what the model was thinking at each step. You can see whether it misunderstood the user's request, chose the wrong function, or misinterpreted a result.

You do not need to use the scratch pad for simple single-function calls. It adds tokens to the output and slows generation. Reserve it for multi-step tasks where the model needs to coordinate three or more function calls or where the order of calls matters.

After setting up the basics, here are patterns that matter in production.

Schema Validation Validate tool calls against your function schemas before executing them. The model occasionally hallucinates argument names or types, especially at lower parameter counts. Pydantic makes this straightforward:

from pydantic import BaseModel, ValidationError

class StockPriceArgs(BaseModel):
    symbol: str

def safe_execute(call, tools_map, schemas):
    schema = schemas[call["name"]]
    try:
        validated = schema(**call["arguments"])
    except ValidationError as e:
        return {"error": str(e)}
    func = tools_map[call["name"]]
    return func(**validated.model_dump())

When validation fails, feed the error message back to the model as a tool response. Hermes 3 can usually self-correct on the next iteration.

Error Handling in the Loop

Functions fail. APIs time out, queries return empty results, and permissions get denied. Always wrap function execution in try/except and return the error as a tool response rather than crashing the loop:

try:
    result = func(**call["arguments"])
except Exception as e:
    result = {"error": f"{type(e).__name__}: {str(e)}"}

The model can reason about errors and either retry with different arguments or explain the failure to the user.

Context Window Management

Each tool call round-trip adds tokens to the conversation. Tool definitions in the system prompt can be 500+ tokens for a handful of functions. Results from API calls can be large. Monitor your total token count and consider these strategies:

• Summarize large tool responses before injecting them (trim API responses to relevant fields)
• Limit max_depth based on your available context window
• For the 8B model, keep total context under 8,192 tokens for reliable output quality

Persistent Storage for Agent Outputs

Function calling agents often produce artifacts that need to outlive the conversation: generated reports, extracted data, processed files. Storing these locally works for single-session experiments, but breaks down when you need to share results with a team, hand off from an agent to a human reviewer, or run multiple agents concurrently.

Fastio provides a workspace layer purpose-built for this. Agents connect through the MCP server and get access to 19 consolidated tools for file operations, search, and AI queries. Intelligence Mode auto-indexes uploaded files for semantic search, so a downstream agent or human can ask questions about what a previous agent produced without re-processing anything.

The free tier includes 50GB storage, included credits per month, and 5 workspaces with no credit card required. For Hermes 3 agents specifically, the pattern is: your agent calls external functions, generates results, uploads artifacts to a Fastio workspace via the API, and a human reviews them through the web UI. Local storage alternatives like SQLite or the filesystem work fine for single-machine prototypes. S3 or Google Cloud Storage handle scale but lack the built-in search and handoff features.