How to Check If Code Is AI Generated with Hermes Agent

Eighty-four percent of respondents in the 2025 Stack Overflow Developer Survey use or plan to use AI tools in development, yet 46% actively distrust the accuracy of AI tool output, more than the 33% who trust it. That adoption-trust gap is the practical reason people keep asking "is this code AI generated" in pull requests, take-home tests, and student submissions.

Checking whether code is AI generated means scoring style, structure, and detector signals, then classifying the sample as likely human, likely AI, or inconclusive. A single online score is not enough. Use this six-step path every time:

1. Capture the sample and context. Save the exact file, diff, or paste. Note language, assignment constraints, repo history, and whether AI tools are allowed.
2. Score stylistic signals. Look for generic variable names, comment density that does not match the author's other work, and textbook-perfect structure with no dead ends.
3. Score structural signals. Check for missing edge cases the author usually handles, over-engineered helpers for a tiny task, or patterns that appear nowhere else in the repo.
4. Run at least one AI code detector. Tools such as MyDetector's AI Code Detector return an AI probability score and a report. Treat the score as one input, not a verdict.
5. Cross-check process evidence. Git history, intermediate commits, oral walkthroughs, and "why this approach" questions matter more than a 78% detector number.
6. Classify with a threshold. Map combined signals to likely human, likely AI, or inconclusive. When process evidence and detector scores conflict, default to inconclusive and ask for a live explanation.

Search demand around this question is real but modest: about 210 monthly US searches for "is this code ai generated," 210 for "ai generated code detector" (CPC around $12.67), and 140 for "ai code detection" (CPC around $17.90). People who land here usually need a decision framework, not another tool list.

Style and structure are weak signals alone. Stack them carefully.

Signals that often appear in AI-generated code

• Uniform polish. Every helper has a docstring, every branch is symmetric, and naming is consistent in a way that does not match the author's prior messy but working code.
• Template explanations. Comments restate what the code already says, or mirror common tutorial phrasing without tying to the product domain.
• Over-abstraction for the prompt. A five-line homework task arrives with factories, strategy objects, and config layers the course never introduced.
• Missing local knowledge. No references to internal libraries, house style, or the bug ticket that motivated the change, even when those are required.
• Sudden skill jump. The same author who struggled with async error handling last week ships perfect cancellation, retries, and typed result wrappers with no intermediate commits.

Signals that do not prove AI authorship

• Clean formatting. Prettier, Black, and gofmt make human code look machine-regular.
• Standard algorithms. Sorting, REST handlers, and CRUD endpoints converge on the same shapes whether a human or a model wrote them.
• Non-native English in comments. Variable English skill is not an authorship detector, and false accusations here are common.
• A high detector score on a short snippet. Short samples are unstable. Prefer whole files or meaningful functions.

Confidence thresholds that keep you honest

Use explicit bands so reviewers do not invent policy on the fly:

• 0–35 combined score: lean likely human unless process evidence strongly conflicts.
• 36–64: inconclusive. Request a walkthrough or a second detector. Do not fail a student or reject a hire on this band alone.
• 65–100 with corroborating process evidence: likely AI. Still allow the author to explain, especially if policy permits AI assistance with disclosure.

Combined score means your own checklist points plus one or more detector probabilities, not a single vendor number. MyDetector describes its AI Code Detector as combining static analysis, style checks, and model predictions into a probability score, and notes that no detector is 100% certain. Independent testing of general AI detectors often finds real-world accuracy far below vendor marketing. Apply the same humility to code.

Nous Research Hermes Agent is an open-source (MIT) agent with a skill system compatible with the agentskills.io standard. Skills live under ~/.hermes/skills/ as knowledge documents the agent loads on demand. Official docs describe progressive disclosure: list skills cheaply, load full SKILL.md content only when needed, then optional reference files.

That model is a good fit for "is this code written by AI" checks. You want a reusable procedure that reads a snippet, calls tools, and returns a structured judgment, not a one-off chat.

What Hermes tools give you for this workflow

Hermes groups tools into toolsets you enable with hermes chat --toolsets "web,terminal" or hermes tools. For a code-origin skill, the relevant categories from the official tools guide are:

• Terminal and files: terminal, process, read_file, patch to open candidate files and run local scripts.
• Agent orchestration: execute_code to run scoring helpers, todo to track multi-file PR checks.
• Web: web_search / web_extract if you call a hosted detector API or documentation endpoint.
• Skills toolset: load your verification skill via slash command once installed.

Terminal backends include local, Docker, SSH, Singularity, Modal, and Daytona. For untrusted student or candidate code, prefer Docker or SSH so analysis stays off your host.

Author a skill with the documented SKILL.md shape

Hermes skills use frontmatter plus markdown sections. A practical verification skill looks like this (create it under ~/.hermes/skills/devtools/ai-code-check/SKILL.md or ask Hermes to draft it with /learn after you walk through the procedure once):

---
name: ai-code-check
description: Score code origin signals and classify likely human, likely AI, or inconclusive
version: 1.0.0
metadata:
  hermes:
    tags: [code-review, detection, integrity]
    category: devops
    requires_toolsets: [terminal]
---

Title: AI Code Origin Check

When to Use
User asks whether a snippet, file, or PR looks AI generated, or wants a structured integrity report.

Procedure
1. Read the target path or pasted snippet with read_file / conversation context.
2. Collect style and structure notes against the checklist in this skill.
3. Optionally run a detector CLI or HTTP API via terminal/execute_code.
4. Weight process evidence the user provides (git log, interview notes).
5. Return JSON: classification, confidence 0-100, signals[], caveats[], next_questions[].

Pitfalls
- Never treat a single detector score as proof.
- Short snippets under ~40 lines are usually inconclusive.
- Allowed AI assistance with disclosure is not the same as undisclosed generation.

Verification
Classification is one of: likely_human | likely_ai | inconclusive.

Official Hermes skill guidance requires a short description, clear procedure, pitfalls, and verification. You can also generate the skill from a live walkthrough with /learn how I just checked this PR for AI-generated code, then edit the saved SKILL.md.

Invoke the skill

Start the interactive CLI with hermes, then run the skill slash command on a path:

hermes
/ai-code-check review src/payments/refund.py against our house style

Or ask in natural language with the skills toolset enabled:

hermes chat --toolsets "skills,terminal,file" -q "Is this code AI generated? Check ./candidate/solution.py"

Stack skills when useful. Hermes allows chaining up to five leading slash skills in one message, for example combining a PR workflow skill with the origin check. For repeated combos, a skill bundle is cleaner than retyping chains.

Structured judgment output

Have the skill always emit the same shape so humans and downstream tools can parse it:

{
  "classification": "inconclusive",
  "confidence": 48,
  "signals": [
    "uniform docstring density atypical for author",
    "detector_score_ai: 0.61"
  ],
  "caveats": [
    "single-file sample",
    "no git history provided"
  ],
  "next_questions": [
    "Walk through why you chose a strategy pattern here",
    "Show intermediate commits if any"
  ]
}

Persist those reports somewhere shared. Local folders and object storage (S3, Google Drive) work for solo reviewers. For agent-plus-human teams, a shared Fast.io workspace with Intelligence Mode keeps reports searchable and citable alongside the original snippets. Hermes can talk to external systems through MCP; Fast.io exposes Streamable HTTP at /mcp and legacy SSE at /sse so an agent can upload the JSON report and the reviewed file without a separate vector database. See the storage for agents overview for how agent workspaces fit the handoff.

Save AI-generated code checks where reviewers can find them

When you ask is this code AI generated, keep the snippet, Hermes skill report, and interview notes in one workspace with version history and Intelligence search. Every org starts with a 14-day free trial.

Start 14-Day Trial

The same skill should not apply the same policy in every context. False positives hurt differently in each setting.

Pull requests

Goal: flag high-risk merges, not police every autocomplete suggestion.

• Run the Hermes skill on the diff, then on whole changed files when the diff is large.
• Weight repo consistency heavily. If the PR continues an existing pattern the author already owns, lower the AI suspicion score even if a detector is noisy.
• Prefer inconclusive + questions over blocking merges when policy allows AI assistance with disclosure. Ask the author to list which tools they used in the PR description.
• Store the skill report next to the PR link. Version history on the report file matters when policy is contested weeks later.

Hermes can read files and run terminal commands, so a practical PR loop is: checkout the branch, run /ai-code-check on changed paths, paste the JSON summary into the review, and only escalate when classification is likely_ai with confidence above your team threshold (for example 70) plus at least two structural signals.

Homework and academic integrity

Goal: protect learning without punishing clean style or non-native English comments.

• Collect process evidence first: scaffold commits, design notes, lab logs.
• Use detectors only as corroboration. A mid-range score without process anomalies should stay inconclusive.
• Offer a live oral defense before any academic penalty. Ask the student to modify a function under observation or explain a failure case they never tested.
• Document every step. If you keep submissions on a drive share or LMS export, fine. If agents help grade at scale, put submissions and skill reports in a workspace humans can audit, for example Fast.io with per-file version history and an append-only audit log.

Interviews and take-homes

Goal: assess how the candidate thinks, not whether they ever opened an AI tool.

• State AI policy up front. If AI is forbidden, say so in the brief and the calendar invite.
• Time-box a live follow-up. Candidates who generated the take-home still struggle when asked to extend it or fix a planted bug.
• Score communication and tradeoffs as hard as detector output. A perfect solution with zero ability to reason about edge cases is a stronger negative signal than a medium detector score.
• Never auto-reject solely on an AI code detector. Use Hermes to prepare a question list from the next_questions field, then have a human run the interview.

False-positive handling playbook

When the skill says likely_ai and the author disputes it:

1. Re-run on a larger sample or prior work from the same person.
2. Run a second detector if available.
3. Hold a 15-minute walkthrough focused on design choices, not "did you use ChatGPT."
4. If explanation holds, reclassify to inconclusive or likely human and update the report.
5. Only then apply policy actions (rewrite request, failed assignment, rescinded offer).

That order protects students and candidates from the worst failure mode of this category: a confident wrong number.

Origin checks produce artifacts: snippets, detector JSON, Hermes skill reports, and interview notes. Choose storage that matches retention and access needs.

Local disk is fine for one reviewer and disposable checks. It fails when another instructor or hiring manager needs the same trail.

GitHub PR comments keep context next to the code but scatter evidence across threads and get lost when repos archive.

S3 or Google Drive work for bulk archives. You still need naming conventions, permission hygiene, and a way for agents to read and write without custom glue.

Fast.io sits in the agentic middle: shared org-owned workspaces, per-file version history, granular permissions, branded shares for sending a report pack to a hiring panel, and Intelligence Mode so you can ask "which take-homes were classified inconclusive last week?" with citations. Agents and humans use the same files. Agents can connect through the consolidated MCP toolset; humans use the UI. Plans start with a 14-day free trial per organization (credit card required), then Starter at $29/mo, Business at $99/mo, or Growth at $299/mo. Real workspace use requires a paid organization subscription after the trial.

A clean Hermes-to-human flow:

1. Hermes runs /ai-code-check on the candidate tree in a Docker terminal backend.
2. The skill writes report.json and a short report.md summary.
3. The agent uploads both plus the original files into a Fast.io workspace (or you drop them via the UI / URL import from Drive or Dropbox).
4. Enable Intelligence so later reviewers can search by meaning, not only filename.
5. Share a time-bounded link with the hiring panel or teaching team.
6. When an agent set up the org for a client engagement, use ownership transfer so a human admin owns the workspace going forward while the agent can retain admin access as configured.

That pattern keeps Hermes as the execution and skill layer, and Fast.io as the durable collaboration layer, without claiming Hermes natively hosts team file sharing.

Detectors are statistical models. They estimate how much a sample resembles training distributions associated with machine generation. They do not see the author's screen.

Common failure modes:

• Edited AI code. A few human refactors often move scores into the inconclusive band.
• Human code that matches common tutorials. Boilerplate looks "synthetic."
• Language and domain shift. A detector strong on Python web code may be noisy on embedded C.
• Policy mismatch. Many teams allow AI for scaffolding if the human owns design and tests. A detector cannot encode that nuance unless your Hermes skill does.

MyDetector positions its AI Code Detector as multi-language analysis with a probability score and report, and states accuracy claims on its marketing pages. Treat vendor accuracy numbers the way you treat any vendor claim: useful as a starting point, incomplete without your own calibration set. Build a small golden set of 20–30 samples you already know are human, AI, or mixed. Run them through your Hermes skill quarterly and adjust thresholds.

Hermes stays useful here because skills are editable procedures, not black boxes. When your policy changes, you update the SKILL.md procedure and pitfalls sections. When a new detector API appears, you add a terminal step. When you need auditability, you write the JSON report to durable storage. The agent is the orchestrator; you own the definition of "likely AI."

Bottom line: ask "is this code AI generated" as a structured investigation. Use Hermes Agent to run that investigation the same way every time. Classify with room for inconclusive. Keep humans in the loop wherever the outcome affects grades, jobs, or trust.