How to Automate Web Scraping With Hermes Agent's Scrapling Skill

The web scraping tools market passed $1 billion in 2025 and is growing at roughly 18% annually, according to The Business Research Company. That growth tracks a shift: scraping is no longer a developer writing BeautifulSoup scripts in a Jupyter notebook. It's becoming an agentic workflow where an AI agent decides what to scrape, picks the right extraction strategy, and routes the output to wherever your team needs it.

Nous Research's Hermes Agent, an open-source framework with over 91,000 GitHub stars, fits this pattern. Its Scrapling skill provides three fetching strategies: fast HTTP fetching for static pages, dynamic JavaScript rendering for single-page applications, and stealth mode with anti-bot bypass for protected sites. Each strategy maps to a different Python class, so the agent (or you) can pick the right tool without switching frameworks.

The gap this guide fills is specific. Most AI scraping tutorials focus on browser-use or custom Playwright scripts, which work fine for one-off jobs. Hermes Agent's Scrapling skill adds a spider framework with concurrent crawling and checkpoint-based pause/resume, turning a one-off extraction into a repeatable, resumable pipeline. If you're scraping five pages, any tool works. If you're scraping five thousand across sessions that might get interrupted, strategy matters.

Scrapling is an optional skill that ships with Hermes Agent but isn't active by default. Install it with one command:

hermes skills install official/research/scrapling

Or from inside a Hermes chat session:

/skills install official/research/scrapling

The skill wraps the Scrapling Python library, which needs its own setup. Choose the installation scope based on what you'll scrape:

HTTP only (static pages, APIs):

pip install scrapling

Full browser automation (JS-rendered and stealth):

pip install "scrapling[all]"
scrapling install

The scrapling install step downloads Chromium browsers for the dynamic and stealth fetchers. Skip it if you only need HTTP fetching.

Verify it works:

scrapling extract get 'https://quotes.toscrape.com' output.md --css-selector '.quote .text'

If that produces a file with quote text, you're set. If not, check that Python 3.10 or later is on your PATH and that the browser install completed without errors.

One thing to note on timeouts: the HTTP Fetcher measures timeouts in seconds, but DynamicFetcher and StealthyFetcher use milliseconds (default 30,000ms). This inconsistency trips people up, so keep it in mind when you start mixing fetcher types.

Scrapling's three fetchers aren't interchangeable. Each handles a different class of website, and picking the wrong one wastes time or fails outright.

HTTP Fetching for Static Pages

Use Fetcher or FetcherSession when the page content exists in the initial HTML response. This covers documentation sites, blogs, product listings, and most APIs. It's the fast option by a wide margin because there's no browser overhead.

from scrapling.fetchers import Fetcher

page = Fetcher.get('https://example.com/products')
titles = page.css('.product-title::text').getall()
prices = page.css('.price::text').getall()

For multiple requests that need to share cookies or session state, use FetcherSession:

from scrapling.fetchers import FetcherSession

with FetcherSession(impersonate='chrome') as session:
    login_page = session.get('https://example.com/login', stealthy_headers=True)
    dashboard = session.get('https://example.com/dashboard')
    data = dashboard.css('.metric::text').getall()

The impersonate='chrome' flag sends headers that match a real Chrome browser, which prevents basic bot detection on sites that check User-Agent strings.

Dynamic Fetching for JavaScript-Rendered Pages

Use DynamicFetcher when the content loads via JavaScript after the initial page load. Single-page applications built with React, Vue, or Angular fall into this category. The fetcher launches a headless browser, waits for rendering, then hands you the fully loaded DOM.

from scrapling.fetchers import DynamicFetcher

page = DynamicFetcher.fetch(
    'https://example.com/spa-dashboard',
    headless=True,
    wait_selector=('.data-table', 'visible'),
    network_idle=True,
)
rows = page.css('.data-table tr td::text').getall()

The wait_selector parameter tells the fetcher to hold until a specific element appears. Without it, you might capture the page before the data loads. Setting network_idle=True waits for all network requests to finish, which is slower but more reliable for pages that make multiple API calls during render.

Speed tip: add disable_resources=True to skip loading fonts, images, and stylesheets. The official docs report roughly 25% faster extraction with this flag enabled, since you're not waiting for assets you don't need.

Stealth Mode for Protected Sites

Use StealthyFetcher when a site actively blocks automated browsers. This includes sites behind Cloudflare Turnstile, Akamai Bot Manager, or similar anti-bot systems. Stealth mode uses fingerprint obfuscation, WebRTC blocking, and canvas hiding to look like a real user.

from scrapling.fetchers import StealthyFetcher

page = StealthyFetcher.fetch(
    'https://protected-site.com/data',
    headless=True,
    solve_cloudflare=True,
    block_webrtc=True,
    hide_canvas=True,
)
content = page.css('.content::text').getall()

Cloudflare solving adds 5 to 15 seconds of overhead per request, but it only activates when the solver detects a challenge page. On pages without active protection, stealth mode runs at roughly the same speed as dynamic fetching.

The CLI equivalents for all three strategies:

scrapling extract get 'https://static-site.com' out.md --css-selector '.content'
scrapling extract fetch 'https://spa-site.com' out.md --css-selector '.data' --network-idle
scrapling extract stealthy-fetch 'https://protected.com' out.html --solve-cloudflare

Persist Your Scraped Data Where Your Team Can Use It

Upload extraction results to a shared workspace with versioning, permissions, and AI-powered search. generous storage, no credit card, MCP-ready for direct agent uploads.

Start 14-Day Trial

Single-page fetching is the starting point. The real power comes from the Spider framework, which handles link following, concurrency, and checkpointing across hundreds or thousands of pages.

Here's a spider that crawls a quotes site, follows pagination links, and yields structured data:

from scrapling.spiders import Spider, Response

class QuotesSpider(Spider):
    name = "quotes"
    start_urls = ["https://quotes.toscrape.com/"]
    concurrent_requests = 10
    download_delay = 1

async def parse(self, response: Response):
        for quote in response.css('.quote'):
            yield {
                "text": quote.css('.text::text').get(),
                "author": quote.css('.author::text').get(),
                "tags": quote.css('.tag::text').getall(),
            }
        next_page = response.css('.next a::attr(href)').get()
        if next_page:
            yield response.follow(next_page)

result = QuotesSpider().start()
result.items.to_json("quotes.json")

The concurrent_requests parameter controls how many pages load in parallel. Set download_delay to add a polite pause between requests. Both are essential for staying within rate limits and not hammering target servers.

Multi-Session Spiders

Some crawls hit a mix of static and protected pages. A multi-session spider lets you route different URLs through different fetcher strategies automatically:

from scrapling.fetchers import FetcherSession, AsyncStealthySession
from scrapling.spiders import Spider, Request, Response

class SmartSpider(Spider):
    name = "smart_crawler"
    start_urls = ["https://example.com/sitemap"]
    concurrent_requests = 5

def configure_sessions(self, manager):
        manager.add("fast", FetcherSession(impersonate="chrome"))
        manager.add("stealth", AsyncStealthySession(headless=True), lazy=True)

async def parse(self, response: Response):
        for link in response.css('a::attr(href)').getall():
            if "protected" in link or "members" in link:
                yield Request(link, sid="stealth")
            else:
                yield Request(link, sid="fast")

The lazy=True flag on the stealth session means the browser only launches when the spider encounters a protected URL. If your crawl is mostly static pages with a few protected ones, this saves significant startup time.

Checkpoint-Based Pause and Resume

Long crawls get interrupted. Network drops, rate limits kick in, or you need to shut down for the night. Scrapling's spider framework supports checkpoint-based pause and resume, so you can stop a crawl mid-run and pick up exactly where you left off. This is particularly valuable for large-scale extraction jobs that run across multiple Hermes Agent sessions.

Getting the page is half the job. Extracting the right data from it is the other half. Scrapling provides CSS selectors, XPath expressions, and find methods that cover everything from simple text extraction to complex DOM traversal.

CSS selectors handle most cases:

page.css('h1::text').get()
page.css('a::attr(href)').getall()
page.css('.product .price::text').getall()

XPath for complex DOM navigation:

page.xpath('//div[@class="content"]/p/text()').getall()
page.xpath('//table//tr[position()>1]/td[2]/text()').getall()

Find methods when you know what you're looking for but not exactly where it is:

page.find_all('div', class_='product-card')
page.find_by_text('Add to cart', tag='button')
page.find_by_regex(r'\$\d+\.\d{2}')

DOM navigation for traversing element relationships:

element = page.css('.target')[0]
parent_text = element.parent.css('::text').get()
sibling_data = element.next_sibling.css('::text').get()
children = element.children

One practical pattern: combine find_by_regex with CSS selectors to extract structured data from semi-structured pages. Find price elements by pattern, then navigate to their parent containers to get the associated product names and descriptions.

Extracted data sitting in a JSON file on your server isn't useful to the rest of your team. The final step in any scraping pipeline is getting the output somewhere accessible, versioned, and searchable.

Local options work for solo projects. Write to a local directory, commit to git, or push to S3. For teams where multiple people (or multiple agents) need to access and act on scraped data, a shared workspace makes more sense.

Fastio works well here because Hermes Agent can write files directly through the Fastio MCP server. Upload extracted datasets as JSON or CSV, and your team gets versioned files with granular permissions, audit trails, and search. Enable Intelligence Mode on the workspace and the uploaded data becomes queryable through AI chat, so a teammate can ask "what were the top 10 products by price from last week's scrape?" without opening a spreadsheet.

The free tier gives you 50GB of storage, included credits per month, and five workspaces with no credit card required. That covers a lot of scraping output before you need to think about paid plans.

Other options worth considering:

• Google Sheets API for small datasets that non-technical teammates need to browse
• Supabase or PostgreSQL for structured data that feeds downstream applications
• S3 or R2 for raw HTML archives where storage cost matters most

The key is building the persistence step into your spider or post-processing script so it runs automatically. An agent that scrapes data and leaves it in /tmp is doing half the job.