How to Set Up Offline Speech Recognition on Raspberry Pi for OpenClaw Voice Agents

Whisper Tiny transcribes a 10-second audio clip in roughly 2-3 seconds on a Raspberry Pi 5 with 8 GB RAM using the faster-whisper runtime. Two years ago, offline voice recognition on a $35 board meant choosing between unusable accuracy and multi-second lag. The performance gap has closed enough to make offline voice agents practical on hardware you can hold in one hand.

Speech recognition on Raspberry Pi converts spoken audio into text using on-device machine learning models like Whisper, enabling voice-controlled applications without sending audio to cloud servers. Most Raspberry Pi voice tutorials still route audio through Google Cloud Speech-to-Text or Amazon Transcribe. That works until you need to process audio in a location without reliable internet, or until your monthly API bill exceeds the cost of the hardware running the pipeline. A fully offline stack avoids both problems.

The key hardware change is the Raspberry Pi 5. Its quad-core Cortex-A76 processor at 2.4 GHz and 8 GB memory option give Whisper enough room to run alongside a wake word engine, a TTS engine, and basic system processes. Previous Pi models could handle lightweight STT with Vosk, but lacked the memory headroom for Whisper models plus the rest of the pipeline.

On the software side, optimized runtimes have closed the speed gap. The original Python Whisper implementation takes about 6 seconds to transcribe 10 seconds of audio on Pi 5. faster-whisper, which uses the CTranslate2 inference engine, cuts that to 2-3 seconds. whisper.cpp, a pure C/C++ port, runs even faster. Both runtimes are open source and compile natively on ARM without GPU dependencies.

For OpenClaw users, this creates an appealing deployment pattern. OpenClaw already supports voice input through USB microphones on Pi hardware. The openclaw-voice-agent project on GitHub demonstrates the full pipeline: Porcupine detects a wake word, audio capture begins, Whisper transcribes the speech, and OpenClaw processes the command. The Pi handles all audio processing locally. Only the transcribed text goes to an LLM API, keeping raw audio on-device.

Choosing an offline speech-to-text engine for Raspberry Pi means balancing accuracy, speed, and memory. Three options dominate the landscape, each designed for different constraints.

Whisper (via faster-whisper or whisper.cpp)

OpenAI released Whisper as an open-source model trained on 680,000 hours of multilingual audio. On Raspberry Pi, two optimized runtimes handle inference: whisper.cpp (a C/C++ port using the ggml library) and faster-whisper (a Python library wrapping CTranslate2). Both produce identical transcription output from the same model weights.

Model selection determines the accuracy-speed tradeoff on Pi 5:

• Whisper Tiny (39M parameters, ~75 MB on disk): Processes speech in 2-3 seconds with faster-whisper. Good enough for voice commands in quiet environments.

• Whisper Base (74M parameters, ~142 MB): Roughly 30% more accurate than Tiny on clean speech. Both Tiny and Base fit within 1 GB of RAM.

• Whisper Small (244M parameters, ~466 MB): Noticeably more accurate with accented speech or background noise. Needs about 2 GB of RAM and is too slow for real-time voice commands on Pi 5. Medium and Large models exceed available memory and are not practical for this hardware.

Use the English-only variants (tiny.en, base.en, small.en) for English-only applications. They skip the language detection step and are slightly faster and more accurate than the multilingual models for English speech.

Vosk

Vosk is purpose-built for embedded devices. Models range from 50 MB to 1.8 GB, and the engine runs on as little as 500 MB of RAM. Where Vosk beats Whisper on Pi hardware is streaming: it provides native real-time streaming with word-by-word output as audio arrives, while Whisper processes audio in chunks and returns results after the full segment completes.

Vosk runs on hardware as small as a Raspberry Pi Zero, something Whisper cannot do. It supports about 20 languages and requires no GPU. The tradeoff is accuracy. On clean audio with a close microphone, Vosk performs well. On noisy recordings, accented speech, or mixed-language input, Whisper produces notably better results.

For voice command applications where the vocabulary is limited and predictable, Vosk is the lighter option. For open-ended transcription where users might say anything, Whisper handles the variety better.

Picovoice Leopard and Cheetah

Picovoice offers two commercial STT engines optimized for edge devices. Leopard handles batch processing, Cheetah does streaming. Both use about 20 MB of storage and run on a single CPU core. Picovoice claims accuracy matching cloud-based APIs while running entirely on-device.

The constraint is pricing. Picovoice's starter plan costs $6,000 per year for up to 100 monthly active devices. The free tier covers development and single-device prototyping. For a personal project or home automation setup, the free tier works. For anything at scale, the license cost is worth evaluating against cloud API pricing.

Which Engine to Choose

For OpenClaw voice agents handling general-purpose commands in a home or office, Whisper Tiny through faster-whisper offers the best balance of accuracy, speed, and zero licensing cost. For always-on streaming with the smallest possible footprint, Vosk fills a gap that Whisper cannot reach on lower-end Pi hardware.

A voice agent that runs Whisper continuously would consume all available CPU processing audio it does not need to transcribe. Wake word detection solves this by running a lightweight model that listens for a trigger phrase and only activates the STT pipeline when it hears one. The wake word engine uses a fraction of the CPU that Whisper needs, so it can run indefinitely on battery-powered or always-on hardware.

Porcupine by Picovoice

Porcupine is the most widely tested wake word engine on Raspberry Pi. It uses under 4% of a single CPU core on a Pi 3 and about 1 MB of memory. On a Pi Zero, it runs in real time with near-zero overhead. Picovoice benchmarks show Porcupine as 11x more accurate and 6.5x faster than competing engines on Pi 3 hardware, while consuming 45% less memory.

Porcupine ships with pre-trained wake words including "jarvis," "terminator," "porcupine," and "computer." Custom wake words require the Picovoice Console, where you type your desired phrase and download a platform-specific model file. For Raspberry Pi, select the ARM architecture matching your board: 32-bit for Pi Zero, 64-bit for Pi 4 and Pi 5.

The openclaw-voice-agent project uses Porcupine as its default wake word engine. The flow works like this: Porcupine listens on the microphone continuously, detects the wake phrase, triggers audio capture for a configured duration, then passes the captured audio to Whisper for transcription.

Porcupine's free tier allows custom wake words and unlimited built-in keywords, enough for personal projects and single-device deployments.

openWakeWord

openWakeWord is the fully open-source alternative. It builds on Google's pre-trained speech embedding model as a frozen feature extractor, training small classification heads on top using 100% synthetic speech data. No real audio recordings are needed to create a custom wake word.

Performance is competitive with Porcupine. A single core of a Raspberry Pi 3 can run 15-20 openWakeWord models simultaneously in real time. On the Dinner Party Corpus dataset (roughly 5.5 hours of far-field speech in noisy multi-speaker environments), the bundled "alexa" model outperforms Porcupine's equivalent.

The tradeoff is setup complexity. openWakeWord requires Python, ONNX Runtime, and model files you train or download. Training a custom wake word involves generating synthetic speech through a TTS engine, processing it through the embedding pipeline, and training a small classifier. The process is documented on the project's GitHub repository, but it is not as turnkey as Porcupine's web console.

For OpenClaw deployments, the choice comes down to convenience versus licensing. Porcupine gives you a working wake word in minutes through the free tier. openWakeWord gives you full control with zero licensing concerns, at the cost of a longer initial setup.

Store and search your voice agent's output across sessions

Shared storage with automatic semantic indexing. Your OpenClaw agent's transcripts, files, and reports stay searchable and shareable. Starts with a 14-day free trial.

Start 14-Day Trial

This section covers practical setup for running Whisper offline on Pi 5 hardware. The two viable runtimes are whisper.cpp and faster-whisper. They differ in language ecosystem, build dependencies, and integration patterns.

Choosing a Runtime

whisper.cpp is a C/C++ implementation that compiles natively on ARM without Python or GPU dependencies. Clone the repository, build with make, download a ggml-format model file, and run the binary directly against audio files. It is the fast option for bare-metal Pi 5 deployments and works well for standalone transcription tasks.

faster-whisper is a Python library that wraps CTranslate2, an inference engine that quantizes and optimizes transformer models. It is 4-5x faster than the original Python Whisper implementation and uses less memory. Install it via pip, and it integrates naturally with Python-based voice agent pipelines. For OpenClaw voice agents, faster-whisper is the more practical choice because the agent toolchain is already Python.

Hardware Recommendations

The Raspberry Pi 5 with 8 GB RAM is the recommended baseline for running Whisper alongside other pipeline components. The 4 GB variant can run Whisper Tiny alone, but runs out of memory when you add a wake word engine, TTS, and any local LLM. For a fully local pipeline running Whisper plus Phi-3 Mini (quantized to Q4), the LLM alone consumes 1-1.5 GB and Whisper models add another 700 MB to 2 GB depending on size.

A USB microphone is the simplest audio input. Omnidirectional USB microphones work for desk setups where the speaker is within arm's reach. For far-field voice capture across a room, a multi-element array microphone with beamforming provides better results by filtering directional noise.

Audio quality has a larger effect on transcription accuracy than model size. A clear recording through a decent microphone with Whisper Tiny will outperform a noisy recording through Whisper Small. Budget for the microphone before upgrading the model.

Practical Tips

Use the .en model variants (tiny.en, base.en) for English-only applications. They skip the language detection step and produce slightly faster, more accurate results for English speech.

If your voice agent only needs to recognize a limited set of commands rather than open-ended dictation, consider pairing Whisper with a post-processing step that maps transcriptions to known intents. Whisper sometimes hallucinates words on short or quiet audio clips. Matching against a known command vocabulary filters out these errors before they reach OpenClaw.

For batch transcription of recorded files rather than live audio, whisper.cpp with the Whisper Base model is the sweet spot. Base gives you about 30% better accuracy than Tiny while still processing faster than real time on Pi 5.

Test your pipeline with different microphone distances and ambient noise levels before deploying. Whisper's accuracy drops noticeably when the speaker is more than a meter from the microphone in a room with background conversation or HVAC noise. A simple gain calibration step at startup, where you ask the user to say a test phrase, helps catch microphone issues early.

For persistent deployments, configure the Pi to run the voice pipeline as a systemd service that starts on boot. This way, the voice agent is ready as soon as the Pi powers on without needing SSH access or a display connected.

OpenClaw is an open-source AI assistant framework that gives an LLM the ability to run tools, manage workflows, and interact through messaging channels. On a Raspberry Pi, OpenClaw acts as a lightweight controller: it receives transcribed text from the speech pipeline, routes it to an LLM API for processing, and executes the resulting actions locally.

The openclaw-voice-agent project demonstrates one complete implementation. The architecture follows a straightforward loop:

1. Porcupine listens continuously for the configured wake word
2. Audio capture begins when the wake word is detected
3. Whisper transcribes the captured audio to text
4. The transcription is sent to OpenClaw's local HTTP endpoint
5. OpenClaw processes the command through the configured LLM provider
6. The response is spoken back through Piper TTS (offline) or gTTS (requires internet)

OpenClaw runs as a local Node.js gateway using your AI provider credentials. The gateway handles the interface between the voice pipeline and whichever LLM you configure, whether that is Anthropic, OpenAI, or another provider. No separate OpenClaw API key is needed for the local gateway.

End-to-End Latency

The full pipeline latency depends on whether the LLM runs locally or remotely. With a cloud LLM API, the bottleneck shifts from on-device processing to network round-trip time. Expect 3-5 seconds from spoken command to spoken response when the STT and TTS run locally and only the LLM call goes over the network.

With a fully local stack (Whisper for STT, Phi-3 Mini for the LLM, Piper for TTS), benchmarks on Pi 5 with 8 GB RAM show 15-25 seconds end-to-end without streaming optimization. When TTS output streams during LLM generation, speaking words as they are produced rather than waiting for the complete response, perceived latency drops to 8-12 seconds.

For most voice agent use cases, the hybrid approach is more practical. The Pi handles wake word detection and speech-to-text offline, so raw audio never leaves the device. Only the transcribed text goes to the API. This preserves audio privacy while keeping response times fast enough for conversational use.

Persistent Storage for Voice Agent Artifacts

Voice agents that do real work produce artifacts: transcripts, downloaded files, configuration exports, generated reports. OpenClaw's local workspace gives each agent a directory for session state, but that storage lives on the Pi's SD card. Reimage the card or move to a different Pi, and the data is gone.

Local options like NFS mounts or S3-compatible object storage work but require you to manage infrastructure separately. They also give you no built-in search or access control. Fast.io takes a different approach. Instead of bare storage, you get an intelligent workspace where uploaded files are automatically indexed for semantic search. Enable Intelligence Mode, and anything your voice agent writes becomes queryable by other agents or by humans through the same interface. No separate vector database or ETL pipeline needed.

Plans start with a 14-day free trial (Solo at $29/month, Business at $99/month), with storage, monthly credits, and workspaces scaled to each tier. Fast.io exposes 19 MCP tools via Streamable HTTP at /mcp, so your OpenClaw agent can read, write, search, and manage files through the same API surface it uses for other tools. For a voice agent that logs transcripts or generates reports, having searchable persistent storage means you can query past conversations by meaning rather than grepping through flat files.

The MCP documentation is at mcp.fast.io, and the agent storage guide is at fast.io/storage-for-agents.