How to Configure txtai Storage Solutions for Embeddings Databases

txtai storage holds two things: vector embeddings and source content. Standard databases store text and numbers. txtai works differently because it handles high-dimensional vectors along with your metadata.

Think of it like RAM versus a Hard Drive. The vector index (RAM) is built for speed. The document store (Hard Drive) handles reliability and capacity. txtai connects them. You find the specific result (vector search) and then pull the full document (retrieval) without crashing your system.

Your storage choice changes search speed. An in-memory index is fine for a personal chatbot, but a production RAG (Retrieval-Augmented Generation) system needs a real architecture. You need embeddings to load fast, even as your data grows.

By default, txtai runs in memory or on local files. For agents processing terabytes of data, you need persistent storage. This lets you scale your search index separately from your app.

Helpful references: Fast.io Workspaces, Fast.io Collaboration, and Fast.io AI.

The vector index runs the search. It matches queries to documents. Choose your backend based on speed needs and data size.

Hnswlib (Default) txtai uses Hnswlib by default. This library runs approximate nearest neighbor search and stores indices as local binary files. It works well for datasets under 1 million vectors where speed is the goal. Hnswlib builds a graph structure for fast traversal. It works great when the index fits in RAM. But since it's file-based, managing it across servers is difficult.

Faiss For larger datasets, txtai uses Faiss (Facebook AI Similarity Search). Faiss supports GPU acceleration and quantization. This reduces the memory usage of your vectors by up to 75%. It also includes clustering and product quantization to save space without losing much accuracy. This helps when storage costs add up.

Faiss also supports on-disk storage for indices that are larger than RAM. This "on-disk" mode uses memory mapping (mmap) to access the index without loading it entirely. It trades some speed for the ability to handle billion-scale datasets on modest hardware.

PostgreSQL (pgvector) If you already use PostgreSQL, the pgvector extension lets you store embeddings directly in your relational database. Benchmarks show pgvector mixes speed and simplicity, so you don't need to manage a separate vector database.

Give Your AI Agents Persistent Storage

Stop rebuilding indices on every restart. Get 50GB of free, persistent storage for your AI agents and vector databases.

Get Free Agent Storage

The vector index finds the location. Document storage holds the content, such as text, images, or JSON.

SQLite (Embedded) txtai uses SQLite as the default content store. It's serverless, fast, and lives in a single file next to your vector index. This works for single-agent deployments or local testing. SQLite is zero-config. No database server, no user permissions. The file just appears in your working directory, making it easy to back up or move.

DuckDB (Analytical) For heavy analytical workloads, you can configure txtai to use DuckDB. DuckDB runs column-oriented queries, making it faster when you need to filter search results by metadata fields (e.g., "find documents from 2024").

Client-Server Databases For multi-agent systems, local files like SQLite handle concurrent reads but fail at concurrent writes. txtai connects to external PostgreSQL, MySQL, or MariaDB servers. This allows multiple agents to write to the same knowledge base at the same time without file locking issues. Moving to a client-server model also centralizes management. You can use tools like pgAdmin to inspect data or run SQL queries without slowing down your agents.

Stateful workloads like vector databases cause problems in containers. txtai fixes this by supporting cloud object storage for archiving and loading indices.

You can configure txtai to save compressed indices directly to S3, Google Cloud Storage, or Azure Blob Storage. This allows for a stateless architecture. Agents start, pull the latest index from the cloud, and begin serving queries immediately.

For example, an indexing agent can run nightly to rebuild the vector store and push it to S3. Search agents then pull the new index, ensuring they always have fresh data.

This setup helps with data pipelines. Just as you version control code, you should version control embeddings. By treating your vector index as an artifact that you build, test, and deploy, you can roll back to previous versions if needed.

Configuring the right storage backend happens in your embeddings object. Here are common patterns for different scales.

Local Prototyping (Default) For simple scripts, the default setup writes to a local directory. This is the zero-dependency route. You define the path, and txtai handles the rest.

### Basic configuration
config = {
    "path": "sentence-transformers/all-MiniLM-L6-v2",
    "content": True
}

This keeps everything self-contained. It is perfect for testing or single-user desktop apps where latency is strictly local.

PostgreSQL for Teams When you move to production, switch the content backend to a standard SQL database. This configuration tells txtai to use an external PostgreSQL instance.

### Postgres configuration
config = {
    "path": "sentence-transformers/all-MiniLM-L6-v2",
    "content": "postgresql://user:password@localhost:5432/vectors"
}

By pointing the content parameter to a connection string, you move storage management to a dedicated server. This allows for concurrent writes and better data integrity.

Cloud Archiving To use cloud storage, you configure the cloud parameter. This tells txtai where to push the built index.

### Cloud configuration
config = {
    "path": "sentence-transformers/all-MiniLM-L6-v2",
    "cloud": {
        "provider": "s3",
        "container": "my-bucket",
        "key": "indices/v1"
    }
}

This pattern is needed for containerized environments like Kubernetes, where local disk is ephemeral. It allows your agents to stay stateless while accessing persistent, versioned data.

Storage is often the main bottleneck for autonomous agents. Agents need a place to keep indices, logs, and memories that lasts after a restart.

Why Fast.io Fast.io provides a persistent filesystem for AI agents. With our free agent tier, you get 50GB of storage that your agents can access via standard file protocols or our MCP server.

• Universal Access: Mount your storage as a drive or access it via API.
• MCP Integration: Use the Fast.io MCP server to let Claude or custom agents manage txtai indices directly.
• Zero Ops: No need to manage S3 buckets or IAM roles. Fast.io handles the infrastructure.

By storing your txtai indices in a Fast.io workspace, you build a shared memory layer that any authenticated agent can access.

Shared memory lets agents work together. For example, a "Researcher" agent can scrape the web and store embeddings. A "Writer" agent then mounts that workspace to query them. This lets each agent do its job without waiting on the other.