Flowise Document Store Limits: Overcoming Upsert Ceilings for Large Corpora

The default Flowise document store limit is 50MB per uploaded file, governed by the FLOWISE_FILE_SIZE_LIMIT environment variable in Flowise server configurations. When teams attempt to ingest large enterprise document archives into visual flow pipelines, Node.js process memory exhausts long before hitting vector database quotas. Flowise document store limits are memory, file size (typically 50MB), and batch upsert constraints that restrict how many documents Flowise can parse and embed into vector stores during flow execution.

In Flowise, Document Stores provide a centralized data management interface for retrieval-augmented generation (RAG) applications. Instead of configuring separate document loaders, text splitters, and vector store nodes inside every individual chatflow, developers configure a shared Document Store as documented in Flowise Documentation. A Document Store accepts files or API data sources, partitions the raw text into manageable chunks using a text splitter, passes those chunks to an embedding model, and upserts the resulting dense vectors into an external vector database such as Pinecone, Qdrant, or PostgreSQL pgvector.

While this architecture simplifies visual orchestration for smaller proof-of-concept projects, it encounters strict throughput and capacity limits when handling large document collections. The table below outlines documented file upload and ingestion limits across major AI interfaces as of November 2026:

Attaching large document libraries directly to conversational interfaces introduces practical capacity boundaries across every modern AI platform. Anthropic documents that Claude web chat accepts up to 20 files at up to 500MB each while Claude Projects accepts files up to 30MB each with no fixed file-count cap as detailed in Anthropic Claude upload specifications (checked on 2026-09-14). However, the practical ceiling on a Claude Project remains the context window. When an organization attempts to attach a multi-gigabyte corpus across dozens of contracts or product specifications, prompt tokens saturate and inference degrades. Flowise developers face an even earlier boundary: local server process memory and default container file size ceilings.

Most competitor guides focus exclusively on vector database configuration, such as tuning Pinecone pod dimensions or Chroma collection metadata, rather than addressing the initial document store parsing and payload bottleneck in Flowise. Before a single vector embedding reaches a database, Flowise must receive the file over HTTP, buffer the binary payload in memory, extract raw text, and split it into chunks. That upfront ingestion pipeline is where flows break down.

• CrewAI File Limits: Ingestion Caps and Large-Corpus ArchitectureBuilt-in file tools in CrewAI load raw text directly into agent working memory, creating an operational CrewAI file...
• CSV File Size Limits: AI Assistant Upload Caps, Parsing Ceilings, and FixesCSV file size limits vary from strict row boundaries in spreadsheets to memory and token constraints in AI chat...
• Google Gemini Document Upload Limits: 10-File Caps, Sizes, and WorkaroundsThe Google Gemini document upload limit restricts web interface users to a maximum of 10 attached files per prompt up...
• ChatGPT Projects File Limit: Plan Caps, Workarounds, and Large-Corpus SearchThe ChatGPT Projects file limit restricts workspace knowledge to 5 files on Free, 25 on Plus, and 40 on Enterprise,...
• Custom GPT File Limit: Knowledge Base Caps and Large-Corpus SearchOpenAI restricts Custom GPT Knowledge bases to a hard ceiling of 10 files, 512MB per file, and 2 million tokens per...
• Gemini Attachment Limit: File Caps and Multi-Document WorkaroundsThe Gemini attachment limit restricts uploads to 10 files per prompt (subject to availability) in the Gemini web and...

More on this subject: Agent File and Document Workflows (397 guides)

Flowise crashes when upserting large documents because the underlying Node.js V8 runtime runs out of memory during in-memory text parsing and unbatched vector transmissions. Flowise is built on Node.js and TypeScript, translating LangChain and LlamaIndex primitives into visual nodes. By default, the Node.js V8 engine enforces a strict heap memory limit, typically between 1.4GB and 2GB on standard deployments, and up to 4GB on 64-bit systems. When an administrator or API client uploads a massive document, the server attempts to execute binary extraction, text chunking, and embedding generation inside this single execution loop.

Two distinct bottlenecks cause Flowise instances to crash or hang during document store processing: process heap exhaustion during file parsing and gateway timeouts during vector upserts.

Node.js Heap Exhaustion During File Parsing

Flowise document loaders rely on in-memory JavaScript parsing libraries such as pdf-parse for PDF documents, text extractors for Microsoft Word files, and CSV parsers. When an incoming file arrives at the server, Flowise loads the entire binary buffer into system memory.

For a 400-page scanned PDF or a technical manual containing embedded vector graphics and tables, parsing requires unpacking compressed page streams and converting complex layout objects into plain text strings. This operation creates hundreds of thousands of intermediate string fragments and AST nodes. When parsing multi-hundred-page PDFs in Flowise worker threads, Node.js heap crashes are common. The V8 garbage collector cannot free intermediate string objects rapidly enough to keep pace with the allocation rate. As memory usage climbs toward the process boundary, the server halts with a fatal out-of-memory error:

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

In production architectures using background job workers (such as BullMQ backed by Redis) or containerized pods in Kubernetes, memory accumulation presents an additional challenge. Background workers retain session graphs and loader caches in memory across consecutive flow runs. Without scheduled worker recycling, memory consumption creeps upward until the host operating system terminates the container.

Vector Upsert Ceilings and Batching Constraints

Once text is extracted, Flowise splits documents into chunks and dispatches them to an embedding model before writing the resulting vector arrays to an external vector database. This stage introduces a secondary failure point: vector upsert payload limits.

Every vector database enforces payload and batch thresholds on its write APIs. For instance, Pinecone enforces a maximum request payload size of 2MB or 1,000 vector records per request. When a large document produces 3,000 or 5,000 chunks, attempting to upsert all vectors in a single HTTP request exceeds the database gateway limit. Even when a database accepts larger payloads, generating thousands of embeddings sequentially inside an active canvas run causes the HTTP connection to stall.

If an upsert request exceeds reverse proxy or browser timeout limits (typically 60 to 120 seconds), the client receives an HTTP 504 Gateway Timeout error. Although the background process might still be running, the visual interface loses synchronization, leaving the Document Store in an incomplete or corrupted state. To prevent these failures, the recommended vector upsert chunk batch limit is 100 to 500 records. Partitioning upserts into batches of 100 to 500 records keeps network payloads well beneath gateway ceilings and allows the event loop to yield between network dispatches.

Administrators running self-hosted Flowise instances can raise default file size ceilings, expand Node.js heap space, and tune text splitter parameters to handle moderately larger files. While these configuration changes do not transform Flowise into an enterprise data warehouse, they prevent immediate crashes when processing documents between 50MB and 100MB.

Configuring FLOWISE_FILE_SIZE_LIMIT

Flowise restricts incoming file sizes across document loaders, chat attachments, and Document Store uploads through the FLOWISE_FILE_SIZE_LIMIT environment variable. To increase this threshold, define the variable in your .env configuration file or container environment:

PORT=3000
FLOWISE_FILE_SIZE_LIMIT=100mb

When deploying Flowise through Docker Compose, pass the environment variable directly within the service specification:

version: '3.8'

services:
  flowise:
    image: flowiseai/flowise:latest
    restart: always
    environment:
      - PORT=3000
      - FLOWISE_FILE_SIZE_LIMIT=100mb
      - NODE_OPTIONS=--max-old-space-size=4096
    ports:
      - "3000:3000"
    volumes:
      - ~/.flowise:/root/.flowise

Expanding Node.js Heap Allocation

Increasing FLOWISE_FILE_SIZE_LIMIT without expanding process memory accelerates heap crashes. To allow the V8 engine to allocate additional memory before invoking the garbage collector, set the NODE_OPTIONS environment variable with --max-old-space-size:

export NODE_OPTIONS="--max-old-space-size=4096"

Setting --max-old-space-size=4096 expands the heap limit to 4GB, which provides sufficient overhead for parsing moderate-sized PDF files and managing intermediate chunk arrays. On dedicated virtual machines with 16GB or more of RAM, you can increase this value to 8192 (8GB).

Adjusting Text Splitter Parameters

Oversized chunk configurations degrade retrieval accuracy and create bloated vector payloads. In the Flowise Document Store interface, configure the attached Text Splitter (such as the Recursive Character Text Splitter) with standard production parameters:

• Chunk Size: Set to 1000 characters. This produces concise passages that fit comfortably within embedding model context limits without truncating key information.
• Chunk Overlap: Set to 200 characters. Overlapping consecutive chunks preserves semantic continuity across section boundaries.

Keeping chunk sizes around 1000 characters avoids generating single vectors that represent sprawling, unfocused text blocks.

Reverse Proxy and Ingress Timeouts

If Flowise runs behind an Nginx reverse proxy, Cloudflare tunnel, or AWS Application Load Balancer, the proxy will terminate uploads that exceed its default client body size or timeout window. Update your Nginx server block to accommodate the expanded boundaries:

server {
    listen 80;
    server_name flowise.internal.example;
    client_max_body_size 100M;
    location / {
        proxy_pass http://localhost:3000;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;
        proxy_read_timeout 300s;
        proxy_connect_timeout 300s;
        proxy_send_timeout 300s;
    }
}

Configuring client_max_body_size 100M; and extending read timeouts to 300 seconds prevents premature HTTP 413 (Payload Too Large) and HTTP 504 (Gateway Timeout) disconnections during long-running upsert jobs.

Scale Flowise Retrieval Across Large Document Collections

Connect your Flowise agents to persistent workspaces that keep company files indexed and searchable via remote MCP, avoiding local payload crashes. Monthly plans start with a 14-day free trial that requires a credit card.

Start 14-Day Trial

Raising environment variables and heap allocations helps individual files pass through Flowise, but it introduces severe architectural friction as document libraries scale into thousands of files. Teams managing corporate knowledge bases face an architectural choice: should Flowise function as both the ETL data pipeline and the agent runtime, or should ingestion live in an external indexed workspace?

The Hidden Costs of Canvas-Based Document ETL

Using Flowise Document Stores as the primary ingestion engine for enterprise repositories introduces three operational liabilities:

• Compute and Memory Contention: When a Flowise instance performs OCR, PDF parsing, and text splitting on the same machine that serves live agent chats, upsert jobs starve interactive users of CPU cycles and memory. A single heavy document upload can degrade response latency across all active chatflows.
• Re-Embedding Costs on Pipeline Edits: If you adjust your chunking strategy, switch embedding providers, or reorganize vector metadata in a Document Store, Flowise must re-read, re-chunk, and re-embed the source documents. For a corpus of several thousand files, re-indexing costs hundreds of dollars in model API fees and requires hours of compute time.
• Synchronization Latency: Documents in corporate environments change continuously. Flowise document stores do not continuously track file modifications in Google Drive, Dropbox, Box, or OneDrive. Updating a changed document requires manual re-upload or custom API scripting to purge stale vectors and insert replacement records.

Decoupling File Ingestion from Agent Graph Execution

A more resilient architecture decouples storage and indexing from agent flow execution. Instead of uploading raw files into Flowise and forcing Node.js workers to parse binary buffers, organizations maintain their document corpus in an external intelligent workspace.

In this decoupled model, the document corpus resides inside an intelligent workspace on Fastio Workspaces. Files enter the workspace through direct upload or through Cloud Sync, which connects Google Drive, Dropbox, Box and OneDrive folders to a workspace, one-way or two-way, on a schedule or on demand. When Intelligence Mode is enabled on the workspace, incoming documents are automatically indexed with hybrid search, combining full-text search, semantic search, and search-by-metadata-value.

Fastio leaves every vendor's own upload limit exactly where it is; what it adds is a searchable place for the files that do not fit into local canvas memory. Flowise agents do not ingest, parse, or chunk the source files. Instead, the agent connects to the workspace over the Model Context Protocol (MCP) and queries pre-indexed document passages via remote tool calls. This pattern eliminates Node.js heap exhaustion, avoids vector database payload limits, and delivers relevant citations directly to the agent. For agent runtime integration details, explore the Fastio Agent Guide.

Connecting Flowise to an external document index through the Model Context Protocol allows agent flows to query multi-gigabyte document archives without parsing binary files locally. Flowise provides a Custom MCP tool node that connects to remote MCP servers using Streamable HTTP.

Follow this procedure to connect your Flowise agent to a pre-indexed Fastio workspace:

Step 1: Ingest and Index Documents in Fastio

1. Create a workspace in Fastio and enable Intelligence Mode in workspace settings. Intelligence Mode automatically creates hybrid search indexes for all uploaded documents.
2. Populate the workspace with your reference documents. You can upload large archives directly or configure Cloud Sync to connect existing Google Drive, Dropbox, Box, or OneDrive folders.
3. Verify that documents are indexed by submitting a test search query in the Fastio interface to confirm semantic retrieval.

Step 2: Generate an API Key for Agent Access

Flowise connects to remote services as an autonomous server client. Generate an API credential with access to your target workspace:

1. In Fastio, open your organization settings and select API Keys.
2. Create a new scoped API key with read permissions for the target workspace.
3. Copy the generated key. You will pass this token in the HTTP Authorization header from Flowise.

Step 3: Add the Custom MCP Tool in Flowise

1. Open your Flowise dashboard and create or open an agent canvas (such as a Tool Agent or ReAct Agent).
2. In the node palette on the left, open the Tools category and drag the Custom MCP node onto the canvas.
3. Configure the Custom MCP node parameters:
   • Server URL: Enter https://mcp.fast.io/mcp/tools. Fastio hosts its remote MCP server over Streamable HTTP at this endpoint for general agent runtimes and tool clients. Complete setup instructions are available in the Fastio MCP Documentation.
   • Custom Headers: Provide your authentication header as a JSON object:

     {
       "Authorization": "Bearer YOUR_FASTIO_API_KEY"
     }
    

• Transport Protocol: Select Streamable HTTP. Do not configure Server-Sent Events (SSE) or local CLI bridges; the endpoint speaks Streamable HTTP directly.

Step 4: Attach the Tool to Your Agent Flow

1. Connect the output of the Custom MCP tool node to the tools input handle of your Agent node.
2. Attach a conversational model and a memory buffer to the agent node.
3. Save the chatflow. When the flow initializes, Flowise queries https://mcp.fast.io/mcp/tools to discover available workspace tools, exposing the consolidated MCP toolset.

Step 5: Test Semantic Retrieval Without Local Ingestion

Open the Flowise chat preview and query your document collection:

1. Submit a question requiring specific information from your document corpus, such as querying technical specifications, policy rules, or contract terms.
2. In the thought trace, observe the agent invoking the storage tool with the search action.
3. The remote MCP server executes a hybrid search across the pre-indexed workspace and returns ranked text passages with document titles and path citations.
4. The agent synthesizes an accurate answer grounded in your files, completing the retrieval turn without loading a single binary file into the Flowise Node.js process.

Review Fastio Pricing for deployment options. Monthly plans start with a 14-day free trial that requires a credit card. The table below details resource allowances across plans:

By offloading document storage, parsing, and vector indexing to an external workspace, your Flowise deployment remains lightweight, responsive, and completely immune to local document store size limits.