eDiscovery Data Processing: Architecture, Workflows, and Load File Staging

Dumping raw forensic disk images and multi-gigabyte email archives straight into a legal review platform guarantees two outcomes: review bills that spiral out of control and reviewers bogged down by operating system binaries and duplicate email threads. The engineering discipline that prevents this breakdown is eDiscovery data processing, the automated pipeline that extracts, deduplicates, filters, and structures electronically stored information before any attorney reads a single document.

In technical terms, eDiscovery data processing is the automated stage in the litigation lifecycle where collected raw electronic files are ingested, unpacked from containers, de-duplicated, filtered against known file hash lists, and extracted into searchable text and metadata for review. Within the Electronic Discovery Reference Model (EDRM), processing bridges the operational gap between Collection and Review. While collection focuses on acquiring forensically sound copies of custodian data without altering timestamps, processing normalizes, culls, and indexes those records for legal evaluation.

To execute defensible data reduction and prepare files for legal assessment, eDiscovery data processing follows a structured five-step pipeline:

1. Ingestion and Container Extraction: Unpacking archive formats, email databases, and compound document files to isolate individual records while preserving parent-child family relationships.
2. DeNISTing and File Filtering: Comparing cryptographic hashes against the National Institute of Standards and Technology reference data set to eliminate non-evidentiary system files, followed by applying agreed-upon file type filters.
3. Deduplication: Computing cryptographic hashes across files and normalized email components to suppress redundant copies globally or within individual custodian collections while cataloging all associated custodians.
4. Text Extraction and Optical Character Recognition (OCR): Extracting Unicode and ASCII text streams, unearthing hidden document metadata, and generating searchable text layers from scanned images and flattened PDFs.
5. Metadata Indexing and Load File Staging: Capturing standard electronic file attributes, assigning control numbers, establishing relational database pointers, and generating standard load files for review environments.

Without automated processing, legal teams face massive overhead during linear document review, where attorneys evaluate records one by one for relevance and privilege. Because legal review represents the largest operational expenditure in modern civil litigation, every extraneous system file, duplicate email chain, or unindexed scanned document that enters the review platform directly inflates case expenses.

The architectural objective of eDiscovery data processing is turning unstructured custodian data into a predictable, queryable database. By unpacking complex container formats, isolating individual messages, and generating structured metadata fields, litigation support teams create a clean foundation for early case assessment and attorney review.

• Casepoint eDiscovery: Cloud Platform Review, Legal Hold, and Review ToolsEvaluating enterprise eDiscovery software requires assessing how well preservation workflows bridge into document...
• DISCO eDiscovery Pricing: Ingestion Costs, Hosting Fees, and Review EconomicsDISCO eDiscovery pricing combines monthly gigabyte hosting fees, processed data measurements, and all-inclusive AI...
• Modern eDiscovery Tools: Technical Evaluation and Workflow GuideEvaluating modern eDiscovery tools requires understanding how software categories map across the Electronic Discovery...
• Litigation Management Software: Architecture and Workflows for Modern Legal TeamsLitigation management software centralizes matter intelligence, evidentiary documents, deposition transcripts, and...
• Nextpoint eDiscovery Review: Features, Pricing, and Litigation WorkflowsNextpoint is a cloud-based eDiscovery and litigation support platform that combines evidence processing, document...
• Reveal eDiscovery: Platform Review, AI Capabilities, and Review WorkflowsLitigation discovery bottlenecks rarely occur during document tagging alone; they emerge across the entire data...

More on this subject: Legal Teams (147 guides)

Electronically stored information rarely arrives as loose, individual documents. Corporate data collections arrive packed inside complex containers and archive formats. A single custodian collection might contain Microsoft Outlook databases (.pst and .ost files), Unix mailbox stores (.mbox), individual message files (.msg and .eml), Lotus Notes archives (.nsf), compressed file archives (.zip, .rar, .7z, .tar, and .gz), or raw forensic disk images.

Opening and expanding these containers requires specialized ingestion engines that unpack nested files without altering underlying timestamps or file properties.

Recursive Container Unpacking

Processing engines parse compound files recursively. Consider a standard corporate email thread: an executive sends an email message (Level 1 Parent) carrying an attached ZIP archive (Level 2 Child). Inside that ZIP archive sits an Excel workbook (Level 3 Grandchild), which itself contains an embedded Word document object and an attached image (Level 4 Great-Grandchildren).

A failure in recursive extraction at any intermediate layer risks omitting critical evidence from the litigation record. The processing engine must parse container headers, extract each embedded binary stream, assign a persistent tracking identifier, and write the record to a temporary working directory while maintaining the relational tree back to the top-level parent email.

When compound file expansion encounters password-protected archives, corrupted segments, or archive bombs designed to consume disk space, the engine routes the offending record to an exception log rather than terminating the entire processing queue.

Preserving Family Relationships

In civil litigation, documents do not exist in isolation. If an employee writes an email stating that the updated pricing structure is attached, the email and the attached spreadsheet form a single evidentiary unit known as a document family.

Preserving family integrity is mandatory under standard discovery protocol:

• If a child attachment is deemed responsive to a discovery request, the parent email must be reviewed alongside it to provide conversational context.
• If a parent email contains communications with outside legal counsel, privilege review must evaluate whether the legal privilege extends to the child attachments or whether the attachments must be produced while redacting the parent.
• If a child file contains sensitive non-party personal data, redactions must be applied without severing the link to the parent email.

Processing engines preserve these relationships by generating relational database keys during container expansion. The engine assigns a unique Document Control Number (DocID) to every item, a Family ID (FamilyID) shared across the parent and all descendants, a Parent ID (ParentID) linking each child directly to its immediate container, and an Attachment Range indicating the start and end control numbers of the complete family group.

True File-Type Identification via Header Inspection

Operating system file extensions are superficial labels that cannot be trusted during evidentiary processing. Custodians may rename files intentionally or accidentally, changing an executable (.exe) or database file (.sqlite) into a text file (.txt) or log file (.log). Furthermore, modern office formats like .docx, .xlsx, and .pptx are technically compressed ZIP containers containing XML structures.

Defensible processing engines ignore the file extension and inspect the binary file header, examining the initial bytes (known as magic numbers or file signatures) to determine true MIME types:

• Portable Document Format files begin with %PDF (hexadecimal 25 50 44 46).
• Modern Microsoft Office OpenXML files and standard ZIP archives begin with PK\x03\x04 (hexadecimal 50 4B 03 04).
• Legacy Microsoft Compound File Binary formats (DOC, XLS, PPT) begin with hexadecimal D0 CF 11 E0 A1 B1 1A E1.
• JPEG images begin with hexadecimal FF D8 FF.

By matching binary headers against an authoritative file signature dictionary, the processing system identifies the true nature of every file. If an extension mismatch is detected, the engine flags the file in the metadata log for forensic scrutiny while applying the correct extraction parser.

Collecting data from corporate laptops, cloud drives, and file servers sweeps up hundreds of gigabytes of non-evidentiary files. Operating system components, program files, font libraries, and identical email copies inflate storage footprints and create unbillable noise for legal reviewers. Automated data reduction eliminates this extraneous volume before human review begins.

DeNISTing and the NSRL Reference Data Set

DeNISTing is the automated process of identifying and removing known, non-evidentiary computer system and application files by matching their digital signatures against an authoritative reference repository. The governing standard is the National Software Reference Library Reference Data Set (RDS), maintained by the National Institute of Standards and Technology (NIST).

The NSRL RDS contains cryptographic hash values (specifically MD5 and SHA-1 signatures) for millions of known software files from commercial operating systems (Windows, macOS, Linux), desktop productivity suites, web browsers, and device drivers. The National Software Reference Library collects software profiles and incorporates them into a Reference Data Set of digital signatures to identify known files.

During ingestion, the processing engine calculates cryptographic hashes for each collected file and queries the NSRL database:

• If a file matches a known NSRL hash, such as an operating system DLL or standard desktop wallpaper, the engine flags it as a NIST hit and purges it from the review set.
• If a file does not match the NSRL database, it is retained for further review.

Because NSRL files originate from commercial software distributions and contain no user-created content or business communications, removing them carries zero spoliation risk.

File-Type and Date-Range Culling

Beyond DeNISTing, litigation teams apply agreed-upon culling criteria established during Rule 26(f) meet-and-confer negotiations:

• File-Type Filtering: Eliminating file extensions that are irrelevant to the dispute, such as audio files (.mp3, .wav), video streams (.mp4, .mov), or system log files, unless the litigation specifically concerns multimedia assets or IT infrastructure logs.
• Date-Range Culling: Restricting the data population to the operative timeframe of the dispute. Files with creation, modification, and email transmission timestamps falling outside the agreed-upon date window are isolated in an excluded partition.

Cryptographic Deduplication

Deduplication identifies identical electronic records to ensure that attorneys review each unique document only once.

For loose files such as standalone PDFs, Word documents, or spreadsheets, deduplication computes a cryptographic hash (MD5, SHA-1, or SHA-256) of the entire binary file. If two files share identical hashes, their contents are mathematically identical, even if they have different file names or reside in different directory paths.

Email deduplication requires normalized email hashing because raw email files carry unique transport headers (Received: lines) across recipients. The engine normalizes and concatenates key components:

• Sender email address
• Normalized recipient lists (To, CC, and BCC)
• Transmission date and time (standardized to UTC)
• Normalized subject line (stripping "RE:", "FWD:", and extra whitespace)
• Cryptographic hash of the email body text and attachment hashes

If this composite string matches an existing record in the database, the engine identifies the email as an exact duplicate.

Custodial Versus Global Deduplication

Litigation teams configure deduplication rules to operate at either the custodial level or the global level:

• Custodial Deduplication: Duplicates are eliminated only within each individual custodian collection. If Custodian A has three copies of the same email, two copies are suppressed. If Custodian B also received that email, Custodian B retains a copy.
• Global Deduplication: Only one instance of an identical document or email is retained across the entire matter, producing the lowest document count and minimizing review costs.

Preserving Custodian Mapping

When global deduplication suppresses a duplicate from Custodian C, counsel preparing for a deposition might mistakenly assume Custodian C never received the document. To maintain defensibility, processing software creates an ALL_CUSTODIANS or DUPLICATE_CUSTODIAN metadata field, appending Custodian C name, email address, and original file path to the master document record.

Email Threading and Near-Duplicate Analysis

Email threading groups related emails into structured conversational threads, identifying inclusive emails that contain all prior replies, forwards, and attachments. Reviewing only inclusive emails allows attorneys to evaluate complete conversational context while suppressing redundant intermediate messages. Near-duplicate analysis applies text similarity algorithms to identify draft contracts or modified agreements that share substantial text blocks, enabling consistent batch coding.

Making electronic records searchable requires extracting text layers and metadata attributes from binary file formats. In modern litigation, if text cannot be searched by keyword or indexed for concept clustering, it is practically invisible.

Direct Native Text Extraction

For digital-native documents (Word files, Excel spreadsheets, PowerPoint decks, HTML files, and text-based PDFs), processing engines extract the embedded Unicode (UTF-8, UTF-16) or ASCII text streams directly from the application data.

Native text extraction must capture both visible body copy and hidden document layers:

• Tracked changes, revisions, and inline editorial markups.
• Hidden worksheets, hidden rows, and hidden columns in financial spreadsheets.
• Underlying mathematical formulas in spreadsheet cells rather than calculated display values.
• Presenter notes, speaker commentary, and off-slide text in presentation decks.
• Hidden document properties, including author identities, company names, creation templates, and total editing time.

Failing to extract these hidden layers during processing can cause legal teams to overlook critical case facts or face court scrutiny under Rule 34 of the Federal Rules of Civil Procedure.

Optical Character Recognition (OCR)

Scanned paper documents, fax transmissions, flattened PDF agreements, and screenshot images (PNG, JPEG, TIFF) lack a native text layer. Processing engines pass image files through optical character recognition (OCR) pipelines:

1. Image Normalization: Deskewing rotated pages, removing digital speckles, and adjusting contrast to maximize character recognition accuracy.
2. Character Recognition: Converting graphical letterforms into machine-encoded text streams.
3. Text Layer Embedding: Generating a searchable text layer mapped directly to original page image coordinates, allowing reviewers to highlight search hits on the document viewer.
4. Confidence Scoring: Calculating recognition confidence metrics to flag low-resolution scans for manual inspection.

Defensible Exception Handling and Remediation

In any large-scale enterprise discovery project, a portion of files will fail automated processing due to encryption, corrupted binary headers, unsupported proprietary database formats, archive bombs, or zero-byte files.

When an exception occurs, the processing engine isolates the record in a defensible Exception Log documenting:

• Custodian name and device origin
• Original file path and file name
• Native file size and MD5 cryptographic hash
• Specific error code and failure description
• Processing timestamp

Legal operations teams review the exception log to interview custodians for missing passwords, deploy specialized extraction utilities, or document unrecoverable files to fulfill meet-and-confer disclosure duties under Rule 26(f).

Stage and organize litigation files before document review

Set up secure matter workspaces, ingest evidence with branded Receive links, and extract structured fields with Metadata Views. Monthly plans start with a trial of up to 30 days (credit card required); annual plans have no trial. Starter plans begin at $9.99 per month.

Start 30-Day Trial

The culmination of eDiscovery data processing is the generation of load files. A load file is a structured data package that bridges the processing engine and the legal review platform, providing the relational instructions required to display text, metadata, images, and native files in a unified review workspace.

The Anatomy of eDiscovery Load Files

Standard eDiscovery productions and internal review imports rely on three coordinated file formats:

1. Concordance .DAT Metadata File: A delimited text database where each row represents a distinct document record and each column represents an extracted metadata attribute. Standard legal databases use dedicated ASCII control characters to prevent data corruption. Concordance documents its defaults as follows:
   • Column Delimiter (Comma): ASCII character 020 (the pilcrow, ¶), marking the field break between columns.
   • Field Qualifier (Quote): ASCII character 254 (þ, Latin small letter thorn), wrapping fields that contain text and spaces.
   • New Line: ASCII character 174 (®), marking a manual line break inside a single field.
   • Record Separator: Standard carriage return and line feed (CRLF), with a final carriage return required to load the last record.
2. Opticon .OPT or .LFP Image Cross-Reference File: A comma-delimited text file that maps document Bates numbers to page-level TIFF or PDF image files. Each line specifies the Bates number, volume identifier, relative file path, document break indicator (Y denoting the initial page of a document; blank for subsequent pages), and page count.
3. Extracted Text Directory (TEXT): A folder structure containing individual .txt files named after document control numbers (such as DOC-0001042.txt). The .DAT file includes a pointer column linking each document record to its corresponding text file, enabling full-text search indexing.

Core Metadata Field Architecture

A defensible processing export captures system-level file attributes and conversational email properties in standardized database columns.

Pre-Review Staging: Controlling Costs Before Production

Pushing unprocessed or semi-processed data directly into enterprise document review platforms creates severe operational and financial inefficiencies. Linear review tools bill law firms and corporations on recurring per-gigabyte monthly hosting rates, active user seat licenses, and processing surcharges. Ingesting raw collections directly into review databases forces organizations to pay recurring hosting fees for irrelevant operating system files, junk data, and duplicate records.

To prevent these unnecessary expenses, legal operations teams establish a pre-review staging architecture. In this staging layer, discovery specialists validate load file integrity, test search term reports, sample responsiveness, and cull extraneous records before promoting clean datasets into linear review.

Modern cloud workspaces like Fast.io workspaces provide an ideal substrate for pre-review staging and solutions for legal teams:

• Direct Evidence Intake: Instead of relying on fragile FTP servers or insecure email attachments, litigation teams deploy Fast.io branded Receive shares. Corporate clients, custodians, and third-party forensic specialists can upload large disk images and archive containers directly into an org-owned workspace from a browser without creating an account.
• Cloud Import: Legal teams can pull discovery sets directly from Google Drive, Dropbox, OneDrive, Box, or public URLs. Cloud import transfers data directly across cloud infrastructure without downloading terabytes of evidence to local office workstations, eliminating bandwidth congestion and local disk constraints.
• Automated Workspace Intelligence and Hybrid Search: Enabling Intelligence Mode on a staging workspace indexes documents upon arrival for hybrid search, combining full-text and semantic search. Legal teams can test proposed keyword strings, identify over-inclusive terms, and evaluate conceptual groupings during early case assessment before committing data to review databases.
• Structured Extraction with Metadata Views: For specialized document sets such as contracts, invoices, or board minutes, teams use Metadata Views to turn documents into structured, queryable spreadsheets. Natural language prompts define typed schemas (extracting dates, counterparties, financial amounts, or Bates numbers) across PDFs, Word files, spreadsheets, and scanned pages without building brittle OCR templates.
• Defensible Custody and Access Control: Fast.io provides granular permissions at the organization, workspace, folder, and file levels. Per-file version history and an append-only audit log record every upload, access, and export event, preserving complete chain-of-custody documentation. Fastio runs on cloud infrastructure partners, including Google Cloud Platform and Cloudflare, that are certified to industry-leading security standards, with encryption in transit and at rest.

By establishing a disciplined pre-review staging workspace, legal teams eliminate data bloat, validate metadata consistency, and ensure that downstream attorney review focuses entirely on relevant, case-critical evidence.