Programmatic Data Ingestion: Clay API File Upload Guide

According to McKinsey research, approximately 33% of tasks within sales and sales operations can be automated using existing technology [McKinsey Sales Automation Report]. Despite this massive automation potential, manual data transfer and fragmented pipelines still slow down go-to-market execution. In high-volume outbound prospecting, sales teams must import lead lists containing thousands of contacts from public registries, web scrapers, and customer databases.

When implementing these workflows, developers often run into a common architectural challenge: standard API requests designed for single-row operations fail to scale. Sending individual HTTP POST requests for thousands of leads quickly triggers rate limits, consumes API credits, and causes timeout errors. A standard single-row enrichment execution takes too long and wastes resources.

To address these limitations, programmatic systems must use bulk ingestion. The Clay API file upload is a programmatic method for bulk-ingesting large JSONL datasets into Clay tables using presigned PUT URLs or the Clay CLI. Rather than sending individual data rows one by one, developers compile lead records into a single file and upload it in a single operation. This programmatic bulk transfer is standard practice for modern sales operations, enabling teams to initiate large enrichment routines without running into rate limits or execution timeouts.

JSONL (JSON Lines) formatting is standard for batch go-to-market processing. Unlike a standard JSON array, which requires parsing the entire file into memory at once, a JSONL file contains independent JSON objects on each line. This format allows ingestion engines to stream and process records line-by-line. If a single line contains formatting errors, the remaining lines can still load successfully, making the ingestion pipeline stable and predictable. By using JSONL files, developers can easily format data collected from custom lead scraping agents or automated forms, preparing it for direct import into enrichment routines. For broader strategic context on RevOps planning, teams can review the McKinsey Sales Report.

To upload a JSONL file to a routine in Clay, you must first obtain a temporary upload destination. The Clay API does not accept large files directly on its main processing endpoints. Instead, it provides a dedicated endpoint that issues a presigned PUT URL pointing to secure object storage.

The developer makes a POST request to the routine upload-url endpoint. The HTTP request must include the target routine identifier in the URL path and your personal API key in the authorization header.

Step 1: Request the Presigned URL

The first step in the programmatic upload pipeline is making a POST request to the routine's upload-url endpoint.

curl -X POST "https://api.clay.com/public/v0/routines/rt_lead_enrich_01/run-batch/upload-url" \
  -H "clay-api-key: clay_key_prod_8989" \
  -H "Content-Type: application/json"

The response is a JSON object containing two fields: the presigned PUT upload URL and a unique file identifier.

Expected JSON Response Format

{
  "upload_url": "https://clay-user-uploads.s3.amazonaws.com/uploads/file_98765abcd?AWSAccessKeyId=AKIAIOSFODNN7EXAMPLE&Signature=vjbyPxybdZaNmGa%2ByT272YEAiv4%3D&Expires=1700000000",
  "file_id": "file_98765abcd"
}

The upload_url is a secure, expiring S3 URL authorized specifically for a PUT upload. The file_id is an opaque tracking string that represents the uploaded resource within the workspace. You will need this identifier in the subsequent step to trigger the actual batch routine. By separating the upload destination request from the file transfer, the API keeps its core endpoints free from processing heavy file uploads, maintaining low latency for other synchronous API operations. This architectural separation prevents network congestion on the primary API endpoints when multiple developers trigger bulk transfers simultaneously.

With the presigned upload URL and file identifier in hand, the next phase is executing the actual file transfer. Because the upload URL is presigned for a PUT method, you perform a direct binary upload without needing to supply your Clay API key in the request headers for this specific transfer. The S3 endpoint verifies the signature query parameters in the URL to authorize the upload.

Step 2: Upload the JSONL File

You can execute this upload from a terminal or a script by sending a PUT request with the file contents. In curl, the -T flag uploads the file as binary data.

curl -X PUT -T company_leads.jsonl \
  "https://clay-user-uploads.s3.amazonaws.com/uploads/file_98765abcd?AWSAccessKeyId=AKIAIOSFODNN7EXAMPLE&Signature=vjbyPxybdZaNmGa%2ByT272YEAiv4%3D&Expires=1700000000"

Once the upload command finishes with a successful status (HTTP 200 OK), the JSONL data resides in Clay's staging bucket. The file is now ready for processing, but the enrichment routine has not yet started.

Step 3: Trigger the Batch Run

To start the batch enrichment, you must send a final POST request to trigger the routine, passing the file identifier in the JSON request body.

curl -X POST "https://api.clay.com/public/v0/routines/rt_lead_enrich_01/run-batch/start" \
  -H "clay-api-key: clay_key_prod_8989" \
  -H "Content-Type: application/json" \
  -d '{
    "file_id": "file_98765abcd",
    "webhook_id": "wh_lead_completed_55"
  }'

The optional webhook_id parameter specifies a destination URL where Clay will send a notification when the batch run completes. This approach avoids the need to poll the API for updates, allowing your application to remain passive until the enrichment data is ready.

The following Node.js script coordinates this three-step workflow, automating the request for the upload URL, the PUT upload of the JSONL file, and the batch execution request.

import fs from 'fs';
import fetch from 'node-fetch';
async function ingestClayBatch(routineId, filePath, apiKey, webhookId) {
  const headers = {
    'clay-api-key': apiKey,
    'Content-Type': 'application/json'
  };
  try {
    const urlResponse = await fetch(
      `https://api.clay.com/public/v0/routines/${routineId}/run-batch/upload-url`,
      { method: 'POST', headers }
    );
    if (!urlResponse.ok) throw new Error('Failed to fetch upload URL');
    const { upload_url, file_id } = await urlResponse.json();
    const fileStream = fs.createReadStream(filePath);
    const stats = fs.statSync(filePath);
    const putResponse = await fetch(upload_url, {
      method: 'PUT',
      headers: { 'Content-Length': stats.size.toString() },
      body: fileStream
    });
    if (!putResponse.ok) throw new Error('Failed to upload file to storage');
    const startResponse = await fetch(
      `https://api.clay.com/public/v0/routines/${routineId}/run-batch/start`,
      {
        method: 'POST',
        headers,
        body: JSON.stringify({ file_id, webhook_id: webhookId })
      }
    );
    if (!startResponse.ok) throw new Error('Failed to start batch execution');
    const runResult = await startResponse.json();
    return runResult;
  } catch (error) {
    console.error('Batch ingestion error:', error.message);
    throw error;
  }
}

By consolidating these operations into a single script, developers can integrate lead enrichment directly into their internal GTM tools.

Coordinate your Clay API uploads in Fastio workspaces

Store your lead files and outputs in a shared workspace with automatic versioning, full search indexing, and MCP endpoints. Get started with a 14-day free trial on our Starter plan at $29/mo, Business plan at $99/mo, or Growth plan at $299/mo.

Start 14-Day Trial

For developers who prefer terminal-based automation or cron jobs, the Clay CLI provides an alternate path for coordinating workflows. Once installed via npm, you authenticate your environment and check your connection using the identity command.

CLI Credentials and Identity Verification

clay login
clay whoami

The CLI is a JSON-first interface, meaning it returns structured JSON outputs that you can pipe into utilities like jq to parse identifiers and tokens. While the CLI is useful for managing webhooks, querying table schemas, and validating connection states, actual file uploads are still routed through S3-compatible URLs to handle large payloads efficiently.

When scaling data pipelines, timeout configurations are a critical factor. In Clay's routine architecture, bulk upload timeouts default to 600,000 milliseconds for batch execution [Clay Routine Documentation]. If your enrichment routine, API enrichment loops, or data transfers exceed this 10-minute threshold, the execution terminates automatically, resulting in incomplete datasets.

To prevent execution failures, developers must implement the following design practices:

• Splitting a large JSONL file into smaller segments ensures that individual executions complete well within the default millisecond timeout limit.
• Using webhooks instead of polling the results endpoint continuously saves bandwidth and processing power.
• Adjusting rate limits inside the routine prevents third-party APIs from stalling, avoiding retries that extend execution time.
• Tracking routine run statuses using the results endpoint helps log failed rows and identify performance bottlenecks.

For complete configuration details, refer to the Clay Routine Documentation.

Before you can run a bulk upload, you need a system to collect, verify, and store the raw lead files. Managing these files on local disks or temporary servers introduces security risks and synchronization issues, especially when multiple agents and developers work together.

For basic storage, organizations often turn to options like local file storage, Amazon S3, or Google Drive. However, these services operate as simple folders, requiring developers to write custom code to handle version history, search indexing, and team collaboration.

Fastio provides an intelligent workspace platform designed to coordinate these data pipelines. Instead of treating storage as simple cloud folders, Fastio workspaces serve as a shared collaboration layer where human GTM teams and autonomous agents work together. When a scraping agent writes raw JSONL lead files to a shared workspace, the file is automatically versioned, maintaining a detailed file version history that ensures concurrent agent edits remain auditable.

To make lead data useful before sending it to Clay, developers can use Metadata Views inside their workspace. Fastio's Metadata Views turn raw documents into a live, queryable database. Users describe the fields they want extracted in natural language, and the AI designs a typed schema (supporting Text, Integer, Decimal, Boolean, URL, JSON, and Date & Time). The platform scans files like PDFs, images, or presentations, extracts the target data, and populates a spreadsheet grid. This structured database is detailed at the Metadata Views product page.

By integrating Fastio workspaces with Clay enrichment pipelines, teams benefit from:

• Clean cloud import allows GTM teams to pull lead files from Google Drive, OneDrive, Box, or Dropbox via OAuth without local network I/O.
• Automated indexing enables hybrid search across filenames and document contents, combining full-text matching with semantic retrieval.
• Scoped API access and the Model Context Protocol (MCP) server let agents read, write, and query files programmatically, which is configured via the Fastio Storage for Agents interface.
• Secure ownership transfer permits external developers to build workspaces, configure schemas, and hand off control via a secure claim link.

Fastio plans fit any team size, starting with the Starter plan at $29/mo, the Business plan at $99/mo, and the Growth plan at $299/mo [Fastio pricing plans]. Every organization starts with a 14-day free trial that requires a credit card [Fastio free trial], allowing you to test the workflow engine, RAG chat, and Metadata Views. Compare details on the Fastio pricing page.