NIST AI Safety Institute Super Intelligence Evaluation Standards: Federal Protocols

Clarifying the Institutional Structure Inside NIST

The AI Safety Institute was established within NIST, a bureau of the U.S. Department of Commerce, following presidential directives in late 2023 to evaluate frontier dual-use foundation models. In June 2025, the Department of Commerce announced that the institute would operate as the Center for AI Standards and Innovation (CAISI). While the administrative title was revised to emphasize technical innovation and voluntary industry collaboration, federal agency solicitations and technical practitioners continue to refer to its testing protocols as the NIST AI Safety Institute evaluation standards.

The institute operates under a specific institutional mandate. Unlike a standalone regulator, CAISI does not have rulemaking authority; NIST's traditional role is to develop voluntary standards, guidelines, and measurement science that industry and government can adopt. The institute does not issue commercial operating licenses, levy regulatory fines, or inspect commercial software repositories without consent. Instead, it serves as the federal government's technical measurement laboratory for advanced computing systems.

Its core operational responsibilities span five primary activities:

1. Empirical Capability Evaluations. Developing automated test suites and expert red-teaming methodologies to measure model capabilities and safety limits.
2. Voluntary Safety Standards. Formulating technical guidance, test benchmarks, and evaluation criteria in collaboration with academic and industry partners.
3. Pre-Deployment Lab Evaluations. Executing voluntary testing agreements with frontier model developers to evaluate advanced foundation models before general release.
4. Interagency Technical Support. Providing capability assessments and testing guidance to federal departments, including the Department of Defense (DOD), Department of Homeland Security (DHS), Department of Energy (DOE), and Health and Human Services (HHS).
5. International Standards Alignment. Coordinating testing methodologies with international partner bodies, such as the United Kingdom AI Security Institute (UK AISI).

Federal Super Intelligence Terminology Under Executive Order 14434

Federal procurement and compliance documentation underwent a terminology transition on September 29, 2026, when the President signed Executive Order 14434, titled "Inaugurating the Era of Super Intelligence". Understanding how this presidential directive interacts with technical evaluation standards requires distinguishing between administrative policy and computer science theory.

Under Section 2(a) of the order, executive departments and agencies are instructed to use the terms "Super Intelligence" and the abbreviation "SI" in place of "Artificial Intelligence" and "AI" across official correspondence, public communications, websites, reports, policy documents, and non-statutory federal publications.

Outside federal policy, the abbreviation SI designates the International System of Units (Système International d'Unités), the sacroiliac joint in orthopedic medicine, and Sports Illustrated in commercial journalism. Within federal executive communications, SI designates the administrative vocabulary for computing systems.

Technical leads and defense contractors must separate two independent concepts that share identical wording:

1. Federal Super Intelligence (SI). An administrative designation established by presidential executive order for existing computing systems operated across federal agencies. Under Section 3(a) of Executive Order 14434, the terms Super Intelligence and SI mean the technologies and systems covered by the statutory definition of artificial intelligence in 15 U.S.C. 9401(3). In federal administration, SI and AI name the exact same systems: statistical classifiers, predictive algorithms, and language models currently in active use.
2. Laboratory superintelligence (often abbreviated ASI). A theoretical computer science concept describing a hypothetical future system that would surpass the capabilities of the most capable human experts across nearly every field, generally situated beyond artificial general intelligence (AGI). Commercial laboratories such as Meta Superintelligence Labs, Microsoft's MAI Superintelligence Team, and Safe Superintelligence Inc. explicitly pursue this milestone as an engineering vision. This hypothetical software does not exist today.

Section 2(b) of Executive Order 14434 leaves previously issued regulations, existing Presidential actions, ongoing contracts, federal grants, and historical records unchanged. Under Section 3(b) of the order, the Assistant to the President for Science and Technology (APST) must submit proposed legislative language for a federal statutory definition of Super Intelligence to the President within 60 days of the order, which falls on November 28, 2026. This submission is an internal recommendation to the President, not an enacted law. As of October 2026, no statutory definition of Super Intelligence exists in the United States Code. Section 4(c) provides that Executive Order 14434 creates no right or benefit enforceable at law and imposes no mandates on private commercial enterprises.

Automated Benchmarking Suites vs. Expert Human Red Teaming

A central principle of the NIST AISI evaluation methodology is that automated benchmark test suites and human red teaming serve complementary functions. Automated benchmarks run standardized prompt sets through target models to produce quantitative scores across predefined metrics, testing syntactic safety compliance and factual accuracy against structured reference databases. Automated suites provide high reproducibility and immediate regression detection during fine-tuning. However, automated tests suffer from prompt saturation: once a test set is public, model alignment can be trained to pass those specific queries without resolving underlying vulnerabilities. Automated tests also cannot adapt dynamically to multi-turn model responses.

To address these limitations, AISI couples automated benchmarking with expert human red teaming. The institute deploys specialized teams composed of offensive cyber operators, molecular biologists, chemical defense experts, and adversarial machine learning researchers. These practitioners conduct interactive, multi-turn elicitation sessions using several advanced adversarial techniques:

• Adversarial Persona Framing. Constructing complex academic, defense, or historical research scenarios to test whether guardrail filters can be bypassed through hypothetical problem formulation.
• Multi-Turn Decomposition. Breaking prohibited requests into benign, isolated sub-queries across multiple conversational turns, then assembling the dangerous output without triggering automated safety classifiers.
• Obfuscation and Token Smuggling. Using base64 encoding, foreign languages, cipher substitutions, or symbolic logic representations to evade keyword filters and input monitors.
• Context Window Overloading. Flooding the context window with technical terminology, benign literature, or competing instructions to weaken system prompt constraints.

Measuring Dual-Use Uplift in Chemical and Biological Scenarios

Among the most sensitive evaluation domains overseen by the institute is the measurement of dual-use uplift in chemical, biological, radiological, and nuclear contexts. In safety science, dual-use uplift refers to the degree to which a computing system provides actionable, specialized assistance that measurably enhances an actor's ability to execute a dangerous task compared to conventional information sources.

Evaluating uplift requires moving beyond simple keyword matching. A model that recites publicly available historical information regarding chemical treaties or high-school biology principles poses minimal incremental risk. The institute's evaluation protocols focus instead on whether the model acts as an actionable force multiplier.

To measure uplift empirically, the institute uses controlled human-subject experimental designs:

1. Cohort Separation. Researchers establish a control group composed of individuals with relevant technical education who receive access to standard internet search engines, scientific textbooks, and academic journals. A matched treatment group receives access to the target foundation model.
2. Task Performance Testing. Both groups solve complex scientific challenges, such as troubleshooting laboratory synthesis workflows, identifying alternative chemical precursors that bypass commercial screening, or optimizing fermentation protocols.
3. Metric Tracking. Evaluators measure task completion time, protocol viability, troubleshooting efficiency, and the prerequisite expertise required to achieve successful outcomes.

If the cohort equipped with the model achieves viable results faster, identifies dangerous synthesis pathways previously inaccessible to non-experts, or successfully troubleshoots laboratory roadblocks, the model demonstrates measurable uplift. The institute coordinates directly with scientific national laboratories, including Lawrence Livermore, Los Alamos, and Sandia, as well as HHS and DOE, ensuring evaluation rubrics reflect physical laboratory realities.

Voluntary Testing Agreements and Pre-Deployment Access Protocols

The operational mechanism governing frontier model assessments rests on voluntary Memoranda of Understanding (MOUs) established between the AI Safety Institute and leading developers, including OpenAI and Anthropic. Under these bilateral agreements, developers grant AISI technical staff access to major new model checkpoints prior to their commercial release.

Pre-deployment testing protocols follow strict operational safeguards:

• Isolated Testing Enclaves. Evaluations are conducted within secure, isolated computing environments to protect proprietary model weights and prevent unauthorized data leakage.
• Direct Checkpoint Access. AISI researchers evaluate raw base models, intermediate fine-tuned checkpoints, and deployment-ready models equipped with system prompts, input filters, and output classifiers, assessing whether safety alignment has altered underlying capabilities or merely added filtering layers.
• Standardized Evaluation Batteries. Models are evaluated against comprehensive benchmark suites covering offensive cybersecurity capabilities, biological weaponization risks, autonomous agency, and persuasive influence.
• Collaborative Remediation. When testing uncovers critical vulnerabilities, AISI provides technical feedback so developers can retrain safety classifiers, reinforce alignment guardrails, or implement additional deployment mitigations before commercial release.

The institute also coordinates directly with the United Kingdom AI Security Institute, sharing adversarial testing prompts and conducting synchronized pre-deployment assessments to maintain technical consistency across allied jurisdictions.

Translating AISI Evaluations into Federal Acquisition and TEVV Pipelines

While frontier developers participate in direct pre-deployment testing with AISI, government contractors and enterprise software vendors face a different operational challenge: demonstrating that models integrated into federal agency workflows comply with government safety expectations.

Federal agencies do not conduct de novo foundation model evaluations for every software acquisition. Instead, departments such as the Department of Defense, Department of Homeland Security, and Department of Veterans Affairs integrate AISI measurement science into their existing Software Testing, Evaluation, Verification, and Validation (TEVV) lifecycles.

When preparing proposals or system authorization packages for federal agencies, contractors should anticipate four core evaluation requirements:

1. Model Provenance and Evaluation Cards. Documenting the foundation model's provenance, training cutoff, version identifier, and prior participation in recognized evaluation regimens, such as AISI testing or third-party audits.
2. Adversarial Robustness Testing. Providing test logs demonstrating resilience against prompt injection, data exfiltration, and tool hijacking, structured according to guidelines like NIST AI 100-2.
3. Domain-Specific Guardrail Verification. Demonstrating that application-level guardrails block policy violations and maintain factual grounding across sensitive agency workflows.
4. TEVV Lifecycle Alignment. Mapping technical deliverables to formal agency TEVV gates prior to operational fielding and securing an Authority to Operate (ATO).

Contractors must note that Executive Order 14434 applies directly to executive departments and agencies, mandating Super Intelligence terminology across non-statutory documents without compelling private vendors to alter commercial product names. If an agency solicitation contains ambiguous technical terminology, vendors should seek clarification directly from the contracting officer (CO) or their legal counsel.

Empirical Testing Metrics vs. Process Governance Registers

The fundamental difference between organizational frameworks and AISI evaluation standards lies in how risk is measured. Process governance relies on qualitative assertions, such as documenting that an engineering team conducts code reviews or maintains an acceptable use policy. Empirical evaluation standards rely on quantitative testing metrics that measure model behavior under stress.

When evaluating models for federal deployment, technical teams measure four specific quantitative indicators:

• Attack Success Rate (ASR). The proportion of adversarial prompts that successfully elicit prohibited content or bypass system safety guardrails across single-turn baseline attacks and multi-turn iterative jailbreaks.
• Dual-Use Uplift Delta. The quantitative performance gap between a control group relying on standard information resources and a treatment group operating the target model when attempting high-consequence scientific or technical tasks.
• Prompt Injection Vulnerability Rate. The frequency with which indirect prompt injection vectors, such as adversarial text embedded within ingested files or database records, successfully override system instructions and hijack agent tool execution.
• Calibration and Hallucination Error. The statistical divergence between a model's stated confidence score and its empirical factual accuracy across domain-specific benchmark queries.

Continuous Red Teaming and Post-Deployment Safety Monitoring

Passing an initial pre-deployment evaluation does not guarantee persistent system safety. In production federal environments, software systems evolve continuously: underlying models receive upstream checkpoint updates, application developers connect new external APIs and retrieval databases, and operational data distributions shift over time.

Federal evaluation guidelines emphasize that testing must continue throughout the software lifecycle. Continuous safety monitoring incorporates three operational practices:

1. Automated Canary Testing. Running automated adversarial evaluation suites against production API endpoints on a scheduled basis to detect regressions introduced by fine-tuning or upstream vendor updates.
2. Telemetry and Distribution Monitoring. Tracking input prompt distributions, output refusal rates, and tool execution logs to identify emerging attack patterns or anomalous user interactions.
3. Structured Incident Reporting. Establishing standardized procedures for logging and escalating security anomalies, prompt injection attempts, or unexpected model failures to system administrators and federal oversight personnel.

By combining pre-deployment evaluations with continuous operational telemetry, agencies and contractors maintain visibility over system behavior in live mission environments.