NIST and Scale AI Launch DualChem Benchmark for Dual-Use Chemistry Risks

The National Institute of Standards and Technology (NIST) and Scale AI have released DualChem, a benchmark for evaluating how artificial intelligence models respond to chemistry questions that may have both beneficial and harmful applications. Developed through a Collaborative Research and Development Agreement (CRADA) announced in February 2025, DualChem is the first benchmark produced through Scale AI’s partnership with NIST’s Center for Advancing Innovation and Standards for Super Intelligence (CAISSI). The announcement describes a research collaboration, not a procurement action: it provides no contract value, solicitation, new award, or specified follow-on vehicle.

DualChem is intended to help assess both model capability and safety in realistic, ambiguous situations. Earlier benchmarks, according to materials Scale shared with FedScoop, have focused on explicit harmful requests; DualChem instead examines questions where the potential use is less clear. The benchmark asks how much assistance a model provides a user who may have limited prior expertise, as well as whether the model’s safeguards identify and appropriately handle potential dual-use risk. As Scale’s global head of policy and government relations, Max Fenkell, told FedScoop, the effort gives government “another tool on how to assess AI risk.”

The benchmark contains 600 expert-curated questions spanning molecular design, synthesis procedures, and information management. Questions were reviewed by Ph.D.-level evaluators. Scale research lead Madhu Sehwag described the motivating issue as: “We wanted to study how do models handle dual-use scenarios in chemistry, and how much uplift that a novice can get by just using [a large language model] to produce something that potentially can be helpful.” Here, “uplift” means the additional assistance a model provides beyond a user’s existing knowledge. Sehwag gave protein synthesis as an illustrative type of question, noting that a process could be used to create either a beneficial or harmful material.

For GovCon teams, DualChem is a signal of federal interest in practical methods for AI model evaluation, risk measurement, and safety testing—not evidence that NIST has established a compliance requirement or is seeking vendors through a new procurement. Scale reported evaluating roughly 30 frontier models against the benchmark. Those models were generally about 50% accurate on dual-use chemistry questions, with top-performing models at roughly 67% accuracy. The company also reported significant variation between model assessments and expert judgments of question severity; even when models identified a query as potentially dual-use, they still answered in most scenarios. Contractors working in AI assurance, evaluation, red teaming, and safety research can use the benchmark as an example of the kinds of test design and risk questions federal research may examine, while avoiding claims that DualChem is a government-mandated standard.

• NIST CAISSI and Scale AI developed DualChem under a CRADA announced in February 2025; no contract value or procurement vehicle was reported.
• The benchmark includes 600 questions across molecular design, synthesis procedures, and information management.
• Questions were curated by experts and reviewed by Ph.D.-level evaluators to assess performance and safety in ambiguous dual-use cases.
• The benchmark focuses on realistic chemistry questions with possible beneficial or harmful applications, rather than only explicit harmful requests.
• Scale evaluated roughly 30 frontier models; reported results were generally around 50% accuracy, with top models at roughly 67%.
• Scale reported variation between model assessments and expert judgments of question severity, and said models often answered even after flagging potential dual-use risk.
• AI evaluation, safety-testing, and assurance contractors can use DualChem as a reference for emerging government research priorities, but the release does not establish a mandatory standard or compliance rule.
• No solicitation, follow-on opportunity, or new award was specified; procurement teams should treat the release as a research and market signal rather than an active bid opportunity.