Researching and Improving AI Agents for Biology
OncologyBench: Benchmarking multiagent AI systems on grounded systems in oncology research and treatment
Preprint · OncologyBench25 evals · 40.0% top scoreRead paper
Agentic AI is currently being deployed across oncology by patients, caregivers, researchers, and healthcare providers to compress analysis timelines, increase decision confidence, and provide reasoning support. The promise of agentic AI is clear, but general deployment requires trusted evaluation benchmarks that mimic the clinical journey. We introduce OncologyBench, a verifiable benchmark of 25 multistep-oncology evaluations that tests the ability of 31 AI model-harness combinations. Each evaluation is built from a set of deterministically graded analytical questions (Tasks), that form the input data to make a final clinically relevant conclusion (Decision), in prose, that is graded by a rubric. The data used for these evaluations is from publicly available manuscripts, case studies, or databases. In addition, we utilized a consented patient case study data for the express purpose of benchmarking. Evaluations span multiple decision categories that may occur along a clinical oncology journey. To replicate this, multiple agents work through the Tasks in parallel and their outputs are provided to a final agent to reason over and make a Decision. Our results indicate, highest performance scores for Opus 5 & 5.5, Sonnet 5.5 at 29–40% and followed by Kimi K3 at 24% among the 31 AI model-harness combinations that we tested. We found that the models with the strong interpretation and analysis skills still showed weakness in reasoning. In a separate experiment we found decision and answer quality varied depending on the persona of the question asker. With the known limitations of our work, we believe our current Oncology Benchmark effectively measures the performance of these frontier AI models being utilized by different personas.