-
Benchmarking AI Agents for Addressing Scientific Challenges Across Scales
Authors:
Tianyu Liu,
Allen Xin Wang,
Antonia Panescu,
Lisa Xinyi Chen,
Wenxin Long,
Xinyu Wei,
Yueqian Jing,
Ziyao Zeng,
Jihang Chen,
Sihan Jiang,
Ziqing Wang,
Siyi Gu,
Siyu Chen,
Xinyang Hu,
Haoran Shao,
Leqi Xu,
Wangjie Zheng,
Zhiyuan Cao,
Ada Fang,
Botao Yu,
Kunyang Sun,
Rex Ying,
Arman Cohan,
Qingyu Chen,
Lingzhou Xue
, et al. (8 additional authors not shown)
Abstract:
AI agents are increasingly being developed to accelerate scientific discovery, yet their practical capabilities in real research settings remain poorly understood. Existing benchmarks for AI agents rarely capture the complexity, heterogeneity, and extended reasoning required by scientific work, whereas benchmarks for scientific tasks often reduce research to static, direct problems and provide lim…
▽ More
AI agents are increasingly being developed to accelerate scientific discovery, yet their practical capabilities in real research settings remain poorly understood. Existing benchmarks for AI agents rarely capture the complexity, heterogeneity, and extended reasoning required by scientific work, whereas benchmarks for scientific tasks often reduce research to static, direct problems and provide limited support for interactive evaluation. Here, we introduce SciAgentArena, a systematic benchmark for evaluating AI agents in real-world scientific research scenarios drawn from emerging needs across multiple domains. SciAgentArena comprises approximately 200 tasks with stepwise verification and an interactive, agent-agnostic environment for assessing diverse AI agents. Using this benchmark, we find that current agents can contribute effectively to well-specified data-analysis workflows, particularly when the task structure and evaluation criteria are clear. However, their performance remains uneven across scientific contexts: agents struggle to generate genuinely novel insights, sustain self-directed exploration, and formulate robust solutions for open-ended research questions. We further characterize common failure modes across agents and identify opportunities for improving their reliability, autonomy, and scientific reasoning. Together, SciAgentArena provides a practical framework for measuring progress in AI agents for science and for guiding the design of future agents capable of addressing complex scientific challenges. Full codes, tasks, and datasets can be accessed via this link: https://sciagentarena.github.io/.
△ Less
Submitted 10 June, 2026;
originally announced June 2026.
-
Zatom-1: Towards a Multimodal Foundation Model for 3D Molecules and Materials
Authors:
Alex Morehead,
Miruna Cretu,
Antonia Panescu,
Rishabh Anand,
Maurice Weiler,
Tynan Perez,
Samuel Blau,
Steven Farrell,
Wahid Bhimji,
Anubhav Jain,
Hrushikesh Sahasrabuddhe,
Pietro Lio,
Tommi Jaakkola,
Rafael Gomez-Bombarelli,
Rex Ying,
N. Benjamin Erichson,
Michael W. Mahoney
Abstract:
General-purpose 3D modeling in chemistry encompasses molecules and materials, requiring both generative and predictive capabilities. However, most existing AI approaches are optimized for a single domain (molecules or materials) and a single task (generation or prediction), which limits representation sharing and transfer. We introduce Zatom-1, a cross-domain, general-purpose model architecture th…
▽ More
General-purpose 3D modeling in chemistry encompasses molecules and materials, requiring both generative and predictive capabilities. However, most existing AI approaches are optimized for a single domain (molecules or materials) and a single task (generation or prediction), which limits representation sharing and transfer. We introduce Zatom-1, a cross-domain, general-purpose model architecture that unifies generative and predictive learning of 3D molecules and materials. Zatom-1 is a deliberately simplified Transformer trained with a multimodal flow matching objective that jointly models discrete atom types and continuous 3D geometries. This approach supports scalable pretraining with predictable gains as model capacity increases, while enabling fast and stable sampling. We use cross-domain generative pretraining as a universal initialization for downstream multi-task prediction of properties, energies, and forces. Empirically, Zatom-1 outperforms or competes with specialized baselines on both multi-task generative and predictive benchmarks in data-controlled settings, while improving generative inference speed by more than an order of magnitude. Our experiments demonstrate positive predictive transfer between data domains from joint generative pretraining: modeling materials during generative pretraining improves molecular property prediction accuracy. Open-source code and model weights are freely available at https://github.com/Zatom-AI/zatom.
△ Less
Submitted 13 May, 2026; v1 submitted 24 February, 2026;
originally announced February 2026.
-
Advancing AI Research Assistants with Expert-Involved Learning
Authors:
Tianyu Liu,
Simeng Han,
Hanchen Wang,
Xiao Luo,
Pan Lu,
Biqing Zhu,
Yuge Wang,
Keyi Li,
Jiapeng Chen,
Rihao Qu,
Yufeng Liu,
Xinyue Cui,
Aviv Yaish,
Yuhang Chen,
Minsheng Hao,
Chuhan Li,
Kexing Li,
Yinsheng Lu,
Xinyu Wei,
Qinzhe Xing,
Antonia Panescu,
Mengbo Wang,
Vibha Annaswamy,
Alicia Sanchez,
Jack Cloherty
, et al. (5 additional authors not shown)
Abstract:
Large language models (LLMs) and large multimodal models (LMMs) promise to accelerate biomedical discovery, yet their reliability remains unclear. We introduce ARIEL (AI Research Assistant for Expert-in-the-Loop Learning), an open-source evaluation and optimization framework that pairs a curated multimodal biomedical corpus with expert-vetted tasks to probe two capabilities: full-length article su…
▽ More
Large language models (LLMs) and large multimodal models (LMMs) promise to accelerate biomedical discovery, yet their reliability remains unclear. We introduce ARIEL (AI Research Assistant for Expert-in-the-Loop Learning), an open-source evaluation and optimization framework that pairs a curated multimodal biomedical corpus with expert-vetted tasks to probe two capabilities: full-length article summarization and fine-grained figure interpretation. Using uniform protocols and blinded PhD-level evaluation, we find that state-of-the-art models generate fluent but incomplete summaries, whereas LMMs struggle with detailed visual reasoning. We later observe that prompt engineering and lightweight fine-tuning substantially improve textual coverage, and a compute-scaled inference strategy enhances visual question answering. We build an ARIEL agent that integrates textual and visual cues, and we show it can propose testable mechanistic hypotheses. ARIEL delineates current strengths and limitations of foundation models, and provides a reproducible platform for advancing trustworthy AI in biomedicine.
△ Less
Submitted 6 April, 2026; v1 submitted 3 May, 2025;
originally announced May 2025.