Scientific peer review is increasingly challenged by the scale and complexity of modern research output, particularly in its ability to assess reproducibility. Evaluating reproducibility necessitates reconstructing experimental dependencies, methodological choices, data flows, and result-generating procedures, often exceeding the capacity of human reviewers.
Addressing this, Agentic Reproducibility Assessment (ARA) formalizes reproducibility assessment as a structured reasoning task over scientific documents. Given a research paper, ARA leverages AI agents to extract a directed workflow graph that links sources, methods, experiments, and outputs. It then evaluates the reconstructability of this workflow using both structural and content-based scores to provide a comprehensive reproducibility assessment.
The generalizability and consistency of ARA were demonstrated through experiments on 213 ReScience C articles, which represent the largest cross-domain benchmark of human-validated computational reproducibility studies to date. The system showed consistent workflow reconstruction and assessment across various Large Language Models (LLMs), model temperatures, and scientific domains. ARA achieved approximately 61% accuracy on three benchmarks. Notably, it reported the highest accuracy on ReproBench (60.71% versus 36.84%) and GoldStandardDB (61.68% versus 43.56%), highlighting its substantial potential to complement human review at scale and facilitate the evolution of next-generation peer review processes.
