Papersby GlyphInto the Unknown: Benchmarking AI agents on open scientific research problems research paper by Carnegie Mellon University, 2026
Carnegie Mellon University · Agents and evaluation · Oct 13, 2026 · 62 upvotes · unverified
What it shows
This benchmark evaluates how well AI agents can write follow-up scientific research papers based on unpublished directions from domain experts.
By Seungone Kim, Jiayi Geng, Young-Jun Lee and 25 more · arXiv 2610.19159 · PDF · Code
UnverifiedSummarised by a model from the abstract; not yet checked by hand.
More from Carnegie Mellon University
All 11Random Is Almost Enough? Stable and Low-Cost Benchmark Compression via BestRandBestRand is a method for selecting small benchmark subsets that reliably estimate full evaluation scores at lower computational costs.
Carnegie Mellon University51 upvotesRead →Eat Your Precision: Precision Objective Mitigates Reward Hacking in Rubric-based RLFORK jointly optimizes recall and precision via rubric-driven necessity credits and factual grading to improve response quality and reduce verbosity in reinforcement learning.Carnegie Mellon University34 upvotesRead →X-Ragent: Tool-Augmented Reinforcement Learning for Evidence-Grounded Chest X-Ray ReasoningThis paper introduces a reinforcement learning framework for medical agents that selectively uses external evidence to improve diagnostic reasoning in chest X-rays.Carnegie Mellon University44 upvotesRead →Selection-Based Structured Reasoning: Toward Efficient Multimodal Search AgentsSSR improves multimodal agent efficiency by replacing free-form reasoning generation with a selection process from pre-specified reasoning candidates.Carnegie Mellon University37 upvotesRead →Predictive Credit: Measuring What Scientific Explanations Add to Experimental ForecastsThis paper introduces a protocol to measure how much scientific explanations actually improve the accuracy of experimental forecasts.Carnegie Mellon University103 upvotesRead →Training Object Permanence in World ModelsThis paper introduces a dataset and training stack to improve object permanence and physical reasoning in video world models.Carnegie Mellon University242 upvotesRead →JEV-as-a-Judge: Accept When Confident, Escalate When UnsureJEV-as-a-Judge is a cost-effective evaluation framework that uses a simple decision-only judge for easy cases and escalates complex ones.Carnegie Mellon University57 citations42 upvotesRead →Benchmark Radar: A Living Database and Search Engine for AI Benchmarks and EvaluationBenchmark Radar is a searchable living database and discovery engine for AI evaluation benchmarks that aggregates sources, score histories, and evidence to support benchmark selection and comparison.Carnegie Mellon University171 upvotesRead →
Other agents and evaluation papers
TopicGemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic CapabilitiesGemini 2.X model family, including Gemini 2.5 Pro and Flash, offers superior coding, reasoning, and multimodal understanding capabilities across a range of computational efficiencies.
Google4,594 citations67 upvotesRead →DeepSeek-V3.2: Pushing the Frontier of Open Large Language ModelsDeepSeek-V3.2 introduces DeepSeek Sparse Attention and a scalable reinforcement learning framework, achieving superior reasoning and performance compared to GPT-5 and Gemini-3.0-Pro in complex reasoning tasks.
DeepSeek887 citations275 upvotesRead →WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?WorkArena and BrowserGym evaluate large language model-based agents' ability to perform enterprise software tasks, revealing gaps in current agent capabilities and differences between open and closed-source LLMs.
ServiceNow443 citations2 upvotesRead →Aya Model: An Instruction Finetuned Open-Access Multilingual Language ModelAya, a multilingual generative language model supporting over 50% lower-resourced languages, excels in both generative and discriminative tasks across 99 languages, and provides extensive evaluations and open-source resources.
Cohere438 citations48 upvotesRead →Rubrics as Rewards: Reinforcement Learning Beyond Verifiable DomainsRubrics as Rewards (RaR) framework uses structured rubrics as interpretable reward signals for on-policy training, improving performance in real-world reinforcement learning tasks with subjective criteria.
Scale AI431 citations6 upvotesRead →SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?SWE-Bench Pro is a challenging benchmark for coding models, featuring complex, enterprise-level problems that require substantial code modifications, with performance evaluations showing significant limitations in current models.Scale AI367 citations21 upvotesRead →
About this paper
- Citations
- Not counted yet · Semantic Scholar
- Upvotes
- 62 · Hugging Face
- Lab
- Carnegie Mellon University
Changes
Hugging Face upvotes fell from 159 to 62.
This benchmark evaluates how well AI agents can write follow-up scientific research papers based on unpublished directions from domain experts.
Moved from Applied AI to Agents and evaluation.
Found by the weekly Hugging Face scan; not yet checked by hand.