Papersby GlyphBeyond Static Leaderboards: Predictive Validity for the Evaluation of LLM Agents research paper by IBM, 2026
IBM · Agents and evaluation · Jun 18, 2026 · 2 citations · 42 upvotes · unverified
What it shows
Aggregate-score leaderboards in agent benchmarks fail to capture deployment-relevant dimensions and show rank instability, necessitating new evaluation frameworks based on predictive validity and out-of-distribution criteria.
By Dhaval C. Patel, Kaoutar El Maghraoui, Shuxin Lin and 58 more · arXiv 2606.19704 · PDF
UnverifiedHugging Face's summary; not yet checked by hand.
More from IBM
All 5Retrieval Augmented Generation CompositionalityThis study shows that individual RAG techniques do not always work well together and provides a framework to evaluate full retrieval pipelines.
IBM34 upvotesRead →Asking for What Was Never Requested: Horizontal and Vertical Proactivity in AgentsQ&D trains agents to proactively ask for information that is necessary for task completion but not explicitly requested by the user.IBM41 upvotesRead →From Static Templates to Dynamic Runtime Graphs: A Survey of Workflow Optimization for LLM AgentsLLM-based systems use executable workflows that interleave various computational components, with recent approaches organized by workflow structure determination timing and optimization dimensions.IBM34 citations57 upvotesRead →TOUCAN: Synthesizing 1.5M Tool-Agentic Data from Real-World MCP EnvironmentsToucan, a large publicly available tool-agentic dataset, enhances the performance of LLM agents by providing diverse, realistic, and complex multi-tool and multi-turn interactions.IBM69 citations29 upvotesRead →
Other agents and evaluation papers
TopicGemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic CapabilitiesGemini 2.X model family, including Gemini 2.5 Pro and Flash, offers superior coding, reasoning, and multimodal understanding capabilities across a range of computational efficiencies.
Google4,594 citations67 upvotesRead →DeepSeek-V3.2: Pushing the Frontier of Open Large Language ModelsDeepSeek-V3.2 introduces DeepSeek Sparse Attention and a scalable reinforcement learning framework, achieving superior reasoning and performance compared to GPT-5 and Gemini-3.0-Pro in complex reasoning tasks.
DeepSeek887 citations275 upvotesRead →WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?WorkArena and BrowserGym evaluate large language model-based agents' ability to perform enterprise software tasks, revealing gaps in current agent capabilities and differences between open and closed-source LLMs.
ServiceNow443 citations2 upvotesRead →Aya Model: An Instruction Finetuned Open-Access Multilingual Language ModelAya, a multilingual generative language model supporting over 50% lower-resourced languages, excels in both generative and discriminative tasks across 99 languages, and provides extensive evaluations and open-source resources.
Cohere438 citations48 upvotesRead →Rubrics as Rewards: Reinforcement Learning Beyond Verifiable DomainsRubrics as Rewards (RaR) framework uses structured rubrics as interpretable reward signals for on-policy training, improving performance in real-world reinforcement learning tasks with subjective criteria.
Scale AI431 citations6 upvotesRead →SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?SWE-Bench Pro is a challenging benchmark for coding models, featuring complex, enterprise-level problems that require substantial code modifications, with performance evaluations showing significant limitations in current models.Scale AI367 citations21 upvotesRead →
About this paper
- Venue
- arXiv.org
- Citations
- 2, 0 influential · Semantic Scholar
- Upvotes
- 42 · Hugging Face
- Lab
- IBM · on Companies · on Acquisitions · on Quarterly · on Paydays · on TechConf
Changes
Citations rose from 1 to 2.
A first count of 0 influential citations.
A first count of 1 citations.
Found by the weekly Hugging Face scan; not yet checked by hand.