Papersby GlyphAgents and evaluation
Agents, harnesses, benchmarks and how to measure them. 147 papers from 25 labs.
RoboColosseum: Evaluating Physical Intelligence at Scale In-The-WildRoboColosseum is an evaluation framework that tests robot AI models across different environments and physical setups.
Hugging Face50 upvotesRead →WebHarbor: Docking Real Websites for Evolving GUI Agent EnvironmentsWebHarbor provides realistic, containerized website environments for training and testing computer-using web agents across complex online tasks.
Microsoft28 upvotesRead →Random Is Almost Enough? Stable and Low-Cost Benchmark Compression via BestRandBestRand is a method for selecting small benchmark subsets that reliably estimate full evaluation scores at lower computational costs.
Carnegie Mellon University51 upvotesRead →Co-Design Bench: Multimodal Collaborative Coding Evaluation for Agentic ModelsCo-Design Bench evaluates how effectively coding agents update software applications based on user requests with text, images, and other media.
Alibaba (Qwen)23 upvotesRead →UserArena: Benchmarking Interactive User Simulation for Long-Horizon Shopping AgentsUserArena introduces a real-world interactive benchmark with long-horizon shopping trajectories and an evaluation framework, alongside On-Policy Trajectory Distillation to improve simulator fidelity for agentic recommendations.Alibaba (Qwen)55 upvotesRead →Agent World Model in the Wild: Computer-Use Agent Environment SynthesisThis pipeline creates controllable, open-source environments to safely train and test AI agents on complex, long-horizon computer-use tasks.Microsoft69 upvotesRead →CLASHarness: Benchmarking Multi-Agent Conflict in Computer Use via a Controllable and Verifiable Generation PipelineThis paper introduces a benchmark and generation pipeline to evaluate how agents resolve conflicts and verify information in multi-agent computer-use scenarios.
Google36 upvotesRead →Into the Unknown: Benchmarking AI agents on open scientific research problemsThis benchmark evaluates how well AI agents can write follow-up scientific research papers based on unpublished directions from domain experts.Carnegie Mellon University62 upvotesRead →Harness^2: Recursive Agent Harnessing for an Open WorldThis paper introduces a recursive framework that allows agents to automatically improve their own execution harnesses for better performance on novel open-world tasks.Google2 citations123 upvotesRead →Scaling Agentic Data for Long-Horizon Terminal IntelligenceThis paper introduces a system to automatically generate high-quality training data for long-horizon terminal agents to improve their autonomous problem-solving capabilities.Alibaba (Qwen)1 citations147 upvotesRead →Stateless Language Agents: Scaling Long-Horizon Automated ResearchThis paper introduces a framework where agents remain stateless, allowing a central harness to manage research state and improve long-horizon task efficiency.
Stanford University1 citations30 upvotesRead →Just-in-Time Memory: Learning to Curate Task-Adaptive Memory for LLM AgentsKeeps an agent's raw past runs and writes a task-specific memory only when a new task arrives, rather than deciding up front what to keep.
Salesforce2 citations56 upvotesRead →JEV-as-a-Judge: Accept When Confident, Escalate When UnsureJEV-as-a-Judge is a cost-effective evaluation framework that uses a simple decision-only judge for easy cases and escalates complex ones.Carnegie Mellon University57 citations42 upvotesRead →The Tasteful Agent: Measuring and Improving Taste in Long-Horizon TasksTaste-Bench measures an agent's ability to make good long-horizon decisions by evaluating choices at critical decision forks in trajectories.Microsoft1 citations162 upvotesRead →HappyWorld-BenchHappyWorld-Bench evaluates the reliability and consistency of generated worlds as agents interact with them over time.Alibaba (Qwen)57 upvotesRead →OSWorld-Pro: Process-based Evaluation for Computer Use AgentsOSWorld-Pro provides a procedural evaluation framework for computer-use agents by breaking tasks into subgoals and tracking progress.
NVIDIA28 upvotesRead →RRSI: Regularized Recursive Self-Improvement of Agent HarnessesRRSI keeps self-improving agent harnesses from memorising their training tasks, so gains carry over to new benchmarks.Google32 citations221 upvotesRead →RecreationWorld: Scalable and Verifiable Environments for Hybrid Computer-Use AgentsRecreationWorld is a framework for training and evaluating hybrid computer-use agents that must discover and recreate software behavior.Alibaba (Qwen)2 citations78 upvotesRead →Verifiable Social Reasoning for LLM AssistantsFuse is a multi-agent simulation framework that allows for the verifiable evaluation of an assistant's social reasoning capabilities.Google2 citations67 upvotesRead →Root-Cause Attribution Is a Search Problem: Continual Search for Long-Horizon Agent FailuresContinual Search improves automated root-cause attribution in long agent execution traces by iteratively prompting diagnosis until unresolved evidence is found.
Scale AI5 upvotesRead →Benchmark Radar: A Living Database and Search Engine for AI Benchmarks and EvaluationBenchmark Radar is a searchable living database and discovery engine for AI evaluation benchmarks that aggregates sources, score histories, and evidence to support benchmark selection and comparison.Carnegie Mellon University171 upvotesRead →Studying Without a Syllabus: Task-Agnostic Environment PreprocessingAn agent can explore unfamiliar environments without task-specific guidance to build reusable artifacts that reduce later inference costs, though larger study budgets do not always improve results.Scale AI1 citations4 upvotesRead →WearableQA: A Benchmark for Health Reasoning over Real-World Wearable DataWearableQA is a benchmark of multiple-choice questions derived from real longitudinal wearable data that evaluates large language model reasoning across data and health dimensions.
Meta43 upvotesRead →EVOHARNESSBENCH: Can Your Agents Keep Pace with an Evolving Harness?The study introduces a benchmark to evaluate LLM agents under evolving tool, skill, and agent harnesses, revealing persistent gaps in retention, adaptation, and harness-induced forgetting.Salesforce7 citations32 upvotesRead →Terminal-Universe: Turning Agent Trajectories into Scalable Terminal EnvironmentsTerminal-Universe reconstructs executable workspaces from agent trajectories to synthesize diverse training tasks and improves post-training performance through supervised fine-tuning.Alibaba (Qwen)13 citations244 upvotesRead →AgentJudgeBench: A Multi-Difficulty Benchmark for Evaluating LLM Judges on Agentic Tool-CallingAgentJudgeBench reveals that LLM judges face structural reliability limits on dependency-driven agentic tool-calling workflows, with alignment degrading by difficulty and ground-truth exposure yielding mixed effects.
ServiceNow4 citations19 upvotesRead →WikiSkill: Compiling Agent Experience into Persistent Knowledge for Skill EvolutionWikiSkill co-evolves reusable agent skills with a persistent knowledge base to systematically accumulate experience and improve performance across models.Google23 citations34 upvotesRead →AutoSaddler: Automatic Harness Optimization with Durable Updates from Agent Execution TracesAutoSaddler automatically improves LLM agent harnesses via offline failure-driven optimization, boosting performance on long-horizon benchmarks.Microsoft22 citations63 upvotesRead →EnvHarness: Awakening Static Worlds for Agent LearningEnvHarness and EnvRigger dynamically reshape static environments via programmable plugins to target agent weaknesses and improve reinforcement learning co-evolution.Google19 citations174 upvotesRead →Agent Lightning v1.0: Towards Harnessed Agentic RLAgent Lightning v1.0 enables reproducible reinforcement learning for arbitrary agent harnesses, substantially improving coding-agent performance with minimal data and compute.Microsoft11 citations35 upvotesRead →Can We Defend Against AI-Generated Video Attacks on Real-World Crisis Events? A Systematic Evaluation of Detectors, Generators and Social DisseminationA new benchmark for AI-generated video detection reveals that current detectors fail to generalize across realistic crisis-related videos and become less reliable as content spreads socially.Hugging Face1 citations285 upvotesRead →HarnessOpt-Bench: Evaluating LLMs at Harness OptimizationHarnessOpt-Bench measures how well frontier LLMs iteratively improve agent harnesses under constrained evaluation budgets, revealing substantial variation across models and tasks.Scale AI12 citations35 upvotesRead →LongHorizon-Harness: Advancing Long-Horizon Agents for Real-World TasksLongHorizon-Harness improves long-horizon agent performance by explicitly tracking verified task states outside context via a manage-execute-audit loop.Alibaba (Qwen)14 citations151 upvotesRead →DarwinX: Evolving Agent Harnesses Through Natural SelectionDarwinX evolves agent harnesses via population selection with frozen models, improving verified performance across benchmarks without benchmark-specific patches.Salesforce16 citations116 upvotesRead →Model or Harness? An Interaction-Centric Taxonomy for Localizing Agent FailuresAn interaction-centric taxonomy localizes agent failures to specific component interactions to guide targeted repairs across diverse architectures.Scale AI11 citations10 upvotesRead →StateAct: Program State, before Pixels, for Long-Horizon Computer-Use AgentsStateAct improves computer-use agents by grounding actions, verification, and memory in direct program state rather than screenshots, boosting success rates while reducing cost.Salesforce6 citations62 upvotesRead →Molt: A Scalable PyTorch-Native Training Framework for Agentic Reinforcement LearningMolt is a compact PyTorch framework for agentic reinforcement learning that enables efficient asynchronous training of multimodal and mixture-of-experts policies with minimal overhead.NVIDIA1 citations33 upvotesRead →NVIDIA-labs OO Agents: Native Python Object-Oriented AgentsNOOA treats AI agents as Python objects whose methods and fields define actions, state, and prompts, enabling deterministic testing and LLM-driven runtime completion within a unified programming model.NVIDIA5 citations36 upvotesRead →RESOURCE2SKILL: Distilling Executable Agent Skills from Human-Created Multimodal ResourcesRESOURCE2SKILL converts diverse multimedia tutorials and artifacts into structured, executable skills for software agents, improving performance through hierarchical retrieval, composition, and online acquisition.Microsoft3 citations141 upvotesRead →SynthDocBench: Controlled Benchmark for Long-Context Visual Document UnderstandingSynthDocBench is a synthetic long-context visual document benchmark that isolates failure modes in vision-language models, revealing sharp length degradation, positional sensitivity, and chart comprehension breakdowns.ServiceNow1 citations68 upvotesRead →ASPIRE: Agentic /Skills Discovery for RoboticsASPIRE is a continual learning system that autonomously develops and refines robot control programs through iterative exploration, achieving superior performance and zero-shot generalization in manipulation and household tasks while enabling sim-to-real transfer.NVIDIA114 citations25 upvotesRead →SWE-INTERACT: Reimagining SWE Benchmarks as User-Driven Long-Horizon Coding SessionsSWE-Interact presents a testbed that evaluates coding agents in realistic multi-turn, user-driven software engineering scenarios, revealing significant gaps between single-turn performance and interactive task completion.Scale AI23 citations7 upvotesRead →TUA-Bench: A Benchmark for General-Purpose Terminal-Use AgentsTUA-Bench presents a comprehensive benchmark for evaluating general-purpose terminal-use agents across diverse digital activities and specialized workflows, revealing significant performance gaps among current frontier agents.Meta9 citations48 upvotesRead →Qwen-Image-Agent: Bridging the Context Gap in Real-World Image GenerationA unified agentic framework called Qwen-Image-Agent is proposed to address the context gap in text-to-image generation by progressively constructing complete generation context through planning, reasoning, searching, and memory mechanisms.Alibaba (Qwen)18 citations55 upvotesRead →The Verification Horizon: No Silver Bullet for Coding Agent RewardsVerification challenges in AI agents arise from the difficulty of aligning proxy signals with human intent, requiring adaptive verification systems that evolve alongside generative capabilities.Alibaba (Qwen)11 citations53 upvotesRead →Qwen-AgentWorld: Language World Models for General AgentsLanguage-based world models enable agentic environment simulation across multiple domains and enhance general agent performance through scalable simulation and improved downstream task performance.Alibaba (Qwen)28 citations165 upvotesRead →Beyond Static Leaderboards: Predictive Validity for the Evaluation of LLM AgentsAggregate-score leaderboards in agent benchmarks fail to capture deployment-relevant dimensions and show rank instability, necessitating new evaluation frameworks based on predictive validity and out-of-distribution criteria.
IBM2 citations42 upvotesRead →Playful Agentic Robot LearningEmbodied robots learn reusable skills through self-directed play and exploration, then apply these skills to improve performance on downstream tasks without additional training.
UC Berkeley37 citations49 upvotesRead →Zone of Proximal Policy Optimization: Teacher in Prompts, Not GradientsZone of Proximal Policy Optimization (ZPPO) improves knowledge distillation by using reformulated prompts that help students learn from both correct and incorrect responses, enhancing performance especially at smaller model sizes.NVIDIA4 citations64 upvotesRead →SpatialClaw: Rethinking Action Interface for Agentic Spatial ReasoningSpatialClaw is a training-free framework that uses code as an action interface to enable flexible, stateful spatial reasoning in vision-language models, achieving superior performance across diverse 3D/4D spatial reasoning tasks.NVIDIA23 citations113 upvotesRead →