Papersby GlyphSWE-INTERACT: Reimagining SWE Benchmarks as User-Driven Long-Horizon Coding Sessions research paper by Scale AI, 2026
Scale AI · Agents and evaluation · Jun 29, 2026 · 23 citations · 7 upvotes
What it shows
SWE-Interact presents a testbed that evaluates coding agents in realistic multi-turn, user-driven software engineering scenarios, revealing significant gaps between single-turn performance and interactive task completion.
By Mohit Raghavendra, Anisha Gunjal, Aakash Sabharwal and 1 more · arXiv 2606.30573 · PDF · Code
Hugging Face's summary; not yet checked by hand.
More from Scale AI
All 25Root-Cause Attribution Is a Search Problem: Continual Search for Long-Horizon Agent FailuresContinual Search improves automated root-cause attribution in long agent execution traces by iteratively prompting diagnosis until unresolved evidence is found.
Scale AI5 upvotesRead →SteerDuplex: Steerable Duplex Speech Dialogue ModelsA full-duplex speech model that follows spoken instructions on tone, persona, pace and voice, with a benchmark of 390 prompts to test it.Scale AI3 citations6 upvotesRead →Studying Without a Syllabus: Task-Agnostic Environment PreprocessingAn agent can explore unfamiliar environments without task-specific guidance to build reusable artifacts that reduce later inference costs, though larger study budgets do not always improve results.Scale AI1 citations4 upvotesRead →HarnessOpt-Bench: Evaluating LLMs at Harness OptimizationHarnessOpt-Bench measures how well frontier LLMs iteratively improve agent harnesses under constrained evaluation budgets, revealing substantial variation across models and tasks.Scale AI12 citations35 upvotesRead →Model or Harness? An Interaction-Centric Taxonomy for Localizing Agent FailuresAn interaction-centric taxonomy localizes agent failures to specific component interactions to guide targeted repairs across diverse architectures.Scale AI11 citations10 upvotesRead →Not Every Rubric Teaches Equally: Policy-Aware Rubric Rewards for RLVRPOW3R is a policy-aware framework for reinforcement learning with rubric-based rewards that adapts criterion weights during training to improve policy optimization while preserving human-defined criteria importance.Scale AI13 citations5 upvotesRead →Reward Hacking in Rubric-Based Reinforcement LearningResearch examines reward hacking in rubric-based reinforcement learning, identifying verifier failure and rubric-design limitations as key sources of divergence between training and evaluation metrics.Scale AI33 citations4 upvotesRead →HiL-Bench (Human-in-Loop Benchmark): Do Agents Know When to Ask for Help?Frontier AI agents struggle with judgment calls about when to seek help, leading to poor performance on incomplete or ambiguous tasks despite having sufficient capabilities.Scale AI28 citations4 upvotesRead →
Other agents and evaluation papers
TopicGemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic CapabilitiesGemini 2.X model family, including Gemini 2.5 Pro and Flash, offers superior coding, reasoning, and multimodal understanding capabilities across a range of computational efficiencies.
Google4,594 citations67 upvotesRead →DeepSeek-V3.2: Pushing the Frontier of Open Large Language ModelsDeepSeek-V3.2 introduces DeepSeek Sparse Attention and a scalable reinforcement learning framework, achieving superior reasoning and performance compared to GPT-5 and Gemini-3.0-Pro in complex reasoning tasks.
DeepSeek887 citations275 upvotesRead →WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?WorkArena and BrowserGym evaluate large language model-based agents' ability to perform enterprise software tasks, revealing gaps in current agent capabilities and differences between open and closed-source LLMs.
ServiceNow443 citations2 upvotesRead →Aya Model: An Instruction Finetuned Open-Access Multilingual Language ModelAya, a multilingual generative language model supporting over 50% lower-resourced languages, excels in both generative and discriminative tasks across 99 languages, and provides extensive evaluations and open-source resources.
Cohere438 citations48 upvotesRead →Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual EvaluationGlobal-MMLU addresses cultural biases and translation artifacts in multilingual datasets by involving professional annotators and offering culturally distinct subsets for more accurate model evaluation.Cohere236 citations21 upvotesRead →The BrowserGym Ecosystem for Web Agent ResearchBrowserGym provides a standardized environment for evaluating web agents using LLMs, facilitating consistent benchmarking and improving agent development and analysis.ServiceNow200 citations24 upvotesRead →
About this paper
- Venue
- arXiv.org
- Citations
- 23, 4 influential · Semantic Scholar
- Upvotes
- 7 · Hugging Face
- Lab
- Scale AI · on Companies · on Acquisitions · on awards · on breaches · on Attested · on Rounds · on Downtime · on labs · on Leaders · on marketplaces · on Allied · on Paydays · on quotes · on salaries · on Nines · on socials · on startups · on terms · on Titlewave
Changes
Citations rose from 21 to 23.
Influential citations rose from 1 to 4.
Citations rose from 10 to 21.
Citations rose from 8 to 10.
Influential citations rose from 0 to 1.
Citations rose from 7 to 8.
A first count of 0 influential citations.
A first count of 7 citations.
Added to the list.