Papersby GlyphScale AI research papers
A company publishing mostly on agents and evaluation, best known here for Rubrics as Rewards.25 papers · 2,091 citations · latest Sep 2026
Root-Cause Attribution Is a Search Problem: Continual Search for Long-Horizon Agent FailuresContinual Search improves automated root-cause attribution in long agent execution traces by iteratively prompting diagnosis until unresolved evidence is found.
Scale AI5 upvotesRead →SteerDuplex: Steerable Duplex Speech Dialogue ModelsA full-duplex speech model that follows spoken instructions on tone, persona, pace and voice, with a benchmark of 390 prompts to test it.Scale AI3 citations6 upvotesRead →Studying Without a Syllabus: Task-Agnostic Environment PreprocessingAn agent can explore unfamiliar environments without task-specific guidance to build reusable artifacts that reduce later inference costs, though larger study budgets do not always improve results.Scale AI1 citations4 upvotesRead →HarnessOpt-Bench: Evaluating LLMs at Harness OptimizationHarnessOpt-Bench measures how well frontier LLMs iteratively improve agent harnesses under constrained evaluation budgets, revealing substantial variation across models and tasks.Scale AI12 citations35 upvotesRead →Model or Harness? An Interaction-Centric Taxonomy for Localizing Agent FailuresAn interaction-centric taxonomy localizes agent failures to specific component interactions to guide targeted repairs across diverse architectures.Scale AI11 citations10 upvotesRead →SWE-INTERACT: Reimagining SWE Benchmarks as User-Driven Long-Horizon Coding SessionsSWE-Interact presents a testbed that evaluates coding agents in realistic multi-turn, user-driven software engineering scenarios, revealing significant gaps between single-turn performance and interactive task completion.Scale AI23 citations7 upvotesRead →Not Every Rubric Teaches Equally: Policy-Aware Rubric Rewards for RLVRPOW3R is a policy-aware framework for reinforcement learning with rubric-based rewards that adapts criterion weights during training to improve policy optimization while preserving human-defined criteria importance.Scale AI13 citations5 upvotesRead →Reward Hacking in Rubric-Based Reinforcement LearningResearch examines reward hacking in rubric-based reinforcement learning, identifying verifier failure and rubric-design limitations as key sources of divergence between training and evaluation metrics.Scale AI33 citations4 upvotesRead →HiL-Bench (Human-in-Loop Benchmark): Do Agents Know When to Ask for Help?Frontier AI agents struggle with judgment calls about when to seek help, leading to poor performance on incomplete or ambiguous tasks despite having sufficient capabilities.Scale AI28 citations4 upvotesRead →SciPredict: Can LLMs Predict the Outcomes of Scientific Experiments in Natural Sciences?SciPredict benchmark reveals that large language models struggle to accurately predict scientific experiment outcomes and cannot reliably assess prediction confidence, unlike human experts who show better calibration and performance when experiments are deemed predictable.Scale AI4 citations4 upvotesRead →MCP-Atlas: A Large-Scale Benchmark for Tool-Use Competency with Real MCP ServersMCP-Atlas is a large-scale benchmark for evaluating tool-use competency in LLMs, featuring 36 real servers and 220 tools across 1,000 realistic multi-step tasks with claims-based scoring.Scale AI65 citations1 upvotesRead →Agentic Rubrics as Contextual Verifiers for SWE AgentsAgentic Rubrics enable efficient and scalable verification for software engineering agents by creating context-aware checklists that outperform traditional methods while maintaining interpretability.Scale AI30 citations13 upvotesRead →PropensityBench: Evaluating Latent Safety Risks in Large Language Models via an Agentic ApproachA new benchmark framework, PropensityBench, evaluates models' likelihood to engage in risky behaviors when equipped with simulated dangerous capabilities, highlighting the need for dynamic propensity assessments in AI safety.Scale AI22 citationsRead →ResearchRubrics: A Benchmark of Prompts and Rubrics For Evaluating Deep Research AgentsResearchRubrics is a benchmark for evaluating deep research agents, using expert rubrics to assess their factual grounding, reasoning, and clarity across diverse, complex tasks.Scale AI96 citations10 upvotesRead →MoReBench: Evaluating Procedural and Pluralistic Moral Reasoning in Language Models, More than OutcomesMoReBench and MoReBench-Theory provide benchmarks for evaluating AI's moral reasoning and decision-making processes, highlighting the need for process-focused evaluation and transparency in AI systems.Scale AI25 citations2 upvotesRead →TutorBench: A Benchmark To Assess Tutoring Capabilities Of Large Language ModelsTutorBench is a dataset and benchmark for evaluating the tutoring skills of large language models, showing significant room for improvement in adaptive explanations, feedback, and active learning.Scale AI13 citations2 upvotesRead →Chasing the Tail: Effective Rubric-based Reward Modeling for Large Language Model Post-TrainingRubric-based rewards mitigate reward over-optimization in reinforcement fine-tuning by leveraging off-policy examples while maintaining reward reliability.Scale AI62 citations20 upvotesRead →SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?SWE-Bench Pro is a challenging benchmark for coding models, featuring complex, enterprise-level problems that require substantial code modifications, with performance evaluations showing significant limitations in current models.Scale AI367 citations21 upvotesRead →Rubrics as Rewards: Reinforcement Learning Beyond Verifiable DomainsRubrics as Rewards (RaR) framework uses structured rubrics as interpretable reward signals for on-policy training, improving performance in real-world reinforcement learning tasks with subjective criteria.Scale AI431 citations6 upvotesRead →EnigmaEval: A Benchmark of Long Multimodal Reasoning ChallengesEnigmaEval introduces a benchmark of complex multimodal puzzles to evaluate language models on implicit knowledge synthesis and multi-step deductive reasoning.Scale AI21 citationsRead →MultiChallenge: A Realistic Multi-Turn Conversation Evaluation Benchmark Challenging to Frontier LLMsMultiChallenge evaluates large language models on multi-turn conversations, identifying four key challenges that current models struggle with, despite high performance on existing benchmarks.Scale AI207 citations1 upvotesRead →Refusal-Trained LLMs Are Easily Jailbroken As Browser AgentsBrowserART, a test suite for red-teaming browser agents, reveals that LLMs trained to refuse harmful instructions in chats often fail to do so when equipped with web browser capabilities.Scale AI82 citationsRead →Planning In Natural Language Improves LLM Search For Code GenerationPLANSEARCH, a novel search algorithm, improves performance in coding benchmarks by generating a diverse set of natural language plans, outperforming existing methods through increased diversity in solutions.Scale AI101 citationsRead →LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks YetMulti-turn human jailbreaks reveal significant vulnerabilities in large language model defenses, achieving high attack success rates and exposing flaws in machine unlearning mechanisms.Scale AI186 citationsRead →A Careful Examination of Large Language Model Performance on Grade School ArithmeticEvaluation of large language models on a new math benchmark, GSM1k, reveals that many models exhibit signs of overfitting to the existing GSM8k benchmark, leading to performance drops on GSM1k.Scale AI255 citations33 upvotesRead →
About and links
Topics: Agents and evaluation 17, Training and scaling 2, Reasoning 2, Multimodal and robotics 1, Applied AI 1, Retrieval and data 1, Foundation models 1
Scale AI on Companies (companies.fru.dev)Also onAcquisitions·awards·breaches·Attested·Rounds·Downtime·labs·Leaders·marketplaces·Allied·Paydays·quotes·salaries·Nines·socials·startups·terms·Titlewave