Papersby GlyphThe BrowserGym Ecosystem for Web Agent Research research paper by ServiceNow, 2024
ServiceNow · Agents and evaluation · Dec 6, 2024 · 200 citations · 24 upvotes
What it shows
BrowserGym provides a standardized environment for evaluating web agents using LLMs, facilitating consistent benchmarking and improving agent development and analysis.
By Thibault Le Sellier De Chezelles, Maxime Gasse, Alexandre Drouin and 17 more · arXiv 2412.05467 · PDF · Code
Hugging Face's summary; not yet checked by hand.
More from ServiceNow
All 21Apriel Voice 27B Preview: An Open Model for Speech, Reasoning, and Tool UseApriel Voice 27B is an open AI model that directly processes spoken speech to converse, reason, and use external tools.
ServiceNow37 upvotesRead →AgentJudgeBench: A Multi-Difficulty Benchmark for Evaluating LLM Judges on Agentic Tool-CallingAgentJudgeBench reveals that LLM judges face structural reliability limits on dependency-driven agentic tool-calling workflows, with alignment degrading by difficulty and ground-truth exposure yielding mixed effects.ServiceNow4 citations19 upvotesRead →StarHarness: Evolving Harnesses with Stratified Search for Enterprise EnvironmentsStarHarness evolves fixed-weight agent harnesses via stratified task pools and hidden selection to improve enterprise tool-use performance and cross-model transfer.ServiceNow3 citations41 upvotesRead →SynthDocBench: Controlled Benchmark for Long-Context Visual Document UnderstandingSynthDocBench is a synthetic long-context visual document benchmark that isolates failure modes in vision-language models, revealing sharp length degradation, positional sensitivity, and chart comprehension breakdowns.ServiceNow1 citations68 upvotesRead →EVA-Bench: A New End-to-end Framework for Evaluating Voice AgentsEVA-Bench presents a comprehensive evaluation framework for voice agents that simulates realistic conversations and measures performance across multiple voice-specific failure modes using novel accuracy and experience metrics.ServiceNow20 citations77 upvotesRead →Do Enterprise Systems Need Learned World Models? The Importance of Context to Infer DynamicsEnterprise discovery agents that read system configuration at runtime outperform traditional world models in configurable environments where dynamics change over time.ServiceNow2 citations65 upvotesRead →Apriel-Reasoner: RL Post-Training for General-Purpose and Efficient ReasoningApriel-Reasoner is a 15B-parameter language model trained with reproducible multi-domain reinforcement learning to improve reasoning efficiency and accuracy across diverse tasks while reducing inference costs.ServiceNow12 upvotesRead →Therefore I am. I ThinkReasoning models appear to encode action choices before beginning textual deliberation, as evidenced by early decision detection and activation steering effects.ServiceNow2 citations31 upvotesRead →
Other agents and evaluation papers
TopicGemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic CapabilitiesGemini 2.X model family, including Gemini 2.5 Pro and Flash, offers superior coding, reasoning, and multimodal understanding capabilities across a range of computational efficiencies.
Google4,594 citations67 upvotesRead →DeepSeek-V3.2: Pushing the Frontier of Open Large Language ModelsDeepSeek-V3.2 introduces DeepSeek Sparse Attention and a scalable reinforcement learning framework, achieving superior reasoning and performance compared to GPT-5 and Gemini-3.0-Pro in complex reasoning tasks.
DeepSeek887 citations275 upvotesRead →Aya Model: An Instruction Finetuned Open-Access Multilingual Language ModelAya, a multilingual generative language model supporting over 50% lower-resourced languages, excels in both generative and discriminative tasks across 99 languages, and provides extensive evaluations and open-source resources.
Cohere438 citations48 upvotesRead →Rubrics as Rewards: Reinforcement Learning Beyond Verifiable DomainsRubrics as Rewards (RaR) framework uses structured rubrics as interpretable reward signals for on-policy training, improving performance in real-world reinforcement learning tasks with subjective criteria.
Scale AI431 citations6 upvotesRead →SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?SWE-Bench Pro is a challenging benchmark for coding models, featuring complex, enterprise-level problems that require substantial code modifications, with performance evaluations showing significant limitations in current models.Scale AI367 citations21 upvotesRead →A Careful Examination of Large Language Model Performance on Grade School ArithmeticEvaluation of large language models on a new math benchmark, GSM1k, reveals that many models exhibit signs of overfitting to the existing GSM8k benchmark, leading to performance drops on GSM1k.Scale AI255 citations33 upvotesRead →
About this paper
- Venue
- Trans. Mach. Learn. Res.
- Citations
- 200, 17 influential · Semantic Scholar
- Upvotes
- 24 · Hugging Face
- Lab
- ServiceNow · on Companies · on Acquisitions · on Quarterly · on Paydays · on TechConf
Changes
Influential citations fell from 18 to 17.
Citations rose from 193 to 200.
Influential citations rose from 16 to 18.
Citations rose from 176 to 193.
Citations rose from 169 to 176.
Influential citations rose from 14 to 16.
Citations rose from 159 to 169.
A first count of 14 influential citations.
A first count of 159 citations.
Added to the list.