Papersby GlyphWorkArena++: Towards Compositional Planning and Reasoning-based Common Knowledge Work Tasks research paper by ServiceNow, 2024
ServiceNow · Reasoning · Jul 7, 2024 · 114 citations · 2 upvotes
What it shows
WorkArena++ benchmark evaluates the effectiveness of LLMs and VLMs in workplace tasks, revealing challenges and providing a mechanism for model fine-tuning.
By Léo Boisvert, Megh Thakkar, Maxime Gasse and 6 more · arXiv 2407.05291 · PDF
Hugging Face's summary; not yet checked by hand.
More from ServiceNow
All 21Apriel Voice 27B Preview: An Open Model for Speech, Reasoning, and Tool UseApriel Voice 27B is an open AI model that directly processes spoken speech to converse, reason, and use external tools.
ServiceNow37 upvotesRead →AgentJudgeBench: A Multi-Difficulty Benchmark for Evaluating LLM Judges on Agentic Tool-CallingAgentJudgeBench reveals that LLM judges face structural reliability limits on dependency-driven agentic tool-calling workflows, with alignment degrading by difficulty and ground-truth exposure yielding mixed effects.ServiceNow4 citations19 upvotesRead →StarHarness: Evolving Harnesses with Stratified Search for Enterprise EnvironmentsStarHarness evolves fixed-weight agent harnesses via stratified task pools and hidden selection to improve enterprise tool-use performance and cross-model transfer.ServiceNow3 citations41 upvotesRead →SynthDocBench: Controlled Benchmark for Long-Context Visual Document UnderstandingSynthDocBench is a synthetic long-context visual document benchmark that isolates failure modes in vision-language models, revealing sharp length degradation, positional sensitivity, and chart comprehension breakdowns.ServiceNow1 citations68 upvotesRead →EVA-Bench: A New End-to-end Framework for Evaluating Voice AgentsEVA-Bench presents a comprehensive evaluation framework for voice agents that simulates realistic conversations and measures performance across multiple voice-specific failure modes using novel accuracy and experience metrics.ServiceNow20 citations77 upvotesRead →Do Enterprise Systems Need Learned World Models? The Importance of Context to Infer DynamicsEnterprise discovery agents that read system configuration at runtime outperform traditional world models in configurable environments where dynamics change over time.ServiceNow2 citations65 upvotesRead →Apriel-Reasoner: RL Post-Training for General-Purpose and Efficient ReasoningApriel-Reasoner is a 15B-parameter language model trained with reproducible multi-domain reinforcement learning to improve reasoning efficiency and accuracy across diverse tasks while reducing inference costs.ServiceNow12 upvotesRead →Therefore I am. I ThinkReasoning models appear to encode action choices before beginning textual deliberation, as evidenced by early decision detection and activation steering effects.ServiceNow2 citations31 upvotesRead →
Other reasoning papers
TopicChain-of-Thought Prompting Elicits Reasoning in Large Language ModelsAsking a model to write out its intermediate steps (chain of thought) sharply improves its math and logic answers.
Google23.8k citations16 upvotesRead →DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language ModelsDeepSeekMath 7B improves mathematical reasoning through enhanced data pre-training and Group Relative Policy Optimization, achieving high scores on MATH benchmark without external tools.
DeepSeek10.6k citations152 upvotesRead →DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement LearningReinforcement learning taught a model long step-by-step reasoning on par with OpenAI o1.DeepSeek5,972 citations458 upvotesRead →DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code IntelligenceDeepSeek-Coder-V2, a Mixture-of-Experts language model, excels in code-specific tasks by enhancing coding and mathematical reasoning capabilities while expanding language support and context length.DeepSeek530 citations73 upvotesRead →MolmoAct: Action Reasoning Models that can Reason in SpaceAction Reasoning Models (ARMs) integrate perception, planning, and control to enable adaptable and explainable robotic behavior, achieving superior performance across various tasks and settings.
Ai2232 citations45 upvotesRead →ThinkAct: Vision-Language-Action Reasoning via Reinforced Visual Latent PlanningThinkAct, a dual-system framework, uses reinforced visual latent planning to enable few-shot adaptation, long-horizon planning, and self-correction in embodied AI tasks by bridging high-level reasoning with low-level action execution.
NVIDIA192 citations43 upvotesRead →
About this paper
- Venue
- Neural Information Processing Systems
- Citations
- 114, 7 influential · Semantic Scholar
- Upvotes
- 2 · Hugging Face
- Lab
- ServiceNow · on Companies · on Acquisitions · on Quarterly · on Paydays · on TechConf
Changes
Influential citations rose from 6 to 7.
Citations rose from 111 to 114.
Citations rose from 96 to 111.
Citations rose from 93 to 96.
Citations rose from 88 to 93.
A first count of 6 influential citations.
A first count of 88 citations.
Added to the list.