Papersby GlyphCodeT5+: Open Code Large Language Models for Code Understanding and Generation research paper by Salesforce, 2023
Salesforce · Architectures · May 13, 2023 · 836 citations · 7 upvotes
What it shows
CodeT5+, a family of flexible encoder-decoder LLMs initialized with frozen LLMs and trained with a mixture of pretraining objectives, achieves state-of-the-art performance across various code-related tasks.
By Yue Wang, Hung Le, Akhilesh Deepak Gotmare and 3 more · arXiv 2305.07922 · PDF · Code
Hugging Face's summary; not yet checked by hand.
More from Salesforce
All 34Self-Retrospection Distillation: Turning Post-hoc Experiences into Prior ForesightSelf-Retrospection Distillation improves agent performance by using hindsight from completed trajectories to supervise foresight predictions during training.
Salesforce1 citations197 upvotesRead →Just-in-Time Memory: Learning to Curate Task-Adaptive Memory for LLM AgentsKeeps an agent's raw past runs and writes a task-specific memory only when a new task arrives, rather than deciding up front what to keep.Salesforce2 citations56 upvotesRead →Flattening Every Memory Peak in Long-Context Mixture-of-Experts TrainingBounds the four memory peaks that crash long-context mixture-of-experts training, from expert dispatch to optimizer state, with fixed-size GPU schedules.Salesforce19 upvotesRead →RISE: Recursive Improvement via Self-Extrapolating Policy DistillationRISE improves language model post-training by recursively generating dense token-level supervision from the model's own reinforcement learning trajectory via self-extrapolation, avoiding external teachers.Salesforce5 citations17 upvotesRead →EVOHARNESSBENCH: Can Your Agents Keep Pace with an Evolving Harness?The study introduces a benchmark to evaluate LLM agents under evolving tool, skill, and agent harnesses, revealing persistent gaps in retention, adaptation, and harness-induced forgetting.Salesforce7 citations32 upvotesRead →Random Attention: Rethinking KV Cache Eviction for Efficient ReasoningRandom eviction of reasoning tokens matches selective KV cache compression because reasoning traces are self-protecting through redundancy, making scoring unnecessary once prompts are preserved.Salesforce5 citations187 upvotesRead →DarwinX: Evolving Agent Harnesses Through Natural SelectionDarwinX evolves agent harnesses via population selection with frozen models, improving verified performance across benchmarks without benchmark-specific patches.Salesforce16 citations116 upvotesRead →StateAct: Program State, before Pixels, for Long-Horizon Computer-Use AgentsStateAct improves computer-use agents by grounding actions, verification, and memory in direct program state rather than screenshots, boosting success rates while reducing cost.Salesforce6 citations62 upvotesRead →
Other architectures papers
TopicAttention Is All You NeedIntroduced the Transformer, the attention-only architecture behind nearly every large language model since.
Google201k citations142 upvotesRead →Mamba: Linear-Time Sequence Modeling with Selective State SpacesA selective state space model that scales linearly with sequence length and matches Transformers on language.
Carnegie Mellon University10.2k citations152 upvotesRead →VGGT: Visual Geometry Grounded TransformerVGGT, a feed-forward neural network, efficiently infers multiple 3D attributes from single or multiple views, outperforming alternatives and enhancing downstream tasks without post-processing.
Meta2,132 citations40 upvotesRead →DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language ModelsThe DeepSeekMoE architecture improves expert specialization in Mixture-of-Experts models by segmenting experts and isolating shared ones, achieving better performance and computational efficiency compared to GShard and other models.
DeepSeek1,268 citations65 upvotesRead →HuggingFace's Transformers: State-of-the-art Natural Language ProcessingTransformers library provides state-of-the-art Transformer architectures and pretrained models for natural language processing tasks with a unified API and emphasis on extensibility and robust deployment.
Hugging Face251 citations34 upvotesRead →Hymba: A Hybrid-head Architecture for Small Language ModelsHymba, a family of small language models with a hybrid-head architecture combining transformer attention and state space models, achieves state-of-the-art performance with improved efficiency and reduced cache size.
NVIDIA126 citations50 upvotesRead →
About this paper
- Venue
- Conference on Empirical Methods in Natural Language Processing
- Citations
- 836, 70 influential · Semantic Scholar
- Upvotes
- 7 · Hugging Face
- Lab
- Salesforce · on Companies · on Acquisitions · on Quarterly · on Paydays · on TechConf
Changes
Citations rose from 813 to 836.
Citations rose from 800 to 813.
Citations rose from 792 to 800.
A first count of 70 influential citations.
A first count of 792 citations.
Added to the list.