Papersby GlyphCodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation research paper by Salesforce, 2021
Salesforce · Architectures · Sep 2, 2021 · 2,633 citations · 3 upvotes
What it shows
CodeT5, a unified encoder-decoder Transformer model, improves code understanding and generation by leveraging semantic information from identifiers and user comments.
By Yue Wang, Weishi Wang, Shafiq Joty and 1 more · arXiv 2109.00859 · PDF
Hugging Face's summary; not yet checked by hand.
More from Salesforce
All 34Self-Retrospection Distillation: Turning Post-hoc Experiences into Prior ForesightSelf-Retrospection Distillation improves agent performance by using hindsight from completed trajectories to supervise foresight predictions during training.
Salesforce1 citations197 upvotesRead →Just-in-Time Memory: Learning to Curate Task-Adaptive Memory for LLM AgentsKeeps an agent's raw past runs and writes a task-specific memory only when a new task arrives, rather than deciding up front what to keep.Salesforce2 citations56 upvotesRead →Flattening Every Memory Peak in Long-Context Mixture-of-Experts TrainingBounds the four memory peaks that crash long-context mixture-of-experts training, from expert dispatch to optimizer state, with fixed-size GPU schedules.Salesforce19 upvotesRead →RISE: Recursive Improvement via Self-Extrapolating Policy DistillationRISE improves language model post-training by recursively generating dense token-level supervision from the model's own reinforcement learning trajectory via self-extrapolation, avoiding external teachers.Salesforce5 citations17 upvotesRead →EVOHARNESSBENCH: Can Your Agents Keep Pace with an Evolving Harness?The study introduces a benchmark to evaluate LLM agents under evolving tool, skill, and agent harnesses, revealing persistent gaps in retention, adaptation, and harness-induced forgetting.Salesforce7 citations32 upvotesRead →Random Attention: Rethinking KV Cache Eviction for Efficient ReasoningRandom eviction of reasoning tokens matches selective KV cache compression because reasoning traces are self-protecting through redundancy, making scoring unnecessary once prompts are preserved.Salesforce5 citations187 upvotesRead →DarwinX: Evolving Agent Harnesses Through Natural SelectionDarwinX evolves agent harnesses via population selection with frozen models, improving verified performance across benchmarks without benchmark-specific patches.Salesforce16 citations116 upvotesRead →StateAct: Program State, before Pixels, for Long-Horizon Computer-Use AgentsStateAct improves computer-use agents by grounding actions, verification, and memory in direct program state rather than screenshots, boosting success rates while reducing cost.Salesforce6 citations62 upvotesRead →
Other architectures papers
TopicAttention Is All You NeedIntroduced the Transformer, the attention-only architecture behind nearly every large language model since.
Google201k citations142 upvotesRead →Mamba: Linear-Time Sequence Modeling with Selective State SpacesA selective state space model that scales linearly with sequence length and matches Transformers on language.
Carnegie Mellon University10.2k citations152 upvotesRead →VGGT: Visual Geometry Grounded TransformerVGGT, a feed-forward neural network, efficiently infers multiple 3D attributes from single or multiple views, outperforming alternatives and enhancing downstream tasks without post-processing.
Meta2,132 citations40 upvotesRead →DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language ModelsThe DeepSeekMoE architecture improves expert specialization in Mixture-of-Experts models by segmenting experts and isolating shared ones, achieving better performance and computational efficiency compared to GShard and other models.
DeepSeek1,268 citations65 upvotesRead →HuggingFace's Transformers: State-of-the-art Natural Language ProcessingTransformers library provides state-of-the-art Transformer architectures and pretrained models for natural language processing tasks with a unified API and emphasis on extensibility and robust deployment.
Hugging Face251 citations34 upvotesRead →Hymba: A Hybrid-head Architecture for Small Language ModelsHymba, a family of small language models with a hybrid-head architecture combining transformer attention and state space models, achieves state-of-the-art performance with improved efficiency and reduced cache size.
NVIDIA126 citations50 upvotesRead →
About this paper
- Venue
- Conference on Empirical Methods in Natural Language Processing
- Citations
- 2,633, 397 influential · Semantic Scholar
- Upvotes
- 3 · Hugging Face
- Lab
- Salesforce · on Companies · on Acquisitions · on Quarterly · on Paydays · on TechConf
Changes
Influential citations rose from 386 to 397.
Citations rose from 2,557 to 2,633.
Citations rose from 2,528 to 2,557.
Influential citations rose from 383 to 386.
Citations rose from 2,510 to 2,528.
A first count of 383 influential citations.
A first count of 2,510 citations.
Added to the list.