Smart Paste for context-aware adjustments to pasted code

🌈 Abstract

The article discusses the development and deployment of "Smart Paste", a tool that uses AI to provide context-aware adjustments to pasted code, in order to streamline the code development workflow at Google. It covers the analysis of user behavior, the model training process, the UI design exploration, and the overall impact and adoption of the feature.

🙋 Q&A

[01] Overview of Smart Paste

1. What is the purpose of the Smart Paste tool? The purpose of Smart Paste is to provide AI-generated, context-aware adjustments to pasted code in order to streamline the code development workflow. It predicts the next state of the code environment and suggests modifications to the pasted code to better fit the surrounding context.

2. What were the key findings from the analysis of user behavior? The analysis of the Google monorepo revealed that:

• ~25% of pasted code is immediately modified, with edits ranging from small fixes to more sophisticated adaptations
• Making such changes often breaks the code-authoring flow and slows down the development process
• 28% of all pastes have no follow-up edits, with variance among programming languages from 23% to 41%

3. How was the training data for the Smart Paste model prepared? The training data was extracted from the Google monorepo and associated comprehensive logs, using a set of heuristics to constrain the notion of post-paste adjustments to those in close proximity to the original paste location. The dataset included examples with no follow-up edits, so the model could learn to output "no edits needed" as well.

[02] Model Development and Deployment

1. What model was used as the starting point for Smart Paste? The model used for Smart Paste was pre-trained on coding-related tasks from DIDACT, a large sequence model that has enabled tools for code review, build repair, and more.

2. How did the team optimize the model for speed and accuracy? The team experimented with different model sizes and properties, aiming to strike the right balance between suggesting entirely accurate edits and showing speculative suggestions that might provide some value but require follow-up changes. They settled on a latency of ~150 ms, which resulted in the model showing a suggestion in 56% of the evaluation cases.

3. What UI patterns were explored for displaying the suggested edits? The team explored various UI patterns, including:

• Grayed-out text (similar to code completion)
• Full diff view
• Automatically applying the suggested edits and highlighting the changes
• Showing the inserted and removed snippets directly in the editor as an inline diff

They ultimately chose the inline diff approach, as it allowed users to remain in control while providing a sufficient overview of the suggested changes.

[03] Impact and Future Plans

1. What was the overall adoption and impact of Smart Paste? According to the article, 6.9% of all pastes in the IDE utilized Smart Paste, with a 42.5% acceptance rate, which significantly streamlined the developer workflow.

2. What are the future plans for Smart Paste? The team plans to revisit the auto-apply mode for small, high-confidence changes, as they found that suggestions with fewer than 10 characters have nearly half the acceptance rate compared to suggestions containing 10 to 100 characters. They also plan to explore the model's capabilities in providing helpful suggestions outside of the copy-paste interactions, such as automatically adapting all references to a property after renaming and copy-pasting.