Software Engineering
At Google, we pride ourselves on our ability to develop and launch new products and features at a very fast pace. This is made possible in part by our world-class engineers, but our approach to software development enables us to balance speed and quality, and is integral to our success. Our obsession for speed and scale is evident in our developer infrastructure and tools. Developers across the world continually write, build, test and release code in multiple programming languages like C++, Java, Python, Javascript and others, and the Engineering Tools team, for example, is challenged to keep this development ecosystem running smoothly. Our engineers leverage these tools and infrastructure to produce clean code and keep software development running at an ever-increasing scale. In our publications, we share associated technical challenges and lessons learned along the way.
Recent Publications
Taming the Variants Multi-Architecture Continuous Testing at Google
Chandrakanth Chittappa
Ali Esmaeeli
Laura Macaddino
Sam Manfreda
David Margolin
Dharma Naidu
Sabuj Pattanayek
Sachin Sable
Ruslan Sakevych
Dushyant Acharya
Adrian Berding
Kevin Crossan
Wolff Dobson
Abhay Singh
19th IEEE International Conference on Software Testing, Verification and Validation (ICST) 2026, Daejeon, Republic of Korea, IEEE
Preview abstract
Enterprises are increasingly adopting multiple general-purpose computer architectures in the data center. This leads to new testing challenges as it creates demand to qualify the software for the additional architectures. Naively double-testing all software for both architectures is costly and unnecessary. Further, reconfiguring CI/CD to take advantage of the new architecture can be non-trivial at scale. This paper introduces CI/CD variants and an optimized testing cycle to solve these twin challenges. We empirically evaluate our solution's impact on human and machine expenses using 44k projects at Google on real production data. First, we estimate saving ~25% of machine expenses at the negligible cost of a few delayed breakage detections per day. Second, we estimate a 90+% reduction in human cost for migrating the configuration. All features described in this paper are now Generally Available at Google and we report this as an empirical case study in scaling CI/CD to new architectures.
View details
Towards AI as a Collaborative Partner: A Taxonomy of AI Agent Behavior in Software Engineering
Proceedings of the 3rd ACM International Conference on AI-Powered Software (AIware '26), ACM, Montreal, QC, Canada (2026) (to appear)
Preview abstract
The ongoing transition of Large Language Models (LLMs) in software engineering from one-shot code generators into agentic partners requires a shift in how we define and measure success. While models are becoming more capable, the industry lacks a clear understanding of the behavioral norms that make an interactive software engineering (SWE) agent effective in collaborative software development in the enterprise. This work addresses this gap by presenting a taxonomy of desirable SWE agent behaviors, synthesized from 91 sets of developer-defined rules for SWE agents and validated through interviewing 15 experienced professional developers. In this taxonomy, we identify four core expectations: Adhere to Standards and Processes, Ensure Code Quality and Reliability, Solve Problems Effectively, and Collaborate with the Developer. These findings offer a concrete vocabulary for aligning SWE agent behavior with developer preferences, enabling researchers and practitioners to move beyond correctness-only benchmarks and start designing evaluations that reflect the socio-technical nature of professional software development in enterprises.
View details
Towards A Human-in-the-Loop Framework for Reliable Patch Evaluation using an LLM-as-a-Judge
Renyao Wei
Michele Tufano
José Cambronero
AI-SQE '26: Proceedings of the 1st International Workshop on AI for Software Quality Evaluation - Judgment, Metrics, Benchmarks, and Beyond, ACM (Association for Computing Machinery), New York, NY, USA (2026), pp. 19 - 28
Preview abstract
Reliable evaluation is crucial for advancing Automated Program Repair (APR), but prevailing benchmarks that rely on execution-based evaluation methods (pass@k) often fail to capture the patch quality required for real-world adoption. This creates a significant gap between automated metrics and true patch validity (valid@k), a discrepancy observed across several state-of-the-art techniques.
To develop a scalable solution for measuring valid@k, we first study the human evaluation process itself. While manual assessment can determine validity, we find it suffers from poor inter-rater reliability (Fleiss' Kappa k=0.307). Our foundational insight is that this inconsistency is largely resolved when evaluators use a shared, high-quality rubric, which significantly improves agreement.
Building on this finding, we propose an LLM-as-a-Judge framework that operationalizes rubric-guided evaluation at scale. Our method employs a human-in-the-loop workflow where an LLM first generates a candidate rubric for a given bug, which a human expert then reviews and refines into a "golden" evaluation standard. This golden rubric is then used by an LLM judge to assess the validity of candidate patches. In an evaluation on 48 bugs and 115 patches, our LLM judge demonstrates substantial agreement with the consensus of human developers. This work contributes a scalable and reliable methodology for approximating valid@k, providing a much-needed high-fidelity signal for measuring true progress in the field of automated program repair.
View details
From Correctness to Collaboration: A Human-Centered Taxonomy of AI Agent Behavior in Software Engineering
Extended Abstracts of the 2026 CHI Conference on Human Factors in Computing Systems (CHI EA ’26), ACM, New York, NY, USA (2026)
Preview abstract
The ongoing transition of Large Language Models in software engineering from code generators into autonomous agents requires a shift in how we define and measure success. While models are becoming more capable, the industry lacks a clear understanding of the behavioral norms that make an agent effective in collaborative software development in the enterprise. This work addresses this gap by presenting a taxonomy of desirable agent behaviors, synthesized from 91 sets of user-defined rules for coding agents. We identify four core expectations: Adhere to Standards and Processes, Ensure Code Quality and Reliability, Solve Problems Effectively, and Collaborate with the User. These findings offer a concrete vocabulary for agent behavior, enabling researchers to move beyond correctness-only benchmarks and design evaluations that reflect the realities of professional software development in large enterprises.
View details
Preview abstract
In "Elephants, Goldfish and the New Golden Age of Software Engineering," the author discusses how AI is changing knowledge work, especially software development. Written from the perspective of April 2026, the article points out that while AI speeds up coding, it can also quickly generate a lot of mistakes and messy code if it isn't carefully managed by human oversight and clear processes.
The paper outlines a practical approach to working with AI, broken down into three main sections:
Using AI as a Tool, Not a Toy:
The author notes that people often get poor results by asking AI to do everything in a single prompt. Instead, users should have back-and-forth conversations with AI to question assumptions, set clear grading rules, and guide the research. The main point is that humans must still provide the final judgment; AI is simply a way to speed up and record that thinking.
The Elephant-Goldfish Model:
As AI creates more code than humans can easily read, written design documents become more important than the code itself. To keep AI on track, the author suggests a two-part method:
* The Elephant: A long chat session where the human and AI discuss ideas and write a detailed design document *before* any code is written. This session holds all of the project's background information and decisions.
* The Goldfish: A brand-new AI chat session with no memory. The human asks this "goldfish" to read the design document. If the goldfish cannot understand the plan based only on that document, the document needs more details.
* Only after the design document is clear enough for the goldfish to understand does the human ask the AI to write the code based on those strict instructions.
* Managing AI and the Future of Work:
The author expects that regular employees will soon act like managers, overseeing multiple AI helpers. Because of this, workers need to learn basic management skills, like how to delegate tasks and set clear boundaries. Also, since AI will handle routine chores, humans will need to practice focusing for longer periods to do deeper, harder thinking. Ultimately, a worker's value will come from their planning and decision-making skills, rather than their ability to type code.
View details
Dynamic Cogeneration of Bug Reproduction Test in Agentic Program Repair
Michele Tufano
José Cambronero
Renyao Wei
Grant Uy
34th ACM International Conference on the Foundations of Software Engineering (FSE) (2026)
Preview abstract
Bug Reproduction Tests (BRTs) have been used in many Automated Program Repair (APR) systems, primarily for validating fixes and aiding fix generation. In practice, when developers submit a patch, they often implement the BRT alongside the fix. Our experience deploying agentic APR reveals that developers desire a BRT within AI-generated patches to increase their confidence. However, canonical APR systems tend to generate BRTs and fixes separately, and focus on producing only the fix in the final patch. In this paper, we study agentic APR in the context of cogeneration, where the APR agent is instructed to generate both a fix and a BRT in the same patch. We evaluate the effectiveness of different cogeneration strategies on 120 human-reported bugs at Google and characterize different cogeneration strategies by their influence on APR agent behavior. We develop and evaluate patch selectors that account for test change to select patches with plausible fixes (and plausible BRTs). Finally, we analyze the root causes of failed cogeneration trajectories. We show that cogeneration allows the APR agent to generate BRTs for at least as many bugs as a dedicated BRT agent, without compromising the generation rate of plausible fixes, thereby reducing engineering effort in maintaining and coordinating separate generation pipelines for fix and BRT at scale.
View details
×
![]()