Franjo Ivancic
Franjo's research interests include software engineering, automated software testing, static and dynamic program analysis, software verification, model checking, as well as formal modeling and analysis of cyber-physical systems. Before joining Google, he was a Senior Researcher at NEC Laboratories America in Princeton, NJ from 2003-2013. He received his Ph.D. and MSE degrees in Computer and Information Science from the University of Pennsylvania in Philadelphia, PA. Earlier, he received his diploma (Dipl.-Inform.) degree from the Rheinische Friedrich-Wilhelms-University in Bonn, Germany, for his research performed at the Fraunhofer Institute in St. Augustin, Germany. He received the Morris and Dorothy Rubinoff dissertation award from the University of Pennsylvania. More information is available at www.franjo-ivancic.info.
Authored Publications
Sort By
Towards A Human-in-the-Loop Framework for Reliable Patch Evaluation using an LLM-as-a-Judge
Renyao Wei
Michele Tufano
José Cambronero
AI-SQE '26: Proceedings of the 1st International Workshop on AI for Software Quality Evaluation - Judgment, Metrics, Benchmarks, and Beyond, ACM (Association for Computing Machinery), New York, NY, USA (2026), pp. 19 - 28
Preview abstract
Reliable evaluation is crucial for advancing Automated Program Repair (APR), but prevailing benchmarks that rely on execution-based evaluation methods (pass@k) often fail to capture the patch quality required for real-world adoption. This creates a significant gap between automated metrics and true patch validity (valid@k), a discrepancy observed across several state-of-the-art techniques.
To develop a scalable solution for measuring valid@k, we first study the human evaluation process itself. While manual assessment can determine validity, we find it suffers from poor inter-rater reliability (Fleiss' Kappa k=0.307). Our foundational insight is that this inconsistency is largely resolved when evaluators use a shared, high-quality rubric, which significantly improves agreement.
Building on this finding, we propose an LLM-as-a-Judge framework that operationalizes rubric-guided evaluation at scale. Our method employs a human-in-the-loop workflow where an LLM first generates a candidate rubric for a given bug, which a human expert then reviews and refines into a "golden" evaluation standard. This golden rubric is then used by an LLM judge to assess the validity of candidate patches. In an evaluation on 48 bugs and 115 patches, our LLM judge demonstrates substantial agreement with the consensus of human developers. This work contributes a scalable and reliable methodology for approximating valid@k, providing a much-needed high-fidelity signal for measuring true progress in the field of automated program repair.
View details
Dynamic Cogeneration of Bug Reproduction Test in Agentic Program Repair
Michele Tufano
José Cambronero
Renyao Wei
Grant Uy
2026
Preview abstract
Bug Reproduction Tests (BRTs) have been used in many Automated Program Repair (APR) systems, primarily for validating fixes and aiding fix generation. In practice, when developers submit a patch, they often implement the BRT alongside the fix. Our experience deploying agentic APR reveals that developers desire a BRT within AI-generated patches to increase their confidence. However, canonical APR systems tend to generate BRTs and fixes separately, and focus on producing only the fix in the final patch. In this paper, we study agentic APR in the context of cogeneration, where the APR agent is instructed to generate both a fix and a BRT in the same patch. We evaluate the effectiveness of different cogeneration strategies on 120 human-reported bugs at Google and characterize different cogeneration strategies by their influence on APR agent behavior. We develop and evaluate patch selectors that account for test change to select patches with plausible fixes (and plausible BRTs).
Finally, we analyze the root causes of failed cogeneration trajectories. We show that cogeneration allows the APR agent to generate BRTs for at least as many bugs as a dedicated BRT agent, without compromising the generation rate of plausible fixes, thereby reducing engineering effort in maintaining and coordinating separate generation pipelines for fix and BRT at scale.
View details
Zero-Config Fuzzing for Microservices
Wei Wang (王薇)
Andrei Benea
38th IEEE/ACM International Conference on Automated Software Engineering (2023)
Preview abstract
The microservice paradigm is a popular software development pattern that breaks down a large application into smaller, independent services. While this approach offers several advantages, such as scalability, agility, and flexibility, it also introduces new security challenges. This paper presents a novel approach to securing microservice architectures using fuzz testing. Fuzz testing is known to find programming errors in software by feeding it with unexpected or random inputs. In this paper, we propose a zero-config fuzz test generation technique for microservices that can maximize coverage of internal states by mutating both the incoming requests and the backend responses from dependent services. We successfully deployed our technique to over 95 % of C++ services built on Google's internal microservice platform. It reported and got fixed thousands of errors in real-world microservice applications.
View details
Reducing Time-To-Fix For Fuzzer Bugs
Rui Abreu
Hadi Ravanbakhsh
Ramesh Viswanathan
36th IEEE/ACM International Conference on Automated Software Engineering (2021)
Preview abstract
At Google, fuzzing C/C++ libraries has discovered tens of thousands of security and robustness bugs. However, these bugs are often reported much after they were introduced. Developers are provided only with fault-inducing test inputs and replication instructions that highlight a crash, but additional debugging information may be needed to localize the cause of the bug. Hence, developers need to spend substantial time debugging the code and identifying commits that introduced the bug. In this paper, we discuss our experience with automating a fuzzing-enabled bisection that pinpoints the commit in which the crash first manifests itself. This ultimately reduces the time critical bugs stay open in our code base. We report on our experience over the past year, which shows that developers fix bugs on average 2.23 times faster when aided by this automated analysis.
View details
SunDew: Systematic Automated Security Testing
2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST), IEEE, {3-3}
Preview abstract
At Google, tens of thousands of security and robustness bugs have been found by fuzzing C and C++ libraries. The various aspects of the SunDew project, one of the projects working on automated scalable techniques related to fuzzing at Google, are presented: how to fuzz, what to fuzz, and how to deal with discovered bugs. First, a distributed fuzzing infrastructure is presented. It allows to cooperatively utilize multiple test generation techniques. Then, a system for automated fuzz driver generation, named FUDGE, is described, which automatically generates fuzz driver candidates for libraries based on existing client code. Running large-scale fuzzing services also causes lots of bugs and vulnerabilities to be reported. Various techniques are presented to provide feedback to developers to reduce the time a known security bug remains open. Finally, challenges and opportunities to incorporate security testing into the general software development workflow are highlighted.
View details
FUDGE: Fuzz Driver Generation at Scale
Domagoj Babic
Yaohui Chen
Tim King
Markus Kusano
Caroline Lemieux
Wei Wang
Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ACM
Preview abstract
At Google we have found tens of thousands of security and robustness bugs by fuzzing C and C++ libraries. To fuzz a library, a fuzzer requires a fuzz driver—which exercises some library code—to which it can pass inputs. Unfortunately, writing fuzz drivers remains a primarily manual exercise, a major hindrance to the widespread adoption of fuzzing. In this paper, we address this major hindrance by introducing the Fudge system for automated fuzz driver generation. Fudge automatically generates fuzz driver candidates for libraries based on existing client code. We have used Fudge to generate thousands of new drivers for a wide variety of libraries. Each generated driver includes a synthesized C/C++ program and a corresponding build script, and is automatically analyzed for quality. Developers have integrated over 200 of these generated drivers into continuous fuzzing services and have committed to address reported security bugs. Further, several of these fuzz drivers have been upstreamed to open source projects and integrated into the OSS-Fuzz fuzzing infrastructure. Running these fuzz drivers has resulted in over 150 bug fixes, including the elimination of numerous exploitable security vulnerabilities.
View details
Replay without Recording of Production Bugs for Service Oriented Architectures
Nipun Arora
Jonathan Bell
Gail Kaiser
Baishakhi Ray
Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering, ACM (2018), pp. 452-463
Preview abstract
Short time-to-localize and time-to-fix for production bugs is extremely important for any 24x7 service-oriented application (SOA). Debugging buggy behavior in deployed applications is hard, as it requires careful reproduction of a similar environment and workload. Prior approaches for automatically reproducing production failures do not scale to large SOA systems. Our key insight is that for many failures in SOA systems (e.g., many semantic and performance bugs), a failure can automatically be reproduced solely by relaying network packets to replicas of suspect services, an insight that we validated through a manual study of 16 real bugs across five different systems. This paper presents Parikshan, an application monitoring framework that leverages user-space virtualization and network proxy technologies to provide a sandbox “debug” environment. In this “debug” environment, developers are free to attach debuggers and analysis tools without impacting performance or correctness of the production environment. In comparison to existing monitoring solutions that can slow down production applications, Parikshan allows application monitoring at significantly lower overhead.
View details
ARC++: Effective Typestate and Lifetime Dependency Analysis
Xusheng Xiao
Naoto Maeda
Aarti Gupta
Deepak Chhetri
ISSTA, ACM (2014), pp. 116-126
