John Penix

John Penix

Works on deploying automated tools into developer workflows. He is on the Steering Committee of the IEEE/ACM International Conference on Automated Software Engineering, and has worked in the Automated Software Engineering R&D group at NASA's Ames Research Center before joining Google. He holds a Ph.D. in Computer Engineering from the University of Cincinnati.

Research Areas

Authored Publications
Sort By
  • Title
  • Title, descending
  • Year
  • Year, descending
    Techniques for improving regression testing in continuous integration development environments
    Sebastian Elbaum
    Gregg Rothermel
    Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, ACM (2014), pp. 235-245
    Preview abstract In continuous integration development environments, software engineers frequently integrate new or changed code with the mainline codebase. This can reduce the amount of code rework that is needed as systems evolve and speed up development time. While continuous integration processes traditionally require that extensive testing be performed following the actual submission of code to the codebase, it is also important to ensure that enough testing is performed prior to code submission to avoid breaking builds and delaying the fast feedback that makes continuous integration desirable. In this work, we present algorithms that make continuous integration processes more cost-effective. In an initial pre-submit phase of testing, developers specify modules to be tested, and we use regression test selection techniques to select a subset of the test suites for those modules that render that phase more cost-effective. In a subsequent post-submit phase of testing, where dependent modules as well as changed modules are tested, we use test case prioritization techniques to ensure that failures are reported more quickly. In both cases, the techniques we utilize are novel, involving algorithms that are relatively inexpensive and do not rely on code coverage information -- two requirements for conducting testing cost-effectively in this context. To evaluate our approach, we conducted an empirical study on a large data set from Google that we make publicly available. The results of our study show that our selection and prioritization techniques can each lead to cost-effectiveness improvements in the continuous integration process. View details
    Introducing Continuous Systematic Testing of Evolving Software
    Mary Jean Harrold
    Darko Marinov
    Stephen Oney
    Mauro Pezzè
    Adam Porter
    Per Runeson
    Shin Yoo
    Dagstuhl Seminar Proceedings, Schloss Dagstuhl-Leibniz-Zentrum für Informatik (2010)
    Preview abstract In today's evolutionary development of software, continuous testing is needed to ensure that the software is still functioning after changes. Test automation helps partly managing the large number of executions needed, but there is also a limit for how much automated tests may be executed. Then systematic approaches for test selection are needed also for automated tests. This manuscript defines this situation and outlines a general method and tool framework for its solution. Experiences from different companies are collected to illustrate how it may be set into practice. View details
    Experiences Using Static Analysis to Find Bugs
    Nathaniel Ayewah
    David Hovemeyer
    J. David Morgenthaler
    William Pugh
    IEEE Software, 25 (2008), pp. 22-29
    Preview abstract Static analysis examines code in the absence of input data and without running the code, and can detect potential security violations (e.g., SQL injection), runtime errors (e.g., dereferencing a null pointer) and logical inconsistencies (e.g., a conditional test that cannot possibly be true). While there is a rich body of literature on algorithms and analytical frameworks used by such tools, reports describing experiences with such tools in industry are much harder to come by. In this paper, we describe FindBugs, an open source static analysis tool for Java, and experience using it in production settings. FindBugs does not push the envelope in terms of the sophistication of its analysis techniques. Rather, it is designed to evaluate what kinds of defects can be effectively detected with relatively simple techniques and to help us understand how such tools can be incorporated into the software development process. FindBugs has become very popular, downloaded more than 500,000 times and used by many major companies and software projects. We report on experience running FindBugs against Sun’s JDK implementation, using Findbugs at Google where it has been used for more than a year and a half and incorporated into their standard development process, and preliminary results from a survey of FindBugs users. View details
    Predicting Accurate and Actionable Static Analysis Warnings: An Experimental Approach
    Joseph Ruthruff
    J. David Morgenthaler
    Sebastian Elbaum
    Gregg Rothermel
    Proceedings of the International Conference on Software Engineering, ACM (2008), pp. 341-350
    Preview abstract Static analysis tools report software defects that may or may not be detected by other verification methods. Two challenges complicating the adoption of these tools are spurious false positive warnings and legitimate warnings that are not acted on. This paper reports automated support to help address these challenges using logistic regression models that predict the foregoing types of warnings from signals in the warnings and implicated code. Because examining many potential signaling factors in large software development settings can be expensive, we use a screening methodology to quickly discard factors with low predictive power and cost-effectively build predictive models. Our empirical evaluation indicates that these models can achieve high accuracy in predicting accurate and actionable static analysis warnings, and suggests that the models are competitive with alternative models built without screening. View details
    Using FindBugs on Production Software
    Nathaniel Ayewah
    J. David Morgenthaler
    William Pugh
    YuQian Zhou
    Proc. OOPSLA'07, ACM, Montréal (2007)
    Preview
    Evaluating Static Analysis Defect Warnings on Production Software
    Nathaniel Ayewah
    William Pugh
    J. David Morgenthaler
    YuQian Zhou
    Proceedings of the 7th ACM SIGPLAN-SIGSOFT workshop on Program analysis for software tools and engineering, ACM Press, New York, NY, USA (2007), pp. 1-8
    Preview abstract Classification of static analysis warnings into false positive, trivial or serious bugs: Experience on Java JDK and Google codebase View details