This award is funded in whole or in part under the American Rescue Plan Act of 2021 (Public Law 117-2).

Software’s increasing impact on daily life means there is a need high quality, correct software. The most common approach to check for software correctness is through software testing. The implicit assumption is that a failing test signals the presence of a bug. However, this fundamental assumption is invalidated by the presence of flaky tests, which are tests that nondeterministically pass or fail when run on the same code. Intuitively, there is a nondeterministic component in the code under test or environment that is expected or even desired (and therefore not a true bug in the software), but a developer may have written a flaky test that does not take this nondeterminism into account, resulting in the test to sometimes pass and sometimes fail. When there are flaky tests in the test suite, a developer can no longer trust their test results. If the test failures no longer guarantee the presence of a true bug, developers may then choose to ignore all test outcomes instead of investigating further. At this point, testing become effectively useless, and developers risk critical bugs slipping into the released software. Flaky tests are a common problem faced by developers in industry and the open-source community.

The goal of this project is to mitigate the negative effects of flaky tests. To accomplish this goal, this project will (1) develop techniques to more efficiently detect flaky tests by leveraging code evolution, (2) develop specialized testing environments that can reduce flaky test failures without needing to modify code under test, and (3) enhance software engineering techniques that rely on testing to become flaky-test aware so as to not be misled by unreliable test failures. Beyond traditional flaky tests, this project will also (4) investigate and develop techniques to detect and repair tests that are flaky with respect to nonfunctional properties, such as execution time. Finally, given the parallels between flaky tests and difficulties in computational reproducibility, this project will also (5) develop techniques to increase the reproducibility of computational research results based on insights from the work on flaky tests.