Research Paper Proposes Improved Performance Anomaly Detection Methods for Mozilla

A new research paper submitted to arXiv examines statistical change point detection techniques as potential improvements to Mozilla's current performance anomaly detection system. The study, authored by Diego Elias Costa and colleagues, addresses limitations in Perfherder, Mozilla's performance engineering management system that uses a Student's T-test-based approach to identify software regressions across hundreds of daily code changes.

The researchers analyzed one year of Mozilla performance data and identified significant gaps in the current detection method: 12.5% of generated alert groups are false positives, while approximately 6.8% contain regressions that the automated system missed. These findings motivated the empirical evaluation of 25 change-point detection (CPD) methods and 15 ensemble approaches as alternatives to Mozilla's existing approach.

To support their research, the team constructed a ground-truth dataset comprising 174 performance time series manually annotated by eleven Mozilla performance engineers. This dataset represents one of the first practitioner-annotated benchmarks for change point detection in performance engineering. The experimental evaluation revealed that while offline and hybrid CPD methods improve recall compared to Mozilla's current method, they do so at the cost of reduced precision.

The researchers found that ensemble voting strategies provide a more effective solution, achieving an 11% improvement in F1-score over Mozilla's existing approach while offering more consistent performance across different scenarios. The team validated their findings through a practitioner survey and documented lessons learned from integrating the best-performing methods into Mozilla's performance engineering system. This work aims to enhance Mozilla's ability to detect software performance regressions more accurately while reducing false alerts in continuous integration pipelines.