Merits of organizational metrics in defect prediction: An industrial replication

Abstract

Defect prediction models presented in the literature lack generalization unless the original study can be replicated using new datasets and in different organizational settings. Practitioners can also benefit from replicating studies in their own environment by gaining insights and comparing their findings with those reported. In this work, we replicated an earlier study in order to investigate the merits of organizational metrics in building defect prediction models for large-scale enterprise software. We mined the organizational, code complexity, code churn and pre-release bug metrics of that large scale software and built defect prediction models for each metric set. In the original study, organizational metrics were found to achieve the highest performance. In our case, models based on organizational metrics performed better than models based on churn metrics but were outperformed by pre-release metric models. Further, we verified four individual organizational metrics as indicators for defects. We conclude that the performance of different metric sets in building defect prediction models depends on the project’s characteristics and the targeted prediction level. Our replication of earlier research enabled assessing the validity and limitations of organizational metrics in a different context.