Improving performance of defect predictors using confirmation bias metrics

Abstract

Software defect prediction models help managers to prioritize their testing effort. Algorithms, which are used to learn defect predictors, have reached a ceiling such that further improvements may only come from increasing information content of input data. The main goal of this research is to build defect predictors which are learnt from developers’ levels of confirmation bias, which is defined as tendency of people to seek for evidence to verify hypotheses rather than seeking for evidence to refute them. Our main goal is to overcome the ceiling effect in defect prediction performance. As a first step, we define a methodology to measure/quantify confirmation bias and then we perform the following empirical analysis: i) investigation of factors which have the potential to affect confirmation bias levels of software developers and testers, ii) empirical analysis of how confirmation bias affects software developer and tester performance, iii) a benchmark analysis comparing performance of defect predictors which use combinations of static code, confirmation bias and churn metrics, iv) defining a methodology to build defect predictors which can learn using incomplete confirmation bias metric values as input. Our results on industrial data show that: i) no effect of experience in software development/ testing has been observed on confirmation bias, whereas hypotheses testing and reasoning skills affect confirmation bias, ii) confirmation biases of developers lead to an increase in defects, while testers’ confirmation bias causes an increase post-release defects, iii) using only confirmation bias metrics, we can build defect predictors with higher or comparable prediction performance when compared to defect predictors that are learnt by using only churn metrics or only static code metrics, iv) promising results can also be obtained by using incomplete confirmation bias metric values to learn defect predictors.