On Searching Relevant Studies in Software Engineering

BACKGROUND: Systematic Literature Review (SLR) has become an important research methodology in software engineering since 2004. One critical step in applying this methodology is to design and execute appropriate and effective search strategy. This is quite time consuming and error-prone step, which needs to be carefully planned and implemented. There is an apparent need of a systematic approach to designing, executing, and evaluating a suitable search strategy for optimally retrieving the target literature from digital libraries. OBJECTIVE: The main objective of the research reported in this paper is to improve the search step of doing SLRs in SE by devising and evaluating systematic and practical approaches to identifying relevant studies in SE. OUTCOMES: We have systematically selected and analytically studied a large number of papers to understand the state-of-the-practice of search strategies in EBSE. Having identified the limitations of the current ad-hoc nature of search strategies used by SE researchers for SLR, we have devised a systematic approach to developing and executing optimal search strategies in SLRs. The proposed approach incorporates the concept of ‘quasi-gold standard’, which consists of collection of known studies and corresponding ‘quasi-sensitivity’ into the search process for evaluating search performance. We report the case study and its finding to demonstrate that the approach is able to improve the rigor of search process in an SLR, and can serves as the supplements to the guidelines for SLRs in EBSE. We plan to further evaluate the proposed approach using several case studies with varying topics in software engineering.