The value of mapping studies – A participant-observer case study

Background: We are strong advocates of evidence-based software engineering (EBSE) in general and systematic literature reviews (SLRs) in particular. We believe it is essential that the SLR methodology is being used constructively to support software engineering research. Aim: This study aims to assess the value of mapping studies which are a form of SLR that aims to identify and categorise the available research on a specific topic. Methods: We use a multi-case, participant observer case study using five examples of studies that were based on preceding mapping studies. Results: We identified 13 unique benefits that can accrue from basing research on a preceding mapping study of which only 2 were case specific. We also identified 9 problems associated with using preceding mapping studies of which two were case specific. Conclusions: Mapping studies can save time and effort for researchers and provide baselines to assist new research efforts. However, they must be of high quality in terms of completeness and rigour if they are to be a reliable basis for follow-on research.


INTRODUCTION
In 2004-5, Kitchenham, Dybå and Jørgensen wrote three papers suggesting that the concept of evidence-based practice, (as initially developed in medicine, and subsequently adopted by many different disciplines including economics, psychology, social science and most health care disciplines) should be adopted in software engineering (Kitchenham et al., 2004, Dybå et al. 2005, and Jørgensen et al., 2005).By analogy with medicine, they suggested that evidence-based software engineering (EBSE) should be concerned with the aggregation of empirical evidence and should use systematic literature reviews (SLRs) as a methodology for performing unbiased aggregation of empirical results.Based on the 5 stages in evidence-based medicine, Kitchenham et al. (2004) suggested equivalent stages in EBSE.Stages 1 to 4 are 1) to construct an answerable question; 2) to track down evidence to answer the question; 3) to critically appraise the evidence, and 3) to use the evidence to address the question.Stage 5 is rather different in nature.It is about seeking ways to improve the way in which we undertake evidence-based software engineering and is the rationale for this paper.
One of the main technologies underpinning EBSE is a rigorous procedure for searching research literature called a systematic literature review (SLR).SLRs are secondary studies (i.e.studies that are based on analyzing previous research) used to find, critically evaluate and aggregate all relevant research papers (referred to as primary studies) on a specific research question or research topic.The methodology is intended to ensure that the literature review is unbiased, rigorous and auditable.The basic SLR methodology is similar, irrespective of discipline using it; although medical standards emphasize metaanalysis (a means of statistically aggregating the results from different studies of the same phenomena) more than other disciplines (see for example, Petticrew and Roberts 2005, Khan et al. 2003, Kitchenham and Charters, 2007).
We are currently undertaking a program of case study-based research that is aimed at better understanding the role of systematic literature reviews (SLRs) in software engineering (Brereton et al. 2007).This is part of the Evidence-based Practices Informing Computing (EPIC) project which is funded by the UK Engineering and Physical Sciences Research Council.In this paper we assess the value that mapping studies (also called scoping studies) provide to the research community.Mapping studies use the same basic methodology as SLRs but aim to identify all research related to a specific topic rather than addressing the specific questions that conventional SLRs address (Budgen et al., 2008b).
A standard systematic review is driven by a very specific research question that can be answered by empirical research, for example "Are algorithmic cost models more accurate than expert judgementbased estimates?"(Jørgensen, 2004).This research question drives the identification of appropriate primary studies, informs the data extraction process applied to each included primary study, and determines the aggregation of the extracted data.
In contrast a mapping study reviews a specific software engineering topic and classifies the primary research papers in that specific domain.The research questions for such a study are quite high level and include issues such as which sub-topics have been addressed, what empirical methods have been used, and what sub-topics have sufficient empirical studies for a more detailed systematic review (Petersen et al., 2008, Budgen et al, 2008b).Thus, mapping studies can be of great potential importance to software engineering researchers by providing an overview of the literature in specific topic areas.Although at the extreme, mapping studies and systematic reviews have rather different goals, there is often an overlap.Some systematic reviews include a classification system to organise relevant literature followed by a more detailed description of the research within each category.The important difference is that a conventional systematic review makes an attempt to aggregate the primary studies in terms of the research outcomes and investigates whether those research outcomes are consistent or contradictory.In contrast, a mapping study usually aims only to classify the relevant literature and aggregates studies with respect to the defined categories.Differences between mapping studies and conventional SLRs are summarised in Table 1.
The research question addressed by this case study is: "how do mapping studies contribute to further research?"The "case" in this study is a follow-on research activity that has been based on the results of a mapping study.In order to address the research question we will consider two sub-questions: • RQ.S1 What are the advantages and disadvantages of basing research on a previous mapping study?• RQ.S2 What makes a mapping study suitable for supporting further research activities?

RELATED RESEARCH
In a sense the answer to our research question is self-evident: They provide a categorized citation list that researchers can use to undertake more detailed SLRs.A typical example of this is the mapping study undertaken by Sjøberg et al (2005) that identified 103 human-centric experiments and quasi-experiments published in 13 leading journals and conferences.This set of papers has been used in numerous subsequent systematic reviews that have investigated aspects of empirical software engineering (i.e.Dybå et al., 2006, Hannay and Jørgensen, 2008, Hannay et al., 2007, Kampenes et al., 2007and Kampenes et al., 2009).In this example, the subsequent SLRs were all undertaken by members of the original research team because the list of 103 papers was not published in any of the journal papers.As an example of a different use of mapping study results, Sjøberg provided a random sample of the primary studies which Budgen et al. (2008a) used as the basis of a study of the value of structured abstracts.In contrast to Sjøberg et al.'s study, the mapping study of cost estimation papers undertaken by Jørgensen and Shepperd (2007) cited all the papers found by the study and provided open access to the database holding details about each primary study.The database is available to any interested researchers (www.simula.no/BESTweb)subject to obtaining a user name from Jørgensen.
With respect to sub-question 1, a problem with software engineering mapping studies is already clear, i.e. some mapping studies do not cite all the identified primary studies (e.g.Bailey et al., 2007;Gómez et al., 2006;Bellini et al., 2008).In addition some mapping studies do not report the classification for each primary study (i.e.only report aggregations such as the number of primary studies in each category), for example Catal and Diri (2009).In two studies, Kitchenham et al. (2009aKitchenham et al. ( , 2009c) identified 12 software engineering mapping studies published between 2004 and June 2008 (excluding the Jørgensen and Shepperd study and Sjøberg et al.'s study and their follow-on studies) and in only two cases were both the primary studies all cited and the classifications clearly linked to the specific studies.This limits the value of the study to other researchers (sub-question 2).A common reason for this is space restrictions in conference papers and journals.An interesting example of how to avoid this problem is found in Neto et al. (2008) where details about all the primary studies were included in an appendix published as a Web Extra associated with the specific issue of IEEE Software.

METHOD
We used a case study methodology to investigate our research questions (Yin, 2003).In this example our "case" is a research activity following on directly from a preceding mapping study.We considered a total of five cases, so the study is a multiple-case study.We ourselves were involved in many of the studies, so the case study is a participantobserver study.In addition to the individual studies, we also asked Magne Jørgensen about the way in which the BESTweb database was being used.

Case Selection and Case Study Roles and Procedures
The individual cases comprise follow-on activities undertaken at Durham University and overseen by Budgen, plus both follow-on activities undertaken as part of the EPIC study and overseen by Kitchenham.Kitchenham was responsible for: • Contacting Magne Jørgensen about use of BESTweb.

•
Circulating questionnaires to staff involved in two follow-on studies undertaken as part of the EPIC project.

•
Analyzing the responses to questionnaires • Preparing the case study report.
Budgen was responsible for collecting information concerning three follow-on research activities performed at Durham University.Brereton was responsible for checking the classification of responses and aggregation of information.
Participants in the case studies were asked to complete a questionnaire for each study The questionnaire is shown in Appendix A. One mapping study had two related follow-on activities, but researchers were only asked to complete one form.However, issues that related to only one of the followon activities were identified separately when the data were analyzed.

3.2
Data Analysis and Interpretation Information obtained from Jørgensen directly addresses our overall research question.The questionnaire responses were analyzed to summarise: • The individual characteristics of each case considering both the original mapping study and the follow-on activity.This allows us to identify whether there are any contextual issues that influence follow-on activities and addresses subquestion 2 • Any problems that accrue from follow-on research.This directly addresses sub-question 1. • Any benefits that accrue from a previous mapping study.
When analyzing the data we separated generic benefits and problems, from benefits and problems that were case specific.For each problem we suggest some remedial action.

Response from Jørgensen
In terms of advantages of mapping studies, BESTweb has been the basis of other studies by Jørgensen and Shepperd, and  • There were 120 users of the BESTweb.

•
He and his co-worker at the Simula Laboratory use the BESTweb library in most studies to find relevant studies for the related work section (which is sort of a mini-review).

•
He had used BESTweb in three papers where a review was a separate part of the research.

•
He recalled that several researchers had commented on the usefulness of the database.
In particular, two PhD students said they had saved a lot of work through this database.
In a subsequent communication (31/10/09) Jørgensen reported that Ira Monarch and colleagues at SEI were data mining the BESTweb abstracts as a means to do "reviews"/meta-studies with the purpose of finding connections between research topics in effort estimation.

Case Study details
The five case studies are described in Table 2.The positive issues raised by the researchers are presented in Table 3 and the negative issues are presented in Table 4.In terms of coverage of initial mapping study contextual factors: • We have studies undertaken by both experienced and novice researchers.

•
In none of the cases was the follow-on research activity anticipated when the initial mapping study was undertaken.

•
Our cases include studies that extended a previous mapping study; a study that was concerned with methodology; and studies that undertook a more detailed study of the literature.However, we do not have an example of a follow-on activity comprising a conventional SLR, although the UML follow-on did aggregate outcomes of metric studies.

•
With respect to involvement in the follow-on study, in three cases the follow on study was performed by the same lead researcher, in two cases it was not.However, in all cases some of the researchers were the same.Thus, we do not have an example of a mapping study follow-on performed by a completely independent research group.
In terms of response to our questionnaire, two researchers out of a total of 11 did not complete the forms.By an oversight, one was not invited to complete a questionnaire.We do not know why the other researcher failed to complete the questionnaire.This affected three of the case studies because one of the researchers worked on two follow-on activities.It must also be noted that in several cases researchers completed forms for more than one case study In terms of the benefits of mapping studies, there were 36 reports of benefits addressing 13 separate types of benefit.However, 17 of the 36 reports related to the two benefits explicitly mentioned in the questionnaire (see Section 3.1) and five of the participants did not identify any benefits other than those suggested in the questionnaire.Researcher R1 identified 8 additional benefits, R9 identified 6, R5 identified three, and R2 and R7 identified one each.Two of the additional benefits were specific to the particular follow-on activity, and another two were limited to updating a mapping study.
In terms of the problems associated with follow-on activities, there were 10 reports of problems corresponding to 9 individual types of problem.Seven were general problems while two were specific to a particular follow-on activity.Only four of the 11 researchers identified potential problems, with a single researcher contributing 6 of the 10 reports.

DISCUSSION
This section answers our research questions and identifies limitations of the study.

RQ.S1 The advantages and disadvantages of basing research on a previous mapping study
Most researchers agreed that the use of a preceding mapping study both saved time for subsequent studies and provided an understanding of the literature.However, they noted a number of other advantages: • In the case of an extension to a mapping study, the protocol may be reusable; if the same researchers undertake the study they will be experienced with the procedures; time trends can be investigated; and a set of known studies exists which can be used to validate search strings used for automated searches.

•
The mapping study can identify clusters of research studies that are suitable for more detailed study.

•
The mapping study may identify the need for more primary studies.

•
The researchers themselves gain a very good overview of the literature.
Researchers identified fewer disadvantages than advantages and one case (the follow-on from the Design Patterns study) reported no disadvantages.Nonetheless some important problems were identified: • If the search process used by the original mapping study was restricted, there are likely to be limitations to the original study that will negatively impact subsequent research activities.For example in two of our cases the searches were restricted because the original mapping studies were MSc projects that needed to be completed in a short time period.A particular problem that can occur in this situation is missing primary studies due either to search string limitations or lack of snowballing for extra studies.In addition, the classification scheme may be over-simplistic or incorrectly used.

•
If the follow-on research activity is an extension of the preceding mapping study, it is important to be aware of process changes that could reduce the comparability of the initial and follow-on mapping studies.

•
Mapping studies may underestimate the number of primary studies needed for subsequent SLRs.This occurs when multiple primary studies are reported in a single paper.This is of particular importance if a mapping study is being used as a means to assist resource estimation for future SLRs.It is equally important to be sure that the policy for dealing with duplicate reports of the same study is appropriate i.e. whether duplicate reports have been identified and appropriately handled.For example, neither the Jørgensen and Shepperd mapping study (2007) nor the Sjøberg et al. (2005) mapping study removed primary studies that reported the same study.This may be acceptable for a mapping study but might not be for a follow-on activity, particularly an SLR.

•
If follow-on activities are delayed, even a high quality mapping study will need to be brought up to date before the follow-on research can be started.

RQ.S2 What makes a mapping study suitable for supporting further research activities
We have already noted that a mapping study cannot readily support further research activities, except by the original researchers, unless all the references are cited and the classification information for each study is reported.An additional issue is that the mapping study must be of high quality.It must have been based on: • A stringent search process including automated searches, manual searches of critical sources (particularly topic specific conferences), snowballing of primary study references, and direct communication with important researchers and research groups.

•
A well-defined and reliable classification system.
If the study is not known to be of high quality, or is known to have been undertaken in a restricted timescale, the mapping study report needs to be reviewed critically to confirm its suitability as a basis for further research activities.In the event that the mapping study is not of sufficient quality, it should be used as the starting point for a more detailed mapping study.In particular, it will provide a list of known primary studies against which subsequent searches and automated search strings can be validated.

RQ How do studies contribute to further research?
If a high quality fully reported mapping study has been performed, there are many ways in which the outcomes can be used, in addition to undertaking conventional SLRs based on clusters of related primary studies.A high quality mapping study can be used as: • A baseline against which research trends can be tracked over time.

•
A justification for further primary studies when there are few (or no) relevant empirical studies.

•
As a means of identifying relevant literature for the "related research" section in other primary studies (section 4.1).

•
A baseline for empirical research of various kinds (e.g.Budgen et al., 2009, and section 4.1).

•
A set of known references which other mapping studies and SLRs can use to validate their own searches.

Study Limitations
Most of the data discussed in this paper comes from mapping studies and follow-on activities undertaken by EPIC team members.Since we are generally in favour of systematic mapping studies there is a danger that we would be more likely to report benefits than problems.Furthermore, the questionnaire was very simple but the positive and negative questions were slightly unbalanced.The question related to benefits offered some examples, which were readily accepted by most of the respondents whereas the question related to problems did not have any examples.This lack of balance was necessary because it was simple to identify potential benefits but we had no clear indication of what would constitute potential problems.However, since we identified 9 unique problems compared with 13 unique benefits, the lack of balance does not seem to have led to a serious failure to identify problems.
In terms of generality, our case studies are restricted to studies where researchers involved in an initial mapping study were also involved in a subsequent research activity (although the lead researcher changed in two cases).The large number of authors citing the Jørgensen, and Shepperd paper suggest that independent researchers can utilize highquality mapping studies successfully.However, we would expect problems, if they occur, to be more severe.For example, Beecham (2009) needed to undertake a very detailed comparison to understand why a mapping study she performed found a different set of primary studies to those found by another mapping study with the same basic research question.

CONCLUSIONS
Our results indicate that mapping studies can be of significant benefit to researchers in establishing baselines for further research activities.A baseline can be used in a variety of ways, either as the starting point for investigating research trends or the starting point for conventional SLRs.However, it is important to recognize that although mapping studies may claim to follow a rigorous research process, not all follow the process closely enough to ensure that their results are trustworthy.In particular, although undertaking mapping studies may provide a useful educational experience for MSc students (Kitchenham et al., 2009d), the outcomes of mapping studies performed under the time and effort restrictions imposed on students may be limited.Thus, researchers intending to use a preceding mapping study as the basis for further research must take care to critically review the quality and suitability of the mapping study research procedures before depending on its results.