Scrum and Productivity in Software Projects: a Systematic Literature Review

Background: Agile methods have increasingly attracted interest in the Software Industry. SCRUM is currently one of the most studied agile method, because of both its novelty and the assumption that SCRUM is able to improve project productivity. Objective/Method: This article describes a systematic literature review, which aims to find scientific evidence of the correlation between the use of SCRUM and productivity in Software Projects. Results/Conclusion: Among 274 primary studies, this research selected 28 papers presenting strong evidence on the research questions. According to these results, the relation between SCRUM and productivity may be positive.


INTRODUCTION
Agile methods have increasingly attracted interest in the Software Industry.A survey performed in the United Stated and Europe has shown that 14% of software companies are already using agile methods, and 49% of the companies which know something about agile methods are interested in adopting them.[13].This increasing interest may be due to a belief that Agile Methods are powerful development alternatives, supposedly able to avoid project problems such as low productivity, schedule delays, high costs, lack of people motivation, and others, which still are significant problems in the Software Engineering field [13].
SCRUM is currently one of the most studied agile method, because of both its novelty and the assumption that SCRUM is able to improve project productivity.SCRUM is an agile project management method, which uses an empirical process based on flexibility, adaptability and productivity.SCRUM sets high-frequency management activities in order to track real-time problems during a project development process.SCRUM is usually adopted in organizations which have not achieved success by using traditional development or management methods, and it is also usually combined to other development methodologies, such as XP, according to the organization needs.This article describes a systematic literature review, which aims to find scientific evidence of the correlation between the use of SCRUM and productivity in Software Projects.Even though there are studies such as [13] seeking for evidence on the benefits, limitations and implications of using agile methods in software projects, those studies are mainly about Extreme Programming.
This research follows guidelines defined by [16,17] and looks to the literature to answer the following research questions:

SYSTEMATIC LITERATURE REVIEW
This research was conducted in three main steps: planning, execution and data analysis [15].In the first step, the research protocol was designed, including the definition of the queries terms, choice of scientific resources databases, design of the document selection criteria, and quality assessment procedures.In the second step, the queries were executed, the material was read and selected according to its relevance, and then they were assessed according to their quality.The last step was the data analysis, where the results were organized according to each research question, in order to answer them.The three research steps are better described in the following sessions.

Planning
The search terms used in this SLR were developed using the same steps as in [16] and [17], and are summarized in Table 1.After executing the search query, the resulting documents were independently selected by three different researchers, according to the procedure described in Table 2.The selection criteria are also specified in Table 3.

Table 2: Document Selection Procedure
Step 1.1 The three researchers (R1, R2, R3) independently read all titles and abstracts.Then, they selected only the relevant articles and discarded the others.
Step 1.2 The three researchers (R1, R2, R3) independently read the introduction and conclusion of the chosen articles.Then, they selected only the relevant articles and discarded the others.
Step 1.3 The articles selected so far were compiled, in order to remove duplications.Each article was completely read by two researchers (R1+R2, R1+R3, R2+R3).
Step 1.4 The researchers filled a Quality Assessment form for each article, and elaborated a summary containing their key-points such as objectives and findings.
In order to assess the quality of the selected articles, a Likert-scale form was designed covering five characteristics: research validity, threats to validity, relevance, applicability, and consistency of the presented evidence.Each researcher has assessed all selected articles, so the article grade is the sum of the two grades, assigned in the Step 1.4.After grading, the articles were classified in four types: Very Good (between 31-40), Good (between 21-30), Regular (between [11][12][13][14][15][16][17][18][19][20] and Poor (between 0-10).

Table 3: Document Selection Criteria
The articles must be available among the selected resources; be written in English.
have some study about SCRUM.present some result on the relation between the use of SCRUM and software projects productivity.clearly describe its methodology.... have sufficient data for the analysis.... have been completed and concluded.... be unique (have no duplication).
It is important to report that the researchers also had ad-hoc help from specialists in agile methodologies during the document selection steps, which has added some quality and reliability to this selection process.

Execution
The first search retrieved 274 studies from the chosen scientific databases.After performing the document selection procedure, 28 relevant articles were selected.Table 4 shows the distribution of these articles among the resources.This SLR did not restrict the period of publications, although all selected studies were carried out between 2000 and 2009, as shown in Figure 1, which therefore portraits the relevance this theme has recently acquired.

Figure 1: Temporal Distribution of selected studies
Table 5 shows the final classification of the selected articles, according to their respective grades.It is important to notice in Table 5 there are no Poor articles, which means that the found evidence have a reasonable level of reliability.This fact also confirms the efficacy of the document selection procedure.

Data Analysis
Among the selected studies, this research found relevant evidence to satisfactorily answer its two research questions, as described below.
(RQ1) Is the use of SCRUM related to productivity improvement in software development projects?
The productivity improvement was discussed in 14 out of the 28 selected articles, from which 5 are Very Good, 7 are Good and 2 are Regular.It indicates that the answer of this question may be positive.Even though each study has its particularities, all of them have evidenced some productivity benefit in software engineering projects using SCRUM methodology.It is important to notice that most SCRUM implementations are usually combined with another software development methodology or process model.Therefore, further research may identify what others methods, which combined to SCRUM, have generated evidence of productivity improvement in software development projects.
(RQ2) What are the other performance outcomes, which might be confounded with productivity, related to the use of SCRUM in software development projects?
Among the selected articles, the productivity was the most frequently cited outcome of using SCRUM methodology.However, some relevant evidence refer to outcomes also related to project performance which may be confounded to productivity, such as customer satisfaction, product and process quality, team motivation, and cost reduction.They are properly summarized in Table 6.This study did not find sufficient evidence to consider other project aspects, such as communication, reliability, cohesion, and business value, as a significant outcome, even though they seem to be incidental benefits, of using SCRUM [4,10,13,19,20,21,23].This research data is still being analyzed in order to address other complementary questions which could not be covered in this short paper, such as identify the context of organizations, project, teams and people in which SCRUM have been related to productivity and also what are the other methodologies and models which have been combined to SCRUM and have contributed to productivity.
It is important to notice, however, that most of the evidence on this theme is recent.A possible explanation for having only a few relevant studies in 2009 is that relevant publications and proceedings have not yet been indexd in the search engines.Hence, even though the selected studies are considered good evidence, further research should replicate this study, covering a broader sample of studies, which would enable wider analyses and evidence comparison.Accessing only five sources can be considered as a threat to validity of his research, because many other sources seem to be relevant but, for access restrictions, their papers could not be retrieved and analyzed.Future research must address this problem too.

Table 1 :
Research Terms Then, the search was performed only in those search engines and digital libraries available on the Internet and which have partnership with the Universidade Federal de Pernambuco (Federal University of Pernambuco): ACM Portal, IEEE, Scopus, EI Compendex and Science Direct.

Table 4 :
Distribution of studies among resources

Table 6 :
Outcomes of using SCRUM methodology