Meta-analysis in Second Language Research: Choices and Challenges

We describe the history and current status of the meta-analytic enterprise. The advantages and historical criticisms of meta-analysis are described, as are the basic steps in a meta-analysis and the role of effect sizes as chief coins of the meta-analytic realm. Advantages of the meta-analytic procedures include seeing the "landscape" of a research domain, keeping statistical significance in perspective, minimizing wasted data, becoming intimate with the data summarized, asking focused research questions, and finding moderator variables. Much of the criticism of meta-analysis has been based on simple misunderstanding of how meta-analyses are actually carried out. Criticisms of meta-analysis that are applicable are equally applicable to traditional, nonquantitative, narrative reviews of the literature. Much of the remainder of the chapter deals with the processes of effect size estimation, the understanding of the heterogeneity of the obtained effect sizes, and the practical and scientific importance of the effect sizes obtained.

Wider use in psychology of confidence intervals (CIs), especially as error bars in figures, is a desirable development. However, psychologists seldom use CIs and may not understand them well. The authors discuss the interpretation of figures with error bars and analyze the relationship between CIs and statistical significance testing. They propose 7 rules of eye to guide the inferential use of figures with error bars. These include general principles: Seek bars that relate directly to effects of interest, be sensitive to experimental design, and interpret the intervals. They also include guidelines for inferential interpretation of the overlap of CIs on independent group means. Wider use of interval estimation in psychology has the potential to improve research communication substantially. ((c) 2005 APA, all rights reserved).