Quantitative research methods and study quality in learner corpus research

Abstract

This study aims to provide the first empirical assessment of quantitative research methods and study quality in learner corpus research. We systematically review quantitative primary studies referenced in the Learner Corpus Bibliography (LCB), a representative bibliography of learner corpus research maintained by the Learner Corpus Association which contained 1,276 references when the current study began. Each primary study in the LCB was coded for over fifty features representing six dimensions: (a) publication type (i.e. conference paper, book chapter, journal article), (b) research focus (e.g. lexis, grammar), (c) methodological features (e.g. keyword analysis, error analysis, use of reference corpus), (d) statistical analyses (e.g. X², t-test, regression analysis), and (e) reporting practices (e.g. reliability coefficients, means). Results point to several systematic strengths as well as many flaws, such as the absence of research questions, incomplete and inconsistent reporting practices (e.g. means without standard deviations), and lack of statistical literacy (i.e. LCR studies generally overrely on tests of statistical significance, do not report effect sizes, rarely check or report whether statistical assumptions have been met, and rarely use multivariate analyses). Improvements over time, however, are clearly noted and there are signs that, like other related disciplines, learner corpus research is slowly undergoing methodological reform.