27
views
0
recommends
+1 Recommend
0 collections
    0
    shares
      • Record: found
      • Abstract: found
      • Article: found
      Is Open Access

      Hidden multiplicity in exploratory multiway ANOVA: Prevalence and remedies

      research-article

      Read this article at

      Bookmark
          There is no author summary for this article yet. Authors can add summaries to their articles on ScienceOpen to make them more accessible to a non-specialist audience.

          Abstract

          Many psychologists do not realize that exploratory use of the popular multiway analysis of variance harbors a multiple-comparison problem. In the case of two factors, three separate null hypotheses are subject to test (i.e., two main effects and one interaction). Consequently, the probability of at least one Type I error (if all null hypotheses are true) is 14 % rather than 5 %, if the three tests are independent. We explain the multiple-comparison problem and demonstrate that researchers almost never correct for it. To mitigate the problem, we describe four remedies: the omnibus F test, control of the familywise error rate, control of the false discovery rate, and preregistration of the hypotheses.

          Related collections

          Most cited references29

          • Record: found
          • Abstract: not found
          • Article: not found

          Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing

            Bookmark
            • Record: found
            • Abstract: not found
            • Article: not found

            Empirical Bayes Analysis of a Microarray Experiment

              Bookmark
              • Record: found
              • Abstract: found
              • Article: not found

              An Agenda for Purely Confirmatory Research.

              The veracity of substantive research claims hinges on the way experimental data are collected and analyzed. In this article, we discuss an uncomfortable fact that threatens the core of psychology's academic enterprise: almost without exception, psychologists do not commit themselves to a method of data analysis before they see the actual data. It then becomes tempting to fine tune the analysis to the data in order to obtain a desired result-a procedure that invalidates the interpretation of the common statistical tests. The extent of the fine tuning varies widely across experiments and experimenters but is almost impossible for reviewers and readers to gauge. To remedy the situation, we propose that researchers preregister their studies and indicate in advance the analyses they intend to conduct. Only these analyses deserve the label "confirmatory," and only for these analyses are the common statistical tests valid. Other analyses can be carried out but these should be labeled "exploratory." We illustrate our proposal with a confirmatory replication attempt of a study on extrasensory perception.
                Bookmark

                Author and article information

                Contributors
                aoj.cramer@gmail.com
                Journal
                Psychon Bull Rev
                Psychon Bull Rev
                Psychonomic Bulletin & Review
                Springer US (New York )
                1069-9384
                1531-5320
                15 September 2015
                15 September 2015
                2016
                : 23
                : 640-647
                Affiliations
                [ ]Psychological Methods, Department of Psychology, University of Amsterdam, Amsterdam, The Netherlands
                [ ]Faculty of Science and Information Technology, School of Psychology, University of Newcastle, Callaghan, New South Wales Australia
                [ ]Data Analytics, Price Waterhouse Coopers, Amsterdam, The Netherlands
                Article
                913
                10.3758/s13423-015-0913-5
                4828473
                26374437
                ee0709dd-f5a6-46c8-9a2f-bc68406b786d
                © The Author(s) 2015

                Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

                History
                Categories
                Article
                Custom metadata
                © Psychonomic Society, Inc. 2016

                Clinical Psychology & Psychiatry
                multiway anova,factorial anova,multiple comparison problem,type i error,sequential bonferroni,benjamini–hochberg procedure,familywise error rate,false discovery rate,preregistration

                Comments

                Comment on this article