For Publishers
DiscoveryMetadataPeer reviewHostingPublishing
For Researchers
JoinPublishReviewCollect
Register
Blog
About
Search
Advanced search
My ScienceOpen
Search
For Publishers
For Researchers
Blog
About
13
views
26
references
 Top references
cited by
196
    
0 reviews
 Review
0
comments
 Comment
0
recommends
+1 Recommend
0 collections
    0
    shares
      1,606
      similar
       All similar
      • Record: found
      • Abstract: not found
      • Article: not found

      Beyond Power Calculations : Assessing Type S (Sign) and Type M (Magnitude) Errors

      Author(s): 1 , 2 , 3
      Publication date Created:
      Publication date (Electronic):
      Journal: Perspectives on Psychological Science
      Publisher: SAGE Publications

      Read this article at

      ScienceOpenPublisherPubMed
      Bookmark
          There is no author summary for this article yet. Authors can add summaries to their articles on ScienceOpen to make them more accessible to a non-specialist audience.

          Abstract

          Statistical power analysis provides the conventional approach to assess error rates when designing a research study. However, power analysis is flawed in that a narrow emphasis on statistical significance is placed as the primary focus of study design. In noisy, small-sample settings, statistically significant results can often be misleading. To help researchers address this problem in the context of their own studies, we recommend design calculations in which (a) the probability of an estimate being in the wrong direction (Type S [sign] error) and (b) the factor by which the magnitude of an effect might be overestimated (Type M [magnitude] error or exaggeration ratio) are estimated. We illustrate with examples from recent published research and discuss the largest challenge in a design calculation: coming up with reasonable estimates of plausible effect sizes based on external information.

          Related collections

          Most cited references26

          • Record: found
          • Abstract: found
          • Article: not found

          Power failure: why small sample size undermines the reliability of neuroscience.

          Katherine S. Button, John Ioannidis, Claire Mokrysz … (2014)
          A study with low statistical power has a reduced chance of detecting a true effect, but it is less well appreciated that low power also reduces the likelihood that a statistically significant result reflects a true effect. Here, we show that the average statistical power of studies in the neurosciences is very low. The consequences of this include overestimates of effect size and low reproducibility of results. There are also ethical dimensions to this problem, as unreliable research is inefficient and wasteful. Improving reproducibility in neuroscience is a key priority and requires attention to well-established but often ignored methodological principles.
          Article 0 comments Cited 1133 times – based on 0 reviews     Review now
            Bookmark
            • Record: found
            • Abstract: found
            • Article: not found

            False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant.

            Joseph Simmons, Leif Nelson, Uri Simonsohn (2011)
            In this article, we accomplish two things. First, we show that despite empirical psychologists' nominal endorsement of a low rate of false-positive findings (≤ .05), flexibility in data collection, analysis, and reporting dramatically increases actual false-positive rates. In many cases, a researcher is more likely to falsely find evidence that an effect exists than to correctly find evidence that it does not. We present computer simulations and a pair of actual experiments that demonstrate how unacceptably easy it is to accumulate (and report) statistically significant evidence for a false hypothesis. Second, we suggest a simple, low-cost, and straightforwardly effective disclosure-based solution to this problem. The solution involves six concrete requirements for authors and four guidelines for reviewers, all of which impose a minimal burden on the publication process.
            Article 0 comments Cited 784 times – based on 0 reviews     Review now
              Bookmark
              • Record: found
              • Abstract: found
              • Article: not found

              Puzzlingly High Correlations in fMRI Studies of Emotion, Personality, and Social Cognition.

              Edward Vul, Christine Harris, Piotr Winkielman … (2009)
              Functional magnetic resonance imaging (fMRI) studiesofemotion, personality, and social cognition have drawn much attention in recent years, with high-profile studies frequently reporting extremely high (e.g., >.8) correlations between brain activation and personality measures. We show that these correlations are higher than should be expected given the (evidently limited) reliability of both fMRI and personality measures. The high correlations are all the more puzzling because method sections rarely contain much detail about how the correlations were obtained. We surveyed authors of 55 articles that reported findings of this kind to determine a few details on how these correlations were computed. More than half acknowledged using a strategy that computes separate correlations for individual voxels and reports means of only those voxels exceeding chosen thresholds. We show how this nonindependent analysis inflates correlations while yielding reassuring-looking scattergrams. This analysis technique was used to obtain the vast majority of the implausibly high correlations in our survey sample. In addition, we argue that, in some cases, other analysis problems likely created entirely spurious correlations. We outline how the data from these studies could be reanalyzed with unbiased methods to provide accurate estimates of the correlations in question and urge authors to perform such reanalyses. The underlying problems described here appear to be common in fMRI research of many kinds-not just in studies of emotion, personality, and social cognition.
              Article 0 comments Cited 354 times – based on 0 reviews     Review now
                Bookmark
                All references

                Author and article information

                Journal
                Title: Perspectives on Psychological Science
                Abbreviated Title: Perspect Psychol Sci
                Publisher: SAGE Publications
                ISSN (Print): 1745-6916
                ISSN (Electronic): 1745-6924
                Publication date Created: November 17 2014
                Publication date Created: November 2014
                Publication date (Electronic): November 17 2014
                Publication date (Print): November 2014
                Volume: 9
                Issue: 6
                Pages: 641-651
                Affiliations
                [1 ]Department of Statistics and Department of Political Science, Columbia University
                [2 ]Clinical Epidemiology and Biostatistics Unit, Murdoch Children’s Research Institute, Parkville, Victoria, Australia
                [3 ]Department of Paediatrics and School of Population and Global Health, University of Melbourne
                Article
                DOI: 10.1177/1745691614551642
                PubMed ID: 26186114
                SO-VID: 564c044d-e43c-48b8-a7f7-65fe6fc51ebb
                Copyright © © 2014
                License:

                http://journals.sagepub.com/page/policies/text-and-data-mining-license

                History

                Data availability:

                Comments

                Comment on this article

                scite_

                Similar content1,606

                See all similar

                Cited by312

                See all cited by

                Most referenced authors283

                See all reference authors