For Publishers
DiscoveryMetadataPeer reviewHostingPublishing
For Researchers
JoinPublishReviewCollect
Register
Blog
About
Search
Advanced search
My ScienceOpen
Search
For Publishers
For Researchers
Blog
About
48
views
23
references
 Top references
cited by
153
    
0 reviews
 Review
0
comments
 Comment
0
recommends
+1 Recommend
0 collections
    0
    shares
      215
      similar
       All similar
      • Record: found
      • Abstract: found
      • Article: found
      Is Open Access

      Sample size considerations for the external validation of a multivariable prognostic model: a resampling study

      research-article

      Read this article at

      ScienceOpenPublisherPMC
      Bookmark
          There is no author summary for this article yet. Authors can add summaries to their articles on ScienceOpen to make them more accessible to a non-specialist audience.

          Abstract

          After developing a prognostic model, it is essential to evaluate the performance of the model in samples independent from those used to develop the model, which is often referred to as external validation. However, despite its importance, very little is known about the sample size requirements for conducting an external validation. Using a large real data set and resampling methods, we investigate the impact of sample size on the performance of six published prognostic models. Focussing on unbiased and precise estimation of performance measures (e.g. the c‐index, D statistic and calibration), we provide guidance on sample size for investigators designing an external validation study. Our study suggests that externally validating a prognostic model requires a minimum of 100 events and ideally 200 (or more) events. © 2015 The Authors. Statistics in Medicine Published by John Wiley & Sons Ltd.

          Related collections

          Most cited references23

          • Record: found
          • Abstract: found
          • Article: not found

          General cardiovascular risk profile for use in primary care: the Framingham Heart Study.

          Ralph B. D’Agostino, Ramachandran S. Vasan, Michael J. Pencina … (2008)
          Separate multivariable risk algorithms are commonly used to assess risk of specific atherosclerotic cardiovascular disease (CVD) events, ie, coronary heart disease, cerebrovascular disease, peripheral vascular disease, and heart failure. The present report presents a single multivariable risk function that predicts risk of developing all CVD and of its constituents. We used Cox proportional-hazards regression to evaluate the risk of developing a first CVD event in 8491 Framingham study participants (mean age, 49 years; 4522 women) who attended a routine examination between 30 and 74 years of age and were free of CVD. Sex-specific multivariable risk functions ("general CVD" algorithms) were derived that incorporated age, total and high-density lipoprotein cholesterol, systolic blood pressure, treatment for hypertension, smoking, and diabetes status. We assessed the performance of the general CVD algorithms for predicting individual CVD events (coronary heart disease, stroke, peripheral artery disease, or heart failure). Over 12 years of follow-up, 1174 participants (456 women) developed a first CVD event. All traditional risk factors evaluated predicted CVD risk (multivariable-adjusted P<0.0001). The general CVD algorithm demonstrated good discrimination (C statistic, 0.763 [men] and 0.793 [women]) and calibration. Simple adjustments to the general CVD risk algorithms allowed estimation of the risks of each CVD component. Two simple risk scores are presented, 1 based on all traditional risk factors and the other based on non-laboratory-based predictors. A sex-specific multivariable risk factor algorithm can be conveniently used to assess general CVD risk and risk of individual CVD events (coronary, cerebrovascular, and peripheral arterial disease and heart failure). The estimated absolute CVD event rates can be used to quantify risk and to guide preventive care.
          Article 0 comments Cited 2184 times – based on 0 reviews     Review now
            Bookmark
            • Record: found
            • Abstract: found
            • Article: not found

            Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD): Explanation and Elaboration

            Karel Moons, Douglas G. Altman, Johannes Reitsma … (2015)
            The TRIPOD (Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis) Statement includes a 22-item checklist, which aims to improve the reporting of studies developing, validating, or updating a prediction model, whether for diagnostic or prognostic purposes. The TRIPOD Statement aims to improve the transparency of the reporting of a prediction model study regardless of the study methods used. This explanation and elaboration document describes the rationale; clarifies the meaning of each item; and discusses why transparent reporting is important, with a view to assessing risk of bias and clinical usefulness of the prediction model. Each checklist item of the TRIPOD Statement is explained in detail and accompanied by published examples of good reporting. The document also provides a valuable reference of issues to consider when designing, conducting, and analyzing prediction model studies. To aid the editorial process and help peer reviewers and, ultimately, readers and systematic reviewers of prediction model studies, it is recommended that authors include a completed checklist in their submission. The TRIPOD checklist can also be downloaded from www.tripod-statement.org.
            Article 0 comments Cited 1181 times – based on 0 reviews     Review now
              Bookmark
              • Record: found
              • Abstract: found
              • Article: not found

              Assessing the generalizability of prognostic information.

              A Justice, K. Covinsky, J A Berlin (1999)
              Physicians are often asked to make prognostic assessments but often worry that their assessments will prove inaccurate. Prognostic systems were developed to enhance the accuracy of such assessments. This paper describes an approach for evaluating prognostic systems based on the accuracy (calibration and discrimination) and generalizability (reproducibility and transportability) of the system's predictions. Reproducibility is the ability to produce accurate predictions among patients not included in the development of the system but from the same population. Transportability is the ability to produce accurate predictions among patients drawn from a different but plausibly related population. On the basis of the observation that the generalizability of a prognostic system is commonly limited to a single historical period, geographic location, methodologic approach, disease spectrum, or follow-up interval, we describe a working hierarchy of the cumulative generalizability of prognostic systems. This approach is illustrated in a structured review of the Dukes and Jass staging systems for colon and rectal cancer and applied to a young man with colon cancer. Because it treats the development of the system as a "black box" and evaluates only the performance of the predictions, the approach can be applied to any system that generates predicted probabilities. Although the Dukes and Jass staging systems are discrete, the approach can also be applied to systems that generate continuous predictions and, with some modification, to systems that predict over multiple time periods. Like any scientific hypothesis, the generalizability of a prognostic system is established by being tested and being found accurate across increasingly diverse settings. The more numerous and diverse the settings in which the system is tested and found accurate, the more likely it will generalize to an untested setting.
              Article 0 comments Cited 259 times – based on 0 reviews     Review now
                Bookmark
                All references

                Author and article information

                Journal
                Journal ID (nlm-ta): Stat Med
                Journal ID (iso-abbrev): Stat Med
                Journal ID (doi): 10.1002/(ISSN)1097-0258
                Journal ID (publisher-id): SIM
                Title: Statistics in Medicine
                Publisher: John Wiley and Sons Inc. (Hoboken )
                ISSN (Print): 0277-6715
                ISSN (Electronic): 1097-0258
                Publication date (Electronic): 09 November 2015
                Publication date (Print): 30 January 2016
                Volume: 35
                Issue: 2 ( doiID: 10.1002/sim.v35.2 )
                Pages: 214-226
                Affiliations
                [ 1 ] Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, Botnar Research Centre University of Oxford Windmill Road Oxford OX3 7LDU.K.
                Author notes
                [*] [* ] Correspondence to: Gary S. Collins, Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, Botnar Research Centre, University of Oxford, Windmill Road, Oxford OX3 7LD, U.K.

                E‐mail: gary.collins@ 123456csm.ox.ac.uk

                Author information
                http://orcid.org/0000-0002-2772-2316
                Article
                Publisher ID: SIM6787 Other ID: SIM-15-0170.R2
                DOI: 10.1002/sim.6787
                PMC ID: 4738418
                PubMed ID: 26553135
                SO-VID: c1113950-bb10-4762-a819-987c5bc666fc
                Copyright © © 2015 The Authors. Statistics in Medicine Published by John Wiley & Sons Ltd.
                License:

                This is an open access article under the terms of the Creative Commons Attribution License, which permits use, distribution and reproduction in any medium, provided the original work is properly cited.

                History
                Date received : 10 March 2015
                Date revision received : 02 October 2015
                Date accepted : 12 October 2015
                Page count
                Pages: 13
                Funding
                Funded by: Medical Research Council
                Award ID: G1100513
                Funded by: Medical Research Council Prognosis Research Strategy (PROGRESS) Partnership
                Award ID: G0902393/99558
                Categories
                Subject: Research Article
                Subject: Research Articles
                Custom metadata
                source-schema-version-number 2.0
                component-id sim6787
                header-id sim6787-hdr-0001
                cover-date 30 January 2016
                details-of-publishers-convertor Converter:WILEY_ML3GV2_TO_NLMPMC version:4.7.5 mode:remove_FC converted:28.01.2016

                ScienceOpen disciplines: Biostatistics
                Keywords: prognostic model,sample size,external validation
                Data availability:
                ScienceOpen disciplines: Biostatistics
                Keywords: prognostic model, sample size, external validation

                Comments

                Comment on this article

                scite_

                Similar content215

                See all similar

                Cited by225

                See all cited by

                Most referenced authors1,489

                See all reference authors