Questioning the application of risk of bias tools in appraising evidence from natural experimental studies: critical reflections on Benton et al., IJBNPA 2016

There is no author summary for this article yet. Authors can add summaries to their articles on ScienceOpen to make them more accessible to a non-specialist audience.

Abstract

We recently read the article by Benton et al. which reviewed risk of bias in natural experimental studies investigating the impact of the built environment on physical activity (Benton et al., 2016; Int J Behav Nutr Phys Act 13:107). As a technical exercise in assessing risk of bias to understand study quality, we found the results of this study both interesting and potentially useful. However, it prompted a number of concerns with the use of risk of bias tools for assessing the quality of evidence from studies exploiting natural experiments. As we discuss in this commentary, the rigid application of such tools could have adverse effects on the uptake and use of natural experiments in population health research and practice.

Related collections

Most cited references 17

Record: found
Abstract: found
Article: found

Is Open Access

Health effects of the London bicycle sharing system: health impact modelling study

James Woodcock, Marko Tainio, James Cheshire … (2014)

Objective To model the impacts of the bicycle sharing system in London on the health of its users. Design Health impact modelling and evaluation, using a stochastic simulation model. Setting Central and inner London, England. Data sources Total population operational registration and usage data for the London cycle hire scheme (collected April 2011-March 2012), surveys of cycle hire users (collected 2011), and London data on travel, physical activity, road traffic collisions, and particulate air pollution (PM2.5, (collected 2005-12). Participants 578 607 users of the London cycle hire scheme, aged 14 years and over, with an estimated 78% of travel time accounted for by users younger than 45 years. Main outcome measures Change in lifelong disability adjusted life years (DALYs) based on one year impacts on incidence of disease and injury, modelled through medium term changes in physical activity, road traffic injuries, and exposure to air pollution. Results Over the year examined the users made 7.4 million cycle hire trips (estimated 71% of cycling time by men). These trips would mostly otherwise have been made on foot (31%) or by public transport (47%). To date there has been a trend towards fewer fatalities and injuries than expected on cycle hire bicycles. Using these observed injury rates, the population benefits from the cycle hire scheme substantially outweighed harms (net change −72 DALYs (95% credible interval −110 to −43) among men using cycle hire per accounting year; −15 (−42 to −6) among women; note that negative DALYs represent a health benefit). When we modelled cycle hire injury rates as being equal to background rates for all cycling in central London, these benefits were smaller and there was no evidence of a benefit among women (change −49 DALYs (−88 to −17) among men; −1 DALY (−27 to 12) among women). This sex difference largely reflected higher road collision fatality rates for female cyclists. At older ages the modelled benefits of cycling were much larger than the harms. Using background injury rates in the youngest age group (15 to 29 years), the medium term benefits and harms were both comparatively small and potentially negative. Conclusion London’s bicycle sharing system has positive health impacts overall, but these benefits are clearer for men than for women and for older users than for younger users. The potential benefits of cycling may not currently apply to all groups in all settings.

0 comments Cited 101 times – based on 0 reviews      Review now

Bookmark

Record: found
Abstract: not found
Article: not found

Improving Causal Inference: Strengths and Limitations of Natural Experiments

G Dunning (2008)

0 comments Cited 71 times – based on 0 reviews      Review now

Bookmark

Record: found
Abstract: found
Article: found

Is Open Access

The effect of changing the built environment on physical activity: a quantitative review of the risk of bias in natural experiments

Jack S. Benton, Jamie Anderson, Ruth F Hunter … (2016)

Background Evidence regarding the association of the built environment with physical activity is influencing policy recommendations that advocate changing the built environment to increase population-level physical activity. However, to date there has been no rigorous appraisal of the quality of the evidence on the effects of changing the built environment. The aim of this review was to conduct a thorough quantitative appraisal of the risk of bias present in those natural experiments with the strongest experimental designs for assessing the causal effects of the built environment on physical activity. Methods Eligible studies had to evaluate the effects of changing the built environment on physical activity, include at least one measurement before and one measurement of physical activity after changes in the environment, and have at least one intervention site and non-intervention comparison site. Given the large number of systematic reviews in this area, studies were identified from three exemplar systematic reviews; these were published in the past five years and were selected to provide a range of different built environment interventions. The risk of bias in these studies was analysed using the Cochrane Risk of Bias Assessment Tool: for Non-Randomized Studies of Interventions (ACROBAT-NRSI). Results Twelve eligible natural experiments were identified. Risk of bias assessments were conducted for each physical activity outcome from all studies, resulting in a total of fifteen outcomes being analysed. Intervention sites included parks, urban greenways/trails, bicycle lanes, paths, vacant lots, and a senior citizen’s centre. All outcomes had an overall critical (n = 12) or serious (n = 3) risk of bias. Domains with the highest risk of bias were confounding (due to inadequate control sites and poor control of confounding variables), measurement of outcomes, and selection of the reported result. Conclusions The present review focused on the strongest natural experiments conducted to date. Given this, the failure of existing studies to adequately control for potential sources of bias highlights the need for more rigorous research to underpin policy recommendations for changing the built environment to increase physical activity. Suggestions are proposed for how future natural experiments in this area can be improved. Electronic supplementary material The online version of this article (doi:10.1186/s12966-016-0433-3) contains supplementary material, which is available to authorized users.

0 comments Cited 50 times – based on 0 reviews      Review now

Bookmark

All references

Author and article information

Contributors

David K. Humphreys: +44 (0)1865 280339 , david.humphreys@spi.ox.ac.uk

Jenna Panter: +44 (0) 1223 746884 , jrp63@medschl.cam.ac.uk

David Ogilvie: +44 (0) 1223 769197 , dbo23@medschl.cam.ac.uk

Journal

Journal ID (nlm-ta): Int J Behav Nutr Phys Act

Journal ID (iso-abbrev): Int J Behav Nutr Phys Act

Title: The International Journal of Behavioral Nutrition and Physical Activity

Publisher: BioMed Central (London )

ISSN (Electronic): 1479-5868

Publication date (Electronic): 19 April 2017

Publication date PMC-release: 19 April 2017

Publication date Collection: 2017

Volume: 14

Electronic Location Identifier: 49

Affiliations

[1 ]ISNI 0000 0004 1936 8948, GRID grid.4991.5, Department of Social Policy and Intervention, , University of Oxford, ; 32 Wellington Square, Oxford, OX1 2ER UK

[2 ]ISNI 0000 0004 1936 8948, GRID grid.4991.5, Green Templeton College, , University of Oxford, ; 43 Woodstock Road, Oxford, OX2 6HG UK

[3 ]ISNI 0000000121885934, GRID grid.5335.0, MRC Epidemiology Unit and Centre for Diet and Activity Research (CEDAR), , University of Cambridge, ; Box 285, Cambridge Biomedical Campus, Cambridge, CB2 0QQ UK

Article

Publisher ID: 500

DOI: 10.1186/s12966-017-0500-4

PMC ID: 5397808

PubMed ID: 28424086

SO-VID: f72c4823-ad2f-4b28-9e75-8ae4895a7926

License:

Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

History

Date received : 27 January 2017

Date accepted : 3 April 2017

Custom metadata

ScienceOpen disciplines: Nutrition & Dietetics

Keywords: natural experiments,built environment,physical activity,risk of bias

Data availability:

ScienceOpen disciplines: Nutrition & Dietetics

Keywords: natural experiments, built environment, physical activity, risk of bias

Questioning the application of risk of bias tools in appraising evidence from natural experimental studies: critical reflections on Benton et al., IJBNPA 2016

Read this article at

Abstract

Related collections

Special issue: Historical narratives of public health in the built environment

Most cited references 17

Health effects of the London bicycle sharing system: health impact modelling study

Improving Causal Inference: Strengths and Limitations of Natural Experiments

The effect of changing the built environment on physical activity: a quantitative review of the risk of bias in natural experiments

Author and article information

Contributors

Journal

Affiliations

Article

History

Categories

Custom metadata

Comments

Comment on this article

Similar content 242

Cited by 8

Most referenced authors 405