Why is impact measurement abandoned in practice? Evidence use in evaluation and contracting for five European Social Impact Bonds

There is no author summary for this article yet. Authors can add summaries to their articles on ScienceOpen to make them more accessible to a non-specialist audience.

Abstract

Despite broad consensus on the importance of measuring “impact,” the term is not always understood as estimating counterfactual and causal estimates. We examine a type of public sector financing, “Social Impact Bonds,” a scheme where investors front money for public services, with repayment conditional on impact. We examine five cases in four European countries of Social Impact Bonds financing active labor market programs, testing the claim that Social Impact Bonds would move counterfactual causal impact evaluation to the heart of policy. We examine first how evidence was integrated in contracts, second the overall evidence generated and third, given that neither contracts nor evaluations used counterfactual definitions of impact, we explore stakeholders’ perspectives to better understand the reasons why. We find that although most stakeholders wanted the Social Impact Bonds to generate impact estimates, beliefs about public service reform, incentives, and the logic of experimentation led to the acceptance of non-causal definitions.

Related collections

Most cited references 56

Record: found
Abstract: found
Book: not found

The Discovery of Grounded Theory : Strategies for Qualitative Research

Barney G. Glaser, Anselm L. Strauss (1967)

Most writing on sociological method has been concerned with how accurate facts can be obtained and how theory can thereby be more rigorously tested. In The Discovery of Grounded Theory, Barney Glaser and Anselm Strauss address the equally Important enterprise of how the discovery of theory from data--systematically obtained and analyzed in social research--can be furthered. The discovery of theory from data--grounded theory--is a major task confronting sociology, for such a theory fits empirical situations, and is understandable to sociologists and laymen alike. Most important, it provides relevant predictions, explanations, interpretations, and applications. In Part I of the book, -Generation Theory by Comparative Analysis, - the authors present a strategy whereby sociologists can facilitate the discovery of grounded theory, both substantive and formal. This strategy involves the systematic choice and study of several comparison groups. In Part II, The Flexible Use of Data, - the generation of theory from qualitative, especially documentary, and quantitative data Is considered. In Part III, -Implications of Grounded Theory, - Glaser and Strauss examine the credibility of grounded theory. The Discovery of Grounded Theory is directed toward improving social scientists' capacity for generating theory that will be relevant to their research. While aimed primarily at sociologists, it will be useful to anyone Interested In studying social phenomena--political, educational, economic, industrial-- especially If their studies are based on qualitative data.

0 comments Cited 566 times – based on 0 reviews

Bookmark

Record: found
Abstract: not found
Article: not found

The Constant Comparative Method of Qualitative Analysis

Barney G. Glaser (1965)

0 comments Cited 322 times – based on 0 reviews      Review now

Bookmark

Record: found
Abstract: found
Article: not found

Is Open Access

Understanding and misunderstanding randomized controlled trials

Angus Deaton, Nancy Cartwright (2018)

Randomized Controlled Trials (RCTs) are increasingly popular in the social sciences, not only in medicine. We argue that the lay public, and sometimes researchers, put too much trust in RCTs over other methods of investigation. Contrary to frequent claims in the applied literature, randomization does not equalize everything other than the treatment in the treatment and control groups, it does not automatically deliver a precise estimate of the average treatment effect (ATE), and it does not relieve us of the need to think about (observed or unobserved) covariates. Finding out whether an estimate was generated by chance is more difficult than commonly believed. At best, an RCT yields an unbiased estimate, but this property is of limited practical value. Even then, estimates apply only to the sample selected for the trial, often no more than a convenience sample, and justification is required to extend the results to other groups, including any population to which the trial sample belongs, or to any individual, including an individual in the trial. Demanding ‘external validity’ is unhelpful because it expects too much of an RCT while undervaluing its potential contribution. RCTs do indeed require minimal assumptions and can operate with little prior knowledge. This is an advantage when persuading distrustful audiences, but it is a disadvantage for cumulative scientific progress, where prior knowledge should be built upon, not discarded. RCTs can play a role in building scientific knowledge and useful predictions but they can only do so as part of a cumulative program, combining with other methods, including conceptual and theoretical development, to discover not ‘what works’, but ‘why things work’.