This study is concerned with how impact from research and innovation (R&I) programmes is accounted for in impact evaluation reports. Establishing causal links between a research funding instrument and different effects, poses well known methodological difficulties. In the light of such challenges, textual accounts about causal links ought to be carefully written. Nevertheless, impact evaluation reports have a tendency towards unwarranted simplification as far as impact inferences are concerned. In this study, we illustrate how such simplifications – versions of the narrative device ellipsis – are accomplished. Using examples from three Swedish impact evaluation reports, we focus on the constituent components of longer impact accounts, that of the impact argument, to analyze the various ways that impact is narratively achieved through simplification. We believe this analysis can contribute to the methodology of impact evaluation, as well as spread light on some the difficulties in the historiography of innovation in general.
Abbott, H. (2002) Narrative, Cambridge University Press, Cambridge.
Bal, M. (2002) Narratology: Introduction to the Theory of Narrative, University of Toronto Press, Toronto.
Cartwright, N. (2007) Hunting Causes and Using Them: Approaches in Philosophy and Economics, Cambridge University Press, Cambridge.
Cartwright, N. and Hardie, J. (2012) Evidence-Based Policy, Oxford University Press, New York.
Chatman, S. (1978) Story and Discourse: Narrative Structure in Fiction and Film, Cornell University Press, Ithaca NY.
Donovan, C. (2011) ‘State of the art in assessing research impact. Introduction to a special issue,‘ Research Evaluation, 20, pp. 175–79. doi:10.3152/095820211X13118583635918.
Donovan, C. and Hanney, S. (2011) 'The 'Payback Framework' explained,' Research Evaluation, 20, pp. 181–83. doi:10.3152/095820211X13118583635756.
Eriksson, T., Roback, K., Lundqvist, M., Levin, L. and Carsson, P. (2011) Effektanalys Av Forskningsprogramme Inom Material Från Förnyelsebara Råvaror [Effect Analysis of Research Programmes on Materials from Renewables], VINNOVA, Stockholm.
Fischer, F. (2003) Reframing Public Policy: Discursive Politics and Deliberative Practices, Oxford University Press, Oxford.
Fischer, F. and Forrester, J. (1993). The Argumentative Turn in Policy Analysis and Planning, Duke University Press, Durham NC.
Fischer, F. and Gottweis, H. (eds) (2012) The Argumentative Turn Revisited: Public Policy as Communicative Practice, Duke University Press, Durham NC.
Forster, E.M. (1927) Aspects of the Novel, Harcourt, Brace & Co, New York.
Gasper, D. (1996) ‘Analysing policy arguments,‘ European Journal of Development Research, 8, pp. 36–62. doi:10.1080/09578819608426652.
Gates, E. and Dyson, L. (2017) ‘Implications of the changing conversation about causality for evaluators,‘ American Journal of Evaluation, 38, pp. 29–46. doi:10.1177/1098214016644068.
Gertler, P., Martinez, S., Premand, P., Rawlings, L. and Vermeersch, C. (2016) Impact Evaluation in Practice, Second, Inter-American Development Bank and World Bank, Washington DC.
Girfalco, L. (1991) Dynamics of Technological Change, Van Norstrand Reinhold, New York.
Gläser, J. and Laudel, G. (2015) ‘A bibliometric reconstruction of research trails for qualitative investigations of scientific innovations,‘ Historical Social Research, 40, pp. 299–330.
Granhag, P., Strömwall, L. and Allwood, C.-A. (2000) ‘Effects of reiteration, hindsight bias, and memory on realism in eyewitness confidence,‘ Applied Cognitive Psychology, 14, pp. 397–420. doi: 10.1002/1099–0720(200009)14:5<397::AID-ACP658>3.0.CO;2-M.
Guba, E.G. and Lincoln, Y.S. (1989) Fourth-Generation Evaluation, Sage Publications, Thousand Oaks CA.
Hajer, M. and Laws, D. (2006) ‘Ordering through discourse’ in Moran, M., Rein, M. and Goodin, R. (eds) Oxford Handbook of Public Policy, Oxford University Press, Oxford, pp. 251–68.
Hogwood, B. and Gunn, L. (1984) Policy Analysis for the Real World, Oxford University Press, New York.
Kontigo. (2016) Effektanalys Av Vinnväxt-Programmet: Analys Av Effekter Och Nytta [Effect Analysis of the Vinnväxt-Programme: Analysis of Effects and Utility], VINNOVA, Stockholm.
Kostoff, R. (1993) ‘Semiquantitative methods for research impact assessment,‘ Technological Forecasting and Social Change, 44, pp. 231–44. doi:10.1016/0040–1625(93)90070-N.
Lothe, J. (2000) Narrative in Fiction and Film, Oxford University Press, Oxford.
Majone, G. (1989) Evidence, Argument and Persuasion in the Policy Process, Yale University Press, New Haven CT.
Mansfield, E. (1998) ‘Academic research and industrial innovation: an update of empirical findings,‘ Research Policy, 26, pp. 773–76. doi:10.1016/S0048–7333(97)00043–7.
Megill, A. (2007) Historical Knowledge, Historical Error: A Contemporary Guide to Practice, University of Chicago Press, Chicago.
Mohr, L. (1995) Impact Analysis for Programme Evaluation, Sage, Thousand Oaks CA.
National Academy of Sciences. (1999) Evaluating Federal Research Programmes: Research and the Government Performance and Results Act, National Academies' Press, Washington DC.
O'Keefe, C. and Head, R. (2011) ‘Application of logic models in a large scientific research programme,‘ Evaluation and Programme Planning, 34, pp. 174–84. doi:10.1016/j.evalprogplan.2011.02.008.
Pawson, R. and Tilly, N. (1997) Realistic Evaluation, Sage Publications, London.
Pen, J. (1985) Among Economists, North Holland, Amsterdam.
Penfield, T., Baker, M., Scoble, R. and Wykes, M. (2014) ‘Assessment, evaluations, and definitions of research impacts: a review,‘ Research Evaluation, 23, pp. 21–32. doi:10.1093/reseval/rvt021.
Popper, K.R. (1963) Conjectures and Refutations: The Growth of Scientific Knowledge, Routledge, London.
REF (2014, 02) Assessment framework and guidance on submissions, available from http://www.ref.ac.uk/pubs/2011–02/[accessed Jun 2017].
Roe, E. (1994) Narrative Policy Analysis: Theory and Practice, Duke University Press, Durham NC.
Scriven, M. (2008) ‘A summative evaluation of RCT methodology: & an alternative approach to causal research,‘ Journal of MultiDisciplinary Evaluation, 5, pp. 11–24.
Smith, K. and Larimer, C. (2017) The Public Policy Theory Primer, Westview Press, Boulder CO.
Steel, D. (2008) Across the Boundaries: Extrapolation in Biology and Social Science, Oxford University Press, New York.
Stern, E. (2013) ‘Editorial,‘ Evaluation, 19, pp. 3–4. doi:10.1177/1356389012473814.
Thomas, D. (2006) ‘A general inductive approach for analyzing qualitative evaluation data,‘ American Journal of Evaluation, 27, pp. 237–46. doi:10.1177/1098214005283748.
VINNOVA (2017) available from www.vinnova.se [accessed Jun 2017].
von Bahr, J. (ed). (2014) Hälsoekonomisk Effektanalys Av Forskning Inom Programmet Innovationer För Framtidens Hälsa. [Health Economic Effect Analysis of Research within the Programme ‘Innovation for Future Health‘], VINNOVA, Stockholm.
Weiss, C.H. (1998) ‘Have we learned anything new about the use of evaluation?‘ American Journal of Evaluation, 19, pp. 21–33.
White, M. (2010) ‘A contribution to current debates in impact evaluation,‘ Evaluation, 16, pp. 153–64. doi:10.1177/1356389010361562.
Wilson, J. (1973) Political Organizations, Basic Books, New York.
Wooding, S., Hanney, S., Buxton, M. and Grant, J. (2005) ‘Payback arising from research funding: evaluation of the arthritis research campaign,‘ Rheumatology, 44, pp. 1145–56. doi:10.1093/rheumatology/keh708.