Achieving impact: impact evaluations and narrative simplification

If you have found this article useful and you think it is important that researchers across the world have access, please consider donating, to ensure that this valuable collection remains Open Access.

Prometheus is published by Pluto Journals, an Open Access publisher. This means that everyone has free and unlimited access to the full-text of all articles from our international collection of social science journals. Furthermore Pluto Journals authors don’t pay article processing charges (APCs).

Prometheus

Volume 35, Issue 3

Record: found
Abstract: found
Article: found

Is Open Access

Achieving impact: impact evaluations and narrative simplification

research-article

Author(s): Tomas Hellström ^a , Christina Hellström ^b

Publication date (Print): 1 September 2017

Journal: Prometheus

Publisher: Pluto Journals

Bookmark

Abstract

This study is concerned with how impact from research and innovation (R&I) programmes is accounted for in impact evaluation reports. Establishing causal links between a research funding instrument and different effects, poses well known methodological difficulties. In the light of such challenges, textual accounts about causal links ought to be carefully written. Nevertheless, impact evaluation reports have a tendency towards unwarranted simplification as far as impact inferences are concerned. In this study, we illustrate how such simplifications – versions of the narrative device ellipsis – are accomplished. Using examples from three Swedish impact evaluation reports, we focus on the constituent components of longer impact accounts, that of the impact argument, to analyze the various ways that impact is narratively achieved through simplification. We believe this analysis can contribute to the methodology of impact evaluation, as well as spread light on some the difficulties in the historiography of innovation in general.

Content

Author and article information

Contributors

Tomas Hellström

Journal

Journal ID (doi): 10.2307/j50022063

Journal ID (jcode): prometheus

Title: Prometheus

Publisher: Pluto Journals

ISSN: 0810-9028

ISSN: 1470-1030

Publication date (Print): 1 September 2017

Volume: 35

Issue: 3 ( doiID: 10.1080/prometheus.35.issue-3 )

Pages: 215-230

Affiliations

[ ^a ]School of Economics and Management, Lund University, Lund, Sweden

[ ^b ]Department of Psychology, Göteborg University, Göteborg, Sweden

Article

Publisher ID: 08109028.2018.1522829

DOI: 10.1080/08109028.2018.1522829

SO-VID: 31fcaac4-a178-4b69-b594-6eb1be1982ad

License:

All content is freely available without charge to users or their institutions. Users are allowed to read, download, copy, distribute, print, search, or link to the full texts of the articles in this journal without asking prior permission of the publisher or the author. Articles published in the journal are distributed under a http://creativecommons.org/licenses/by/4.0/.

Custom metadata

lang eng

ScienceOpen disciplines: Computer science,Arts,Social & Behavioral Sciences,Law,History,Economics

References

Abbott, H. (2002) Narrative, Cambridge University Press, Cambridge.
Bal, M. (2002) Narratology: Introduction to the Theory of Narrative, University of Toronto Press, Toronto.
Cartwright, N. (2007) Hunting Causes and Using Them: Approaches in Philosophy and Economics, Cambridge University Press, Cambridge.
Cartwright, N. and Hardie, J. (2012) Evidence-Based Policy, Oxford University Press, New York.
Chatman, S. (1978) Story and Discourse: Narrative Structure in Fiction and Film, Cornell University Press, Ithaca NY.
Donovan, C. (2011) ‘State of the art in assessing research impact. Introduction to a special issue,‘ Research Evaluation, 20, pp. 175–79. doi:10.3152/095820211X13118583635918.
Donovan, C. and Hanney, S. (2011) 'The 'Payback Framework' explained,' Research Evaluation, 20, pp. 181–83. doi:10.3152/095820211X13118583635756.
Eriksson, T., Roback, K., Lundqvist, M., Levin, L. and Carsson, P. (2011) Effektanalys Av Forskningsprogramme Inom Material Från Förnyelsebara Råvaror [Effect Analysis of Research Programmes on Materials from Renewables], VINNOVA, Stockholm.
Fischer, F. (2003) Reframing Public Policy: Discursive Politics and Deliberative Practices, Oxford University Press, Oxford.
Fischer, F. and Forrester, J. (1993). The Argumentative Turn in Policy Analysis and Planning, Duke University Press, Durham NC.
Fischer, F. and Gottweis, H. (eds) (2012) The Argumentative Turn Revisited: Public Policy as Communicative Practice, Duke University Press, Durham NC.
Forster, E.M. (1927) Aspects of the Novel, Harcourt, Brace & Co, New York.
Gasper, D. (1996) ‘Analysing policy arguments,‘ European Journal of Development Research, 8, pp. 36–62. doi:10.1080/09578819608426652.
Gates, E. and Dyson, L. (2017) ‘Implications of the changing conversation about causality for evaluators,‘ American Journal of Evaluation, 38, pp. 29–46. doi:10.1177/1098214016644068.
Gertler, P., Martinez, S., Premand, P., Rawlings, L. and Vermeersch, C. (2016) Impact Evaluation in Practice, Second, Inter-American Development Bank and World Bank, Washington DC.
Girfalco, L. (1991) Dynamics of Technological Change, Van Norstrand Reinhold, New York.
Gläser, J. and Laudel, G. (2015) ‘A bibliometric reconstruction of research trails for qualitative investigations of scientific innovations,‘ Historical Social Research, 40, pp. 299–330.
Granhag, P., Strömwall, L. and Allwood, C.-A. (2000) ‘Effects of reiteration, hindsight bias, and memory on realism in eyewitness confidence,‘ Applied Cognitive Psychology, 14, pp. 397–420. doi: 10.1002/1099–0720(200009)14:5<397::AID-ACP658>3.0.CO;2-M.
Guba, E.G. and Lincoln, Y.S. (1989) Fourth-Generation Evaluation, Sage Publications, Thousand Oaks CA.
Hajer, M. and Laws, D. (2006) ‘Ordering through discourse’ in Moran, M., Rein, M. and Goodin, R. (eds) Oxford Handbook of Public Policy, Oxford University Press, Oxford, pp. 251–68.
Hogwood, B. and Gunn, L. (1984) Policy Analysis for the Real World, Oxford University Press, New York.
Kontigo. (2016) Effektanalys Av Vinnväxt-Programmet: Analys Av Effekter Och Nytta [Effect Analysis of the Vinnväxt-Programme: Analysis of Effects and Utility], VINNOVA, Stockholm.
Kostoff, R. (1993) ‘Semiquantitative methods for research impact assessment,‘ Technological Forecasting and Social Change, 44, pp. 231–44. doi:10.1016/0040–1625(93)90070-N.
Lothe, J. (2000) Narrative in Fiction and Film, Oxford University Press, Oxford.
Majone, G. (1989) Evidence, Argument and Persuasion in the Policy Process, Yale University Press, New Haven CT.
Mansfield, E. (1998) ‘Academic research and industrial innovation: an update of empirical findings,‘ Research Policy, 26, pp. 773–76. doi:10.1016/S0048–7333(97)00043–7.
Megill, A. (2007) Historical Knowledge, Historical Error: A Contemporary Guide to Practice, University of Chicago Press, Chicago.
Mohr, L. (1995) Impact Analysis for Programme Evaluation, Sage, Thousand Oaks CA.
National Academy of Sciences. (1999) Evaluating Federal Research Programmes: Research and the Government Performance and Results Act, National Academies' Press, Washington DC.
O'Keefe, C. and Head, R. (2011) ‘Application of logic models in a large scientific research programme,‘ Evaluation and Programme Planning, 34, pp. 174–84. doi:10.1016/j.evalprogplan.2011.02.008.
Pawson, R. and Tilly, N. (1997) Realistic Evaluation, Sage Publications, London.
Pen, J. (1985) Among Economists, North Holland, Amsterdam.
Penfield, T., Baker, M., Scoble, R. and Wykes, M. (2014) ‘Assessment, evaluations, and definitions of research impacts: a review,‘ Research Evaluation, 23, pp. 21–32. doi:10.1093/reseval/rvt021.
Popper, K.R. (1963) Conjectures and Refutations: The Growth of Scientific Knowledge, Routledge, London.
REF (2014, 02) Assessment framework and guidance on submissions, available from http://www.ref.ac.uk/pubs/2011–02/[accessed Jun 2017].
Roe, E. (1994) Narrative Policy Analysis: Theory and Practice, Duke University Press, Durham NC.
Scriven, M. (2008) ‘A summative evaluation of RCT methodology: & an alternative approach to causal research,‘ Journal of MultiDisciplinary Evaluation, 5, pp. 11–24.
Smith, K. and Larimer, C. (2017) The Public Policy Theory Primer, Westview Press, Boulder CO.
Steel, D. (2008) Across the Boundaries: Extrapolation in Biology and Social Science, Oxford University Press, New York.
Stern, E. (2013) ‘Editorial,‘ Evaluation, 19, pp. 3–4. doi:10.1177/1356389012473814.
Thomas, D. (2006) ‘A general inductive approach for analyzing qualitative evaluation data,‘ American Journal of Evaluation, 27, pp. 237–46. doi:10.1177/1098214005283748.
VINNOVA (2017) available from www.vinnova.se [accessed Jun 2017].
von Bahr, J. (ed). (2014) Hälsoekonomisk Effektanalys Av Forskning Inom Programmet Innovationer För Framtidens Hälsa. [Health Economic Effect Analysis of Research within the Programme ‘Innovation for Future Health‘], VINNOVA, Stockholm.
Weiss, C.H. (1998) ‘Have we learned anything new about the use of evaluation?‘ American Journal of Evaluation, 19, pp. 21–33.
White, M. (2010) ‘A contribution to current debates in impact evaluation,‘ Evaluation, 16, pp. 153–64. doi:10.1177/1356389010361562.
Wilson, J. (1973) Political Organizations, Basic Books, New York.
Wooding, S., Hanney, S., Buxton, M. and Grant, J. (2005) ‘Payback arising from research funding: evaluation of the arthritis research campaign,‘ Rheumatology, 44, pp. 1145–56. doi:10.1093/rheumatology/keh708.

Comments

Comment on this article

[1] Abbott, H. (2002) Narrative, Cambridge University Press, Cambridge.

[2] Bal, M. (2002) Narratology: Introduction to the Theory of Narrative, University of Toronto Press, Toronto.

[3] Cartwright, N. (2007) Hunting Causes and Using Them: Approaches in Philosophy and Economics, Cambridge University Press, Cambridge.

[4] Cartwright, N. and Hardie, J. (2012) Evidence-Based Policy, Oxford University Press, New York.

[5] Chatman, S. (1978) Story and Discourse: Narrative Structure in Fiction and Film, Cornell University Press, Ithaca NY.

[6] Donovan, C. (2011) ‘State of the art in assessing research impact. Introduction to a special issue,‘ Research Evaluation, 20, pp. 175–79. doi:10.3152/095820211X13118583635918.

[7] Donovan, C. and Hanney, S. (2011) 'The 'Payback Framework' explained,' Research Evaluation, 20, pp. 181–83. doi:10.3152/095820211X13118583635756.

[8] Eriksson, T., Roback, K., Lundqvist, M., Levin, L. and Carsson, P. (2011) Effektanalys Av Forskningsprogramme Inom Material Från Förnyelsebara Råvaror [Effect Analysis of Research Programmes on Materials from Renewables], VINNOVA, Stockholm.

[9] Fischer, F. (2003) Reframing Public Policy: Discursive Politics and Deliberative Practices, Oxford University Press, Oxford.

[10] Fischer, F. and Forrester, J. (1993). The Argumentative Turn in Policy Analysis and Planning, Duke University Press, Durham NC.

[11] Fischer, F. and Gottweis, H. (eds) (2012) The Argumentative Turn Revisited: Public Policy as Communicative Practice, Duke University Press, Durham NC.

[12] Forster, E.M. (1927) Aspects of the Novel, Harcourt, Brace & Co, New York.

[13] Gasper, D. (1996) ‘Analysing policy arguments,‘ European Journal of Development Research, 8, pp. 36–62. doi:10.1080/09578819608426652.

[14] Gates, E. and Dyson, L. (2017) ‘Implications of the changing conversation about causality for evaluators,‘ American Journal of Evaluation, 38, pp. 29–46. doi:10.1177/1098214016644068.

[15] Gertler, P., Martinez, S., Premand, P., Rawlings, L. and Vermeersch, C. (2016) Impact Evaluation in Practice, Second, Inter-American Development Bank and World Bank, Washington DC.

[16] Girfalco, L. (1991) Dynamics of Technological Change, Van Norstrand Reinhold, New York.

[17] Gläser, J. and Laudel, G. (2015) ‘A bibliometric reconstruction of research trails for qualitative investigations of scientific innovations,‘ Historical Social Research, 40, pp. 299–330.

[18] Granhag, P., Strömwall, L. and Allwood, C.-A. (2000) ‘Effects of reiteration, hindsight bias, and memory on realism in eyewitness confidence,‘ Applied Cognitive Psychology, 14, pp. 397–420. doi: 10.1002/1099–0720(200009)14:5<397::AID-ACP658>3.0.CO;2-M.

[19] Guba, E.G. and Lincoln, Y.S. (1989) Fourth-Generation Evaluation, Sage Publications, Thousand Oaks CA.

[20] Hajer, M. and Laws, D. (2006) ‘Ordering through discourse’ in Moran, M., Rein, M. and Goodin, R. (eds) Oxford Handbook of Public Policy, Oxford University Press, Oxford, pp. 251–68.

[21] Hogwood, B. and Gunn, L. (1984) Policy Analysis for the Real World, Oxford University Press, New York.

[22] Kontigo. (2016) Effektanalys Av Vinnväxt-Programmet: Analys Av Effekter Och Nytta [Effect Analysis of the Vinnväxt-Programme: Analysis of Effects and Utility], VINNOVA, Stockholm.

[23] Kostoff, R. (1993) ‘Semiquantitative methods for research impact assessment,‘ Technological Forecasting and Social Change, 44, pp. 231–44. doi:10.1016/0040–1625(93)90070-N.

[24] Lothe, J. (2000) Narrative in Fiction and Film, Oxford University Press, Oxford.

[25] Majone, G. (1989) Evidence, Argument and Persuasion in the Policy Process, Yale University Press, New Haven CT.

[26] Mansfield, E. (1998) ‘Academic research and industrial innovation: an update of empirical findings,‘ Research Policy, 26, pp. 773–76. doi:10.1016/S0048–7333(97)00043–7.

[27] Megill, A. (2007) Historical Knowledge, Historical Error: A Contemporary Guide to Practice, University of Chicago Press, Chicago.

[28] Mohr, L. (1995) Impact Analysis for Programme Evaluation, Sage, Thousand Oaks CA.

[29] National Academy of Sciences. (1999) Evaluating Federal Research Programmes: Research and the Government Performance and Results Act, National Academies' Press, Washington DC.

[30] O'Keefe, C. and Head, R. (2011) ‘Application of logic models in a large scientific research programme,‘ Evaluation and Programme Planning, 34, pp. 174–84. doi:10.1016/j.evalprogplan.2011.02.008.

[31] Pawson, R. and Tilly, N. (1997) Realistic Evaluation, Sage Publications, London.

[32] Pen, J. (1985) Among Economists, North Holland, Amsterdam.

[33] Penfield, T., Baker, M., Scoble, R. and Wykes, M. (2014) ‘Assessment, evaluations, and definitions of research impacts: a review,‘ Research Evaluation, 23, pp. 21–32. doi:10.1093/reseval/rvt021.

[34] Popper, K.R. (1963) Conjectures and Refutations: The Growth of Scientific Knowledge, Routledge, London.

[35] REF (2014, 02) Assessment framework and guidance on submissions, available from http://www.ref.ac.uk/pubs/2011–02/[accessed Jun 2017].

[36] Roe, E. (1994) Narrative Policy Analysis: Theory and Practice, Duke University Press, Durham NC.

[37] Scriven, M. (2008) ‘A summative evaluation of RCT methodology: & an alternative approach to causal research,‘ Journal of MultiDisciplinary Evaluation, 5, pp. 11–24.

[38] Smith, K. and Larimer, C. (2017) The Public Policy Theory Primer, Westview Press, Boulder CO.

[39] Steel, D. (2008) Across the Boundaries: Extrapolation in Biology and Social Science, Oxford University Press, New York.

[40] Stern, E. (2013) ‘Editorial,‘ Evaluation, 19, pp. 3–4. doi:10.1177/1356389012473814.

[41] Thomas, D. (2006) ‘A general inductive approach for analyzing qualitative evaluation data,‘ American Journal of Evaluation, 27, pp. 237–46. doi:10.1177/1098214005283748.

[42] VINNOVA (2017) available from www.vinnova.se [accessed Jun 2017].

[43] von Bahr, J. (ed). (2014) Hälsoekonomisk Effektanalys Av Forskning Inom Programmet Innovationer För Framtidens Hälsa. [Health Economic Effect Analysis of Research within the Programme ‘Innovation for Future Health‘], VINNOVA, Stockholm.

[44] Weiss, C.H. (1998) ‘Have we learned anything new about the use of evaluation?‘ American Journal of Evaluation, 19, pp. 21–33.

[45] White, M. (2010) ‘A contribution to current debates in impact evaluation,‘ Evaluation, 16, pp. 153–64. doi:10.1177/1356389010361562.

[46] Wilson, J. (1973) Political Organizations, Basic Books, New York.

[47] Wooding, S., Hanney, S., Buxton, M. and Grant, J. (2005) ‘Payback arising from research funding: evaluation of the arthritis research campaign,‘ Rheumatology, 44, pp. 1145–56. doi:10.1093/rheumatology/keh708.