147
views
0
recommends
+1 Recommend
1 collections
    0
    shares

      If you have found this article useful and you think it is important that researchers across the world have access, please consider donating, to ensure that this valuable collection remains Open Access.

      Prometheus is published by Pluto Journals, an Open Access publisher. This means that everyone has free and unlimited access to the full-text of all articles from our international collection of social science journalsFurthermore Pluto Journals authors don’t pay article processing charges (APCs).

       
      • Record: found
      • Abstract: found
      • Article: found
      Is Open Access

      Achieving impact: impact evaluations and narrative simplification

      research-article
      a , b
      Prometheus
      Pluto Journals
      Bookmark

            Abstract

            This study is concerned with how impact from research and innovation (R&I) programmes is accounted for in impact evaluation reports. Establishing causal links between a research funding instrument and different effects, poses well known methodological difficulties. In the light of such challenges, textual accounts about causal links ought to be carefully written. Nevertheless, impact evaluation reports have a tendency towards unwarranted simplification as far as impact inferences are concerned. In this study, we illustrate how such simplifications – versions of the narrative device ellipsis – are accomplished. Using examples from three Swedish impact evaluation reports, we focus on the constituent components of longer impact accounts, that of the impact argument, to analyze the various ways that impact is narratively achieved through simplification. We believe this analysis can contribute to the methodology of impact evaluation, as well as spread light on some the difficulties in the historiography of innovation in general.

            Content

            Author and article information

            Contributors
            Journal
            10.2307/j50022063
            prometheus
            Prometheus
            Pluto Journals
            0810-9028
            1470-1030
            1 September 2017
            : 35
            : 3 ( doiID: 10.1080/prometheus.35.issue-3 )
            : 215-230
            Affiliations
            [ a ]School of Economics and Management, Lund University, Lund, Sweden
            [ b ]Department of Psychology, Göteborg University, Göteborg, Sweden
            Article
            08109028.2018.1522829
            10.1080/08109028.2018.1522829
            31fcaac4-a178-4b69-b594-6eb1be1982ad
            © 2017 Pluto Journals

            All content is freely available without charge to users or their institutions. Users are allowed to read, download, copy, distribute, print, search, or link to the full texts of the articles in this journal without asking prior permission of the publisher or the author. Articles published in the journal are distributed under a http://creativecommons.org/licenses/by/4.0/.

            Custom metadata
            eng

            Computer science,Arts,Social & Behavioral Sciences,Law,History,Economics

            References

            1. Abbott, H. (2002) Narrative, Cambridge University Press, Cambridge.

            2. Bal, M. (2002) Narratology: Introduction to the Theory of Narrative, University of Toronto Press, Toronto.

            3. Cartwright, N. (2007) Hunting Causes and Using Them: Approaches in Philosophy and Economics, Cambridge University Press, Cambridge.

            4. Cartwright, N. and Hardie, J. (2012) Evidence-Based Policy, Oxford University Press, New York.

            5. Chatman, S. (1978) Story and Discourse: Narrative Structure in Fiction and Film, Cornell University Press, Ithaca NY.

            6. Donovan, C. (2011) ‘State of the art in assessing research impact. Introduction to a special issue,‘ Research Evaluation, 20, pp. 175–79. doi:10.3152/095820211X13118583635918.

            7. Donovan, C. and Hanney, S. (2011) 'The 'Payback Framework' explained,' Research Evaluation, 20, pp. 181–83. doi:10.3152/095820211X13118583635756.

            8. Eriksson, T., Roback, K., Lundqvist, M., Levin, L. and Carsson, P. (2011) Effektanalys Av Forskningsprogramme Inom Material Från Förnyelsebara Råvaror [Effect Analysis of Research Programmes on Materials from Renewables], VINNOVA, Stockholm.

            9. Fischer, F. (2003) Reframing Public Policy: Discursive Politics and Deliberative Practices, Oxford University Press, Oxford.

            10. Fischer, F. and Forrester, J. (1993). The Argumentative Turn in Policy Analysis and Planning, Duke University Press, Durham NC.

            11. Fischer, F. and Gottweis, H. (eds) (2012) The Argumentative Turn Revisited: Public Policy as Communicative Practice, Duke University Press, Durham NC.

            12. Forster, E.M. (1927) Aspects of the Novel, Harcourt, Brace & Co, New York.

            13. Gasper, D. (1996) ‘Analysing policy arguments,‘ European Journal of Development Research, 8, pp. 36–62. doi:10.1080/09578819608426652.

            14. Gates, E. and Dyson, L. (2017) ‘Implications of the changing conversation about causality for evaluators,‘ American Journal of Evaluation, 38, pp. 29–46. doi:10.1177/1098214016644068.

            15. Gertler, P., Martinez, S., Premand, P., Rawlings, L. and Vermeersch, C. (2016) Impact Evaluation in Practice, Second, Inter-American Development Bank and World Bank, Washington DC.

            16. Girfalco, L. (1991) Dynamics of Technological Change, Van Norstrand Reinhold, New York.

            17. Gläser, J. and Laudel, G. (2015) ‘A bibliometric reconstruction of research trails for qualitative investigations of scientific innovations,‘ Historical Social Research, 40, pp. 299–330.

            18. Granhag, P., Strömwall, L. and Allwood, C.-A. (2000) ‘Effects of reiteration, hindsight bias, and memory on realism in eyewitness confidence,‘ Applied Cognitive Psychology, 14, pp. 397–420. doi: 10.1002/1099–0720(200009)14:5<397::AID-ACP658>3.0.CO;2-M.

            19. Guba, E.G. and Lincoln, Y.S. (1989) Fourth-Generation Evaluation, Sage Publications, Thousand Oaks CA.

            20. Hajer, M. and Laws, D. (2006) ‘Ordering through discourse’ in Moran, M., Rein, M. and Goodin, R. (eds) Oxford Handbook of Public Policy, Oxford University Press, Oxford, pp. 251–68.

            21. Hogwood, B. and Gunn, L. (1984) Policy Analysis for the Real World, Oxford University Press, New York.

            22. Kontigo. (2016) Effektanalys Av Vinnväxt-Programmet: Analys Av Effekter Och Nytta [Effect Analysis of the Vinnväxt-Programme: Analysis of Effects and Utility], VINNOVA, Stockholm.

            23. Kostoff, R. (1993) ‘Semiquantitative methods for research impact assessment,‘ Technological Forecasting and Social Change, 44, pp. 231–44. doi:10.1016/0040–1625(93)90070-N.

            24. Lothe, J. (2000) Narrative in Fiction and Film, Oxford University Press, Oxford.

            25. Majone, G. (1989) Evidence, Argument and Persuasion in the Policy Process, Yale University Press, New Haven CT.

            26. Mansfield, E. (1998) ‘Academic research and industrial innovation: an update of empirical findings,‘ Research Policy, 26, pp. 773–76. doi:10.1016/S0048–7333(97)00043–7.

            27. Megill, A. (2007) Historical Knowledge, Historical Error: A Contemporary Guide to Practice, University of Chicago Press, Chicago.

            28. Mohr, L. (1995) Impact Analysis for Programme Evaluation, Sage, Thousand Oaks CA.

            29. National Academy of Sciences. (1999) Evaluating Federal Research Programmes: Research and the Government Performance and Results Act, National Academies' Press, Washington DC.

            30. O'Keefe, C. and Head, R. (2011) ‘Application of logic models in a large scientific research programme,‘ Evaluation and Programme Planning, 34, pp. 174–84. doi:10.1016/j.evalprogplan.2011.02.008.

            31. Pawson, R. and Tilly, N. (1997) Realistic Evaluation, Sage Publications, London.

            32. Pen, J. (1985) Among Economists, North Holland, Amsterdam.

            33. Penfield, T., Baker, M., Scoble, R. and Wykes, M. (2014) ‘Assessment, evaluations, and definitions of research impacts: a review,‘ Research Evaluation, 23, pp. 21–32. doi:10.1093/reseval/rvt021.

            34. Popper, K.R. (1963) Conjectures and Refutations: The Growth of Scientific Knowledge, Routledge, London.

            35. REF (2014, 02) Assessment framework and guidance on submissions, available from http://www.ref.ac.uk/pubs/2011–02/[accessed Jun 2017].

            36. Roe, E. (1994) Narrative Policy Analysis: Theory and Practice, Duke University Press, Durham NC.

            37. Scriven, M. (2008) ‘A summative evaluation of RCT methodology: & an alternative approach to causal research,‘ Journal of MultiDisciplinary Evaluation, 5, pp. 11–24.

            38. Smith, K. and Larimer, C. (2017) The Public Policy Theory Primer, Westview Press, Boulder CO.

            39. Steel, D. (2008) Across the Boundaries: Extrapolation in Biology and Social Science, Oxford University Press, New York.

            40. Stern, E. (2013) ‘Editorial,‘ Evaluation, 19, pp. 3–4. doi:10.1177/1356389012473814.

            41. Thomas, D. (2006) ‘A general inductive approach for analyzing qualitative evaluation data,‘ American Journal of Evaluation, 27, pp. 237–46. doi:10.1177/1098214005283748.

            42. VINNOVA (2017) available from www.vinnova.se [accessed Jun 2017].

            43. von Bahr, J. (ed). (2014) Hälsoekonomisk Effektanalys Av Forskning Inom Programmet Innovationer För Framtidens Hälsa. [Health Economic Effect Analysis of Research within the Programme ‘Innovation for Future Health‘], VINNOVA, Stockholm.

            44. Weiss, C.H. (1998) ‘Have we learned anything new about the use of evaluation?‘ American Journal of Evaluation, 19, pp. 21–33.

            45. White, M. (2010) ‘A contribution to current debates in impact evaluation,‘ Evaluation, 16, pp. 153–64. doi:10.1177/1356389010361562.

            46. Wilson, J. (1973) Political Organizations, Basic Books, New York.

            47. Wooding, S., Hanney, S., Buxton, M. and Grant, J. (2005) ‘Payback arising from research funding: evaluation of the arthritis research campaign,‘ Rheumatology, 44, pp. 1145–56. doi:10.1093/rheumatology/keh708.

            Comments

            Comment on this article