Introduction
The nature of contemporary scientific inquiry has been the subject of considerable debate over the last 25 years. Several major works published in the mid-1990s and early 2000s argue that scientific practices are in the midst of a transformation. This transformation is the result of greater integration with other spheres in society – particularly industry, government and civil society – that are pushing scientists to engage in transdisciplinary, issue-based and collaborative research at the expense of what is seen as more independent, disciplinary and curiosity-driven inquiry (Gibbons et al., 1994; Slaughter and Rhoades, 1997; Etzkowitz and Leydesdorff, 1997; Nowotny et al., 2001; Rip, 2004). While there is substantial variety in the arguments presented in the transformationalist literature (which we will address more thoroughly in the following section), a common theme across most works is that scientists and universities are under increasing pressure to demonstrate and enhance the ‘relevance’ of their research to actors outside the traditional academic community (Hessels et al., 2009). However, the transformationalist argument has been contested by others who argue that the narrative is too simplistic (detailing what the authors either hope or fear will happen to science), that it is historically inaccurate (i.e. that the ideal of secluded science has been an historical anomaly) or that such changes are taking place at the fringes of academic science rather than at its core (Shinn, 1999, 2002; Mirowski and Sent, 2008; Hansen, 2009). For critics, this is less a story of transformation than of incremental and variegated evolution as scientists adapt to changes in funding environments and social expectations without radically altering their practices.
Empirical research on this debate and the questions it raises has tended to focus on either the micro- or the macro-level. Micro-level research usually involves in-depth case studies of research groups or organizations, which are often presented as convincing illustrations of transformation (Tuunainen, 2005, p.284; Wyatt, 2015). Macro-level analyses are focused mostly on shifts in national research policy and funding (e.g. Metcalfe, 2010; Leisyte, 2011). While both types of study are valuable, there is a need for more meso-level comparative research that goes beyond single case studies while simultaneously looking at how policies are implemented or translated into practice. Research at the meso-level allows us to understand better the relationship between policy and practice across several sites or cases, looking at both the expressions and the limits of different aspects of the transformationalist thesis as they actually appear across multiple contexts and areas of research.
In this paper, we take a comparative approach in examining the effects of Canada’s Networks of Centres of Excellence (NCE) programme and its explicitly transformational mandate on 14 NCE-sponsored research networks. Founded in 1989, the NCE programme has been described as ‘the most dramatic change in the nation’s science policy in a hundred years … [one that is] explicitly designed to change the norms of science’ in Canada (Fisher et al., 2001, p.299). Its primary goal has been to support research initiatives based on partnerships among Canadian academics and industry, government, and/or civil society groups that address problems and issues of national interest and concern (Halliwell and Smith, 2011). The NCE programme has grown significantly since its inception, and is now one of the largest and most prestigious funding programmes in the country (Atkinson-Grosjean, 2006). Yet, little is known about how the rhetoric and directives of the NCE programme are implemented (or not) by the research groups that work with NCE funding. The aim of this paper is to investigate and compare how multiple NCE projects define ‘societal relevance’, particularly in the light of the rhetoric, policy directives and performance assessments coming from the central funding agency. We do this by analysing reports, public statements and web materials of the 14 NCE projects as well as those of the governing funding body.
We came to this research question through personal experience. Both authors were involved with an NCE project whose members struggled to determine what exactly it meant to pursue greater societal relevance, how to engage better with potential users of their research, and how to measure and evaluate their efforts and outcomes. These difficulties were not caused by lack of commitment or desire, but appeared to be the result of confusion over what the transformationalist mandate meant and how it could be fulfilled. These experiences prompted us to question how other NCE projects were tackling these same challenges in the context of the ongoing debate over the extent to which a transformation is underway in the nature of scientific inquiry.
The transformationalist thesis and its critics
Arguments about the transformation of science have revolved around several key concepts advanced by authors in the mid-1990s and early 2000s, including Mode 2 science (Gibbons et al., 1994), post-normal science (Funtowicz and Ravetz, 1993), triple helix (Etzkowitz and Leydesdorff, 1997), academic capitalism (Slaughter and Rhoades, 1997) and strategic science (Rip, 2004). The ideas behind these concepts are summarized in Table 1. This list is not exhaustive, however, and other relevant concepts not directly considered here include reflexive modernization (Beck et al., 1994), the enterprise university (Marginson and Considine, 2000), and post-academic science (Ziman, 2000). There are notable similarities across the concepts in Table 1, particularly the argument that scientists and scientific processes are being increasingly exposed to pressures and forces emanating from other spheres of society, as well as to actors and institutions that judge the relevance and impact of scientific findings according to market, social and political/governance values. The differences across the concepts are also important. Tuunainen (2005) argues that the main distinction among them is the degree of change implied. The notions of Mode 2, post-normal science, and the triple helix imply a marked break with a previous era of scientific practice and/or a transformation of the mandates and missions of universities relative to other institutions. In contrast, the concepts of academic capitalism and strategic science can be seen as contested changes in the landscape of scientific governance that prompt reaction among researchers (such as adaptation and/or resistance) rather than radical change in orientation or behaviour.
| Concept | Main arguments | Key sources |
|---|---|---|
| Mode 2 science | Mode 1 science (disciplinary, curiosity-driven, inward-looking and secluded from society) is being replaced by Mode 2 science that is transdisciplinary, application-oriented, participative and intentionally responsive to markets, politics and/or culture. Knowledge is evaluated on its social robustness or public acceptability as much as by methodological robustness. The lines between science and other institutions blur, and are more easily transgressed by all types of actor | Gibbons et al. (1994); Nowotny et al. (2001) |
| Post-normal science | Real-world problems have become more complex, blending natural and human variables. Higher uncertainty means higher political stakes for scientific facts and arguments. The importance of ‘normal’ puzzle-solving science diminishes in the face of a post-normal science that takes human values into account. The peer community is extended to include non-scientists, and stakeholders are taken more seriously as contributors and evaluators of knowledge. Holistic methods and philosophies challenge the reductionist approach to generating knowledge | Funtowicz and Ravetz (1993) |
| Triple helix | The mandates, values and practices of universities increasingly overlap with those of states and private industry, creating new hybrid entities. Technology transfer and business incubation take on prominent roles in research practice and support. The capitalization of knowledge becomes a more acceptable goal (or even the dominant motive) for researchers, who adopt the characteristics and attitudes of entrepreneurs | Etzkowitz and Leydesdorff (1997) |
| Academic capitalism | University and government administrators push researchers to accept structures and policies based on market principles, including competitiveness, productivity and profit orientation. Universities are losing their special status in society and becoming much like other entrepreneurial organizations. Government funding declines in favour or public–private issue-based hybrid projects and institutes | Slaughter and Rhoades (1997) |
| Strategic science | Governments increasingly see science as a tool for furthering national interests. Research funding is targeted to practical problem-solving, reflecting the priorities and ideologies of government sponsors. Excellence and relevance are considered together in allocating funding. Differentiation of universities, departments and networks is encouraged, based on perceived comparative advantage | Rip (2004, 2011) |
Critics of the transformationalist thesis generally focus on three interrelated points: (1) that the narratives are overly simplistic and do not reflect context-specific conditions or the haphazard and uneven pace of change; (2) that transformationalist arguments are empirically thin and/or historically inaccurate, failing to recognize the fact that science has a long history of application to real-world problems and embeddedness in other spheres of society; and (3) that transformationalist concepts leave little room for the agency of scientists, who are quite capable of thinking for themselves and are known to anticipate and adapt to outside pressures without radically altering their behaviours and practices (see Weingart, 1997; Shinn, 1999, 2002; Hansen, 2009). These criticisms converge on an interesting point made by Nowotny, Hall and Gibbons (originators of the Mode 2 concept) that the idea of transformation has been more popular among politicians and civil servants ‘struggling to create better mechanisms to link science with innovation’ than with practising scientists themselves (Nowotny et al., 2003, p.179). As we will discuss below, from its inception, the NCE programme has very much been a case of politicians and bureaucrats pushing the idea of transformation onto researchers from the top down. Funding agencies are among the most powerful institutional actors in contemporary science, given their substantial influence over the criteria for allocating funds and evaluating research projects and outcomes (Rip, 1994; Lave et al., 2010; Lane and Rogers, 2011). The responses of individual scientists to these pressures are not well understood, but occupy the whole spectrum from enthusiastic acceptance to outright resistance, with substantial indifference, confusion and opportunistic adaptation between these two poles (see Singh et al., 2014; Thoren and Breian, 2016). The motivation of our research is to understand how these institutional pressures affect the practice of research groups caught up in the top-down drive for greater societal relevance.
Transformation by design: Canada’s Networks of Centres of Excellence programme
Canada’s NCE programme was launched in 1989 at a time of national uncertainty about the country’s intellectual and industrial future. The Canada–US free trade agreement had just been signed, leading some to speculate that Canadian industry would suffer from greater competition with US firms, and that high-end research talent would be tempted to migrate south as part of a wide-ranging ‘brain drain’. In this context, the NCE programme was intended to produce ‘big science in a small country’ by funding interdisciplinary networks of researchers across multiple universities focused on specific research areas or problems (Fisher et al., 2001, p.323). The NCE programme has been strongly influenced by the transformation ideal, particularly the ideas of Mode 2 science (Halliwell and Smith, 2011). Michael Gibbons, one of the leading proponents of the Mode 2 concept, served as an advisor to the programme’s architects, and was a member of the committee that evaluated the first NCE proposals (Fisher et al., 2001, p.312).
The NCE programme is run jointly by the Ministry of Industry and Canada’s three major research councils (with respective emphases on the natural sciences, humanities and social sciences, and health sciences). The research councils coordinate the peer review of applications for funding, while the Ministry of Industry makes final decisions on which projects merit support. The Ministry also hosts the powerful NCE Secretariat, which is responsible for coordinating and communicating policy priorities and directives across the NCE programme, as well as evaluating the performance of individual NCEs. Overall, the NCE programme gives priority to proposals that demonstrate high potential for producing scientific research that leads to ‘real-life policies, products and services’, and/or those that make a demonstrable contribution to Canadian economic and social development (NCE, 2012a). Applicants and contributors are expected to develop partnerships with government, private industry and/or civil society groups as collaborators in the research. For instance, applicants are required to seek direct and in-kind funding from outside groups, and to include non-academics on the NCE’s board of directors (NCE 2012a). Once funded, an NCE becomes relatively autonomous – free to call for and evaluate proposals for specific projects, hire staff, and come up with its own internal policies and governance strategies – although it must follow the general governance directives determined by the NCE Secretariat. NCEs are funded for seven years, with the possibility of one renewal following evaluation of all aspects of the network, including research output, partnerships, commercial and policy-relevant products, and governance. NCEs may continue after 14 years, but are expected to be self-funded by then through income from patents, commercial products, and the licensing of technology and intellectual property (Atkinson-Grosjean, 2006, p.159).
Methods
As mentioned earlier, our research investigates how ‘societal relevance’ is defined and pursued by 14 different NCE projects, compared with the official definitions and measures emanating from policy authorities (in this case, the NCE Secretariat), as a window into ongoing debates about the transformation of contemporary scientific practices. At the conceptual stage of this project, we considered several possible methodological approaches. Preliminary requests for interviews were sent to representatives of the 14 NCEs (see Table 2), but few positive responses were received. This problem was compounded by the policy of the Canadian government at the time (headed by Conservative prime minister Stephen Harper) that forbade civil servants from speaking directly to journalists or academic researchers (Young and Coutinho, 2013), which meant that no members of the NCE Secretariat could be interviewed.
| NCE | Area of research |
|---|---|
| Allergen | Allergy diseases and allergy-related socio-economic consequences |
| ArcticNet | Impacts of climate change and modernization in the coastal Canadian Arctic |
| Auto 21 | Automotive innovations |
| Canadian Arthritis Network (CAN) | Arthritis and arthritis-related socio-economic consequences |
| Canadian Institute for Photonic Innovations (CIPI) | Photonics innovations |
| Canadian Stroke Network (CSN) | Stroke-related disease and related socio-economic consequences |
| Canadian Water Network (CWN) | Fresh water management |
| Carbon Management Network (CMN) | Fossil fuel industry and other stationary emitters of carbon dioxide |
| Geomatics for Information Decisions (GEOIDE) | Geomatics; computer mapping; spatial information |
| Graphics, Animation and New Media (GRAND) | Digital media; gaming and interactive simulation; animation, graphics and imaging |
| MPrime | Development of mathematical tools vital to knowledge-based economy |
| NeuroDevNet | Causes of neurological deficits and brain development in children |
| PrioNet | Prions and prion-related diseases |
| Stem Cell Network (SCN) | Cell-based therapy and therapeutics |
To maintain the comparative breadth of the study, we chose to focus on public documents and texts from the NCE Secretariat and the 14 NCEs. This method relies heavily on website posting and archives. Social scientists are increasingly seeing websites as important sites of cultural and organizational research (Pauwels, 2012; Ackland, 2013; Varis, 2016). New fields, such as Internet studies and digital ethnography, conceptualize online environments as arenas for social action and expression of comparable importance to corporeal environments (Pauwels, 2005, 2012; Garcia et al., 2009; Coleman, 2010; Varis, 2016). In these fields, websites are no longer considered mere appendages of organizations, but critical tools for community-building and legitimation (Hine, 2005; Coleman, 2010). Internally, the loose, geographically-dispersed nature of NCE projects means that websites are often an important channel of communication among researchers and project administrators, via such tools as links, guidelines, pages, reporting tools and news feeds (Hine, 2005). As such, they are critical for communicating expectations, establishing practices and sharing experiences (Coleman, 2010). Externally, websites are important points of contact with potential and actual partners, as well as with the curious public at large. It follows that websites contain important statements of justification and explanation of NCE activities. We agree with other authors who argue that websites are not mere repositories of neutral information, but windows into cultural values and practices from which an understanding of organizational priorities and imperatives can be discerned (Lovejoy and Saxton, 2012; Pauwels, 2012). Website research is particularly compelling in our case because each NCE is required to develop, maintain and update a comprehensive website as a condition of funding.
Research began with the collection of the entire public contents of each NCE’s website, as well as that of the NCE Secretariat. The contents were collected over two calendar years (2011 and 2012). Each website was visited several times during the study period, and updates to key pages were also collected. The websites varied in structure, suggesting that each was developed independently and not according to any pre-set or shared template. The collected text was subjected to two rounds of inductive coding (Thomas, 2006). To minimize potential observer bias, coding was conducted collaboratively by both authors using a consensus model, meaning potential codes were accepted only on agreement of both analysts.
Coding focused on two phenomena: (1) how ‘relevance to outside actors’ is defined by each NCE; and (2) how the performance of NCEs with respect to relevance is measured and evaluated by the Secretariat. Statements about scientific relevance were found mostly on web pages containing the NCE’s mission statement, research descriptions, training programmes, FAQs, event notices and summaries, and descriptions of partnerships and linkages to non-academic groups. The first round of content coding involved careful reading of each web page to extract information on the ‘who, how and why’ associated with statements about scientific relevance and engagement within each network. The ‘who’ involved cataloguing the researchers, collaborators, partners and sponsors involved in producing or disseminating knowledge. This included information on both individuals and groups (private, government and civil society), as well as the hierarchies and role(s) they play within the network or relative to its activities. The ‘how’ involved cataloguing network practices and procedures for the conduct of research, and the communication, disclosure and/or sharing of research findings. The ‘why’ involved recording reasoning and justifications for NCE programmes and strategies, as well as statements about expected outcomes for network members, partners and/or the public at large. The resulting data were then subjected to a qualitative sorting analysis, which is an exploratory technique in which thematically similar quotations and information are grouped together to investigate the presence of possible ‘core themes’ that reach across multiple (but not necessarily all) cases (Morse, 1994). The second round of coding involved a re-examination of the web pages, using the core themes previously identified as ‘retrieval codes’ (Olsen, 2012).
The annual reports of the NCE Secretariat were subjected to the same two-stage coding process described above, with particular attention paid to: (1) ‘what’ is being measured (what is designated an indicator of performance, such as number of patents or partnerships); and (2) ‘how’ it is being measured (using type of indicator, such as numerical counts, descriptive case studies, and/or quotations and narratives). The longitudinal dimension also gives us information about how the themes have been conceptualized and how these measures have (or have not) changed over time.
Findings
First we consider findings from our analysis of how relevance is defined by the NCEs and the Secretariat. This is follow by discussion of findings regarding the measurement and evaluation of performance and outcomes.
Defining relevance: comparing the NCE Secretariat and 14 networks
The public documents and statements collected from the 14 NCEs and the NCE Secretariat website were subject to two rounds of inductive coding. The first round of coding resulted in the identification of four core themes associated with the conceptualization of relevance: (1) the role of science in society; (2) collaboration and partnership; (3) commercialization; and (4) knowledge sharing and disclosure. Table 3 summarizes the four core themes as well as the indicators used to identify each one in the second round of coding. We describe each core theme in turn.
The first core theme to appear in multiple documents from both the NCE Secretariat and individual networks involves claims about the role of the NCE and the scientific knowledge it generates in broader society. As discussed earlier, an important part of the transformational narrative is that university science as it has traditionally been practised – in cloistered laboratories housed in departments and divided by discipline – is no longer adequate or appropriate in an era of rapid social, economic and technological change. This position is particularly reminiscent of the prescriptive arguments made by proponents of Mode 2 science and post-normal science.
The rhetoric of the NCE Secretariat reflects these ideas, but gives them a decidedly capitalist spin. For example, Suzanne Fortier (NCE, 2012b), the president of the NCE Secretariat at the time of data collection, stated that:
It is no secret that Canada is among the world’s best when it comes to scientific excellence. Through the NCE program, we are making even greater strides in putting that research into practice for the benefit of all Canadians. Over the past year, the NCE Networks and Centres have increased their focus on building a solutions-based economy in partnership with industry, hospitals, provincial governments, communities, and many others.
The quotation also illustrates the NCE Secretariat’s repeated use of nationalist language to justify its emphasis on relevance to non-academic (and particularly economic) actors. These arguments are in line with the concepts of the triple helix and of strategic science, in which scientific resources are seen as key assets to be marshalled for specific projects to enhance national interests. The NCE Secretariat portrays the programme as a tool for harnessing Canadian scientific potential to support political and economic priorities, arguing that Canadian scientists and universities must be:
… encouraged to [help build] a more competitive and sustainable Canadian economy with the help of science and technology. … The most important role of the Government of Canada is to ensure a competitive marketplace and create an investment climate that encourages the private sector to compete against the world on the basis of their innovative products, services, and technologies. Canada must maximize the freedom of scientists to investigate and of entrepreneurs to innovate. (Industry Canada, 2007, p.4)
Our research found that each individual NCE made at least one statement about the broader societal role of its research activities and findings. These statements are summarized in Table 4 using the specific words, terms and phrases found on each project’s website. While the terms vary, we note the dominance of economic and market-based concepts in discussions of the role of scientific knowledge in society. Nearly every NCE makes a direct claim about economic benefits associated with the network in the form of either revenue generation or cost reduction. Even health-related NCEs, which have a broader mandate than some of the technology-focused NCEs, make frequent direct reference to economic gain. For example, CAN, whose mandate is to ‘improve the quality of life of people with arthritis’, describes its more tangible benefits as: ‘decreas[ing] the personal, societal and economic burden of the disease and promot[ing] the growth of the Canadian economy through arthritis R&D in areas of unmet need’ (CAN, 2012a).
| NCE | The stated societal role of the NCE and its knowledge |
|---|---|
| Allergen | Reduce costs; position Canada in the forefront of innovation; provide socio-economic benefits for Canada |
| ArcticNet | Northern industrial development; Canadian economic and social benefits |
| Auto 21 | Canadian economic and social benefits; competition in the global economy; enhancement of quality of life |
| CAN | Improved quality of life; personal and socio-economic benefits; creation of tangible outputs; decreasing the personal, societal and economic burden of arthritis |
| CIPI | To improve Canada’s position at innovation; benefits to the country; to make Canadian industry competitive in the international marketplace; to generate wealth; to enhance quality of life; to solve specific problems |
| CSN | To decrease physical, social and economic burdens; to maximize health and economic benefits |
| CWN | To increase economic prosperity; to improve public good of Canadians; solution-oriented research |
| GEOIDE | Substantial benefits to society and the economy; to consolidate the geomatics industry |
| GRAND | Solution-driven products and services; to enhance Canada’s digital media sector |
| MPrime | High impact on Canada’s economy; benefit of all Canadians |
| NeuroDevNet | Improvement of lives of children, families and communities |
| PrioNet | To enhance human and animal health; socio-economic benefits for all Canadians; to make Canada a leader in prion research |
| SCN | To relieve the health care burden; to provide economic opportunities |
Note: All terms and phrases are direct quotations. Distinct quotations are separated by a semi-colon.
The second core theme in the NCE websites involves claims about collaboration and partnership. The NCE Secretariat demands collaboration as a condition of funding. NCEs are required to be inter-university organizations, uniting geographically-distant researchers and departments into a cohesive research network, held together by common research goals, shared research projects and rituals, such as regular workshops and conferences. In an earlier analysis of NCE documents, Fisher et al. (2001, p.322) concluded that these ‘programme documents convey a sense that the country can no longer afford researchers who isolate themselves in the academy, pursuing esoteric problems at public expense’. All NCEs are keen to demonstrate a high level of inter-university collaboration.
Collaborations beyond academia, however, are another matter. The Secretariat defines partners (sometimes called ‘end users’) broadly, stating that Networks must produce ‘leading-edge research findings that are relevant to the needs of the user sector (e.g. private and public sectors, non-governmental organizations, and others)’ (NCE 2012a) but also that Networks are to ‘work with end users to accelerate the creation and application of new knowledge’ (NCE 2012c). The number and types of partners participating in the network were found to vary with each NCE. Some NCEs, such as Auto 21, have a tight relationship with industry (in this case, the automotive industry). Others, such as the Canadian Water Network (CWN) are predominantly partnered with government, while still others have close partnerships with civil society groups (see Table 5).
| NCE | Descriptions of partnership and collaboration in each NCE |
|---|---|
| Allergen | Unite public and private partners; networking; partnership; academic–industry liaison |
| Auto 21 | Federal, industry and provincial support; public/private sector collaborative research |
| ArcticNet | Bring together scientists, managers, Inuit organizations, northern communities, government and private sector |
| CAN | Avoid individual research efforts; to bring experts together; multi-institutional partnership; collaborative research |
| CIPI | Collaboration between researchers and industry; university–industry collaboration |
| CSN | Collaborative effort |
| CWN | To catalyze strong partnerships; against individual research projects; collective insight and research |
| GEOID | To promote collaboration; multidisciplinary collaboration |
| GRAND | Actively engage researchers; to develop valuable industry relations |
| MPrime | To bring together scientists and industry; collaboration |
| NeuroDevNet | Involving individuals and organizations in research |
| PrioNet | Connecting stakeholders with research organizations |
| SCN | Collaboration and partnerships |
Note: All terms and phrases are direct quotations. Distinct quotations are separated by a semi-colon.
Equally important is how the NCEs conceptualize and structure collaboration in practice. For example, some NCEs have moved to involve non-academic collaborators in key research and administrative activities, while others engage with them in more restrictive ways only at the conclusion of the research process. There is a good deal of variation between these extremes however, leading us to suggest that strategies for collaboration should be considered a continuum. At the more restrictive end of the continuum, partners are assumed to have a passive role as receivers of knowledge and information. They are the targets of education and awareness campaigns, and are invited to consult with NCE-funded experts about the results of their work. For example, the CAN network describes its relationship with users this way:
[CAN provides] services aimed at facilitating research, innovation, commercialization, and research knowledge communication. These services are available to CAN members, CAN partners and the broader arthritis stakeholder community. From the early development of a product to post marketing surveillance, we offer access to a wide-range of expert resources, with services that are personalized and bundled to meet the needs of our users. (CAN, 2012b)
This contrasts with the more progressive end of the continuum, where relations between scientists and non-academic collaborators are more complex and communication is multi-directional. Here, partners are invited to sit on key committees, to participate in the drafting of research priorities, evaluations of funding proposals and internal peer review processes. ArcticNet provides the clearest illustration of this type of engagement. This NCE’s charter contains commitments to the ‘bilateral exchange of knowledge, training and technology’ and to ‘involve Northerners in the scientific process’ (ArcticNet, 2012a). A group called the ‘Inuit Research Advisors’ (IRA) participates in various stages in the evolution of research designs and the production of scientific knowledge. IRA’s official role is to ‘assist and advise researchers’ in the proposal development, dissemination of research results and development of projects of the NCE. In this, the NCE aims to ‘represent a new way of knowledge sharing and engagement’ with relevant partners and communities (ArcticNet, 2012b).
CWN stands perhaps at the midpoint of the spectrum. This network has developed a partnership programme, the Consortia Program, which is ‘designed to engage end users, and findings confirm that they are actively involved. End users and academics are about equally represented among the most-connected participants in the Consortia Program’. Unlike ArcticNet, CWN’s Consortia Program is not directly involved in research, but is rather a forum for networking and consultation. This is a case in which partners are invited to engage with the NCE and its researchers, but in a non-central role that does not directly affect the research process.
The third core theme is commercialization of research outcomes. The NCE Secretariat is clear that the programme’s mandate is to increase the commercialization of scientific knowledge, stating that:
A key NCE program objective is to advance Canadian economic and social development. Accordingly, every effort must be made to have the results of Network-funded research exploited in Canada, for the benefit of Canadians. Benefit to Canada is defined as incremental Canadian economic activity and improved quality of life in Canada. Maximum benefits would be derived from the creation of high-quality jobs in Canada and this should be an important goal of any commercialization activity. (NCE, 2012a)
| NCE | Descriptions of commercialization by each NCE |
|---|---|
| Allergen | Speed to market; economic benefits; dissemination of research |
| Auto 21 | To market better products; maximization of returns |
| ArcticNet | Potential for commercialization |
| CAN | Innovation and commercialization; development of products |
| CIPI | To commercialize applications; to make new commercial products |
| CSN | To commercialize research advances; to create products that produce revenue |
| CWN | To capitalize investment in research |
| CMN | Technologies from conception to commercialization |
| GEOID | Commercialization of results |
| GRAND | Commercialization engine |
| MPrime | Commercialization of research outcomes |
| NeuroDevNet | To capitalize on genomic advances |
| PrioNet | To support commercialization of research |
| SCN | Commercialization of the research |
All terms and phrases are direct quotations. Distinct quotations are separated by a semi-colon.
The fourth core theme involves strategies for knowledge sharing and disclosure to potentially interested parties beyond the scientific community and core partners. Such terms as ‘knowledge translation’, ‘knowledge exchange’ and ‘knowledge mobilization’ have become common shorthand for attempts to maximize the chances of knowledge being adopted and used by non-scientific actors in the broader public sphere (see Fazey et al., 2012). The scholarly literature around each of these terms asserts that there are significant barriers to the communication and uptake of scientific knowledge that take substantial effort to overcome. Knowledge that is directly carried across social and organizational barriers (via knowledge brokers and/or boundary organizations) is thought to have a greater chance of societal impact, as is knowledge that is presented in narrative form (similar to storytelling) or translated directly into possible applications (Fazey et al., 2012; Young et al., 2016; Minke-Martin, 2016).
Our analysis suggests that this is a significant point of divergence and difference among the NCEs, and between the NCEs and the Secretariat. Table 7 shows that despite its prominent rhetorical place in the overall NCE programme, there is very little consistency or consensus on what is meant by knowledge sharing and its related terms. Our review of the Secretariat’s annual reports (discussed in detail in the next section) found references to ‘knowledge translation’ in the NCE programme since at least 1998. However, despite the longstanding use of knowledge terms in official documents, we found that individual NCEs conceptualize the term in at least four different ways:
- (1)
as the conversion of scientific knowledge into specific practices, tools and technologies to be applied outside of academia;
- (2)
as the exchange or transfer of information among expert and non-expert (or consumer) communities;
- (3)
as a necessary step for a more efficient and evidence-based public policy-making process; and
- (4)
as the creation of a ‘learning community’ through the communication of synthesized knowledge created by the NCE’s experts.
| NCE | Descriptions of knowledge sharing and disclosure |
|---|---|
| Allergen | Knowledge translation (discovery and development link); knowledge and technology exchange and exploitation (KTEE); mandatory disclosure of potential commercial results; translation as development of trials and adoption of best practices |
| ArcticNet | Translation into assessments, policies and strategies; knowledge sharing and engagement |
| Auto21 | [No information provided] |
| CAN | Research knowledge communication; knowledge translation and exchange resources; exchange of information between researchers and receptor organizations |
| CIPI | Knowledge exchange among stakeholders; dissemination and exchange of scientific knowledge; workshops and student exchanges |
| CMN | Knowledge sharing linking researchers and practitioners |
| CSN | To translate research findings (policy-making and improved public management) |
| CWN | KT/KB strengthens science-policy links; increased coordination and sharing of information; knowledge mobilization |
| GEOIDE | Sharing expertise; supporting educational programmes |
| MPrime | Knowledge transfer (patents, licences); knowledge exchange |
| NeuroDevNet | Knowledge translation (dissemination of ‘easy-to-understand’ knowledge); to build bridges and cultivate innovation; provision of services; synthesization of knowledge; to translate knowledge into clinical practices |
| PrioNet | Knowledge mobilization; translation and transmission of scientific knowledge |
| SCN | Translation of knowledge into therapies, commercial products and public policy |
These understandings of knowledge translation are not mutually exclusive, and several NCEs deploy two or more conceptualizations simultaneously. For instance, the SCN claims to act as ‘a catalyst for Canadian research that translates stem cell research into new therapies, commercial products and public policy’ (SCN, 2012), thus touching on the first and third concepts described above. CWN (2012) draws on the second, third and fourth of these in its knowledge translation and brokering plan (KT/KB), aimed at:
… those working at the science–policy interface, operating as knowledge brokers and/or translators, or seeking to develop contacts and greater awareness in this field … [by providing] the opportunity to develop new networks and knowledge, exchange best practices, and contribute to an active and growing KT/KB community.
Variability in understandings and practices of knowledge translation and the more recent terms – ‘knowledge mobilization’, ‘knowledge exchange’ and ‘knowledge transfer’ – are not necessarily problematic. Indeed, this variability can be interpreted as a sign of strength, as NCEs grapple with various strategies for reaching their different non-academic audiences. However, close reading of the relevant documents suggests a good deal of confusion remains. Part of the problem appears to stem from the conceptual vagueness and imprecision of the terms themselves. Lack of consensus definitions means that these terms are frequently used interchangeably and somewhat haphazardly. For example, the NCE Mprime uses the term ‘knowledge transfer’ in a very similar way to what SCW or SCN label ‘knowledge translation’. Similarly, PrioNet refers to ‘knowledge mobilization’ as the outcome of ‘both the scientific investigation process and the social processes of transmission and translation of scientific knowledge’ (PrioNet, 2012), while CWN (2012) refers to it as:
… various activities through which the people who use research findings for decision-making (end users and decision-makers) and the people who do research (researchers and their graduate students or assistants) work together and share knowledge throughout the research process to improve the application, uptake and relevance of research for decision making.
The conceptual imprecision, vagueness and variability in how these terms are deployed suggests that the pathways to broader societal impact are not as clear to the Secretariat and many of the individual NCEs, as are, say, the pathways to commercialization discussed previously. Broader societal impact via knowledge sharing and disclosure are key tenets of several transformationalist arguments, particularly Mode 2 and post-normal science. We argue that the difficulties shown by the Secretariat and the NCEs in addressing this issue signals an important limit in the programme’s push for transformation in scientific practice – a theme we return to later.
Measuring and assessing Mode 2 activities
Next we address findings about how the Secretariat measures and evaluates the performance, relevance and impact of individual NCEs. The purpose of this part of our research is to understand how the Secretariat assesses whether NCEs are meeting the transformational mandate of the programme, and thus indirectly how it recognizes and rewards individual NCEs for achieving better ‘relevance-oriented’ outcomes in its research activities and partnerships. The core sources for this analysis are the annual reports produced by the NCE Secretariat between 1998 and 2014. These reports are intended to summarize the activities of individual NCEs as well as the overall programme, and to communicate achievements to network members, political authorities and the public at large.
An overview of the range and type of measures and indicators used in the annual reports is presented in Table 8. It is important to note at the outset that the terminology and measures in these reports have changed very little over time. The changes that have occurred have tended to be minor, such as the addition of such newly-minted terms as ‘knowledge mobilization’ after 2010. We also note that the annual reports became highly succinct from 2012, presenting far less data than in previous years (shrinking from 50–75 pages to approximately five–six pages). This may mask recent changes in how the NCEs are evaluated, but we found no evidence of such changes in our broader review of Secretariat documents.
Sources: NCE (2009) and NCE Secretariat, Annual Reports, 1998–2014.
Our main finding from the annual reports is that the measures used by the NCE Secretariat to evaluate programme success focus strongly on some aspects of the transformationalist narrative, while mostly – and in some cases completely – ignoring other key dimensions central to its own rhetoric. For example, while numerous measures exist for economic impacts of NCEs, such as number of jobs created, number of spin-off companies created and ‘demonstrated impacts of network innovations on existing industries’, there are no explicit measures of civil society, community or end-user engagement, particularly the progressive multilateral forms of upstream involvement pursued by such NCEs such as ArcticNet.
Overall, the Secretariat’s measures of performance and impact tend to privilege ‘hard indicators’ – activities that can be captured numerically – while downplaying or ignoring ‘soft indicators’ that are more difficult to quantify (Bouckaert and Van de Walle, 2003). For example, data on collaborations and partnerships, which our document analysis showed to be rich and highly variable across NCEs, are represented in the annual reports as numerical counts of joint activities undertaken, such as projects, publications, numbers of researchers involved in the NCE, committees struck and outside financial contributions. The softer details of partnerships and collaborations appear only selectively as showcase descriptions of fewer than 100 words interspersed throughout the documents.
A consequence of the focus on hard indicators is that some aspects of ‘relevance’ are given greater prominence in the public measuring and evaluation of NCE projects, while other aspects are ignored, diminished or relegated to background. This is perhaps clearest in the cases of commercialization and knowledge sharing. Both were shown in the previous section to be rhetorically important to the NCE mission, but find themselves treated quite differently in the annual reports. Commercialization is a goal that is highly amenable to quantitative measurement, and Table 8 shows that commercialization outcomes are measured across multiple dimensions, including number of patents and licence agreements, revenue increase, new products and services, new companies, and technology transfers. By contrast, knowledge sharing activities and outcomes (including such sister terms as ‘knowledge translation’, ‘knowledge mobilization’, ‘knowledge exchange’, etc.) are not directly measured. The absence of measures means that the evaluation and promotion of knowledge sharing activities in the NCE annual reports is incomplete. In practice, it means that a given NCE’s success in knowledge sharing is judged only by proxy – often using the same criteria as those for commercialization and collaboration and partnerships. For example, the 2004–05 annual report of the NCE Secretariat states the following:
The NCE program exists to mobilize knowledge translation. In 2004–2005, there were more than 7,000 researchers and highly qualified personnel involved in NCE projects and activities. NCE scientists were issued 37 patents (up from 31 in 2003–2004), published 5,673 papers in refereed scientific journals (up from 3,564) and were granted or were negotiating 48 licenses. Six spin-off companies were created to translate NCE-generated knowledge into practice. (NCE, 2005, p.18)
This quotation illustrates the problem of imprecision in the measurement and evaluation of these softer, more qualitative processes and outcomes. In the absence of good measures to capture activities that are not readily counted, the Secretariat reverts to narrower metrics that are based predominantly on traditional scientific output and market outcomes. The full range of engagement and knowledge sharing activities – particularly those involving community and civil society groups –is not recognized in these public appraisals of NCE projects.
Discussion and conclusions
The aim of this research has been to investigate how NCE projects have defined and sought to enhance the societal relevance of their work alongside the rhetoric, policy directives and performance assessments used by the NCE Secretariat to influence and enforce these goals. The findings outlined in the previous sections lead us to several conclusions. First, our research demonstrates that both the NCE Secretariat and the individual NCE projects are deeply rhetorically committed to transformationalist ideals. The four core themes identified in our analysis of communications from the NCE projects and Secretariat are broadly consistent with the core concepts reviewed at the outset of this paper (see Table 1). Our study found that nearly all projects have adopted strong position statements on such issues as the role of NCE science in society, collaborations and partnerships, commercialization, and the importance of broader knowledge sharing and transfer. The idea that science should directly benefit Canada and Canadians also plays a prominent role in these communications. Such rhetorical endorsements are significant, because they communicate a set of expectations to scientists, partners and the general public that NCE science is to be conducted in a particular way, one that implies a break from the practices of classic or traditional science.
However, our second main finding is that this rhetoric – much of which originates from the NCE Secretariat – has not resulted in coherent or consistent practices across NCE projects. On some of the core themes, the NCEs are highly congruent in their words and actions. On commercialization, for instance, most of the NCEs examined use similar language, strategies and measures to describe their plans and achievements. In contrast, the NCEs were shown to differ markedly on issues such as engagement with partners and potential users, and practices for broader knowledge sharing, transfer and mobilization. The differences among NCEs on these dimensions were marked enough for us to suggest a continuum of approaches: at one end are NCEs that see partners and potential users solely as consumers of research and expertise, and at the other end are NCEs that attempt to involve directly non-academic actors in multiple facets and stages of NCE research and activities, with the goal of facilitating multi-directional communication and learning.
There are two possible interpretations of this finding. Both are likely to be partly correct. On the one hand, it is likely that individual NCEs are tailoring their efforts to the problems and audiences specific to their area of research. ArcticNet, for instance, has the most developed system for partner engagement, but this is perhaps unsurprising given its focus on Canada’s High North with its readily identifiable population of stakeholders. Considering the range of NCEs and their areas of research, experimentation to find the best strategic fit for engagement and knowledge sharing ought to be encouraged. On the other hand, there is evidence to suggest that the observed variety reflects a fundamental confusion over what high-minded concepts such as ‘knowledge mobilization’, ‘collaboration’ and ‘public engagement’ actually mean in practice (see Stilgoe et al., 2014; Young et al., 2016). NCEs use the terms in varying and sometimes contradictory ways, at times using them with reference to routine and not at all transformative activities, such as conference presentations and supporting educational programmes (see Table 7). This implies that, in practice, the pursuit of greater societal relevance is proceeding somewhat haphazardly through the NCE programme, despite strong rhetorical congruence.
Our third major finding is that there is a significant imbalance in how the NCE Secretariat measures and evaluates the success of individual NCEs and the umbrella programme more generally. Our analysis of annual reports between 1998 and 2014 revealed a strong preference for quantifiable hard indicators, most of which emphasize economic and commercial impacts, over softer indicators that touch more on processes and qualitative factors, such as levels of engagement and multi-directional learning. Given that the annual reports are public documents intended to justify and promote the NCE programme, we argue that this imbalance leads to an overemphasis on certain types of ‘relevance’ – particularly economic achievements – while downplaying or backgrounding other aspects, such as knowledge sharing, non-economic impacts and community involvement. The paucity of measures of key transformational terms, such as ‘knowledge translation’ and ‘knowledge mobilization’ is also striking, and again suggests that these concepts are still not well understood or consistently used even by the institutions that promote them as the basis for transformational scientific endeavours.
We conclude this paper with a brief discussion of what our findings mean for the academic debate on transformationalism. This research was inspired by our participation in an NCE network, in which we observed fellow researchers and leaders struggle to define what exactly is meant by pursuing greater societal relevance, better engagement with potential users of research, and measuring and evaluating such efforts and outcomes. These experiences prompted us to engage in a meso-level analysis across multiple similar cases. Meso-level comparative research allows us to look beyond single case experiences, while also examining how macro-level phenomena, such as national policies, are implemented by creative actors who may endorse, resist or re-interpret different aspects of policies in their translation into practices. Ultimately, our findings support some of the arguments made by both proponents and critics of the transformationalist thesis. Consistent with the arguments of proponents, our research confirms the conclusions of prior micro- and macro-level studies of the NCE programme, which have argued that the NCE programme has had a significant impact on Canadian science, pushing scientists to change their practices in pursuit of greater connections with, attention to and alignment with non-academic actors (Fisher et al., 2001; Metcalfe, 2010; Halliwell and Smith, 2011). However, our analysis also shows that the implementation of these ideals by NCE networks and researchers has been uneven. While the NCE Secretariat and individual NCEs have developed consistent means of pursuing and measuring commercial success and collaborations among scientists, on other dimensions, particularly knowledge sharing and the engagement of (non-commercial) potential users, experiences are more varied and ultimately more limited. Although experimentation is ongoing in several NCEs, these efforts are not being adequately recognized by existing measures of network and programme success. By narrowing the scope of measurement, recognition is denied for certain types of experimentation. In line with critics, these findings suggest that transformationalist arguments are too simplistic and fail to differentiate among different types and paces of change (Shinn, 2002; Hansen, 2009). From this perspective, transformation looks more incremental than radical, more rhetorical than substantive.