Lessons from working across fields to develop a framework for informed choices

In late 2018, Iain Chalmers, Andy Oxman and others from the Informed Health Choices team convened a cross-field forum to develop a generic framework of key concepts for thinking critically about claims, research and choices about interventions, with the aim of supporting ‘informed choices’. We define an informed choice as one that is based on critical understanding of the relevant available evidence. This paper describes the process of that cross-field engagement, and reflects on how consensus was reached on the generic framework. Working in an alliance of 24 researchers from across fields to develop the Key Concepts for Informed Choices framework, we learned three lessons about crossfield working: (1) there was much agreement, despite diversity of views and experiences; (2) the applications of our work were broader than we could have imagined; and (3) we identified a wide range of problems that we have in common when making informed choices. Here we describe our experience of working together to develop the framework, and draw out lessons for others who may be involved in similar cross-field initiatives.

• There is considerable commonality in the evidence-informed approaches that exist in a wide range of fields.
• We need to document the processes of cross-field engagements, and not just their outputs.

Background
When Iain Chalmers, Andy Oxman and others from the Informed Health Choices team (www. informedhealthchoices.org) convened a cross-field forum to develop a generic framework to support people in making choices informed by critical understanding of the relevant available evidence (that is, 'informed choices'), they did not know how such an interdisciplinary group would work. A generic framework to support these informed choices was subsequently produced and published elsewhere (Aronson et al., 2019). This companion paper describes how we collaborated, the underlying discussions, and our experiences of working together across disciplines and fields (see Box 1 for definitions). The project had an inherently experimental feel from the outset. When we began this task, for instance, we were not sure whether we would all agree on the nature of the problem we were working to solve, whether we could identify enough shared items for inclusion in our framework, or whether we could produce something that was applicable across the full range of potential uses that our fields represented.
There is a growing body of research on the value of breaking down traditional boundaries in academia (see, for example, Heinze et al., 2009). This is combined with increased awareness of the need to work in transdisciplinary and cross-field ways to prepare our students for the future, and to generate useful knowledge for decision making (Sharples et al., 2017). Several organisations, including the Global Research Council (2016), the Wellcome Trust (2017) and the Research Council of Norway (2019) have encouraged interdisciplinary innovation, but there is little understanding of how such innovation can occur. To gain a sense of the challenges that interdisciplinary work imposes, one must first understand that each discipline trains its followers to engage with ideas and with peers, using frameworks that often differ fundamentally from discipline to discipline. For example, psychology very broadly focuses on individuals and their rights, responsibilities, norms and practices, while sociology prioritises the collective, and focuses on a shared community-level identity and centres of control. Educational research operates generally under an assumption that each learner, classroom and school is unique. A clinician, on the other hand, seeks to individualise the treatment of a patient with some condition, based on the average results of research in a population. These examples demonstrate the widely varying paradigms within which different academic disciplines operate, and provide some idea of why cross-disciplinary work is more unusual and more challenging than one might expect.
Evidence-based approaches focus on informing real-world choices. Helping individuals, communities, citizens and policymakers tackle important problems often requires solutions that span more than one field (Wilkins and Cooper, 2019). Improving evidence-informed decision making for a young child's health, for example, requires input from primary schools, public health practitioners, clinicians and families. Addressing issues such as high youth unemployment requires evidence on economic growth, higher education and learning, business development, entrepreneurship and innovation. The more complex the problem, the greater the need to break down traditional disciplinary boundaries.
The SARS-CoV-2 coronavirus disease (COVID-19) pandemic has highlighted the need for crossfield collaboration. For example, addressing the accompanying 'infodemic' (WHO, 2020a) requires more than epidemiology. Meanwhile, the recommendation to randomise school closures to improve certainty about their effects on both health and non-health outcomes  might be strengthened with collaborations involving colleagues from education and economics. There has been criticism that some national pandemic response teams are limited by their failure to include, for example, economists or behavioural or social scientists, while others have been challenged to be more transparent in their processes (see, for example, Clark, 2020). Meanwhile, the COVID-19 Evidence Network to support Decision-making (COVID-END) is bringing together several fields to help decision makers find the best available evidence, and to help researchers coordinate their efforts, rather than duplicating them (www. mcmasterforum.org/networks/covid-end).
Perhaps encouragingly, a large number of disciplines within broad fields, such as health, education and development (see Table 1), have committed to a shared goal of ensuring that decisions are informed by the best available evidence, and have established their own norms and practices for supporting evidenceinformed decision making. However, apart from a few exceptions, such as in health promotion and other allied health professions, these parallel initiatives seldom work together. This is despite a recognition of the potential for cross-field activities. Academic discussion and debate across evidence-based fields has largely taken place in silos, with limited cross-field discussion or fertilisation of ideas. For example, we know that parallel discussions have taken place in field-specific evidence-related conferences, such as the Cochrane Colloquia and the Collaboration for Environmental Evidence conferences, with little or no awareness that other experts in a different field are focusing on solving the same problems -the development of a multitude of methods for the synthesis of qualitative research is just one example (see Barnett-Page and Thomas, 2009). It was the combination of the shared commitment to the use of evidence across fields, and the recognition that the fields rarely work together, that motivated this crossfield initiative.

Interdisciplinarity and informed choices
The Informed Health Choices (IHC) project team has developed key concepts for thinking critically about health-care claims, research and choices (Chalmers et al., 2018). In part because it is hard to change adults' behaviours, they have focused largely on working with young people (Semakula et al., 2020). This reflects a general trend in this kind of work towards initiatives among schoolchildren or undergraduates (Oxman et al., 2021). The challenge, however, is not one that applies only in the health-care field, nor only to young people. Claims are made in the public arena every day about topics such as nutrition, ways of guaranteeing success in examinations, and how to make money quickly. Unfortunately, people often fail to question such claims, that is, to ask for or question supporting evidence. In recognition of the commonality of the problem of ensuring that decisions are informed by the best available evidence, and in the belief that solutions to our common problems are better sought together, the authors volunteered to join a cross-field group which convened in late 2018. Our aim was to develop a generic framework for assessing claims and supporting informed choices. The group included 24 researchers from a wide range  Table 1). What we had in common was that we all work to support the use of evidence in decision making in our respective fields. This paper shares lessons of interest to researchers and others working in cross-disciplinary collaborations. Our aim in sharing our observations is to engage further with what we have learned and identify areas for improvement to enable greater trans-disciplinary and cross-field work in the future.

Approach
We developed the shared framework in three phases: before, during and after a two-day meeting in Oxford (UK) in December 2018. Before the meeting, the IHC framework was circulated (Chalmers et al., 2018). All were encouraged to bring reflections on this to the meeting. This included sharing preliminary elements of the Informed Choices framework with colleagues, and getting their inputs into which elements they saw as relevant to their discipline. After the meeting, we continued to refine and develop our thinking by editing the emergent framework and preparing a report on our work (Aronson et al., 2019), and developing website content (https://thatsaclaim.org). Table 2 summarises aspects of our engagement.
Because of her personal interest in boundary-spanning initiatives, Ruth Stewart took notes about the process of working together across fields, which have directly contributed to this paper. In the subsequent months, she documented and shared key observations with the group by email, and invited others to reflect on these and help identify key lessons from the process of our cross-field working. Our shared reflections and contributions were distilled and grouped into themes, and are shared in this paper.

The resulting framework
Our cross-field work resulted in a framework that includes many concepts that are important to both laypeople and professionals aiming to make or support informed choices, within at least 14 fields. The concepts are in three overarching groups: (1) claims of effects that 'should be supported by evidence • Show respect for all contributions • Encouragement to avoid field-specific language; take care in using technical language, and develop a common language that transcends the boundaries of the individual fields • Identify common ground • Take our own notes, including those about problems with working together across fields and lessons learned After the meeting • Participation in continuing engagement by email, refining the shared framework • Inclusive authorship and contributions to a shared paper based on the framework • Continuing engagement in each of our fields to tailor, adapt and exemplify the key concepts for our specific contexts • Disseminate within fields, and to general audiences, and actively promote its use from fair comparisons'; (2) comparisons that should be 'fair comparisons, designed to minimise the risk of systematic errors (biases) and random errors (the play of chance)'; and (3) choices that 'depend on judgements about the problem, the relevance (applicability or transferability) of evidence available and the balance of expected benefits, harm and costs' (Aronson et al., 2019: 304). Each overarching group is unpacked within the cross-field framework reported in full and available online (https:// thatsaclaim.org).

Findings
Our findings about the process of developing this framework reflect three broad themes: 1. the extent and nature of agreement over the problems we were trying to solve, and when diversity of experiences and views were voiced 2. the extent to which we identified common problems and prioritised them for inclusion in the framework we were developing, while also recognising differences 3. the breadth of applicability of the framework we developed.
In addition, we found several themes that cut across the elements of the Informed Choices framework that we developed. These cross-cutting themes are summarised in Table 3.
The extent and nature of agreement about the problems we were trying to solve On the first day, we focused on understanding the work to date by those who had convened the meeting (the IHC project), and to develop shared understanding of the value of a cross-field framework. All Claims: Claims about effects should be supported by evidence from fair comparisons. Other claims are not necessarily wrong, but there is an insufficient basis for believing them.
Common observation that interventions get implemented in good faith based on a belief that something works, but belief is not enough to substantiate the claim, and resources can be wasted and harm caused by such claims.
Claims are not made in a vacuum, but are influenced by a wide range of other factors, and often become the accepted norm, even if contrary to the evidence base.
Small-scale studies are common across disciplines, and are part of the problem.
There is a need for examples of harms that can occur, and resources wasted, when you are not making decisions based on evidence, to make the point in a powerful way so that people will take notice.
Comparisons: Studies should make fair comparisons, designed to minimise the risk of systematic errors (biases) and random errors (the play of chance).
There is a need for common terms for describing study designs that make fair comparisons, and that are designed to minimise biases.
There is a lack of high-quality evidence on which to base decisions at all levels, from practice to policy.
Choices: What to do depends on judgements about the problem, the relevance (applicability or transferability) of the evidence available, and the balance of expected benefits, harms and costs.
Recognition of the trade-offs that decision makers make, and that these take place at several levels, balancing multiple benefits, harms and costs.
Cognitive bias seems to be a stumbling block. participants acknowledged early on that our shared purpose centred on helping individuals and groups make well-informed decisions about interventions, primarily by improving the quality of communication of scientific information. We quickly recognised the value of working together, and the need to enable individuals to think critically about claims. At no point was the goal of developing a shared framework to support the use of evidence for informed choices challenged.
Despite the breadth of backgrounds and experiences within the group, several of us had taken part in cross-disciplinary work previously, and we self-selected to an extent in accepting the invitation to take part in this initiative. We were therefore each already convinced about the value of working across traditional boundaries. Discussions at the meeting were rich, varied and wide ranging. We agreed that across all fields, choices (when options are considered) and decisions (when one action is chosen) are wide ranging and differ from one another in degree, scale and nature. Discussions across fields about the need for a common framework helped us to think more broadly about the nature of the informed choices people make, and the kinds of claims with which they are faced. We learned from both our commonalities and our differences.
We also recognised the need to consider all levels of decision makers, from individuals making choices for themselves and their families, to health-care professionals and government policymakers making choices on behalf of individuals and whole populations. We recognised the extent to which public policy in its broadest sense (that is, policy that affects the public, whatever its source) is often shaped by a wide range of considerations. This included discussion of how policymakers understand public debate. We acknowledged that their decisions are shaped by perceived public support, official considerations, and a complex interaction between what the evidence suggests and how much policymakers are willing to risk political failure in following the evidence. We also perceived barriers for researchers/scientists in whether they inform policy, and how that occurs. We discussed the role that misinformation can play, and agreed on the importance of tackling it at all levels.
Discussions across fields helped us to see how the framework we were developing could be useful beyond health choices. For example, we discussed the strength of the available evidence and agreed that small-scale studies, which are common across disciplines, are part of the problem behind many misleading claims. We found examples from each of our fields, for example, a systematic review of management research that found a predominance of small studies with low internal validity (Barends et al., 2014).
The commonalities across fields not only related to experiences and ideas, but also to the degree of applicability of the framework across those fields. We all agreed that a mechanistic hypothesis, although important, is not on its own enough to justify implementation of an intervention until it has been tested.
A key observation was that interventions, implemented in good faith based on a theory that something should work, can lead to wasted resources and even to harm. We did nevertheless observe that the development of theory, often using tools such as logic models, is a common precursor to any experiment, with differing levels of importance in different fields.
We observed that informed choices are not made in a power vacuum. The power and influence behind some ideas are forceful. For example, microloans to people living in poverty, with the intention of supporting them to generate income through small businesses, are usually provided at very high interest rates, and an overwhelming body of evidence suggests that this does more harm than good. Nevertheless, the desire to find solutions for poverty makes the intervention hugely attractive and very popular. Muhammad Yunus -considered by many as the founding father of microfinance -was awarded the Nobel Peace Prize in 2006. It is difficult to challenge accepted truths and change norms.
Other examples include the lack of global support for electric cars (Paine et al., 2006), the decades before the harms of smoking were accepted (Warner, 1991), and the limited action against the sugary drinks industry (Public Health England, 2019). Sometimes those with influence, whether big pharma, big donors or political parties, put their weight and their money behind an intervention, and it takes a lot to counter this. Most recently, we have observed persistence of the belief that hydroxychloroquine is an effective treatment for COVID-19, despite high-quality evidence to the contrary, following support for it from US President Donald Trump (Khuroo et al., 2020). A few public initiatives specifically challenge accepted truths, including Ask for Evidence (www.askforevidence.org). We agreed that all decision makers, including at national and international levels, have multiple factors to balance in decision making.
We found common ground when considering the important roles that the media (including social media) play in influencing the choices that people make, and explored how to counter the power of misinformation in all forms of media. Another shared challenge related to the power of commercial interests, and how well-resourced (but not necessarily evidence-based) organisations can use their financial weight to drive policies and interventions that are extremely difficult to challenge without considerable funding and effort.
The potential value of a generic framework of key concepts to inform choices was agreed, as reflected in a willingness to press on with understanding applications in different fields, and implications for the detailed sections of the framework (see more below).

How we identified and prioritised common problems for inclusion in the framework
When we considered adapting the original Informed Choices tool, which was developed for decisions in health care, into a framework for use across different fields, one of the main topics we discussed related to wording, the use of particular terminology and semantics, as opposed to differences in the concepts themselves. Colleagues in different disciplines use different jargon to characterise studies that focus on comparisons -randomised clinical trials, randomised social experiments, cluster-randomised trials, placerandomised trials. For example, in most disciplines, a randomised trial is referred to as an 'experiment', but in the domain of management, an experiment means 'trying something out to see what happens', and is a synonym for 'pilot'. Nevertheless, it was understood that the phrases embody a common idea, that of fair comparisons in estimating outcomes. In another example, colleagues in each discipline alluded in varied ways to the tentativeness of evidence. We agreed that this concept of uncertainty is fundamental, and a reason for doing more than one study and for learning from mistakes.
Once we had aired, discussed and recognised the subtle differences in how we use words such as 'trial', 'experiment', 'decision', 'policy' and 'harm', we moved on to discussing the framework itself. This helped us to identify common factors that shape decisions, which in turn were included in the new crossfield framework.
We observed that across all fields, decision making involves trade-offs at several levels, balancing multiple benefits, harms and costs. We learned that in some fields it is routine to consider harms and costs (medicine, dentistry, veterinary medicine and international development, for example). In other fields, benefits, harms and costs might take longer to become apparent, and harms may not even be considered, for example, in education (Zhao, 2017). With some decisions it is necessary to balance benefits, harms and costs across multiple people, groups or populations of other species, for example, in environmental management. In some fields, such as planetary health, some people for whom this balance must be considered may not even have been born yet. Costs vary enormously across fields. They include both fiscal costs and opportunity costs, and they can be immediate or long term. In agriculture, for example, intensification through increased use of pesticides and/or fertilisers may meet increasing food production requirements, but can cause harm in the wider environment. In the longer term, there may also be cost implications, particularly to future generations, as damage to natural support services, such as soils or pollinators, can have economic consequences.
Opportunity costs vary according to context: what we have to give up in order to adopt a course of action can vary for each individual, and from group to group and place to place. In resource-poor settings, the opportunity cost of seemingly trivial actions can be enormous. Furthermore, we recognised that across fields, decisions can be made at many levels (personal, for family or friends, in a school or hospital, in a city, regionally, nationally or internationally), and the distribution of harms, benefits and costs varies with context. Who gains and who loses can sometimes be as important as maximising an outcome, if not more so. In evidence-based management, this is known as 'stakeholder evidence' (yet another term which means different things in different fields) -evidence that allows analysis of impact on different groups and introduces an ethical dimension. The need for fair comparisons to enable balancing of options cuts across fields and levels of decision being made.
Language commonly generates tension in all fields; for example, the contrast between warning language and neutral statements -'Beware false news' versus 'This is what an evidence-based claim looks like'. We agreed that the framework should contain statements that are as neutral as possible about the value of evidence, which could be adapted in different fields and for different uses. For example, the framework states: 'Comparisons of interventions should be fair' (Aronson et al., 2019: 304), rather than 'Beware comparisons that are unfair'.
We developed the option for each field of enhancing the framework and sharing additional fieldspecific information. We agreed that claims are not made in a vacuum, and we discussed the importance of ethical issues. However, we decided not to include them in the framework, because they can be context specific. We agreed that irrespective of the field, there is a need for relevant examples of what happens when decision making is not evidence based, so that people will take notice. These have been included in field-specific sections of the project website (www.thatsaclaim.org).

The breadth of applicability of the framework
While we discussed how we might adapt the framework for different fields, there was never any suggestion that the framework would not be useful in any specific field. Rather, there was a lot of common ground across fields. Participants described cross-field work as a pleasure, reflecting a new way of working without disciplinary boundaries, engendering optimism. We felt reassured that we all speak the same broad language, and that the principles of evidence-informed decision making (and thus the framework that we developed) apply across all fields. To date, the project website includes field-specific enhancements for 8 of the 14 fields involved in the development of the framework (www.thatsaclaim.org).
Finally, there was a shared recognition that the framework is not an endpoint for any of our fields. Implementation of decisions is an essential aspect of informed choices, and is often lacking, whether we are discussing implementation of individual or policy choices. Policy choices in any field require implementation.

Discussion
Cross-field, interdisciplinary approaches are possible, and the learning from them is hard to overestimate. With COVID-19 dominating many current policy debates, and the proliferation of related research requiring synthesis, there are calls for disciplinary breadth (Stewart et al., 2020;WHO, 2020b). The lessons from our work are highly relevant for anyone convening these forums. Agencies that aim to tackle other global priorities, from climate change to Black Lives Matter, all require cross-field engagement. We hope that in sharing our experiences, we can contribute a little to these engagements.
An increasing number of organisations and initiatives support the use of evidence in decision making, as well as seeking to address misinformation about claims. Fact-checking organisations, such as Africa Check, Ask for Evidence and iHealthFacts, are on the rise around the world. The potential to apply, reflect and collectively learn from cross-field engagements is considerable, and increasing in importance.
We know of many other cases in which agreement has been more difficult to achieve. In 2003, the International Campaign to Revitalise Academic Medicine (ICRAM), led by an interdisciplinary working party, accepted the challenge of reinventing academic medicine, but found the task too difficult (ICRAM, 2004). Some within the group suggested that they had not understood the problem (Clark, 2005). Relative to ICRAM, our focus was much more specific, and we began with the common ground of evidence synthesis in supporting decision making.
It has also been suggested that because disagreement on committees is rife, consensus in health care is reached only on 'bland generalities that represent the lowest common denominator of debate and are embalmed as truths' (Buetow et al., 1997: 269). However, we were not a 'committee' responsible for reaching a consensus, but had freedom to engage and to choose whether or not the resulting framework was useful for our own fields. We were also encouraged to adapt, and either broaden or narrow, the key concepts for our own fields. There was no pressure from a funder or other authority that required a consensus, or even an output.
We suspect that respect for one another also encouraged constructive dialogue. We consistently shared concrete examples from our work, and this paper reports just a few of those. We believe that voicing opinion alone would not have been sufficient to reach consensus, and that the examples enabled us to find common ground.
In conclusion, despite representing broad and varied interests, and belonging to traditionally siloed academic disciplines, we were able to reach agreement on a framework. Through the process of working together, we came to appreciate the breadth of applicability of the evidence-based approach. Readers who find the boundaries of traditional disciplines frustrating should be encouraged by our experience.
On how unusual it has been for cross-field working to take place in some areas of the evidence approach: Stewart, R. and Oliver, S. (2006) 'Reviewing the potential for critical appraisal training to cater for professional practice'. Medical Teacher, 28 (2)