How do we know what works? Evaluating data on the extent and impact of young people’s involvement in English health research


The case for children and young people’s involvement in health research has been well documented, but less attention has been paid to the experiences and impact of involving children and young people rather than adults. This paper explores these issues in relation to a project undertaken for the James Lind Initiative on the availability, quality and reliability of existing data on young people’s involvement in studies supported by the National Institute for Health Research (NIHR). The project found that there was no systematic way of getting reliable and comparable information about which young people are involved in health research in England, what form that involvement may have taken or the impacts of involvement. There is a need to better collate and disseminate evidence on young people’s involvement in research, using both existing systems and processes, and the possible development of new metrics and measures. Not having this information risks children and young people’s involvement continuing to develop in a piecemeal fashion. Further research is needed about how, when and where children and young people are involved in health research, and about the impact of involvement on research and on the children and young people involved.


Introduction Background
Health research, as defined by the National Institute for Health Research (NIHR), is 'research which aims to improve the current and future health and care of the population' by 'providing research evidence to inform health and care practice' (NIHR, n.d.). Public involvement in health and social care research refers to research 'carried out "with" or "by" members of the public, rather than "to", "about" or "for" them' (Hayes et al., 2012). In other words, working with people with relevant lived experience in the design, delivery and dissemination of research, as opposed to them being involved as research participants, that is, as sources of data. This can take many forms, for example, 'working with research funders to prioritize research, offering advice as members of a project steering group, commenting on and developing research materials, undertaking interviews with research participants' (INVOLVE, 2020). Involvement can take place at all stages of the research process, including: commissioning; management (for example, membership of a project steering committee); research design and the development of proposals; data collection and analysis; reporting (for example, co-authoring reports or lay summaries); and dissemination (for example, presenting at conferences or co-authoring journal articles). It can also be at different levels of power-sharing, including: one-off consultations seeking people's views on health research; collaboration (an ongoing partnership between researchers and members of the public/patients); co-produced research, in which power and responsibility are shared from the start to the end of a project; and user-led research (research that is actively controlled, directed and managed by service users/members of the public and their organizations) (Brady and Graham, 2019).
Involving those who are the focus of health research has been found to have a positive impact on what is researched, how research is conducted, and the impact of research findings on services and in the lives of those involved (Staley, 2009). By making use of people's knowledge, lived experience and networks, the argument is that researchers can provide more relevant, higher-quality research that is more widely communicated (Barber et al., 2011;Stewart and Liabo, 2012). All publicly funded health research in the UK is now expected to have some element of public involvement (Evans et al., 2014). In 2014, an independent panel conducted a review of public involvement in the NIHR, Breaking Boundaries, which led to a report entitled Going the Extra Mile (NIHR, 2015). The report called for a new vision, mission and set of principles to underpin public involvement in research, and for public involvement in the NIHR to be focused on six common goals: • Opportunities to be involved in research are visible and seized on by the public.
• It is standard practice for the public and professionals to work together.
• The experience of patients, service users and carers is valued.
• Public involvement is a required part of high-quality research.
• Evidence of what works is easily available and can be put into practice (our emphasis). • The NIHR has maintained its global presence and influence for working with the public.
The role of the NIHR Clinical Research Network (CRN) is to provide support for researchers to deliver studies to 'time and target', and to provide the infrastructure to make research 'work' in the NHS. In April 2014, the NIHR CRN underwent significant reorganization intended to streamline and simplify the processes that support research delivery.
The NIHR CRN encourages researchers to involve the public in the early stages of research development. The CRN provides advice on public involvement, and works collaboratively with NIHR partners to ensure continuity for the patients and carers involved, so that they can share their insights throughout the research process.
But while the body of knowledge on the rationale, methods and contextual factors for public involvement has expanded over the past decade, the quality of reporting is often inconsistent, which limits understanding of how it works, for whom and why (Staniszewska et al., 2017). A perceived lack of data to demonstrate the impact of involvement upon research (Brett et al., 2014) has resulted in a drive to outline standards of good involvement practice (UK Standards for Public Involvement, n.d.) and an increasing interest in measuring the impact of public involvement (for example, Evans et al., 2014;Wilson et al., 2015;Crocker et al., 2017).

The case for involving children and young people
The majority of children and young people involved in health research in the NIHR and elsewhere are between the ages of 8 and 21, so this is the age range generally implied in this article.
The case for children and young people's involvement has been well documented (for example, Percy-Smith and Thomas, 2010), and it is a popular concept in many healthcare organizations (Weil et al., 2015). There are two principal arguments for involving children and young people in health research: a rights-based moral argument that it is the 'right' thing to do; and an impact-or evidence-based argument that involvement has benefits for the children and young people participating, for research, and for the services and policies that draw on this research evidence (Brady and Graham, 2019).
Public involvement is seen as a right when a group of people might be affected by the research topic, or 'as part of good governance in ensuring transparency of decisions and accountability in the use of public funds' (Oliver et al., 2015: 206). The rights-based argument for children and young people's involvement centres on the United Nations Convention on the Rights of the Child (UNCRC; United Nations, 1989), which established international recognition that all children have the right to the highest possible standards of health care and the right to have a say in decisions that affect them (Alderson, 2014). The UNCRC sets out children's rights in terms of both their protection and their participation in society, and 'asserts children's right to have a voice in decision-making, as well as rights to freedom of thought and expression' (Percy-Smith and Thomas, 2010: 1). The key article relating to participation, Article 12, states that: 'States Parties shall assure to the child who is capable of forming his or her own views the right to express those views freely in all matters affecting the child' (United Nations, 1989).
The CRC-informed understanding that children and young people should be involved in decisions that affect them is increasingly reflected in law, regulation, policy and research guidance (Franklin and Sloper, 2005). Furthermore, Article 13 states that children have the right to seek, receive and impart information and ideas of all kinds, and Article 24 states that they have the right to the best possible health and access to good-quality health care. The realization of children's rights requires their translation into policy and practice, as well as children and young people's participation in conceptualizing and realizing these rights (Spronk, 2014). Children's rights have been reflected in increasing interest in children and young people's involvement in research (for example, INVOLVE, 2016;Powell and Smith, 2009;Brady and Graham, 2019), both as sources of data and -the focus of this article -through their active involvement in the research process.
The evidence-based argument centres on the idea that involving those who are the focus of health research has a positive impact on what is researched, how research is conducted, and the impact of research findings on services and in the lives of those involved (Brett et al., 2014;Staley, 2009), as discussed above. Research that actively involves young people, if used to inform decision making or policy formulation, should lead to policies and services that reflect children's priorities and concerns Fleming and Boeck, 2012) and enhance the opportunity for optimal health outcomes (Jamal et al., 2015). But less attention has been paid to the experiences and impact of involving children and young people than to the involvement of adults in research (Brady and Preston, 2017;Bird et al., 2013;Wilson et al., 2015). While evaluation frameworks for involvement have been developed for adults, it is still unclear whether they can be used effectively with children and young people (Parsons et al., 2017).

children and young people's involvement in practice
Involving children and young people is not the same as involving adults, and it requires different considerations. Researchers and others can make assumptions about children and young people's abilities to understand and make useful contributions to research, or that their need for protection from harm overrides their right to choose whether or not to be involved (Brady and Graham, 2019). In our experience, and the growing body of literature cited throughout this paper, it is entirely possible for children and young people to make meaningful contributions to even the most complex health research studies. The challenge is in having skilled facilitators who can provide children and young people with the support and training they need to make an informed contribution, and who can act as 'translators' to ensure that researchers communicate with children and young people in ways that work for them, and that children and young people's contributions are framed in ways that are relevant to the research in question (for example, see the GenerationR groups discussed below). Methods and techniques demonstrated to be effective in adult involvement cannot necessarily be translated to children and young people (Bate et al., 2016). On a practical level, children and young people's availability tends to differ from that of adults (for example, they can only meet during school holidays or at weekends), they may be involved for shorter periods of time (for example, until they go off to further education or employment, or are no longer deemed to be a young person), and they are often dependent on adults to enable them to be involved (see 'Rights, power and control', below).
The dominant model for children and young people's involvement is through the forum of a young persons' advisory group (YPAG), which is also the focus of the project discussed in this article. GenerationR is a national YPAG made up of local YPAGs across the UK. Developed in 2006 by what was formerly known as the NIHR Medicines for Children's Research Network, it is a widely praised model for involving children and young people in the design and development of paediatric health research. Internationally, the European Young Persons' Advocacy Group (eYPAGnet; www.eypagnet.eu) is a virtual consortium of YPAGs that aims to support the development of new YPAGs within Europe, and to provide the necessary infrastructure to support meaningful and valued involvement of children and young people in clinical trial design and health research. The International Children's Advisory Network (iCAN; www.icanresearch.org) is a worldwide consortium of children's advisory groups that works to promote and develop greater global understanding about the importance of the paediatric patient and caregiver voice in health care, clinical trials and research.
But YPAGs are not the only model to involve children and young people, and recent studies have highlighted the need to develop more flexible, children-and young people-centred approaches (Mawn et al., 2015) alongside more 'traditional' YPAGs as described above, which tend to be based in hospitals and/or academic institutions. This can be particularly helpful for researchers seeking to involve children and young people who are disadvantaged, seen as more vulnerable or otherwise less frequently heard (for example, Forsyth et al., 2019;Mitchell et al., 2019).

Rights, power and control
Children's rights are about more than children and young people's right to have a say in matters that affect them, as discussed above. How and when children and young people are involved is largely determined and controlled by adults (Boyden and Ennew, 1997). The extent to which children and young people's involvement is active, authentic and ongoing in research depends both on the approach researchers choose to take, and whether they are able to create an environment in which children and young people feel comfortable to engage in open and honest dialogue on how they can best be involved (Brady and Graham, 2019). The working relationships between adults and children and young people also need to be understood, and reported more fully to explore all participants' perceptions about the process and outcomes of involvement (Dovey-Pearce et al., 2019).

Diversity and inclusion
In considering who is involved, it is also important to remember that children and young people are not a homogeneous group. Age and other aspects of social background, such as race and ethnicity, disability, social class, family background and use of services, 'intersect as aspects of who [young people] are, their social position, and what researchers need to consider in designing research approaches appropriate to the young people they wish to involve' (Clavering and McLaughlin, 2010: 604).
There are disparities in the characteristics of children and young people likely to participate, the types of decisions they are involved in making, and the extent to which this involvement is meaningful and effective (Cockburn, 2005;Percy-Smith, 2010). The mechanisms of formal involvement can privilege the already privileged (Crowley, 2015), and the voices of children and young people deemed to be more 'vulnerable' are often absent from the literature on children and young people's participation in research (Powell and Smith, 2009;Mawn et al., 2015). However, a growing literature (for example, Forsyth et al., 2019;Brady and Lester, 2018) has begun to address this by reporting on attempts to develop more inclusive involvement with 'less frequently heard' children and young people. As discussed above, YPAGs are the dominant model for children and young people's involvement in research, but YPAGs can be convened in children and young people's spaces, such as schools or youth centres, as well as children and young people coming into adult spaces, such as academic and/or clinical settings (for example, Dennehy et al., 2019).

Evidencing involvement
It is important to consider not just why, and how best, to involve children and young people in health research, but also how best to document this, as well as who is involved. The reporting of children and young people's involvement is patchy, inconsistent and often includes informal sharing of knowledge between practitioners in the field, rather than consistent collection of comparable data. NIHR INVOLVE, now part of the NIHR Centre for Engagement and Dissemination, recently undertook a mapping exercise and survey (Sonpal et al., 2019) in order to find out more about how, when and where children and young people are involved in English health and social care research. This project identified a diverse range of active YPAGs, and other involvement of children and young people across England, and explored the benefits and challenges faced by health and social care researchers involving children and young people in their work. But it was a scoping, rather than a comprehensive mapping, exercise, and there is still a need to collate, understand and disseminate more comparable evidence on the nature, extent and impact of children and young people's involvement in research (Brady and Preston, 2017). Knowing more about which children and young people are involved in research would enable researchers and research bodies to ensure that involvement is inclusive and diverse. Having comparable information across studies on when and how children and young people are involved in research, and the impact and outcomes of that involvement, would enable shared learning and the development of best practice (Brady and Preston, 2017). Not having this information risks children and young people's involvement continuing to develop in a piecemeal fashion.
So, in summary, the current body of knowledge leads to several suggestions: • Involving children and young people in research is not the same as involving adults, and requires different considerations, as well as working with third parties (gatekeepers). • There are disparities in the characteristics of children and young people likely to participate, the types of decisions they are involved in making, and the extent to which this involvement is meaningful and effective. • More needs to be done to find out who is involved in research, how and whether different approaches disadvantage marginalized, 'vulnerable' and/or less frequently heard young people.

The background to this study
In September 2013, the GenerationR Alliance of YPAGs was inaugurated at a meeting at the Science Museum, and a report of this meeting, containing 11 recommendations for future developments, was accepted by the Director General of NIHR, Professor Dame Sally Davies and supported by the NIHR-funded James Lind Initiative (GenerationR, 2014). Three recommendations underpin the project discussed below, particularly recommendation 10: • Recommendation 9: Develop an effective communications strategy that showcases involvement activities and sustains the objectives highlighted at GenerationR. • Recommendation 10: Develop a systematic way to measure the impact of involvement activities. • Recommendation 11: Work with the education sector to promote clinical research education in schools.
Understanding and evaluating the impact of public involvement is becoming an increasingly important issue, as discussed above. An evaluation of the GenerationR YPAGs by the National Children's Bureau (Wallace and Eustace, 2014) found that the YPAGs had contributed to a significant number of research proposals, almost half of which informed research at the funding application stage (48 per cent), and 38 per cent after projects had gone through ethical review and were adopted into the NIHR portfolio (Wallace and Eustace, 2014). However, the National Children's Bureau evaluation only focused on studies that had worked with the GenerationR YPAGs, and did not look at studies on the NIHR CRN portfolio, which may have involved children and young people in other ways. The evaluation also identified the need to 'develop clear streamlined and effective monitoring systems to enable the gathering of high-quality data and collation of findings across the network' (Wallace and Eustace, 2014: 30), echoing Recommendation 10 in the GenerationR report.
exploring the feasibility of evaluating children and young people's involvement in NIhR studies In 2017, the authors were commissioned by the James Lind Initiative to explore children and young people's involvement in the design and delivery of research studies adopted by the NIHR Clinical Research Network: Children (CRN Children) (Brady and Preston, 2017). The purpose of this project was to find out what information was available on how and when children and young people were involved in research, and what difference this involvement makes to research, and to the children and young people involved. The project was divided into two stages. Stage 1 involved an assessment of the availability, quality and reliability of data on children and young people's involvement in the CRN Children portfolio. We analysed data available through the NIHR CRN Central Portfolio Management System (CPMS), which is a national information system of study data. Data were extracted from 1 April 2015 to August 2016. We also reviewed data captured from the CRN Closed Study Evaluation Survey, and data collected within individual GenerationR YPAGs.
Stage 2 involved a qualitative exploration through telephone and face-to-face conversations with 12 key stakeholders, including senior public involvement staff within the NIHR, and involvement professionals including GenerationR YPAG coordinators. These semi-structured conversations explored informants' views on monitoring and evaluation systems used to record involvement activities, the feasibility of systematically collecting and collating data on the nature and impact of children and young people's involvement, and the key opportunities and challenges in doing so.

Key findings
In Stage 1, assessing the availability, quality and reliability of information, one of the difficulties we found was accessing the data from the CRN CPMS because only certain personnel within the CRN had permission to access the data. However, once permission was obtained, reports indicated that from the 563 adopted studies held within the CPMS, only 48 studies had completed the Patient and Public Involvement and Engagement (PPIE) activities sections.
All 48 studies were non-commercial studies. More detail was provided in the free-text public involvement section, but only 13 out of the 48 studies completed this section. The 13 completed responses indicated that members of the public were involved in the prioritization of the research and commissioning the research, and were involved throughout the study. Only 1 of the 13 responses expanded on who had been involved, and in that case, it was a parent.
It was therefore not possible to identify through CPMS data a single study in the CRN Children portfolio that had involved children or young people. This does not mean that no children and young people were involved in studies, but rather that it was not possible to determine whether those involved were children or young people rather than adults.
The CRN Closed Study Evaluation Survey captures feedback from study teams at the point of study closure (when a study is no longer recruiting) about their experience of working with the CRN. It included two questions about PPIE in the research process, and researchers' perceptions of the impact of this involvement. Similarly to accessing data within the CPMS, the data can only be accessed by the divisional portfolio team and is not routinely collected. A more detailed statistical analysis of the surveys would require encouraging all researchers to complete the form, as it was also not possible to identify which studies, if any, had involved children and young people rather than adults, or whether or not researchers accessed YPAGs, due to the limited number of surveys completed. A question that addressed 'who' was involved would also need to be added to the survey, as opposed to focusing on the stages of involvement, as discussed in the introduction.
Following the reorganization of the CRN, and changes in sponsorship of GenerationR YPAGs, each group now uses different techniques to monitor, evaluate and report group activities. Often this is done in isolation, fitting around busy workloads and managing different expectations for reporting to those sponsoring individual YPAGs. Most of the information collected focuses on the stages of research (priority setting, design, management, dissemination and so on) in which people had been involved.
In Stage 2, informants reported that past attempts to gather systematic information about, and to map, public involvement activities across the NIHR had been patchy and met with limited success. However, in addition to the opportunities within existing systems already discussed, ResearchFish was mentioned as offering potential opportunities. ResearchFish is an external online system used by the NIHR since 2012, which requires award holders, on an annual basis, to submit data about their research outputs, outcomes and impacts (www.ResearchFish.net/). Questions on public involvement have been asked of all award holders since 2016. These focus on the stages at which there has been public involvement and researchers' perceptions of the benefits of involvement. This is the first time that public involvement questions had been systematically asked of researchers in the regular reporting process. However, it is unclear exactly what data are captured and analysed, but also, more importantly, how these data are going to show the impact on NIHR-funded studies and on those who get involved in the design and delivery of health research. Informants felt that there was scope for further analysis of free-text responses. The addition of demographic questions on who is involved would also be helpful.
The people we spoke to thought that it was important for the NIHR to ask some new questions about who is involved in research and how, as well as what difference this involvement makes. They suggested we could do this by looking at what people say in funding applications and reports about public involvement, as well as by adding new questions about who is involved in research to the reports that researchers have to fill in. People also pointed out that most information collected on young people's involvement is written by researchers, and that there needs to be more opportunities for young people to have a say on how they have been involved, and the difference they think it has made to them and to the research.
In summary, this work suggests that there is currently no way of identifying through existing CRN data collection systems which studies have involved children and young people, the ways in which they have been involved, and what impact any involvement may have had. Furthermore, several questions remain about systems and processes of recording children and young people's activities on studies not on the CRN portfolio, and at different stages of the research cycle.
Several recommendations came out of this research: • Any reviews or development of public involvement questions in the NIHR Integrated Research Application System, ResearchFish, and the NIHR standard application form and monitoring information should consider the need for the collection and collation of information on who is involved in research and how, as well as on impact. • In its work on measuring the impact and developing standards for public involvement following Going the Extra Mile (NIHR, 2015), the NIHR should consider the need to identify when children and young people, as distinct from adults, are involved in research. • Alongside other demographic information, the NIHR should routinely collect information on the ages of those involved as public reviewers, on advisory boards and in other roles, as currently being piloted by the NIHR Central Commissioning Facility. • The GenerationR steering committee and Alliance should consider whether and how it may be possible to capture and collate data on YPAG activity, including reviving the study monitoring form from the National Children's Bureau evaluation (Brady and Preston, 2017).

conclusions
Children and young people have a right to be involved in things that affect them, including research. Involvement appears to have benefits for the children and young people participating, and for research, and it should lead to policies and services that reflect children and young people's priorities and concerns Fleming and Boeck, 2012), and enhance the opportunity for optimal health outcomes (Jamal et al., 2015). But this article has outlined how more needs to be done to provide consistent and comparable evidence of these benefits and impacts, and the realities and challenges of involvement in practice. In our view, it is entirely possible for children and young people to make meaningful contributions to even the most complex health research studies. But in order to better understand this, and to develop practice further, research is also needed on how, when and where children and young people are involved in health research. Many of the issues raised by this exploratory project are relevant to the involvement of adults as well as children and young people. The current climate of limited budgets, and changes to NHS services and commissioning, has implications for both adults' and children and young people's involvement in research, both for the continuation of YPAGs and other projects, and in ensuring that learning is captured or shared. Effective involvement requires critical reflection and shared learning, as well as an understanding of the wider systems and structures that can facilitate or present barriers to participative practice. If we knew more about who is involved in research, and how and whether different approaches disadvantage marginalized, 'vulnerable' and/or less frequently heard children and young people, this could make a useful contribution to wider debates on diversity and inclusion in public involvement, and how to go beyond (but also not exclude) the 'usual suspects'.
How and when children and young people are involved is largely determined and controlled by adults who have the power to put into practice (or not) decisions that children and young people are involved in making. Any work to develop measures of the impact of children and young people's involvement in research should also consider how to include the views of the children and young people involved, and not just those of the adults involved. This includes the production of research reports, articles and other outputs, but also consideration of how metrics and measures of public involvement can include the public voice. Under the UNCRC, children and young people have a right to have a say in matters that affect them (Article 12), but they also have the right to seek, receive and impart information and ideas of all kinds (Article 13). In our view, this includes contributing to the evaluation of, and dissemination of learning from, public involvement.

acknowledgements
The authors would like to thank the James Lind Initiative for commissioning this project, and in particular to acknowledge the support of Sir Iain Chalmers, coordinator of the James Lind Initiative, who identified the need for this project, secured funding and provided encouragement throughout. We also want to thank Sandy Oliver, Professor of Public Policy at the UCL Institute of Education, for her invaluable support as adviser to this project.
Last, but by no means least, we would like to thank all those who provided access to data and support in Stage 1, and the stakeholders who contributed their time, knowledge and considerable expertise in Stage 2.
Notes on the contributors louca-Mai Brady is a senior research fellow at UCL Institute of Ophthalmology -where she supports eyeYPAG, a young people's advisory group for paediatric eye and vision research -and an independent research and public involvement consultant and facilitator. She has particular interests in children and young people's involvement in health services and research, including supporting young people's involvement as young advisers and as co-researchers in health and social care research.
Jennifer Preston is the Senior Patient and Public Involvement and Engagement (PPIE) Manager at the University of Liverpool. Her main role is to deliver a strategy for the involvement and engagement of children and young people and families in paediatric health research, both in the UK and internationally. She leads the GenerationR Alliance of young people's advisory groups, and she is a founding member of the European Young Persons' Advisory Group Network (eYPAGnet).