Going beyond the one-off: How can STEM engagement programmes with young people have real lasting impact?

A major focus in the STEM public engagement sector concerns engaging with young people, typically through schools. The aims of these interventions are often to positively affect students' aspirations towards continuing STEM education and ultimately into STEM-related careers. However, most schools engagement remains in the form of short one-off interventions, which obviously have limited scope for real lasting impact. In this paper we discuss various different programmes of repeated interventions with young people, assessing what impacts can realistically be expected. Short series of interventions, rather than one-offs, appear to also suffer some limitations in the types of impacts achievable. However, deeper programmes that interact with both young people and those that influence them seem to be more effective. We discuss how developing a Theory of Change and considering young people's wider learning ecologies are required in enabling such lasting impacts. Finally, we raise several sector-wide challenges to implementing and evaluating these emerging approaches.


Introduction
As we aim to move towards more impactful engagement and deeper learning, we have to carefully consider and evaluate the different types of engagement that we develop. As the sector has moved from the more didactic 'Public Understanding of Science' towards more egalitarian constructivist approaches, we still see the dominance of traditional interventions such as lectures. We are aware of the limitations of the traditional "listen to me" lecture-style event (e.g. Freeman, 2014;Marbach-Ad, 2000;Marbach-Ad, 2001) and in 2013, The Times Higher Education noted over 700 studies determining that lectures were less effective than other teaching strategies (Gibbs, 2013). As such, there are definite moves in undergraduate teaching away from such traditional lecturing with shifts in: the format of lecturing, such as segmenting and including discussion-based approaches (e.g. Iowa State University, 2020); increased use of active learning strategies (e.g. Durham University, 2019); and even the redesigning of learning spaces or "build pedagogy" (Monahan, 2002;Elkington, 2019). However, learners and teachers will often state a preference for lectures over active learning (Deslauriers, 2019). Students feel that they learn more from lectures, even although the evidence suggests otherwise, perhaps partly caused by the additional cognitive effort required by active learning methods (Deslauriers, 2019).
Against this backdrop, in trying to develop STEM public engagement projects that are more impactful and instil deeper learning we are also faced with the additional resources, costs, relationship building and time required to make programmes happen. Engaging schools and young people with STEM has long been a priority for universities and researchers, perhaps driven by concerns over the perceived low numbers of young people opting for studies in STEM subjects (e.g. Smith, 2004). However, it has been recognised that practitioners do not always adequately consider their engagements' audience and/or methods (e.g. Thorley, 2016). To gather a flavour of current practice, during a session at the 2019 Interact symposium (Archer, 2019) attendees were solicited using an anonymous live online interactive survey about the schools engagement activities they undertake and what the perceived aims of them are. Seventeen university researchers and engagement professionals took part. Figure 1 shows the distribution of types of activity, where the respondents could select from any number of the following: A. Stall/stand: drop-in activities for schools at STEM or careers fairs B. Talk: a typically one/two lesson slot featuring a predominantly one-way interaction C. Workshop: a typically one/two lesson slot with mostly two-way interaction and often hands-on activities for students D. Masterclass/taster: half-day or day-long activities which may be comprised of talks and/or workshops E. Summer school: several-day to week-long activities often involving some project work as well as talks and/or workshops F. Extended programme: multiple interventions with the same group of students over a protracted period of time Not unsurprisingly, talks and workshops scored the highest whereas both summer schools and extended programmes were significantly ( < 0.0019, { } = 0.0033 using the Bonferroni correction) less common than these two options. Bear in mind this was with an audience who are already heavily bought-in to engaging with schools due to their attendance at a day-long symposium on STEM public engagement. The results highlight that the majority of interactions that university staff undertake with school students are one-off interventions that are typically only about an hour long. This point is backed up by reporting across the South East Physics Network (a network of 9 university physics departments with a collaborative outreach and public engagement programme, http://www.sepnet.ac.uk). Figure 2 shows the distribution of engagement durations (i.e. the average time each individual is interacted with, as opposed to the duration of the entire event) for schools events across the network in the academic year 2018-2019, demonstrating a significant peak in probability density between 0-1 hours long with the values dropping off substantially as duration increases.  The attendees of the Interact session were also asked what they hoped to achieve (i.e. the aims or intended impacts) through their school engagements via an open question. Performing a thematic analysis of the qualitative results, it was possible to categorise the majority of answers into the following: • Change school students' aspirations (9 people), with the word "inspire" often used • Enhance students' awareness or understanding of STEM (6 people), often in the context of primary research • Tackle societal biases in STEM (4 people), most often gender Note that some responses covered more than one of these aims. Other stated motivations outside of these themes included "access to a student population for [research] studies", to "build relationships", and to deliver "meaningful content". The three main themes from our coding are in general agreement with those determined by Thorley (2016) when surveying 190 UK physicists as to why they think scientists are being asked to engage more with the public. However, they noted more emphasis on awareness or understanding rather than changing aspirations in their subsequent quantitative results concerning engaging school students, which appears slightly different to those presented here. Nonetheless, it is fair to say that these three themes in general can indeed explain the main aims for scientists (be they researchers or STEM engagement professionals) in working with young people.
One-off experiences, whether a school trip, a show, a speaker, a video or some other interaction, while popular, are limited in the kinds of impacts they might have. There is evidence that they can support content learning, for instance, but this is more likely to come in the form of 'factoids' than deep understanding. Likewise, meeting a scientist can certainly provide students with an increased awareness of what a particular career involves or the range of careers that might be available, and even what courses might be required to progress, but it is unlikely to have a significant effect on aspirations. Research into young people's aspirations (cf. Archer & DeWitt, 2017, among many others) highlights that aspirations are complex and multifaceted and evolve over time. They are influenced by a range of factors, including experiences at home, in the school and in the wider community, as well as background factors that are interwoven with the way they experience school, engagements with science, and everything else. They are also closely linked to identity -to what or whom young people can see themselves becoming, and what type of person they (and others) perceive them to be. Consequently, it should not be surprising that one-off experiences, while 'inspirational', are highly unlikely to significantly impact aspirations and educational trajectories. At the same time, because aspirations develop and are maintained (or not) over time, there can be a role for such experiences in providing additional support in their maintenance. Moreover, when considered from the perspective of learning ecologies, any potential impact of a one-off experience can be extended by linking to other experiences that young people may have, both shorter term and longer in duration. In other words, while one-off experiences are not necessarily problematic in and of themselves, it is important to be realistic about what they can achieve -and to maximise the opportunities they do offer by linking to other aspects of young people's lives and experiences.
Given these limitations of one-off interventions, several organisations and practitioners have been moving towards programmes of repeated-interventions with the same group of young people in order to maximise the likelihood of impacts on their aspirations towards STEM. This paper provides a commentary on some of the different approaches that have emerged, the potential benefits and limitations of them, and what as a sector might be required in order to move forward effectively. These topics arose from discussions at a session on repeated interventions at the 2019 British Interactive Group Event ).

Short series
The next logical step on from a one-off intervention is to instead deliver a short series of (typically a few) interventions. Here we discuss some different examples of such series delivered within the academic year and their evaluation thus far. We note that other programmes working on interventions across multiple years (though with only one session per academic year) are starting to emerge, but we are not aware of sufficient published evidence of the impacts of these types of initiatives yet.
'I'm a Scientist, Get Me Out of Here' (https://imascientist.org.uk/) involves an online chat between students (between ages 9-18) and scientists followed by more extended Q&A over the span of typically 2 weeks. They commissioned an evaluation using the framework of science capital to understand what impact it might be having and what might be contributing to that impact (DeWitt, 2019). Perhaps not surprisingly, one of the biggest impacts was on how young people perceived scientists -as normal, regular people, with hobbies, families and interests outside of science. Such perspectives on scientists are similar to those shared by individuals with higher levels of science social capital -who know people (e.g. family, friends' parents) who work in science. While there was limited evidence that this awareness completely changed aspirations of young people, there was an increased willingness among many to consider the possibility of pursuing science further. Of course, the realisation of any longer-term or significant impact on aspirations is dependent on many experiences that may (or may not) happen after. But this increased openness to the possibility of pursuing science, which seems to be influenced by young people realising that scientists are people 'like me', can have a role to play.
A similar shift in perceptions of scientists was found among primary school pupils participating in 'Scientist of the Week', an intervention developed by the NUSTEM team at Northumbria University. This is a five week, teacher-led intervention, in which a new scientist was 'introduced' to the students each week using presentation slides, classroom posters and postcards to take home. In the materials provided to teachers there was a short paragraph describing the work and life of each scientist which included three key attributes of that person, attributes which both contrasted with stereotypical views of scientists (e.g. curious, openminded, creative) and represented characteristics that young children could imagine themselves possessing and often likely already possessed. In other words, these attributes also communicated that these scientists were 'like me'. Evaluation of this project provided encouraging evidence that young people's perceptions of scientists were shifting -after the intervention pupils were more likely to use non-stereotypical words than stereotypical words when asked to describe a scientist (Shimwell et al., in preparation).
Earlier it was noted that week-long type activities like summer schools are less common than most one-off schools outreach or engagement activities. It is worth noting that these sorts of activities can be thought of to some extent also as similar to a short series or "intensive one-off", since they are still highly focused down into a specific period of time i.e. often a single week. It appears that summer schools, while enjoyable to the participants, only cause moderate changes to the overall makeup of students' likelihood to apply to a selective university in surveys immediately afterwards (e.g. Universify Education, 2018). There is also a critical lack of longitudinal studies at present, as highlighted in a recent review of the higher education widening access sector (Robinson and Salvestrini, 2020). Such studies are necessary in demonstrating whether increased uptake of higher education actually occurs and whether there is causality in summer schools themselves (and not other factors) leading to improved progression. We also note that summer schools often have severely limited places, making them highly competitive, and even with targeting of participants using widening participation criteria, they are likely selecting predominantly those who are already highly bought into the subject of the summer school. Therefore it is fair to characterise summer schools as high cost, small reach, and only moderate impact, particularly when it comes to positively influencing students' STEM aspirations.
Across the South East Physics network two programmes targeting Year 8 (12-13 year-old) students using multiple interventions have been implemented. The first of these is the Connect Physics programme, a series of three workshops spread out across the academic year which address the following aspects: 1. What is physics? The breadth of the subject and how it all connects together 2. Why do physics? The skills and range of careers open to physicists 3. How do we do physics? The process of the scientific method The aim of Connect Physics is to raise (or at least maintain) the science capital of children at this age (Archer et al., 2015). An evaluation of the pilot of the programme in the academic year 2017-2018 was commissioned externally. The final report (Hope-Stone Research, 2018) showed that the workshops were enjoyed by participants but unfortunately the data was not robust enough to demonstrate any of the aspirational impacts that were hoped for. The key issues were that individual students' responses across the year could not be linked, and there was very little crossover between students answering the initial survey and those that answered the final one. Furthermore, responses could not be separated out by demographic informationso it was not possible to explore whether more or less impact was being had on certain groups.
As the programme has become more established, further in depth evaluation is being conducted (in the academic year 2019-2020). This evaluation data will allow linking of individual students' responses, exploration of themes emerging alongside demographic information, and comparisons with control data from Year 8 students at the same schools not taking part in the programme. It is hoped that this will allow a more robust assessment of the impact of this multiple intervention programme.  Jeavans and Jenkins, 2018), unfortunately the programme did not result in any significantly changed attitudes in before vs after surveys, either within gender or in gender variations, other than an increased feeling that physics is difficult by both genders. This latter change, however, is probably a common outcome over the course of Year 8 anyway (cf. DeWitt et al., 2019) and therefore unlikely to be due to Shattering Stereotypes. Following the external evaluators' recommendations, the programme was revised in 2017/18 and a second pilot was launched in 2018/19, going on until 2020. Shattering Stereotypes currently consists of three workshops, aiming to raise awareness of what gender stereotypes are, in the context of a student's everyday life and a student's possible career path. The project also aims to empower students so they can identify and challenge situations where they are presented with these stereotypes. The three workshops shaped around these aims, again similarly spread out across the academic year in their delivery, are: issues around gender stereotyping to various target groups, including teachers, parents and younger students, giving them the opportunity to take ownership over the project.
The interventions use existing resources from the IOP and the Women in Science and Engineering campaign (i.e. the People Like Me quiz). Schools are also offered options for teachers' involvement, to raise awareness of how gender stereotyping might affect their teaching. Parental engagement is also one of the programme's secondary objectives which has proven challenging in the past, so the second pilot aims to identify new ways of including parents in the discussion. A similar approach to the revised Connect Physics evaluation is being taken with the second pilot of Shattering Stereotypes to allow for more robust analysis.
The final example we present here is The Royal Institution's 'Ri Masterclass' programme. These short series of STEM extra-curricular enrichment workshops have been running since 1981. Initially focused on mathematics at secondary level (age 13-14), it has expanded to include both primary and older age-groups as well as engineering and computer science. Teachers select students who they feel should attend. Though initially aimed at the most able, Masterclass organisers now encourage teachers to choose those they feel would benefit most. While only a few places per school are available, teachers are also encouraged to attend and share their experiences back at school. In 2019, 164 Masterclass series ran across the UK for 6276 students, with over 100,000 students having attended since 1981. The aims of the programme (Royal Institution, 2018), which align with the science capital research (Archer et al., 2015), are: -To improve attitudes towards and understanding of these subjects, their applications and their relevance in the wider world -To allow participants to explore topics interactively and in depth outside of what they would see in the classroom over an extended period of engagement -To enable participants to meet a range of subject experts and enthusiasts, showcasing a variety of careers -To enable participants to meet like-minded peers from different schools across the local area Evaluation has revealed some increases in confidence and positive influences on reported levels of interest in the subjects and future subject choices (Royal Institution, 2018& Barmby et.al., 2008. However, these surveys were a small sample of the students involved and it has been difficult to perform meaningful longitudinal evaluation. In addition, due to the franchise model of the programme, the quality of provision can vary across the country. A significant number of students choose to attend follow-up events, however, which is positive (Royal Institution, 2018).
As with one-off interventions, a short series of engagements does appear to have some impact. However, practitioners need to be realistic as to what types of (likely relatively limited) impacts are actually achievable from these programmes.

Deeper programmes
Research suggests that programmes with deeper engagement than simply a short series can lead to considerable increases in students' aspirations as well as other outcomes (Robinson and Salvestrini, 2020). These programmes often combine several different approaches in supporting and interacting with students. Here we detail a few different examples.
The NUSTEM primary partnerships are an ongoing collaboration that works with children, teachers and families to support a broadening of aspirations, with a particular focus on STEM careers. There are currently 33 schools in the partnership, with around 12 having been in partnership for over five years. NUSTEM delivers regular activities in the classroom, but also supports teachers to deliver STEM career focused activities of their own through Continuing Professional Development (CPD) and resource development (e.g. the STEM Person of the Week series mentioned previously) as well as working with parents and carers. Over the first three years of the partnership the gender difference in career aspirations between girls and boys decreased significantly, with more children saying that they would be interested in a broader range of careers .
Working in partnership with several organisations, Durham University has developed a programme supporting young people in becoming Science Ambassadors for their own communities (https://www.dur.ac.uk/science.outreach/national/ambassadors/), which is adapted depending upon the needs and interests of the young people. Throughout the year-long programmes, teams of girls from primary schools work together with community, university and business partners developing their own plans to work with and inspire their peers and community. This has proved successful, even in some of the most deprived areas in England. Ambassadors grow particularly in confidence as well as slightly gaining more enjoyment from presenting. While no statistically significant rises in the numbers stating that they could be a scientist in the future have occurred, schools do report increased parental engagement in Science Ambassador run activities. More work needs to be done on analysing the longer-term impact on the young people and their communities though.
Urban Advantage is a formal-informal partnership between eight informal science learning institutions in New York City and the NYC Department of Education. It works with middle school students (ages 11-14) and focuses on hands-on science inquiry, supported via CPD for teachers and classroom resources, as well as visits to informal science institutions and family workshops. Recent evaluations (e.g. Weinstein et al., 2014) found that participation in Urban Advantage was associated with higher attainment in science (especially among disadvantaged pupils) and increased student confidence in science (aspirations were not measured).
The Science Capital Teaching Approach (Godec et al., 2017) was developed out of the Enterprising Science project in the UK. It takes a social justice approach to science teaching and is based on a foundation of broadening what counts (whose voices and experiences matter and have a place) in the science classroom. This approach was co-developed with 43 secondary school science teachers over four years. It is a CPD model, involving training teachers in the approach via Saturday sessions and ongoing classroom support during the year. Importantly, it was implemented in science classrooms, meaning that students experienced the approach in an ongoing fashion, over the course of a school year. Research on the approach found a statistically significant increase in the science capital score of young people whose teachers had participated in the project. It also found an increase in aspirations to study science post-16 and improved attitudes to science. Teachers also reported improved behaviour and attainment (Archer et al., 2018).
Finally, a number of protracted programmes for secondary and sixth form students to experience undertaking cutting-edge scientific research have emerged in recent years. These have the potential to address the disconnect between science education and what professional scientists actually do (Braund and Reiss, 2006;Yeoman et al., 2017), which may have a role to play in affecting students' aspirations towards STEM subjects. Unfortunately, such programmes are rare globally, with those that do exist often lacking active support from researchers and/or targeting of disadvantaged groups (Bennett et al., 2018;Archer, in preparation a). One example which avoids both of these issues is the PRiSE (Physics Research in School Environments, http://www.qmul.ac.uk/spa/researchinschools) programme at Queen Mary -a scalable framework for 6-month-long independent research projects based on current physics research topics. PRiSE is supported throughout by active researchers and has equitably involved diverse groups of 14-18 year-old London students (Archer, in preparation a). While in some regards this may be seen as similar to the short series, since each project only has a limited number of interventions between researchers and students, the key difference is that there is significant activity between these interventions where students and their teachers work on these research projects. Therefore, the interventions can be thought of as more of a support mechanism for a wider programme, empowering the students to complete their investigations and present them at dedicated student conferences to other PRiSE students, teachers, peers, family, and friends. Longitudinal evaluation (Archer, in preparation b) at the 6-month and 3-year stages has shown lasting effects on students' STEM-related confidence and skills. Furthermore, there is evidence suggesting participating students are more likely to continue with STEM as a direct result, e.g. preliminary longitudinal evidence 3 years later has revealed greater uptake (to a statistically significant level despite the small sample size) from PRiSE students of both physics and STEM subjects at university than would be expected of physics students nationally. Beyond simply affecting students, PRiSE has been developing teachers' professional practice and even affecting the profile of STEM across their schools. This is further reinforced by the significant repeated buy-in from schools.
All these programmes share a number of characteristics that likely contribute to their impacts. They are long term -over the course of a school year or more. They involve frequent and repeated activities which students encounter as part of their ordinary learning experience. They both work with and support key influencers in the students' lives, e.g. teachers are leading regular activities. The activities of the interventions are closely integrated with the curriculum in terms of subject content and skills development. They also link together different facets of learners' experience, both inside and outside of the classroom. In understanding, however, how these different aspects contribute to impact upon students, we need to consider the wider picture rather than simply the programmes themselves.

The big picture
While it is clear that long-term interventions are needed to even begin to influence young people's aspirations and identification with science (or anything else, for that matter), such efforts are beyond the scope of many organisations. They are also extremely difficult to develop, deliver and maintain if the delivery is outside of school, simply because young people do not have to be there. Thus, it becomes important to consider how to maximise potential impacts of the one-offs, short series or even deeper programmes of engagement. This is where the notion of learning ecosystems or ecologies can be helpful (e.g. Brofenbrenner, 1979). A learning ecology is the context (physical, social, cultural) in which learning takes place. Ecological perspectives on learning acknowledge that young people (and adults, for that matter) learn across a range of contexts, as well as over time, and all of this is influenced in a complex way by previous experiences, background factors, and what follows any given experience. An example diagram of a learning ecology model is shown in Figure 3, where the individual lies at the centre and the layers surrounding have decreasing amounts of direct interaction/influence with them. Such models serve to remind us that our interventions -whether one-offs, short series, or longer -do not operate in isolation. Moreover, they can be more powerful as learning experiences by linking to other elements in the learning ecosystem. That is, by making a conscious and concerted effort to form links among organisations, young people may be guided towards other experiences that reinforce what they have learned in our activities (Bevan, 2016;Traphagen and Traill, 2014). Doing so can also help us refine our efforts -define our niche -so that it better fits what is needed. In the United States, such initiatives are gaining momentumand, indeed, a STEM Funders Network has formed to support such efforts in a concerted way (https://stemecosystems.org/about-the-stem-funders-network/). Figure 3: The learning ecology model of Brofenbrenner (1979). From Rhodes (2013).
It is perhaps tempting for organisations that want to effect behaviour or aspiration change to develop an intervention that is based on 'common-sense', but without considering the learning ecology that the child or young person is situated within, without identifying a realistic way in which the desired change could be achieved by the intervention, and with no research base underpinning the intervention. One way to overcome this temptation is to use a Theory of Change approach. This was initially developed to evaluate complex initiatives (Sullivan and Stewart, 2006) and support long-term behaviour change. A key benefit of using a Theory of Change is that it starts with the end goal, and then requires the identification of intermediate outcomes through a process of backwards mapping that will, over time, lead to that goal. In this way causal chains can be identified that link an intervention with (evidence-based) steps that should eventually lead to the desired change . This process is an iterative one, and allows the production of a Theory of Change diagram, such as that shown in Figure 4, which depicts the changes involved in achieving the goal of an intervention. Challenges to the sector The recent review of Robinson and Salvestrini (2020) concerning programmes of engagement with young people advocated for further research to disentangle their individual components in order to better understand the most impactful elements. However, this is perhaps a too reductionistic approach to developing engagement programmes, given the context of learning ecologies and Theories of Change for impacting on young people's STEM aspirations. Indeed, we have presented several examples demonstrating that the aims of our engagements require considerable effort beyond that which is typically being undertaken at present, and even then they remain complicated and difficult to realise. Therefore, there are many challenges for the sector even with the emergence of these deeper programmes of engagement.
A common theme in many of the repeat interventions programmes discussed is the difficulty in evaluating what impacts, particularly the higher-level aspirations-type ones, the interventions are having. Often changes may only be subtle and without the right types of quantitative and/or qualitative analysis, it can appear as if no impact is being had. The Theory of Change shown in Figure 4 identifies three groups of stakeholders that are key to increasing the number of young people choosing a STEM career: families, teachers (schools) as well as the children and young people themselves. However, it is noticeable that there is another set of stakeholder groups that do not appear on the diagram: companies, industrial sector bodies, and other STEM organisations. Realistically, there is another Theory of Change that should be developed to explore the organisational changes that need to take place within companies to make them appealing places to work, and address the retention issues that mean a (not insignificant) proportion of young people who enter the STEM workforce do not remain there (Smith & White 2019). The STEM engagement sector (including university outreach organisations) can be seen as 'working' indirectly for these additional stakeholders and doing some of their pre-recruitment work for them. This could lead to a certain moral ambiguity if young people are being encouraged to enter sectors that are (still) not always welcoming to them.
Another issue that can be identified from the Theory of Change in Figure 4 is that the learning ecology is extremely challenging to affect. Many of the interventions in STEM engagement continue to focus on the young person, particularly around the points of 'choice' between GCSEs and A-levels. Given the relatively short-term nature of industrial and academic planning where five years is considered long term this is not surprising. There has been some shift in age downwards with recommendations for organisations to work in primary schools -albeit at the upper end of the age range (HEFCE and OFFA 2013). However, while a few interventions include teachers, very few include parents or carers. In many respects this is understandable, parents of teenagers are 'hard to reach', particularly as young people may actively work to avoid some forms of involvement from their parents in their education (Deslandes and Cloutier 2002). As young people progress through the education system it becomes more likely that parents and carers will work during the day, and have limited capacity to attend evening events. There has not been, as yet, concerted effort to identify how best to engage and include parents in the STEM engagement learning ecology. It is clear that different aims, messages, and activities are needed as students progress through their education. Figure 5 shows a potential way of characterising these throughout their educational journey. However, this raises questions as to who is best placed to be delivering these evolving messages. For example, it is arguable that a university academic is perhaps not the most effective person in discussing (or being a suitable role model for) the wealth of careers beyond academia that continued STEM education can provide. Similarly, many science communicators and engagement professionals do not have the depth of research knowledge or expertise that might be able to sustain a young person's interest through cutting-edge STEM. This highlights that we need to work more collaboratively, forming partnerships across all those who engage young people, in order to identify who is best placed to deliver the very different types of engagement necessitated at the various points along a young person's educational journey. Such wider collaborative working could, hopefully, shift away the focus from solely the impacts that individual programmes/organisations are having and rather promote thinking about how (via Theories of Change and improved evaluations) we are collectively having positive impacts upon young people's learning ecologies.