Measuring the effectiveness of knowledge creation as a means of facilitating evidence-informed practice in early years settings in one London borough

This paper examines our use of knowledge-creation activity as a way of developing evidence-informed practice among a learning community of 36 early years practitioners in one London borough. It also seeks to illustrate how we approached the idea of measuring evidence use and our engagement with, and adapted use of, two separate measurement scales: the ‘ladder of research use’ and Hall and Hord’s (2015) Levels of Use scale. In doing so we examine the ‘trustworthiness’ of our approaches to measuring evidence use, which we explored via in-depth semi-structured interviews. Our findings would appear to be encouraging, suggesting that knowledge-creation activity provides an effective way of communicating research and keeping it top of mind; also that our interview data would appear to support the trustworthiness of our measurement scales as a means to ascertain levels of evidence use. At the same time the approach we have developed does have its limitations, namely that it is only really applicable to situations where researchers are working regularly with practitioners on areas of practice development, where the general desire is that these areas should become evidence-informed. However, we suggest that, in school systems such as that in England – where the expectation is that schools or alliances of schools should lead their own professional development activity, often in partnership with universities – it is likely that these instances will soon be increasing in number. Facts, characteristics, and rules are defined so clearly and objectively … that they can be recognised without reference to the concrete situations in which they occur. On the contrary, the rules can be generalised to all similar situations, which the novice might conceivably confront. At the novice level, facts, characteristics, and rules are not dependent on context: they are context independent.


Objectives
This paper has three key aims. First, it examines our attempts to use knowledge-creation activity as a way of developing evidence-informed practice among a learning community of 36 early years practitioners in one London borough. Second, situating the idea of effective evidence use within Flyvbjerg's notion of expertise, it seeks to illustrate how we approached the idea of measuring evidence use; specifically, how we sought to ascertain whether early years practitioners, having been continually engaged in knowledge-creation activity across a school year, were developing expertise as evidence users. As part of this second aim, we examine our engagement with and adapted use of two separate measurement scales: the 'ladder of research use' and Hall and Hord's (2015) Levels of Use scale. Finally, we explore how we sought to examine the 'trustworthiness' of our approaches to measuring evidence use, via the use of in-depth semi-structured interviews. We conclude by assessing our approach, examining both its strengths and limitations, and also highlighting other contexts and situations in which it might be used.

Evidence-informed practice
We define evidence as including the following: the use of formal research, practitioner enquiry such as action research, and the interpretation of routinely collected data used to inform any action taken. More specifically, we take the notions of formal research and practitioner enquiry to comprise the collection of data that has been gathered via a process of research, that has been interpreted, and that subsequently has or could be used to address a particular issue facing policy-makers or practitioners. Routinely collected data would include that already captured by systems within schools and school systems, such as pupil assessment data (Brown, 2013). Using evidence to aid practice is not without controversy or debate (e.g. see Hargreaves, 1996;Hillage et al., 1998;Tooley and Darby, 1998;Biesta, 2007;Nutley et al., 2007;Cooper et al., 2009;Brown, 2013). Notwithstanding these issues, this paper is grounded in the idea that engaging with evidence is socially beneficial: that using evidence can develop our understanding of the world via an exposure to ideas and concepts we might not ordinarily come across and that, in turn, this exposure potentially enhances the repertoire of 'understanding' upon which we can draw as we develop solutions to the issues of practice.
In itself, being evidence-informed is a somewhat nebulous notion and it is difficult to conceptualize what evidence-informed practice might 'look like'. It is unlikely, for example, that evidence will ever inform practice in a 'pure' form; that acts of practice will be instigated and will follow exactly what was intended by the researcher. Instead, evidence will conjoin with contextual and other practice-related factors and any decision made will be a function of all of these (Virtanen and Tynjälä, 2008;Rexvid et al., 2012;März and Kelchtermans, 2013). This is reflected by England's Department for Education (DfE), which deploys the term 'evidence-based practice' (EBP) and suggests that EBP represents a combination of practitioner expertise and their knowledge of the best external research, and evaluation-based evidence.
The DfE's notion of EBP, however, highlights three key issues in relation to the concept: • a need to conceive of how such combinations of formal and practitioner-held knowledge might be meaningfully realized (i.e. from a theoretical perspective, what represents successful combination of knowledge types) • to think of approaches that will enable us to combine formal and practitioner-held knowledge in accordance with this theory, so that a state of being 'evidence-informed' is achieved • to devise ways of measuring whether we have reached our destination -that is, based on our theoretical perspectives and practical approaches, how might we ascertain whether practitioners are now informed by evidence?
We now use the remainder of the paper to discuss these issues in more detail before examining our attempts to tackle them as part of the Early Years Learning Community project.

Expertise
We suggest that an effective way to conceptualize the combination of knowledge types described above is by considering notions of 'expertise' in evidence use, a position that has its basis in the work of Bent Flyvbjerg. Flyvbjerg (2001)  A novice, for example, is new to particular situations and will, during instruction, learn about facts that correspond to the situation and its characteristics, so is taught or develops 'rules for action '. Flyvbjerg (2001: 11) suggests that, for the novice: Facts, characteristics, and rules are defined so clearly and objectively … that they can be recognised without reference to the concrete situations in which they occur. On the contrary, the rules can be generalised to all similar situations, which the novice might conceivably confront. At the novice level, facts, characteristics, and rules are not dependent on context: they are context independent.
Flyvbjerg argues that, as learners advance from 'novice' and through the levels of 'advanced beginner', 'competent performer', and 'proficient performer', a number of things occur to facilitate the normalization of more instinctual/intuitive behaviour. First, instances of performing in real-life situations increase so that, correspondingly, the number of 'cases' the learner encounters and tackles also increases. Second, recognition of different situations accumulates, as does recognition of the context in which those situations occur. Third, dependency on specific 'rules for action' diminishes as learners are able to interpret and judge how to perform optimally in any given situation. Genuine expertise, however, only occurs as the result of a 'quantum leap' in behaviour and perception, from that of an analytical problem-solver to someone who ' [exhibits] thinking and behaviour that is rapid, intuitive, holistic, interpretive … [expertise that] has no immediate similarity to the slow, analytical reasoning which characterizes rational problem-solving and the first three levels of the learning process' (Flyvbjerg, 2001: 14). In other words, true expertise represents a situation in which experience and formal knowledge are intuitively and holistically combined in order that a problem might be immediately assessed and a solution provided almost without conscious reasoning.
In terms of practitioner evidence use, it is clear that a spectrum of potential evidence-use types is likely to exist. This ranges from novice use -where practitioners slavishly follow 'what works'-type evidence, mandates, or solutions produced on the back of this evidence -to expert use. At this advanced level, practitioners are social actors who intuitively develop responses to situations using an intuitive, holistic reading based on a combination of the formal research knowledge they have adopted to date, an understanding of the specific case they are dealing with, and their understanding of the other environmental factors that might influence their decision. We would argue that the most effective types of use stem from evidence being employed in this expert way. It is most likely to avoid the difficulties reported by a number of professions in marrying centrally prescribed, 'evidence-based' solutions with local context -for example, Rexvid et al. (2012) examine doctors' and social workers' reactions to initiatives to implement 'evidence-based' guidance. It is also most likely to avoid any substantial costs in terms of the re-contextualization required with such marrying (see Cartwright, 2013, who argues that such costs relate to the support factors required to implement solutions in a given setting). By this we do not mean that the 'best available' evidence is slavishly followed. Rather, we suggest it is applied alongside practitioner understanding/data/enquiry in order to frame or define an issue, provide an understanding for how it might be addressed, and/or help ensure that data might be analysed most effectively. A fundamental difference, then, between top-down evidence-informed solutions (see Moss, 2013, for an excellent example) and practitioners' expert evidence use is that, in the latter, the practitioner engages with the formal knowledge that best suits their needs, circumstances, or priorities. In other words, formal research is filtered through a 'contextual window' which determines what research might be considered (for instance, with regards to the nature of the issue and acceptable ways of tackling it).

Professional learning communities
Having established a theory that accounts for how knowledge might be meaningfully combined, we now consider approaches that enable formal and practitioner-held knowledge to be combined in ways consistent with Flyvbjerg's theory of expertise. Here we turn to the notion of professional learning communities. While there is no universal definition of a professional learning community, they are usually depicted as a context in which people involved with, and concerned about, schooling work collaboratively to learn about how they can improve pupil learning or outcomes. For example, such communities have been described by Stoll (2008: 107) as a means through which to build 'learning [in order] to support educational improvement', and which comprise 'inclusive, reflective, mutually supportive and collaborative groups of people who find ways, inside and outside their immediate community to investigate and learn more about their practice'. The notion of professional learning communities thus encapsulates instances where practitioners and researchers might collaborate to facilitate learning about and from formalized/academic knowledge.
A key benefit of the professional learning community may be attributed to the nature of the learning that takes place within it. This is encapsulated by the process of knowledge 'creation', described by Stoll (2008) as one where the producers and users of formal knowledge -who are, respectively, also the users and holders of 'practical' knowledge -come together to create 'new' knowledge (Stoll, however, uses the term 'animation' to describe the collaborative process). Nonaka and Takeuchi (1995) conceptualize this process of creation as one arising from the interactions between tacit and explicit (or informal and formal) knowledge. In particular it is a 'spiralling' that accrues from four sequential types of knowledge conversation, culminating in practitioners 'internalizing' newly created knowledge and intuitively drawing upon it as part of the day-to-day process of developing practice-related solutions. In other words, it results in a state of expertise.
Activities to facilitate knowledge-creation activity are considered in detail in Stoll (2009), and we outline our approach in a general sense further below.

How might we measure evidence use?
A third issue relates to how we might know when any state of being 'evidence-informed' has been achieved. Researchers' ability to measure the effectiveness of methods aimed at facilitating evidence-informed practice has been problematized by Levin (2013: 6-7), who argues that: even if one recognizes multiple forms of research influence, it is very difficult to identify when these have taken place or how much effect they have had … Most of the research assessing research impact relies on one asking people what they did and why, even though it is known that self-reports, whether of belief or behaviour are not reliable. and that: Practitioners themselves cannot readily disentangle the role of particular influences on complex activities such as professional practice or policy choices.
In other words, measures of research use focus on discrete instrumental uses of research (Landry et al., 2003). We argue that the issues raised by Levin are further compounded when incorporating the need to conceive of evidence use as something intrinsically related to expertise. This is because, as will be illustrated below, the levels used to describe expertise seem fiendishly difficult to translate into something that can be surveyed. This presents a tricky problem in relation to our above description of the notion of expertise and its relationship to being evidence-informed: How might we measure evidence use in a way that helps us ascertain whether individuals are learning/developing as expert evidence users?
We now describe our approach to this issue as part of the Early Years Learning Community project conducted in one London borough.

The Early Years Learning Community project
In England, early years provision has undergone significant and unprecedented investment and intervention from governments in recent years. This is in recognition of the critical part that early education can play in improving the life chances of children, particularly those from disadvantaged groups. However, provision for children under 5 is offered within a range of highly diverse settings including nursery classes, private and voluntary nurseries, children's centres and primary schools. Historically, the fragmented and patchy nature of this provision has created difficulties and divisions for children, their families, and practitioners alike. In order to overcome some of these divisions, the Early Years Foundation Stage (EYFS) was established in 2008 and revised in 2012 (DfE, 2014). The EYFS was devised with the following aims, to: • set the standards for children's learning, development, and care from birth to statutory school age at 5 • improve quality and consistency in the early years sector • lay a secure foundation for future learning through individual learning and development planned around the individual needs and interests of the child • provide equality of opportunity • create the framework for partnership working (DfE, 2011).
In England, qualifications of staff working with young children are highly variable and match the diversity of provision in the sector. Latest figures suggest that 73 per cent of staff working in childcare settings are qualified to Level 3, the equivalent of an A level in England. A much lower number, 7 per cent, are qualified to Level 6, or graduate level. In nursery and reception classes within primary schools the figure for graduate-level qualification is higher, with around 40 per cent having qualified teacher status (DfE, 2011). Considerable progress has been made to improve the qualifications of those working with young children. A further challenge, however, is that of ensuring continuing professional development during a period of significant change in the policy landscape and in curricular and pedagogical requirements. It is now widely accepted that good pedagogical leadership makes a difference to the quality of early years provision and in particular to creating an 'enabling environment' for learning as defined by the EYFS (see, for example, Siraj and Hallet, 2014). An enabling environment is one that supports children's play and exploration, and their critical and creative thinking. Rogers (2013) has argued that the concept of an 'enabling environment' does not make sufficiently explicit the importance of relationships and interaction in teaching and learning. Hence she proposes that pedagogical leadership requires practitioners to think in terms of an 'enabling pedagogy', based on the available research evidence of the interactive strategies that best meet the complex learning needs of children in the early years. The project reported here draws upon that idea in developing a model for professional development.
Against this background, the project responded to a specific identified need to improve the skills and knowledge of a diverse group of practitioners in relation to the requirements of the EYFS. In particular the participating local authority and a number of heads of early years settings in the borough argued that support was required with regards to the newly introduced 'Characteristics of effective learning', which provide the context for learning in the EYFS (DfE, 2012). These are: • playing and exploring -children investigate and experience things, and 'have a go' • active learning -children concentrate and keep trying if they encounter difficulties • creating and thinking critically -children have and develop their own ideas, make links between ideas, and develop strategies for doing things.
The main objectives of the community therefore were to improve outcomes for children and adults by: • increasing practitioners' understanding of and confidence in using the newly introduced 'Characteristics of effective learning' • enhancing early years practitioners' teaching skills, particularly in promoting children's 'Creating and thinking critically': children's ability to 'have their own ideas', 'make links', and 'choose ways to do things' -this focus was informed by research evidence indicating that this characteristic is linked to self-regulation, creativity, and motivation (Whitebread et al., 2009) • developing a sustainable model of practitioner development and partnership that will support children's learning, particularly those in disadvantaged groups.
In order to realize these objectives a professional learning community was established comprising 36 practitioners from 18 early years settings whose headteachers and setting managers were keen to support the opportunity for developing the skills of their practitioners in this way. The practitioners worked in either children's centres catering for children from birth to 4, or primary school reception classes working with children aged 4 to 5. Level of qualification ranged from Level 2 nursery assistants to qualified teachers. The aim of the learning community was to assist participating practitioners (two from each setting) in developing a wider repertoire of interactive strategies which better support children's creative and critical thinking. It was envisaged that such strategies would be formed from a combination of formal (academic) knowledge and praxisbased best practice, and their enactment would be observed, critiqued, and improved upon throughout the year via facilitated lesson study -a process involving groups of practitioners in collaborative planning, teaching, observing, and analysis of the learning and teaching in research lessons (Lesson Study UK, 2014). The efficacy of these strategies in meeting their intended outcomes for practitioners and children was then established via an evaluation process where baseline data was collected at the beginning of the project, intermediary data was collected at monthly intervals, and endline (outcome) data was collected at the project's 12-month conclusion. At the same time the authors sought to use the evaluation process to test, first, the learning-community approach as a means of facilitating evidence-informed practice and, second (vitally), as a way of testing potential ways of measuring the success of our approach. In other words, to ascertain whether the knowledge-creation approach had facilitated the development of expertise in terms of evidence use.

Approach to knowledge creation
A knowledge-creation workshop was held during the inaugural professional learning community meeting. Following Nonaka and Takeuchi (1995) and Stoll (2009), one of the researchers -an early years expert -facilitated a discussion that centred on current academic knowledge in relation to effective early years practice, covering the age phase from birth to starting school. This process had, in itself, been constructed via a detailed literature review, specifically undertaken for the project (see Rogers and Brown, 2014). The first session included current research findings on early attachment, attunement, and reciprocal play, and on the development of theory of mind, metacognition, and emergence of pretend play and language. A second session considered this research in relation to pedagogy in early years settings and schools. It included, for example, the latest research on sustained shared thinking, adult strategies for supporting child-led play, and developing an enabling pedagogy (Rogers, 2013). The practitioners were then invited to share their own practical knowledge, for example, data and insight about their settings and current practices, via activities designed to surface this knowledge (see Rogers and Brown, 2014). They were then invited to establish what they wanted to achieve by the end of the project and how they might do so. Specifically, following an approach set out by Earley and Porritt (2014) and starting with the end in mind (the goal they wished to achieve), practitioners were asked how they might relate the presented academic knowledge to their own practical knowledge and that provided by others in their study group, in order to develop strategies towards reaching this end-point.
Following this initial workshop, practitioners from the 18 partner settings were paired into six groups of three from a mix of children's centres and reception classes. This was a deliberate strategy, which allowed all practitioners on the project to observe pedagogy and learning across the entire early years age range and to strengthen networks between different early years provision in a sector that is widely recognized to be fragmented. Each setting hosted one visit per term for their study group.
During each visit, these mixed study groups (and facilitator) spent the morning in the setting observing a specific activity planned by the host practitioner. These activities were designed to be part of normal pedagogical practice, to ensure the least disruption to the children but at the same time to allow the practitioners to explore new strategies stemming from the research. Before each activity, the host practitioner briefed observing practitioners on what they wanted to achieve and on desired outcomes. The group then reflected on the 'success' of the activity, offering suggestions for how it might be improved and planning the next lesson study. In order to ensure research evidence remained 'top of mind', the facilitators were asked specifically to ensure that, as well as improvements grounded in practitioner knowledge, they prompted for potential improvements suggested by the research.
At the end of each lesson study day, all 36 participating practitioners attended an end-ofday seminar to ensure that learning was shared and reflected upon across the group. Where pertinent, the researchers would relate practitioner experiences shared during the feedback back session to the research initially presented.

Methods
The overall purpose of the methodological approach described in this paper (as distinct from those approaches employed as part of the overall evaluation) was to ascertain how effective the knowledge-creation process was in instilling an evidence-informed ethos within a specific group of practitioners. To do so required us to establish ways of measuring impact in ways that addressed earlier critiques of this approach, as noted above. Specifically, as well as addressing Levin's points, we wished to measure the extent to which practitioners had become expert in using evidence. As can be seen from Figure 1, however (see page 254), the descriptors of each of Flyvbjerg's levels of expertise seemed tricky to operationalize; it was difficult to turn these into scales that would be meaningful to respondents, which tackled the specific situation we were asking them to consider, and would enable them to employ evidence as part of developing strategies to improve their practice. Ultimately, therefore, this quest led us to approach the issue of measurement in two distinctive ways, with a subsequent phase of in-depth semi-structured interviews and observation work, to provide additional insight and data regarding the validity and reliability of our favoured approach.

The ladder of research use
Our first approach saw us draw upon a study successfully undertaken by Landry et al. (2003), who surveyed government officials from Canadian and provincial public administrations in order to examine the extent to which they employed academic research as part of the policy process. This study based its measurement of research use on an adapted version of the Knott and Wildavsky (1980) 'ladder of research' use (see Landry et al., 2003: 194); one recognized for its reliability (see, for example, Cherney et al., 2012). Correspondingly, Landry et al.'s study is one grounded in the assumption that evidence use must be examined in terms of cumulative levels, or stages, of both cognitive process and action. In other words, evidence use does not just happen, it occurs as a process comprising many stages via a movement from its 'reception' ('I received the university research pertinent to my work') to 'influence' ('university research results influenced [my] decisions'), via the stages of 'cognition', 'discussion', 'reference', and 'effort' ('I made efforts to favour the use of university research results'). This notion that use occurs gradually over time and is dependent on its being meaningfully 'received' or engaged with led us to feel that the scale would provide a good basis from which to ascertain how well research was initially communicated in the knowledge-creation workshop, and the extent to which users were inclined to employ the research findings.
Our scale was amended, however, to reflect the situation in which it was to be applied: that we were studying practitioners not policy makers, and asking practitioners to consider the research just presented which they had been asked to employ in the development of their strategies in their study groups. This led us to employing the following descriptor terms: 1. Reception -the research was well communicated 2. Cognition -I understood the findings of the research 3. Discussion -I discussed with others within my study group how the research might be used 4. Reference -I could relate the research findings to my setting 5. Effort -I used the research in subsequent exercises (when thinking about the approaches I might use in my setting) 6. Influence -I intend to apply/have applied the research as part of my approach.
In order to collect data against our scale, a self-completion survey was employed at the end of the knowledge-creation workshop, with the researchers in situ as the survey was completed so that we might address any queries regarding interpretation. Specifically, respondents were asked to consider all six elements and indicate the extent to which they agreed using five-point Likert items ranging from 'strongly agree' to 'strongly disagree'. The five-point scale provided subtlety of understanding since it is grounded in the notion of knowledge-use being a process containing multiple stages, rather than an event that can be measured in a yes/no binary way.
Respondents were also asked to respond to additional open-response questions, so that we could ascertain what other factors had influenced their choice of strategies. These questions were: • 'Why do you want to use the research?' • 'What else was being discussed by the study group?' • 'Did you also incorporate this into your strategies? If yes, how?' • 'Was there anything else that influenced your choice of strategies? If yes, please state what.' Our initial survey thus recorded practitioners' early thoughts about the strategies they hoped to create, and enabled us to ascertain the extent to which these had been influenced by the research that had been shared and engaged with, along with other potential influences and the relative strength of such influences. In total, 33 of 36 participants completed the survey.

Levels of Use scale
While our scale appeared to be useful for assessing the effectiveness of the knowledge-creation activity as a means of communicating research and as a process of stimulating interest and the desire to use evidence, it failed in terms of providing an ongoing measure. In particular, it did not enable us to determine whether practitioners' level of expertise in using evidence increased over time. In other words, it did not allow us to understand the effectiveness of the evidence combination we had produced. We could not judge, for example, whether practitioners were being guided slavishly by the evidence or had combined it in a more intuitive, holistic way with their wider practical experience so that their overall ability to understand or tackle particular situations had been enhanced. It also failed to provide a way of assessing changes in expertise over time so that we could not judge, for example, whether practitioners had begun as novices and gradually improve their expertise. As a result, our second approach to measuring evidence use involved attempts to find and, if required, amend a scale that could be more closely related to Flyvbjerg's levels of expertise. Following Cherney et al. (2012) we sought out scales with an ability to identify how utilization is related to various ongoing, complex, and indeed socially situated decision-making processes. Our attempts culminated with a decision to use the Levels of Use scale developed by Hall and Hord (2015) as part of their wider work in developing the Concerns-Based Adoption Model (CBAM), a tool developed in the late 1970s in order to explore people's reaction to change and, in particular, to change caused by the introduction of new innovations. The Levels of Use scale represents eight classifications relating to how people act or behave in response to a newly introduced change or innovation. Specifically, it reflects the observation that, just because it has been suggested or mandated that an innovation should be used, it doesn't mean that this will necessarily be the case, or that all individuals will employ the innovation in the same way; some users will 'stumble along' and others may use the innovation to achieve ground-breaking ends (Loucks et al., 1975). This idea -that there are different typologies of use occurring as a function of users' ability to incorporate a new innovation into their practice -led us to believe that the Levels of Use scale could provide a way of understanding how notions of expertise might be expressed. In addition, it enabled us to develop a scale that draws on the essential features of Hall and Hord's existing and extensive learning: for example, their insight that an eight-point scale is more likely to capture the subtlety of the different ways in which innovations are used (Loucks et al., 1975).  Flyvbjerg (2003) and Hall and Hord (2015) Flyvbjerg: Expertise that accrues through learning Non-use Novice i. 'rigid adherence to taught rules or plans' ii. no exercise of 'discretionary judgment' Advanced beginner i. limited 'situational perception' ii. all aspects of work treated separately with equal importance Competent i. 'coping with crowdedness' (multiple activities, accumulation of information) ii. some perception of actions in relation to goals iii. deliberate planning iv. formulates routines Proficient i. holistic view of situation ii. prioritizes importance of aspects iii. 'perceives deviations from the normal pattern' iv. employs maxims for guidance, with meanings that adapt to the situation at hand Expert i. transcends reliance on rules, guidelines, and maxims ii. 'intuitive grasp of situations based on deep, tacit understanding' iii. has 'vision of what is possible' iv. uses 'analytical approaches' in new situations or in case of problems

Non-use
The user has little or no knowledge of the innovation, no involvement with the innovation, and is doing nothing towards becoming involved

Orientation
The user has recently acquired or is acquiring information about the innovation and/or has recently explored or is exploring its value orientation and its demands upon user and user system.

Preparation:
The user is preparing for first use of the innovation Mechanical use: The user focuses most effort on the short-term, day-to-day use of the innovation with little time for reflection. Changes in use are made more to meet user needs than client needs. The user is primarily engaged in a stepwise attempt to master the tasks required to use the innovation, often resulting in disjointed or superficial use.

Routine:
Use of the innovation is stabilized. Few if any changes are being made in ongoing use. Little preparation or thought is being given to improving innovation use or its consequences.

Refinement:
The user varies the use of the innovation to increase the impact on clients [pupils/students] within immediate sphere of influence. Variations are based on knowledge of both short-and long-term consequences for clients [pupils/students].

Integration:
The user is combining her/his own efforts to use the innovation with related activities of colleagues to achieve a collective impact on clients within their common sphere of influence

Renewal:
The user re-evaluates the quality of use of the innovation, seeks major modifications of or alternatives to the present innovation to achieve increased impact on clients [pupils/students], examines new development in the field, and explores new goals for her/himself and the system. As a result, Hall and Hord's approach and scale had an impact on our understanding as to how expertise might be measured. For example, as Figure 1 indicates, while there does seem to be a level of congruence between Flyvbjerg and Hall and Hord, the expertise scale as it currently stands has to be 'stretched out' since certain stages of expertise appear to apply to a number of Hall and Hord's categories. Conversely, the language used in the Hall and Hord scale required substantial modification so that it effectively captured the expertise being displayed by given types of use. For example, that a user was able to modify and augment an innovation in order to obtain greater impact implies a high level of connection between formal knowledge (the intervention) and tacit knowledge (practical understanding of the situation in hand); it implies a high level of expertise. Of similar interest was whether modifications were undertaken consciously or unconsciously, with the latter suggesting even greater levels of expertise. At the same time, this modification meant that we were able to tackle some of the critiques aimed at this scale, such as Landry et al.'s (2003) suggestion that the Levels of Use scale focuses on specific uses of research: we ensured that our revised scale -which drew on but did not simply replicate the essential features of Hall and Hord's work -looked at changes in behaviour occurring as a result of practitioners' ways of engaging with research as opposed to actual instances of research use. To be clear, the purpose of the work was to produce a new scale, not simply to use the existing scale offered by Hall and Hord which did not quite meet the needs of the study in question. Our new and adapted scale effectively combines two existing pieces of work to measure evidence use among, in this case, early years practitioners. We deploy here the parts that best enabled us to measure expertise in evidence use.
Correspondingly we developed the following descriptors based on Hall and Hord's Level of Use scale that sought to elicit notions of expertise in relation to evidence use (see Figure 2).

Integration to Renewal
Proficient to Expert PROFICIENT+: I regularly adapt the strategies suggested by the research in order to make them even more effective or so that they can apply to a number of situations within my setting.
-Expert EXPERT: I frequently use the research strategies we learnt about but I now incorporate them into my day-to-day practice in an automatic rather than conscious way, as I tackle specific situations and issues.
Again, respondents were asked to consider all eight elements and to indicate the extent to which they agreed with them, using a scale that ranged from 'strongly agree' to 'strongly disagree'. This enabled us to establish a more nuanced understanding as to whether a level of expertise had been reached and, as a result, an understanding of 'typologies' of evidence use being displayed by respondents. The survey was administered in December 2013, some four months after the beginning of the project. In total, 34 of 36 participants completed the survey.

In-depth semi-structured interviews
To explore evidence-use typologies emerging from the survey in more detail, we conducted indepth semi-structured interviews. The purpose of the interviews was two-fold: first, to explore the actions and behaviours of individuals at different levels of expertise (as indicated via their survey responses); second, to explore the 'trustworthiness' of the scale by asking questions to assess comprehension of the questions and whether the questions were measuring what they were designed to measure. In total, 10 interviews were undertaken.

Adapted ladder of research use
Beginning with results from our amended ladder of use scale, it can be seen in Table 1 that all respondents agreed or strongly agreed that the research was well communicated (stage 1); comprehension (stage 2) was also high, with 94 per cent agreeing or strongly agreeing that they understood what had been communicated (no respondents disagreed or strongly disagreed with this statement). This pattern continues until stage 4 of the ladder ('I can see how the research related to my setting'), where 3 per cent disagree; correspondingly, this same 3 per cent disagreed that they felt they had started to use the research in subsequent exercises (stage 5) or had any intention of using the research moving forward (stage 6). Conversely, 97 per cent suggested they had actively discussed the research presented (stage 3); 94 per cent could see its relevance and had discussed the research in the exercises throughout the day (stages 4 and 5); and 97 per cent expressed the intention to use findings as they developed approaches and strategies for their setting (stage 6). To follow up on stage 6, we asked an open-ended question: 'Why do you want to use the research?' and inductively coded the responses. Here, the majority (46 per cent) suggested that use was related either to enhancing the 'general' or generic quality of their practice; meanwhile 20 per cent wanted to use the findings to augment specific areas of their work that they felt needed improving. In addition, 6 per cent wanted to use the research to help them improve the practice of others in their setting. Other responses included a desire to better understand a particular aspect of child development (6 per cent) and to develop evidence-informed practice (3 per cent), while one more cynical respondent suggested that the findings would be useful to back up changes that had already been made. Finally, we examined what else had been discussed within the group exercises. Here, responses centred mainly on 'general discussion about the EYFS' (32 per cent), with others subjects low in frequency, but ranging from 'extending children's interests' to 'notions of quality'. Importantly, however, just over a third of respondents (36 per cent) indicated that these discussions were pertinent and did influence how they developed ideas and strategies for their settings. The same number suggested that their strategies were also informed by their practitioner knowledge, that is knowledge of their children and setting.

Adapted levels of use scale
Findings from our combined expertise and levels of use scale appear in Table 2. In administering the survey we explained to participants that each question built on from the one before. If they felt they had reached a specific level already, participants were asked to agree (or strongly agree) with the statement. This process continued until respondents reached a level that did not reflect their engagement or behaviour in the way described. As can be seen, and as might be expected after four months of activity, the majority of respondents (94 per cent) indicated that they had made attempts to use at least some of the research presented as part of their day-to-day practice. Following this, however (and again, as might be expected given the likelihood that there will generally be fewer people at the highest levels of expertise), there is a gradual decline in the percentage of respondents agreeing or strongly agreeing with the statements regarding positive research use. For instance, as we progress through the questions: 94 per cent agree or strongly agree that their use was around 'orientation', that is in considering how research findings might be implemented; this falls to 77 per cent for 'preparation' (making plans to use findings); while 65 per cent suggest they are 'proficient' (beginning to tailor research findings so they are contextually appropriate). At the highest levels of expertise only 47 per cent agreed (nobody strongly agreed) that they were regularly adapting the strategies suggested by the research in order to make them even more effective ('proficient+'); likewise only 35 per cent agreed (nobody strongly agreed) with the statement relating to full expertise: 'I have used the research strategies we learnt about so often now that I barely need to even think about them; they've just become part of my day-to-day practice.'

Conclusions
The findings presented here represent only an initial attempt to establish ways of measuring: (i) meaningful evidence use, defined as the result of combining formal and practitioner knowledge such that this leads to expertise; and (ii) the effectiveness of knowledge-creation activity as a means of establishing meaningful evidence-use. Nonetheless, based on the results of the two surveys outlined above, our findings would appear to be encouraging. Not only does knowledge-creation activity seem to provide an effective way of communicating research and keeping it 'top of mind', but it seems to enable practitioners to develop expertise in using evidence by helping them combine formal knowledge with practice-based or tacit knowledge in practical ways. Simultaneously, our interview data appears to triangulate and verify the trustworthiness of measures used in the surveys as a way of measuring levels of expertise in evidence use (Lincoln and Guba, 1985). It provides us with confidence that not only is our approach to measurement 'valid' but, as a result, it meaningfully represents the level of expertise captured: we are confident in our claim that knowledge creation works to establish expertise in evidence use. In addition, by using both the ladder of use scale and an amended levels of use scale, we believe we have tackled the issues regarding 'self report' raised by Levin (2013) above. Namely, rather than examining participants' perceptions of instrumental research use (what people believe they did) we ask participants to make statements in relation to their present behaviour -how they are currently responding in relation to an ongoing activity. This ensures we are looking at research use as process rather than event (Landry et al., 2003).
At the same time, the approach we have developed to assess expertise in evidence use does have its limitations, namely that it is only really applicable to situations where researchers are working regularly with practitioners on areas of practice development, where the general desire is that these areas should become evidence-informed. In other words, our approach measures the effectiveness of activity to engage practitioners with evidence rather than, more generally, whether practitioners are expert evidence users. However, this does not mean that our approach has relevance only to relatively few instances of initiatives or activity. On the contrary, in school systems such as that in England -where the expectation is that schools or alliances of schools should lead their professional development activity often in partnership with, or drawing on the help of, universities -it is likely that these instances will soon be increasing in number (Greany, 2014). Similarly, the scale could be equally meaningful as a way to measure the extent to which practitioners have engaged with the findings of practitioner enquiry or action research. As such, we suggest that the scale we have set above will soon play a vital part in helping establish whether 'self-improving' school systems (Hargreaves, 2010) are also those that are meaningfully evidence-informed.

Notes on contributors
Dr Chris Brown was a John Adams Research Fellow at the Institute of Education (IOE) in London and was recently appointed to a lectureship in the IOE's London Centre for Leadership in Learning. He has written two books and several papers on how evidence can aid the development of policy and practice, and has presented on the subject at a number of international conferences both in Europe and North America. He was recently awarded a significant grant by the Education Endowment Foundation to work with 100 primary schools in England to increase their use of research. Sue Rogers is Professor of Early Years Education at the Institute of Education in London. Through her research, she is keen to understand pedagogy in early years education, especially the role of play in educational contexts. She has published widely in the field, including three books, and led a range of funded projects in early years settings and primary schools.