Does experience with digital storytelling help students to critically evaluate educational videos about history?

Educational videos are becoming increasingly important for schools. More and more often, students consume videos on YouTube in order to carry out school tasks. At the same time, the digital world is increasingly influencing perceptions of history. The internet contains numerous examples of how history is instrumentalized. Counterfeiting and manipulation distort historical information and abuse it for political purposes. This article presents the results of a research project on history teaching in a seventh grade (age 12–14) class in Germany. The study’s aim was to find out if creating one’s own videos using the method of digital storytelling generally leads to a more critical evaluation of educational videos. Students produced short videos on the subject of ‘European expansion in the early modern period’. One group was secretly commissioned to portray the Europeans as superior to the indigenous societies of America, thus creating a manipulative video. At the end of the lesson, the students rated the credibility of the videos. In addition, interviews with students were conducted. The aim was to investigate whether students trained in digital storytelling could easily identify biased information. The data were analysed using qualitative text analysis. Findings show that students primarily judge videos based on aesthetic features, rarely adopting a media-critical perspective.


Introduction
Educational videos are becoming increasingly important for schools. Such media are now indispensable both in the classroom and in extracurricular education (Wimmer, 2017: 12). More and more, students are consuming YouTube videos to carry out their school tasks. Recent studies show that as many as 86 per cent of adolescent students use YouTube multiple times a week, and 56 per cent even use it daily (MPFS, 2016: 38). These numbers do not say anything about the positive and negative effects of this media consumption, of course. Digitality can certainly support historical learning. At the same time, the digital world is increasingly influencing perceptions of history. The internet contains numerous examples of how history is instrumentalized. Counterfeiting and manipulation distort historical information and abuse it for political purposes.
Even students are confronted with such manipulative information. In the wake of rising nationalism around the world, these manipulations are often characterized by the devaluation of 'others' and the positive evaluation of one's 'own' group or nation. Thus, history is increasingly being instrumentalized to promote identity politics (Barsch, 2019).
At the same time, amateurs are now able to create professional-looking media. Today, no one needs to be a programmer to build a professional-looking website. Likewise, there is no need for expensive technology to produce videos, slide shows and other visual media -a smartphone is enough. Thus, the 'knowledge' available on the internet (whether genuine scientific knowledge or merely subjective opinion) is growing rapidly. The immediate availability of this increasing knowledge has led to a change in reception habits. Libraries were long regarded as the first point of call when gathering information. Today, information is mainly sought and processed in the digital space (Wineburg and McGrew, 2017;Demantowsky, 2015). History education thus finds itself confronted with challenges. We need to enquire which mental processes are necessary to distinguish manipulative from neutral accounts, and historically valid narratives from those written with no regard for professional criteria. Teachers need reliable information about the skills, abilities and preparation that students require to assess the information content of digital offers about history.
These challenges affect the very core of history teaching. Historical learning is designed to equip students with thinking tools that enable them to participate critically in the culture of history. They must therefore also be enabled to critically question the products of historical culture (Kühberger et al., 2018: 418). The promotion of methodological competencies such as these is crucial if students are to actively engage in the digitized society of the future (Kerber, 2015: 105). The handling of digital media is also becoming increasingly important for the training of history teachers (Haydn et al., 2015: 248). Donnelly (2013: 16) noted, however, 'that most history teachers developed their film implementation and integration strategies through an intuitive process based on a combination of "trial and error", professional experience and knowledge of the learning styles and preferences of their students'.
Against this background, the question of the skills people need to be able to engage critically with the multitude of digital sources is of utmost importance both to schools and to society in general.

The research
Few empirical studies deal with students' abilities to assess specific digital media in the context of historical learning. Some studies focus on the impact of films on students' motivation. Cutajar (2018), for example, noted that engaging with historical films had a positive effect on the historical understanding and motivation of students in secondary education. These films inspired them to ask historical questions. Analysing the films also increased peer interaction between the learners. Other studies found that students critically assess the historical reliability of films. However, when films are used in the classroom in secondary education, many uncritically accept the films' narrative (Donnelly, 2013: 17). Jones and Cuthrell (2011: 83) showed that the critical selection of video content for school learning also poses a challenge for teachers. In his study on secondary education students' use of digital media, Hodel (2013: 250) found that they often use the internet to copy and paste information to augment their historical knowledge. However, he also showed that learners use digital media to build collective narratives (ibid.: 311). John (2013) and Schmitt and Kowski (2011) studied university students' user behaviour and expectations concerning digital media. Their results show that while students certainly used such digital media offers to build knowledge, they often used these offers uncritically. Schwabe (2012: 152) examined the user behaviour of users accessing a regional history website, and among other things concluded that the reception of information on the internet is often characterized by a loss of orientation and by excessive demands. The way that users deal with videos, and especially educational videos, has not been explored empirically so far, although Neitzel (2010) showed that students perceive historical documentaries on television quite critically. Wineburg and McGrew (2017) found that many historians struggle to assess the reliability of digital information. The same can be said of students at universities (Wineburg et al., 2016;McGrew et al., 2018).
However, many of these phenomena are not strictly related to digitality, and studies have noted that students also have problems when working with and interpreting analogue sources, from which they often draw a positivist understanding of history (Beilner, 2002;Langer-Plän, 2003). Schönemann et al. (2011) also found that German secondary school students' skills for dealing with sources were inadequate. In a detailed study, Spieß (2014: 197) noted that few students were able to analyse historical sources properly. He also pointed out that we know almost nothing about the way children and adolescents work in out-of-school contexts (ibid.: 236). Thus, studies show that it is often difficult to get students to engage critically with sources. Nevertheless, this remains an essential goal of history teaching: 'When engaged in finding or evaluating evidence, pupils need to critically evaluate and compare multiple sources, synthesizing what they learn from them into a coherent description, explanation or account' (Van Boxtel and Van Drie, 2013: 45).
The study presented here is based on the hypothesis that experience in designing one's own digital media can be conducive to developing a critical attitude towards other media. The assumption is that collaborative digital work promotes a more critical approach to sources in general. Students are given an 'expanded scope for agency' and required to 'research multiple web sources, critically evaluating these sources. They submit drafts, then peer-review' these drafts (Kalantzis and Cope, 2017: 316-17). Accordingly, the students of the intervention group created historical learning videos based on the method of digital storytelling. Digital storytelling has become increasingly important for academic learning in the context of educational science, as well as in subject didactics, over recent years. This popularity is based on the idea that learners play an active role in digital storytelling. In particular, digital storytelling is believed to increase learners' motivation to engage intensively with a topic (Burmark, 2004). Thus, the aim is to support this active stance through technical support measures and facilitate knowledge building by structuring information and combining text, word and image (Robin, 2008: 222). There are indications that these methods can also promote cooperative learning (De Jager et al., 2017: 2575Ohler, 2013;Frazel, 2010). In addition, the method promises an increase in narrative activity, which is currently used in foreign language teaching in particular: In digital storytelling tasks, the students' texts go through several phases of design, writing, production and editing before the stories are completed, and thus, the learners are able to actively develop their own textual and media skills as well as the narrative strategies they use. (Heinz, 2016: 145; translated by the author) The growing popularity of digital storytelling can also be explained by its strong affinity to youth communication culture, in which platforms such as YouTube are of key importance -an importance that is growing due to mobile devices (Heinz, 2016: 144). As a method, it is also attractive for historical learning because it places a clear focus on narration, which means that digital storytelling can also be used to promote narrative competence (Kerber, 2017). So far, there are no studies on the effectiveness of digital storytelling in history lessons. Nevertheless, the method was chosen because, as described above, it potentially promotes students' narrative abilities and may help them to understand the structures that their peers use to construct their narratives. By themselves creating a narrative based on sources, the students engage with the question of which historical documents or images 'can be used as evidence about what happened in the past' (Van Boxtel and Van Drie, 2013: 44). Since digital storytelling compels students to tell a coherent story, they need to think twice about good reasoning. This is also a fundamental requirement of history teaching: 'In history, evidence is often incomplete or provides conflicting information. This means that there are always multiple interpretations possible. That is why argumentation is a crucial part of historical reasoning' (ibid.: 45). Historical reasoning 'is defined as ... an explanation of a historical phenomenon ... by asking historical questions, contextualising, using substantive and second-order historical concepts, and putting forward claims supported with arguments, which are based on sources used as evidence' (ibid.).
Based on this theoretical framework, the study presented here aims to provide insights into students' basic evaluation skills. The main research question is: does training in digital storytelling lead to a more critical evaluation of educational videos about history by students in secondary education? Sub-questions are: (1) How do students judge the reliability of educational videos about history?
(2) Which (historical reasoning) strategies do students use to evaluate the educational videos?

Method
The study was conducted in the seventh grade in a school in northern Germany. The average age of the student population was 13.4 years. In the intervention group, 10 participants were female, and 15 were male. The students were predominantly middle class. Two of the students had a migrant background. There were no learners with special educational needs. The whole class was involved in creating the videos. However, due to absence on the day of the survey or refusal to participate, only 10 students completed the questionnaire. The data of the three students who created the manipulated video were not considered for the evaluation, as they could not comment without bias on the criteria for the evaluation of the videos. The control group consisted of nine students of the same age from another class. Both groups received a ten-lesson series on 'European expansion in the early modern period'. The focus was on cultural contact between Europeans and the indigenous populations of America. Parallel to their history lessons, the students of the intervention group received training in digital storytelling as well as technical support to produce the videos. They learnt how to develop a storyboard, how to work with video software and how to use recording techniques. Over the course of the unit, the learners of the intervention group produced a video on the question 'European expansion: A world for all?' in groups of three or four. The videos were to be created based on the content covered in the lesson. The pool of sources was the same for all learners. However, the groups were also free to research sources and information themselves. The videos were to be about three minutes long. Learners were encouraged to create instructional videos that depicted historical events, taking into consideration specific criteria such as source orientation, multiperspectivity and controversy. The groups could choose the design of their videos freely and created them as a homework assignment outside regular classroom teaching. They were asked not to talk to their classmates about the content of their own video. However, one group was given a secret mission different from the rest of the class: it was told to adopt a decidedly pro-European perspective in the video. We made this explicit assignment because we assumed that all other groups would show the Europeans' behaviour towards the indigenous peoples as unjust and aggressive -which they indeed did. The group that had received the 'secret mandate' to present the video with a pro-European perspective was made up of the higherperforming students in the class.
The control group from another class had regular lessons on 'European expansion'. They did not receive digital storytelling training and did not produce a video. Surveying both intervention and control groups aimed to answer the two research questions: (1) how do students evaluate videos?; and (2) what influence does digital storytelling training have on video rating?
The videos produced by the intervention group were very varied in both content and aesthetics. While many videos used creative techniques (such as a stop-motion film using toy figurines and a cartoon including many images), the 'manipulated' video was very fact-oriented. In it, a narrator explained the historical events using mainly dates and 'facts'. While the narrative was partly supplemented by pictures, often the screen was just black while the speaker was talking. The film ended with a clear verdict on the historical events: 'Our conclusion at the end of the video is that the Europeans were very good and the indigenous population was pretty bad to them.' (All student statements are translated by the author.) At the end of the lesson, each group working together was shown two videos by other groups. The allocation was not random: the 'manipulated' video was always included. No group rated their own video. We took this step to ensure that each group was confronted with two different perspectives when assessing the historical events in the videos. The same procedure was chosen for the control group. Here, there was no need to ensure that their own video was not rated, since the students had not created one themselves. Accordingly, they rated the videos created by the students in the intervention group.
The students filled out an evaluation form after viewing each video. This form consisted of a four-level Likert scale (agree/reject) rating these items: • the video contained all the important historical information • it's fun to watch the video • the video showed history the way it happened • my assessment of events was the same as that of the video • the story in the video follows a meaningful structure.
The five statements were chosen because their range allowed the students to refer to their knowledge as well as to aesthetic criteria. The evaluation form helped to answer the question of how students judge the reliability of educational videos about history.
These individual evaluations served as key questions for subsequent interviews with all learners. Each team that had worked together on a video was interviewed in a group interview to elicit more detailed reasons for their responses to the questionnaire. The interviewers led the students through supplementary questions such as 'Why did you find video X more credible?' or 'What would have made the video even better?' By choosing this approach, we wanted to reveal strategies of historical reasoning with a focus on evaluating the reliability of the videos both on a subjective and on an intersubjective level. As all the students of the intervention group had gone through the process of video production, we wanted to find out whether this led to a more critical approach to the information presented. In particular, we aimed to determine whether the students recognized the one-sided perspective of the pro-European video, as all interviewees were familiar both with the historical content and the process of digital storytelling. The interviews were to provide in-depth information about which strategies of historical reasoning students used to rate the videos. We also conducted interviews with three groups from the control group.

Data analysis
The individual questionnaires based on the Likert scale were evaluated according to the frequency of the answers given. The aim was to develop general statements on the evaluation of the videos from a comparative perspective.
By analysing the group interviews, we hoped to gain deeper insights into the students' historical reasoning and their criteria for evaluation. These interviews were analysed using the method of qualitative text analysis (Mayring, 2010;Schreier, 2014). We anticipated that the categories thus generated would shed light on the criteria according to which the students evaluated the reliability of the videos. For this purpose, the transcripts were first encoded openly in several rounds in order to identify concepts for the formation of categories and subcategories. The open coding revealed numerous strategies that the students adopted when assessing the examined videos. These strategies were combined into categories, on the basis of which we once again carried out several coding cycles. We were able to identify four main categories. These are listed below by quantity, including the subcategories (see Table 1): • aesthetic criteria (21 references in the interviews) • formal criteria (15) • historical criteria (=source criticism and other aspects of historical thinking) (4) • moral criteria (2).
Among the aesthetic criteria we included, for example, 'pictures make the video credible' and 'music distracts', as well as aspects such as 'humour is dubious' or the opposite, 'humour is good'. The formal criteria include, for example, the subcodes 'explanatory narrative, common thread' or 'quantity of information'. We were particularly interested in historical criteria. However, only a few students mentioned such criteria. Specifically, these were 'facts', 'limits of perspective' and 'validity'. We identified 'moral criteria' only in two passages. 'Moral criteria' means that videos are rated as more reliable if the film's speaker judges incidents from today's point of view. One group rated the manipulated video as more reliable than any other video, simply because it offered a moral judgement. The quality of this judgement was not reflected upon, however.

Results
All interviewees had become familiar with the content of 'European expansion' according to the history curriculum in lessons delivered directly before the interviews. The intervention group also gained theoretical and practical knowledge in digital storytelling. The starting conditions were therefore comparable at least on the level of content. First of all, in the interviews, most aspects of the manipulated video were rated as 'better'. Humour (positive) Humour enhances the video. 'If this is as boring as the first one, where they just talked, without any real concern. In the second, they are so, it was just funnier.' (C2_P2) 4 Humour (negative) Humour has a negative effect on the seriousness of the video. 'I think the first one is better too, because, as J. said, the second one is so much funnier and the first one is so serious and more information.' (E4_P3) Sub-question 1: How do students judge the reliability of educational videos about history?
The analysis of the individual questionnaires revealed hardly any differences in the way the individual videos were evaluated. The experience with digital storytelling also seems to have had no influence. All the videos are rated rather positively. Even if we look only at the rating of the 'manipulated' video, there are scarcely any differences between the two groups (see Figure 1).

Figure 1: Evaluation of the 'manipulated' video by survey group and control group
We see that the assessments that differ most mainly refer to the affective criterion 'fun', while the technical assessments are almost identical for the manipulated and the other videos. Both groups show a relatively high agreement with item 3 ('The video showed history the way it happened'). This is surprising. In the group interviews, the students stated that the story would have been different from their point of view.
Sub-question 2: Which (historical reasoning) strategies do students use to evaluate the educational videos?
The previous knowledge of the subject built up through the lessons had no influence on how the vast majority of respondents rated the video. The external form (in the categories aesthetic and formal criteria) clearly outweighs other criteria. Moreover, the quantity of information (without reflection upon the quality) has an impact on the rating. One group from the control group said that the depiction of the indigenous population as aggressors seemed problematic and wrong: C1_P1: Mm [mumble], the conclusion was, I believe, in the first video, that, so it was portrayed as saying that the Christians are the good guys and everyone else is against them … Well, in the end, that was where he was, I believe, no idea what tribe, but then some tribe then tore the Bible from his hand and threw it to the ground, then there came to fight, and then you said yes, the conclusion from this video would be that [they] are actually against Christians and Christians are the good guys. C1_P3: It was not like that.
However, this finding did not affect the overall rating. Towards the end of the interview, the group was asked to vote for the winning video in a fictitious award ceremony, 'The Best Video Tutorial', and deliver a laudatory speech about it. The manipulated video was chosen: C1_P1: I liked the first video better because I got it explained a bit more precisely, even if it was not very informative, but it was a lot more ... C1_P2: Yeah, that came over better somehow. C1_P1: Yes, because the other video was just more the characters and so on shown and more music displayed. C1_P2: And more attention was paid to incidental things. C1_P1: Yeah, because, look, if you're there, so in the first video it's just roughly summarized, in the second with details and the rough, where the thread actually runs, was omitted yes. That's why I find the first a little better. C1_P3: Yeah, me too, because the second one you always had this music and that kept distracting you from the actual video, so you should find out what, what they should say, what happened that way. I just found the first extreme, I found the first better, because there we received more information, the second just said, yes, was childish.
The students' own experiences with digital storytelling appeared to have no influence upon the evaluation criteria. Although the procedure for producing one's own videos was presented in great detail, experience with the selection of sources did not lead students to take these methodological aspects into account when evaluating other films.

Summary results
Formal-aesthetic criteria determine students' judgements on the reliability of videos about history. The students barely pursue historical reasoning strategies as defined above. Learners only ask minor historical questions and reflect only slightly on the evidential value of the sources. Historical knowledge seems to play only a minor role in the evaluation of videos. Historical judgements (or moral criteria) have no bearing on the evaluation. The mere mention of dates and names seems to increase credibility, as does making a personal statement -even if the respondents possess knowledge that contradicts this statement. Overall, this means that training in digital storytelling does not lead to a more critical evaluation of the viewed videos. Our initial hypothesis has thus been disproven.

Discussion
As already indicated, the informative value of the present study is limited. As mentioned above, the group that had received the 'secret mandate' to present the video was made up of the higher-performing students in the class. Possibly, therefore, the reference to a historical 'framework' was more clearly perceptible. But perhaps the very task of adopting a certain perspective in one of these rather contradictory sources led the Does experience with digital storytelling help students? 77 students to engage particularly intensely with film production. Although students were asked not to talk about their content while creating the videos, we cannot rule out the possibility that this happened anyway. If such information was leaked, it could have had an impact on the rating of the video created by the high-performing students. The video might have been rated better because the classmates knew it was created by a stronger group. In general, we need to consider that students rated videos made by their classmates. This could have resulted in a more positive judgement overall.
In addition to the small sample (two classes of a school), the respondents' age needs to be taken into account. This review included several components, along with historical content as well as (and maybe even mainly) aesthetic features. All the students seemed to struggle with historical judgement (see also Mehr, 2012). The same may also apply to adults. However, for students of this age group, where historical skills are still at an early stage of development, the distinction between professional and formal-aesthetic content is even more difficult than for more experienced learners. Some students realized that the presentation of events in the manipulated video was contentious. However, this did not lead the video to be rated more poorly in general. This may show how difficult it is to harness expertise and judgements as a basis for concrete actions. Insofar as it is precisely (historical) educational videos that play a major role in students' life-worlds, this provides relevant insights for history lessons. Formal criteria for evaluating sources and representations are nothing new, and they are clearly not enough. Now more than ever, a clearer discussion on the handling of information seems necessary. It seems that although perspectives are recognized by students, they do not necessarily draw conclusions for their individual assessment of information. If aesthetic and formal components have a greater impact on the credibility of information than professional criteria, then this investigation suggests that greater emphasis needs to be placed on judgements.
We cannot draw any general conclusions from this preliminary investigation on the use of educational videos in history lessons. The results indicate that both aesthetic components and formal aspects (numbers, data, facts, 'information') may contribute more to evaluation than technical knowledge and criteria-based assessment. Information, it seems, has not been judged in terms of quality, but above all in terms of quantity. However, the results presented here are too limited for us to draw conclusions about general phenomena from them. Therefore, similar studies on older students are required to understand the effects of maturation and development, and a quantitative study should be initiated to determine whether the observed phenomena are general trends. Although studies suggest that even adults have difficulty evaluating sources critically (see above), there is evidence that maturation can also influence historical reasoning (Cooper, 1994). The results of this study could also be useful for further research. For example, the criteria developed here for evaluating historical videos and images could be transferred to other research questions.
For the practice of history lessons, however, the questions raised here remain of fundamental importance. Well-packaged identity offerings in the media based on historical narratives, which increasingly promote national interpretations of culture and society, need to be taken up and deconstructed in the classroom. It is clear that precisely in times when groups such as the 'Identitarian Movement' target children and adolescents with racist and nationalist ideas, equipping students to deal competently with 'historical culture' is an essential task of history teaching. Insofar as this historical culture is increasingly disseminated through visual media, history lessons in schools need to offer a critical antidote to nationalist narratives. Promoting the understanding of history as a construct among students means above all penetrating and analysing the structure of historical narratives. Ultimately, however, participation in the culture of history also means empowering students to consider their professional judgement sufficiently valuable not to be overridden by formal and aesthetic judgements.
In order to empower students, we need to expose their automatic evaluation strategies based on formal aspects and aesthetic components. As this automatic reliance on formal-aesthetic criteria prevents students from engaging critically with educational videos, one goal of lessons could be to make students more aware of their own 'blind spots'. Historical reasoning does not just mean analytically grasping phenomena of the past. Rather, it also means being able to participate critically in the present-day culture of history. To this end, real, subjective but technically based judgements and reflections on the validity of these judgements need to become the aim of historical learning. One way to support students in critically dealing with information might be to accompany the process of digital storytelling more through metacognitive phases. Thus, this approach supports them not only in creating conclusive historical narratives, but also in proving them with historical evidence. Possibly checklists listing the principles of historical argumentation can be used here to support the production of one's own videos and the evaluation of other videos. This checklist can have the function of a scaffold and enable cognitive reflections to counter the otherwise overwhelming influence of aesthetic considerations.