Students’ Perceptions of Learning Processes as Co-Authors of Digital Tabletop Activities

We conducted a small-scale study in order to explore students’ perceptions of the learning processes when engaged as co-authors of content for collaborative higher order thinking skills learning tasks. We specifically designed the process to allow for self-critique – where authors can observe their creations being solved and therefore understand where they may improve their design. We collected data over a three-day period from a sample of twelve thirteen year olds, working in teams, authoring content for Digital Mysteries (a higher order thinking skills collaborative learning application based on the digital tabletop). The study was structured to follow Bloom’s taxonomy, a continuum of cognitive skills that develop during a learning process. We found that 1) rather than follow this continuum, skills developed in a non-linear manner due to the abstract nature of the authoring activity, and 2) the students’ demonstrated good metacognitive insights into the authoring task, technology and collaborative learning as a whole.


INTRODUCTION
Reflection and feedback are important aspects of learning higher order thinking skills (Baker & Lund, 1997;Collins & Brown, 1988;Hattie & Timperley, 2007). Allowing students to assess each other's work provides opportunities for feedback, reflection and insights into the learning processes and the thinking of students (both those providing the feedback and those receiving it). Accordingly, peer feedback is a common classroom tool for learning (Falchikov, 1995). This self and peer assessment can also be applied to group work (Falchikov, 1991). However, these processes are usually applied to artefacts that only require passive engagement by the students providing the feedback (e.g. written pieces of work, posters, or presentations) (Nicol, 2010), rather than deep engagement (e.g. solving a problem or interacting with the created artefact). Moreover, this depends on the student's ability to articulate their feedback as useful constructive critique.
A more authentic and potentially more useful feedback results from students actively engaging with a created artefact (e.g. interactive content). This engagement with the artefact, allows them to externalize their thinking processes to an external observer (i.e. the authors). Digital technologies have significant advantages when it comes to providing opportunities for creating such interactive content.

Figure 1: Students using mysteries authored by their peers
We present a small-scale study designed to explore students' perceptions of the learning processes when observing their peers engaging with a Higher Order Thinking (Resnick, 1987) activity that they had created. We chose a highly visible task, Digital Mysteries (AS  (figure 1), that is specifically designed to allow externalization of thinking and the promotion of Higher Order Thinking Skills (HOTS). This creates opportunities to receive implicit genuine feedback from peers, through solving the mystery, and consequently allows for self-critique from the authors. We also aimed to see if this gave greater insights into the target task (i.e. Digital Mysteries). I.e., do students gain a greater understanding of the 2 HOTS required to make a good mystery when observing others solve their work?
In order to capture the reflective thinking of participants, we collected materials generated during the study, conducted group interviews and provided a "diary room" where participants could capture their thoughts individually or in groups. We analysed the interview and diary data by conducting a thematic analysis with multiple coders. We also analysed the outputs materials (i.e. Mysteries) generated by the participants and compared our assessment of them with the feedback and impressions of the participants when they observed others solving them.
Our contribution in this paper is to provide a better understanding of the thinking processes involved in creating, and observing the use of, engaging interactive digital content that focuses on the promotion of HOTS.

Bloom's Taxonomy:
The authoring activities were designed to correlate with the cognitive objectives outlined in Bloom's taxonomy (Anderson et al., 2001;B. Bloom et al., 1984), originally developed by Bloom in the 1950s (B. S. Bloom, Engelhart, Furst, Hill, & Krathwohl, 1956). The taxonomy organises cognitive process into a hierarchy: knowledge, comprehension, application, analysis, synthesis, and evaluation. Bloom argues that the hierarchy is a continuum, i.e. a learner cannot comprehend without first remembering it, i.e. progressing from Lower Order Thinking Skills (LOTS) to Higher Order Thinking Skills (HOTS). In particular, we focused on the knowledge dimension as defined in Krathwohl's Revision of Bloom's Taxonomy (Krathwohl, 2002).

Digital Mysteries
Digital Mysteries was chosen as the target application as it is a HOTS collaborative digital tabletop activity where groups construct an argument from evidence as a response to a question or prompt (i.e. a "mystery"). It requires critical thinking, visuospatial externalisation of thinking (i.e. making thoughts visible to peers) and active collaborative engagement (A Kharrufa, Olivier, & Leat, 2010; AS D Leat & Nichols, 1999). A Digital Mystery is presented over three explicit stages (in order to "scaffold" the process (D Wood, Bruner, & Ross, 1976;David Wood & Wood, 1996)). I) a reading -evidence is presented; ii) grouping -evidence is categorised by participant-generated theme (and irrelevant evidence discarded); and iii) sequencing -the argument is constructed by linking individual pieces of evidence together. A reflective stage allows the process to be re-played, and different decisions can be explored.
It exploits the affordances of the digital tabletop as a collaborative mediumi.e. collocated, unhierarchical, visuospatial representation etc. (Dillenbourg & Evans, 2011;AS Kharrufa, 2010;Rick et al., 2009;Scott, Grant, & Mandryk, 2003;Vygotsky, 1978). The process is visible, allowing observers (e.g. a teacher or author) to see thinking and strategies of the group completing the taskallowing self-criticism about the mystery design.
A well-designed mystery allows for multiple interpretations of the evidence, and as such, there is no "right" answer. Performance depends on how arguments are constructedthat is how well the evidence is used, categorized and sequenced. The authoring tool (figure 2) is a standard PC application that allows creating evidence items ("slips") (by adding text and images). The meta data that determines the rest of the task can also be added, i.e. "the question", data about what slips could be grouped together and a series of hints.

THE STUDY
The study took place over three full school days. Participants engaged in initial preparation activities in their school, and then in the University Lab. The students were aged between 13 and 14 and were organised into three groups of four students (12 total). The groups chose social renewal topics for their mysteries.

Day 1: School Classroom
Activity: Familiarisation with mysteries as a learning activity, investigating a theme for the mystery to be created. Students solve a mystery on paper to introduce the concepts. Students then work with their teacher and a researcher to create materials for subsequent use within a digital mystery, initially as Outputs: Video Diaries, Self-document photographs, Short report, Individual interviews.
Knowledge Dimension: Reinforcing Conceptual Knowledge of elements of a larger structure (i.e. a mystery) and how they fit together Moving toward Procedural Knowledge, i.e. methods of inquiry and how subject specific skills and when to use them.

Day 3: University Lab
Activity: Finalise content and using the completed Mystery on the Digital Tabletop. Observe another group solve the finished mystery. Showcase finished mystery to parents in final event.
Outputs: Video Diaries, Self-document photographs, Video of Digital Tabletop Testing, Group Evaluation Questionnaires of observations of other groups completing their mystery, Video of peer review activity, Individual Interviews, Final mysteries.

Knowledge Dimension: Continuing Procedural
Knowledge of what skills to use and when for creating a Mystery. Metacognitive Knowledge about how others and the authors learn, i.e. strategical knowledge, appropriate contextual and conditional knowledge and Self Knowledge.

ANALYSIS
We conducted two analyses. Firstly, a thematic analysis on the interview and diary data. This analysis was deductive, with codes based on Blooms Taxonomy (Anderson, Krathwohl, & Bloom, 2001;B. Bloom, Krathwohl, & Masia, 1984). We theorised that the participants would transition through the stages of the taxonomy (with some overlap). What we discovered instead was that thinking was less rigidly structured, with different kinds of thinking occurring throughout the task.

Video Analysis
Four researchers independently encoded the data by using the categories Factual, Conceptual, Metacognitive and Procedural. We then discussed and formed a consensus on these encodings.

Day One:
We expected the analysis to produce primarily factual knowledge considering that the activity was mainly about preparing contents for the mysteries. Data was collected through short interviews at the end of the day consisting of two questions; "What have you learned?" and "What did you remember". However, when encoding the data, we primarily encountered conceptual, metacognitive and procedural responses.
Day Two: In addition to the interviews, a video diary facility was available to the students. We planned the activities to progress to conceptual and procedural thinking, but there were actually more instances (proportionally) of factual thinking, than on day 1. However, these factual responses were mainly in the video diaries rather than as interview question responses, and were mainly concerned with the technology rather than "on topic" (Social Renewal theme). Procedural and Conceptual thinking came across in the diary entries, particularly concerning collaboration (similarly to the first day's responses), but in the context of the technology. However, metacognitive responses in the diaries showed that the downsides (glitches) to using technology were also apparent: The interviews were more in line with the expectations, with the majority of responses being procedural.
Day Three: By the third day, students were mostly responding as expected, that is by showing metacognitive and procedural thinking. The students engage with the activity of watching others complete their Mystery and there are metacognitive discussions about how other groups approach solving Mysteries. In the interviews, there is a clear metacognitive summation of the whole study from several students along with procedural thinking.

Summary:
The results show that the expected continuum of thinking did not take place, although by the final sessions it had largely conformed to Metacognitive/Procedural. In the initial sessions however, which were designed to illicit factual thinking (i.e. generate evidence), little factual thinking took place. It was only later (when the evidence was being curated into the Digital Mysteries authoring tool) that thinking became more "factual".

Mysteries Analysis
In the second analysis, we compared (crossvalidated) our assessment of the mysteries with the reflections of the users of the mysteries, as well as reflections of the authors observations.
Researchers: Two researchers assessed and scored the mysteries using criteria outlined in the design of Mysteries (David Leat & Nichols, 2000) and Digital Mysteries (AS  Authors: Authors were asked questions on how well the users had completed their mystery, i.e. difficulties, strategies, and whether they had a good solution. Two groups (2 & 3) did not fully comprehend the open-ended concept of a mystery, assuming a correct answer and way to group evidence. They commented negatively on the users who did not follow their expectation. These authors also thought that the users were "overthinking" their mystery. The fact that the users were still able to explore alternatives perhaps shows that it is possible to create a "good" mystery without fully understanding its purpose. Other groups (1 & 4) gave a more insightful response, understanding that their mystery was too "telling" and "leading users to an obvious answer", which was correctly seen as a negative. While largely agreeing with the users about the difficulty of their mysteries, author's tended to disagree about which tasks were harderwith a general consensus being they thought users found grouping hardest (while users thought sequencing was more difficult).

Users of Mysteries:
Users were asked about the content and the structure of the mystery. They were able to identify key questions, and agreed that the evidence provided was appropriate. When it came to grouping and sequencing, users were more inclined to rate the process as "good" if they found it easy, and vice versa. Users pointed out that sequencing was much easier with a temporal aspect, i.e. events that occur along a timeline, rather than abstractly related.

DISCUSSION AND FUTURE WORK
We set out to explore how students perceive the learning processes involved in co-authoring educational content for a collaborative HOTS task and subsequently observing their peers completing the task. In particular, we wanted to explore the selfcritique generated by these observations. We were also interested in how their thinking changed across the study, and whether it would follow the "continuum" presented in blooms taxonomy.
The students' thinking did not follow the expected continuum suggested by Bloom's taxonomy. This may be a side effect of a task such as authoring, which is essentially already a HOTS as it requires the high level concepts of the subject matter to be understood before effective authoring can take place (students where not only required to think about the content, but also prepare meta-content as well such as associated sub-questions, possible groupings, and even hints for users). Conversely, students who did not fully understand the higher level concepts of digital mysteries would instead critique the end users rather than gaining insight into their own work. One explanation of this is that fuller comprehension of the "target" HOTS task is required (or at least highly desirable) in order to be self-critical about the authoring process.
Getting students to observe their peer solving their mysteries gave them better understanding of the concept of 'writing for an audience'. This was demonstrated by comments like 'overthinking' our mystery -from a group who did not seem to fully comprehend the open-ended nature of mysteries. Such understanding of how the audience may react to created content could not have been achieved through creating 'passive' content that does not have elements of interactivity, active engagement, and 'externalization' of thinking while engaging with content. This is shows a clear example of the potential benefits of creating engaging digital content as opposed to more traditional passive content (e.g. written essays).
This small study has presented some interesting possible avenues for further research. In particular, the concept that in order for effective self-criticism to take place in an authoring context, a good understanding of the HOTS goals of the final output is necessary. This suggests that future studies could be designed that test for this at a larger scale (timewise and sample-size wise). E.g. a study with students already very familiar with Mysteries (rather than having the concepts introduced within the study itself) may show clearer insight into this aspect.
There are several additional ways in which the study could be improved. In order to produce a more realistic output (i.e. in line with what most students would experience), the study could be conducted entirely in the classroom, and with students more representative of the achievement levels within schools. Allowing teachers to give more formal assessment of the outputs, and feeding back to students over a longer time period may also produce more concrete conclusions.