Transactive Episodes : Exploring Interaction , Memory and Narrative

This practice-based doctoral research investigates various methods of human-computer tangible interactions and their relationships with interactive storytelling, which will lead to the construction of an interactive narrative model over a research-through-design process. The design process aims to inform designers and makers of novel ways of converting people’s memories into playable stories that are accessible and engaging to general audiences. The outcome can also contribute to the discussion of the role of interactive technologies as a mediator between personal episodic memories and public collective memories.


BACKGROUND AND LITERATURE REVIEW
This research explores the relation between narrative and memory as Transactive Episodes.This concept is loosely based on two notions borrowed from applied psychology: transactive memories, which are the pool of shared moments from individuals on a defined social group, and episodic memories, which are autobiographical moments kept in people's minds (Tulving, 1972;Wegner et al., 1985).Both types of memories aid in building unique personal and social experiences, which are closely related to the practice of storytelling.
Storytelling was originally seen mostly as an oral practice and now has expanded beyond the spoken word into digital multimodal experiences.This allows people to connect with interactive narratives such as games, apps and even films in a similar manner as they do physically with others and the real world.This research considers interactive narrative as a form of digital interactive experience in which users create or influence a storyline through actions (Riedl and Bulitko, 2012).
Interaction, especially Natural, Reality-based and Tangible Interaction is particularly relevant to this research as it seeks to explore ways in which physical manipulation can foster user engagement.Valli (2008) discussed the relationships between people, spaces, and objects enhanced by technology, and frames them in the context of Natural Interaction.According to him, technology should become invisible and offer nearly real interfaces that do not rely on metaphors but the simulation of reality.This is particularly relevant in regards to communicating naturally with the machine by using natural gestures, full body movements or voice control in the same ways humans communicate among themselves.His vision of Natural Interaction calls for the reduction of cognitive load on users, spontaneous interaction (like in real life) and diegetic purpose of the interactive objects.On the other hand, Jacob et al. (2008) discussed a possible way to unify emerging interaction styles such as Virtual Reality, physical manipulation, and mobile devices by building upon the notion of the user's pre-existing knowledge of the real, everyday world.This approach, defined as Reality-based Interaction, presents four themes: naïve physics, the environment, the body, as well as interacting with other people, each one playing a different part in new modes of interaction.
In another thread, it was the work of Ullmer and Ishii (2000) that brought together the concept of Tangible Interaction.They introduced a framework for tangible interfaces that describe the relationships between physical artefacts, representations, and control of digital content.According to them, tangible interfaces explores the gap between input and output, where physical artefacts represent mechanisms for interactive control coupled to digital representations.In parallel, Fishkin (2004) proposed a taxonomy for the tangible interaction paradigm, in which tangibility is a spectrum that describes the way an object is manipulated in terms of embodiment and metaphor.In the case of embodiment, this axis of the spectrum describes how the cognitive distance between the input and the output increases, like in a remote control where the output is somewhere else (in the T.V), or decreases when the output is the input itself like an Etch-a-Sketch drawing toy.The metaphor axis, on the other hand, describes how the actions of the user are analogous to the real world.They can range from no metaphors like in a command line interface, where there is no correlation between the gesture and the outcome, to full metaphor where the virtual system is like the physical system.The Recompose system developed by the Tangible Media Group at MIT is a good example of a full metaphor, as the actuated interface mimics the movement made through motion sensors (Blackshaw et al., 2011).
Extending the previous work on tangible interaction, Shaer et al. (2004) proposed the TAC paradigm based on the physical qualities of the manipulable object as an answer to different conceptual, methodological, and technical challenges identified in tangible interfaces.This model seeks to describe the objects' functionality and structure, as well as their properties.In a similar manner, Ullmer et al. (2005) introduced the Tokens + Constraints system for Tangible Interaction.In this system, physical representations (tokens) are linked to digital content and mapped to an operation.This link can limit the token's physical manipulation (constraint).For instance, in an abacus, although it does not provide any digital content, each bead is a token that represents one unit, while the rods physically limit the manipulation of the bead representing the place value of a number.Other authors have also proposed taxonomies and categories that describe abstract aspects of tangible systems such as Underkoffler & Ishii (1999), practical design considerations such as Van den Hoven et al. (2013), and guidelines that inform mostly technical aspects of implementing a tangible system such as Nunes et al. (2015).In regards to this diversity of research foci in tangible interaction, Mazalek & Van den Hoven (2009), along with Riedenklau (2016) and Angelini (2017) in their Ph.D. dissertations, provide extensive compilations and reviews of multiple tangible interaction frameworks.Nonetheless, among the vast amount of published work in the field in the past 20 years, the work of Holmquist (2000), Mazalek et al. (2002), Tanenbaum et al. (2010), Tek-Jin & Kim (2011) and more recently Harley (2016;2017) are of special interest to this research as they all consider the component of narrative within tangible interaction.Holmquist et al. (2000) discussed the idea of sense of involvement based on computational objects that represent important parts of an interactive narrative.In this perspective, they explored ways to move away exclusively from first-person views by exploring other aspects of the narrative events by using physical objects that embody meaning to the story and situate the user inside it.These diegetic objects heighten the level of involvement with the story and create better ways to foster engagement.Following a similar approach, Mazalek et al. (2002) discussed a method that considers cooperative and social interactive experiences as viewpoints that allow the user to interact and modify a linear narrative in a shared space.According to them, collaborative systems bridge the gap between the physical and digital worlds by allowing the user to engage directly with the narrative.
Using a tangible approach, Tanenbaum et al. (2010) presented the Reading Glove, an interactive system that makes use of RFID-enabled gloves to bound a digital narrative to physical objects as cognitive hyperlinks.According to them, these hyperlinks (either explicit and implicit) are narrative elements such as themes, characters, and locations that help make sense of a story.This is important in the context of non-linear narratives as they situate the user in the story.On the other hand, Tek-Jin & Kim (2011) discussed how imaginary creatures that "reside" inside pillows, can foster physical interaction by providing serendipitous functions in the form of surprises that the user discovers through manipulation.In their research, they concluded that the creation of stories and meaningful narratives allow tangible interaction to provide the artefact with ludic value and invite the user to engage.
Finally, it is the work of Harley et al. (2016) the one that presented a link between tangible interaction and narratives based in part on the work of Ryan (2002).This framework provides a strong narratological foundation and support considering ways in which tangible interaction can enhance a narrative, which is part of the theoretical basis on which this doctoral research will build.According to the authors, the lack of consistency in the way a narrative is described is seen as an opportunity to consider the narrative as the primary element of the interactive experience.This framework identifies seven categories of narrative possibilities, of which the user, the media, the objects, and the narrative itself reflect how the narrative is constructed and expressed.A later work by Harley et al. (2017) extends from this framework and presents a system that considers diegetic tangible objects in VR narratives.In their discussion, they argue that meaningful interaction should be significant to the narrative and that diegetic tangibles-those that exist in space and time of the narrative-can leverage user engagement and communicate information about the story world as well as the elements that are part of it.

RESEARCH QUESTIONS AND METHODOLOGY
Considering the above frameworks and examples, and with a particular interest in multimodality and tangible interaction, the main research question is how post-WIMP 1 methods of interaction can be applied to a model intended for memory-based interactive narratives.Although novel technologies such as smart assistants could be included into these new methods, they fall outside the scope of this research where the focus is on physical, meaningful manipulation of objects that are created with new materials using paper-based computing mechanisms.
Possible answers to the main question will be explored from the paradigm of Research through Design (RtD), as it builds knowledge through the iterative process of design (Findeli, 1998;Frayling, 1993).In the case of this research, the generated artefact has a dual purpose: a means and a context to conduct specific research in the field of tangible interaction design.Additionally, it will contribute to game theorists and game developers and designers involved with the creation of playable stories from the theoretical perspective of Activity Theory.In regards to this, Kuutti (2009) provides a connection between Research through Design and the Activity Theory framework which places the artefact in its theoretical centre.Activity Theory allows integrating complex aspects inherent to interaction design and interactive narratives such as non-linearity and multiplicity into the core model of tools (conceptual and physical), actors/subjects (the player or reader for example) and objects (as the purpose leading to an outcome).
The research will be conducted in two phases driven by four secondary questions (SQ).It will follow an inquiry-based macro-model grounded on the design process itself-namely, analysis, projection, and synthesis (Jonas, 2006), as illustrated in Figure 1.Phase 1 will look at (SQ1) how people tell stories through the use of objects and (SQ2) to what extent objects can support and aid the process of engaging with a narrative.This will be approached from three research activities.First, a literature review that examines frameworks that describe and outline tangible interaction, as well as an examination of other relevant areas such as multimodality, narratology, phenomenology and cognitive psychology.Second, this review will be supported by a series of case studies that will instantiate various connections between tangible interaction and interactive narratives.Lastly, this phase will conclude with a participatory study that seeks to describe the relationship between physical manipulation and storytelling with a tentative set of parameters derived from the previous two activities.In this study, participants will be invited to interact with a storytelling environment with different experimental settings and comment on their experiences.The study goal is to observe participants' interaction behaviours and understand their cognitive and emotional experiences.
Phase 2 will focus primarily on creative research that will explore the creation of an artefact based on the mechanics of paper engineering enhanced with digital components.This exploration will be made in the context of interactive narratives, specifically as a playable story (Ryan, 2009).This creative process aims to find possible answers to (SQ3) the ways physical manipulation and technology can trigger and sustain interactive narratives and (SQ4) how tangible interaction can impact the level of engagement with interactive narratives.Each iteration of the artefact will be treated as a single case and evaluated in a series of participatory studies.

CURRENT STATUS AND INITIAL RESULTS
At the beginning of this research, the focus was to reframe the main question as the scope of it was too wide and vague.This led to the creation of the subset of secondary questions presented in the previous section.At the same time, the literature review concentrated on gaining insights on various interaction modes and later moved into the theories behind storytelling.As the research became more defined, the priority shifted into identifying, discussing and outlining the right methods and methodology.In parallel to the literature review, creative research has focused on exploring paper mechanics and material experimentation, recorded as different entries in a design journal.More specifically, the journal intends to capture sketches made during the ideation process for possible designs, as well as annotated observations of the experiments made with different materials such as thermochromic ink, conductive paint, and other paper mechanisms (Figure 2).The primary role of the design journal is to be used as a data collection tool in which insights and decision-making accounts are registered, in the form of a reflective conversation on the design process (Pedgley, 2007).Among the experiments recorded in the journal is a system of paper interfaces that combine conductive ink, copper tape, and paper mechanisms such as floating layers, pull strips and trellises, which are connected to an Arduino microcontroller that acts as a distance sensor, a potentiometer, a switch, and a speaker (Figure 3).

Initial Results
The first part of Phase 1's participatory study was completed recently, 8 months into the doctoral study.It sought to relate the hedonic qualities of user engagement with existing playable stories, which could inform the design criteria that will be set for the research's own playable story.Eighteen participants were invited to view/play and discuss their individual experiences with different interactive adaptations of Lewis Carrol's Alice in Wonderland.Among the eighteen participants, eleven were Ph.D. students, four undergraduates, and three academic staff at the Hong Kong Polytechnic University.Only one was a native English speaker, while the rest were non-native speakers from Europe, South East Asia and mainland China.Ten were women; eight were men, all between 20 to 35 years old.The content provided to the participants was in English, and they were required to express their experience in this same language.Each adaptation characterized a different medium of interaction: the hypertext medium was represented by Matthias Conrady's Allice Falling, the touch-based medium by Emmanuel Paletz's electronic book The Alice App for iOS devices, and the naturally tangible medium by Robert Sabuda's movable book Alice's Adventures in Wonderland (Figure 4 and 5).Correspondingly, these mediums are examples of the Model-Control-View (MVC), the Natural Interaction model and the Model-Control-Representation (MCRit) as described initially in the background section (Burbeck, 1992;Ullmer & Ishii, 2000;Valli, 2008).
This study involved measuring the participant's engagement by means of O'Brien et al.'s User Engagement Survey (UES) -a self-report instrument that defines the prominence of four unique factors to the process of engagement -each factor measured in a 5-point Likert scale, as well as assessing their language proficiency, both as mediators of the experience (O'Brien et al., 2018).For the latter, the Common European Framework (CEFR) can-do Statements was used to categorize the participants into three groups that describe their abilities concerning reading, listening, speaking and writing (Verhelst et al., 2009).
Each participant was asked to read and play with the first chapter of Alice in Wonderland; Down the Rabbit Hole, which appeared in all adaptations of the story.Although the textual details differed on each adaptation, the narrative covered the same order of events and characters.As they played with the story, they were able to interact or manipulate different elements such as hyperlinks, virtual objects or moving paper mechanisms.Once the participants finished reading the section assigned,  they were asked to complete a modified version of the User Engagement Survey (UES) followed by a semi-structured interview on their experience that focused specifically in their emotions while playing with the story.On average, each participant spent 8 minutes reading and playing, 3 minutes completing the UES survey, while each semistructured interview lasted about 10 minutes.
The outcomes of the study informed on how certain hedonic emotions, including fun, attraction, excitement, satisfaction and frustration, shape the interactive experience with the three types of stories, and illustrated the values associated to each medium.It also demonstrated possible relationships between physical manipulation and storytelling.The movable book as a naturally tangible narrative, for example, received the highest user engagement score, possibly due to the its unfolding visual spectacle, ability to be physically manipulated, and narrative objects that can be interacted with.
It is worth noting that participants who ranked themselves with a superior language proficiency scored higher in the Reward factor of the User Engagement Survey, while the intermediate participants scored higher in the Focused Attention factor (Figure 6).This difference could be due to Intermediate participants making a substantial effort to interpret the text as they were purposefully reading while interacting, whereas, for the Proficient participants, their effort was less, allowing them to enjoy better the viewing/playing experience.The results also seem to indicate that both language proficiency and the ability to interact directly through physical manipulation of certain narrative elements has indeed a positive effect on user engagement.
The observations made and the comments gathered during the semi-structured interviews provide a more pertinent description of the differences on how people engage with each narrative medium.For example, in the hypertext adaptation, the ambiguity-caused by the fragmented plot, the navigator position of the narrative, and certain interactive narrative elements-originated the sense of fun in the participants.In the touch-based adaptation, its manipulative nature, along with the different dynamics embedded in the story-in particular elements like animated graphics, sound, or even a branching in the narrative, stimulated their curiosity and attracted the participants.Meanwhile, in the naturally tangible narrative, attraction was reinforced by the ability to play and manipulate varied parts of the movable book, which has a diegetic effect in the story.Satisfaction-as pleasure or contentment-came from the cognitive stimuli of manipulating the paper mechanisms, fostering imagination and curiosity among the participants.Nonetheless, compared to the other two mediums, this was described from an emotional standpoint as the participants felt happy and involved with the story and the characters, which in turn developed a sense of immersion as the narrative progressed.
Overall, the initial findings made in the study were worthwhile, although limited to the forms of the chosen story adaptations evaluated.Currently the data analysis is reaching the end, after which the findings will be drawn more systematically.Then the insights generated from the findings will be used to design the next stage of the participatory study.

POTENTIAL CONTRIBUTIONS
As noted by Mazalek and Van den Hoven (2009): few researchers in the field have taken their work forward and beyond the publication and into their own work.This presents an opportunity to build theory based on existing concepts and to generate knowledge on different aspects from the perspective of design (Figure 7).
Additionally, few works have established a relationship between interactive narratives, play, and tangible design, while the ones that do, do it from the perspective of early education, serious gaming, or learning through play.In particular, very few works address playable stories as the evolution of a story world that leads to aesthetic pleasure and free play.This presents an interesting opportunity to contribute to theory on interactive narratives (as playable stories) from the standpoint of the Activity Theory framework.This framework focuses on describing the relationships of purposeful interaction between the player/reader (subject) and the narrative (object) mediated by the material aspects of the story (tools).
On the other hand, there is a trend where researchers explore new materials and technologies and blend them into day-to-day objects.Even though there are contributions in the aspect of storytelling, most of them are limited to short narratives or experimental work to validate technology with no storyline, leaving behind the chance to involve non-linear narratives and richer, complex interactive experiences (Delle Monache et al., 2012;Qi & Buechley, 2010).This motivate this research to discover more about how touching and interacting with paper-based mechanisms can trigger and support engagement in the following years of this research project.

Figure 1 :
Figure 1: Research structure and the relationship between the methods, activities, and outcomes.

Figure 2 :
Figure 2: Entry to Design Journal

Figure 6 .
Figure 6.User Engagement score mediated by interactive medium and language proficiency

Figure 7 .
Figure 7. Areas of potential contributions to the research.