Mixed Reality Collaborative Storytelling

This paper describes a collaborative digital storytelling environment for the navigation and visualization of location-based historical information. The environment combines several technologies including data mining, information visualisation and augmented reality. A number of challenges are involved in this context, such as selection from large volumes of data and data visualization of relevant information through to creating a feelings of immersion and presence in the story. The approach encourages users to tell a story to one another through making them put the elements of the story into the correct order. It uses a combination of a mixed reality HMD and a tablet. The results point to the approach encouraging a sense of flow. Spatial Mapping


INTRODUCTION
Information technologies are increasingly being used for exploring historical content and enriching it with semantic information, cultural objects and images [1].The Locale project aims to encourage people to share their stories and experiences with others.At the same time, it encourages people to learn more about the period after WW2.As an overall contribution, Locale proposes a Mixed Reality Environment for storytelling based on the close combination of content processing and augmented reality techniques.This paper shows how the approach is used to build a rich realistic data driven storytelling scenario.It then evaluates the support provided by mixed reality and game-like features to increase the sense of immersion, ease the navigation within a range of stories, and increase the potential of interaction about location-based historical information.

Data mining & Visualization
Despite significant improvements in online and mobile search and retrieval processes, navigating complex information spaces remains a challenge [2].In order to manage the knowledge in a storytelling environment we need first to search for specific records in large volumes of data or miscellaneous references about particular places (content selection), then to extract useful information for selective actions to be performed (preparation and exploitation).In combination with data analytics methods, visualization supports the user in understanding the flow of geo-located events [3].
Linking information between content and users allows us to see how close other users and content are to in their perspectives, or to filter out incorrect or irrelevant information or sources, and constitutes therefore a ground for interacting about stories.Based on this, the underlying rationale is that data must be dynamically organized, transformed and presented according to the meaning we want to extract from it.In the Locale storytelling environment, this approach contributes to two main objectives: (1) search stories based on different criteria e.g. by location, keywords, characters stories or timeline, and (2) explore and find stories based on the location of the user.

User Experience
One of the key objectives of the work presented later is to immerse users within the digital storytelling experience.For this two primary concepts are used, these are flow [4] and immersion [5].Sense of Flow is where an individual feels fully immersed in an experience so that they lose sense of time and space -they are focussed on this activity.Such activities can include computer games, sport or reading a book.Immersion in our work draws on the area of presence research which has previously been explored with respect to location based digital story telling [6].Another area is gamification which is the application of game-like elements to nongaming situations [7], for example to assist in training, learning or in this case storytelling/creation.The example presented here only adopts limited gamification features, as such we do not regard it as a fully gamification application.

Digital Storytelling
Digital storytelling is a form of storytelling that combines various media content and technology such as images, video and virtual words or real-time data.These multimedia elements are blended together using computer software, to tell a story that usually revolves around a specific theme or topic and often contains a particular point of view.Currently there are several languages and tools that allow the creation of digital storytelling.Digital storytelling is also available through virtual reality (see [8], [9], [10]), augmented reality (see [11]) and mobile phones ( [6]).

Background
This research project started with an investigation into the storage, processing, mining and editing of historical heritage content.An advanced data mining back-end system for authoring and sharing multimedia content have been developed.Mixed reality was added to improve the level of engagement with the content and to explore new ways of collaboration.

Case Study
A collaborative digital storytelling application was developed for two users.One (Ut) uses an application on a Tablet and another one (Uh) uses an application on the Microsoft HoloLens headset.This approach draws on work undertaken in [12], which explored the use of collaborative augmented reality games.The story consists of data (test, images, videos and 3D holograms) which are stored in the Locale database.The collaboration process consists of three parts: (i) The first activity is to collaborate to place three holograms of buildings (of the selected scenario) on a virtual map.Ut has detailed information about the position of the buildings and Uh has the virtual map and the holograms of the buildings that should be placed (see Figure 1).(ii) As a second activity, Uh can freely and autonomously interact with the holograms of the buildings: Listen to a description, watch a current picture or read a story of the buildings he/she find interesting.(iii) The third and final activity is to collaborate to collect some parts of a story based on clues.Several holograms of scrolls are available in the environment, each one contains part of a story.Ut have a list of clues and should collaborate with Uh to identify the right parts of the story and to sort them in the correct order.
The approach used combines elements of conventional digital storytelling with collaborative features that encourages users to collaboratively create a story by assembling it together from components (available from within the database).
Some data are collected during the experience, such as the selected scenario, the most interesting building, the type of interaction preferred (audio, photo or text), some additional interaction with other holograms in the context (i.e.: a box with inside a clue).Finally, the application keeps a record as to whether the user assembled the story in the correct order.Basic game-like elements are also available, such as achievements and score.For example, a user can obtain 5 points after listening to a description of Hotel de Ville building, located in the City Hall of Luxembourg, or by reading a story that happened in that location in the period after WW2.Furthermore, the user can obtain an achievement by exploring the surrounding space and interacting with each hologram on the virtual map.For each building, the users have the choice to interact with three types of content related to buildings: listen to some information about the building, look at a picture, or read a story.This is to support the existence of multiple intelligences [13], that is to say to favour equally people that have for example a strong musical, linguistic or visual-spatial intelligence.

USER STUDY
A user study of was carried out with twenty users from our institute who signed ethics form.The study complied with standard ethical procedures.The sample population was composed by people with ages ranging from 20 to 60.The study consisted of three phases and the experience lasted around 30 minutes.During the introductory phase, participants were presented with the informed consent form by the staff and given enough time to ask questions and fill out the Immersive Tendencies Questionnaire (ITQ) form.In addition, participants were asked about their age and gender.Users received also an overview of the study.During the secondary phase, users received basic training in the use of the devices and then they were asked to use the system.They were then asked to fill out the System Usability Scale (SUS) and Flow State Scale (FSS) questionnaire and then the debriefing questionnaire.The questionnaires were identical to those developed by Witmer and Singer [5], Jackson [14] and Brooke [15].System Usability Scale (SUS) -is a scale used for assess system usability.SUS scores have a range of 0 to 100.The mean score for the 10 users with the HoloLens is 70.5 (n=10), with a maximum of 85 and minimum of 37.5.The mean score for the tablet was 71.25 (n=10), with a maximum score of 85 and minimum score of 52.5.Eight results out of ten of the user with the HoloLens has a high SUS score (result > 50), two were quite low (result < 50).This means that most of the people that tried the application were satisfied about usability, but some improvements must be done in order to increase it, as shown by the two low scores.Users of the tablet, were similar with nine results out of ten being high, and only one quite low.

Flow Scale State (FSS)
-the flow scale is a fivepoint scale format questionnaire used to assess sense of flow.As documented in [11], the FSS considers factors such as Autotelic Experience (ENJY), Clear Goals (GOAL), challenge-skill balance (CHAL), concentration on a task (CONC), paradox of control (CONT), unambiguous feedback (FDBK), action-awareness merging (ACT), transformation of time (TRAN), loss of selfconsciousness (LOSS).The results of the FSS Table 2.For each question, the users have the choice to select an answer on a scale from 1 to 5: Strongly Disagree = 1, Strongly Agree = 5.The mean values are: 3.35 for questions with sub-scale ENJY; 3.56 for questions with sub-scale GOAL; 3.83 for questions with sub-scale CHAL; 3.25 for questions with sub-scale CONC; 3.68 for questions with sub-scale CONT; 3.85 for questions with subscale FDBK; 3.68 for questions with sub-scale ACT; 3.88 for questions with sub-scale LOSS.A higher score of the FSS means a higher level of sense of flow.The maximum score reached is 4.25 (feel highly immersed in the experience) and the minimum 2.25.

CONCLUSIONS
This paper has presented a novel approach which combines the use of data mining and visualization to support non-linear storytelling.It extends this through the use of mixed reality to increase the sense of flow and immersion in the story.It uses a collaborative approach to encourage people to "create" the story from components together.With this in mind the SUS score points to the system (both the tablet and HoloLens) being usable.Furthermore, the study participants all indicated they had a moderate chance to feel immersed in different media.The generally positive scores for the flow questionnaire (all >3) point the approach being useful in creating a sense of flow and with that a state of immersion in the story.
While improvements can be made to the system the results indicate that using collaboration as a means to share a story is useful in creating a sense of flow.It also points to the value of using mixed reality and different devices (e.g. a tablet and Hololens) to aid in creating a sense of flow in such experiences.The results also point to the value of using problem solving as a way to share story content.

Figure 1 :
Figure 1: Mixed Reality Scenario (path of three places)

Table 1 :
ITQ Mean of each question for 20 participants

Table 2 :
FSS Mean of each question for 20 participants.