Towards an Interaction Blueprint for Mixed Reality Experiences in GLAM Spaces: The Augmented Telegrapher at Porthcurno Museum

Emerging mixed reality (MR) technologies, such as Microsoft HoloLens, present many exciting opportunities to the heritage sector. For example, the development of immersive experiences within gallery, library, archive, and museum (GLAM) spaces. Previous work on virtual and augmented reality (AR/VR) in such spaces is often limited to meta-layers of information beyond traditional modes. However, the affordances of MR potentially offer new ways to design content beyond AR/VR that can engage users as performers upon the stage of historically resonant and artistically charged places. This work-in-progress paper examines the case of the Augmented Telegrapher under development at Porthcurno Museum. A pilot study with 30 participants examined how to situate the Augmented Telegrapher into the context of a multi-player game, with particular attention given to which interaction style is more suitable: the use of sensors and gestures to manipulate virtual objects, or a tangible user interface grounded by relevant physical objects. Experimentation reveals a statistically significant difference in time taken to complete a data entry task (d = 1.32, p < .01) and data entry errors (d = 1.24, p < .01). This highlights several challenges in the manipulation of virtual objects, most notably the difficulty of manipulating virtual objects using unnatural maps of artificial gestures to intentions. This work-in-progress paper briefly outlines future directions in response to these challenges, paving the way towards an interaction design blueprint for mixed-reality experiences in GLAM spaces.


INTRODUCTION
A mixed reality (MR) experience, as defined and illustrated by Milgram and Kishino [1994], is an experience that combines the real and the virtual.Such combinations of the real and the virtual will normally span different parts of a "continuum of virtuality", whether these are real spaces that have been augmented with virtual assets (augmented reality), or virtual spaces that have been augmented with real-world artefacts (augmented virtuality).Typically such augmentation is supported by a "system that has the following three characteristics: combines real and virtual; is interactive in real time; is registered in three dimensions" [Azuma, 1997, p 356].However, their applications are typified by their ability to facilitate immersion [Sanna & Manuri, 2016].
The ability to facilitate immersion through the use of mixed reality technologies presents a compelling opportunity to the heritage sector.Curators often want to provide visitors to gallery, library, archive, and museum (GLAM) spaces with a new way to engage with their collections.Such technologies, therefore, could enrich these spaces.
Consider, for example, the dimensions of the experience economy: educational; "esthetic"; escapist; and entertainment [Pine & Gilmore, 1999].Many GLAM spaces already target the educational dimension, typified by absorption rathter than immersion, and target the esthetic dimension, typified by passive participation rather than active participation.However, few target the escapist dimension.This dimension usually offers ways for people to assume new personae and exercise agency over the outcome of an activity.
A key challenge, however, is offering the right kinds of interaction to help visitors switch between different modes of exploration as they navigate the artefacts in a GLAM space [Liarokapis et al, 2017].Bekele et al [2018]  This work-in-progress paper describes part of a case study striving to address some of these interaction design challenges: the 'Augmented Telegrapher' at Porthcurno Telegraph Museum.
The overall aim of the project is to construct a blueprint that can be followed by other GLAM space curators as a means of diversifying their potential audiences through the offer of "escapist"type experiences, thus placing visitors in role of performer upon the stage of historically resonant GLAM spaces.

BACKGROUND & RELATED WORK
The reality-virtuality continuum presents a vast landscape of opportunities in cultural computing research.A recent survey shows that mixed reality received more attention by cultural heritage researchers in 2004 -2016 [Bekele et al., 2018].Despite this, the authors acknowledge that further work is needed.Most relevant to this research, the authors identify the needs for more research into Tangible AR.They suggest: "A number of augmented reality applications use tangible interfaces in a much narrower scope than its potential warrants.We hope to see more research that integrates Tangible User Interfaces and augmented reality so that future applications, irrespective of domain, will be able to augment physical objects with virtual content and enable interaction with this content through the augmented objects."[p. 29] User input for mixed reality systems can be reduced down to six distinct modes: tangible, collaborative; device-based; sensor-based; hybrid, and multimodal interfaces [Bekele et al., 2018].The mixed reality toolkit supplied as part of the development tools for the Microsoft Hololens already has affordances for natural UI modes such as speech, gaze, and gesture sensors, with gestures seemingly the preferred method of interacting with holograms.Gesture input for the Hololens has issues.A study by Pollalis et al. [2017] suggests that users had difficulties performing Hololens specific gestures and that it is advisable to have some type of training to onboard users.Zimmer et al. [2018] corroborate this and suggest that users should be given time to learn how to use the Hololens before they are asked to perform more complex tasks.They also acknowledge that prolonged gesturing can lead to fatigue.Another limiting factor of the Hololens when it comes to user input is the limited 35° degree field of view (FOV).

PROTOTYPES
The Augmented Telegrapher is a mixed-reality experience developed for Porthcurno Telegraph Museum.Users participate in an escape the room style puzzle game where the Microsoft HoloLens augments an underground WW2 bunker tunnel (in which the museum is situated) to aid the user in completing the game.This involves sending messages in Morse code using Morse keys as shown in Figure 1.

Figure 1: MR Enhanced Tangible Interface
A system formed around a dichotomic (binary) tree enables novices, without any prior knowledge or experience, to leverage Morse code in their solutions.As shown in Figure 2, each node in the tree represents a character.Each edge represents a choice of dot or dash that maps to left and right respectively.This system situates the most commonly used characters at the top of the tree, with those used less frequently at the bottom.

Figure 2: The Dichotomic Morse Code Chart
The game presents a dichotomic tree to them as a hologram two metres from where they are situated and they must traverse the tree to enter the message.The input and traversal of the tree that is the focus of this study.
Two modes of user input are compared.The first input method utilised the native gesture recognition of the Hololens, more specifically the air tap gesture, to select holograms in the augmented environment.The second input method uses a tangible interface built using an original WW2 Morse key connected.The key is connected to the Hololens over Wi-Fi and transmits dots and dashes depending on the duration of the key press. 3 At this stage of the project, two prototypes of the Augmented Telegrapher have been implemented.
Both leverage Microsoft's Hololens.However, one version places emphasis on interaction with virtual objects using gestures, whilst the other version places emphasis on interaction with tangible interface elements that communicate (via network) with the headset.It is not the intention of the experiment to pit one against the other, as the ultimate solution will likely leverage elements of the physical and the virtual.However, comparing the two aids in understanding the relative differences and constraints conferred by the two interaction styles.Thus, illustrating what types of activity might be suitable for each style.

METHODOLOGY
To examine the difference between the tactile interface and the gesture-based interface, a smallscale pilot experiment was conducted using two different prototypes of the Augmented Telegrapher.
The experiment follows the 'reversal' design [Mahoney 1978].Participants were invited to use both of the interfaces to complete a task that was designed as just a part of the overall experience.This task was the use of the Morse key to send a message using Morse code.
The research team randomly allocated participants to either the tactile interface or the gesture based first.However, each participant ultimately used both interfaces, such that they completed the task twice, each time using a different interface.and involved a practice exercise.In all cases, they were tasked with sending 'AUGMENT' in Morse code over the Augmented Telegrapher.Though, only after a short period of practice.
A power analysis conducted using G*Power, assuming a medium-to-large effect size [Cohen, 1992], suggested a sample size of 30.A sample of 30 participants was obtained.For the purpose of this initial pilot, a convenience sample of undergraduate students at Falmouth University was drawn from a balanced range of design, computing, and art courses.Only five of the testers identified as female and approximately 66% of participants had never used the HoloLens.None of the participants were competent telegraphers, with each having a limited practical knowledge of morse keys and morse code.
The experimental protocol involved the collection of both qualitative and quantitative data.Recordings were taken, both within the headset and external to the headset, and elementary analytics drawn from the trial.Most prominently, these were time taken (up to a 250 second time limit), and the message itself, which was used to count the number of data entry errors.Participants were also encouraged to 'think aloud' throughout the experiment, which drove an interview prior to and after each experimental condition.This enabled the research team to, firstly, gauge each participant's experience with technology, familiarity with tracing binary trees, and prior experience with AR/VR/MR, whilst secondly, to gauge their overall impression of each experience, and determine whether they developed any preferences.

PILOT STUDY RESULTS
Quantitative data were analysed in SPSS by means of a paired t-test, comparing both conditions that the participants experienced.The results, shown above in Table 1, illustrate statistically significant differences on both of the variables of interest that had been captured during the pilot: time taken; and data entry errors.
In both cases, the tangible interface orientated around using a physical Morse key proved preferable to the gesture-based interface oriented around a virtual Morse key.The mean time required to complete the task (including those running out of time at 250 seconds) was 135 seconds (σ = 64.7)for the gesture-based interface and 92 seconds (σ = 43.1)for the tangible interface (p = .001,d = 1.32).The mean number of data entry errors was 6.97 (σ = 4.0) for gesture-based interface and 4.1 (σ = 3.1) for the tangible interface (p = .002,d = 1.24).Only two of the 30 participants completed the task without any errors.Most incurred at least one data entry error regardless of using a physical or virtual Morse key.The authors assessed the qualitative data using inductive coding and thematic analysis.The corpus of comments drawn from the post-experiment statements were fed through VosViewer to identify the key codes.This tool visualises qualitative data, producing a heatmap that illustrates frequency of term by colour and relatedness of terms by proximity.This provides insight into clusters of related terms.These are illustrated in Figure 3.
Four distinct clusters can be observed in the visualisation relating to gestures (Figure 3, left).
The first relates to issues with the field of view, and the difficulties participants encountered seeing the virtual objects at the same time as the binary-tree visualisation.The second corresponds to the gulf of execution and gulf of interaction [see Hutchins, Holland, and Norman, 1985] due to lag in responding to gesture input.The third corresponds to confusion regarding the gaze being the cursor rather than the hand.The fourth corresponds to the mapping of the gestures that the HoloLens could detect to the action of selecting a dot or a dash on the virtual Morse key.
In contrast, the visualisation relating to the tangible interface (Figure 3, right) only illustrates a single cluster; although, some of the terms are distantly related.Unlike the virtual key, where users selected a dot or dash, players operated the Morse key just as a telegrapher would have normally.There was, however, inconsistency in players' understanding of the length of a 'dash' and the minimum gap needed between each letter.As such, participants often made mistakes in timing which lead to them selecting the wrong character accidentally.
Overall, the physical Morse key received quite positive comments from the participants such as "the [physical] key feels nicer and more responsive" [19, M] and "the click is satisfying" [16, M].Others who used the physical key second said they were "much more confident approaching this task" due to it being "a lot simpler to understand with only one movement" [8, F].Some suggested that the interface could offer "better visual cues to show timings" [4, M].Particularly during the practice period before sending specific messages to help prime players to avoid timing-related mistakes.

LESSONS LEARNED
The pilot study presented in this paper compares a tangible and gesture-based interface for the Augmented Telegrapher.This provides some insight and lessons learned about mixed reality interactions in GLAM-spaces: • Using the tangible interface is preferable in contexts that are either time-sensitive or accuracy-sensitive; • Interim feedback, such as visuals cues mediated by the mixed reality, are likely necessary to reduce the gulf of interaction; • and the gesture-based interface might see improvement by carefully positioning visual elements according to field of view, whilst also implementing alternatives to gaze-cursors.
Further work is being conducted to assemble a longer and more cohesive experience.A range of scenarios is being considered, including: the use of a Wheatstone bridge circuit to configure a duplex telegraph cable, sending sophisticated Morse code transmissions, identifying the location of a break in the cable using a galvanometer, and directing a boat to a break in a sea cable.Different interaction styles will be needed to direct participants to different exhibits and through scenarios throughout the space.Exploring these will equip designers with insights that help them craft active and immersive experiences in mixed reality GLAM spaces.

Table 1 :
Results of a Paired T-Test Comparing the Tangible and Gesture-based Interfaces on Time Taken to Complete Task and Data Entry Errors

Figure 3 :
Figure 3: Cluster Visualisation of Qualitative Data Corresponding to the Gesture (left) and Tangible (right) Interface Trials