Blended Experience Narratives

Our work employs conceptual integration, otherwise known as blending, as a tool for designing interactions that work across and between the digital/physical divide. Theoretical approaches to blends can be difficult to navigate and a hard subject to get to grips with, particularly for novice designers. This paper offers a synthesis of the literature around this topic developed into a framework. We use speculative design as a means for iterating solutions that employ physical/digital transitions. Working collaboratively with thirteen undergraduate university students attending a twelve-day Blended Interactions workshop, we developed speculative narrative storyboards over the course of a live theatre festival ultimately presenting proposed solutions as part of the festival itself. Narratives drew on data gathered through participant observation and interviews of stakeholders with data and feedback from over 380 festival visitors. Presenting the storyboards to festival participants provided direct feedback. The student storyboards were professionally illustrated affording us a means for evaluating our framework in further work.


INTRODUCTION
As interaction across the physical and the digital becomes ubiquitous, designers are challenged to conceive methods for working in this arena. This is a particularly challenging paradigm to teach to novice designers. Our work considers blended experiences, focusing on design that integrates the physical and digital into a balanced, unified experience. Blended Experience builds on Fauconnier and Turner's (2008) work on conceptual integration also known as blending theory.

Blended
Experiencedraws from conceptual metaphors proposed by Lakoff and Johnson (1980) and Benyon's (2014) work on blended spaces and blended interactions (O'Neill and Benyon 2015). Our work combines several theoretical approaches (Edmonds et al. 2006, Jetter et al. 2014) that complement interaction design across the physical and digital. Our work extends the notion of blended spaces to that of blended experiences (O'Keefe and Benyon 2015). In this paper we couple blended spaces and conceptual integration (Benyon 2014, Jetter et al. 2012) focusing on and proposing a means for designing for blended experience. Our challenge is not only developing our own understanding of this miscellany of concepts but developing a pedagogic approach.
Our solution was to work collaboratively with students to employ the constructs of Benyon's Blended Spaces Framework as a tool for examining visitor-centred problems during the Edinburgh Fringe Festival. We discovered that many visitors were overwhelmed with choice and wanted an efficient way to find appropriate shows including social activity with other visitors. The majority of promotion was through paper posters and flyers which were found to be inefficient and wasteful.
Our students developed and iterated examples of blended experience through storyboards drawing from the field of design fiction (Flint 2016, Brown et al. 2016) These storyboards are speculative in nature examining potential products and interactions. From this work, we propose a blended experience framework that specifically draws from these speculations enabling a means and method for designing meaningful blended experiences. The resulting experiences support novel interactions with individuals whilst complementing and synchronising with digital content and physical experiences.
There are two objectives for this paper, first to develop and propose reframing Benyon's (2014) Blended Spaces Framework as a Blended Experience Framework, providing an entry level tool that supports designing with and for blended experience. We also present storyboards that constrain design thinking to the Blended Experience Framework constructs and demonstrate a novel means for interrogating this design paradigm.

RELATED WORK
Conceptual Integration was proposed by Fauconnier and Turner (2008) as a the means through which people merge two concepts. One example they use is of a linguistic blend, the word Frenemy. Most people who have English as a first language understand this word the first time they hear it. The apparent simplicity and immediate understanding of this example camouflages the conceptual complexity that drives understanding.
Fauconnier and Turner argue that we take concepts from what they term input spaces. In the case of Frenemy, this is concepts from the input space of friend, someone one thinks of affectionately and seeks out their company and concepts from the input space of the word enemy, a person one dislikes and tries to avoid. Correspondences from these two spaces are projected into a generic space. These projections then create a blend that has features that may not be present in the original concepts. In this case, the blend is frenemy, which can be understood as a person one dislikes but is forced to spend time with. The concept of blending can be a useful tool in interaction design particularly when considering the transition from one mode to another. Schmitz and Quarischy (2009) combine a physical local store (or dorfläden) with interactive technology. Robert et al. (2010) discuss blending realities together in a mixed reality game with robots. Blending has been used in training (Saenz et al. 2015) and as a tool in interactive space (Jetter et al. 2014) Canovas and Manzanares (2014) and Bodker and Clemens (2016) link blends and metaphor. Our work uses storytelling and narrative with a speculative focus (Auger 2013) using blending to explore diverse possible futures through imagined artefacts, stories and worlds (Dunne & Raby 2013)

Blending Theory
Fauconnier and Turner (2008) explain conceptual integration in terms of four constitutive principles.
(i) Composition establishes correspondences between input spaces and brings them together into a blend. (ii) Relations are established within the blend and build on the relationships between the input spaces. (iii) Completion is the process whereby people's cultural and cognitive models are integrated into the blend. (iv) Elaboration is the process whereby the blend is manipulated resulting in new insights.
Blends benefit from the use of 'material anchors' (Hutchins 2005), if one input space is familiar from lived experience rather than understood on an abstract level, its impact is more powerful and improves the blend's efficacy. Hutchins provides a carefully argued contribution to this idea, drawing upon examples of his own work on Micronesian navigators and their use of rising stars and passing islands as material anchors for their approach to seafaring and navigation.
Material anchors aid in binding abstract concepts to reality. We may lay out the ingredients of a meal in a specific order to aid in remembering how to cook the dish and when certain ingredients are added. Manipulating objects in the real world aids memory and cognition. For example, when disassembling an unfamiliar piece of equipment, an engineer may use the space around them to lay the pieces out in the order in which they will be replaced. Hutchins . (ibid p. 1574). tells us: ``Problems that are too complex to hold in the mind as a cultural model, and possibly some that are too complex to express at all in internal conceptual models, can be expressed and manipulated in material structure''.
References to input spaces can be confusing when examining Benyon's work on blended spaces. Fauconnier and Turner understand input spaces to be constituent packets of cognitive understanding that exist within the mind used to navigate life and experience; whereas Benyon specifically discuss the blend of digital information with physical space and the built environment. Benyon's (2014) Blended Spaces Framework applies the conceptual integration concept to the design of mixed reality spaces. Blending of spaces using these constructs results in a new blended space with unique social space, conceptual space and sense of place. 3 Our work on blended experience focuses on the transference of attention from digital to physical and back again. The point at which interaction is most at risk is during the transition from one mode to another. A discussion of transitions and means by which to orchestrate and minimise disruption is discussed in detail by Benford and Giannachi (2011) in their trajectories framework. The trajectories framework focuses on a person's traversal through a narrative driven mixed reality experience. There are transitions between a number of constructs to consider including different aspects of time, e.g times when participants are able to interact, time as it progresses in the real world and the progression of time in a narrative. There are also transitions in roles, e.g. spectator to participant and transitions concerned with physical resources and access to infrastructure. The idealised route through an experience is termed the canonical trajectory whereas the route actually taken is the participant trajectory.

Figure 2: Benyon's Blended Spaces Framework
The design of mixed reality applications was coupled with blending theory by Benyon (2012Benyon ( , 2014Benyon ( , 2019 To guide and support design, he presents blended space as being built of four key constructs: (i) Ontology: These are those Things that make a specific space a place. Things can be physical resources, or conceptual understandings. (ii) Topology: This focuses on the Relationships that can occur between things, people and places. (iii) Agency: The opportunities for People to interact with digital content or objects available in a specific space. (iv) Volatility: How Change in the physical space affects digital content and vice versa, over time.
Considering Blends enables designers to design for interaction between the physical and the digital (O'Keefe and . Bodker and Klokmose (2016) criticise the Blended Spaces framework for taking a simplified view of conceptual integration and blending theory. Conceptual blending is more nuanced than perhaps represented in this framework and context and metaphors change over time. Simplification is to our advantage when the focus of our work is on education, endowing novice designers with effective tools for design. What we aim to encourage is for our designers to maintain a human centred, considerate focus and avoid 'bolting on' technology in a manner that lacks nuance.

Speculative Design
We employ speculation, specifically design fiction in our work. These are imagined futures that can be perceived as inhabiting a space between the distant future and the near present. Speculations extrapolate from existing or possible technologies (Auger 2013) and design fictions often present narratives of these technologies in everyday use. We can learn much from dystopian visions of these products in use (Dalton et al. 2016) however the dominant view of the future in Human Computer Interaction tends to be optimistic (Coulton et al. 2018). Speculation involves careful world building with different entry points afforded by various artefacts (Coulton et al. 2018 Artefacts are offered in a variety of media including film (Flint, 2016, Lindley andCoulton 2014) prose (Blyth and Wright 2006) or as prototypes that can be shown in catalogues with back stories and evaluations (Brown et al. 2016, Søndergaard et al. 2016) Representations and outputs from speculation are broad (Dunne and Raby, 2013) and attempting to mention all types of media would be exhaustive and beyond the scope of this paper.
Our work looks to sequential narratives specifically graphic novels. Our outputs are set in a probable near future exploring technology and human interaction. We focus our designs on real-world settings, investigated in teams. The outputs are storyboards that enable our students to discuss and present possible and plausible interaction (Bodker 1999, Troung et al. 2006) Using storyboards enables one to put imagined products into context as the focus of specific stories (Sturdee et al. 2016) Where our storyboards leave product-design scenarios behind and move into the speculative design arena is how they are part of a wider world beyond the interaction, encompassing people, lives, and an imagined future with diegetic prototypes. The application of these imagined graphical narratives to research can support the adoption of blending by offering an opportunity to explore spaces and lived experiences in an empirical and accessible manner to speculate on interaction in blends and to conduct evaluations around blends. Linking speculation and blends opens novel methodological space. 4

A BLENDED INTERACTIONS WORKSHOP
With the view of exploring and developing a place of practice for designers, we recruited 13 students from our respective institutions to take part in a summer Blended Interactions Workshop. The primary goal of the twelve-day workshop was to apply the Blended Spaces Framework to real-world problems during the 2019 Edinburgh Fringe Festival. The Edinburgh Fringe Festival is the world's largest performing arts festival, featuring over 3,500 shows during the month of August, shows include comedy, theatre, dance, street performances, and more. The festival coupled with our workshop, provided our researchers with hands-on, human-centred challenges when designing emerging user experiences, products and services for numerous festival visitors, performers, and advertisers.
Modelled after an industry-centred interaction design studio, we used human-centred design techniques (Benyon 2019), speculative storyboards (Flint, 2016, Sturdee et al. 2016, and the Blended Spaces Framework as tools to consider how designers produce new digital and physical experiences that work harmoniously, while supporting new interactions and relationships with people The first half of the workshop was devoted to reviewing principles of the Blended Spaces Framework, e.g., ontology, topology, agency and volatility, while employing best-practices in humancentred design research during the festival. After organising our research strategies, we sent out four groups of researchers to various festival locations throughout Edinburgh. We had the opportunity to participate in the festival activities as festival visitors, advertisers, and performers.
These activities provided our research teams with work-sampling and task-analysis opportunities to inform the direction of subsequent surveys and interview questions. For example, students and staff participated as festival visitors by attending festival shows and observed the social spaces after the shows. These experiences were documented with photographs and written short journal entries. We participated as festival advertisers by promoting our own exhibition, as our final workshop outcomes would be on display as a formal Edinburgh Fringe Festival Exhibition.
By promoting our own show on the streets of Edinburgh, we gained a deeper appreciation for the ineffectiveness of the overall event advertising processes, as well as the paper-waste associated with festival marketing. At the conclusion of the workshop, we participated as performers during our exhibition, ``Flyers are Rubbish''. By producing our own show, we were able to report our findings to our users and experience the hectic challenges of what it takes to visit, advertise, and perform.

Festival Research Findings
We collectively captured data from 389 festival visitors (52%), advertiser business (26%) and performers (22%). We also interviewed 57 visitors (46%), advertisers (45%) and performers (9%). We established several festival problem spaces that aligned to our target festival demographics, Institutional research integrity approval was sought and given for data collection.
Our data demonstrated the following assertions.

Performers:
 Rely on street guerrilla-style paper leaflets to promote their performances.  Often do not have concrete nor measurable performance feedback from visitors.  Believe that paper-flyers are the best way to reach their audience. Visitors:  Reject paper flyers because they find them annoying and overwhelming.  Do not trust critics' performance reviews as they often do not know who they are or why their opinion matters.  Would like to socialise and meet new and exciting people around them. Advertisers:  Generate a large quantity of leaflet and poster paper-waste.  Have no accurate method to measure the effectiveness of paper-based advertising.  Have a direct stake in the success of their paper flyer advertisements.
We began to address these problems by coupling speculative story boarding with the Blended Spaces Framework. We set out to explore and speculate how we could:  Identify a more authentic way for visitors to generate performer reviews while democratising the overall performance feedback process.  Identify a fun and social way to streamline how beverages and tickets are used whilst exploring how people congregate around other like-minded people.  Identify sustainable alternatives to paper advertising by employing design strategies for human-to-human interaction.  Consider the use of material anchors in the physical domain of our interactions,

Applying The Blended Spaces Framework
During the second half of the workshop speculative storyboards were developed through an iterative process of critique and feedback with both internal and external faculty reviewers from the local interaction design community. The Blended Spaces Framework was applied to the storyboard process to assess design decisions and iterate concepts. Many new products, services and experiences were proposed; either discarded or improved upon. The storyboard design decision making process was guided by the Blended Spaces Framework.
Ontology is an inventory and summary of things, people and places. For example, show advertisements, paper-flyers, paper-tickets, devices, beverages, and more during the festival. Utilizing ontology in this context provided a general starting point to focus our workshop strategy, tasks, and goals. Beynon's four key constructs applied to the festival translated as: Topology The relationships between people, places, and things. For example, relationships between visitors and paper-based advertisements proved to be limited and wasteful. Topology in this context helped us uncover problems during the festival, such as print-based flyers which are out of touch with festival visitors needs and expectations.
Agency is concerned with people, performers, and digital agents such as messaging and ticketing systems. For example, we observed many visitors to be highly engaged with their mobile devices. Considering agency in this context, provided us with opportunities to consider mobile devices to encourage physical socialization with other visitors.
Volatility is concerned with change and how the different experiences, impact the visitors' sense of place before, during and after an event. Utilizing volatility in context provided opportunities to use change and how it influences preference, interactions, and behaviours.

STORIES OF BLENDED EXPERIENCE
Through the four constructs of the Blended Spaces Framework (ontology, topology, agency, and volatility), seven speculative storyboards were created. From these storyboards three high-fidelity storyboards were professionally illustrated. These three storyboards were felt to best exemplify the notion of Designing with Blends (Manuel and Benyon 2007) The final Stories of Blended Experiences, Laugh Traders, Sustainable Hobnobbing and Fringeship Bracelet examine how the principles of the Blended Spaces Framework are used. Informed by our research investigation and closely considering, Ontology (Things), Topology (Relationships), Agency (People) and Volatility (Change) we illustrate a closely tied balance between digital and physical spaces through our storyboards.

Story 1 Laugh Traders (Figure3)
We introduce Story 1 with Ontology (Things), via festival visitors, poster advertisements and generic poster reviews. On the streets of the festival, visitors Mike and Stewart are sceptical of Comic Jan's poster reviews.
We considered Topology (Relationships) between the audience and the performer. Here, digital interactions are intended to operate in the background leaving the physical space seemingly uninfluenced by technology. Mike and Stewart simply enjoy the show as they should with the technology working in the background.
We considered how Agency (People) is needed to interact providing more authentic reviews for visitors when choosing their next show Laugh Traders has automatically calculated audience reviews.
Laugh Traders has collected laughs during the show and has calculated audience reviews.
We considered how Volatility (Change) can be utilised to provide more authentic reviews for both the audience and performer. Comic Jan can evaluate the highs and lows of her performance through data.
By carefully considering the things (posters and reviews), relationships (visitor and performer), people (interactions between visitor and performer) and change (fluctuations in laughter), we create a closely tied balance between the digital and the physical.
Mike is particularly sceptical of Comic Jan's poster reviews. Stewart and Mike decide to go paperless, by downloading the Laugh Traders app. The application offers unique features, "Be a Festival Reviewer", "Trade Your Laughs for perks and more." At the show, Comic Jan takes centre stage. During the show Stewart chuckles reluctantly, while Mike is laughing hysterically. Shortly after the show, Mike and Stewart feel vibrations in their wrists. Laugh Traders has collected their laughs during the show and has calculated their reviews. Meanwhile, Comic Jan taps on a push notification, "Laugh Traders: Your Public Awaits!". Her Laugh Traders dashboard has calculated all audience laughter. That night Comic Jan revisits her act, making a few changes informed by the audience reaction

Story 2 Sustainable Hobnobbing (Figure 4)
We introduce Story 2 with Ontology (Things); via festival visitors in close quarters, being social with beverages and having fun. Ricardo sees his cup sync to his watch, as the show is soon to begin. We considered how Topology (Relationships) between people, meeting new people and their beverages. The smart pints, acting as material anchors, enable all visitors to quickly identify who is going to which show based on cup colour glow.
We considered how Agency (People) need to interact and socialise in a festival environment. Julia, interested in Ricardo's style, starts a light conversation. The show opens and Ricardo quickly shifts attention to Jimmy, the very friendly event staffer. We considered how Volatility (Change) can be adopted to support a variety of shows without the need for paper tickets. Material anchors (glowing cups) allow staff to manage entry. The glowing glasses change colour admitting other visitors to the appropriately colour coordinated show.
By carefully considering the things (beverages and technology), relationships (strangers with common interests), people (interactions between visitors and event staff) and change of material anchors (colourisation of drinks as entry tickets) through blended experiences, a closely tied balance between digital and physical spaces is created.

Story 3 Fringeship Bracelet (Figure 5)
We introduce Story 3 with Ontology (Things); via festival advertisers bombarding visitors with a gamut of wasteful paper flyers. However, Mauro is about to receive his digital flyer from another visitor who has an extra digital show ticket. We considered Topology (Relationships) between people meeting new people with shared interests. The digital interactions are intended to be visible while remaining discreet. Maya can choose who might want her extra show ticket from common interests.
We considered how Agency (People) selectively socialise in a comfortable public environment. Maya, interested in Mauro's and Erik's profiles, shares her tickets. Meanwhile both Mauro and Erik are given the nature of the material anchor that will help them identify Maya, her glowing yellow smart watch. Within proximity, Maya can use material anchors to verify Mauro with the blue glowing smart watch and Erik with the green glowing smart watch. We considered how Volatility (Change) supports and respects personal decision making. Maya's choice in who she is interested in, can respectfully change from person to person. Maya has the power to continue to share digital tickets with Erik, while discontinuing public engagement with Mauro both physically and digitally.
By carefully considering the things (digital tickets and mobile devices), relationships (meeting new people platonic or romantic), people (the rational and interaction of sharing) and change (empowering change in choices) a closely tied balance between digital and physical spaces is created as a blended experience.

Festival Problems as stories
Story 1 Laugh Traders was borne from the following inferences drawn from data gathered. Many visitors base their choice of show on human-to-human interactions and word of mouth. The visual design of flyers has little influence on drawing attention to a particular show.
Laugh Traders creates an experience that allows comedians to receive real time feedback while democratising authentic audience reviews.
Story 2, Sustainable Hobnobbing, drew from data demonstrating that many people came to the festivals looking to socialise with others. This data point prompted the question whether people were interested in socialising with someone new. Further data points demonstrated that not only were they interested in meeting new people who were around them at the festival but also get to know them and possibly make a connection. Data revealed that though people were keen to meet strangers they rarely acted on this, we sought to develop a solution that could close the gap between those who wanted to meet new people and those who were reticent to initiate contact.
Ricardo buys a novel Smart Pint, which links to his watch. His watch alerts him that it is time for his next show, and he leaves. Julia notices their smart pints are glowing the same colour. They jokingly tap drinks and the smart pints glow brighter, causing Julia to laugh. They engage in conversation and discover they have many tastes in common. At the entrance to the show, everyone has drinks that glow the same colour. Smart pints can be used to gain entry. Ricardo stops to engage in conversation with Jimmy, the door staffer. They swap numbers.
Mauro is trying to avoid people who are handing out paper flyers.
Maya shares her complementary Fringeship Event Tickets to two other international visitors who share her interests. Mauro and Erik receive Free Fringeship Tickets from Maya through their smart watches. Mauro receives an invite to meet up with Maya and Erik in the beer garden across town. Mauro orders drinks for everyone using two complementary beverages and pays for the rest with his bracelet. The strangers meet and enjoy each other's company and they exchange contact details. Manager Ted can now examine novel visitor data produced by the bracelets. Story 3, Fringeship Bracelet drew from data that demonstrated visitors found the quantity of flyers overwhelming and had little or no desire to get more flyers. For visitors, flyers were seen as wasteful and a cause of litter. Conversely, from our data performers perceived flyers as easy and efficient though not specifically cost effective.

DISCUSSION
The first assertion we make is to interpret interaction across the mixed reality continuum within physical space as blended experience. Blended experience adopts the use of input spaces from the original approach but re-frames them as input concepts. We then align these concepts with Benyon's assertion of Ontology, Topology, Agency and Volatility.
We argue the high value of considering blending theory when designing for interaction in and across space. Blending or conceptual integration is a difficult subject to navigate with a large subsection of theoretical approaches attached to it. Not least of this is the original theory's use of the word space meaning a conceptual package and Beynon's adoption of blending for space in terms of the built environment. Through running our workshops, we have developed strategies for using and considering blends as a design tool.
Approaches to design for mixed reality such as Benford and Giannaci's (2011) Trajectory framework are concerned with smooth transitions through differing modes of interaction and between interfaces in the same experience. A blended experience approach considers how to make these transitions seem less intrusive and simply part of the overall experience. The coupling of trajectories and blends are discussed by O' Keefe et al. (2021).

Figure 6: A blended experiences framework
Our built environment changes at a slow pace but the nature of its use and the context in which it is interacted with can change rapidly. An example of this from Edinburgh Fringe is the almost overnight appropriation of university spaces into temporary theatres and performance spaces for the month of August. Digital technology can change rapidly adapting to context and intent. Considering blending for the design of digital experiences in physical space ensures that designers consider the integrated context of the built environment with digital interventions. This alleviates the design of digital interactions that are simply bolted onto environments in a `one size fits all' approach.
Our approach draws from conceptual design, blending theory and concepts of trajectories. These are necessary and appropriate because the nature of many blended experiences is based on interactions with and through digital and physical objects that takes place as people move through physical space. Using our approach coupled with speculative storyboards, provided us with an approach to design for blended experience.
Speculative storyboards are critical to communicate specific blended experience moments and a means for evaluating the relationships between digital and physical spaces. Traditional approaches to design often do not take these new contexts into account, focusing on the medium rather than the interaction and experience. Many systems do not consider how designers produce new digital and physical experiences that work harmoniously, while supporting new interactions and relationships with people. 9 The harmonised balance between people, relationships and technology is vital. Our work on the Edinburgh Fringe Festival illustrated that the last thing a digital experience should do is interfere with the sensitive, and creative use of physical spaces and objects that artists, performers and curators have crafted with their exhibits. Even where an experience is primarily focused on a location, people transition between physical and digital components to appreciate the whole experience

Future Work
Our approach to designing for blended experience provides us with a means for evaluating proposed interactions. Using the speculative storyboards to communicate the proposed interaction allows us to develop a means of interrogating participants framed in the notions of Ontology, Topology, Agency and Volatility. This evaluation not only provides us with a means for interrogating each speculative evaluation in its own right; but also, a formal means for evaluating the framework itself.
We have developed a practical approach for designing blended experiences, by adopting and adapting the Blended Spaces Framework This approach may be further evaluated by utilising it within student coursework assessments within higher education institutions, and further, with new user studies aimed at gathering the opinion of a wider public. We are now able to begin the process of identifying the tightness or threshold of a blended experience through our evaluations. We argue for the reinterpretation of Benyon's Blended Spaces Framework as a Blended Experiences Framework (figure 6) providing an accessible and well-grounded approach to interaction design. The framework is usable by students and novices, as well as experienced designers developing novel interactive experiences in a contemporary context.

6.CONCLUSION
We argue that Benyon's Blended Spaces Framework be reframed as a Blended Experiences Framework. This is an accessible adaptation of Benyon's discussion of Blended Spaces, affording understanding complexities of blending in real world situations. Combined with storyboards, this is a useful tool for designing differing contexts for interaction within physical space.
Blends can offer guidance for researchers and practitioners examining the transitions and experiences between the digital and physical world. By creating and examining storyboards in the style of speculative narratives, set in a near-future Edinburgh Fringe Festival, we demonstrate the efficacy of the method for identifying blended experiences.
Storyboards are low-cost, easy to read and examine, and enable reflection and evaluation from diverse audiences, they can support future iterations of the Blended Experiences Framework, and the methods can be readily adopted in Human Computer Interaction and design settings.
Our first contribution is an evolution of the Blended Spaces Framework to a Blended Experiences Framework. Our second contribution is. the practical demonstration of this framework in a workshop. Finally, we demonstrate the use of speculative design as a tool for examining designs produced through the Blended Experiences Framework.