Designing Blended Spaces

We present an approach to the design of mixed reality spaces that aims to create a more harmonized and unified user experience. We refer to these as blended spaces. Blended spaces draw upon a description of physical and digital spaces in terms of the ontology, topology, volatility and agency. By describing physical and digital spaces in these terms we are able to use the process and design principles of conceptual blending to arrive at a design that maximizes the relationships between the spaces. It also guides the development of the touch points between the physical and digital spaces. We discuss the user experience in blended spaces and briefly allude to the significant philosophical implications that blended spaces of the future will have to deal with.


INTRODUCTION
Mixed reality comes in a number of forms, spanning the reality spectrum described by Milgram and Kishino (Milgram and Kishino, 1994) from digitally enhanced physical spaces to physically enhanced digital spaces (Figure 1).An example of the former would be a vineyard covered with sensors supplying data about moisture and other growing conditions of the vines.An example of the latter might be the guitar interface to the game Guitar Hero.In the middle lie many combinations of physical and digital objects and spaces, from QR codes on buildings providing information, to GPS triggered events on a smart phone, to augmented reality (AR) overlays on buildings, seen through a phone and using GPS and compass information, to a mixture of maps with realtime video of the physical location integrated into the map.
However, designers have very little advice on how to design for engaging user experiences in mixed reality.There has been a lot of work on tangible interaction, a particular example of augmented reality where the real and the digital are designed as a very tightly-coupled object (Ishii and Ulmer, 1997) but much less on mixed reality spaces.Steve Benford and his colleagues have been looking at mixed reality for some years (Benford, Greenhalgh, Reynard and Kolva 1998) and more recently talk about hybrid spaces and how people move through trajectories of hybrid experiences (Benford, Giannachi, Kolva and Rodden, 2009).Wagner et al., (Wagner, Broll, Jacucci, Kuutii, McCall, Morrison, Schmalstieg, and Terrin, 2009) also discuss this need to recognize that people are not simply present in mixed reality spaces, they move through different experiences and engage in rich, new social phenomena.
Figure 1 The space of mixed reality (after Milgram and Kishino) Our interest is in going beyond mixed reality to investigate spaces in which the real and the virtual are brought together into a unified, designed user experience; a blended space.A blended space is a mixed reality environment of any scale in which the real and the virtual worlds have been carefully and considerately brought together with some content, or access to content (Benyon, 2012).The physical space can be described in terms of the objects that exist in the space, the topographical relationships between those objects, the people that populate and move through the space and the volatility of the space.The digital space can be described in terms of its ontology and topology (information architecture), the agents and other people in the space and the volatility of the space to change.
A blended space consists of a physical space and a digital (or information) space that have been brought together to create the opportunities for new experiences.These may be designed for entertainment such as the work of Blast Theory (Blast Theory, 2012).They may be primarily for information such as many of the augmented reality apps that overlay information onto the real world according to a person's direction and location.They may be cyberphysical systems where a physical space is covered in computational devices.They may be purposedesigned rooms in which technologies have been integrated to provide particular functionality (Benyon and Mival, 2012).
In this paper we see how a description of spaces in this way allows us to focus on key correspondences between physical and digital spaces.These can be used in a process known as 'designing with blends' (Imaz and Benyon, 2005) that is based on conceptual blending (Fauconnier and Turner, 2002) a theory of cognition that shows how new insights come about by blending mental constructs, or mental spaces, into new ideas.Fauconnier and Turner provide a number of principles and guidelines for understanding how to design 'at a human scale'.We take these ideas and apply them to the design of mixed reality, blended spaces.
In section 3 we present a case study of deigning a poetry garden that shows how the blended space arises from the characteristics of the physical space and digital content and the correspondences between them.In section 4 we go beyond design to look at the user experience (UX) of blended spaces and how blending gives rise to a multifaceted and multilayered UX.The conclusion in section 5 highlights the significance of blended spaces for future interactions.

DESIGNING WITH BLENDS
Fauconnier and Turner's book The Way We Think introduced their ideas on a creative process that they called conceptual blending.They argued that cognition could be seen in terms of mental spaces.Developing and extending the ideas of Lakoff and Johnson on the importance of metaphor to thought (Lakoff and Johnson, 1999) where metaphor is seen as a mapping between two domains, Fauconnier and Turner see blending in terms of four mental spaces (domains).Two input spaces have something in common with a more generic space.Blending is an operation that is applied to these two input mental spaces which results in a new, blended space.The blend receives a partial structure from both input spaces but has an emergent structure of its own (Figure 2).
Part II • Techniques for designing interactive systems 206 know a computer window is different from a window in a house.It shares the idea of looking into a document, as you might look into a house, but when you open it, it does not let the fresh air in.It is only ever a window into, or onto, something.Moreover, it has a scroll bar, which a window in a house does not.In a similar fashion we know that the trash can, or the recycling bin, on a computer screen is not a real trash can.The connotations of recycling files are really quite complex.
The contribution that Fauconnier and others have made (e.g.Fauconnier and Turner, 2002) is to point out that what we call 'metaphors' in design are really blends.A blend takes input from at least two spaces, the characteristics of the domain described by the source and the characteristics of the target that we are applying it to.So a computer window takes elements from the domain of house windows and elements of the functioning of a computer trying to get a lot of data onto a limited screen display.The metaphor of a folder is a blend from the domain of real folders that you keep papers in and the domain of computer files which have a physical location on a disk.
Figure 9.5 illustrates this idea of a blend.The blend that results from bringing two domains together in this way will have some features that were not in the original domains.Blends have an emergent structure that results from bringing two sets of concepts (from the source and target domains) together.So, the computer window has different features from both a real window and the computer commands that preceded it.The rolodex concept that arose during the HIC case study (Chapter 6) had emergent properties.These concerned how people could navigate in the 'virtual' rolodex (as opposed to a real, physical one) and how the search results were presented as opposed to how they were presented following a traditional query.It is exactly these emergent properties that makes one design better than another.For example if we take the concept of a window in a computer operating system, this is a blend of the concept of a window (as in a house) and the concept of computer operations.This results in a new mental space of 'window' that now includes things such as a scroll bar, a minimize button and so on that you would not associate with a window in a house.The blended space of a computer window has inherited some shared characteristics from the generic space of 'looking onto something', but now has its own characteristics.
The main principles of blending, are that the projections from the input spaces make new relationships available that did not exist in the original inputs, and that our background knowledge in the form of cognitive and cultural models allow the composite structure to be experienced in a new way.The blend has its own emergent logic and this can be elaborated to produce new ideas and insights.This blended space may then go on to be blended with other mental spaces.
Hoshi and Waterworth (2011) use conceptual blending to design devices for elderly people.Using concepts such as background awareness, peripheral perception, etc. they bring together the input spaces of technologies (such as sound, light, airflow, etc.) with information sources such as notifications to produce a blended space of an ambient information systems.They call this tangible presence.Markussen (2009) uses blending in this way to design a blood-taking device for a hospital.He contrasts the emotional appraisal of the device (it is rounded, and curvy and looks generally very pleasant) with its use and the unpleasantness of the needle taking the blood sample.Jetter, Geyer, Swartz, and Reiterer (2012) use blending theory to design meeting rooms.Imaz and Benyon [21] have applied the ideas of conceptual blending to analyze developments in HCI and software engineering.They argue that designers need to reflect and think hard about the concepts that they are using and how these concepts affect their designs.They emphasise the physical grounding of thought by arguing that designers need to find solutions to problems that are 'at a human scale'.Drawing upon the principles of blends suggested by Fauconnier and Turner (Fauconnier and Turner, 2002) they present a number of design principles.These include designing to preserve an appropriate topology for the blended space, allowing people to unpack the blend so that they can understand where the new conceptual space has come from.There are principles for compressing the input spaces into the blended space, aiming for a complete structure that can be understood as a whole (the integration principle) and for keeping the blend relevant and at a human scale.
Imaz and Benyon present an abstract design method that shows how conceptual blending can be used during the analysis phase to understand the issues of a particular situation and how they can be used during the design stage to produce and critique design solutions.For example, they discuss the existence of the trash can on the Windows desktop.Here the designers have chosen not to enforce the topology principle (since trash cans normally go underneath a desk), but have instead promoted the integration principle, keeping the main functions of the interface together in a 'desk top' metaphor.(Though this example is long gone now that it is a recycling bin which seems to break many principles of conceptual blending).
Bringing the ideas of conceptual blending together with the discussion of the physical and digital spaces leads to a conceptualization of blended spaces that is illustrated in Figure 3.We have a generic way of talking about spaces -ontology, topology, volatility and agents.This is the generic space of spaces and places that is projected onto both the physical and the digital spaces.The correspondences between the physical and the digital are exploited in the design of the blended space.The job of the designer, then is to bring the spaces together in a natural, intuitive way to create a good user experience.

Figure 3 Conceptual blending in mixed reality spaces
One other consideration is important in the design of blended spaces.The physical and the digital rarely coexist.There are anchors, or touch points where the physical is linked to the digital, but there are many places where the physical and the digital remain separate.QR codes or GPS are examples of anchor technologies that bring the physical and the digital together.An iPad running some augmented reality software will take a person into a blended space for the period that they are looking at or through the device, but the connection is lost once they turn to talk to a friend when they return to the physical space, or make some adjustments to the software when they move into the digital space.Thus people move between the physical, the digital and the blended spaces as in the idea of a hybrid trajectory [ (Benford, et al, 2009).
The blended space encompasses a conceptual space of understanding and making meaning and this is where the principles of designing with blends play their part.People need to be aware of both the physical and the digital spaces, what they contain and how they are linked together.People need to understand the opportunities afforded by the blended space and to be able to unpack the blend to see how and why the spaces are blended in a particular way.People need to be aware of the structure of the physical and the digital, so that there is a harmony; the correspondences between the objects in the spaces.The overall aim of blended spaces is to design for a great user experience, at a human scale.

THE POETRY GARDEN
In order to illustrate the process, we will describe the development of a poetry garden in St. Andrew Square in Edinburgh.St. Andrew Square is a public park in the centre of Edinburgh.We were commissioned by the Unesco Edinburgh City of Literature to come with suggestions for how to turn this into a poetry garden.
Our first activity was to try to understand the physical space and how it was used, so we undertook several hours of observation at different times of day.We then conducted a number of brainstorming sessions to come up with possible digital augmentations for the space and finished with the design of a blended space that we felt would provide an appropriate experience.

The Physical Space
As a central location, it used both as a crossing between shops on the north-east side and on the south-west.However, it is also a space where people sit and rest, eat lunch or play with their children.There are several areas sheltered by trees and other open grass areas.The square has a café towards the northeast corner and an attractive garden and pond around the tall central statue of a local dignitary.Throughout the year, different city events such as the book festival or science festival use this square as a showroom.
The space is a highly dynamic or volatile area.Although a large number of people walk through the square, relatively few stay more than half an hour.The coffee shop is yet another centre of interest.People, especially during the day are using this shop as a natural stopping point.
The St. Andrew Square surroundings are busy all day and in the evening with different people coming from all over the world and all age categories.Depending on time of day and year, the demographic aspect of the square changes considerably.Tourists visit, young people from nearby schools visit, people sit down for lunch, commuters hurry through in the morning and mothers with their children visit.

The Conceptual Space
Conceptually the space needed to serve people who walked through the garden regularly, tourists and those who frequented more rarely.It has to serve the needs of those having a quick lunch and those who had more time sit at length.There was also an important consideration as to whether people would simply consume the content or whether they could contribute their own poems, comments or other content to the garden

The Digital Space
The technology we used in this design is varied, involving speakers, projectors, QR codes, AR projections, GPS, mobile devices and private network systems.The content was, of course, poetry but there was much discussion about what sort of poetry was most appropriate for which purposes.This was left undecided until a later stage of the project.

The Blended Space
The concepts that were delivered to the client included the following.4) With this idea, digital poems would be created, embedded in QR codes and the visitors would take the poems with them.This will help the experience in two ways.One is to take something physical, so that people would have something physical to remind them of their visit; the digital experience persists after the journey.The second point is the need for a technological device to decode the poem for them.So whatever the users would do, they need the phone or mobile device to read the poem written there.Therefore, the experience is partly digital and physical.

Carry a Poem (Figure
Sound Showers (Figure 5) This idea involves a bench and localized speakers to create a serendipitous poem experience.Motion sensors will detect the user sitting down, therefore increasing the volume on the speakers.From the user's perspective, resting and enjoying a poem is a simultaneous action.6) This proposal used an AR marker to trigger the projection of a short video of a poet reading a poem.Using the fiducial marker made the character appear as if he were in the square.

Project a Poem AR (Figure
The 3D QR sculpture (Figure 7) This sculpture could be used as a QR code when the visitor was in the correct alignment.It would take the visitor to a web page where poems were displayed.

Summary
The design of the poetry garden aimed to maximise a blended topology, fitting the different ways into the digital space into the physical space of the garden.The AR projection fitted into the stone seats near the pond, the sound showers went into the sheltered areas, the sculpture on the main area, carry a poem used QR codes and these were located near the entrances to the park.The design accommodated the volatile nature of the space through natural and rapid methods of access to the content, and the QR sculpture added collaboration between people as they tried to line up a QR reader with the sculpture as it would only act as a QR code from a certain angle.The integration principle was also important as we wanted people to see the garden as a whole and to enjoy the way the digital portals were distributed across the space.

EXPERIENCE IN BLENDED SPACES
Blended spaces should be designed at a human scale.That is to say that they should be designed putting people first, using concepts that people understand rather than being driven by technological issues.Designing blended spaces is about developing the correspondences between the physical and digital spaces so that people can see and experience the movement across the physical and digital spaces.Designs should recognize the multi-layered nature of interaction design, from the basic layers of visceral experiences to the cognitive interactions necessary to understand and engage with the space, to the ideas and emotions that these give rise to.
In another example of a blended space we have developed a fictional narrative of Robert Louis Stevenson's (RLS) last day in Edinburgh.Using QR codes to link the physical and the digital spaces, the narrative took people through some of RLS's favourite places in Edinburgh and provided relevant sections from his writings about the places.The ontology, in this case, consisted of the physical locations in Edinburgh that RLS wrote about.The digital space used QR codes and a smart phone to deliver the content in the form of text from RLS's book Picturesque Notes.Although this was written over 100 years ago, the physical locations are almost exactly as RLS experienced.
The topology is dictated by the physical topology of Edinburgh.The locations selected were formed into a 0.7 mile walk.The digital topology simply had a page of text for each location.The experience was designed as a single-person experience with no interaction with other people necessary.However the narrative brought in RLS himself as he appeared to lead the visitor along the walk and share his thoughts of the spaces from 114 years ago.The focus on the locations as the key objects in the ontology meant that it was a non-volatile space.There was no change in the locations themselves nor in the quotations.
The experience of taking the walk moves the user through and conceptual experiences that is designed to bring about a new, blended experience.There is the self, physical and digital spaces and experience of inhabiting and moving through that space.There is the physical level of being in Edinburgh, in the outdoors, walking up narrow alleys and over cobbled streets.There is the feel of the phone, perception of the QR code and physical interaction to find the touch point and enter the digital space.Here there are the words of RLS, describing very similar weather and similar views that you experience today.
Building on this experience we are now developing a multi-layered experience called 'historical echoes'.This uses GPS to trigger a visitor's first experience of becoming aware of digital content.Visitors' phones vibrate to make them aware that content is nearby.Tapping the phone brings in an audio experiencewhispers indicate that content is nearby and get louder as the visitors approach the anchor, the portal into the digital world.Once they are near enough to see an AR tag they can scan with the phone and go further into the detail of the content.An animation smoothes the transition between the real and the digital and into the digital space.
Thus experiences in blended spaces can multi-layered as well as following the hybrid trajectories discussed by Benford and his colleagues (Benford, et al. 2009).Moving up and down in the blended space as well as in and out of the physical and digital.

Conclusions
We feel that this is very much the beginning of work in blended spaces.At present the phone and the need for markers such as QR and AR can get in the way of a great blended experience.The user has to step out of the flow of the whole experience in order to align the phone and code, when a more seamless touch point between the physical and the digital would provide a better, blended experience.And this will happen as technologies continue to improve.
But there is a bigger issue that underlies the concept of blended spaces.Place is the medium in which we make ourselves.To be is to be in a place and to be able to move and act in the space.Until now this meant a physical place, but with blended spaces comes a new sense of place.
Blended spaces mean that people have an extended presence; from their physical location into digital worlds.The examples we have explored here take people into quite traditional digital content such as text and video, but soon we will be taking people into highly realistic digital spaces when it becomes difficult to tell real from digital.The blended spaces of the future will be like the Photoshop of today.Mashed up places providing surreal or unreal experiences.

FigureFigure 2
Figure 9.5 Illustrating the concept of a blend

Figure
Figure 4 Carry a Poem

Figure 6
Figure 6 Project a Poem reading