Designing a semantic sketchbook to create opportunities for serendipity

Serendipity is where unexpected circumstances and an insightful ‘aha’ moment result in a valuable outcome. We discuss how interactive systems can support the process of serendipity: from making new connections, to projecting and exploiting their potential value. We focus in particular on how technology can support reflection – which is an important part of the serendipity process. By considering findings from a set of empirical studies and a set of design principles aimed at encouraging reflection, we present an early stage digital ‘Semantic Sketchbook’ which was designed with the aim of supporting reflection (as well as other aspects of the process of serendipity). We discuss how our ‘Semantic Sketchbook’ has the potential to create opportunities for serendipity and the next steps we intend to take in developing it and evaluating its success.


INTRODUCTION
Serendipity is where unexpected circumstances and an insightful 'aha' moment result in a valuable outcome (Makri & Blandford, 2012).It is an important aspect of our work and everyday lives and has played a vital role in the history of research and innovation (Van Andel, 1994).Yet as we spend ever increasing amounts of time living digitallymediated lives, conventional information searching and filtering can lock us into predefined loops based on previous searches and user profiles, where new, seemingly unrelated data may pass us by (Pariser, 2011).The potential for serendipitous information discovery to address this problem has been proposed (Gup, 1998;LeClerc, 2010).
SerenA: Chance encounters in the space of ideas supports this desire to create opportunities for serendipity.SerenA, a £1.87MRCUK funded interdisciplinary project, is developing a rich understanding of serendipity and prototyping interactive systems that connect researchers to other researchers and resources, to discover things they 'did not know they needed to know.' SerenA will be realised as a Google Android smartphone app that uses the paradigm of a physical notebook to support digital note-taking in the form of text, image, drawing and sound.An underlying semantic reasoning system will present potentially serendipitous suggestions to users as notifications, based on its understanding of the individual researcher user, who inputs content to the semantic sketchbook as well as to a user model through providing data sources (e.g. a Twitter ID or a webpage with a list of publications).The user can choose to act upon these suggestions, which may be time, location, content or context driven, through the app itself (e.g.read a publication abstract, and then link to the full article via a Web browser).
In this paper we outline the first SerenA prototypea mobile semantic sketchbook -as a platform for discussing how digital technology might create opportunities for serendipity on mobile Internet devices.Although we argue that it is difficult if not impossible to directly 'engineer' serendipity in interactive systems, we posit that technology has the potential to create opportunities for serendipity.Using an empirically-grounded process model of serendipity, we identify key parts of the process where design may be effective in aiding users' perceptions and realisation of serendipitous connections.The main focus of our discussion, however, is concerned with ways to encourage reflection on serendipitous experiences (which is

DESIGNING FOR SERENDIPITY
Serendipity is a slippery phenomenon, but many attempts have been made to deconstruct and demystify it (e.g.Van Andel, 1994;André et al, 2009;Makri & Blandford, 2012).Several studies have also explored the concept of technologymediated serendipity (e.g.Beale, 2007;Leong et al, 2008).However, it has been highlighted that 'designing for serendipity' is an oxymoron (Van Andel, 1994) and therefore what we should aim to do is design not to directly 'engineer' serendipity itself, but to create opportunities that users might perceive to be serendipitous.
SerenA user studies, including a diary study, interviews, and focus groups (Sun et al, 2011), informed the development of an empiricallygrounded definition and process model of serendipitous discovery (see figure 1).According to this model (Makri & Blandford, 2012) the process of serendipity involves making a new mental connection.The connection must involve both unexpected circumstances (Makri & Blandford, 2012;Sun et al, 2011) and insight (Makri & Blandford, 2012).Forward-facing projections are made on the potential value of the connection and actions are then taken to exploit the value.After an iterative process of projecting further value to be gained from the connection and taking further actions to exploit the value, the process culminates in a valuable, unanticipated outcome.Sun et al. (2011)  Design, we argue, can support three important elements of the serendipity process in figure 1: (i) Making connections -design can create digital spaces that are conducive to an open-minded approach by presenting and/or visualising information in ways that maximise the potential for unexpected and potentially valuable information to be noticed and examined (and therefore a connection made).Whilst we aim to support each of these aspects of the serendipity process within SerenA, this paper focuses on reflecting on the value of outcomes as this is a key part of the process that semantic sketchbooks have the potential to support.

Supporting making connections
Open-mindedness and designing for emotion (Norman, 2004) are critical aspects of a broader desire for design which is playful, creative or delightful.These design approaches may well support 'making new connections' by subtly encouraging a disengaged state of mind where people's minds become open to new opportunities.Delightful design is an experience that engages the user more fully, an experience which provokes 'ensoulment' (Nelson & Stolterman, 2003) -an emotional response to a product which results in a deeply moving feeling of being significantly changed.This approach is also congruent with Hassenzahl's work on 'joy of use' and 'beauty' in product design and usability (Hassenzahl, 2004).This sense of immersion in design, we argue, might promote or lead to open-mindedness which, in turn, can result in the making of new connections.Indeed, it has been proposed that 'enticing curiosity' (André et al, 2009) and introducing creativity and play (Thudt et al, 2012) in interactive systems might create opportunities for serendipity.

Supporting exploiting the value of connections
In order to make connections or suggestions and present them to the user, SerenA maintains a model, which includes a user's knowledge, goals, information needs and interests.This user model is represented semantically using standard RDF 1 triples and representations that are widely used on the web such as the DBpedia ontology 2 .The user model "[tracks] existing domain knowledge" (an approach proposed by André et al, 2009) and makes 'leaps of inference' between known and inferred data.Inferences can be made using triples for both explicit rules and semantic inference processes.The rules and inference processes serve several purposes; they extend the user model; they form connections between the user model and the resources which may be suggested to the user and they highlight potential suggestions and compare them to the user model in order to propose suggestions that might be perceived by users as being to some degree 'unexpected.' In order to propose suggestions that users will want to take forwards and try to 'exploit', it is necessary for SerenA to find a 'sweet spot' between presenting suggestions that are closely-related to the users' existing interests (which may well be considered valuable, but not unexpected) and suggestions that are only loosely-related to the users' existing interests (which may well be considered unexpected, but not valuable).

Supporting reflection on the value of the outcome
Reflection is an important part of Makri & Blandford's (2012) serendipity process model.Forward-facing reflections are needed to project the potential value of the outcome of serendipity (and therefore spur actions to exploit the value of the outcome).Backward-facing reflections are needed to reflect on the value of the outcome and determine whether any further actions might be taken to make the outcome even more valuable.SerenA supports the reflection process by providing a 'thinking' or 'reflection' space within the Semantic Sketchbook.This is an area to capture thoughts, connections, and annotate SerenA's suggested connections across modalities.
The design of the Semantic Sketchbook takes Csikszentmihalyi's theory of Flow (1975) into account.This theory describes the positive sensation felt when in the midst of an optimal experience.Like serendipity, Flow can be 1 Resource Description Framework: www.w3.org/RDF 2 wiki.dbpedia.org supported by an open, exploratory mindset and positive attitudes and outcomes (Finneran, 2005).André et al. (2009) note that many systems that explicitly try to induce serendipity do so in the background, citing examples which do not require the user "to invest any effort, or even expectation, in the system" (310).SerenA provides a holistic approach to creating opportunities for serendipity; encouraging both reflection and the 'cognitive leap' required to make connections.SerenA is conceptualised as a semantic sketchbook and digital mobile companion, proffering unsolicited ideas and suggestions over time (rather than acting as a response-centred assistant).There is, however, a tension inherent in smartphone usage; users have become used to instantaneous responses to their interactions through technology such as instant search results and Instant Messaging.SerenA, by contrast, supports longitudinal investment and exchange -a slow maturation of the system's capabilities, not unlike the incubation of ideas over time.This is akin to Slow Technology (Hallnas, 2001;Sengers et al. 2005), which opens up time for reflection rather than removing time from the users by attempting to maximise efficiency.Gaver's (2003) seminal article on ambiguity in design recognises the potential benefits in embracing user interpretation by mirroring the complexity and ambiguity of the everyday world, and "allow [ing] designers to engage users with issues without constraining how they respond" (p.2033).Makice (2010) identified four qualities of ambiguous systems; imprecision, playfulness, reappropriation, and provocation.The use of subtle ambiguity, along with the principles of Slow Technology, may indeed provoke reflection and interpretation when embedded into the design of SerenA.We will evaluate the success of these interventions in due course.

THE SEMANTIC SKETCHBOOK
The use of paper-based physical notebooks and sketchbooks is well documented in scientific and creative practices (Tabard 2008;Brereton 2009).However, sketchbooks are not simply an externalisation of memory or repository, but encapsulate a playful, open space which prompts reflection, affords the reconfiguration of ideas and manifests itself as a 'receptacle of becoming' in the in-between state of idea gathering and formation (Unwin, 2009;O'Neill, 2011).The Semantic Sketchbook allows the user to combine their own notes and annotations with the suggestions made by SerenA.As users develop their emergent ideas with the Sketchbook, SerenA uses these ideas to provide further suggestions in a virtuous circle and so the sketchbook is developed in partnership between SerenA and its users.
SerenA's use of a mobile diary study to elicit incidents of serendipity from researchers (Sun et al, 2011) suggested the value of capturing potentially serendipitous experiences as an aide memoire and suggested the importance of reviewing and reflecting on serendipitous experiences.This was one of the reasons why we incorporated functionality to support reflection in our semantic sketchbook.

Semantic Sketchbook Functionality
O'Neill ( 2011) examined the functionality of physical sketchbooks in creative practice with respect to replicating their roles in digital technology.Seven key functions were identified for digital sketchbook design; Start-Up/Navigation, Searching, Archiving, Sharing, Editing, Drawing and Collage.Several of these functions are present in the Semantic Sketchbook (even though we did not design the sketchbook with these functions in mind, but with the need to support the serendipity process in fig.1).The Semantic Sketchbook (see figure 2) also follows Thudt et al's (2012) suggestions for encouraging serendipity, i.e. (1) by providing multiple visual access points (through viewing notes within notebooks, tags or goals) (2) highlighting adjacencies (by reordering and juxtaposition), and (3) providing flexible visual pathways for exploring.
The Semantic Sketchbook allows users to:

Underlying System Development
The Semantic Sketchbook is implemented as a decoupled front-end Android app (which this paper focuses on) and a back-end agent software architecture system which incorporates the user model and generates the suggestions from the Semantic Web.The principles of the Semantic Web are used for publishing rich linked open data and applying semantic labels to taxonomise fields of research.Users convey their thoughts and ideas directly to the sketchbook by adding notes in the form of text, images and sound (and pointers to web resources that they have previously found valuable).All suggestions made by SerenA are annotated with semantic information and when a user adds a suggestion to a sketchbook this semantic information is added to their user model.
Semantic information from the notes created by users is gathered in two main ways; users may tag keywords and goals within the body of the text itself (see figure 2), or the information can be gathered by applying Natural Language Processing (NLP) to the content of textual notes.Considerable research exists in identifying keywords within texts and mapping them onto semantic web vocabularies and is available in software such as DBpedia Spotlight (Mendes et al, 2011).Further semantic information can be gained by inspecting the Web resources that users have noted and mining the semantic content attached to these resources.These strategies are all used in SerenA to gather lists of the user's short and longer-term interests and form a rich user model that has the potential to result in valuable (and unexpected) suggestions.
Research is also underway into identifying a user's goals from textual expressions, so that these can be added to the user model (Piao et al, 2012).By allowing users to express their goals both directly and indirectly within the sketchbook, we hope to maximise the potential for providing unexpected and valuable suggestions.Direct expression of user goals also provides us with useful data for evaluating the success of the inferences made.

CONCLUSION AND NEXT STEPS
We have illustrated how an empirically-grounded process model of serendipity can inform the requirements for tools that aim to create opportunities for serendipity.Future work will involve conducting user studies with iterative prototypes of the semantic sketchbook -focusing not only on the usability of the interface, but the system's success in resulting in experiences that are subjectively perceived by users to be serendipitous (or simply unexpected, insightful and/or valuable).
Also, in keeping with Thudt et al.'s (2012) suggestion that 'enticing curiosity' and introducing creativity and play in interactive systems can create opportunities for serendipity, we would also like to devote time to incorporating elements of play, creativity, and 'delight' into future versions of the sketchbook.Indeed, we intend to develop a second SerenA smartphone application with a more experimental 'delightful' interface.We hypothesise that these design interventions might demonstrate that 'delightful design' not only has the potential to impact on users' affective states, but also on their perceptions of their experiences of serendipity.
In addition, a richer understanding of the impact of context may be helpful in supporting users to exploit the connections made by SerenA (Bental et al, 2012).For example, presentation and visualisation of the connections made by SerenA to explain why SerenA has presented users with specific suggestions may be important in encouraging or discouraging users to act on (i.e.exploit the potential value of) a suggestion.The level of 'transparency' provided by SerenA may also be important to users' perception of serendipity in general.Will they be willing to 'suspend their disbelief' even if they know how SerenA creates its 'magic'?Or will this serve to spoil the illusion of unexpectedness and insight?
Several studies also note the importance of environmental context, where the timing or synchronicity of events and ideas influence their perceived value (André at al. 2009;Thudt et al. 2012;Van Andel, 1994).SerenA's mobile Android app will enable time and location critical suggestions or connections to be made (such as the possibility of attending a nearby exhibition or talk that is taking place today and that the system thinks you will find both unexpected and valuable).
We are also examining the notion of environmental context in our user studies; we are currently conducting an experimental study to investigate how mobile users respond to notifications that either do or do not relate to the user's fields of interest, at various times of the day.
Techniques for evaluating the Semantic Sketchbook (and indeed all serendipity-related technology) will need to be novel and developed from scratch.It is particularly important when evaluating 'serendipity systems' to recognise the subjective nature of serendipity and, specifically, that success will be evaluated based on subjective user perceptions of the occurrence of serendipity (and/or of key aspects of serendipity such as unexpectedness, insight and value - Makri & Blandford, 2012).As part of our evaluation, we will evaluate whether the design principles outlined in this paper have impacted user behaviour.In addition, comprehensive testing will be conducted around SerenA's user model, automated NLP and explicit user goal creation (i.e. the 'semantic' elements of the sketchbook), to assess how goal annotating might fit into contextual work practices.
By designing to create opportunities for serendipity, based on a rich empirically-grounded understanding of the phenomenon, we will support users in making new connections, projecting and exploiting the value of those connections and reflecting on the value of the outcome.By designing for reflection in particular, we will provide an 'incubation space' for connections between people and ideas to brew and, when the time is right, flourish.This, we hope, will help users to seize opportunities for serendipity (both technology and non-technology mediated) and maximise the chances of these unexpected, insightful connections resulting in a valuable outcome.

Figure 1 :
Figure 1: The process of serendipity (ii) Exploiting the value of connections -design can ensure high-quality (i.e.potentially valuable) suggestions by taking synchronicity (i.e.location and time) into account.Design can also provide tools to support users in following up on connections.(iii) Reflecting on the value of the outcomedesign can aid reflection by introducing longitudinal incubation for ideas (e.g. through the digital sketchbook concept), providing space for ambiguous interpretation and re-appropriation.
multimodal notes (i.e.text, hyperlinks, images, audio, and video).(ii) Add metadata to notes (e.g.Geodata, tags, goals).(iii) Edit and delete existing notes.(iv) Access notes in a variety of ways (e.g. by groupings, date, title, tag, or goals).(v) Customise sketchbook covers and titles.(vi) Import, export, move and share notes between sketchbooks.

Figure 2 :
Figure 2: The Semantic Sketchbook.From left to right: Notebook view (creating a new notebook and personalising the cover), Goal list (keywords and goals), Sort By: Visual and date, Keywords and Multimodal notes with Goals (Tags in the body of the text).