Having Fun With Evaluation

This paper briefly considers some issues in the evaluation of HCI applications through documenting the use and evaluation of a digital video tool StoriXXX used for editing digital resources for 'digital storytelling' and comments on its multiple use as a form of 'bricolage'. It provides a brief overview of use and evaluation in two community contexts, as part of a long term engagement in creating digital storytelling applications, drawing from elements of film and media practice to point to various interpretations of story creation that also provide insights into the evaluation process in HCI.


INTRODUCTION
This short paper documents some early aspects of an evaluation -usability testing, design and redesign -of an application, StoriXXX, designed for the creation, editing and sharing of digital photographic materials as part of a process of 'digital storytelling'.As such we believed it might contribute something to our understanding of aspects of digital storytelling and the development and public use of the growing digital archive.In particular we were interested in the extent to which the different uses of the application might provide some substance, some examples and some understandings of process, to the rather bare bones of van Dijck's (2008) [13] well known suggestion that the malleability of photographic imagery means that the image is now less likely to be understood as providing fixed testimony of times past, but can be more aligned with current and ongoing imaginative reconstructions of memory.
Relatedly, but beyond this initial, and perhaps obvious, concern we were especially intrigued by the possibilities the application offered for considering different, perhaps contradictory, approaches to evaluation (2).Evaluation, when it is thought about at all, is rarely considered much 'fun'.Rather, it is generally regarded as a necessary, if frequently tedious, and boring exercise, tagged on to the end of a project when all the interesting work, the design and development work, is already effectively finished.So, whilst Computing in general and HCI in particular, has gone through various phases and alternate paradigms, evaluation often seems to have remained doggedly the same.So, put simply, we wondered just how far we might push this application in the course of its evaluation and what lessons we might learn from it, not just for design but, importantly, for the process of evaluation itself.Put even more simply, we determined to 'have some fun' with this application and see what we might learn in the process.
Evaluation and its relationship to design has always been a tricky issue, as Bannon (1) pointed out many, many years ago.But in recent years the broadening scope of HCI and the developing emphasis on usability couched in terms of such complex and multiple notions as 'fun', 'play' and 'enjoyment' rather than mere utility has meant that the challenge to HCI design and evaluation has increased -"..systems that can be interpreted in multiple ways allow individual users to define their own meanings for them, rather than merely accept those imposed by designers..".In 'Staying Open to Interpretation' Sengers and Gaver [12] suggest that when devices can be used in multiple, sometimes contradictory, ways, in ways not necessarily specifically envisaged by the original designers, then this, whilst not necessarily a problem also requires that design and evaluation strategies also adjust to face these new challenges. of which two appear especially relevant to StoriXXX and its design ambitions in the process of the creation and support of digital storytelling: 'clearly specifying usability without constraining use' and 'supporting a space of interactions around a topic'where design shifts "to supporting and intervening in the processes of designer, system, user and community meaning making".They similarly point to the need for new evaluation strategies, taking multiple and evolving uses into account, where evaluation similarly shifts "from determining whether an authoritative interpretation was successfully communicated to identifying, coordinating, stimulating and analyzing processes of (evaluative) interpretation in practice" and that accordingly, "systems might best be evaluated by gathering and presenting a variety of assessments from a diverse population of interpreters, allowing outsiders to get a rich and layered view of how the system is used, the roles it plays and the cultural implications it suggests."This is the approach that we have adopted with StoriXXX, quite deliberately pushing it into different modes, different genres of storytelling in order that the system might be evaluated according to multiple, perhaps contradictory criteria.In particular what we had in mind was some notion of unexpected or unanticipated use; what has sometimes been called 'innofusion' (Fleck) where a device or system gets used in new and different ways that the designer had not anticipated but which resonate with an individual user's circumstances and needs.This emphasis on multiple interpretations meant that what we have found particularly interesting is the use of StoriXXX to encourage forms of bricolage in digital storytelling -the piecing together of various still images and different video clips in order to tell different 'stories' and the challenges this presents to our attempts at evaluation.This is not to deny the claim that such digitized photographic materials retain their connection to past events, even as their digital form facilitated their use in new stories that reshape the significance of those events when narrated in the present, but to point instead to the myriad ways in which such applications as StoriXXX, permit, encourage even, the reconstruction and retelling of events and to ultimately support their over-arching claim that such processes, and the production of such artifacts can help sustain and nurture some aspects of civic community organization.The two particular communities we are currently engaged with in this work are the 'Salford Lads' Club in Salford and the village community of Wray near Lancaster; building on Goldsmith's previous work with the Salford Lad's Club in 'Storycircle' [6] and our own ongoing technology deployment and research work with a rural village of Wray as an example of a 'living lab' and it is from video materials developed by and with these communities that our examples and digital stories are developed as we invited them to use, explore and 'mess around' with the technology.

STORIXXX FRAMEWORK
Full technical details of the StoriXXX framework are beyond the scope of this short paper and can be found in [7].It suffices here to give a brief outline that highlights the most important technical aspects of the platform.StoriXXX comprises four elements: 1. Mediaplex -a content and metadata management system -whic h is responsible for ingesting media assets, a process which involves transcoding at five different quality levels and cutting the resulting file into "chunks" of around 2-10MB.This makes it possible to display video on different hardware platforms at an appropriate quality for the available resources and means that only the required part of the video needs to be downloaded.
2. MARS -a media asset referencing system, which makes it possible to create and manage editdecision lists (EDLs).An EDL is an "an expression of composition of one content object (e.g. a video) from parts of one or more other assets" [Mu et al 2013].Thus an EDL may specify that frames "20-30 of video 1" are followed by "frames 10-15 of video 2" which are then followed by "frames 1-10 of video 1", for example.
3. SCN -smart content networking.This is a series of measures to cache content close to a user, or group of users, to reduce the need to download repeatedly video resources when editing.

USING STORIBOARD TO CREATE STORIES
Central to the design of Storiboard is simplicity of use.The intention was not to provide a complete film editing suite, but instead a mechanism for casual users to produce "stories", comprising compositions of edits or cuts of available video material.In addition, because of the web-based nature of Storiboard, and its support for groups of users, these stories can be created and edited by communities of people rather than just a single individual.
Having created a project, building a story in Storiboard roughly equates to the following steps: 1. Decide what narrative you wish to convey.
2. Search for and locate an available video that helps to convey that narrative.
3. Select the part of the video that accomplishes the narrative goal of this element of the story.

4.Repeat until satisfied.
It is possible to re-order selected clips and so this process is not entirely linear.

CREATING STORIES
Digital storytelling is the process by which people develop and share various stories that are created through the use of various digital media technologies -photographs, video, animation, sound, music, and text -that facilitate a range of non-traditional, non-linear and interactive story forms.For some, such as Couldry, [3] such digital storytelling has important social, even liberatory and democratic consequences.Our interest was rather more prosaic focusing on how we might provide various digital resources to facilitate both the individual and communal forms of digital storytelling.
The act of combining short clips of video in order to tell what a typical user might consider a "story" is not necessarily an entirely intuitive one.Those with a passing knowledge of film theory will recognize that this notion of narrative creation is consistent with the standard concepts of editing, or montage.In his introductory text, Monaco [10] states: "While maintaining its [montage's] basic meaning, it also has more specific usages of: • A dialectical process that creates a third meaning out of the original two meanings of the adjacent shots; and • A process in which a number of short shots are woven together to communicate a great deal of information in a short time." It is reasonable to consider that the assumptions embedded in the design of Storiboard's functionality essentially conflated these various meanings of montage, and yet at the same time, ignored them completely.In other words, in the hands of a sophisticated, or natural, story teller, the functionality offered by Storiboard allows for the creativity demonstrated by the early 20th century Russian film maker, and theorist, Sergei Eisenstein.Whilst in other hands, the functionality simply provides the ability to juxtapose disparate clips of video, whose 'disparate' character is overcome through the use of a combining narrative story.It may be inferred from this brief discussion that, whilst continually being improved, the goals of StoriXXX's technical infrastructure are relatively well understood, but support for what users wanted and want to do is evolving quickly, as more users are engaged.During early user studies, various distinct methods, and various different 'interpretations' of story creation have emerged.

The Moving-Photo Album
A " moving photo " is both meaningless and oddly natural in the digital, mobile-phone camera enabled, age.The photo album, however, is an enduring concept -something we are all familiar with, if not, at least occasionally bored by -a sequence of images that relay experience through order and juxtaposition and, if presented in person, accompanied by a vocal narrative.During initial user-engagement studies, this notion of "story" creation emerged immediately and, apparently, naturally.In contrast to the Eisenstein-like notion of storytelling through the combination of bits of video alone, in this mode, the narrative is relayed by the narrator and the composed videos illustrate and "back-up" the story being told.For obvious reasons, think of a friend talking you though some holiday snaps: "This is where we did X, Y, Z." only in this case there are moving rather than still images.See Fig. 2 for an illustration of this, where a user talked us through a story he had created -in this case a documentary style story of the activities that took place in the Salford Lads Club, the various people who worked there, stictched together with (Youtube) clips and stills of famous 'old boys'.This feature is now more fully supported -with the ability for multiple users to add narrative commentaries to "published" (i.e.non-editable) video stories.

The Nod Towards Silent Movies
The second mode of storytelling to emerge from initial user engagement trials has involved conveying what might be called "directive or explanatory narrative" via onscreen text.Only retrospectively did it become obvious that this paralleled the use of text storyboards used in silent movies.Figs. 3 and 4 shows a rather informal example.In this mode, the relative lack of sophistication of the user or the method of simple montage is overcome by conveying the over-riding narrative using on-screen dialogue.As with the "moving-photo album", this mode of storytelling emerged from the use of Storiboard but involved a very different perspective on what might be involved in creating a 'digital story' a perspective that emphasized using the various resources provided by StoriXXX in a deliberately 'artistic' or 'filmic' style rather than the 'documentary' style outlined earlier.Our user describes the process: "I began to play with the Storiboard ..

to create a story. Not surprisingly, because of the intoxicating experience of the drumming, that dominated my initial thinking at the expense of the scarecrowsthe obvious focus of the Scarecrow Festival! However, I found a sort of balance by combining short clips of loud, dynamic drumming, with even shorter clips -almost stills, .. After combining a number of these different clips together, I was quite pleased with the result in terms of "rhythm" -loud movement followed by quiet stillness, repeated a number of times. However, there was clearly, as yet, no story at all. Suddenly, a means of addressing the lack of functionality to add narration occurred to me. And it was serendipitous because what I was actually doing was trying to add a "title" -.. I hand wrote a short title -in two parts, joking about Wray being invaded by scarecrows and drummers -it was trivial, but sufficient to continue the process."
This account of using StoriXXX beautifully captures precisely the dynamic, improvisation nature of digital storytelling and, of particular interest, tells a very different story of process than the algorithmic steps of the process outlined earlier.The quote illustrates the dynamic interface between story 'creation' software and story 'telling'.

The Alternate Version
So far our evaluation of StoriXXX, whilst emphasizing some of the fun elements of an evaluation, have not really touched on any functionalities that are not generally available in a host of other, similar video-editing applications.In this final section we examine a more unusual aspect of StoriXXX, its encouragement and facilitation of the communal sharing and re-use of video resources: whereby the stories and accompanying images developed by others can be reused and redeployed to tell new and sometimes rather different stories.In this next example some frames from a documentary style video have been reused and reframed using an alternate commentary to project a humourous account of 'The Discovery of Colour'.We begin with the original video -the "proper" club movie.

CONCLUSION: HCI, EVALUATION AND 'OVERCOMING MONSTERS'
This short paper documents an awful lot of 'messing around' with the technology.As such it contrasts somewhat with traditional, heavily functional and, frankly often dull, HCI evaluations.Given the emphasis on 'storytelling' we are somewhat reminded of Booker's (2) 'The Seven Basic Plots' that outlines what he regards as the small number of 'basic stories' in the world.Looking at the range of conference papers, HCI might similarly be described as having a fairly restricted number of 'basic stories' of which the 'evaluation story' is just one, albeit an admittedly important one.In particular, and without exaggerating too much, we wonder whether the HCI 'evaluation story', or at least the traditional evaluation story, has some similarities, some commonality, with Booker's 'overcoming monsters' basic plot.Like the 'overcoming monsters' plot -although with an unfortunate lack of beautiful princesses -traditional evaluation stories begin with an 'anticipation' stage and then generally progress through the 'dream' stage, the 'frustration' stage' the 'nightmare' stage' and conclude with the 'miraculous escape' and success and the eventual 'domestication' or 'taming' of the 'monster' technology (for an exception see Gaver et al [8]) -though HCI evaluation stories are rarely, if ever, conveyed in such an exciting fashion.Moreover, as systems increasingly are required to fulfill multiple uses however such a simple story, and such a simple functional evaluation, becomes increasingly unsatisfactory.
This short paper has been generally concerned with the process of evaluation; but particularly concerned with how exactly we might go about evaluating an application designed and used for digital storytelling.In terms of the evaluation of StoriXXX documented in this note we can clearly argue that its architecture allows casual users to "dabble" in producing a range of stories, a range of 'interpretations' without a significant learning curve.Initial user engagement studies suggest that the system is as easy to use as was intended.Only further studies will show whether that ease of use translates to "ease of story creation".
Of course, what is missing from this short paper, for simple reasons of space and focus is an understaning of how this work relates to other storytelling applications in HCI such as those developed by Bidwell et al (3) and Frohlich et al (7) and their accounts of sharing stories in international community contexts.Instead, this paper shows how a web-based digital story creation software can support differing, creative, and improvisation usage as part of a process of digital storytelling and playing on a tension between usability and openness to interpretation.Such usage constitutes a form of bricolage that needs to be assessed and evaluated using criteria proposed by Sengers & Gaver's (12) paper 'Staying Open to Interpretation' in particular : 'clearly specifying usability without constraining use' and 'supporting a space of interactions around a topic' -.We provide some initial reflections and insights gained from our trials of the StoriXXX application; lessons and insights into the changing character and processes, and especially the representation of collective memory that may well be at odds with some prevailing characterizations of digitization as, for example, undermining aspects of memory work.We have found, instead, that digitization and the use of an application such as StoriXXX may contribute to actually enriching and expanding the process of telling stories and of associated memory recall.We find that successful, multiple use of digital resources seemingly depends on various forms of 'bricolage' -forms of pragmatic customization and use of the available materials and that the term, 'bricolage', which originally referred to making do with tools that were available to address an immediate, local and contingent problem, adequately summarizes successful StoriXXX use.The 'messing about' with the technology that we encouraged and documented as a feature of evaluation is in accord with many people's experience of technology, with what McCarthy and Wright characterize as 'felt-life' and as a form of motivated creation.As McCarthy and Wright argue such an approach: "..suggests questions about a person's experience with technology, the irreducible relationship between a person and technology" and in its concern with exploring the experiencing of technology, offers an alternative and rather different approach to both design and evaluation.

4 .
Storiboard -a web-based user interface for creating stories.From the user's perspective, parts of available videos are composed together in Storiboard to create the desired sequence.These sequences are translated to an EDL which describes which parts of the stored video clips are shown, how they are shown and in what order.
Fig 1 shows a typical screen when carrying out this process.

Fig 1 .
Fig 1.A typical Storiboard screen when editing a story.

Fig 2 .
Fig 2. A user talking us through their story.

Fig 5 .
Fig 5. "... this is just going through pictures of different people who've come into the club ... doing football ..."