Designing Novel Applications Inspired by Emerging Media Technologies

The field of Human-Computer Interaction provides a number of useful tools and methods for obtaining information on end-users and their usage context to inform the design of computer systems, yet relatively little is known on how to go about designing for a completely novel application where there is no user base, no existing practice of use available at the start. The success of the currently available HCI methodology that focuses on understanding users’ needs and establishing requirements is well-deserved in making computing applications usable in terms of fitting them to end-users’ usage contexts. However, too much emphasis on identifying user needs tends to stifle other more exploratory design activities where new types of applications are invented in order to discover or create new activities currently not practiced. In this paper, we argue that a great starting point of novel application design is not the problem space (trying to rigorously define the user requirements) but the solution space (trying to leverage emerging computational technologies and growing design knowledge for various interaction platforms), and we build a foundation for a pragmatic design methodology supported by the authors’ extensive experience in designing novel applications inspired by emerging media technologies.

The contribution of Human-Computer Interaction in developing usable computer applications is now well-recognised in both academia and industry.A project developing a large library management system starts with the task of understanding user needs, interviewing librarians and observing their usage of the existing system to be replaced; companies refer to their "usability labs" claiming that the customer user experience is their number one priority; academic publications routinely include a section on "user evaluation" reporting that their test users found the developed system efficient and enjoyable to use.
In the HCI community much emphasis has been put on driving application development based on the information obtained from end-users and their wishes, complaints and actual usage, helping inform the design by establishing the requirements for the system to be built.Consequently many useful tools, methods, and procedures have been developed, adopted and are practiced today that help obtain this type of data, to serve the rationale and justification for the subsequent design process.An initially designed system is then placed in the target environment and a series of usability evaluations with real users are conducted to iteratively refine the system to make it fit to the user needs and context in which it is to run, making up the core methodology of user-centred design.
Although this strong emphasis on the input from the study of users is what makes the HCI work so well (in making the designed application fit to enduser needs), such a method assumes that there are already existing applications and well-understood practices of use from which requirements can be established and refinements occur.An investigation into a novel application and its possible usage in the future is problematic in adopting this user/usageinformed design method because (i) we don't necessarily know what activity/domain and context the developing application is supposed to support, and (ii) there is no user-base or existing practice of use to solicit the requirements from.The hardbuilt tradition of starting by asking and observing end-users does not seem to work well when we try to design a completely novel, unprecedented application that nobody is currently using.
Interview, questionnaire, observation, and the currently popular ethnographic techniques abundantly found in the HCI literature today can be understood as tools to better understand end-user needs and their contexts in order to help design a more usable system in terms of fitting the designed artefact better to user needs, wishes and expectations.However, very few concepts, tools or procedures are available to help design novel applications that do not accompany end-users or existing practices of use.Envisionment, brainstorming and scenarios are typically mentioned in recent HCI textbooks (e.g.Benyon (2010); Shneiderman et al. (2009); Sharp et al. (2007)) to help create new ideas, innovative features or novel functionality for target applications but even these are heavily geared towards the improvement of existing solutions rather than inventing completely new usage or inventing novel applications.
In short, starting with end-users with the focus on making a system fit their needs and tasks is desirable in designing the next version of an existing application or incrementally improving current solutions, but seems not very effective for designing novel applications since the source of information the design process so heavily relies on is either non-existent or insufficient.Furthermore, the inevitable novelty in novel applications means that the initial design of such an application cannot be guided by past exemplars to draw on our established understanding and experiences of what worked and what didn't.
Despite these methodological problems, technology research laboratories focusing on developing novel computational tools still try to use the requirementsdriven or problem-focused HCI methodology as the justification for their research; grant proposals for developing emerging technologies still emphasise the rigorous study of user needs and establishing the requirements as their starting point; researchers without enough resources to conduct rigorous user studies will, rather guiltily, go on to design novel applications that demonstrate great technical potential but find it difficult to publish due to the lack of 'user-centredness' in terms of satisfying/fulfilling the user needs."We invented this tool just because we can" is considered the worst possible rationale for a technology R&D project.
In this paper, we argue that when engaging in designing novel applications that will impact our future usage, rather than today's usage, of technologies and activities, the attempt to use currently advocated HCI methodology is not cost-effective, and from there we frame an alternative, pragmatic design methodology that starts with looking into the emerging technologies and interaction platforms as the basis.We identify three types of required knowledge (on user activity/task, on technology, and on interaction design) in engaging in any design project, and assert that the relative significance and balance amongst these knowledges as conventionally agreed should be different depending on the kinds of design projects we are involved in.This is based on our 12-year experience in designing novel applications that incorporate emerging multimedia technologies, which serve to support our methodological stance.

EXAMPLES OF NOVEL APPLICATIONS
By "novel applications" we mean those applications where there is no existing user base, and no existing practice of use today.Consequently, designing a novel application means designing not to support any of the activities/tasks people do today, but rather to discover or create new activities/tasks of what people might do in the future by way of developing applications that support those activities/tasks first.
One of the best examples of such a novel application we detail here is LifeLogging.With the development of sensor-rich digital cameras such as SenseCam, a new activity of visually capturing one's whole day and reviewing it each evening can be conceived.One of the major problems in such a scenario is that the amount of captured photos can be huge (with a SenseCam, the number of photos collected in a typical day can be well over 2,500), making the reviewing of those photos very time-consuming and tedious.By employing some of Multimedia techniques currently investigated we can help reduce the burden of such a review task, for example by grouping the captured photos into meaningful chunks of "events", identifying more unique events from regular or mundane ones, and selecting most representative photos from the groups of photosall automatically.How then should the results of such techniques be presented to and interacted with the end-users?We envisaged a scenario where a SenseCam wearer's photos are uploaded to a Web server each evening, automatically processed and then using a special browser the wearer can review what happened during the day.With the browser we developed (Lee et al. (2008a)), the user selects a particular day and is presented with a photo montage of the day that turns thousands of photos into a single page of an interactive, visually summarised template.We mapped the uniqueness of each event as calculated by the system to the size of photos (i.e. more important an event, the larger the photo that represents that event), and displayed varying sizes of photos in an easily-scannable manner.The user can move the mouse cursor over an event photo which will then slide-show all photos in that event sequentially, and search for other similar events that happened in the past.As there is no such an application in the market and no such usage has been considered by general public as anything useful or beneficial, it is not possible to rely on interviews, user studies or patterns of previous systems to inform the design of such an application.Will general public relate to this technology-driven idea of visually recording their (and everybody else's around them) daily activities and store on a web server?
Other representative novel applications we developed include a mobile news update service that automatically indexes daily broadcast TV news and provides its users with a personalised highlights of TV news stories on their mobile devices (Gurrin et al. (2004)); an interactive TV where a TV viewer can use a conventional remote control to interact with other remote viewers, search and browse the contents of TV shows with advanced video analysis tools while watching the current channel (Lee et al. (2008b)); a collaborative search tabletop where two users sitting around a table can together search for video clips with their finger touch (Smeaton et al. (2006)); an object search application where a user draws a contour of an object in a picture and the system retrieves other objects similar to it using our object matching algorithm (Sav et al. (2006)); an online museum explorer where the photos of exhibited artifacts at the museum taken by the visitor are uploaded and automatically grouped by individual artifacts using our edge-matching technique (Blighe et al. (2008)), and many more.
The significance of these novel applications is that they were born out of technological possibilities combined with strong interaction platform-specific knowledge (explained in Section 4.3), and not of rigorous requirements engineering or end-user engagement at the start.And yet their base usability is ensured as each of them was designed to support and exploit its interaction platform characteristics, and with this the opportunities are visibly demonstrated to brainstorm and discuss more specific, realistic usage situations to fit people's work and leisure.

EMERGING COMPUTATIONAL TECHNOLOGIES
There are a great variety of computational technologies currently being researched in technology laboratories around the world that will no doubt shape the future of people's interaction with media.Multimedia, for example, is a technically-oriented research community focusing on developing computational tools to automatically understand media content (text, audio, image and video) in order to provide efficient retrieval.Some examples of Multimedia tools include video summarisation which automatically analyses a video stream to extract only those segments of video that are most "important" and generates a shorter, "highlight" version of the video; automatic face indexing which uses face recognition techniques to identify not only the location of a face in a photo but the identity of the person, thus allowing retrieval of a particular person or group of people in a photo collection.What is noticeable in the numerous experimental applications frequently demonstrated in this community (for example in the annual VideOlympics (Snoek et al. (2004)) where a variety of Multimedia demonstrators are showcased) is that they usually exhibit a poor level of usability, partly because the issue of usability is often an after-thought in these projects but mainly because the HCI knowledge in designing for unprecedented applications is not well understood or not accessible to this community.However, the potential for future exploitation of these demos is staggering.
Other promising computational technologies include Semantic Web, Information Retrieval, Artificial Intelligence, Virtual Reality and Social Networking and their sub-fields to name but a few.
Hidden amongst these emerging technologies and their complex demos are the potential benefits that could well be exploited into lifestyle-changing innovative applications that will create completely new activities and new ways of doing things.However, technology researchers do not know how to bring their developments into the real world; HCI researchers do not offer useful concepts, tools or mechanisms to channel these exciting developments into novel applications that are feasible and usable enough to relate to real world situations.This lack of know-how is very obvious when we look at the HCI literature which barely addresses today's variety of new technological advancements and potential ways to harness such development to inform the design of future applications.

REQUIRED KNOWLEDGE
In framing the act of novel application design and suggest alternative methodology, first we describe three types of knowledge required to design any application.

Activities and Tasks
The knowledge on activities and tasks represents our understanding of where a system is to operate, including the specific user groups and their needs, the tasks to be supported, and the context where the interaction will occur.Considered "problem space," this knowledge is very much emphasised today in an application design process.HCI's stance is that the full understanding of the domain where the application is to operate is the most important step in the development process, serving as the requirements from which all subsequent design and evaluation and consequently the project's success or failure depends upon.Due to the importance of this, many useful and reliable methods, tools and procedures have been developed and practiced today that solicit this knowledge.Contextual Inquiry and Cultural Probes, for example, are two very popular methods today to help understand the domain knowledge of a particular environment as the basis for establishing system requirements.
While this is clearly an important knowledge to be had in order to develop those applications that support people's activities and needs of today, there is a definite problem in trying to use this knowledge as the basis for developing novel applications that nobody is using today.How can we start defining the user needs and requirements when we do not even know what activity it would be that the system is to support?"Design exploration" which tries to find out what is possible, what would be desirable or ideal, is quite a different activity to "design practice" which tries to build a system that satisfies a specific group of users in a specific context (Fallman et al. (2008)).In the recent article "Technology first, needs last" (Norman (2010)), Don Norman goes on to say that the ethnographic observational studies and other deep and rich study of people's lives is of very little use in coming up with innovative and novel products.

Inventory of Technologies
In designing a conventional application (i.e., one that has an existing user-base and already-established practice of use, such as a word processor or Web browser), there is usually an inventory of technologies that could be used that we know are reliable, robust and accurate.The processing power of modern computers, database, graphic-rendering capability, video streaming, etc. are some of the mainstream or "proven" technologies we currently employ in developing a computer-based application.
Apart from these proven technologies, there are numerous emerging technologies as mentioned in the previous section currently studied.Although not mature enough to be commercially viable -they are by definition "experimental tools" in research labsthese emerging technologies will eventually become better understood, more accurate and robust as a result of on-going research in the corresponding fields, and will start appearing in the mainstream inventory of technologies.We highlight here that a great starting point for designing novel applications can be from the technology camp that we know its agenda and progress, rather than either from the end-user camp that we do not know what the novel activity would be, or from both camps at the same time that makes the endeavour costly and the matching in the middle difficult.Thus starting from the technology camp, the task of grounding it to real-world problems becomes the main concern, and interaction design knowledge (independent of task or domain knowledge) is required to ensure that the designed novel application exhibits a clear base usability that conforms the special characteristics of the platform, regardless of the specific activities and tasks it is expected to support.

Interaction Design Knowledge
Regardless of the specific tasks or activities to be performed in a specific domain area, there are generic usability issues that arise from the fact that a human user, with his/her inherent capabilities and limitations, is interacting with a particular interaction platform that exhibits a particular set of affordances.Consider a typical menu system on a desktop PC application today: usually at the top of the window we have a bar with menu items such as 'File', 'Edit', 'Window', 'Help', and when an item is selected, a sub-menu appears below with a list of items.This menu style, regardless of its application area (word processor, banking system, company home page, etc.), is a result of past 3 decades of incremental refinement, now considered generally usable.It is "generally usable" despite all different application areas because of the fact that the user is using a keyboard and mouse in an office desktop setting.There is a generic, activity-unspecific, base usability that can be established for an interaction device without understanding the specific application area, and different interaction platforms/devices require different set of design knowledge specifically for that interaction.For example, the mobile platform has been around for past 10 years, starting with early PDAs and mobile phones and now interaction-rich smartphones such as Apple iPhone and Android phones.Designing for mobile interaction requires a very different set of approach, knowledge and skill set from that for desktop PC, mainly because the small screen size, awkward input mechanism and the expected distraction during use make the affordances of the mobile platform very different from those of desktop PC.
While desktop PCs and mobiles are the two dominant interaction platforms today, it will be no false optimism to expect that in the near future, we will be interacting with other more novel forms of interaction devices: touch-sensitive tabletops, interactive TVs, interactive public display walls, 3-D immersive environments and other embedded appliances are currently being developed and experimented with, to become commonplace in the future where people will be interacting with these platforms more frequently in their daily lives than with desktop PCs or mobiles.
By the time emerging computational technologies as described in the previous section become mature and ready to come out of the laboratories and into the real-world, we will have many options in choosing available interaction platforms.Thus, in designing novel applications today, we should put more effort in coupling the developing technologies with these novel interaction platforms instead of only with desktop PCs or mobiles.
For desktop interaction design, we have abundant amount of design knowledge, experience and skill set accumulated over the past 30 years with more than a dozen textbooks, plenty design guidelines and tips available today; for mobile interaction design, we have one academic textbook and a quickly growing number of conference papers tackling various aspects of mobile interaction, and with commercial success such as iPhone (meaning a growing user-base) we will have a sharp increase in the knowledge; for other novel platforms, we have very little knowledge available but as researchers investigate them in their laboratories, the knowledge base is growing for each interaction platform.More details of the current status of the knowledge accumulation for each of these interaction platforms is described elsewhere (Lee and Smeaton (2009)).
The point to be made here is that this design knowledge available for different interaction platforms is part of the solution space which we can leverage without clearly defining the problem space.Furthermore, unlike the inventory of technologies, design knowledge embodies end-users' cognitive and physical capabilities/limitations allowing designing "usable" application not in the sense of making the system fit to what users want to do today but in the sense of ensuring clear visibility of what the system can do and how to operate it, in line with the idea of designing artefacts open to interpretation (Sengers et al. (2006)) "to have clear usability but the ultimate purpose, meaning, and usefulness of the device is left open for users to decide."

RE-BALANCING THE EMPHASIS
We believe that an important step towards effective novel technology exploitation and streamlined channelling into usable product ideas is the shift of emphasis in knowledge from the user activity to interaction design, and the key to such a shift is a specialisation or the branching of interaction design knowledge/skill by each of the emerging interaction platforms described above.
From this perspective, the relative importance between the kinds of knowledge required to design a novel application is quite different from that required for designing a conventional application: without detailed requirements engineering or concrete domain knowledge, we can still design a feasible and usable novel application informed by interaction design knowledge.It might not perfectly fit to any particular existing domain or activity, it might not make much sense in terms of day-to-day use today -but the initial invention stage of a novel application design need not try too hard to identify the concrete domain and usage, because once an innovative application comes into being, subsequent user studies can then be conducted to address user needs-specific issues with today's powerful HCI tools and methods.Thus a way to "combine the best of both worlds" (Ljungblad and Holmquist (2007)) of invention (idea generation) and inquiry (study of people) is, in our view, to start with invention followed by inquiry, with the invention specifically designed to exploit a novel technology and exhibit strong generic usability in order to maximise the output of the subsequent inquiry process.There is no reason why we should try to create a perfect application in one go, that exhibits novel functionalities and fulfills specific user needs at the same time.
While conventional HCI wisdom tends to make us start from the problem space (to support specific activities and tasks), we argue in this paper that considering the kinds of knowledge (and lack of it) available to us, starting from the solution space could be effective when it comes to designing novel applications.Discussions on how future HCI should be different from today's practice started appearing only recently (Benyon et al. (2008); Davern and Wilkin (2008)).In the HCI 2020 forum held in 2007 with some of the leading HCI researchers and practitioners to draw a strategic roadmap for the future of HCI (Harper et al. (2007)), one of the directions addressed was to explore new design methods in response to the fast-changing technological environment we are in today.We believe that our argument in this paper provides one way to move towards that direction, but it requires re-balancing of conventionally perceived weight among the three kinds of knowledge (activity/task, interaction design and technology) in the equation of user-centred design.
Users are an important source of information for getting feedback on existing applications and improving them, but often they are not in the best position to provide novel concepts and ideas that lead to a major innovative leap.The emphasis on understanding the domain and fitting the system to it is a healthy element for design practice that improves the usability of today's applications, but an over-emphasis on "starting with end-users" could stifle potentially innovative solutions at the beginning of a lifecycle of tomorrow's applications.Similarly, we should avoid the trap of only creating what a usability test today can measure (Olsen (2007)) or blindly following the evaluation doctrine available today (Greenberg and Buxton (2008)).We as HCI community should be the supporter of technology-focused communities by offering them optimal methodologies that help them exploit their immense potential in a cost-effective manner, not the enforcer to condemn their general lack of grounding and impose a rigid set of methods optimised for refining existing applications.There are simply too many novel possibilities in the solution space today to keep insisting the problem-to-solution strategy, and we believe that the lack of knowledge in future activities and tasks should not be the bottleneck in exploring these possibilities.