Reconfigurable Multimedia Environment

This paper presents a bespoke solution for a realtime reconfigurable audiovisual routing system for capture and playback within a pervasive multimedia environment. Having the ability to capture user interaction and experience is essential when conducting usability research. At the University of Sussex a Higher Education Funding Council for England (HEFCE) funded research lab has been developed to allow user centred research to be conducted. Within the lab a solution was required which would let researchers record from multiple camera sources as well as playback video feeds to many display devices. The desired solution would let researchers have control over audiovisual routing, allowing them to have design flexibility over their experiments without the need for dedicated technical support. The solution presented removes the need for the user to have any knowledge of the underlying hardware within the lab by creating a software layer used to re-route audiovisual capture, output, and playback. It allows for rapid setup giving users more freedom with experiment design, enabling them to design practically any type of experiment for large groups of people. Furthermore, experiments have a more realistic outcome as designs can be modified during runtime.


I. INTRODUCTION
Usability audiovisual capture systems have been developed for many years.Mainly systems are custom designed for installation within a bespoke environment.Recently, however, some off the shelf solutions have begun to appear in the market.Morae [1] and Silverback [2] are examples of applications used for usability testing and market research.They can connect to a single video camera or webcam and capture a video stream directly onto the user's hard drive.Morae can also be used with UserVue [3] to enable live streaming of captured usability footage.
These applications are used for small-scale usability research.If capturing video from multiple sources, or complex audiovisual integration is necessary, they are not capable of delivering the required performance.For any large-scale audiovisual capture system a bespoke solution is usually the only solution.Google [4] has recently developed a system to address this type of problem with their distributed user experience capture system.This bespoke designed system was developed to support dozens of researches that could potentially be active in labs all over the world.This allows Google to coordinate and share information from their head office to any of its usability labs.The solution developed uses a Mac Pro for recording video streams (located within test lab) and a web-based video delivery system (controlled remotely from Google headquarters).Commercial software is used to capture any video feed connected to the Mac Pro and once a capture is finished it is uploaded to Google's UseTube servers.This can then be accessed from any web browser using any computer across the globe.
These types of systems are primarily designed for observing user interaction using a single desktop/laptop computer.Their limits are reached when the user needs to observe and record a large group of people potentially interacting with a variety of equipment or one person in a ubiquitous, technology rich, environment.Larger-scale audiovisual capture systems have predominantly only been developed for observing interaction within larger environments, e.g.smart homes.Within the world of pervasive computing environments, smart homes such as The Ubiquitous Home developed by the National Institute of Information and Communications Technology in Japan [5] and the Smart Room developed by Technical University of Catalonia [6] have developed capture systems to record data relating to their real-life living experiments.
The Technical University of Catalonia Smart Room has been designed as a space where "researchers can test multimodal analysis and synthesis developments in the area of human computer interaction" [6].It also acts as a facility for providing data for technology development and evaluation.However these types of systems fail to address the issue of reconfiguration.The system setup is rigid and all the data from the various feeds are stored for analysis once the experiment or event has taken place.Reconfiguration is impossible due to the inability to access, route or analyse audiovisual data feeds in realtime.
This research consists of two key elements which together create a system which allows researchers to rapidly reconfigure an audiovisual capture system for capture and playback within a pervasive multimedia environment.The first is the development of a highly configurable realtime environment for creating "wizard of Oz" [7] type usability experiments and the second is the development of an intuitive and sophisticated capture environment to record and analyse results from the usability experiments.

II. USER CENTRED RESEARCH LAB
The User Centred Research (UCR) Lab at the University of Sussex, was originally designed and developed for conducting user experiments to further understanding of natural language computing.Consisting of two experiment rooms along with a control room, it allows parallel user observed research to be conducted without the need for human-to-human interaction.Within the lab a solution was required which would let researchers playback video feeds to many display devices as well as record from multiple sources.Furthermore the solution would enable researchers to have control over audiovisual routing, allowing them to have design flexibility over their experiments without the need for dedicated technical support.Within the experiment rooms there are large amounts of both audiovisual recording and playback equipment.The recording equipment consists of 12 cameras and multiple cabled and radio microphones, giving researchers the ability to record the activity within any part of the space.The playback equipment consists of two 40" plasma televisions, two projectors, four Mac Pro computers, three iMac computers and a large number of speakers.The equipment is all controlled from a control room, which allows experiments to be conducted from a separate isolated location.Due to the versatility of the lab it can be used for many areas within usability research.

A. Overview
The reconfigurable audiovisual routing system includes a Mac Pro, video matrix, audio matrix, 10 flat panel monitors and a number of dedicated capture devices as shown in Fig 1 With the system a user is able to reconfigure all the audiovisual equipment within the UCR lab.The system front-end consists of a novel graphical user interface (GUI) and an audio sound desk.The backend of the system consists of one 32 channel audio matrix and one 32 channel video matrix.Using the system a user can control the video matrix allowing them to route a video signal from any input device to any output device including the capture hardware.The audio is handled manually via a 32 channel audio matrix.This allows users to control video and audio feed independently from one another.The system front-end was created for the Mac OS X environment using the Cocoa API.At the heart of the software is the Core Data framework.This framework provides "generalised and automated solutions to common tasks associated with object life-cycle" [8].Using the Core Data framework a managed object model was created which defined the input and output objects, their attributes as well as the object relationships.In this way a virtual device can be created within software that can then be manipulated by the user without ever having knowledge of the underlying hardware.
Although similar capture systems have been produced before [9] the way in which the user controls this system is unique.This is achieved with the novel GUI, which presents the user with a top down view of the lab layout (Fig 2).Using this GUI the user is able to drag the mouse from an input device and drop it onto an output device.The software layer then communicates with the video matrix backend and configures the necessary routing.Previous to the reconfigurable audiovisual routing system a user was required to have previous knowledge of the underlying hardware.With the new system this requirement is removed, the user need now only have a simple understanding of using a GUI based computer, and, if a device needs relocating, where to plug it in.This allows the system to be operated by any researcher with little to no training.The system can also be seen as a simple one box solution.The same system can be rolled out to any other research lab requiring a reconfigurable audiovisual routing system for capture and playback with only the purchase of necessary hardware.

B. Audio/Video Sensors
The multi-sensor network of microphones and video cameras provides high quality data for user interaction analysis.There are several types of microphone installed within the lab such as tabletop, ceiling suspended and lapel clip.The configuration of these microphones is not defined.Using numerous connection points located throughout the labs a researcher can configure the location of the microphones to achieve best results for their given experiment.This type of re-configurability continues with the cameras.Apart from four main cameras located within the middle of the lab, all other cameras can be relocated to any place within the lab that has a connection point available.Connecting a device is as simple as plugging it in then selecting the connection point number using the system front-end.This gives the researcher complete control over the layout design of the sensors but making it as simple as plug and go.

C. Front-end GUI
The reconfigurable audiovisual routing system frontend can be used in two ways, design mode and run mode.When setting up the lab the devices (e.g.camera, plasma screen) are connected to the pre-installed connection points located throughout the lab.These connection points feed back to the video matrix and audio matrix located within the control room.When in design mode the user replicates the physical layout of the lab by adding various devices to the GUI and moving them into the corresponding place.This lets the user visualise the space on screen, simplifying the task of real-time reconfiguration.By selecting a device the inspector panel can be used to specify its settings (e.g.name, type, connection point number).As focus of an experiment shifts around the room the user can more easily switch camera to keep the subject in shot, this handled with run mode.
Run mode (Fig 2 ) gives the user freedom to connect any of the input devices to any of the output devices with a simple and fast drag and drop of the mouse.Speed is essential with a real-time reconfiguration system so being able to visualise the space combined with the natural drag and drop selection solution means that a user can quickly react to events happing within the environment.During experiments users can react to a subjects interaction with the environment by routing different audiovisual feeds to the various monitors within the space.If, for example a subject asks to see their favourite television show on the plasma screen, it can be played out using one of the Mac computers and routed instantly through to the screen.Using this "wizard of Oz" [7] type technique complex experiments can be set up without the need to write any sophisticated computer applications to control the environment.

IV. CONCLUSION
The system described within this paper provides a real-time audiovisual routing system for reconfiguring the UCR lab environment as well as capturing user interaction data.In order to gather the necessary information the system described needed to be capable of capture and playback from any video or display device located within the UCR lab.The chosen system architecture proved suitable for video capture from multiple streams in real-time.Furthermore the integration of the various audiovisual devices could not have been possible without the chosen system architecture.
The reconfigurable audiovisual routing system allows rapid setup of the lab, which gives users more freedom with experiment design enabling them to design practically any type of experiment for large groups of people.Furthermore experiments have a more realistic outcome as experiment designs can be modified during run-time.
Previous to the system setup would have been a laborious task carried out by a dedicated engineer.With the adoption of the reconfigurable audiovisual routing system a user with little knowledge of audiovisual capture systems can handle setup and obtain superior results with more accurate data.Although the one box solution has been implemented the purchase of specific hardware limits its widespread adoption and the overall cost of the system makes it viable only to large institutions.
During testing, various parts of the system were installed within the Media Technology Lab [11] recording studios.In order to access the various hardware and software capabilities of the system various tests were completed by the studio engineers.Feedback from the engineers has been very positive, however as the system is more heavily used within the UCR lab more feedback will be obtained leading to the system being updated and improved.
What is apparent is that the software integrates extremely well with the various hardware components and presents users with a simple front-end allowing them to focus on their experiment rather than the equipment.To further integrate the system, the audio configuration could be incorporated into the reconfigurable audiovisual routing system software thereby increasing the functionality and software options available to the user.This reconfigurable audiovisual capture system is intended to further knowledge into usability and natural language research by providing researchers with a state of the art facility.