Applying soniﬁcation to improve accessibility of point-and-click computer games for people with limited vision

Computer game accessibility for people with limited vision is a challenging problem because many games, such as point-and-click games, heavily rely on mouse interaction for game world exploration. But mouse interaction requires the sense of sight for orientation and navigating through the user interface. Solutions have been proposed to this problem but usually they come at a signiﬁcant increase in the cost (e.g. haptic devices). This paper explores the use of general soniﬁcation techniques applicable to a broad set of games to generate acoustic hints that help limited vision users to locate the interactive elements in the game world by simplifying the use of the mouse. This is a ﬁrst step to improve point-and-click computer games accessibility with little extra development cost and the initial soniﬁcation experiments with limited vision people show promising results.


INTRODUCTION
When interacting with a computer, people with limited vision usually prefer to use a keyboard (normal or Braille) instead of a mouse, as they control the position of the cursor in the screen more easily.This prevents them from enjoying programs designed to be used with the mouse.This is the case of many computer games and other multimedia applications.One good example is point-and-click games where the world is explored by moving the cursor over the scene and interactions with objects and characters are triggered by clicking the mouse.
Alternative interaction methods have been proposed for users with limited vision.Some games can be controlled using the voice, the keyboard or haptic devices (Westin, 2004); other games include some acoustic feedback that mitigates the effects of lacking visual feedback.However, a drastic change in the interaction mechanism may impact the engagement in the games since both characteristics are tightly linked in video games.Additionally, these approaches usually involve an extra cost that must be assumed either by the developer or the user.We propose a new approach where acoustic information is generated within the game to simplify the mouse use for players with limited vision without increasing the cost or requiring the acquisition of new devices.
Moreover, most of the game accessibility approaches proposed have been developed ad-hoc for a specific game.Instead we propose to introduce these features as configurable characteristics in a game authoring tool: <e-Adventure>.Two are the main advantages: on the one hand, the cost of developing accessible games decreases for new products as the technology becomes reusable; on the other hand developers' awareness about accessibility in games increases by enhancing its visibility.Nonetheless our approach poses also a greater challenge since the accessible technology developed needs to be conceptualized, abstracted and generalized in a way that is easily configurable to fit different games.This paper is structured as follows: in section 2 we introduce the genre of point-and-click games and the <e-Adventure> game platform in the context of the work being presented.In section 3 we present an overview of different techniques currently used to provide acoustic information to help people with limited vision to interact with a computer game.The solution we propose to make point-and-click games more accessible is described in section 4. We have conducted a preliminary evaluation of our approach with visually limited users, and the result of those tests will be presented in section 5.In the last section (6) we will discuss the conclusions we have extracted from this study and propose some lines of future work.

POINT-AND-CLICK COMPUTER GAMES AND THE <E-ADVENTURE> PLATFORM
The <e-Adventure>1 platform (Moreno-Ger et al. (2008); Torrente et al. (2010)) is a Java-based game authoring tool aiming to facilitate the integration of educational point-and-click adventure games and simulations in the educational system.The <e-Adventure> game editor was designed for non professional educational game designers (see Figure 1).Therefore <e-Adventure> allows for developing educational games at a reasonable cost without requiring programming any code or a deep understanding of the gaming field.<e-Adventure> is composed by two applications: the game editor and the game engine.The game editor is the authoring tool used to create the games (see Figure 1).The game engine is the application that executes the games once the development is completed.<e-Adventure> games can be distributed as standalone Java applications but also in web environments as Java Applets.
<e-Adventure> is specially oriented to the creation of point-and-click adventure games (similar to Monkey Island TM or Myst TM sagas) since this genre is one of the more adequate for educational applications (Amory et al. (1999); Dickey (2005)), although it can be used to develop other kinds of 2D games.
<e-Adventure> games, like most point-and-click adventure games, are composed by interactive elements that are laid out on a 2D scenario.Interactive elements can be objects or animated characters (see Figure 2), or even areas of the game scenario (see Figure 1).
The exploration of the game scenario to discover the interactive elements is done by moving the cursor around the screen.When the mouse points an interactive element visual feedback is provided, as it is usual in point-and-click games.By clicking on any interactive element a contextual menu with available interactions is displayed.Interactions can be trigged by clicking on the appropriate button.
<e-Adventure> games can be in "third person" or "first person" modes.In third person mode the player is represented by an avatar: an animated character that is under the player's control.When the player interacts with an element her/his avatar will execute that action in the game world.For instance, if the player clicks on a "Talk to character" option, the avatar will get close to the selected character and establish a conversation.In first person mode the player interacts directly with the game environment without needing an avatar, perceiving that he/she is part of the game world.

AUDIO GAMES AND SONIFICATION
Audio games are a complete new family of games in which the information provided to the user consists mainly in sound (Friberg and Gärdenfors, 2004).Although it is a relatively recent field of research, several design recommendations have already been formulated (Gärdenfors, 2003) and both commercial and experimental games have been created to explore this field.They rely extensively on the sonification techniques that we will briefly describe bellow.
Sonification (Hermann, 2008) is a generic term for the action of generating acoustic information from data.Its commonly accepted definition is: "the use of non-speech audio to convey information" (Kramer et al., 1999), may this information be a diagram (Alty and Rigas, 2005), a graph or other similar items.
In recent years several sonification techniques have been developed.Some of the more relevant (Hermann, 2002) are described below: • Audification.Technique for translating data into sound, in which data values are converted to sound samples to create an audible acoustic signal.It is used in diverse fields such as seismology or time series analysis.
• Earcons are "audio messages used in the user-computer interface to provide information and feedback to the user about computer entities" (Blattner et al., 1989).They are usually composed of synthetic tones and can be used to perform tasks like navigating a menu by assigning an earcon to every option of its tree-like hierarchy (Barfield et al., 1991).
• Auditory icons are "everyday sounds designed to convey information about events by analogy to everyday sound-producing events".A common example is the sound of a crumpled paper to indicate the sending of a file to the recycle bin.
Not all these techniques are fully applicable to point-and-click applications or games.Some of them (e.g.earcons) are too complex to be easily generalized and integrated in authoring tools devised for non professional designers.For the same reason developing a solution for game exploration that is applicable to a large set of games based on sonification techniques is challenging.Nonetheless some of the concepts described above have been applied in our approach (e.g.auditory icons), as it is described in section 4.

INTRODUCTION OF SONIFICATION IN THE <E-ADVENTURE> PLATFORM
This section describes how sonification has been introduced in the <e-Adventure> platform as a configurable feature.Both game engine and game editor were enhanced to include this option.Sections 4.1 and 4.2 describe the sound system implemented in the game engine.Section 4.3 describes how the sound system of the game engine can be fine tuned and configured for each game using the game editor.

General approach
Our goal is to help the user in locating a relevant element of the game scene by finding the origin of a sound.For that purpose we have used 3D positional audio.Currently this technology is used extensively in the game industry to provide an immersive acoustic environment complementing the visual presentation of the game scenes for several genres of games (e.g.FPS or First-Person Shooter).
In the 3D environment of a game some elements of the game world emit sounds: this can be ambient sound to enhance the feeling of realism, sounds associated with other game characters (e.g.enemies or allies) or sounds generated by objects to interact with.Depending on the distance from the game element to the player these sounds are perceived with a different intensity.Perception is also altered by changing the pitch to provide information on whether such elements (e.g.characters) are moving.
To apply 3D sound to 2D games we have mapped a 2D scene into a 3D environment according to the following rules, depicted in Figure 3: • The cursor takes the role of the player (or her/his avatar if it is a third-person game), and the sounds generated by the different interactive elements of the scene are perceived from its position.Thus, moving the cursor is equivalent to moving the player in a conventional game.We assume that the player is facing the scene and moves along the x-y plane.
• The interactive elements are located in the same plane than the cursor.The source of the sound they emit is their geometrical center.
In the current implementation of the system that sound is a pure sinusoidal tone, but other alternatives are under study.
With this configuration, the sounds generated by the different interactive elements of the scene and their changes as the cursor moves can be used to locate the elements the user is able to interact with.This approach has been developed using OpenAL (Hiebert, 2006) and the Lightweight Java Game Library (LWJGL) to integrate OpenAL in <e-Adventure>, which is a Java-based application.

Acoustic clues
After a first iteration the next problem was identified: with a "standard" audio system (classic stereo, 2+1 or headphones), sound source position identification based only on the loudness (volume) of the sound is fairly good with respect to x axis, but not so precise with respect to y axis.Therefore, more acoustic clues were introduced to improve the vertical precision.
We propose to use changes in the pitch of the sound (Sodnik et al., 2010) to help in that respect.Additional acoustic information, like confirmation and alarm sounds, can also help in finding the relevant areas of the scene or in relocating the player in case he/she gets disoriented.

Loudness
In typical 3D games, sound experiments an attenuation that is proportional to the distance from the sound origin, following a rolloff model that tries to emulate the natural behavior of the sound in open spaces.This type of attenuation can sound very realistic in 3D environments but, when applied to a 2D scene, tends to minimize excessively the sound loudness at a distance relatively short from its origin, also forcing the listener (cursor) to be located with great precision very near the sound origin for it to be heard at full volume.OpenAL provides several models for attenuation by distance, and we have chosen the linear distance clampled attenuation model as it permits to adjust the loudness variation in a controlled fashion.In this model, a reference distance needs to be specified, defined as the distance at which the sound is emitted with loudness equal to one (1.0).For distances lower than the reference distance the loudness is clamped to that same value, and for distances higher than the reference distance the sound is attenuated linearly from 1.0 to 0.0 at a maximum distance than can also be specified.
In our implementation, the reference distance is assigned a value equal to the minimum dimension (width or height) of the rectangular region associated to the sound emitted.If we want the sound to be heard in all places around the scene, the value assigned to the maximum distance should be the diagonal dimension of the game scene.
For experimented users wishing to try a more challenging setup a lower value can be used for this maximum distance, so not all the sounds can be heard all around the screen.

Pitch
We use changes in pitch to increment the vertical accuracy of the sound origin perception.
In our system the pitch range can be defined for every interactive element, so it is possible to experiment with different ranges without rebuilding the main program.The default values are 2.0 and 0.5, and the sound is emitted with a pitch that decreases linearly with the vertical distance from the sound source (see Equation 1).Therefore, if the cursor is far from the sound origin, the sound is emitted with a lower pitch that increases as the cursor approaches the sound source position (Rigas and Alty, 2005).
where: MaxVertDist is the maximum vertical (y) distance of the scene.

Confirmation sound
In 3D audio settings, sound is generated in a point of the 3D space and propagates omnidirectionally away from the origin.In this model, propagation takes the shape of a sphere, which reverts to a circle if we consider the intersection with the x-y plane.In pointand-click games each interactive element can be bounded by a rectangle.Therefore there could be areas of an interactive element in which the sound is perceived with different acoustic values although they belong to the same object.
A confirmation sound (i.e. an auditory icon selected by the game designer) when the cursor enters the bounds of an interactive element is also provided as an additional clue.When this occurs, the original spatial sound emitted for the region is muted and a new background sound is emitted with no positional information, so the user hears it in the middle of the audio system (speakers, headphones, etc.).

Alarm sound (out of bounds situation)
If no action is taken, when the cursor gets out of the game window the last sound emitted keeps sounding unaltered until the cursor re-enters the window.While this issue may not happen in modern computer games that run in full screen mode, it can be a problem in web-oriented games (like <e-Adventure> games) that are embedded in a web-page along with other contents.This effect can disorient users as the references to the source of the sound are lost.To deal with this problem two different solutions have been adopted: • When the cursor exits the game window all sounds are stopped and a new alarm sound (auditory icon) is emitted.As with the confirmation sound for reaching an interactive element, this alarm is emitted as a background sound with no positional information so it is heard in the middle of the audio system (speakers, headphones, etc.).When the cursor re-enters the game window this alarm sound is stopped and all the previously emitting sounds are resumed.
• If the user gets disoriented and she/he is not able to re-enter the game she/he can press a configured key to return the cursor position to the game window center.

Configuring the system with the authoring tool
Acoustic information can be added to the games at a reasonable cost for game creators in <e-Adventure>.In its current implementation, the game designer only needs to select the confirmation sound file that each interactive element will emit when the mouse cursor is over.The designer can optionally adjust several parameters of the sinusoidal tone used to locate the interactive elements: • Range.Sound can be emitted in all the area of the interactive element (default) or in a subset, for special "secret areas" or "achievements" in the game.This is important for game designers to keep the engagement level of the game as highest as possible for players with limited vision.
• Maximum (default 2x) and minimum (default 0.5x) pitch modifications for the sound, with respect to base one.
For the spatial audio to work properly, sound files have to comply with the following requirements: • Monoaural.3D audio API used in the implementation does not work with stereo or multichannel sounds.
• Seamless: When activated, the sounds of the different areas are emitted constantly, so they repeat in an endless loop.To avoid audible clicks when repeating the sound, the sound files have to be processed so that the amplitude of the waveform at its end has to be equal to the amplitude at its beginning.

APPLICABILITY
Improving point-and-click games accessibility using general non game specific sonification techniques is a very challenging problem.Now we are determining the applicability of these techniques in <e-Adventure> games.We have done initial tests with limited vision users with promising results but more comprehensive evaluation is needed to better determine in which games it is applicable and what the limits are.For instance, to determine the maximum number of interactive elements of a scene for which this approach is valid.
To evaluate our system we are creating a test application consisting of a variable number of scenes with 1 to 5 interactive elements in each of them.These interactive elements are randomly generated (size and position), non-overlapping and emit a sound covering the entire scene.The objective is to locate a specific interactive element of the scene ("exit"), identified by a door sound.After finishing the test, the user answers a survey questioning several aspects of the sonification system like usefulness, difficulty or impact in the game atmosphere.
Preliminary results show that, after a short training period, users are able to locate the "exit" element easily for scenes with a low number of interactive elements.Next step in the project is to do a comprehensive evaluation of the approach in collaboration with the Spanish National Blind Organization (ONCE; Organizacion Nacional de Ciegos).

CONCLUSIONS AND FUTURE WORK
In this paper we have presented our approach to use general sonification techniques to improve point-and-click game accessibility for limited vision users.The main goal is to seek low-cost general techniques that could improve the accessibility of computer applications in general and of computer games in particular.One of the main ideas behind our approach is to include the accessibility features in game authoring tools to make them reusable for different scenarios.
Our approach also has some limitations.The scope is narrowed to point-and-click games and as a consequence this approach is not applicable to many of the contemporary console and PC games that can be found in the market.However, it is still applicable to a full range of computer games like Flash or webbased games.Moreover, this approach could also be applied to other point-and-click applications beyond the gaming field.
Next step in the project is to conduct a more comprehensive evaluation in collaboration with the ONCE to evaluate our approach and to better determine the extent to which it is applicable.One of the more important aspects that will be evaluated is the impact that acoustic signals can have in the immersion and engagement of the games.Additionally, a different acoustic clue based on altering the timing of the sound is under study.
In addition several enhancements are being incorporated to the <e-Adventure> game authoring tool to facilitate the authoring process.For example, in the current version sound files have to be processed manually prior to importing them in the game to guarantee they loop seamlessly.As the game creator probably will not be a specialist in audio editing, an enhancement is planned to apply crossfading automatically if necessary in the sound file processing.

Figure 1 :
Figure 1: <e-Adventure> game editor.Red and green rectangles represent interactive areas of the screen.

Figure 2 :
Figure 2: Screenshot of an <e-Adventure> third person game.When the mouse cursor is placed over an interactive element (the notebook) visual feedback is provided (highlight).When the notebook is clicked a contextual menu with possible interactions is displayed (three options in this case).

Figure 3 :
Figure 3: Spatial audio in a 2D plane