Towards Ubiquitous Awareness Tools for Blind People

Blind people are deprived from a wide set of information about the surrounding environment. This is a reality for places, objects and people. In particular, if social norms are disrespected, the blind person is likely to miss out the presence and absence of people in the same room or passing by. Current mobile devices provide a bundle of sensors that are able to provide more information about its user’s whereabouts and people nearby. We focus our research on improving the implicit awareness these users have access about their surrounding environment. We interviewed 19 blind people to understand the limitations and needs they have in indoor and outdoor environments both in orientation and social tasks. Based on our findings, we developed a prototype system able to recognize people nearby and notify the user about their presence. A preliminary probe with 3 blind people revealed that such awareness is a requirement and shed light about novel scenarios pertaining recognition, augmentation and sharing of information about people, places and objects. Blind People, Pervasive Accessibility


INTRODUCTION
A sighted person is able to constantly scan the surrounding environment.Even if no particular attention is given, one can easily be aware of people passing nearby.This awareness is also patent when recognizing locations or objects.Conversely, a blind person has limited information about the people and things nearby.Concerning consciousness about people, blind people are subjected to others complying with the established social norms (e.g.identifying oneself when entering or leaving a room).As to locations and objects, blind people are generally familiarized with previously explored locations and paths but have problems with novel contexts and the dynamics of public outdoor locations and places.Once again, a blind person sees herself dependent on others to have more information about the surrounding environment and the novel items in it.There have been efforts to increase the accessibility to mobile devices [Kane 2008] and through their built-in sensors, enable explicit access to bits of the environment (e.g.color identification [Dominguez 2011]).However, these devices provide a lot more capabilities than the ones currently explored.Further, they are always available with its user, being aware of interactions, proximities and paths.
Our research is focused on improving awareness of the surrounding environment, both explicitly and implicitly, by taking advantage of the available sensors on mainstream mobile devices.In a sense, we intend to provide a replacement for a part of information that reaches a sighted person that it is usually denied to a blind one.In this paper, we present an interview study with 19 blind people shedding light about the expectations, limitations and needs of blind people in this context.Building on it, we present a mobile prototype system able to recognize people nearby resorting to Bluetooth.Further, the system enables adding notes to people which are then triggered when that person is nearby.The system was evaluated with three blind people working as an elicitation for further scenarios.The feedback received along with the desirable scenarios outputted from a focus group with the three participants is reported.Results showed that the target population welcomes higher degrees of implicit and explicit awareness tools, particularly late-blind people who feel to be lacking knowledge of their whereabouts.

RELATED WORK
Other projects have tried to bridge the gaps arising from the absence of sight through technology.Particularly, there have been efforts to improve orientation, location, object recognition and proximity (mostly to obstacles) awareness.[Ghiani 2008] presented Museum Guides, a multimodal and location aware system for indoor environments specifically designed for visually impaired people.It provides flexible orientation support and information based on their location.Similar approaches are becoming mainstream in museums around the world.[Sánchez 2010] presented a low-cost solution that helps users navigating in outdoor environments allowing them to receive information based on their whereabouts.[Nicolau 2009] devised a system for indoor navigation giving specific attention to landmarks while ignoring details in the paths.Other line of research as focused on identifying objects.[Dionisi 2012] resorted to RFID technology to tag and find objects particularly in a home environment.E-notes [Brown 1996] allows the user to place a stick-e note at a physical point of interest.When the user returns to that position, the stick-e note is triggered.Some other examples of usage are given by the authors: the adjacency of a person to other physical objects and when the temperature is below a certain level.The stick-e note approach offers a mechanism for the creation of contextaware applications.[Dominguez 2011] reported a system that allows a blind person to identify the color of an object by using a mobile phone camera.[Dumitras 2006] has provided a mobile text recognition system.Bigham et al. [Bigham 2010] have presented VizWiz, a tool that leverages automatic recognition systems in a first instance and then crowd workers to answer the user's doubts in context.The blind person takes a picture and writes a sentence about the picture.The authors report an average response time of 30 seconds.Pertaining social connections and communities, the work presented by [Stewart 2008] allows visually impaired people to generate orientation notes and share them between one another.This project takes the orientation to another level by adding a socially maintained online database containing information about Points of Interest.Similarly, OurWay [Holone 2008] allows people, particularly wheelchair users, to create and augment geospatial data.This user generated content provides a basis for computing satisfactory routes, from one location to another, matching the user's preferences and needs.
Although not aimed at blind people, PeopleTones is a buddy proximity system [Li 2008] resorting to wireless signals.Currently, several mobile applications resort to location-based services to understand proximity between friends and acquaintances 1 .
1 http://highlig.ht/(Last visited on 14/06/2013) Most of the aforementioned applications focus on very specific problems (e.g.color identification) and are so deemed to be performed on demand.The broader ones, like VizWiz, are highly explicit.Awareness of the surrounding environment by sighted people is not intrinsically on demand.Herein, we explore the need for awareness tools for blind even in situations where they do not perform it explicitly.

UBIQUITOUS AWARENESS TOOLS
Current mobile devices, particularly since the advent of the iPhone and VoiceOver2 , are providing blind people with access to social, leisure and productivity applications as never before.Other operating systems and devices followed.Still, these gadgets enclose a set of powerful localization, communication, storage and processing tools that may give place to richer context awareness tools for blind people.In particular, there is a lacking effort in providing tools for implicit awareness: cues on the surrounding environment in context.

Preliminary Interviews
We performed an interview study with blind people aiming at understanding the limitations and needs they face in social environments.The interview covered a basic profile and mobile usage set of questions along with others more focused on scenarios where awareness of the surrounding is limited.In particular, we intended to understand how blind people deal with novel and known settings, with the dynamics of social environments and with context awareness in general.The interviews were quite broad and aimed mostly at eliciting areas where research is needed.

Participants
Nineteen (19) blind participants with ages ranging from 18 to 50 were recruited from a formation centre for blind adults.The degree of education varied from having the 4 th grade to college graduates.All participants use a mobile phone daily, and, although 10 out of 19 had used a touchscreen before, only one owns one of such devices.Eighteen users own devices with Nuance Talks3 screen reader while one participant uses Apple's VoiceOver.None of them uses GPS or cameras.The most commonly used applications are the basic communication ones, contact management, clock and alarm.Only one participant used social networks on the mobile device, two usually resorted to Bluetooth file sharing.

Main Findings
Most of the interview revolved around awareness questions particularly pertaining orientation in new and old environments, difficulties faced with social interactions and workarounds used in their daily lives.We asked about what sources of discomfort they are usually faced with and which of them are more disruptive.The main cause of discomfort was the lack of knowledge about the people surrounding them, both whom and how many.Interestingly enough this aspect was majorly stressed by late blinders.We also questioned them about how aware are they about people in their surroundings; if they can easily perceive when someone leaves or enters the area.Although all agreed that this is a common flaw, part of the users stated to be sometimes able to perceive who is entering the room because they recognize the sound made by movement patterns of people they daily socialize with (e.g., the way the person handles the white cane).The majority stated that they sometimes understand when someone passes by or sneaks in/out but they cannot understand who that person is if she doesn't introduce herself.In general, all participants stated that one of their first actions when reaching the formation centre was to ask the receptionist about the presence of their closest friends and if he knew about their whereabouts.
Early-blind people, on the other hand, reported to be comfortable with the unawareness about people nearby.On the other hand, despite their experience, they reported to be uncomfortable with the lack of knowledge about their location and the spatial relationship to objects and places around them, in places that they don't know well or that had things moved.Some users reported that they could infer useful information from what they hear but that they felt lost when there was too much noise, disrupting their sense of navigation and perception.As expected, users reported that new environments were more demanding in regards to attention and also the need to search for reference points.One participant stated that he commonly finds safe returning points where he knows he can back up to if he gets lost.The amount of free space in these new environments is also a factor; they claim big open spaces are harder to navigate as well as cluttered places, with too many objects, which damage the acoustics of the room and thus damage echolocation.When questioned about the impact in changes in the environment, most participants pinpointed misplacing or moving objects out of place.Pertaining note taking and prompts, the most techsavvy participants stated to resort to the mobile device to store notes or reminders.
All participants stated to be eager to have deeper knowledge about their surroundings although some stressed that they would desire the tool to be inconspicuous and subtle.

Proof-of-Concept Prototype
Based on our findings, and taking in consideration the state of research, we focused our attention on awareness about people and information about them.Our goal with this prototype was to elicit new scenarios and, together with the users, find new settings where mainstream technology could support them by adding layers of information to the knowledge they have about their surroundings.A prototype system was developed for Android devices.The user interface was built to provide gestural access to items in the menus (as in [Kane 2008].Text input was also provided with a navigational approach [Guerreiro 2008].All audio feedback was offered through SVOX Classic TTS.The application features a proximity detection system that can tell about people in the surrounding environment and a notification system that allows users to create notes and be warned at their convenience about them.

Recognizing People
To allow for the detection of people nearby, the system uses Bluetooth as a discovery mechanism.Using the unique MAC address of each device, the application can associate that identifier to any contact that is present in the user's phone.When using the application, it periodically searches for devices in its vicinity; when one device is detected it goes through the application database to search for its identifier to figure out if it belongs to someone known to the user or not.The user can navigate through the list of recognized and unrecognized devices, associate identifiers to contacts in the phone, and see when someone was nearby.The device filters fast on and off device detections trying to avoid cases where users are simply on the edge of detection range or are simply gone for short periods of time.Notifications are offered via vibration patterns.At this point, personalized patterns have been planned but not implemented in this version.The obvious drawback of the system relies in the usage of Bluetooth and the need for people to have it in discoverable mode.However, our goal here was to assess the benefits of such an approach.

Adding information to people
To augment the information stored about people nearby, we included the possibility to append notifications to contacts.The system allows recording audio notes or writing text notes.Whenever a device with associated notes is detected the feedback to the user is slightly different (different vibration pattern).The user is then able to receive feedback of people nearby and read notes associated with that contact.The inclusion of this feature in the prototype was performed to elicit behaviours of augmentation of information to people but also to assess the interest of having notes associated with places and objects.

PRELIMINARY ASSESSMENT AND PROBING
To assess the benefits of our approach and the aforementioned awareness tools, we performed a preliminary evaluation with blind people in the same formation centre.Our goal with this first prototype was to use it as a probe to elicit other usage scenarios.Also, we wished to understand if the users perceived benefits in such tools.

Procedure
Three participants were recruited to use the application during one full day at the formation centre.Fifteen other blind people were invited to take part of the study by maintaining their devices discoverable.The three participants started the day by joining us in a brief training session (30 minutes) to understand how to operate the application and the devices they were handed.They were instructed to use those devices during the day and to try out the application capabilities without disrupting their daily activities.They were told that the goal was to perceive the usage of such tools.They met us afterwards for a post-test interview.

Materials
Three devices running Android 2.3 were handed to the participants with our application installed and running.The other volunteers used their own devices with Bluetooth turned on and on discoverable mode.To focus the attention on the benefits and scenarios of these tools we previously associated all identifiers with the contacts in the phone.The application logged all usage data.

Results
All three participants managed to navigate the application without unexpected issues.Although we have previously matched identifiers and contacts of the volunteers, the participants were still notified about unrecognized devices (other people in or visiting the formation centre that might had the Bluetooth discoverable).They sought to understand the device owners and, when successful, proceeded to perform the matching on their own.On the other hand, given the decreased timespan of the evaluation session, the users experimented but did not use this feature much.
Overall, the users were satisfied with the fact that the application did provide them with information regarding people that were close to them and that it was able to identify them.They felt compelled to explore the area further in order to try and find people that were in the building.
Post-Test Interviews.These interviews revolved around the performance of the application and how the users interacted with it.Questions were made about the goals of the tool, feedback provided, along with open questions motivated by the observed usage in the test logs.
All participants reported to have experienced enriched knowledge about their surroundings.They stated to have an easier time identifying when people arrived a room.However, they also reported that there was a lag in the recognition that sometimes led to a late notification.They all found the information useful but also pointed out that sometimes it was too much to the point where they started doubting what exactly was the warning about.One user stated to be important to give relevance to people (e.g., white/black lists) and to be able to set modes (meeting and silent modes).
Group Discussion.We performed a meeting with all three participants and tried to ask them about scenarios where they thought these new tools could be used in or extrapolated to.We received several suggestions of scenarios and general usability of such a system.One of the stressed features was the need for more configurations.For example, one of the participants said he would like "a sub list of contacts where one could chose who to be warned about and who to ignore" and other added "we might just want it to be on certain times of the day", reinforcing the aforementioned need for restrictions.Also, to ease navigation and usage of the tool in a more intimate and subtle way, they asked for shortcuts and patterns that enabled them to recognize actions and act without having to push the device.
One suggestion of a scenario brought up in the discussion focused on outdoor scenarios and mentioned the ability to identify events or obstacles that were located on the street.This could be so that others that would follow after could be warned about navigation issues or, to a more general use, be able to leave tips of how to navigate in new environments like knowledge of reference points that they rely on.Another suggestion involved providing more information about places they go into; one user said "It would be nice not having to ask for what exists on a menu when reaching a restaurant".Another suggestion involved the stop signs and semaphores on the street: although some of them have a sound system to help out blind people most are not the best way to help; one user suggested to relay this information directly to the application.
Being able to share the notes was pointed out, even though they did not get to use them much.
They stated they would use them to warn their families or friends (similarly to what their sighted relatives already do on boards).They also stated they would like to set a reminder to buy something from the local food store if they pass by it.
In the overall, users revealed to find the application an important asset to scan their surrounding environment.Also, they saw opportunities in adding information to places and people.

CONCLUSIONS
While some information can be made accessible to blind people through mainstream technologies (e.g.screen readers), there are few projects that enable them to receive information beyond the explicitly acquired through audio or touch.This is a severe limitation that endangers the understanding of the surrounding environment and the competence in social arenas.To the loss of a blind person, there is a lack of systems that can offer him knowledge about who is around him, who is he passing by, who has left the room, which store is nearby or what is written in the news board at work.
In this paper, we shed light about the limitations, needs and desires of blind people pertaining awareness of the surrounding environments.Particularly, we put a focus on social environments seeking to provide more information about people around the target users.We developed a proof-ofconcept prototype enabling the detection of people nearby and the association of audio and textual information to those.A notification system was created to warn the users about proximity and pending notifications.A preliminary probe with target users suggested that such a tool is valuable and shed light about other possible scenarios featuring awareness to places, objects and people.
We will pursue the scenarios that include increased awareness, focusing on the ability to share notes and adding notes to places.Also, we will focus on the usability of such a system to allow personalized patterns and inconspicuous interaction.

ACKNOWLEDGMENTS
This work was funded by FCT, through project PTDC/EIA-EIA/117058/2010 and the Multiannual Funding Programme.