SmartAbility: Detection of reduced physical abilities through smartphone sensors

This paper describes a developed Android application to accompany the SmartAbility Framework, 
in order to recommend technologies for people with reduced physical ability. The framework is a 
culmination of previously conducted research, including requirements elicitation and technology 
trials. The application is based on a previous prototype version that required manual input of user 
abilities. The presented version detects the action that users are able to perform independently 
through the adoption of sensor technologies that are built-into Android devices, such as the 
accelerometer and step counter. The knowledge contained within the framework is subsequently 
used to derive recommendations of technology that could be suitable for the user. Future 
enhancements to the application will enable complete automatic detection, without the requirement 
for manual input. The exploitation of the SmartAbility application is anticipated to increase 
technology awareness amongst the user community, as well as providing a means for assistive 
technology manufacturers to promote their products.


INTRODUCTION
The SmartAbility Framework originated from previous research into assistive technology in the form of requirements elicitation, technology feasibility trials, controlled usability evaluation and validation (Whittington et al. 2015). Assistive technology is an ever evolving market (Gallagher and Petrie 2013), as one billion people worldwide have reduced physical ability (The World Bank 2017) that affects their interaction with society.
The framework was developed by considering analysis of physical conditions through the International Classification of Functioning, Disability and Health Framework (ICF) (World Health Organization 2001), where it was realised that physical conditions can affect users differently, depending on the severity. This became evident in an experimentation involving iOS Switch Control (Whittington and Dogan 2016), as the Range of Movement (ROM) of the neck was the main determinant for the suitability of Switch Control. Therefore, it was established that technology suitability is dependent on the actions that users can perform independently, i.e., user abilities. The aim of the framework is to recommend technologies and interaction mediums based on those user abilities. The SmartAbility Framework consists of four elements: Physical Conditions, Abilities, Interaction Mediums and Technologies, fully described by Whittington et al. (2018). Validation of the framework was performed in two phases involving people with reduced physical ability and domain experts from healthcare and technology. A prototype Android application was subsequently developed to enable manual input of user abilities through a touch interface, in order to obtain recommendations of technologies. This interaction method had the disadvantage of potentially being challenging for users with reduced finger dexterity. This paper focuses on the development of the second version of the SmartAbility Android application that automatically detect user abilities through the utilisation of existing ability-based sensor technologies, integrated with smart devices, e.g., accelerometers, gyroscopes and step counters. An Android application was developed due to the prevalence of documentation regarding sensor technologies in the operating system, compared to other mobile operating systems such as iOS.

RELATED WORK
It was necessary to consider physical conditions, abilities, interaction mediums and sensor technologies, in order to develop the framework and subsequent Android application.
The worldwide standard for disability classification is the ICF (World Health Organization 2001), which establishes a standard language for defining disability. Associated research conducted by Andrews (2014) established that the motor control aspects of disability can also be classified using the Downton Scale and highlighted the impairments that could be potential contraindications of the disability (i.e., limited movements). ROM (Keilhofner 2006) was also discovered to be an effective method of measuring the movements that users can perform independently. In relation to daily activities, Gates et al. (2015) performed a study to quantify the acceptable ROM. The results of this study will contribute to user abilities in the SmartAbility application by defining the levels of ease of action in terms of Easy, Difficult and Impossible, for each ability. The movement will be considered 'Easy' when the user can meet the defined range by Gates et al. (2015), 'Difficult' when the user can only meet the boundaries of this range, and 'Impossible' when no movements are detected. For vertical head movements, the range is between 0º and 108º, horizontal head movements between -65º and 105º, elbow movements between 0º and 121º, shoulder movements between 0º and 118º (Macdemid et al. 2014), wrist movements between -40º (Flexion) and 38º (Extension) and ankle movements between 0º and 56º (Brockett and Chapman 2016).
A systematic literature review into sensor technologies of smart devices, highlighted that the built-in accelerometer and gyroscope can provide an accurate measurement of the motion, location and orientation of the device. Accelerometers measure axis-based motion to determine the direction of the device and are constructed from microscopic crystal structures that become stressed when exposed to accelerative forces (Nield 2017). Using a standard sensor coordinate system, moving the device to the right, generates a positive X acceleration value, a forward motion results in a positive Y value and an upward motion is indicated by a positive Z value. However, as rotation cannot be determined by accelerometers (Virtual Reality Society 2017), gyroscopes are used to calculate the orientation of objects as angular velocity in revolutions per second (RPS). In smart devices, Micro-Electro-Mechanical Systems (MEMS) can detect angular velocity as rotations around the X, Y and Z axes (Ronzo 2018). It is possible to detect other user characteristics through sensors, including gesture recognition and detecting steps or movements, such as tilting, shaking, rotating or swinging.

SMARTABILITY ANDROID APPLICATION
The SmartAbility Android application will incorporate built-in sensors of smart devices that were identified as being feasible through identification of available programming libraries and the current functionality of mobile devices. The Android operating system was selected for the SmartAbility application, due to the available libraries compared to the Apple iOS operating system. The available libraries were mapped onto the user abilities in the framework, an extract of which is shown in Table 1. The choice of developing the application in Android is supported by evidence that over 80% of mobile device sales are Android supported devices (Statista 2017), thus resulting in a significantly larger potential user base, compared to iOS.
The application consists of a welcome screen, where the user is able to log-in or register and commence the evaluation of their user abilities. There is an administrative section to the application that enables new technology recommendations to be added and user accounts to be managed. Technologies can be added to the framework by including name, short description, external website hyperlink and an image. The required user abilities can then be mapped through a series of checkboxes. Every user of the application can register by providing an email address and password. The user accounts store the technologies that are recommended to a particular user, based on their abilities. Therefore, users can return to the application after the assessment is completed, in order to retrieve their recommendations. Additionally, users can have their recommendations sent to their email address.
After the user has either registered or logged in to the SmartAbility application, the user's ability to walk is firstly captured. This feature utilises the step counter and instructs the user to hold their phone against their chest whilst they walk for one minute, as shown in Figure 1. The application counts the number of steps taken in this duration. Following the completion of the assessment of walking ability, the application determines whether the user is able to smile for 5 seconds. To collect data in reference to facial responses and landmark detection through the RGB Camera, the faceDetector.Face Application Programming Interface (API) is used. The command getlsSmilingProbability() returns a value between zero and one, depending on the probability that the user is smiling. The third assessment determines whether the user can blink for 5 seconds and the getls(Left/Right)EyeOpenProbability() command establishes the visibility of the user's eyeballs and returns a value between zero and 1. The Android Voice-to-Text Translator is used for the fourth assessment, whereby the user is instructed to repeat a phrase that is shown on the interface. Android then determines whether the user's speech matches this phrase.

Figure 2: Significant motion sensors adopted to detect the user's ability to move limbs
A number of sensors on the Android platform allow the motion of the device to be captured, including acceleration, rotation and step detection. The motion sensors return a multi-dimensional array of sensor values for each defined sensor event, in terms of the X, Y and Z axes. The subsequent four assessments utilise the Android significant motion sensors in order to determine the user's ability to move their ankle, shoulder, wrists and elbows (as shown in Figure 2). These assessments are conducted by instructing the user to place the mobile device on top of the relevant limb, to perform the movements instructed on the interface. The significant motion sensor returns a value to indicate the extent of the movement performed by the user. The application is able to detect head movements through use of the getEulerY() and getEulerZ() commands, that enable detection of the current orientation of the user's head, by returning the current rotation on the Y and Z axes respectively (Moroney 2015). The commands getls(Left/Right)EyeOpenProbability() can be used to return a value between 0 and 1, dependent on a user's eyeball visibility.
The next evaluation implements the MotionEvent class to capture data regarding the user's finger interaction with the touch screen, when one or several fingers are placed on the screen. The detection is achieved by the GestureDetectorCompat and MotionEventCompat classes to override the onTouchEvent callback, which is initiated when contact with the screen is made (Android Developer 2017). The GestureDetector class is able to detect all common gestures, such as down, long press and fling. The user is requested to perform each of these tasks in turn, so that the ability to move their fingers can be assessed.
The user's ability to suck and blow is elicited by the subsequent assessment interface that instructs the user to suck and blow into the microphone of the device. This output can be collected by the Android Noise Threshold Sensor, where the threshold decibel level is set to determine the user's ability to suck and blow air. The strength of the user's ability to perform these actions can be determined through an increase in volume or amplitude. The final interface of the application is an assessment of the abilities that cannot currently be detected through built-in sensor technologies, including gazing, moving the tongue and the user's visual acuity (i.e. seeing). These abilities are obtained by requesting the user to select checkboxes to indicate whether the ability is Easy, Difficult or Impossible to perform, as illustrated in Figure 3.

Figure 3: Checkboxes to elicit abilities not currently feasible to detect through built-in sensor technologies
The data collected from the sensors, including the checkbox responses, is analysed. Based on the mappings between abilities and technologies defined in the application code, suitable technologies are recommended. These mappings were elicited from the knowledge contained within the original SmartAbility Framework. The suitable technologies are then provided to the user through an output interface, i.e. the technology name, description, image and website hyperlink, in order for the user to obtain further information. The technologies that are currently recommended by the Framework include a smartphone, head mounted display, eye tracking and switch, operated through a variety of compatible interaction mediums, including eye tracking, fingers and chin.

DISCUSSION
Built-in sensor technologies of Android devices enable the SmartAbility application to automatically detect user abilities. The application is based on the SmartAbility Framework that was derived, based on the knowledge obtained through previously conducted requirements elicitation, feasibility trials, controlled usability evaluations and literature concerning physical conditions, abilities and technologies. The key finding being that technology suitability is determined by user abilities rather than physical conditions. Prior to the development of the application, a systematic literature review was conducted to ascertain the capabilities of sensor technologies to detect user abilities (Whittington et al. 2018). The results of the review informed the design for the application that was based on the first version, which relied on manual input to detect the actions that users could perform. The second version enables these abilities to be detected mainly through automatic detection. However, some abilities remain, requiring manual input, e.g. tongue movements. The technologies that are recommended vary, depending on the individual. Therefore, there will not be a 'single solution to fit multiple needs', which is comparable to the 'One Size Fits All' Information Technology concept (Adams 2017).

FUTURE WORK AND CONCLUSIONS
The SmartAbility Framework and the associated application will continue to evolve through the addition of interaction mediums and technologies. This will ensure that SmartAbility remains relevant to the user community of people with reduced physical ability. The future development stages of the application involve the automatic detection of abilities that are not currently supported by built-in sensor technologies. In the future, the aim of the application is to detect all user abilities without manual input, except for registering and logging in. To maximise the number of potential beneficiaries of the framework, the application will be disseminated freely to the user community, manufacturers and domain experts. This will be achieved through digital distribution platforms or websites of organisations associated with reduced physical ability. Manufacturers will be offered a customised application, including logos, product images and descriptions, which will result in a promotional tool for their products.
The impact of the application will be measured through validation, in which the usefulness of the technology recommendations will be measured, by considering the abandonment rates (Leckie 2010) of the technologies after certain timeframes. The presented SmartAbility application therefore facilitates the automated detection of user abilities through the integration of sensor technology, including accelerometers, gyroscopes and step counters. It is anticipated that the continued application development will increase technology awareness amongst people with reduced physical abilities, resulting in an improved quality of life.