Automatic Detection of User Abilities through the SmartAbility Framework

This paper presents a proposed smartphone application for the unique SmartAbility Framework that supports interaction with technology for people with reduced physical ability, through focusing on the actions that they can perform independently. The Framework is a culmination of knowledge obtained through previously conducted technology feasibility trials and controlled usability evaluations involving the user community. The Framework is an example of ability-based design that focuses on the abilities of users instead of their disabilities. The paper includes a summary of Versions 1 and 2 of the Framework, including the results of a two-phased validation approach, conducted at the UK Mobility Roadshow and via a focus group of domain experts. A holistic model developed by adapting the House of Quality (HoQ) matrix of the Quality Function Deployment (QFD) approach is also described. A systematic literature review of sensor technologies built into smart devices establishes the capabilities of sensors in the Android and iOS operating systems. The review defines a set of inclusion and exclusion criteria, as well as search terms used to elicit literature from online repositories. The key contribution is the mapping of ability-based sensor technologies onto the Framework, to enable the future implementation of a smartphone application. Through the exploitation of the SmartAbility application, the Framework will increase technology amongst people with reduced physical ability and provide a promotional tool for assistive technology manufacturers.


INTRODUCTION
There is a growing market for assistive technologies (Gallagher and Petrie 2013) to improve quality of life, as one billion people worldwide have reduced physical ability (The World Bank 2017) that affect their daily interactions with society and the environment.Assistive technology can utilize smart technologies, which have proliferated over recent years including smartphones, tablets, mobile devices and SmartTVs.The involvement of the intended user community of people with reduced physical ability can be used to assess the extent to which these technologies can provide enhancements to quality of life.The SmartAbility Framework evolved from previously conducted requirements elicitation, technology feasibility trials, controlled usability evaluations and Framework validation (Whittington and Dogan 2016a).Requirements elicitation ascertained the tasks deemed to be challenging for people with reduced physical ability and technologies were evaluated to determine their potential to provide assistance with daily tasks (Whittington et al. 2015b).
During the development of the Framework, it was necessary to consider types of physical conditions through analysis of the International Classification of Functioning, Disability and Health Framework (ICF) (World Health Organization 2001).This resulted in the identification of the user abilities that could be affected by physical conditions.Through the conduction of a head tracking experimentation with iOS Switch Control (Whittington and Dogan 2016b), it was established that Range of Movements (ROM) was the main determinant of technology suitability.The Framework was subsequently developed in two versions that were validated by involving the user community of people with reduced physical ability and technology and healthcare domain experts.A prototype smartphone application for the Framework was developed, enabling users to manually input their abilities, in order to obtain interaction medium and technology recommendations.This paper firstly introduces the elements of the SmartAbility Framework and summarises the findings from a two-phased validation process, including the presentation of a conceptual model for the SmartAbility Framework based on the Quality Function Deployment (QFD) concept (Akao 1990).It then focuses on conducting a systematic literature review centred on ability-based sensor technologies in Android and iOS devices.These mobile operating systems have 95% of the worldwide market share (StatCounter 2018).The review answers the following research questions: 1. How can user abilities be detected through built-in sensor technologies of Android and iOS devices?
2. Which mobile operating systems support the monitoring of user abilities?
3. Are there user abilities that cannot currently be detected through Android and iOS devices?
4. How can ability-based sensor technologies be identified and mapped to the SmartAbility Framework?
An updated smartphone application is proposed that generates technology recommendations through the automatic detection of user abilities.

RELATED WORK
The development of the SmartAbility Framework considered the domains of physical conditions and abilities, interaction modalities, Human-Centred Design (HCD) and sensor technologies.The first element of the Framework was devised by analysing existing classifications of disability and the methods in which the abilities of the user could be measured.
It was necessary to review currently-available technologies through the conduction of feasibility trials and usability evaluations, to establish the Interaction Mediums and Technologies elements of the Framework.The Framework development followed an HCD approach by involving the intended user community of people with reduced physical ability and experts from healthcare and technology domains.

Physical Conditions and Abilities
The World Health Organization developed the worldwide standard for disability classification, known as the International Classification of Functioning, Disability and Health Framework (ICF) (World Health Organization 2001).The purpose of the ICF is to establish a standard language for defining disability, which is recognised by Kostanjsek (2011) as 'a complex interaction between the person and their environment', rather than characterising individuals.Andrews (2014) analysed the relationship between the ICF, Downton  (Keilhofner 2006) could be one indication of user abilities in terms of the movements that the user is able to perform independently.The typical method of measuring ROM is using a goniometer that provides an accurate assessment of the extent to which an individual can move their joints between 0º and 360°.Whereas in the SmartAbility Framework, ROM (and other physical abilities) are categorised into three Likert Scales of ease of action; Easy, Difficult and Impossible.
The development of the Framework is an example of ability-based design, where the potential of humans is maximised by focussing on their abilities (Wobbrock et al. 2011).The seven principles of the ability-based design concept defined by Kelley and Hartfield (1996) are: (1) Ability, (2) Accountability, (3) Adaptation, (4) Transparency, (5) Performance, (6) Context and ( 7) Commodity.The first relevant principle to the SmartAbility Framework is Ability, which advises designers to focus on ability and not disability, thereby identifying the actions that users can perform.Secondly, the Adaptation principle that recommends interfaces to either be self-adaptive or user-adaptable to suit users' abilities.The third related principle is Context, which refers to the capabilities to sense user abilities.The Commodity principle can also be applied to the Framework, as the technology recommendations should be low cost and affordable to the user community.

Interaction Mediums and Technologies
It is recognised that the user community of people with reduced physical ability typically have a more diverse ROM than able-bodied users (Biswas and Robinson 2008) and therefore, these types of users may require more sophisticated methods of interaction compared to traditional mediums such as touch-based.One potential medium is braincomputer interaction that can be achieved through an electroencephalogram (EEG) by attaching electrodes to the scalp.Alternatively, 'Sip 'n Puff' technology can be used to facilitate interaction through sucking and blowing for users with insufficient dexterity to operate joysticks.EEG, 'Sip 'n Puff' and other currently-available mediums were included in the Interaction Mediums element of the Framework, which were considered to be affordable and manageable for users with reduced physical disability.Sloan et al. (2010) identified that these factors are typical 'barriers to progress for assistive technologies'.A number of technologies exist that can be utilised as an assistive technology to, "increase, maintain, or improve the functional capabilities of persons with disabilities" (Assistive Technology Industry Association 2018).Many technologies provide interaction through a variety of mediums, including joystick, voice or eye-based interaction.Head-mounted displays are typically designed for use in sports, e.g. the Recon Jet (Recon Instruments 2018).However, these could be utilised as assistive technology for users who are unable to operate touch-based mediums.

Sensor Technologies of Smart Devices
Recent developments in smart devices have included the integration of sensors to detect the location and motion of the device and the physical characteristics of the user.Most smartphones have a built-in accelerometer and gyroscope that together provide an accurate measurement of motion and orientation.
Accelerometers measure axis-based motion to determine the direction of the device and are constructed from microscopic crystal structures that become stressed when exposed to accelerative forces (Nield 2017).The increase in stress levels produce voltages that are interpreted by the accelerometer, to calculate the velocity of the device in terms of speed and direction.A standard sensor coordinate system is adopted, whereby moving the device to the right generates a positive X acceleration value and moving the device forward generates a positive Y acceleration value.The Z acceleration value is generated when the device moves upwards and is calculated by subtracting the acceleration due to gravity of 9.81 m/s 2 from the overall value.A limitation of accelerometers is that it is not possible to accurately determine the rotation (Virtual Reality Society 2017), hence the requirement for gyroscopes.
Gyroscopes calculate the orientation of objects as angular velocity in revolutions per second (RPS).A common use of gyroscopes is in aircraft to determine altitude and position.Smart devices use small Micro-Electro-Mechanical Systems (MEMS) gyroscopes are used to detect angular velocity as rotations around the X, Y and Z axes (Ronzo 2018).
Smart devices also have the ability to detect the face of a user from real-time video and images.It can also distinguish faces at different orientations and expressions, and identify specific landmarks on the face.This is achieved on the Android platform through the Face Detection API (Nguyen 2016).On detection of a human face, the API generates a Face object which allows a number of methods to be performed, including the facility to obtain the position, width and the height of the face.The orientation and rotation of the face can be determined through the execution of the getEulerY() and getEulerZ() methods.Landmarks can be detected from the Face object, including the mouth, ears, nose and eyes, as well as whether the user's eyes are open and their mouth is smiling.
Other user characteristics that can be detected through sensors include; touch gestures (involving one or more fingers placed on the touch screen), as well as detecting steps and movements such as tilting, shaking, rotating or swinging.These characteristics can be captured through the accelerometer and gyroscope.

SMARTABILITY FRAMEWORK
The SmartAbility Framework aims to provide technology solutions to suit the abilities of the user.The Framework was validated in two phases involving people with reduced physical ability and domain experts from computing and healthcare background.

SmartAbility Framework Version 1
The Framework originated from a conceptual model consisting of four 'pillars'; User, Environment, Technology and Context.These pillars considered the factors that would be important when deriving technology recommendations.The User Pillar described the physical conditions and impairments that can exist amongst people with reduced physical ability.The Environment Pillar related to the locations that may be challenging and the Technology Pillar contained products that were currently available to be used as assistive technologies.The final Context Pillar highlighted tasks that could be performed by users.
The four pillars evolved into the first Version of the Framework, consisting of six elements: Disabilities, Range of Movement, Movement Characteristics, Interaction Mediums, Technologies and Tasks.This Version was validated at the Mobility Roadshow and with a focus group of domain experts who suggested a number of enhancements, including a reduction in elements, simplified terminology and enhanced mappings between elements.The methodology and development of Version 1 of the Framework is discussed fully by Whittington andDogan (2016a, 2016b), with a summary provided in section 3.1.2below.

Framework Validation
Validation was performed in two phases, (1) semistructured interviews of people with reduced physical ability and (2) a focus group of domain experts.
The semi-structured interviews were performed at the Mobility Roadshow that enabled the Framework to be validated with the intended user community, as well as manufacturers of assistive technologies and powerchairs.This phase of the Framework validation was conducted by 19 male participants and 16 female participants, aged between 12 and 75 years who had varying physical conditions, including cerebral palsy (n=7), rheumatoid arthritis (n=7), muscular dystrophy (n=4) and Post-Traumatic Stress Disorder (n=1).
A paper-based version of the Framework was completed by the participants where the initial elements described the effect on their abilities due to their conditions.The knowledge contained within the Framework provided recommendations from the final two elements.Following the recommendations, the participants completed a questionnaire that elicited their feedback on the Framework, through questions regarding their previous knowledge of technology and the usefulness of the recommendations.All 35 participants provided positive feedback on the Framework and stated that the recommendations were helpful to improve their awareness.The manufacturers provided excellent feedback through analysing the structure and content of the individual elements, the House of Quality (HoQ) model for the Framework derived from the Quality Function Deployment (QFD) concept and the data dictionary.The focus group contained six domain experts from technology and healthcare backgrounds, including software engineers, academics and nurses.Elaborated scenarios in the form of fictional characters were provided to the participants that were devised from the physical conditions of the participants at the Mobility Roadshow.However, the names of the characters were changed to maintain anonymity.The Framework was utilised by the domain experts in an identical method to the participants at the Mobility Roadshow, whereby recommendations were obtained by the completion of the initial two elements.The participants provided feedback on the elements by highlighting any additions or corrections.An open discussion was held to obtain general feedback on the Framework, which was deemed to be positive overall.Suggestions were made of potential domains where the Framework could assist the user community, including rehabilitation medicine for patients who have experienced traumatic events, paediatrics and increasing awareness amongst healthcare professionals.
The modifications suggested from the two phases included that the rationale behind the Framework was positive by focusing on the abilities of the user, rather than those that are not possible to perform.It was also suggested that the terminology should be simplified to enable ease of understanding for users without domain knowledge.Additional interaction mediums and technologies should be included.A reduced number of elements were recommended to simplify the Framework.

SmartAbility Framework Version 2
The second consolidated version of the SmartAbility Framework evolved from the validation of Version 1.The consolidated Framework consisted of a reduced number of elements: Physical Conditions, Abilities, Interaction Mediums, Technologies, that contained a mixture of colour coded symbols and Likert scales.The mappings within the elements were illustrated by distinct identifiers in the form of colour codes, symbols and checkmarks to characterise the unique elements.
A model (shown in Figure 1) was developed to supplement Version 2 of the Framework that was developed based on the House of Quality (HoQ) matrix of the Quality Function Deployment (QFD) tool.QFD is a quality tool that was developed as a method for capturing the 'voice of customer' to build products that consider customer satisfaction prior to development (Akao 1990).QFD has previously been applied to identify customer needs for public services in a smart city (Zawati and Dweiri 2016).The tool consists of four phases: product planning (known as the HoQ matrix), product design, process planning and production planning.The HoQ matrix contains six sections: Customer Requirements, Planning Matrix, Technical Requirements, Interrelationships, Roof and Targets.
The SmartAbility HoQ model utilises the symbols contained within each element that illustrate the mappings.The Customer Requirements section of the HoQ was replaced with the interaction mediums that correspond to the required abilities that are stated instead of the Technical Requirements section of the model.The centre of the HoQ describes the mappings between the Abilities, Interaction Mediums and Technologies.This also includes the target ranges that indicate the minimum abilities required to successfully operate an interaction medium.The Planning Matrix of the HoQ was substituted for the Technologies element that identified whether high or low agility, acuity or clarity were required for interaction.A data dictionary was developed to support the HoQ for the users without medical or technical domain knowledge, which provided definitions of the terminologies and symbols utilised in the model.Literature sources for each mapping were also included in the dictionary.

Framework Elements
The current Version 2 of the SmartAbility Framework contains four elements: Physical Conditions, Abilities, Interaction Mediums and Technologies.Each element contains images that describe the contents to assist users who may not be familiar with the terminology, as well as distinct identifiers to illustrate the mappings and characterise the elements.

Physical Conditions:
The element (shown in Figure2) identifies the range of conditions that can result in reduced physical ability, such as an acquired brain injury or cerebral palsy, with the aim of filtering them into generic categories.The conditions are mapped to abilities based on observations from previously conducted usability evaluations and existing literature.The input to the element is the physical condition of the user which establishes the abilities that could potentially be affected.This will then inform the Abilities element.

Abilities:
The second element considers the specific conditions of users that can affect their 'ease of action' of performing abilities, in terms of Easy, Difficult or Impossible.The element classifies the abilities into regions of the body and contains a traffic light style grading system.The three colours of green, amber and red creates a simple choice for the user that is universally recognised.The ease of action would be considered 'Easy' when the user can meet the defined ranges for daily living activities, 'Difficult' when the user is only able to achieve the boundaries of this range and 'Impossible' when the user cannot perform the ability.Gates et al. (2015) state that the range for vertical head movements is between 0º and 108º, between -65º and 105º for horizontal head movements and 0º and 121º for elbow movements.Macdemid et al. (2014) suggest that the range for shoulder movements is between 0º and 118º, between -40º (Flexion) and 38º (Extension) for wrist movement and it is recommended by Brockett and Chapman (2016) that ankle movements should be between 0º and 56º.
Users can select 'ease of action' which describes each of their abilities, to form the input to the Interaction Medium element and enable recommendations to be made.
Interaction Mediums: This element describes the ability to interact through different mediums and is illustrated in Figure 1.The QFD approach was adopted to devise two symbols that indicate whether the ability is mandatory or non-mandatory for each interaction medium.A white-centred orange circle infers a non-mandatory ability, whereas a solid circle states that the ability is mandatory.In order for interaction mediums to be recommended that have non-mandatory abilities, the user must possess at least one of the abilities.Only the interaction mediums that are suitable for the user are the output of this element and these form the input to the final Technologies element.The mappings between the elements achieve the recommendation of interaction mediums and technologies based on user abilities.As an example, a 'Sip 'n Puff' interaction medium relies on the user to have the ability to both suck and blow and the interaction medium is compatible with smartphones, tablets or momentary switches.The SmartAbility Framework would therefore only recommend 'Sip 'n Puff' for users who are able to suck and blow.
An example of using the Framework would be for an individual with cerebral palsy who is not able to bend their fingers or speak.They would complete the Abilities element, indicating that these abilities are Impossible for them to perform.The Framework would therefore recommend brain, chin, eye, foot, head, Sip 'n Puff, and tongue interaction mediums, which can be used to control a smartphone, tablet, head mounted display, eye tracker, head tracker, electroencephalogram and switch.

AUTOMATIC DETECTION OF USER ABILITIES
Based on the knowledge and mappings contained within Version 2 of the Framework, a prototype Android application was developed.The application elicited the inputs from the user through the completion of a form, whereby the user selected their 'ease of action' for each ability by tapping a green, amber or red radio button, as shown in Figure 4.The output of the application was recommendations consisting of a list of suitable technologies and interaction mediums, supplemented with descriptions, images and external website hyperlinks.The application had the disadvantage of requiring manual input through touch-based interaction, which could be challenging for people with reduced finger dexterity.To evolve the smartphone application further, an updated version is proposed that utilises the sensors built into smart devices, in order to capture user abilities with less manual input.

Methodology
A systematic literature review was conducted into the existing sensors of smart devices, in order to 0answer the four research question stated in Section 1.There are examples of previous literature reviews conducted into the application of smartphone sensor technologies, to monitor fall detection in patients, where accelerometers are used to detect sudden changes in velocity (Vo et al. 2012, Yi et al. 2014).However, a systematic review had not been conducted to map sensor technologies that can detect reduced physical ability.The Scoping and Planning stages of the systematic review were followed, as defined by Siddaway (2018).Inclusion criteria was established, including:  The literature aligns with the research questions.
 The sensor technologies can be implemented on a smartphone platform.
 Statistical results are provided that give indications of the reliability of the sensor technology.
 The literature has been written within the last 5 years.
Example criteria for excluding literature comprised the following: sensor technologies not being compatible with smartphones, monitoring psychological behaviour or being older than 5 years.Once the inclusion and exclusion criteria had been defined, searches were made for literature on a variety of online repositories, including Google Scholar, IEEE Xplore Digital Library, ACM and international institutions and Universities.Search terms were established, including; 'smartphone internal sensors', 'smartphone accelerometer', 'smartphone gyroscope', 'android movement sensor' and ' iOS face detection'.For each search term, a list of relevant publications were derived and utilised to produce the results described in Section 4.2.

Results
Table 1 identifies the sensors that would feasibly detect user abilities, including the accelerometer, gyroscope, voice to text translator, face detection, significant motion sensor, gesture detector and step counter.Based on the review, the Android operating system had the greatest number of algorithms available that were able to detect the necessary user abilities.The iOS operating system could only detect certain abilities, such as the motion of limbs and walking via the step counter.

The
Android Face Detection Application Programming Interface (API) was able to identify human faces in real-time video images and distinguish faces at different orientations and expressions (Nguyen 2016).Therefore, it could be utilised to detect abilities involving the head and facial features.Specific landmarks on a face could be detected, including the eyes, nose and lips.When a human face is detected, the API instantiates a Face object with spatial data in order for landmarks to be added.The position of an object can be ascertained via the methods getPosition(), getWidth() and getId().The orientation of a face can be obtained by executing the methods getEulerY() (to return the rotation of the face around the vertical axis, i.e. looking left or right) and getEulerZ() (to return the rotation of a face around the Z axis, i.e. tilting the head sideways).The Face Detection API is able to return the coordinates of specific landmarks in the face through the getLandmarks() method.The current landmarks that can be detected by this method are: mouth cheeks, ears, nose and eyes.The API can detect a number of states, including whether each eye is open (through the getlsLeft/RightEyeOpenProbability() method) and whether the user is smiling (using the getIsSmilingProbability() method).Each of these methods returns a value between zero and one to determine the probability of a particular facial feature being present, with zero being the lowest probability, and one the highest probability.These three methods could therefore be used to determine the users' blinking and smiling abilities.
Both the Android and iOS operating systems have the ability to detect motion through utilisation of the built-in accelerometer and gyroscope.This enables movements to be measured, such as tilting, shaking, rotating and swinging.The motion sensors can be accessed through multi-dimensional arrays that contain values of the movement in the X, Y and Z axis (measured in m/s 2 ), rotational movements (measured in rad/s) and step number (no units).
Gestures can be detected by both operating systems in two phases.Firstly, data is gathered regarding the touch events on the screen (i.e. the number of fingers that have been placed on the touch screen and their position) and secondly, the data is interpreted to ascertain whether the gesture is supported by the application.The gestures are represented by rotational vectors that detect angular changes in the fingers of the user.
It is currently not feasible to automatically detect sucking and blowing, eye gazing, tongue movements and the visual acuity of the user due to absence of the necessary sensors and algorithms.Therefore, the SmartAbility application would focus on detecting the abilities stated in Table 1.

DISCUSSION
The initial version of the SmartAbility Framework was developed based on knowledge obtained through literature review of physical conditions, abilities and currently-available interaction mediums and technologies.Technology feasibility trials and controlled usability evaluations were conducted to ascertain the technologies that were deemed suitable for people with reduced physical abilities.A key finding from the evaluations showed that technology suitability is determined by the ROM of the user, rather than their physical condition.1 illustrates the mapping of technologies onto the SmartAbility Framework.As a greater number of abilities can be detected with Android, it was concluded that the SmartAbility application to provide automatic detection for user abilities, would be developed on this platform.

FUTURE WORK AND CONCLUSIONS
The SmartAbility Framework will continue to be enhanced with additional interaction mediums and technologies that are mapped to the required abilities.The mappings will be devised through the conduction of further experimentations involving the user community of people with reduced physical ability.Based on the knowledge obtained from the literature review into sensor technologies, the second version of the SmartAbility application will be developed.The input to the application will consist of separate user interfaces for each ability that can be detected.The interfaces will request the user to perform a specific action and the relevant sensor will be activated to ascertain whether the user successfully completes the action.Once the user has attempted to complete all required abilities, a list of suitable interaction mediums and technologies will be provided.This will consist of descriptions, images and external website hyperlinks, to enable users to investigate the products further, potentially resulting in purchases.As these recommendations will vary depending on individual user abilities, there will not be a 'single solution to fit multiple needs'.This can be compared to the 'One Size Fits All' Information Technology concept (Adams 2017), which should be avoided by undertaking a personalised approach to solutions.It is envisaged that the SmartAbility application would evolve into a promotional tool for assistive technology manufacturers.Customised applications will be developed that include manufacturers' logos, product images and descriptions, thereby enabling products to be advertised to people with reduced physical ability.The two SmartAbility applications would be made freely-available to the user community to download via digital distribution platforms or through websites of organisations associated with reduced physical ability.The collaborations with these organisations would be imperative to maximise the number of potential users of the SmartAbility Framework.
Following the exploitation of the two applications, it would be necessary to conduct a further validation phase, to establish the usefulness of the technology recommendations amongst the user community.To determine the contribution to society of the Framework, the abandonment rates of the technologies (Leckie 2010) would be measured after specific timeframes, i.e. within one week, one month or one year.Secondly, another focus group would be formed through the involvement of domain experts and people with reduced physical ability, to obtain feedback on the applications.This would increase analysing the usability and performance through adoption of the System Usability Scale (Brook 1986) and NASA Task Load Index (NASA 2018).
The SmartAbility Framework was established through literature reviews to map user abilities to interaction mediums and technologies and the application of the seven principles of ability-based design.Based on the findings from the systematic literature review of ability-based sensor technologies, a proposed smartphone application will be developed to exploit the sensors that are currently built into Android devices.These include accelerometers, gyroscopes and step counters, to provide technology recommendations, without the requirement of manual user input.The exploitation and continued development of the Framework is anticipated to promote technology awareness amongst people with reduced physical abilities, resulting in improved quality of lives.
The underpinning knowledge of the Framework was obtained through previously conducted requirements elicitation (Whittington et al. 2015b), technology feasibility trials (Whittington et al. 2015a) and controlled usability evaluations (Whittington et al. 2015a).The evaluation results have been published in the IEEE Transactions in Human Machine Systems (Whittington and Dogan 2016c).

Figure 1 :
Figure 1: Interaction Mediums element Technologies: Each of the interaction mediums contained within the previous elements are mapped to compatible technologies, identified to be suitable for people with reduced physical abilities.These mappings were formed from literature reviews and the results of feasibility trials and usability evaluations.Six colour-coded symbols indicate levels of physical agility (motor skills), visual acuity and speech clarity required for successful operations.The coloured version of the Framework includes seven technologies; smartphone, tablet, head mounted display, eye tracker, head tracker, electroencephalogram and switch.Future developments of the Framework will include the

Figure 4 :
Figure 4: Prototype SmartAbility Application requiring manual user input

Table 1 :
Mappings of user abilities to smartphone sensors and algorithmsAutomatic Detection of User Abilities through the SmartAbility FrameworkWhittington • Dogan • Jiang • Phalp 8