Structured Exploration of the User Experience of Mobile Applications

Mobile application is an emerging area of interest, growing exponentially each year. Among the many mobile applications, only a few remain competitive. The success of a mobile application depends on its context of use and user experience (UX). In a pilot study, we tried to find and review user interface (UI) and UX problems by analysing user sentiments towards apps based on their text reviews. In addition to this, to fully understand the cause of the UI and UX problems, the ThinkingAloud method was used. Text reviews cannot be directly used for UI/UX problems and most of the problems were related to the planning-stage of the interaction cycle. Our approach may help app developers/designers to focus on particular UI/UX problems of applications.


INTRODUCTION
The increasing number of the mobile devices has escalated the use of Information and communication technology (ICT).Focus has shifted from Personal Computers (PC) to devices like smartphones and pads.The increasing number of mobile applications on the market provide alternatives for users to accomplish their tasks using apps.Users can easily search for and buy different apps from app stores.In many cases, app stores and web sites provide users with the opportunity to share their opinions about apps in text reviews.Research has shown that 81 per cent of Internet users do online search for product at least once.More than 71 per cent say that online reviews of the product have huge impact for their decision (Pang and and Lee 2008).This indicates how important reviews are.Sentiment analysis have been used for text review using a corpus to map words to the topic of interest (Sangani and Ananthanarayanan) but have not directly considered UI or UX.User reviews have also been used to find information about the need for improvement of features and new requirements (Guzman and Maalej 2014).In this paper, we present the result of a pilot study, exploring how sentiment analysis of user reviews can be used to indicate UI/UX problems.

METHODOLOGY
Initially we selected six apps: Temple Run, Google Map Google Map, Viber, Messenger, Skype, and YouTube.These were considered as 'good' apps by reviewing information provided on-line.We collected data using an online review tool available from https://www.appannie.com/for each of aforementioned apps.Data was collected in January 2016 and more than 600 reviews were collected for each app.We used Python to convert reviews into a corpora.Each corpus was analyzed and classified into positive and negative sentiment words.Due to differences in sample size i.e. number of reviews, we used percentages i.e. the total number of words used from reviews for each app divided by each word frequency of that app and multiplied by hundred.From the corpus of each app, we listed positive and negative sentiment words.For each app, we summed the percentages of positive sentiment words and negative sentiment words.By doing this we get an indicator about what apps that have more positive sentiments from users and which have negative sentiments.We found repetition of common positive and negative sentiment words used among reviews of the six apps.Our objective for finding common words was to explore whether apps have common UI/UX problems related to the words used.Additionally, some sentences provided ambiguous results, containing either a positive sentiment with a negative meaning or vice versa.
For the second stage, we selected two apps that were considered 'bad' apps.Only two apps were considered because user text reviews for these apps were very few compared with 'good' apps.The purpose of this was to find whether users used the same positive or negative sentiment words for text reviews as exemplified by the 'good' apps.
For the third stage, we selected two apps and their web-based desktop versions and compare the UI/UX problems between them.At every stage we used the corpora and sentiment analysis approach.
We then analyzed sentences associated with positive and negative sentiment words of two apps and their desktop web version.Our target was to find UI/UX problems associated with positive and negative sentiment words and compare them.To understand the psychology of each UI and UX problems and to know more about the causes of the problems, we used the Thinking-Aloud method.Five participants took part in this session.All five were familiar with smart mobiles and apps.For the final stage, a detailed analysis of the Thinking-Aloud sessions was done.The purpose of the detailed analysis of Thinking-Aloud for apps and their desktop web versions was to compare the UI and UX problems using User Action Framework (UAF) (Andre et al. 2001) (Hartson and Pyla 2012).

DISCUSSION
Our research shows that good review apps have a more positive sentiment percentage than a negative sentiment word percentage.Text review sentiments and star ratings are not directly related, with a higher star rating than would be expected by the text review, as in (Maks and Vossen 2013).

Figure 1: Apps Sentiment Words Percentage
From the Figure 1 we can see that not all positively rated apps have a higher percentage of positive sentiment words.From Figure 1, it is clear that users have more positive sentiments for game apps, which is a bias.Results show that highly rated apps may not have more high positive sentiment words than negative.We only got eleven common positive sentiment words from the six apps which are love, great, best, good, awesome, amazing, simple, cool, thanks, enjoy, like, well and happy.Only four negative sentiment words were found common among six apps which are hard, crash, bad and fix.Positive sentimental words like "useful" and "helpful" were found for five apps except, Temple Run.These words may be missing from Temple Run due the app being in a different domain.
The results show that to compare different apps, we also need to consider the domain.We did sentiment analysis of two low rated apps; American Airlines and Fingerprint Login.The American Airlines App positive sentiment words were 0.075 per cent and negative sentiment words 1.4 per cent.Fingerprint Login positive sentiment is 1.5 per cent and negative sentiment words 1.19 per cent.This indicates that it is not necessarily that badly rated apps have a higher negative sentiment words percentage than positive sentiment words.Common positive sentiment words for these two apps are like, thanks, great and glad and negative sentiment words are worst and awful.
Analysing sentiment words associated sentences did not provide direct insight into UI and UX problems compared to technical problems.We found the results useful to indicate initial UI and UX problems related to apps.UI/UX issues relating to Gestalt psychology, system feedback, desirability and navigation were found.Sentence analysis showed that a further analysis was needed to understand the UI/UX problems.To do that, we used the Thinking-Aloud method.Before carrying out Thinking-Aloud sessions, we selected two apps and their corresponding web sites for desktop use, aiming to detect differences in UI/UX related problem.We wanted to see if the different and common problems associated UI/UX were related.Google Maps and YouTube were chosen for this.The reason for choosing Google products is they are widely deployed and freely available.Our results show that Google Maps web version has 1.96 per cent positive sentiment words and 1.06 per cent negative sentiment words percentage.YouTube's webpage version has 3.04 per cent of positive and 0.8 per cent negative sentiment word percentages.From the data we can say that YouTube's website is better than the app, in terms of sentiment words.Our sentence analysis shows that the web-version for desktop have few UI and UX problems compared with the app version.Thinking-Aloud performed on Google Maps and YouTube show that most problems related with web were trivial but problems related with apps are more serious in term of UI/UX problems.The results indicate that designers need to work on the planning-stage of the interaction cycle and cognitive affordances (Lewis 1982).The Thinking-Aloud sessions indicated that users might have different perceptions of the same problem.

CONCLUSION
This study contributes to finding UI and UX problems using sentiment analysis, especially for mobile applications.The result shows that analysing user text reviews using sentiment analysis cannot be directly used for finding deep UI and UX problems related to mobile applications, but can be a useful indicator for UI and UX problems.The Thinking-Aloud method was used to further investigate the problems.In the Thinking-Aloud sessions, data was carefully analysed by using Hartson's UAF (Hartson and Pyla 2012), showing that UI and UX problems were mainly in planning phase of the interaction cycle.The results also show that the users' mental models need to be studied more precisely for providing a better user experience with mobile apps.