Correlating Cross-Platform Usability Problems with Eye Tracking Patterns Majrashi ● Hamilton ● Uitdenbogerd

Evaluating the cross-platform usability of multiple interactive systems has become increasingly essential. Despite eye tracking being used to supplement traditional usability assessment, there is little research on its use for cross-platform usability evaluation. Our exploratory study seeks relationship between eye-tracking metrics and cross-platform usability problems. We user-tested three cross-platform services and identified a set of usability problems. We separated the identified problems into traditional and cross-platform usability problems. Some of the cross-platform usability problems were associated with users’ eye-tracking patterns. We found that consistency on many levels is a major problem cross-platform and we recommend some considerations for evaluators to use as indicators to predict possible cross-platform usability problems.


INTRODUCTION
Users currently can switch between multiple interactive systems across devices such as desktop and mobile phone to complete inter-related computing tasks (Majrashi et al., 2015;Seffah et al., 2004).This brings the need for usability evaluation of multiple interactive systems when users carry out related goals across platforms.Hence, a new theme of usability called "crossplatform usability" or "inter-usability" has emerged (Majrashi and Hamilton, 2015).This theme concerns the ease of use of multiple user interfaces whereby users can reuse their knowledge and skills for a given functionality when switching from one device to another.
An eye tracker measures the direction of a user's gaze.User research based on eye tracking mainly relies on the eye-mind hypothesis, that is, when a person looks at a visual display while attempting a task, the location of their gaze point makes reference to the thoughts that are "on top of the stack" of mental operations (Just and Carpenter, 1976).Eye tracking methodology is used widely in usability studies as an important technique for evaluating traditional usability of user interfaces (Ehmke and Wilson, 2007;Jacob and Karn, 2003;Poole and Ball, 2005).
Several eye tracking metrics such as a fixation (the cessation of eye movements while focused on a specific area of the visual view) were employed by usability evaluators to analyze traditional usability.For instance, fixation duration and length metrics have been used to judge if a user encounters difficulty extracting information, or becomes engaged with a particular element (Jacob and Karn, 2003;Poole and Ball, 2005).However, the analysis of eye movement data is generally based on the interpretation of each individual usability evaluator (Ehmke and Wilson, 2007).Hence, Ehmke and Wilson (2007) attempted to establish a framework correlating eye-tracking patterns and usability problems with direct links between the two.We adopt Ehmke and Wilson's (2007) term "eye tracking pattern" to refer to single metrics such as a long fixation on a specific element, or combinations of metrics such as a specific scan path, which means a sequence of fixations and saccades (the quick movements between fixations).
Users can encounter usability problems when interacting with multiple interactive systems that are different to traditional usability problems (such as inconsistency between user interfaces across devices (Denis and Karsenty, 2004)).Since crossplatform usability is a newly emerging theme, there is limited work on using eye-tracking techniques to evaluate it.The purpose of this study is to contribute to the body of knowledge on using eye tracking for cross-platform usability evaluation.We aim to establish a correlation scheme to link cross-platform 2 usability problems to eye-tracking patterns.In our study, we use the term correlation to refer to the link between cross-platform usability problems and eye tracking patterns.Hence, it should not be confused with correlation as a statistical term that indicates the extent to which two or more variables relate to each other.
In this paper, we present an overview of traditional usability and cross-platform usability concepts, and the use of eye tracking techniques in usability studies.Next we provide details of our empirical study.Then, we present our findings and discussion.Finally, we conclude the paper and provide possible future direction of research.

BACKGROUND
In this section, we give an overview of usability, cross-platform usability, eye tracking methodology and metrics in usability studies.

Usability and Cross-Platform Usability
Usability is an important attribute influencing software quality (McCall et al., 1977).Practitioners use a variety of methods such as usability testing to identify usability problems in user interfaces, which can then be addressed to improve the usability of the product.
We use the term cross-platform service to refer to a collection of user interfaces for a single service, encompassing two or more computational platforms for interacting with the service.Users are currently able to access many cross-platform services using multiple platforms, enabling them to complete inter-related tasks.The new paradigm of users' interaction with cross-platform services contributes to the emergence of cross-platform usability as a new concept in the usability field.Denis and Karsenty (2004) defined inter-usability as "the ease with which users can reuse their knowledge and skills for a given functionality when switching to other device".Seffah et al. (2004) used the term horizontal usability to refer to the same notion.In this paper, if we use one of these terms, then we refer to the same concept.Cross-platform (horizontal) usability may have different requirements (e.g., consistency of the service across platforms (Denis and Karsenty, 2004)) when compared to traditional usability, which we also refer to as vertical usability in this paper.If a service has poor cross-platform usability, users may encounter problems.For example, after switching from one user interface to another, users may find it difficult to locate a specific user interface element if it is placed in a location that is different to its location on the previously experienced user interface.Denis and Karsenty (2004) have studied interusability of multi-device systems via theoretical analysis and interviews and found that service continuity as a factor could influence inter-device transitions.They argue that service continuity could have two different dimensions: knowledge continuity and task continuity.Their study suggests design principles such as inter-device consistency that can be applied to user interface designs to support both dimensions of service continuity.

Eye Tracking in Usability Studies
Eye tracking has been applied to numerous fields including cognitive psychology (Dahan et al., 2001;Rayner, 1998), and Human-Computer Interaction (HCI) (Goldberg and Kotval, 1999;Poole and Ball, 2005).According to our review, eye-tracking studies in HCI and usability can be classified into two groups based on their aims.The first group aims to understand user behaviour, including scanning, searching, and viewing behaviour.The second group aims to evaluate a specific user interface feature in terms of usability, such as forms, and navigation.
Studies in usability combine eye tracking with usability testing methods to learn something that cannot be discovered by only observing users while performing tasks (Ehmke and Wilson, 2007;Nielsen and Pernice, 2010).For example, if a user looks back and forth at some point on an interface, this can be an indication that a user might be confused or double-checking before moving on (Goldberg and Kotval, 1999).Eye movement data also can be used to learn which elements on a user interface are harder to recognize (Poole and Ball, 2005).Although eye movement data can help in understanding several usability issues on a specific user interface (Ehmke and Wilson, 2007), the analysis of eye movement needs to be informed by other collected data within the usability testing session, such as users' verbalisations.This is because a specific user behaviour can mean different things.For instance, long fixation durations on a specific area of a user interface can indicate difficulty in extracting information, or that a user is engaged (Just and Carpenter, 1976).
The think aloud protocol is used widely in combination with eye tracking in usability research.There are two main techniques of thinking aloud protocols.The first technique is Concurrent Think Aloud (CTA), in which participants are asked to verbalize their thoughts while they are doing tasks.The second is Retrospective Think Aloud (RTA), in which participants provide a description of their experiences doing the tasks after completing each task or the testing session.Guan et al. (2006) argue that participants can be distracted by concurrently thinking aloud.This can impact the quality of recordings (e.g., looking out of screen when thinking aloud), and thus give biased results.Hence, retrospective thinking aloud can be used in combination with eye tracking to avoid such issues.This is to collect user's thoughts after completing a task on the basis of a video recording of their task performance (Van Den Haak et al., 2003).

Eye Movement Metrics in Usability Studies
Usability researchers have used different eyetracking metrics to analyse eye movement data streams in the context of traditional usability.These include metrics related to fixation, saccade, scanpath, and gaze (the sum of duration of all fixations within a prescribed area).For example, Byrne et al. (1999) used "time to first fixation on object" to measure if it has better attentionattracting properties.Goldberg and Kotval (1999) employed a "number of saccades" metric to measure efficiency in completing a task.That is, a high number of saccades indicate wide searching, which means time inefficiency.They also used a "scanpath duration" metric, which can indicate high or less efficient scanning if a long duration is recorded.A gaze metric can be used to compare two areas of interest or attention spread between targets (Renshaw et al., 2003).Many other eyetracking metrics have also been used to gain insight into the usability of a specific user interface.For further extensive review of these metrics, refer to Ehmke and Wilson (2007); Jacob and Karn (2003); Poole and Ball (2005).
Eye-tracking metrics have been compared to other usability techniques such as expert reviews and performance measures (e.g., task completion times) (Ehmke and Wilson, 2007).Ehmke and Wilson (2007) conducted an exploratory empirical study to investigate the relationship between eyetracking patterns and usability problems.They could identify how user behaviour and their associated eye tracking patterns are related to specific usability problems.For example, one identified behavior recorded as "There are many fixations across the page without very long fixations and no fixations on some important link" was correlated with the usability problem of "A user is coming to a page that does not contain options relevant for his task".Ehmke and Wilson's (2007) work produced a general scheme for correlating eye-tracking data with usability problems, but they suggested the proposed 'pattern-problem' correlations in their study need to be validated using focused experimental studies.
Since cross-platform usability is a new usability theme, there are few studies on the use of eye tracking metrics in the context of cross-platform usability evaluation.In our earlier work (Majrashi et al., 2014), we developed an initial framework for using eye movement metrics in the context of cross-platform usability and provided some possible interpretations of general cross-platform usability based on some eye tracking metrics.For example, after transitioning from one interface to another, if the time to first fixation on a relevant target or component to the interrupted task is short, this could mean that the user would continue or resume the task seamlessly and easily.However, Majrashi et al. (2014) did not provide a direct correlation schema between user behaviours and their related eye tracking patterns cross-platform, or between cross-platform usability problems and cognitive processes.

METHOD
We carried out an exploratory study to find links between the eye-tracking metrics and crossplatform usability problems.We conducted a user study on three different cross-platform shopping services (Amazon, eBay, and Souq), explained further in Section 3.2.Users attempted horizontal tasks across two different user interfaces of a single service.A horizontal task refers to at least two inter-related subtasks in which each sub-task is attempted using a specific user interface.In this section, we describe our experimental design including users and setup, cross-platform services, horizontal tasks and procedures.

Users and Setup
The 31 volunteer participants were students at RMIT University in Australia.The demographics and service experience of the participants are shown in Table 1.

Table 1: Profile of study participants (numbers in the brackets refers to number of participants associated with each specific profile).
All participants had basic computer skills and had been using the Internet for more than 3 years.
Each participant undertook three horizontal tasks.
For each horizontal task a user needed to interact with two different user interfaces of a service.
Figure 1 shows the eye tracking lab setup.Eye movements to the different screen components across each user interface of a service were collected using a Tobii T60 eye tracker.This type of eye tracker uses the reaction of near-infrared lights in the eyes to enable non-intrusive tracking of gaze position on a computer screen.The device also allows the capture of detailed information on timing.
The eye tracker is integrated in a high-resolution 17-inch monitor (1280 x 1024 pixels).It has cameras and speakers, which are incorporated into the monitor to avoid distracting the participant.Using the eye tracker, participants interacted with two versions (desktop and mobile) of user interfaces when attempting each horizontal task.
For mobile user interface eye tracking, there are several different eye-tracking options now available, including head-mounted eye trackers (such as eye tracking glasses from SMI1 ), standalone eye trackers (such as Tobii's stand-alone eye tracker 2 ), and desktop monitor-based eye trackers, in which an emulator is installed in the eye tracker computer for the target smartphone (Tobii Technology Inc, 2010).However, we used an emulator on a black background on the Tobii's desktop monitor-based eye tracker (See Figure 1).While eye-tracking systems do currently exist for mobile devices, they still tend to have problems with accuracy (Hohlfeld et al., 2015).Cuadrat et al. (2012) listed many advantages of using desktop monitor-based eye tracker over other techniques.
For example, desktop eyetrackers can allow a high degree of head movement, and recording mouse inputs and keyboard interaction so there is no need for an external camera.Furthermore, the desktop monitor-based eye tracker technique has a high degree of accuracy and quality of recording when tracking the eye.In addition, the technique provides reliability for gaze data-based research results.
Figure 1 shows the laptop which participants used to answer online pre-test and post-transitioning questionnaires.We designed a pre-test questionnaire to collect demographic data.The post-transitioning questionnaire consisted of statements requiring responses from a five-point Likert format that ranged from "strongly disagree" (1) to "strongly agree" (5).This questionnaire was undertaken after a user completed two sub-tasks that involved moving from one device to another.The statements in the post-transitioning questionnaire concerned different aspects such as inter-platform consistency and continuity.An example of a statement from the questionnaire is "I noticed inconsistencies between user interfaces cross-platform".Users' responses to the five-point Likert scale statements were used as indications of possible usability problems.With the posttransitioning questionnaire, users could also report cross-platform usability problems.
There was an observer monitor (see Figure 1), which was used to show a "track status" box.The track status box allowed adjustments to be made between the participant's and the eye tracker's position.The eyes of the participants are displayed as two white dots in the track status box.As recommended by the eye tracker manufacturer, eyes should be in the center of the box and the distance indicator should display a value between 50 and 80 cm.During the tests, we frequently checked the status bar through the observer monitor to ensure it remained within the recommended value.In the lab, the observer sat next to and slightly behind the user, so as to not encourage conversation.We also ensured that the observer's monitor was not in the user's line of vision, to avoid distracting users.

Cross-Platform Services
For this study, we selected three cross-platform services, which are Amazon3 , eBay4 , and Souq5 .The selected services were all from the same domain (online shopping).The Amazon, eBay and Souq cross-platform services allow users to search for, browse and buy products online.Figure 2 shows the Souq cross-platform service as an example of the tested services.We refer to the services as A = Amazon, E = eBay, and S = Souq.

Horizontal Tasks (HT)
The horizontal tasks have been selected to be representative for all three services.There were three horizontal tasks.Users of each cross-platform service attempted exactly the same tasks.An example of a horizontal task is as follows: • HT1: Subtask 1: Using the (Amazon / eBay / Souq) service, find all items under the baby-feeding category.
• HT1: Subtask 2: Using the (Amazon / eBay / Souq) service, add a baby-feeding item to your shopping cart.

Testing Procedures
Each participant engaged in the following procedures during this study experiment: • Answer the pre-test questionnaire, • Attempt horizontal task 1: Perform sub-task 1 using desktop or mobile website, while eye movement is captured, Perform sub-task 2 using desktop or mobile website, while eye movement is captured, Answer the post-transitioning questionnaire, Retrospectively think aloud and provide a description of experiences while doing the two sub-tasks.(We recorded participants' thinking aloud).
• Attempt horizontal tasks 2 and 3 with the same procedure used for horizontal task 1.
Each participant attempted three horizontal tasks.
For each horizontal task, we asked participants to interact with a different cross-platform service (Amazon, eBay or Souq).
We also recorded notes of problems that participants encountered.
To minimise order effects, we used a basic Latin square design to rotate tasks, services and device order.This resulted in a block of 18 trials with different orders.After finishing the first eighteen trials, we restarted at the beginning of the block for the remaining 13 participants.

Identifying Usability Problems
We started our analysis by identifying traditional and cross-platform usability problems.Traditional (or vertical) usability problems are associated with only one user interface version (desktop or mobile).
That is, a user's interaction with one version of a user interface (such as the desktop version) is not impacted by their interaction with another version of the user interface (such as the mobile version).An example of a vertical usability problem is: an error message is too small and cannot be recognised from the mobile user interface.Such a problem is considered to be a vertical usability problem since it is not a consequence of interacting with another version of the user interface.A cross-platform (or horizontal) usability problem can be identified after a user transfers from one user interface to another.That is, it can only be identified in the transferred-to user interface since it is a result of interaction with multiple user interfaces.For example, if a user found a specific element in a specific location in a desktop user interface and attempted to find the same element in the same spot on the mobile user interface after the transitioning process but the mobile interface did not provide it at the same location, then this is considered a cross-platform usability problem.The reason behind identifying vertical usability problems in our study since we are studying cross-platform usability is that we needed to make sure there was no overlap between problems.Hence, we could correlate eye-tracking patterns that only link to cross-platform usability problems without having vertical usability problems affect eye-tracking patterns.
To identify usability problems (vertical and horizontal), we analysed the raw data (questionnaires, retrospective verbalisations, and observations) for each tested service.We then extracted all usability problems and separated these problems into vertical and horizontal usability problems.We also identified the user interface version (mobile or desktop) in which the vertical usability problem occurred.Similarly, we identified the order of user interfaces (desktop to mobile or mobile to desktop) when extracting a horizontal usability problem.This was to allow the separation of eye-tracking patterns associated with horizontal problems based on the sequence of user interfaces.
We aggregated the usability problems from different protocols for each participant, which resulted in many redundant usability problems identified per participant.Therefore, we deleted duplicates of vertical usability problems from the data for each participant per task.Duplicate problems across tasks and across participants were kept.

Correlating Horizontal Usability Problems with Eye Tracking Patterns
For the purpose of correlating usability problems (referring to horizontal problems, which are our focus in this study) with eye tracking patterns, we followed the methodology described by Ehmke and Wilson (2007).We first examined usability problems encountered by each participant for each task in each service and recorded the eye-tracking patterns associated with the problem as text-based descriptions.This resulted in three separate files (one file per task) of problems and their associated eye-tracking patterns per participant.The eyetracking patterns can be mixtures of eye tracking metrics, which are shown in Figure 3.
We described an eye-tracking pattern only during the period that the usability problem occurred, to ensure that the pattern is related to the identified problem.Furthermore, as we had already identified vertical usability problems, we became aware of these problems.That is, if there is overlap between a vertical and a horizontal problem, we did not record the eye-tracking pattern for the horizontal usability problem.For example, after transferring from one user interface to another, one user encountered two problems (vertical and horizontal) with the same user interface element, and reported the vertical problem as "font size is too small for menu item"6 , and the horizontal problem as "some menu items are not in the same order as the previous interacted-with user interface".In this case, we did not record the eye-tracking pattern for the horizontal problem since the vertical problem could have affected the users' eye movement pattern.

Figure 3: Eye tracking metrics used in descriptions of eye tracking patterns
We described the eye-tracking pattern based on users' behaviour on the transferred-to user interface.For the purpose of this study, we divided user behaviours into two types, which could be described as eye-tracking patterns.The first type is where the user behaviour on a page is related to a problem on the page as a whole.For example, Figure 4 shows part of user behaviour on the homepage of the Amazon desktop website, which can be described in association with the page.

Figure 4: Example of user eye tracking behaviour over the Amazon homepage
The second type is the user behaviour for a particular element (e.g., menu, icon etc) on the page if the problem was related to the element.For instance, Figure 5 shows user behaviour for the sort element on the search result page of the eBay desktop website, which can be described in association with the element.

Figure 5: Example of user eye tracking behaviour indicating an issue with the sort element on the eBay service page
The next step was that we generalised the horizontal usability problems.We examined the produced files of problems and their associated eye-tracking patterns, which we created for participants as discussed previously.We placed related problems and their associated eye-tracking patterns under generalised problem categories.
The generalisation process included all identified related problems for all participants for all tested services.For example, problems related to inconsistent labeling of items between the two user interfaces of each service were placed under one generalised problem category, which is "inconsistent labeling".
Finally, we aggregated eye-tracking patterns for each generalised usability problem based on the textual description of the patterns.That is, patterns with the same descriptions under the same usability problem were aggregated to produce a generalised pattern to reduce their numbers.In some situations, where there are differences between patterns descriptions, we kept all of them.Thus, some horizontal usability problems have more than one eye tracking pattern associated with them.If a specific eye-tracking pattern occurred only once under a specific problem, we did not include it.Overall, the horizontal usability problems were reduced to 11 generalised problems for all tasks, with 13 generalised eye patterns associated with them.

RESULTS AND DISCUSSION
In this section, we present and discuss our findings from our exploratory research.
In Table 4, we present the correlation of cross platform usability problems with eye tracking patterns (CPUP-ETP).We show the generalised problem (with number of occurrences) and description of the problems, and the eye tracking patterns associated with it.Two of the problems (1 and 10) are presented with their two associated eye tracking patterns.We excluded eye-tracking patterns that occurred only once among participants for a generalised problem.
Table 4 also outlines occurrences of eye tracking patterns associated with the problems for each order of interaction (desktop mobile and mobile desktop) across the tested services.As can be seen from Table 4, each cross-platform usability problem is associated with a sequence of patterns, and not with only a single eye tracking pattern.

Cross-Platform (Horizontal) Usability Problem (CPUP) Eye Tracking Pattern (ETP) in the Transferred-to
User Interface Frequency 7

Occurrences: 35
Most of users found homepages of Amazon and eBay services not consistent cross-platform in the design for example, in terms of design patterns, placing of elements and so on.These confused users and delayed continuing on tasks after the transition.

Occurrences: 7
Some users expected to find a specific item in a specific group in the transferred-to user interface based on their experience with the previous interacted-with user interface, but the item is missing from the group.
ETP3: Initially, many short fixations on the group where users expect to find the item, followed by many short fixations across the page.

Inconsistent grouping
An item is not grouped with other items from the transferred-from user ETP4: In the beginning, no fixation on the targeted item with fixations on items that are not within a S (D M): 3

Occurrences: 6
interface and grouped with other items from the transferred-to interface.Thus, participants when using the transferred-to interface were confused.
group across the page.This followed by further scanning on the different items, which are grouped together with focusing on the same area where the item is located in the transferred-from interface.

Occurrences: 3
Submenu items are placed in a different high-level menu item from each platform.This delayed some users from continuing on the horizontal task when transferred to the second user interface.
ETP5: Initially, fixations on submenu items of the high-level menu item selected from previous interacted-with user interface, which are irrelevant from the current user interface.This followed by a high number of fixations across the different menu items, and sub-items with some regressions to subitems of the high-level item selected from the previous user interface when targeted item is not found yet.

Occurrences: 5
The transferred-to user interface displays elements of a page in different orders compared to the same page from the transferred-from user interface.Thus, some users identified this as a problem as they expected to see user interface elements in the same order cross-platform.ETP6: In the beginning, when users looked for elements in the page, users fixated on element(s), which have approximately the same order that user experienced from previous user interface, and this then followed by a high number of fixations on the different elements across the page.

Occurrences: 42
The transferred-to user interface displays items / options (e.g., submenu items, sort options, search results) within a specific element / group in different orders compared to the order of the same items within the element from the transferred-from user interface.Many users identified this as a problem as they expected to see items within a specific element in the same order cross-platform.ETP7: In the beginning, when users looked for elements on the page, users fixated on an item, which has approximately the same order that the users experienced from the previous user interface, and then this followed by a high number of fixations across the different items within the element / group with some regressions to the item that has the same order of the item selected from the previous user interface when targeted item is not found yet.

Occurrences: 26
The transferred-to user interface displays elements (e.g., buttons, icons) within a specific page in different side of the page (e.g., top, bottom, right, left) comparing to where these elements are placed from the corresponded page in the transferredfrom user interface.

Occurrences: 2
The transferred-to user interface displays elements (e.g., buttons, and icons) in a page which does not correspond to the page where elements are placed on from the transferred-from user interface.For example, one platform displays a button from the homepage and the other displays it from another page, which is located deeply.ETP9: On the corresponded page, many short fixations on the place where a participant expected to find the targeted element based on a pervious interaction with the transferred-from user interface, and this then followed by a high number of fixations across the page on different elements on the page with a few changes of scanpath direction (topbottom, bottom-top).Further scanning and reading across the page also occurred before leaving the page.

Occurrences: 5
Some buttons/icons are different in shape from each platform, which confused some users.ETP10: Many short fixations on the area of the page where the buttons/icons are expected, looking for the same shape of buttons/icons based on previous interaction and then fixations on different elements across the page.This followed by regressions to the same area with further scanning with some long fixations on specific items.

A (D M): 1 E (D M): 1 E (M D): 3
ETP11: In the beginning, scanning across the items within a group where item is expected, with a short fixation on an item which has the same order of the selected item from the previous user interface.This sometime accompanied by changing in scanpath direction.Then, this followed by further scanning and reading on items.

Occurrences: 26
Different labels are used for the items (e.g., menu items) from each platform.These delayed users from continuing on migrated task easily.ETP12: Reading of the most relevant item that has different label, followed by a few further scanning on other items and regressions to the most relevant item with the different label.

S (D M): 2 A (D M): 1 11. Inappropriate user interface presentation for continuity Occurrences: 17
Relevant elements to the migrated task are less attractive from the homepage of the transferred-to user interface.Thus, some users were not able to restart the interrupted task quickly.In general, for each problem, most participants behaved similarly when experiencing the problem that resulted in shared eye tracking patterns.In addition, when users encountered a specific crossplatform usability problem with the desktop or mobile user interface but with a different order of user interfaces, they still behaved similarly.This can be seen with several problems that were encountered with different orders of user interface (refer to Table 4).For example, users encountered inconsistent ordering of items within a specific element or group cross-platform 42 times across the three services.Most (90.47 per cent) users who encountered this problem behaved similarly (refer to Table 4,problem 6).This means that the same user behavioural patterns were recorded 38 times with different orders of user interfaces.With the order "desktop mobile", the pattern was recorded seven times with Amazon, eight times eBay and 10 times with Souq.The pattern was also recorded 12 times with eBay, and once with Souq for the order "mobile desktop".Four users behaved differently when encountering the problem so we did not include their behaviours in Table 4 as mentioned previously.
Some of the observed eye tracking patterns are similar to patterns found previously.For example, the eye tracking pattern associated with problem 7 is similar to the pattern "Many short fixations across page where information is expected" identified by Ehmke and Wilson (2007).

Impact of the First-Interacted with User Interface on User Behaviour after the Transition
Based on pattern descriptions (the third column in Table 4), it is clear that all eye tracking patterns (except ETP2, ETP13), which were associated with the problems are influenced by the design of the previously interacted-with user interface.In Table 4 (the third column), the bolded text of eye tracking pattern descriptions indicates the impact of the previous design on the user behaviour when interacting with the transferred-to user interfaces.With problem 2, for example, users focused on a specific group initially, which is the same group in which users found the targeted element within the first interacted-with user interface, but the item was not provided within the same group on the transferred-to user interface.
With problems 7 and 8, many users initially fixated on a specific spot on the transferred-to user interface, because they expected to find the targeted item at the same spot as experienced on the transferred-from user interface.This highlights that analysing users' behaviour on the transferredfrom user interface could be the key for understanding users' behaviour on the transferredto user interface.That is, users reuse their knowledge of the first interacted with user interface when switching to other devices.Hence, evaluators need to be aware of users' behaviour on the first interacted-with user interface to improve their understanding of users' behaviour on the transferred-to user interface to identify any possible horizontal problems.

Duplication of Eye Tracking Patterns
Some users' behaviours could be associated with multiple horizontal problems.For example, user behaviour recorded with ETP8, and ETP9: "many short fixations on the same spot where a participant expected to find the targeted element based on a previous interaction with the transferred-from user interface" were recorded with problems 7 and 8.This might be due to the similarities of the encountered problems, as both problems 7 and 8 were associated with the theme of inconsistent placement of a user interface element.Hence, using eye tracking patterns to predict the associated horizontal usability problem might not be possible all of the time.Therefore, the eye tracking patterns could be used as indicators for problems in the transferred-to user interface.That is, when a specific pattern occurred, further investigation should take place (e.g., evaluator may need to confirm with users).

Is there a problem or not?
The ETP2 "Scanning across the top area of homepage, then heading to use the search box" (refers to Table 4) is general, and could occur in a user typical interaction where no problem is encountered.However, such a pattern could be as a result of the problem (inconsistency of homepages) encountered when users investigated the top area of user interface page.This might be due to the different user behaviours (e.g., strategies to manage a problem) that they might adopt once a problem is encountered.In addition, our investigation for particular pages showed that many users fixated on a specific area of a page on the transferred-to user interface just because it was more attractive to them, according to users' retrospective verbalisations of their experiences of doing the sub-tasks.This could be overlapping with some of the identified eye tracking patterns such as ETP10 where users fixated on the area of the page where the buttons/icons were expected, looking for the same shape of buttons/icons based on their previous interaction.

CONCLUSION AND FUTURE DIRECTIONS
We conducted an exploratory study with three crossplatform services to investigate correlation between eye tracking patterns and cross-platform usability problems.We tested the three services with 31 participants, in which each participant performed three horizontal tasks (one horizontal task per service).This resulted in 174 cross-platform usability problems.The problems were reduced to 11 generalised problems for all tasks, with 13 generalised eye patterns associated with them.We found users behave similarly when encountering a specific usability problem regardless of the order of user interfaces, and whether the interface is desktop-based or mobile-based.For example, when users encountered inconsistent ordering of items within specific elements/groups cross-platform, 90.47 per cent of users behaved similarly (refer to Table 4, problem 6).
In addition, all eye tracking patterns in Table 4 (except ETP2, and ETP13) that were associated with the problems were influenced by the design of the previously interacted-with user interface in which some participants reused their knowledge of the first interaction with the user interface when switching to the other device.This emphasises the need for investigating the user behaviour on the transferred-from user interface to understand the behaviours on the transferred-to user interface and to identify any possible associated problems.An eye tracking pattern could also be associated with more than one cross-platform usability problem (refer to ETP8 and ETP9 in Table 4).An eye tracking pattern could also be a result of the coping strategies that users adopt when they encounter a problem (e.g., ETP2 in Table 4).
The "CPUP-EPT" (refer to Table 4) correlation can be seen as an initial correlation scheme, which can be used by usability evaluators as indicators of possible cross-platform usability problems during the analysis of eye movement data.Besides the use of our correlation scheme, we recommend that further investigation (e.g., with users) is needed when a specific pattern occurs.
The work of Wäljas et al. (2010) on cross-platform user experience found that users might not be very sensitive to consistency because it is not something that users obviously think about or may be that the evaluated services were satisfactorily consistent on all platforms.In our study, we found that consistency on many levels is a major problem cross-platform.Inconsistent home page design and ordering of page elements were the two main sources of problems for our cross-platform user experiments.These both caused our participants to waste time on numerous occasions and are issues which could largely be avoided by better webpage design.
This study can be seen as a step toward developing a comprehensive schema of crossplatform usability problems and eye tracking patterns possibly associated with them.Further experiments are needed to validate our correlation schema of eye tracking and cross-platform usability problems with more focus on specific problems such as inconsistent ordering of elements crossplatform.There might also be a need for investigating the problem-encountering behaviour of users with different characteristics.For example, users with different levels of cross-platform expertise may behave differently.Users' expectations of the degree of data and function distribution cross-device are factors that may affect user behaviours.Some users may expect that the same data and functions can be accessed from the different devices, while others may expect some data and functions not to be accessible from a specific device.An investigation of user behaviours for the different groups of users based on their expectations when experiencing a specific crossplatform problem are also needed, as the user behaviours would be different.
In this study, the evaluated services were from one domain (online shopping).This may limit the generalisability of our findings across other service domains.However, our evaluation results of services from other domains confirmed the existence of the cross-platform usability problems identified in this study across different domains (results under submission).
One limitation of the study is that the analysis was conducted by a single evaluator, leading to potential bias in interpreting the observations.Furthermore, determining the start and the end of a usability problem when recording the eye tracking pattern was also a subjective decision of one evaluator.Nevertheless, our study demonstrates the importance of consistency for cross-platform services, and the value of eye-tracking as an additional tool for assessing cross-platform usability.

Figure 2 :
Figure 2: Souq cross-platform service, ((1) desktop webbased user interface and (2) mobile web-based user interface)) ETP13: In the beginning, high number of short fixations on attractive elements, with no fixation on relevant element(s) to the migrated task.Followed by scanning different elements across the page.A (D M): 7 A (M D): 2 E (D M): 2 E (M D): 1 S (M D): 1

Table 2 :
Table2shows the number of vertical 6 usability problems (aggregated across tasks and participants) identified in each service per type of user interface version (desktop and mobile).The total number of identified vertical usability problems across the different services and the different types of user interfaces is 124.Number of vertical usability problems identified in each service per platform The total number of identified horizontal usability problems across the different services and the different order of user interfaces is 174.

Table 3 :
Number of horizontal usability problems identified in each service per order of user interface for all tasks

Table 4 :
Correlation of cross-platform usability problems with eye tracking patterns (CPUP-ETP)