Comparison of Off-screen Visualization Techniques with Representation of Relevance on Mobile Devices

When exploring map information on mobile devices, relevant points of interest (POIs) are often located off-screen. Despite the existence of several techniques that allow the exploration of the surrounding areas, none represents the POIs’ relevance. Furthermore, when the number of POIs increases, the visualization of the information often becomes unintelligible. This paper presents: our approach to enhancing off-screen visualization techniques with the representation of relevance and cluttering reduction; and a comparative study, using three modiﬁed techniques with the proposed approach, HaloDot, Scaled Arrows and Mini-Map. We concluded that while Scaled Arrows has advantages when analysing the distance to the most relevant objects, the Mini-Map provides better information about the distribution of the POIs, helping users on navigational tasks. Also, the choice of an off-screen technique depends, at least, on two factors: number and geographic distribution of POIs.


INTRODUCTION
The widespread usage of Internet applications, along with ubiquitous geopositioning technology, has fostered the research interest on mobile visualization and geographic information systems (GIS).
Despite the evolution of handheld mobile devices, their reduced display size continues to impose severe usability and visualization restrictions, in particular, in tasks involving the exploration of large amounts of geographic data. When searching for points of interest (POIs) in large maps on mobile devices, some objects are located off-screen. Panning and zooming techniques can be used to explore the surrounding areas that are not visible (Cockburn and Savage 2003). However, these techniques can be cognitively complex and time consuming, if the relevant objects are not visible in the desired level of detail. To overcome these issues, several approaches have been proposed, such as the use of graphical representations along the borders of the display area, to convey information about the distance and the direction to the off-screen objects (Baudisch and Rosenholtz 2003;Burigat et al. 2006;Gustafson et al. 2008). In addition, to avoid cluttering of information on small screens, it is important to use mechanisms that reduce the amount of information displayed, and highlight the most relevant to the user.
Despite the several proposals to represent the relevance of on-screen POIs, none of them are applied to the off-screen visualization techniques. Furthermore, although most off-screen visualization techniques provide information about the distance and direction to the POIs, very few provide information about their relevance and some of them suffer from over-cluttering problems.
Let us consider the scenario where a user wants to find restaurants. Supposing that after seeing an overview of the results, considering Shneiderman's visualization mantra (Shneiderman 1996), the user zooms to the image of Figure 1, to learn more about the POI at the center of the screen. The user then decides to look for other relevant POIs. Using a visualization method, such as the Halo (Baudisch and Rosenholtz 2003), the user learns that there are two POIs equidistant to the location at the center of the screen. However, the user is not able to swiftly decide between POIs, since the Halo technique does not give awareness about the relevance of the off-screen objects. This situation could compromise or, at least, slow down the user's search, as the alternative would be to pan the map to one of the POIs, hoping it to be a highly relevant one, or zoom out and repeat the search.
In this paper, we present a comparative study of three modified off-screen visualization techniques, that deal with the representation of the relevance and the over-cluttering of information.
In section 2, the paper describes several techniques on representation of relevance and off-screen visualization, that deal or not with relevance. Afterwards, in sections 3 and 4, the paper presents our comparative study and the discussion of its results. Finally, in section 5, the main conclusions of this study are drawn and future work is pointed out.

RELATED WORK
The following section describes several techniques on representation of the relevance and off-screen visualization. Moreover, we describe some cluttering methods used in both contexts. Reichenbacher (2007) argues that the inclusion of mechanisms that allow the quantification and perception of the relevance of the various objects may lead to more usable mobile map applications.

Representation of Revelance
The representation of the relevance has been subject of research in several studies. Overall, it is important to use visual attributes with a high probability of guiding the user's attention to convey information (pre-attentive attributes). Swienty et al. (2007) suggest that an object's relevance value can be emphasized by the use of different shapes or colours, to guide the user's attention. In fact, some attributes, like colour, can be used to represent different meanings, one of them temperature (Silva et al. 2011). In addition, previous studies identified colour, motion, orientation and size as attributes that undoubtedly guide the user's attention (Wolfe and Horowitz 2004).
Besides the need to provide a proper symbology regarding the POIs' relevance values, it is crucial to reduce the amount of information displayed on the map. Throughout the literature, several approaches can be found. Furnas (1986) explored the presentation of potentially large structures in displays of reduced size using fisheye views, based on a degree of interest function. This function estimates the interest of the user to visualize a certain object taking into account its global relevance, and what the user is focused at the moment. Swienty et al. (2008) proposed the use of filtering functions, depending on the contexts of use, and taking into consideration the objects' spatial, temporal, and thematic relevance values.
Another approach consists in the use of generalization operators that group symbols that are close to each other (Edwardes et al. 2005). Pombinho et al. (2008) proposed an approach for the visualization of POIs on mobile devices based on a degree of interest function and generalization operators (e.g., aggregation methods). The degree of interest function deals with the user's preferences for different attributes and takes into account the distance between the location of the user and the POIs. The aggregation method considers the existence of a hypothetical grid, based on geographic coordinates, overlaying the map, which divides the geographical space into cells. POIs inside the same cell are grouped into one.

Off-Screen Visualization Techniques
To mitigate the problem of exploring off-screen POIs on small screens, several techniques have been proposed. These can be grouped into: Pan and Zoom, Overview+Detail, Focus+Context, and Contextual Clues (Burigat and Chittaro 2011).
Panning and zooming operations consist, respectively, on methods to drag the visualization window to locations that were not visible, and change the scale (and, sometimes, the detail) of the representation (Cockburn and Savage 2003). Despite their widespread usage, these approaches are limited, in a mobile context. If no clues are given about the location of off-screen objects, the interaction may be time consuming, and disorienting (Burigat et al. 2008).
Overview+Detail techniques consist on the simultaneous display of a detailed view and a small-scaled overview of the information space, usually overlaying the detailed view (Burigat and Chittaro 2013). In this overview, which we will call Mini-Map, a viewfinder Two types of Overview+Detail interfaces can be identified, based on the context presented by the overview. The first shows the entire information space within the overview, i.e., the entire geographical space where the user can retrieve information. Panning the detailed view will result in moving the viewfinder. This approach is common on applications where the information space is limited and/or of known dimensions. The second approach "locks" the viewfinder at the center of the overview and updates the overview's image according to the detailed view's location. The overview shows only the surrounding areas of the detailed view, at a lower zoom level. As a consequence, this approach may not provide all context, and is often used when the information space is too large.
It is argued that these techniques help users keeping track of their current position in the information space, and provide a better understanding of the distribution of the POIs (Burigat and Chittaro 2013). However, they impose an additional cognitive processing by requiring the reorientation when switching between views (Baudisch et al. 2002). Additionally, overview windows require additional space, thus occluding part of the context in the detailed view (Burigat et al. 2008).
Focus+Context techniques, such as fisheye views (Furnas 1986), display information with different levels of detail, without separating the views. Usually, they present a distorted but complete view of the surroundings. However, it is difficult to integrate all information into a single mental model based on different scales and distortions (Baudisch et al. 2002).
Contextual Clues/Views (Burigat and Chittaro 2011) consist of graphical representations, positioned along the borders of the display area, conveying information about the distance and direction to offscreen objects. The basic example is an arrow pointing at the direction of the off-screen object. Labels, the size, or the length of the arrow may be used to express the object's distance to the on-screen location (Burigat et al. 2006). The City Lights technique (Burigat et al. 2006) uses small lines at the border of the screen. Unlike arrow-based techniques, it may convey an abstract representation of the object's distance, using colours to represent specific distance ranges. The Halo technique (Baudisch and Rosenholtz 2003), consists of a circle surrounding the POI's location, large enough to reach the visible border of the screen. Based on the position and size of the visible portion of the arc, the user can infer the location and distance to the off-screen POI.
Despite their effectiveness in representing off-screen objects, when there is a large amount of POIs on the map, the visual clues may overlap, which is a critical issue. To mitigate this problem, some techniques were proposed. EdgeRadar and EdgeSplit create a small overlay region on the four edges of the screen to represent the off-screen space (Hossain et al. 2012), and represent distance with smaller off-screen POIs' compressed proportionally into the border. EdgeSplit, in particular, divides the off-screen space into polygons, associated with POIs, that, once selected, allow for an effective navigation to the POIs (Hossain et al. 2012). Gustafson et al. (2008) proposed the Wedge technique, which represents off-screen POIs with an acute isosceles triangle, with the tip located at the off-screen POI, and the other two corners on-screen. To avoid overlapping, Wegde can change its rotation, intrusion into the screen, and/or angle of the triangle, and still point to the same POI. This technique has later been improved by using transparency over the Wedges, proportional to the POIs distance, and by displaying only the Wedges representing the closest POIs (Burigat et al. 2012).
Although the mentioned techniques provide some information about the objects' distance or direction, none represents their relevance, which is critical when visualizing large amounts of POIs.

Relevance in Off-Screen Visualization
The Dynamic Insets technique (Ghani et al. 2011) considers the concept of relevance and follows a similar approach of the Overview+Detail technique. In this technique, several small-sized views/nodes overlap the map with a detailed view of the off-screen POIs. By choosing between different interest functions, the user can control the visibility of the nodes. Although further studies are required, The HaloDot (Gonçalves et al. 2011) consists on a variation of the Halo technique that includes the representation of the relevance. Like the Halo, this technique provides information about the location and the distance to the off-screen object based on the size and the curvature of the visible portion of the arc. However, unlike the Halo, it provides information about the relevance to the off-screen objects, through the use of a colour+transparency code, reduces the cluttering of arcs with an aggregation method, and improves the direction encoding, by using a small arrow over its arc (see Figure 4(left)).
Despite the several visual variables available, not all are suitable for the representation of the relevance. Even highly pre-attentive variables, like size and line thickness, are not suitable on a mobile device context, since they may induce a higher cluttering of information. Moreover, some techniques already use these variables to represent information (e.g. size to convey distance). As such, it was decided to use colour to represent the relevance of the off-screen objects. Using a "warm-cold" analogy, objects are either coloured in red (warm = high relevance), blue (cold = low), or purple (tepid = medium). Previous studies also suggest that the least relevant objects should be less visible (Swienty et al. 2008). Therefore, transparency is used to emphasize the representation of the relevance (see Figure 3), i.e. the less relevant, the more transparent (Gonçalves et al. 2011).
To minimize the cluttering of the symbology with the Halo, and Contextual Clues techniques, in general, an aggregation method was implemented to combine off-screen symbols, similar to the one described by Pombinho et al. (2008). POIs inside the same "cell" are represented by the same symbol (e.g., one arrow, arc). In addition, overlapping symbols may be merged into a single one. To represent the relevance of the aggregation, the HaloDot uses the colour+transparency code of the most relevant object within the aggregation and text labels close to the symbols, to represent the number of objects. Therefore, the HaloDot technique will always focus on the most relevant information, even when representing several POIs.
Previous results (Gonçalves et al. 2011) reveal a higher efficiency in finding relevant POIs and the users' positive feedback towards the HaloDot's relevance representation, in comparison to the original Halo technique.
Our work follows the approach taken by the HaloDot, and expands it by using the representation of the relevance and the cluttering reduction approaches to optimizing and comparing two other off-screen visualization techniques. In the next section, the techniques evaluated in the user study are described.

TECHNIQUES COMPARED IN THE STUDY
In this study, we have compared the HaloDot (HD) with two other adapted techniques: a Contextual Clues approach (Scaled Arrows -SA) and an Overview+Detail variant (Mini-Map -MM) (see Figure 4). The criteria used for the selection of these techniques was their popularity in current mapping and gaming applications and because they are the most representative of the Contextual Clues and Overview+Detail approaches. Next, we describe the adaptations introduced in the Scaled Arrows and Mini-Map techniques.

Modified Scaled Arrows
The Scaled Arrows technique provides information about the direction and the distance to off-screen POIs (Burigat et al. 2006). The former is given by the arrow's orientation, while the later is given by its size. The size of the arrow is inversely proportional to the object's distance to the screen, i.e., the larger the arrow, the closer the object is.
To represent the relevance of off-screen POIs in the Scaled Arrows technique we applied the same colour+transparency code as in HaloDot. In addition, a similar aggregation method was used, merging arrows that are too close or overlap each other. As a consequence, each arrow is coloured according to the relevance value of the most relevant object it represents, and uses text labels to represent the number of POIs within an aggregation (see Figure 4 (mid)).
Another concern was the shape of the arrow. We selected the one considered to be the less intrusive. Due to the small size of mobile devices' screens, this is an important issue. Choosing the less intrusive arrow can be an advantage for the users' performance, since more space will be available to visualize information.

Modified Mini-Map
To adapt the Mini-Map technique, we used the colour+transparency code of HaloDot both in the overview's and the detailed view's POIs (see Figure 2). As the users tend to interact with the device on the lower parts of the screen more often, we decided to draw the Mini-Map on the upper left corner of the screen, to prevent interaction errors (see Figure 4 (right)).
Comparing with Contextual Clues, Overview+Detail techniques tend to suffer less from cluttering issues. Thus, we have not considered aggregation methods in this technique. In fact, given the existence of the overview, the application of an aggregation algorithm could raise some visualization/interaction issues that are, currently, beyond the context of our study.
In this study, we have chosen an overview with a fixed viewfinder motivated by two reasons. First, we are considering that the information space is the entire map (a large geographic area). Second, presenting the information on an overview with varying detail could hinder the users' navigation.

USER STUDY
To deepen the analysis of relevance representation in off-screen visualization techniques, we have conducted a usability study to compare the efficiency among these new approaches: HaloDot, Modified Scaled Arrows and Modified Mini-Map. For a simplification of language, we shall address the last two techniques as Scaled Arrows and Mini-Map.
Another objective of this study is to investigate the effects of the number of POIs on the users' performance. Previous studies considered this factor as a basis for their experiments, however, none considered the concept of relevance. Since the use of relevance clues may act as a visual filter (optimized by the aggregation algorithms of the Contextual Clues techniques), the number of POIs may reveal an effect on the users' performance.

Hypothesis
Based on the characteristics of the techniques and the objectives of the experiment, our hypotheses were the following: (H1) It is expected that the majority of the users prefer the Mini-Map technique. This approach provides a direct visualization of the POIs, while the others require a mental association between the graphical clues and the direction/location/relevance of the off-screen POIs. This hypothesis is also supported by the usage of similar techniques in games and GIS; (H2) It is also expected that, with a large amount of POIs, the users prefer the Scaled Arrows technique over the HaloDot. Despite the aggregation algorithm, the HaloDot still takes more screen space than the Scaled Arrows technique; (H3) Analysing the task completion times, it is expected that the representations of off-screen objects with lower relevance values do not harm the user's performance, despite of their number; (H4) Analysing the task completion times, on scenarios with a large amount of POIs, it is expected that users locate relevant POIs faster with the Mini-Map technique. When searching for several POIs, the users need to retain information about the locations they have already explored. Due to the overview and the viewfinder, the users may retain that information more easily with this technique.

Participants and Materials
The study had the participation of 19 volunteers (11 male, 8 female). Their age ranged from 21 to 49 years old (M = 27, SD = 8.132). Among these, 16 users had some familiarity with Web map applications for planning routes or searching POIs. In addition, 8 participants had already some familiarity with off-screen visualization techniques, namely, arrow and Overview+Detail techniques, used in video game environments.
The study was carried out on a touch-screen HTC Desire, running the Android OS 2.2, featuring a 1GHz processor and a 3.7-in touch screen with 480x800 of resolution.

Tasks
The study consisted of two tasks, preceded by a training session. The tasks selected were based on some previous works (Gonçalves et al. 2011;Burigat et al. 2006;Gustafson et al. 2008) and consisted on a mobile scenario where the users had to carry out spatial tasks involving POIs on a city map, some of them located off-screen.

Task 1 -Proximity
This task compares the effects of the relevance clues on the three techniques for the determination of the nearest relevant off-screen POI. To carry out this task, the users had to select the representation of the off-screen object that represented the closest POI with a high relevance value, i.e., tap the (red) arrow, arc, or Mini-Map point.

Task 2 -Navigation
In this task, the users were asked to navigate and select all highly relevant POIs on the map. Our aim is to investigate the effects of the number of POIs and the techniques on the users' performance in finding the most relevant objects. For that, the users had to tap on the visible POIs. This is a common spatial task in mobile map scenarios, when users want to find the most relevant POIs after a search.

Experimental Design
In this experiment, we have considered two independent variables: Visualization technique (Vt) -with three levels: HaloDot, Scaled Arrows, and Mini-Map; and Number of Relevant POIs per total number of POIs (NrP) -with four levels: 3 highly relevant POIs in 10, 6 in 20, 6 in 40, and 10 in 40.
The experiment followed a within subjects design and all participants carried out each task individually. At the beginning of the study, they were briefed about the objective of the experiment and the tasks they would be asked to perform. After that, they watched a demonstration for each technique and were free to explore the three visualization techniques so they could get used to them, to the device itself, and to clarify any doubts.
After this training phase, the users performed the Proximity task. Since the scenarios presented in this task consisted of static images, the users simply had to tap on arrows, arcs or points located on-screen. Thus, panning and zooming operations were not needed/enabled. As a result of the aggregation methods, on a static image, the two last levels of the variable number of relevant POIs per total number of POIs were visually similar to the second. Thus, only the first two levels of this variable were used. In the first configuration, 3 POIs in 10 had a high relevance value (above 0.8 in a 0 to 1 range). In the second, 6 POIs in 20 had a high relevance value. Each participant performed 6 trials: (3 Vt) x (2 NrP).
In the Navigation task, users were presented with a map that could be dragged to find the off-screen POIs. Zooming operations were not enabled, so they performed the task relying only on the off-screen visualization techniques. We considered the four levels of the variable number of relevant POIs per total number of POIs. In the first configuration, 3 highly relevant POIs were presented, in a total of 10. In the second, 6 in 20. In the third, 6 in 40, and, in the last one, 10 in 40. Each user performed 12 trials: (3 Vt) x (4 NrP).
Although each task began with the configurations with fewer POIs, followed by those with a higher number, the order in which the techniques were presented was randomized, to prevent, as much as possible, learning effects, without compromising the users' ability to compare the techniques. In addition, the following dependent variables were considered: • Task completion times; • Users' preferences: at the end of each task, the users were interviewed to collect their suggestions, difficulties, and opinions about the techniques, to identify the one they preferred, and to order them relatively to the ease of use; • Errors committed: in the Proximity task, an error is a point, arc or arrow selected that does not represent the closest highly relevant off-screen object to the center of the screen. In the Navigation task, it is an object that did not have a high relevance value.

Task Completion Times
For the analysis of the users' task completion times, we subjected the data to the Shapiro-Wilk test of normality. Since no data transformation could provide a normalized dataset, we decided to follow a non-parametric approach by applying the Aligned Rank Transform (ART) method, followed by a full factorial ANOVA (Wobbrock et al. 2011). When significant effects were detected, we performed the Bonferroni test for pairwise comparisons. Figure 5 shows the users' mean times for the Proximity task, for the six combinations of the two independent variables (visualization technique and number of POIs). Although no significant interactions were detected between visualization and number of points, the tests revealed a significant effect of the visualization technique on task completion time (F(2, 108) = 6.847, p = 0.002). The pairwise comparisons tests revealed a statistically significant difference between the Scaled Arrows and the other techniques (p = 0.02 and p = 0.017 comparing with the HaloDot and Mini-Map technique, respectively).
For the Navigation task, since selecting 6 or 10 points of interest requires more time than selecting 3, we divided the total task time by the number of selected points, to provide more meaningful data for the analysis. Figure 6 shows the users' mean times of completion for the task divided by the number of objects to select.
We did not detect any significant interaction between the visualization technique and number  In terms of visualization technique, the pairwise comparison tests revealed significant differences between the HaloDot and the Mini-Map (p = 0.032). In terms of number of points, significant differences were detected between the first, second and third configurations, with the fourth (p <0.001, for all cases).

Users' Preferences
We applied Friedman's test, followed by a Wilcoxon Signed Rank test, for pairwise comparison, with a Bonferroni correction, to compare the differences between the users' preferences before the study and for each configuration of each task.
The first configuration of the Proximity task (see   pairs of results revealed significant differences between the Scaled Arrows and the HaloDot techniques (p = 0.002). In the second configuration (see Figure 8), statistically a significant difference between the users' preferences were also detected (X 2 (2) = 10.478, p = 0.005), with the pairwise comparison test revealing, again, a significant difference between the Scaled Arrows and the HaloDot techniques (p = 0.003). Figure 9 shows the users' preferences in the navigation task. While in the first configuration no differences were detected, in the second one, statistically significant differences where found (X 2 (2) = 10.829, p = 0.004), particularly, between the HaloDot and the other techniques (p = 0,001 and p = 0,016, comparing with Scaled Arrows and Mini-Map, respectively). The third configuration, also revealed significant differences between the users' preferences (X 2 (2) = 13.549, p = 0.001), in particular, between the HaloDot and the other two techniques (p = 0.001 and p = 0.002 comparing it with the Scaled Arrows and the Mini-Map). In the fourth configuration, the results revealed again significant differences (X 2 (2) = 20.090, p<0.001), in particular, also between the HaloDot and the other techniques (p <0.001 in both cases).

User errors
Although the error results did not allow to perform a more complex analysis, we decided to report the number of errors committed. In the Proximity task, in total, 7 errors were committed when using the Scaled Arrows technique, 6 by the same user, who confused the meaning of the size of the arrows, with the size of the HaloDot's arc. In total, 19 errors were committed using the HaloDot, 9 by the same user. Only 1 error was committed with the Mini-Map technique. In the Navigation task, no errors were committed.

DISCUSSION
The analysis of the results reveals small variations depending on the tasks and/or configurations. In these situations, analysing the users' feedback and their actions during the tasks, was valuable to the following discussion.
After the training phase, 65% of the users preferred the Mini-Map technique. These users commented that this technique was the most "visually pleasant", and intuitive for analysing the POIs' distribution. 15% also justified their preference based on their familiarity with this technique in video games.
In the Proximity task, users were most efficient and shown a significant preference for the Scaled Arrows technique. They stated that the size allowed them to compare the distance to the POIs more easily. Contrarily, the HaloDot was classified as the most complex to use, thus being the least preferred. Overall, these results reinforce some previous studies (Burigat and Chittaro 2011) and suggest that Scaled Arrows are the more usable to calculate and compare the distance to off-screen POIs.
In the Navigation task, in all configurations but the first, a significant preference was detected for the Scaled Arrows and the Mini-Map techniques, over the HaloDot. Alongside with those of the previous task, these results support partially our first hypothesis (H1: Mini-Map would be the preferred technique) but clearly support our second hypothesis (H2: Scaled Arrows would be preferred over HaloDot).
In the second and third configurations (see Figure 6), although the number of relevant POIs was the same in both tasks (6), the total number of objects was 20 and 40, respectively. However, no significant differences were found between the results of these configurations. In addition, the users commented that they have not found any increasing difficulty between the two configurations, and that, despite the number of POIs, they were still able to visualize the symbology of the most relevant POIs. These results support the fact that, even with a larger amount of less relevant object's representations, the interaction and visualization of the data was not harmed, therefore, supporting our third hypothesis (H3: performance would not be significantly affected by less relevant POIs).
The analysis of the configurations with a larger number of POIs (i.e. 6 highly relevant POIs in 40 and 10 in 40) reveals that users were significantly faster with the Mini-Map, in comparison to the HaloDot. In turn, these results support our fourth hypothesis (H4: faster task completion times using Mini-Map).
Overall, during the Navigational task, although the Mini-Map technique got better average results, the users pointed out three main problems. The first consists in the need to split the attention between the Overview+Detail screens. In this case, however, it did not result in a worse performance. Second, if the POIs are located outside the overview, the user has no information about their existence. Finally, the overview may overlap relevant POIs located in the detailed view, confusing the users.
The analysis of the fourth configuration (find 10 POIs out of 40) revealed that users were significantly slower in completing the task. During the Navigation task, nearly all users (continuously) forgot which POIs they had already selected. Interestingly, this situation was significantly worse with the HaloDot technique. In fact, some users seemed to focus their attention on the size of the arc, and ignored the appearance of highly relevant objects on-screen. Thus, giving more attention to the relevance of the off-screen POIs rather than the on-screen ones (in theory, more relevant (Reichenbacher 2009)). Still, this is not a surprising result. The intrusiveness of the HaloDot's arc, alongside its constant change of size, means that we have two efficient attributes guiding the users' attention (size and movement) (Wolfe and Horowitz 2004). In comparison, the on-screen representation of the POIs are smaller than the HaloDot's arcs. Therefore, the users' attention may have been more easily captured by the HaloDot, rather than the on-screen POIs. Despite these results, this situation never happened with the other techniques. In fact, the Scaled Arrows and Mini-Map techniques are less intrusive than the HaloDot and therefore, an on-screen POI is as noticeable as an arrow or a POI inside the overview.
Moreover, the number of errors was very small. In the Proximity task, no significant differences between the errors committed with the techniques were detected. Judging by the users' feedback and their errors, we can hypothesize that the HaloDot is the least effective technique. Nevertheless, this assumption needs further investigation. In addition, the lack of errors on the Navigation task supports the results of previously mentioned analysis (Gonçalves et al. 2011). Overall, the combination of colour and transparency is efficient for the representation of the relevance on the off-screen visualization techniques. This code allows the identification of the approximated relevance value of the off-screen objects (based on colour) and the focus onto the most relevant ones (based on transparency), reducing, alongside with the aggregation algorithm, if applicable, the cognitive workload to analyse a map with several objects.
In addition, the analysis of the results allowed to identify, at least, two factors that may have an impact in the choice of an off-screen visualization technique with representation of relevance on mobile devices: Number of POIs: as the number increases, less intrusive techniques seem to be more efficient. The analysis of task completion times and users' preferences supports this fact, since the Mini-Map and the Scaled Arrows techniques aided users performing better, even with a larger amount of POIs.
Geographical distribution/density of the POIs: It is necessary to analyse the location where POIs may be displayed, to decide which technique to use. The Mini-Map technique is the most effective in scenarios where all POIs are located within the overview. However, when some POIs are located outside the overview, the Scaled Arrows technique seems more efficient. This is supported by the users' feedback regarding the Mini-Map. This limitation is dependent on the approach followed for the Mini-Map technique. A possible solution would be using additional clues within the overview, or a dynamic Mini-Map.
Additionally, we hypothesize the existence of a third factor, namely the symbology used to represent POIs. Based on the observations obtained with the HaloDot, we observed that the users got distracted with the arcs of the technique not noticing the appearance of relevant POIs on-screen. To prevent these situations, the off-screen visualization technique must be as less distractive as possible, and the representation of the on-screen POIs must be noticeable enough. Also, if we consider the possibility of having different representations depending on the POIs category (e.g., restaurants, hotels), some visualization techniques could present different results. Despite these observations, further studies should be conducted.

CONCLUSIONS
This paper presents a comparative study between three off-screen visualization techniques that include the representation of relevance: HaloDot, Modified Scaled Arrows, and Modified Mini-Map. The Contextual Clues techniques also include an aggregation method to reduce cluttering.
The results of this study show the users' preference for less intrusive techniques. This fact becomes more evident as the number of POIs increases. In tasks where the user needs to calculate and compare the distance to off-screen POIs, techniques like Scaled Arrows are more helpful. In navigational tasks, Overview+Detail techniques, such as the Mini-Map, provide better results. Despite the number of POIs on the map, the representation of the relevance for the off-screen objects allowed the users to efficiently complete exploration tasks, as the representation of less relevant objects did not harm the users' search for highly relevant POIs.
We also identified two factors that may condition the choice of an off-screen visualization: number of POIs; and geographical distribution of the POIs. On a scenario with many POIs (over 20), Scaled Arrows and Mini-Map techniques provide better results. However, when POIs located outside the visible area are covered by the overview, the Scaled Arrows technique will be more helpful. In a scenario with several POIs, despite the number of graphical clues (i.e. arrow or arc graphics), the use of an intrusive technique may harm the users' visualization, both due to the cluttering of information and the possibility of distracting the user with the adjustment of the visualization, inherent to the technique, after a panning/zooming operation.
Apart from comparing and studying off-screen visualization techniques in a mobile device context, our work contributes to the area of off-screen visualization, by studying the enhancement of techniques with the representation of relevance and cluttering reduction, and by providing new insights to mobile interface designers.
Nevertheless, there are still some issues to investigate. The following steps in this study are: (1) to perform further studies regarding the identified factors, in particular the factor of the symbology used; (2) to study possible representations for (on-screen and off-screen) POIs that convey more information than just the POIs relevance; (3) to explore solutions for the problems detected on the Mini-Map technique; (4) to adapt other techniques with the concept of relevance and compare them; (5) to compare the usability of on-screen with off-screen aggregation methods, to solve cluttering issues.