Augmenting User-Maintained Interaction Through Mode Locking and Reversing

s on Human Factors in Computing Systems, Toronto, Ontario, Canada. ACM Augmenting User-Maintained Interaction Through Mode-Locking and Reversing Katherine Fennedy ● Hyowon Lee ● Insuk Ko ● Yi Ren Tan ● Dongliang Zhang ● Chunxiao Li 11 Guiard, Y. (1987) Asymmetric Division of Labor in Human Skilled Bimanual Action. Journal of Motor Behavior, 19, 4, 486-517. Gutwin, C., Cockburn, A., Scarr, J., Malacria, S. and Olson, S. C. (2014) Faster command selection on tablets with FastTap. Proceedings of the 32nd annual ACM conference on Human factors in computing systems, Toronto, Ontario, Canada. ACM Hamilton, W., Kerne, A. and Robbins, T. (2012) High-performance pen + touch modality interactions: a real-time strategy game eSports context. Proceedings of the 25th annual ACM symposium on User interface software and technology, Cambridge, Massachusetts, USA. ACM Hinckley, K., Guimbretiere, F., Baudisch, P., Sarin, R., Agrawala, M. and Cutrell, E. (2006) The Springboard: Multiple Modes in One SpringLoaded Control. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Montreal, Quebec, Canada, 181-190. ACM 1124801. Holz, C. and Baudisch, P. (2010) The generalized perceived input point model and how to double touch accuracy by extracting fingerprints. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Atlanta, Georgia, USA. ACM InstagramPress. (2016) Introducing Instagram Stories. https://instagrampress.com/blog/2016/08/02/introducing-instagramstories/ (14th March 2018). Johnson, J. and Engelbeck, G. (1989) Modes Survey Results. SIGCHI Bull., 20, 4, 38-50. Johnson, J. (1990) Modes in non-computer devices. International Journal of Man-Machine Studies, 32, 4, 423-438. Kleimola, J., Laine, M., Litvinova, E. and Vuorimaa, P. (2013) TouchModifier: enriched multi-touch gestures for tablet browsers. Proceedings of the 2013 ACM international conference on Interactive tabletops and surfaces, St. Andrews, Scotland, United Kingdom. ACM Lank, E., Ruiz, J. and Cowan, W. (2006) Concurrent bimanual stylus interaction: a study of non-preferred hand mode manipulation. Proceedings of Graphics Interface 2006, Quebec, Canada. Canadian Information Processing Society Le, H. V., Mayer, S., Bader, P., Bastian, F. and Henze, N. (2017) Interaction Methods and Use Cases for a Full-Touch Sensing Smartphone. Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems, Denver, Colorado, USA. ACM Li, Y., Hinckley, K., Guan, Z. and Landay, J. A. (2005) Experimental Analysis of Mode Switching Techniques in Pen-Based User Interfaces. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Portland, Oregon, USA. ACM Matulic, F. and Norrie, M. C. (2012) Supporting active reading on pen and touch-operated tabletops. Proceedings of the International Working Conference on Advanced Visual Interfaces, Capri Island, Italy. ACM Matulic, F. and Norrie, M. C. (2013) Pen and Touch Gestural Environment for Document Editing on Interactive Tabletops. Proceedings of the 2013 ACM international conference on Interactive tabletops and surfaces, St. Andrews, Scotland, United Kingdom, 41-50. ACM 2512802. Odell, D. and Chandrasekaran, V. (2012) Enabling comfortable thumb interaction in tablet computers: a Windows 8 case study. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 56, 1, 1907-1911. Oulasvirta, A., Reichel, A., Li, W., Zhang, Y., Bachynskyi, M., Vertanen, K. and Kristensson, P. O. (2013) Improving two-thumb text entry on touchscreen devices. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Paris, France. ACM Pfeuffer, K., Hinckley, K., Pahud, M. and Buxton, B. (2017) Thumb + Pen Interaction on Tablets. Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, Denver, Colorado, USA. ACM Poller, M. F. and Garter, S. K. (1983) A Comparative Study of Moded and Modeless Text Editing by Experienced Editor Users. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Boston, Massachusetts, USA, 166-170. ACM 801603. Raskin, J. (2000) The Humane Interface: New Directions for Designing Interactive Systems. ACM Press/Addison-Wesley Publishing Co. Rod, J., Li, C., Zhang, D. and Lee, H. (2016) Designing a 3D modelling tool for novice users. Proceedings of the 28th Australian Conference on Computer-Human Interaction, Launceston, Tasmania, Australia. ACM Saffer, D. (2013) Microinteractions: Designing with Details. O'Reilly Media, Inc. Saund, E. and Lank, E. (2003) Stylus Input and Editing Without Prior Selection of Mode. Proceedings of the 16th annual ACM symposium Augmenting User-Maintained Interaction Through Mode-Locking and Reversing Katherine Fennedy ● Hyowon Lee ● Insuk Ko ● Yi Ren Tan ● Dongliang Zhang ● Chunxiao Li 12 on User interface software and technology, Vancouver, Canada, 213-216. ACM 964720. Sellen, A. J., Kurtenbach, G. P. and Buxton, W. A. S. (1992) The Prevention of Mode Errors Through Sensory Feedback. Hum.-Comput. Interact., 7, 2, 141-164. Siek, K. A., Rogers, Y. and Connelly, K. H. (2005) Fat finger worries: how older and younger users physically interact with PDAs. Proceedings of the 2005 IFIP TC13 international conference on Human-Computer Interaction, Rome, Italy. Springer-Verlag Surale, H. B., Matulic, F. and Vogel, D. (2017) Experimental Analysis of Mode Switching Techniques in Touch-based User Interfaces, 3267-3280. Uddin, M. S., Gutwin, C. and Lafreniere, B. (2016) HandMark Menus: Rapid Command Selection and Large Command Sets on Multi-Touch Displays. Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, Santa Clara, California, USA. ACM Vogel, D. and Baudisch, P. (2007) Shift: a technique for operating pen-based interfaces using touch. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, San Jose, California, USA. ACM Vogel, D. and Balakrishnan, R. (2010) Occlusionaware interfaces. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Atlanta, Georgia, USA. ACM WABetaInfo. (2017) WhatsApp to support locked recordings. http://wabetainfo.com/whatsapp-tosupport-locked-recordings/ (14th March 2018). Wagner, J., St, Huot, p. and Mackay, W. (2012) BiTouch and BiPad: designing bimanual interaction for hand-held tablets. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Austin, Texas, USA. ACM Wigdor, D., Forlines, C., Baudisch, P., Barnwell, J. and Shen, C. (2007) Lucid touch: a see-through mobile device. Proceedings of the 20th annual ACM symposium on User interface software and technology, Newport, Rhode Island, USA. ACM Woods, D. D., Johannesen, L. J., Cook, R. I. and Sarter, N. B. (1994) Behind human error: Cognitive systems, computers and hindsight. DAYTON UNIV RESEARCH INST (URDI) OH. Xia, H., Grossman, T. and Fitzmaurice, G. (2015) NanoStylus: Enhancing Input on Ultra-Small Displays with a Finger-Mounted Stylus. Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology, Charlotte, NC, USA. ACM


INTRODUCTION
User-maintained mode (UMM) is an interaction technique in which a mode is maintained only while a user is consciously and actively invoking that mode (e.g. by keeping a button pressed down or holding a lever in a particular direction).The benefits of this interaction, in minimising the moderelated errors due to the continuous effort required by the user when staying in a particular mode, have been proven by many studies (Sellen et al. 1992;Li et al. 2005;Surale et al. 2017).
While UMM interaction has been adopted in recent studies to discover various useful ways to support bimanual interaction for multitouch devices such as tablets (e.g.(Hinckley et al. 2006;Wagner et al. 2012;Foucault et al. 2014;Pfeuffer et al. 2017), the very nature of this particular interaction itself has a lot more to be explored as there are novel supporting features that could complement or augment this interaction.This is especially true in the usage context of multitouch applications where many subtle but potentially-useful interactive features could be easily instrumented via software means in ways to enrich the interactivity and enhance the experience.
In this paper, we explore some of these novel features of UMM interaction by understanding the characteristics of the interaction as found in our surroundings.For example, a piano pedal is pressed down by the player's foot and the effect of it (e.g. a sustained or softer sound) lasts only while the player keeps the pedal pressed down.One of the features found in many pianos today is a way to "lock" the pedal while it is down (by sliding the pedal to the left so that it will latch on a small groove); pressing down a shift key on a keyboard turns the subsequent alphabet key presses to be capitalised only while the shift key is being pressed down, while the "locking" mechanism supported in this case is a separate "Caps Lock" key.Such a locking feature could also be augmented by other novel mechanism for example temporarily triggering the original mode while in the "locked" mode.Such novel features are more difficult to be implemented in physical objects, thus not have been thought of or provided, but are more feasible on software.
2 Seeing the possible wide uptake of UMM interaction for bimanual use of multitouch devices in the coming years, we step back to analyse the nature of user-maintained interaction, brainstorm how new features could complement it in the context of multitouch usage, and investigate the usefulness by designing and implementing three prototypes that incorporate these features and conducting usability testing to validate and reflect on the designs.The contributions of this paper are thus as follows: • Characterise user-maintained interaction; • Identify novel locking mechanism and reversed mode relationship that can complement and augment UMM interaction; • Design visual representations and interaction strategies to support these (see Figure 1 for some examples); • Validate the level of usability through user testing, and • Offer insights into the kinds of application areas where these features of usermaintained technique could be applied to enhance the interactivity.
These contributions will help re-examine the current rather simplistic and uniform ways that most apps on multitouch devices offer their interactivities today, appropriating the user-maintained gesture where a very common usage behaviour of intermittent shifting of modes/functions is needed.

RELATED WORK
Moded interaction in computing was introduced in the early 1980s, long before multitouch platforms came in the market, though it was already a prevalent idea for non-computing devices (Johnson 1990).The rationale for moded interaction has been to increase users' input vocabulary, and hence making the interface more powerful, without making it more complex (Foucault et al. 2014).The same gesture can generate different output, depending on which mode is active, easily increasing the functionality of a few simple gestures without having to incorporate new affordances.The lack of contextual awareness as the main challenge with the moded interaction is often referenced (Poller and Garter 1983;Johnson 1990;Woods et al. 1994).Multiple studies (Li et al. 2005;Surale et al. 2017) have consistently proved that the usermaintained mode (Sellen et al. 1992), sometimes referred to as quasi-mode (Raskin 2000;Saund and Lank 2003;Hinckley et al. 2006;Matulic and Norrie 2013;Foucault et al. 2014) or spring-loaded mode (Johnson and Engelbeck 1989;Degani 1999;Hinckley et al. 2006;Saffer 2013;Pfeuffer et al. 2017), performs better in terms of task completion time, error rate and user preferences as compared to other mode-switching techniques such as persists, once (Hinckley et al. 2006) and inferred (Saund and Lank 2003).
Past studies have also shown that UMM technique is effective in providing users with fast access to command selection (Hamilton et al. 2012;Kleimola et al. 2013;Gutwin et al. 2014;Uddin et al. 2016;Pfeuffer et al. 2017).This is vital because in platforms where screen size is limited, it is not feasible to reserve screen space to represent all possible functionalities especially when many of them are only temporarily invoked.Therefore, many of today's applications have resorted to hiding the set of buttons for advanced functions or tools within a parent button, slowing down the user interaction and reducing the discoverability of such a feature.
UMM plays an effective role in part because it leverages on the common gestures that users are already familiar with.This can be made possible by integrating Guiard's kinematic chain model (Guiard 1987) for bimanual interaction, where the nondominant hand can act first to set the context for the dominant hand to carry out precise gestures.This asymmetric bimanual hand coordination helps to minimise the cognitive and motor costs (Lank et al. 2006) associated to mode switching and retain the intuitive interaction of touch surfaces while expanding users' input vocabulary.Furthermore, UMM technique is in line with Buxton's notion of chunking (Buxton 1995), which highlighted how the continuity of compound tasks is essential to reduce mode confusion.
Since the nature of UMM requires a sustained interaction such as continuously holding onto a button, finger occlusion (commonly referred to as "fat-finger problem 1 ") (Siek et al. 2005; Holz and Baudisch 2010; Vogel and Balakrishnan 2010) is an anticipated issue when implementing this on multitouch devices where the surface to be touched is also the same surface for visual feedback.Various solutions have been proposed so far, from using finger-mounted stylus (Xia et al. 2015) to extending touch surfaces beyond the default screen (Wigdor et al. 2007;Baudisch and Chu 2009;Le et al. 2017).Vogel's Shift pointing technique (Vogel and Baudisch 2007) using a bubble callout is commonly adopted in many apps available today such as in the familiar zoomed-in preview of a magnifying glass while selecting a text.Our design solution takes these into consideration to combat the finger occlusion problem for UMM.
There is an increasing number of studies that adopt UMM interaction (e.g.novel bimanual thumb + pen interaction (Pfeuffer et al. 2017)) that will surely enhance the usability of tablet interaction in the coming years.Most of them take advantage of UMM interaction with the main focus in facilitating intuitive bimanual usage.To our knowledge, none of these past studies focused on the nature of UMM interaction itself to (1) explore and exploit the features that will augment UMM interactivity, and (2) investigate the design issues and opportunities for creating such features in applying it to realworld multitouch scenarios.

MAKING SENSE OF UMM
To understand the purposes for and the characteristics of features in UMM interaction, we start by looking at everyday physical objects around us that employ user-maintained interaction (see Figure 2).While there are many other examples such as the lever of a height-adjustable office chair and fully-automated rifle, we use the examples from Figure 2 and the shift key example as mentioned in the introduction, to illustrate our process of design feature identification.The main purpose of providing a UMM interaction is to allow its user to change a mode (or function) temporarily.
1 Fat-finger problem: the tendency to make errors on a device where the screen element is too small, typically the finger that selects it blocking the user's view of it.Table 1 compares the examples.For each, we describe the distinction between primary and secondary modes offered by UMM technique.
Primary mode refers to the default behaviour/state of a system when the user does not trigger/maintain any action.Secondary mode describes the behaviour/state when UMM is temporarily triggered on the system.We also specify the existence of locking mechanism, useful in instances where a user wishes to persist the triggered secondary mode (see third column in the table ).This is to prevent fatigue caused by long duration of continued physical exertion to maintain the secondary mode (e.g.locking the water flow mode in a drinking fountain while filling a large bottle).
After locking to the secondary mode, what if the user now wants to return to the primary mode temporarily, as if the originally intended primarysecondary relationship has been reversed?This creates opportunities for UMM to be applied to the locked secondary mode, treating it as the new primary mode, resulting in the reversing in the mode relationship.Hence, the last column in Table 1 shows where there may be a potential in having a reversed primary-secondary relationship between the modes that could be useful.
In a way, the reversed relationship is an extension of the locking mechanism, where the user wants not only to use the secondary mode more continuously but to start using the primary mode intermittently while mainly staying in the secondary mode.Currently we are not able to find any existing examples that employ it, and such a reversing mechanism may not make sense in many applications.For example, there would not be any reason for reversing a clipboard's mode relationship so that the user wants primarily to keep the papers released while only occasionally wants to hold the papers together.
However, in the cases of piano pedals, laser pointer and shift key, there might be interesting and novel usage situations for such a reversed relationship even though the potential may not look obvious especially if one only thinks about its typical usage of today.Physical examples we examined here have, with their unmistakable physical feedback, clear indications of what mode a user is in.We now move on to usage patterns expected from the UMM interaction and the above-mentioned features.In Figure 3(a), the left diagram represents a common UMM interaction where a larger circle indicates a primary mode (A) and smaller circle a secondary (B), the latter being temporarily triggered only during the period in which the user consciously maintains that mode with some effort.Upon stopping that effort, the mode gravitates back to the primary one (dotted arrow line).The graph on the right is an expected behaviour pattern in which the user mostly stays in Mode A but occasionally and temporarily triggers Mode B. Figure 3(b) shows the expected behaviour when a locking mechanism is available to lock into Mode B. Once locked, it becomes the only mode available, as Mode A cannot be triggered during this locked period.Only after unlocking, the system can then return to the default usage pattern in Figure 3(a).
While the reversed mode relationship (i.e.reversing what the primary and secondary mode does) is not common, we believe there exist some situations where it will be beneficial in optimising the workflow or simplifying/enriching the interaction, depending on the usage scenarios and context.Figure 4 shows our expected usage behaviour where such a reversing of the mode relationship is offered.Once locked, Mode B becomes the new primary mode while Mode A becomes the new secondary mode, the latter infrequently triggered for brief periods of time.In the digital realm, these behaviours and the mechanisms to support them have a lot more room for exploration, as there are many subtle but impactful interaction details that are present but have not been understood well.Very few apps today utilise UMM interaction (let alone locking and reversing features).Instagram Stories (InstagramPress 2016) features a video recording button that continues recording only while the user keeps pressing the button, and a media slide-show which automatically flips through the photos/videos by default and only pauses if the user continues pressing on them.A recent update of WhatsApp messaging app (WABetaInfo 2017) includes a voice texting feature in which the voice gets recorded only while the button is pressed down.For a longer period of recording, the button could be slided, effectively implementing a locking mechanism of Figure 3(b).These available UMM interactions can benefit from further refinement such as locking mechanism and/or a reversed relationship.Bringing these together, in the next section we summarise the choices of applications and design rationale for providing the UMM with locking and reversing features, specifically for the multitouch tablet use.

DESIGN STRATEGIES FOR UMM
While there are different postures that can influence how the user interacts with a tablet, we mainly focus on the typical bimanual use (Lank et al. 2006;Foucault et al. 2014;Pfeuffer et al. 2017), in which the user uses her non-dominant hand (NDH) to support the weight of the tablet, leaving only the thumb of the same hand and some or all fingers of the dominant hand (DH) for interaction.For illustration purposes, we will use right hand as DH and left hand as NDH, although this may be changed or customised in real applications.While a similar study by Wagner et al. (Wagner et al.) focused on finding the optimal bimanual hand position and gesture, our study is on using a typical bimanual posture to explore the ways to design and incorporate UMM features.We devised two overall design strategies to incorporate UMM features including locking and reversing of the relationship.The simplest implementation is to use a regular button widget to trigger the UMM functionality.However, such a button needs to be located on the edge/side of the screen where the thumb of the hand that is holding the device can easily press without changing the grip (Odell and Chandrasekaran 2012;Oulasvirta et al. 2013).It could be either fixed on the corner or edge, or be floating on the side so that the user can freely move it up and down with the thumb (see Figure 5(a)).An alternative strategy is by using a sidebar where any point on the sidebar located on the edge of the screen can be pressed with the thumb to trigger the UMM functionality (the blue bar in Figure 5(b)).For both strategies, pressing the thumb triggers the secondary mode and lasts only while pressing.While pressing, a small, transient icon guide appears about 5mm beside the thumb, indicating the locking/reversing possibility.The guide appears beside the thumb as the finger occlusion problem becomes a more critical issue in the UMM interaction than in other conventional one-tap interaction, where the user might be pressing that area for varying durations.
Depending on the layout, the nature of the app and of the mode, the guide can appear either vertically or horizontally (e.g. for the button strategy, horizontal guide may be more suitable in general because the button location itself may be vertically adjustable by the user).Releasing the thumb will bring back to the primary mode and the icon guide will disappear.While keeping the thumb pressed to maintain the secondary mode, the user can indirectly (Pfeuffer et al. 2017) slide the thumb along the icon guide in order to lock that secondary mode as the primary mode, and the icon shows a short but effective animation to indicate the locking (see demo video: https://youtu.be/aajsy5_OZOU).
We apply this overall UMM strategy to 3 different applications.

Prototype 1: eBook Reader
Based on a series of brainstorming that focused on day-to-day applications people use that could result in the usage behaviour projected in Figure 4, eBook reading was an obvious choice where the reader spends most time in the full-page view but occasionally flips through the pages or jumps between them (i.e.full-page view is the primary mode while page-flipping view is the secondary mode).We used the sidebar strategy with a subtle, overlapping page-like shaded design (see Figure 1 Analogous to flipping through the physical pages of a book where the reader usually uses one hand to hold tightly on the binded side of the book while using the other hand to flip through the pages, we incorporated the UMM interaction where the user presses one's thumb on the sidebar to enter the temporary mode during which s/he uses the other hand to flip through the pages.This is similar to a proposed tabletop gesture (Matulic and Norrie 2012) but in the context where both hands are free.Upon releasing the thumb, whatever page that is at the centre of the screen will fill in the screen, returning to the primary mode.This is illustrated in Figure 6.Thumb pressing on the sidebar also reveals the lock guide (see Figure 1(c) and Figure 6(b), (c)) just right of the pressed thumb, and the user can slide it up to lock the page-flipping mode and thus reversing the primary-secondary relationship.Once the locking has completed, the user stays in the page-flipping mode (see Figure 6(d)) without pressing the sidebar.We expect there are some cases where the reader might want to mainly stay in this mode: to go through the pages to quickly remind herself what was read, to get the overall feel for the book content before commencing a serious reading, and so on.Furthermore, a fast scrolling mechanism with the page number indication was designed at the bottom of the screen in this mode, to allow quick access to a specific page (see bottom of Figure 6(d).

Prototype 2: 3D Modelling Application
A previous study (Rod et al. 2016) showed the benefits of having a clear distinction between model editing and viewpoint changing mode in tablet-based 3D modelling activities.While making a modification on a 3D object, the user needs to frequently (but for a very short duration) change the viewpoint to check the modified shape from different angles.Then other times, the user may stay mainly in the viewpoint changing mode while only occasionally fixing minor errors on the object.
Here a UMM interaction with the projected behaviour of Figure 4 will be applicable.We used the button strategy (see Figure 5 In the viewpoint change mode, the background shows a grid on a lighter colour 2D plane to visually reinforce the active mode.The UMM button can be moved up and down by the user to suit his or her holding posture.

Prototype 3: Video Browsing Application
For video browsing applications such as YouTube and Vimeo, a behavioural study (Chen et al. 2013) highlights that users spend most of their time switching between videos and between scenes of a video before committing to watching one till completion.In live-streaming platforms such as Twitch, there is a constant competition for visual attention from the users between the video content itself and the active chat box.Most of these applications use complex user-interfaces with overwhelming amount of information squashed or overlaid on the playback area, resulting in reduced viewability and the issue becomes more serious on a device with limited screen space.Hence we designed the UMM concept in the prototype, by providing fast yet temporary switch between the full-screen playback and other information views such as comment feeds or a list of related videos.
On a typical YouTube-like user-interface (see Figure 8(a)), we embedded a small icon at the bottom left of the playback panel to go to fullscreen, the secondary mode.While pressing this button with the thumb, the full-screen view is temporarily maintained and the icon guide appears indicating the possibility of locking the full-screen mode (see Figure 8(b)).The guide has a small, semi-transparent and floating panel behind to stand out from the background.Upon locking, a small 'exit' icon appears at the bottom left of the playback (see Figure 8(d)), touching this with the thumb will temporarily bring the screen to normal view (see Figure 8(a)) but releasing the thumb will bring the screen back to the full-screen playback.
We intentionally varied the design details across the prototypes to suit the distinct context of individual applications and to explore other factors affecting the user experience.

USABILITY TESTING
We conducted a usability testing with the aim of giving us insights into the designed UMM approaches in terms of overall usability, people's reactions, mistakes/errors and other aspects of the interaction.A total of 27 participants (19 males and 8 females) were recruited in the study one by one, each participant spending between 30 and 45 minutes.13 of them are between 22 to 25 years old, 8 of them are between 26 to 30 years old, 4 of them are between 31 to 35 years old, and 2 of them are above 40 years old.For each participant, the researcher introduced and briefly demonstrated each prototype, followed by the participant trying it out to get familiarised with the interaction.Then a series of simple tasks to make use of the UMM button and its features was conducted, followed by filling in a short post-task questionnaire and a brief closing conversation to capture additional feedback.For instance, as shown in Figure 9, the task for eBook prototype required the participants to flip through the pages in order to spot the given image (via the page-flipping view) and subsequently identify the specific source located below the image (via the full-page view).For the 3D modelling prototype, the tasks included changing the angle (via the viewpoint change mode) and changing the colour of the 3D model (via the object modification mode).For the video-browsing prototype, the participants were tasked to find a particular comment (via the default view) while watching a video (via the full-screen view).These tasks required the participants to switch between the different modes and to show us whether the UMM interaction designed made sense to them and straightforward to use.Using a 5-point Likert scale where 1 means "very difficult" and 5 means "very easy", post-task questions were asked upon completing all the tasks for each prototype on the level of usability in terms of learnability, action of swapping from Mode 1 to Mode 2, and vice versa (i.e. from Mode 2 to Mode 1).
The three designed prototypes described in the previous section were implemented using Unity game engine, installed to an 8-inch Samsung Galaxy Tab S2 tablet running Android OS 7. All sessions were video-recorded and the tablet logged all touch gestures for further analysis.The logging mechanism was implemented as a part of the software, which runs in the background and records time-stamped touch points of the widgets.

FINDINGS AND DISCUSSION
Our participants very quickly learned what the UMM buttons do and used them with ease, as witnessed in our observation throughout the sessions.This is also confirmed by their responses to the post-task question on learnability, which scored 4.2 across the three prototypes.As expected, all of them were already familiar with video browsing (on their laptops and/or tablets), and almost all with eBook reading (on smartphones, tablets and/or Amazon Kindle) doing at least a few times a week, although only half of the participants were familiar with 3D modelling task and no one had experience in a tablet-based 3D modelling.Based on this observation on the participants' familiarity with the application areas, the order of the three prototypes presented was (1) video browsing, (2) eBook reading, followed by (3) 3D modelling 2 .

Overall Usage Patterns
Figure 10 shows the proportion of time spent in each mode across 27 participants for each prototype, regardless of how the mode is triggered (i.e. by pressing the UMM button/bar or locking it).
It can be seen from the figure that the proportion does vary from one user to another and particularly interesting is that the more frequently used mode in video browsing and 3D modelling prototype is default view (59%,  = 0.00156) and viewpoint change mode (57%,  = 0.0365) respectively.This is statistically significant despite the fact that the assigned tasks were consistent and do involve both modes for a given prototype.Hence, this warrants the need to facilitate a flexible switching of modes on current and future moded applications, and the proposed features of UMM (mode locking and reversing) could meet such needs.Figure 11 shows the time spent between primary and secondary mode across the 27 participants, which simply means whether a given mode is invoked with or without UMM by each user.While it is reasonable to expect participants to spend a lower proportion of their time on secondary mode as compared to primary mode (because it is physically tiring to continuously hold a button), it is interesting to note that participants spent most (27%) of that little proportion while interacting with the eBook prototype, than with other prototypes (see Figure 11).Despite that, eBook is still rated as the most suitable for UMM features to be incorporated, as compared to the other two 2 The order was not randomised because the benefits of progressively introducing the interaction from the most to the least familiar application (thus capturing the participants' learnability of the features introduced) outweigh any drawbacks of inevitable bias.
application prototypes.Reasons for this will be further analysed in Section 6.3.

Figure 11: Comparison of proportion of time spent by each user (each vertical bar) between primary and secondary modes while interacting with video browsing (top), eBook reading (middle), and 3D modelling (bottom)
prototype.
To get a better sense of what these observations mean in the actual usage, Figure 12 shows segments of the interaction log data plotted from two participants (P9, P16) when using eBook reader (a) and 3D modelling app (b), confirming the mode-switching and locking behaviour similar to Figure 4 and notably the temporary mode switching from the reversed relationship.However, we note that the behaviour shown were biased by the tasks given to the participants.One consistent user behaviour that we observed across the different participants and prototypes was the difficulty in reversing the mode relationship after having locked it.This was further supported from the reduced rating (see Figure 13) when asked how easy it was to swap between the modes in either default or reversed relationship.For example, after locking the full-screen view in the video browsing prototype, users assumed and wanted the 'exit' from the full-screen to be triggered simply by tapping at the button once rather than holding it then sliding it to unlock because "it feels like there is one more step with locking" (P6).Similarly, after locking the page-flipping mode in the eBook prototype, they expected to be able to return to the full-page mode by tapping on a page found by flipping through, rather than temporarily going back to the full-page mode then unlocking it because they "would want to do it very fast" (P3).This was further supported from the significantly reduced rating (see Figure 13) for both video browsing and eBook prototype, as compared to that of 3D modelling.While we had assumed that it would be conceptually easier to treat the default and reversed relationships to be equal in terms of how to go from one to the other, it was found that this reversing mechanism should consider the purpose of that switching raised in that specific application and devise optimal way to do it accordingly.reversing the relationship; user did this because s/he wants to be in this mode 'B' for some time (e.g.flipping through the pages or watching video full-screen) before returning.At this point, instead of a UMM action followed by unlocking the mode, a simpler and more direct action (e.g.single tapping on the page found, or tapping on the 'exit' button in the full-screen mode) could be supported to instantly unlock the mode, returning the system back to the default mode relationship.This support is represented by the "shortcut" arrow in the figure, which will be an alternative that users can make use of, in addition to the initially proposed UMM.

Enhanced Model of UMM Interaction
For instance, the same UMM button can be used to facilitate both the original 2-step and proposed shortcut of unlocking Mode B (or locking Mode A).This enhanced model of interaction can then better accommodate the varying users' mental models while dealing with the relatively more challenging reversed mode relationship.

Application Areas
Overall, our participants found the UMM features most suitable for the eBook reading prototype, followed by 3D modelling and lastly video browsing prototype.Based on our observations, this is because during the secondary mode with the thumb pressing, the other hand only had to do simple finger swiping to flip through the pages and no other more elaborate or intensive actions.What are other tasks where such a workflow frequently happens?
Tasks where there is a short and occasional/frequent sub-task that interrupts the main workflow but not requiring heavy interaction during the interruption are the prime candidates: reading a research paper while occasionally jumping to the reference section as cited from the main text is a good example.Alternatively, staying in reference section while intermittently returning to the main text to locate all in-text citations could be where reversed mode relationship would be useful.The search for this type of tasks can also be approached by re-thinking the kinds of tasks where a mode/state typically considered to be primary has little or no intensive interactivity during that mode/state.For example, when interacting with a tablet or smartphone, we spend most time within a particular app rather than in the main, home screen (although when we are not actually using the device we tend to leave it on home screen).This means that an app currently opened is the primary mode while the UMM interaction can temporarily trigger the home screen as the secondary mode during which the user can select the next app to use.Managing multi-tasking or multi-pages are also good candidates, e.g.jumping between many tabs on a web browser, selecting a worksheet in a spreadsheet app, or viewing a desktop screen to select a file.For any specific cases where the temporary mode should be persisted for long, the locking/reversing mechanisms could be offered.Previewing a document, briefly checking a notification/alert detail, browsing a thumbnail panel of photos before seeing a full-size one, looking at a contact list before choosing one for engaging in a chat, etc. also share this behavioural characteristic, thus worth considering the UMM techniques to enhance the interaction.
Many general usability insights were highlighted during the sessions, including the temporary nature of UMM buttons giving a sense of tentative and worry-free exploration as the consequence of pressing is only temporary and returns to its original state.Comparable to hovering a mouse cursor on desktop widgets (e.g.popping up a tooltip or preview), such a tentative and noncommittal gesture can be leveraged to allow the user to explore the features more without worrying about taking an erroneous action; some found tiring to hold the tablet in horizontal orientation with one hand while using the other hand for touching.While our work assumed a typical bimanual posture, and did not particularly focus on various manners a device can be held, designing for a specific application that employs a UMM will need further considerations in its specific usage and how its associated locking/reversing features could further enhance the experience.
The exploration as described in this paper brought about other novel arrangements in UMM interactivity.For example, Figure 15 shows a possible, extended UMM interaction with 3 modes: starting with Mode A as the default, the user can temporarily go to or lock to either Mode B or C, where the locked mode is the main task the user is doing while occasionally going to the other two for short periods of time.Although past studies have designed interaction to facilitate more than two modes (Hamilton et al. 2012;Foucault et al. 2014;Pfeuffer et al. 2017), none of them implemented locking or reversing of modes, hence only allowing one fixed default/primary mode which in turn might lead to inflexible user experiences.We expect instrumenting UMM features evaluated in this paper will point to many useful possibilities that will enrich our future interaction with multitouch devices, and potentially be applied to other modalities e.g.VR/AR and on-air gesture.

Limitations and Future Work
The premise of UMM interaction in the use of multitouch tablet implies that a particular mode lasts while a part of the screen is pressed down by a finger.Despite its intermittent frequency, the screen is partially occluded while the mode is maintained.This occlusion problem becomes more significant when UMM interaction is applied to multitouch devices with smaller screen real estate (e.g.smartphone and smartwatch), and hence it might make more sense to use non-screen elements (e.g.physical button on the side) or adopt alternative input modality to facilitate UMM.Also, while the focus of the study was to characterise novel UMM features and assess its usability in a qualitative way, a further study could instrument a more quantitative evaluation of the efficiency and overall experience between the apps with and without UMM features.Finally, a longitudinal study could be carried out in the future to capture the day-to-day, longer term usage of the UMM features, which will reveal interesting usability insights.This will help to refine existing features and possibly come up with more novel features that can support effective UMM interaction.

CONCLUSION
Our view is that the multitouch interaction available in today's hardware platforms in the market has a lot more to offer than tapping, swiping and pinching which in essence takes the point-and-click paradigm inherited from their desktop counterpart applications.Furthermore, research efforts which make use of UMM also have a lot more to investigate than simply applying press-and-hold to maintain a temporary mode.While it can be considered a progress in itself to see a desktop software turning into a tablet/smartphone app so that more people could use it more easily, the multitouch capability of these devices will not see its full potential unless the special characteristics it exhibits are more fully understood and mapped to various tasks and usage scenarios that could take advantage of them.We hope that the usermaintained interaction and its associated features for multitouch devices as explored in this paper will help diversify the existing gesture vocabulary and enrich our day-to-day experience with increasingly ubiquitous tablets and smartphones.

Figure 1 :
Figure 1: User-maintained functionalities with locking/reversing mechanism: (a) eBook prototype in reading mode, (b) page flipping mode, and (c) finger occlusion-free locking guide (vertical); (d) 3D modelling prototype in viewpoint change mode, and (e) finger occlusion-free locking guide (horizontal).

Figure 2 :
Figure 2: Examples of physical objects that employ UMM interaction.

Figure 3 :
Figure 3: Diagrammatic view of UMM with/without locking and their expected usage patterns.

Figure 4 :
Figure 4: Usage pattern expected for the reversed mode relationship of UMM.

Figure 5 :
Figure 5: Incorporating the UMM features using (a) a button to show the horizontal locking guide above the thumb; (b) a sidebar to show a vertical guide.
(a) and Figure 6(a)).Anywhere on this bar can be touched to trigger the mode change.The user can move between pages by swiping right/left anywhere on the screen without any mode change.

Figure 6 :
Figure 6: UMM interaction for eBook reader prototype.(a) shows the position of the sidebar in full-page view.(b and c) show a (finger occlusion-free) guide to locking; (d) shows the page-flipping view.
(a)) to incorporate the UMM interaction in accordance with other parts of the interface.The user is by default in the object modification mode (see Figure 7(a)); upon pressing the UMM button (and only while keeping pressing) on the left of the screen, the mode temporarily switches to the viewpoint change, with the horizontal lock guide just above the thumb (see Figure 1(d), (e) and Figure 7(b)).If the user wants to stay in the viewpoint change mode longer, s/he can lock that mode by sliding the thumb to the right (see Figure 7(c)).

Figure 7 :Figure 8 :
Figure 7: UMM interaction for 3D modelling prototype.(a) shows the initial position of the movable button in object modification mode; (b and c) show a (finger occlusion-free) guide for locking the mode; (d) shows the viewpoint change mode locked as primary mode, with the UMM button slightly moved down by the user.

Figure 9 :
Figure 9: A demonstration of a mode-switching task designed to evaluate the eBook prototype.

Figure 10 :
Figure 10: Comparison of proportion of time spent by each user (each vertical bar) between each mode while interacting with video browsing (top), eBook reading (middle), and 3D modelling (bottom) prototype.

Figure 12 :
Figure 12: Interaction log data showing the locking/reversing patterns in eBook (a) and 3D modelling prototype (b).

Figure 13 :
Figure 13: Comparison of participants' usability ratings between the default and reversed mode relationship across the three application prototypes

Figure 14 :
Figure 14: A "Shortcut" to reverse the mode relationship.

Figure 14
Figure14illustrates this point: a user switches the mode (from 'A' to 'b' on the top half of the figure) then locks it (from 'b' to 'B' on the right half of the figure) reversing the relationship; user did this because s/he wants to be in this mode 'B' for some time (e.g.flipping through the pages or watching video full-screen) before returning.At this point, instead of a UMM action followed by unlocking the mode, a simpler and more direct action (e.g.single tapping on the page found, or tapping on the 'exit' button in the full-screen mode) could be supported to instantly unlock the mode, returning the system back to the default mode relationship.This support is represented by the "shortcut" arrow in the figure, which will be an alternative that users can make use of, in addition to the initially proposed UMM.For instance, the same UMM button can be used to facilitate both the original 2-step and proposed shortcut of unlocking Mode B (or locking Mode A).This enhanced model of interaction can then better

Figure 15 :
Figure 15: An envisaged novel usage behaviour for UMM interaction with 3 modes.

Table 1 :
Summary of physical examples that employ UMM interaction.