Complex Systems Models of Cognition for HCI

My research investigates dynamical systems models of human behaviour, evaluating these approaches for HCI research. I focus on how so-called interaction dominant (ID) dynamics in individual behaviour can be used to develop low cost an unobtrusive measures of skill learning, task-engagement, and executive function during interaction. ID dynamics are ubiquitous in human behaviour and indicate the central role of complex, nonlinear network effects. Changes in these ID dynamics have been related to high-level features of human behaviour and experience, many of which are of interest to HCI. I adapt this prior research to the conditions and concerns of HCI.


Overview and Background
My PhD work builds on recent research in cognitive science understanding the body's role in intelligent behaviour. It asks whether we can understand behaviour and experience -for example how engaged or mentally fatigued a person is by observing patterns in their behaviour during technology use. In particular it asks whether we can do this through commonplace components like keyboards, mice, and accelerometers.
I am interested in whether the IDVC may be a source of methodologies for HCI and system design; providing useful, convenient methods for observing and understanding high level features of user behaviour during interaction with technology.
In the past two decades research in the IDVC has progressed considerably, developing new experimental methods to quantify the significance of this "interactivity" in human cognition, and in particular its role in flexible, context-sensitive behaviour (Likens et al. (2017); Anastas et al. (2014); Kelty-Stephen et al. (2016)). I evaluate these methods for the measurement of high-level behavioural features include engagementwith-task, skill learning, and executive function (flexible selfcontrol during the execution of some task). To my best knowledge this is the first work focused on bringing the IDVC approaches into HCI.
Based on the very brief introduction above, many readers may expect that the signatures of interaction dominant dynamics are primarily neural -measured, for example via E.E.G. In fact, while ID dynamics have been been demonstrated in neural signals (Wallot and Kelty-Stephen (2018); Likens et al. (2014)), it is more common to analyse overt behaviours, such as hand movement (Dotov et al. (2017); Likens et al. (2015Likens et al. ( , 2017). I think this makes these approaches particularly useful for HCI research: they do not require specialist equipment and can be measured from overt behaviour directed towards the task.
It is possible to use overt task-directed behaviour, rather than more covert physiological signals, because interaction dominance is fundamentally a multi-scale phenomenon (Ihlen and Vereijken (2010)). It emerges from nested interactions within and between neural and non-neural systems, as well as between these and processes in the environment -including tools and other agents. It has been shown that when systems are organised in such a way the signatures of interactivity (in which we are interested) are reflected in all components which are substantially incorporated into the system's functioning (Huke (2006)). As such, we can observe ID dynamics in behaviour such as the movement of hands on a steering wheel (Likens et al. (2015)), or on a mouse (Dotov et al. (2017)). This makes the approach potentially convenient and unobtrusive; easy to integrate into system design without highly specialised equipment, and without requiring the use of user report, or distracting secondary tasks.
My research questions are: RQ1 Do multifractal signatures in user interaction correlate significantly with high level features of user behaviour and experience: engagement with task, mental fatigue, and skill learning?
RQ2 Do these correlations between the measure and behavioural features hold across mechanically different interaction modalities -at first keyboard use and mouse use.
RQ3 What practical barriers are there to the implementation of these analyses in real world computer systems, and to their application to new interaction modalities.

Research to Date
My research to date has largely focused on experiments to evaluate and develop these measures for use in user experiments and for developing systems which could respond appropriately to user state. Since my focus has been on carefully validating these methodologies and understanding the barriers to use in HCI, I have limited my focus to two interaction modalitiesmouse and keyboard. These were chosen as they are very common modes of interaction, but at the same time mechanically quite different from one another -allowing me to test portability of these multifractal approaches.
My first work built on prior research demonstrating ID dynamics in mouse use behaviour during a skillful computer game (Dotov et al. (2017(Dotov et al. ( , 2010). This research showed a connection between ID dynamics and changes in the "readiness-to-hand" of a tool 1 . Building on this work, I ran 3 user experiments on mouse use (N=28, N=44, N=30). The first replicated prior results while clarifying some issues particular to HCI applications, while the second two focused on hypotheses which had 1 a notion from Heideggerian phenomenology which describes the transparency and adequacy of a tool during use, and which has been influential in UBICOMP and HCI (Winograd et al. (1986) not previously been demonstrated experimentally. The results of these experiments showed that ID dynamics also correlate to task-engagement, and skill learning features pertinent to HCI -while also removing features of prior protocols which might be expected to reduce ecological validity in interaction scenarios.
The data captured also allowed me to focus on practical of implementation in technology -such as the possibility of measuring movement directly through the mouse rather than via an accelerometer.
My current research focuses on keyboard use another modality in which ID dynamics have been demonstrated (Likens et al. (2017); Wallot and Grabowski (2013)). In this work I aim to establish a link between signatures of ID dynamics and mental fatigue -pointing to the possibilities of these techniques being used to identify fatigue in users. I also focus on practical issues for implementation in HCI -e.g. the minimum number of key presses required for effective analysis, and the effect of timing issues in browser environments on data capture and analysis.
My hope is that ID approaches could ultimately be useful for supporting users in text composition tasks, and perhaps also in identifying fatigue in safetycritical environments. If ID signatures can be shown to robustly predict patterns in engagement and fatigue, then this might be used to help adapt and time interventions by software -whether contextual help, suppression of distracting notifications, or the timing of break suggestions.

Contributions, and Future Directions
My PhD makes two kinds of contribution. First it contributes a careful analysis of the use of multifractal analysis to understand user behaviour in two interaction modalities. It establishes approaches for inferring high level features of user behaviour and experience -task engagement, skill on task, mental fatigue -from the multifractal analysis of interaction behaviour. It also identifies practical issues and limitations when applying these techniques on these modalities in real world situations. This work also contributes techniques which will support these techniques in a far wider range of interaction modalities. As the first work to focus on IDVC methodologies in HCI, I have been careful to establish techniques which can be applied across a range of modalities, and which will support future research in new interaction scenarios. For example I contribute a method for parameter tuning which is modality independent, and which can help in applying these techniques to new input data, where there is no prior guidance on parameter selection. This is important since ID dynamics have been shown to be ubiquitous across a wide range of human behaviours -from physically skillful tasks like balancing (Morales and Kolaczyk (2002)), through complex language production (Likens et al. (2017)), and skilled crafting (Nonaka and Bril (2014) up to coordination during a collaborative tasks (Likens et al. (2014)). The workflows, methodologies and guidance established in my thesis will provide a strong base from which future researchers can explore other interaction typeswearables and IoT devices, smartphone data, and a wide range of other systems.
Finally, while my PhD focuses on quite practical issues of user measurement, my interest in the IDVC was inspired by a more ambitious longer term goal: that of understanding agency and autonomy during interaction. In the longest term I hope my research can contribute in a small way to progress towards this understanding. Given the practical focus of my work, this may seem a distant prospect -but work including Juarrero (1999), Wheeler (2005Wheeler ( , 2018 and others (e.g. Chemero (2013); Paolo et al. (2017); Meacham and Prado Casanova (2018)) points to the value of complexity approaches, and interaction dominant formalisms of behaviour for understand how human agency and autonomy arises and is moulded by action in real contexts. Researchers have noted that HCI has tended to lack tools for understanding how agency and intentions are moulded through interaction Hornbaek and Oulasvirta (2017), and I believe the IDVC, and measures of the role of "interactivity" in behaviour offer important resources for addressing this. In part I see the work in my PhD as a necessary baby step towards developing new methods for understanding agency and autonomy in HCI.