Can Video-based Qualitative Analysis Help Us Understand User-algorithm Interaction?

There is growing debate in contemporary life over the roles played by algorithms when we browse online. In particular, concerns are raised that algorithmic processes to index, filter and personalise content can ‘manipulate’ user behaviours in ways that lead to detrimental outcomes both online and offline. This short paper reports on ongoing work to examine how users interact with algorithms when undertaking browsing tasks online. Drawing on insights from ethnomethodology and conversation analysis, this video-based approach produces rich and detailed observations of user behaviour and reveals several relevant patterns of interest. Our findings highlight that both user-algorithm interaction and users’ own practical reasoning play key roles in shaping the browsing experience.


INTRODUCTION: CONTEMPORARY DEBATES OVER ALGORITHM PREVALENCE
Over the last 5 years, discussion and debate has grown over the prevalence of particular kinds of algorithms on online platforms. These debates occur across public [1], academic [2] and policy [3] discourses and crystallise around concerns that algorithmic processes functioning to index, filter and personalise the content seen by users may do so in ways that can lead to detrimental outcomes. Controversies have arisen over: i) the capacity for personalisation mechanisms to push users into filter bubbles in which they only see content that reinforces their existing views and discourages critical thinking [4]; ii) the 'gaming' of news feeds on social media platforms to enable the spread of false or misleading information [5]; iii) search engine results that might unintentionally reflect and reinforce wider societal biases and prejudices [6]; and iv) personalisation results that might systematically disadvantage certain populations on the basis of gender or ethnic status [7]. These controversies suggest that, to some extent, algorithmic processes may be capable of 'manipulating' users, leading them towards certain browsing outcomes and post-browsing attitudes and behaviours.

LITERATURE REVIEW: USER-ALGORITHM INTERACTION
The contemporary debate over the prevalence of algorithms on online platforms highlights the value of developing an in-depth understanding of useralgorithm interaction. This kind of understanding can make a significant contribution to the field of HCI and also underpin public and policy debates over the appropriate governance of online platforms [8]. This is a highly complex and challenging area of work not least because the algorithms used by popular online platforms are treated as proprietary and therefore not made available for examination to researchers. For many years there has been an interest in using aggregate data to improve the design of algorithms. This has been driven by the desire to improve user engagement, given wider accessibility to technologies [9], and also for goal orientations such as commercial interest. In many instances data such as click stream data [10] or web log data [11; 12; 13] have been used to develop more personalised information retrieval for users. Such data and statistical analyses have been used to create user profiles in attempts to optimise search results. As data mining techniques and access to 'big data' have improved, such analytic approaches have become increasingly sophisticated, and we have also seen interest in developing means by which user browsing behaviour can be predicted. Whilst such techniques, which are part of a rich quantitative tradition, offer the capacity to exploit large amounts data to help understand user browsing behaviour, there are some fundamental limitations to their use. Primarily, although these techniques enable the visualisation of broad patterns of interaction, these are often abstracted from their context. They gloss over important features surrounding the reasoning and detail of how and why users browse online in different ways.

VIDEO-BASED QUALIATIVE ANALYSIS TO EXPLORE USER-ALGORITHM INTERACTION IN CONTEXT
Given that we do not have access to the content of the algorithms driving online platforms, an insight into how users experience algorithms in their daily browsing may shed light on their actual impact and influence in context. Some existing work has drawn on eye tracking [14] and interview methods [15] to begin to access the sense-making that underpins user behaviour online. Other areas of analysis draw on ethnomethodology and conversation analysis (EMCA) [16] to take an even more fine-grained approach to understanding human-computer interaction. This approach has produced insightful observational findings that reveal how user accomplishment of practical actions, such as workplace tasks, is highly contingent on technological environment and interaction with coparticipants. [17,18].
In this paper we outline an EMCA-based observational approach which seeks to identify how users interact with certain algorithms when browsing. Is it possible to identify how a user interacts with the outcomes of an algorithmic processin particular to pinpoint moments at which interaction with an algorithm might change the trajectory of a user's action? We report some initial findings to highlight how this approach can identify the occurrence of observable patterns of action and interaction.

Methods
The work described in this paper is undertaken as part of a broader research project, UnBias 1 , which explores the user experience of algorithm-driven online platforms. The study is set within the context of current controversies over the prevalence of algorithms online and seeks to promote fairness in relation to the design, development and use of algorithms [19]. We are particularly interested in studying how people interact with algorithms in order to identify what factors enable trust in algorithms and 1 https://unbias.wp.horizon.ac.uk/. how users might be protected against their potential detrimental consequences.
Data collection involved quasi-naturalistic observations of browsing [20]. Pairs of participant volunteers gave informed consent to undertake simple browsing tasks together; the rationale for pairwork was that participants would be required to articulate their reasoning, that would otherwise be unspoken if browsing alone. The tasks were designed to mimic simple, everyday online practices but for ethical reasons did not require participants to log in to any private accounts. The pairs were given a laptop to use that had been cleared of all previous browsing history and cookies etc.; this ensured that, as far as possible, the participants' browsing experience was not shaped by any pre-existing data. The laptop also ran a screen capture programme so that all the users' online actions were captured as video. In addition, the pairs were audio and video recorded to capture their spoken interactions and bodily movements.
In the first phase of the data collection, 15 pairs of participants were recorded and the resulting data were analysed according to EMCA principles. We focused on working across the dataset to identify instances in which participants could be observed as interacting with the outcomes of an algorithmic processfor instance responding to an autocomplete suggestion when typing a search inquiry or responding to the provision of an indexed set of suggestions following a recommendation inquiry. Here we overview a small number of the emerging findings to demonstrate the value of this methodological approach.

Initial Findings
Our analysis enabled us to begin to understand user-algorithm interaction on a granular level. In this findings section we provide a brief overview of observable patterns occurring when users enter an inquiry into a search platform and select a link or links from the results of that search.
As shown in Figure 1, we can observe a typical sequence made up of three steps. In the preparation step users open a browsing window or tab. In step 2 they type their search term into the bar shown on the page of the search platform. This step may include correction, change or refinementfor instance the correction of a spelling mistake or alteration of a search term being inputted. Change may occur when users abandon inputting their own search team and select one suggested by the autocomplete function of the platforma direct instance in which an algorithmic process shapes user behaviour. Once the the search term has been entered a third step occurs in which the user reviews the results of the search. This typically begins with the scrolling up and down of the results on the page (but rarely includes looking at the second page of results onwards) and concludes with the selection of a link or links. In between there may be further refinement of the search term, for example to add more words and repeat the search. In these instances, the users treat their interactions with the search and content filtering algorithms as producing an insufficient or incomplete outcome for the task at hand.

User practical reasoning
All pairs of participants were given the same tasks to undertake. However, the search terms they entered differed considerably. Much of what they selected to type was shaped by and contingent on discussions they had with each other whilst in the preparation phase of the search. As they opened up a tab or browser window the participants discussed together how they would go about the task and made relevant to the activity their understanding of what the task involved and relevant pre-existing knowledge. For instance, in a task to find flowers to be delivered to a friend, some participants indicated an awareness of the commercial delivery chain Interflora, and agreed to use that as their search term. Others who stated they had no personal experience of ordering flowers online entered different kinds of search term. As they moved on through steps 2 and 3 of the sequence, the interactions between participants and their relevant pre-existing knowledge again influenced the actions they took. This led to significantly different outcomes to the task despite its identical form across pairs. So, whilst user-algorithm interaction might be important to the conduct of browsing, it is not the only relevant feature. The practical reasoning of the userin the form of pre-existing knowledge that is drawn on and interactions with co-participantsalso shapes the online experience. This can mean that very different results are achieved from similar algorithmic processes.

Navigating the search results page
Step 3 of the sequence, in which participants reviewed the results of their search inquiry contained a number of features of analytic interest. One feature relates to the ways in which participants navigated around the page of results shown to them. Once again participants interacted with each other whilst interacting with the online platform being used. As they scrolled up and down the page the cursor provided a shared focal point which enabled them to refer to specific content in a way that could be easily and immediately understood by the other. in their spoken interactions. The pairs frequently articulated the practical reasoning that underpinned their actions -and their interactions with the outcomes of the search and filtering algorithms. For instance, they typically oriented to the order of the search results as important to their selection of linkwith results higher up the page being referred to as more helpful and better to select and items on the second page of results (or lower) as not necessary to look at. In addition, where participants used a search engine, such as Google, that provides a series of advertised posts at the top of the results page, participants often moved straight beyond these to focus on the non-advertised results. This behaviour was often made verbally explicit in comments such as "I would say don't click on the ads" and "That's the first one that's not an advert.".

Selecting viable results
A further point of interest in phase 3 of the sequence relates to the ways in which participants selected links from the results page and viewed them. Typically, participants selected a number of links and opened them in a series of tabs or windows. These were viewed and discussed in relation to the task; pages showing sites that were agreed on as not useful to completing the task were closed and viable options were kept open. Sites of particular potential relevance were often referred back to multiple times with a single, most viable, option used as a kind of reference point [21] against which all other pages and possible solutions were compared. This observation once again highlights the role of users' practical reasoning when interacting with algorithms. The results of the algorithmic processes were interpreted rather than being drawn on straightforwardly or without further examination. Even whilst users attended to the indexing function of the search platforms as highlighting content of hierarchical relevance to the completion of their task, they did not treat the content they saw as providing a definitive answer by itself.

DISCUSSION
This paper has reported some preliminary findings of ongoing video-based analysis to observe to useralgorithm interaction. Although the analysis is in an early stage our emerging findings point to a number of relevant features of interest and highlight the value of our methodological approach.
Focusing on the entry of an inquiry into a search engine to accomplish a particular task, we observe a typical sequence involving actions of preparation, search and review. Instances of user-algorithm interaction can be observed at a number of points such as in the acceptance of an autocomplete search suggestion and the orientation to the order of search results as hierarchically relevant to the task at hand. However, in addition to attending to the results of algorithmic processes as relevant to the accomplishment of their task, participants also draw on their own practical reasoning. The search terms they enter into the site and the ways they respond to the results shown to them are contingent on the expression of their own pre-existing knowledge and the discussions they have with their co-participant. For instance, search terms are agreed upon as a result of discussions about users' lifeworld knowledge and the selection of links to follow relates to shared understanding of advertised results as less 'valid' than non-advertised ones. This combination of user-algorithm interaction and practical reasoning means that very different outcomes are reached even when participants are set an identical task.
We also observe similar findings when analysing other forms of user-algorithm interaction occurring in relation to the other tasks we set the pairs of participants in the study. We found that the results of algorithmic processes played a role in content participants selected as relevant (or otherwise) to the completion of their tasks but that once again this occurred alongside practical reasoning. Participants' accomplishment of their tasks was determined in part by their discussions of what could be reasonably and jointly understood as relevant information to input into a platform or how they navigated through the filtering options on recommendation platforms etc. As previously, the presence of practical reasoning alongside useralgorithm interaction led to highly different outcomes to identical tasks.
These emerging findings are of relevance to contemporary debates and concerns over the prevalence of algorithms on online platforms. They indicate that algorithmic processes play a role in shaping browsing behaviours and that users' online experiences are contingent on how they interact with the outcomes of these processes. This supports suggestions that it is necessary to protect against the capacity for algorithms to 'manipulate' user behaviour where it might lead to detrimental outcomes. Nevertheless, the coincidence of practical reasoning alongside user-algorithm interaction demonstrates that the impact of algorithms on users is highly nuanced and context dependent. Understanding the precise influence algorithmic processes have on user behavioursin both positive and negative formsrequires a great deal of sophisticated analysis.
The conduct of video-based analysis using an EMCA approach enables the detailed examination of user behaviour. This produces rich findings that highlight the context dependent ways in which users orient to the outcomes of algorithmic processes when they are undertaking online tasks. This methodological approach is therefore highly fruitful in helping us identify and understand user-algorithm interaction and explore the ways in which it is interlinked with social processes such as practical reasoning.
Despite this, there are several challenges to this work. The collection of data and data analysis are very time consuming, leading to small datasets in comparison to quantitative studies and concerns over generalisability. Although attempts are made to ensure the 'naturalness' of the data collection setting and the tasks set, the participants are placed in a contrived setting that does not totally mirror their everyday behaviours -in particular they are asked to complete tasks in pairs that they might normally do alone and with more time available to them than in our data collection. Ethical concerns limit the sites that participants can be asked to browse since best practice guidance prohibits the collection of data from private social media accounts; as the operation of algorithms on these platforms is of particular research interest this is a significant barrier to the production of relevant findings.
Moving forwards, a particular challenge to be faced is how to identify with certainty precise moments in which user behaviour is impacted by an algorithm. At times this can be easy to isolatesuch as when a user abandons the entry of a search term and selects an autocomplete option. However, in other cases it is much more difficult. This is due to the complexity of algorithmic processes, our inability to know exactly how these processes are functioning and their different outcomes in relation to preexisting (user) data. This challenge will require further unpacking by researchers working in this area as concerns over the influence of algorithms online continues to grow.

ACKNOWLEDGEMENTS
The research undertaken in this study formed part of the EPSRC funded study 'UnBias: Emancipating