A Design Space for Memory Augmentation Technologies

The pervasive nature of display technologies can enable novel ever-accessible memory aids to address deterioration caused by ageing and cognitive decline. To date, however, memory has largely been treated as a single-unit, and there has been little formal consideration of how to select the most appropriate technology for a given intervention. We build on existing domain knowledge from neuroscience and psychology to suggest a novel design space with two axes: processing level, and display modality. In particular, we consider how augmentations might intervene at a biological, cognitive or meta-cognitive level using head-mounted (private) displays, small-scale (personal) displays, larger public and semi-public displays, and with technology that bypasses the visual channels entirely (e.g. through neural stimulation or non-visual senses). We then provide examples of potential studies to explore these design areas, and discuss future directions this approach to memory augmentation may take. Consideration is also given to the ethics of memory augmentation.


INTRODUCTION
Technology and tools to address physical deterioration that emerges as a result of age or illness are commonplace. More recently, researchers have begun to express a similar vision for technology use to address limitations in cognitive function, including memory (Chen and Jones, 2010;Davies et al., 2015;Harvey et al., 2016;Hodges et al., 2011;Hodges et al., 2006;Iwamura et al., 2014;Le et al., 2016;Mikusz et al., 2018;Rhodes, 1997;Schmidt, 2017). Most commonly, lifelogging devices and other data sources are used to provide cues to help rehearse personal experiences, known as episodic memories (Harvey et al., 2016;Hodges et al., 2011;Hodges et al., 2006;Le et al., 2016). Other memory augmentations tackle concepts such as prospective (Rhodes, 1997) or procedural memory (Seim et al., 2014;Seim et al., 2015).
Despite addressing a variety of types of memory, these augmentations have typically focussed on inthe-moment experiences of memory, particularly for episodic memories. Other understandings of memory drawn from a variety of disciplines (including psychology, neuroscience, philosophy and sociology) are yet to be used as the foundation for memory technologies. This paper aims to formulate a design space that considers the medium through which memory augmentation is presented, and the aspect of memory being augmented. Through this design space we identify where previous memory augmentations have targeted, and where future research may take place. We posit that the design space for memory augmentation is far broader than is reflected in existing literature. Considering memory augmentation through the lens of this design space will allow for further investigation into effective memory augmentation techniques, which have previously been overlooked.
We then consider the unexplored areas of the design space, and present potential studies which would serve as a first step in addressing these research areas. We also discuss ways the design space could be expanded to allow for concepts of memory in other disciplines, as well as some ethical considerations to be made when researching and implementing human memory augmentation.

THE DESIGN SPACE
We suggest two concrete dimensions as a foundation for a taxonomy of existing systems. Firstly, processing levelthe conceptual level at which a given augmentation is operating: Technology may be used to impact one of these levels, or potentially multiple levels. A survey by Madan (2014) gave an overview of approaches to augmenting memory, which focused on techniques which fit into these processing levels. Two examples affecting the biological processing of memories were given-nootropics and brain stimulation. While these are quite different techniques, the former involving the taking of pharmacological agents such as caffeine, and the latter involving the stimulation of neurons, they both impact a person's biological response to memory stimuli. Thus in the present design space, we group these together as impacting the biological level.
Madan also gives an example of augmentation on the cognitive level-that of external aids. These are aids which support cognition in the present, which in our design space represents the cognitive level. This in-the-moment augmentation is what most would consider when thinking of technological, memory augmentation.
Madan's final example is the use of mnemonics. This is a memory technique to aid cognition which people can utilise. We classify this as the metacognitive level. Meta-cognition is the term given to thinking about thinking. In terms of our design space, we consider it the use of technology to augment memorisation techniques, or the monitoring of cognitive abilities.
Our second axis is display modality. Whilst not all memory interventions are visual, the majority are, and displays have played a significant role to date. We therefore build on prior classifications of displays (Muller et al. 2010) to categorise this axis as follows: (i) Non-displays: internal (implantable) and external technology which does not provide visual feedback to the user; (ii) Private/Head-mounted displays: technology with immersive properties which are attached to a single person; (iii) Personal displays: smaller displays, which although potentially shareable, are designed for individual use; (iv) Semi-Public and Public displays: technology targeting a wide audience.
We focus this design space on visual mediums as they are pervasive within HCI, however this is a limitation in terms of working with visually impaired users. While devices such as haptics are included in the non-display level (e.g. Seim et al., 2014), there is no differentiation between those external devices and implantable devices. As such it may be necessary to expand the definition of 'non-displays' to better highlight the use of other sensory displays, such as audio cues and haptics (for further discussion of this please see Section 5).

TRENDS IN MEMORY AUGMENTATION
On reviewing existing literature in memory augmentation technology, we can map these works onto our two axes, creating 12 distinct spaces for augmentation. In Figure 1 we can see these distinct spaces, and where existing work tends to cluster within this model.
On the biological level, memory prosthetic research has begun with the aim of creating implantable devices (non-displays) to aid those with memory impairment (Solis, 2017). Recently, success has been seen in memory implants in epileptic patients (Hampson et al., 2018), although these implants were later removed and so the long-term feasibility of this approach is yet to be understood. Currently, work augmenting biological processes involves invasive procedures and does not utilise existing, readily-available technology. Similarly, work in this area has largely ignored head-mounted displays such as Virtual Reality (VR), Augmented Reality (AR) or Mixed Reality (MR), despite the technology's immersive properties which may be more stimulating than other displays.
More research has been conducted on the cognitive level. For example, Mikusz et al. (2018) used large displays around campus to support student learning of lecture content. While limitations were found regarding the extraneous variables, the study successfully utilised public displays to augment memory on a cognitive level. Likewise, Dingler et al. (2016) placed memory displays in the homes of students revising for exams which were found to encourage participants to study. These displays were in the form of tablets but due to the way they were displayed in the participant's homes, they had the potential to reach multiple occupants placing them on the intersection between a personal and semi-public display. There have also been some studies on this level utilising nondisplays (specifically haptics).
The meta-cognitive level is also relatively sparse compared to the cognitive level, with few interventions targeting these skills. An exception to this is the work by Yang et al. (2020) where they use VR to aid participants in utilising the 'memory palace' (method of loci) memory technique. This is when an individual imagines a location and places the items to be remembered around the location.
Mapping these prior works into our design space [ Figure 1] we can see the areas of augmentation which have received limited attention, and require further research. In terms of processing level, biological augmentation has been largely overlooked, and the main contribution to this level has been invasive technologies. While the task of externally augmenting biological processes is not an easy one, if achieved, it would aid in bypassing the ethical concerns that invasive augmentations raise.

EXPLORING THE DESIGN SPACE
The design space suggests that there are two processing levels that have received limited attention: the biological and meta-cognitive. In terms of display modality, personal displays are the most represented within the research, with few studies investigating the other modalities. In this section, we will discuss some potential ways the under-represented areas of the design space could be investigated in future work. The work in the biological level utilises technology implanted into people to increase the presence of neurotransmitters which aid in the formation of memories. To do this non-invasively, displays would need to elicit the natural production of these key neurotransmitters such as dopamine, serotonin and norepinephrine (Handra et al. 2019). If the display can show an image or give an experience which would elicit an emotion known to stimulate the release of one of these neurotransmitters, then this could augment the biological level in a noninvasive way. However, considerations would need to be made regarding the ethics of such stimulation, particularly in cases where norepinephrine is being stimulated as this is closely associated with fear memory.
The meta-cognitive level is also sparsely researched. The studies at this level have focussed on the method of loci memory technique, by creating memory palaces through VR. However this is a highly individualised set up and may not be suitable for those who experience motion sickness (a common side effect of VR). As such, metacognitive augmentation may lend itself to the use of semi-public and public displays. This could be done by expanding upon the study by Mikusz et al. (2018) where they used public displays to deliver memory cues to students across the university campus. However, instead of just delivering the cues, this system could be used to create a campus-wide memory palace. The displays could present the relevant information but actively tie it to the location to encourage the training of this metacognitive technique.
Despite the recent increase in accessibility of headmounted/private displays, there is still limited work in using these for cognitive memory augmentation purposes. These displays have been suggested to give participants greater senses of immersion compared to desktop computer displays (Shu et al., 2019) and this feeling may create a better learning environment for memory than public or personal displays. Therefore, future research may wish to investigate whether existing cognitive memory aids can be transferred to a head-mounted display. AR may be particularly useful in this regard as push notifications may be presented to the user, without them having to direct their attention away from the environment. For example, an AR shopping list could help the user remember what they needed from each aisle by providing unobtrusive prompts as they navigate the shop.

FUTURE DIRECTIONS AND ETHICAL AUGMENTATION
The ideas in Section 4 are just some examples of future directions for memory augmentation. However, they show the value of mapping relevant research to this design space, as we are able to generate ideas targeting novel research areas. By utilising this design space, future research may identify novel methods of memory augmentation, which will lead to a holistic picture of the ecosystems that best augment human memory. This may then lead the design space to serve practitioners. For example, as dopamine transmission in the brain reduces with age (Bäckman et al., 2006), the practitioner may find interventions for older adults more effective if they augment the biological level to address this deficiency.
Further ideas may still come from the evolution of the design space itself. As noted in Section 2, the present design space does not distinguish within the category of non-displays, leading to invasive technologies such as neural prosthetics (e.g. Hampson et al., 2018) to be grouped with nonvisual, sensory technology such as haptic devices (e.g. Kuznetsov et al., 2009). As research into those fields develops, it may be beneficial to better distinguish between these, as the applications and accessibility of such technologies vary greatly. Further to this, there is some overlap between personal displays and semi-public displays, as exemplified in the work of Dingler et al. (see Section 3). As such, increasing the level of detail explored in the display modality axis may allow for stronger distinctions to be made between these domains.
The processing levels described in the design space are also limited to interpretations from neuroscience and psychology, and as such primarily address the individual. Current understandings of memory are, however, much richer than thisspanning many disciplines (e.g. sociology, philosophy, cultural studies). Whilst these understandings are yet to be the focus of most memory augmentations, as technology develops, it may be valuable to expand the design space to encompass these interpretations of memory. This could be through the addition of new processing levels (e.g. collective memory), or through the addition of new axes. This could enable human memory augmentation on a scale previously thought unachievable, for example by augmenting group memory to aid in the standardisation of technical skills, to reduce the risk of human error.
However, the question of memory augmentation, particularly as it becomes more commonplace within society raises ethical issues. Throughout this paper we have largely overlooked the issue of ethics, beyond that of invasive technology, however memory augmentation itself raises ethical issues. Firstly, the digital divide may limit the accessibility of memory augmentation technology. This may lead to those able to afford the memory aids to have advantages over others, for example in the workplace, where technology may aid individuals in job performance metrics. As such, the goal of this technology should be for it to be universally accessible so that anyone may utilise. This is particularly important in health contexts, where augmentation technology is being used to aid those with cognitive impairment.
Secondly, the act of augmenting one's memory will undoubtedly have knock-on effects, including new potentials for harm and deliberately malicious intervention. In the case of the current design space, augmenting an individual has a limited effect but as the design space opens up to group augmentation, this effect could have widespread consequences. One such consequence could be the use of memory augmentation technology to sway public opinion of national events. If group augmentation is achieved, entire populations could be manipulated into remembering national events differently, causing threats to freedom of thought. As such, the implementations of such technologies must be done with caution and Davies et al., (2015) suggested the need for memory security, to protect memory augmentation technologies from external threats, and protect a person's memories from tampering. This would be similar to the way we use anti-virus technologies to prevent our devices from being hacked. Preventing the hacking of a person's memory would allow these augmentation technologies to be used safely and with confidence that the memories are real. Given this, when considering memory augmentation, the ethical implementation of such tools is a priority as while it has implications for preventing cognitive impairments, it would be a powerful tool if used maliciously.

CONCLUSIONS
This paper aimed to present a novel design space for memory augmentation technologies. Our twoaxis model identifies two key design factors in the creation of technological memory interventions: the display modality and the processing level. The mapping of previous research into this model enabled us to highlight areas in which little research has been conducted, thus allowing a clear insight into the spaces where more work is required.
The proposed directions for exploring this design space highlight the ways memory could be augmented in the future, and shows the benefit of mapping memory augmentation into such a design space. Reflections on future expansions that could be made to the design space also suggest the potential for memory to be considered in an interdisciplinary fashion. However, as we highlight, there are ethical considerations to be made to ensure the safety of such technologies. Overall, the presented design space gives clear insight into current directions of memory augmentation research, and highlights the ways this field may continue to grow.