Integrating Memory Context into Personal Information Re-finding

Personal information archives are emerging as a new challenge for information retrieval (IR) techniques. The user’s memory plays a greater role in retrieval from person archives than from other more traditional types of information collection (e.g. the Web), due to the large overlap of its content and individual human memory of the captured material. This paper presents a new analysis on IR of personal archives from a cognitive perspective. Some existing work on personal information management (PIM) has begun to employ human memory features into their IR systems. In our work we seek to go further, we assume that for IR in PIM system terms can be weighted not only by traditional IR methods, but also taking the user’s recall reliability into account. We aim to develop algorithms that combine factors from both the system side and the user side to achieve more effective searching. In this paper, we discuss possible applications of human memory theories for this algorithm, and present results from a pilot study and a proposed model of data structure for the HDMs achieves.


INSTRUCTION
Development of hardware recording devices together with associated recording software, and reductions in the cost of in digital storage is now allowing vast digital archives of personal life experiences to be captured.These personal archives, which we call Human Digital Memories (HDMs), can contain various types of data created or accessed by the individual.While most of present 'life logging' projects are still confined to recording users' activities in the 'digital world', mainly concerning user's interactions with electronic files (e.g.documents, emails, images, videos, etc.) that they have accessed on their computers [e.g. 2, 3], another strand of work, which is aimed at recording the real life, is beginning to develop.Research in this area usually involves wearing audio or video capturing devices to track the user's behaviour in a laboratory environment [e.g., 4,6].Many potential benefits have been put forward for such systems.According to Sparck Jones [5], these archives can be used in following ways: storing the past information for a person as an 'Deposit'; amplifying people's memory of events as a 'super me', by providing linked relevant information; showing one's past life (or certain aspect of it) to other people; sharing memory of certain information among different people.By far the most common proposition for real life logging is to provide support for people's memory about their past by presenting them with data captured during their daily activities [e.g, 1,6].However, for any of the above applications, it is essential for an individual to be able to locate and retrieve the desired items from them.We aim to explore an efficient way of information retrieval (IR) for this new type of data collection.
We structure this paper as follows: Section 2 introduces some related work on integrating context data in IR of life logs, and basic notions of human memory with an associated memory model, which deals with how information is encoded and retrieved with contextual cues.Based on his model, we explain how memory works when people perform information re-finding tasks.Section 3 presents results from our pilot study on memory of photos; Section 4 gives a brief description of our present data collection work and presents our proposed model of linking and weighting the collected data in retrieval.

BACKGROUND
Present life logging data usually involves multimedia data such as audio, video or static images.Due to the huge amount of data, it would be a time consuming and tedious to look for a certain frame of scene from one year's video recordings, merely by browsing.For the same reason, it is almost impossible for the users to annotate these data sources manually.Lack of textual content in these types of data makes them difficult to retrieve by traditional content-based text IR methods.Current IR techniques for audio and images collections are based on content analysis of low level features.The lower quality of HDM data may significantly reduce the accuracy of automatic semantic annotation with content-based methods.Indexing these types of information with their embedded timestamps may be a solution.Yet, just like other textual types of data, traditional query based searching in IR systems relies too much on the user's ability to recall accurate details about the searching target, such as the key words, titles, and the exact time.
The 2nd BCS-IRSG Symposium on Future Directions in Information Access 2

RELATED WORK
Existing studies have been trying to cope with the problem of indexing features by allowing manual annotation and then searching files with well remembered types of features such as episodic context information [12].Hori et.al. [17] have tried to log people's life with wearable camcorders.They used simultaneously captured context information to help index and retrieval.Apart form the microphone and camera embedded in the camcorder, they also had a GPS device to provide address keys, latitudes and longitudes; Biometric devices such as acceleration sensor, and a gyro sensor and a brain wave analyzer were employed in their system to capture the wearer's motion and mental status information.Face detection technology was also used to indicate the people shown in the captured video.With this context information embedded in the life log video, their system allowed the user to locate and retrieve required episodes of video with information about their corresponding experiences, such as their emotion, location, action, weather and other people shown up.
Other example systems include Mylifebits [7] and MediAssist [14].In Mylifebits, contextual information such as location, people, and date are used, they do not only assist retrieval, but also link events by these data [7].MediAssist uses extended types of both content and contextual information in a photo retrieval system, and allows more flexibility, for example, instead of limiting the date time to accurate numbers, which the user may not necessarily knew even at the time of the occurrence of the event, it enable them to roughly decide the range or period, which is more likely to be perceived and therefore remembered from the time of capture [14].
Significantly personal archives are different from other more traditional information collections in that they comprise information that is or used to be in the owner's memory.Therefore, they possess more potential for interaction with the features recalled from the owner's memory.We assume that if well remembered information can satisfy searching needs, it may to some extent relieve the burden of human memory at the time of re-finding, and make the IR (based on the information that can be recalled) more reliable.Combined with the presentation of related clusters and iterative rounds of searching outlined above, we believe that this can form the basis for improved search techniques suitable for the HDM domain.As very little computer-human interaction literature explores the link between life-logging technologies and human memory, we aim to put more effort in this aspect by exploring the possible application of human memory from cognitive theories or models into technologies for IR from HDMs.To have a better idea of how to utilize human memory for more efficient and less effortful searching, we first look into the information processing of human memory.

Encoding and working memory
Encoding can be defined as a set of operations that people use to code incoming stimuli.These processes modify and organize the arriving data by associating it with information in their current memory.Encoding involves sensory memory, working memory, as well as long term memory.Sensory memory, which is also called sensory registration, stores massive amount of raw data from physical stimuli features very briefly (<1 second) before processing them.This kind of memory decays very quickly and is replaced immediately when new sensory information is registered.Working memory holds limited information (typically limited to 7±2 items) for a longer time (usually no more than a few minutes) while processing them [1].

FIGURE 1: Chunking Structure
Due to the limited capacity and durance of this short term storage, some memory strategies are either intentionally or unconsciously employed to make maximum use of each processing stage.These strategies are particularly important for explicit intentional learning.Chucking is one strategy most frequently observed in learning experiments.It breaks big information sets into small pieces randomly, but usually base on sequential proximity [3].Strategic chunking applied in learning enables short term storage holding 7±2 chunks of items, which can be expanded to smaller units [1].For example, if you were asked to remember a list of random digits, 987398798379831708200812345, it would be very difficult to hold them all in mind to repeat.However, if they were broken into pieces, for instance, 9873987983 happens to be a phone number you remembers well, 17082008 can be considered as a date17-08-2008, then user can chunk them into 9873987983-17-08-2008-12345; The chunked sequence is then much easier to hold in short term memory.The strategy behind this chunking might be to extract the lower level elements from long term memory where the chain of these lower level elements resides.The lower level items are of strong association, thus requiring less cognitive or attentional resources to retrieve them one by The 2nd BCS-IRSG Symposium on Future Directions in Information Access one.This process schema may be able to inspire some algorithms in IR, which for instance, only processes the higher level nodes (chunks).

Storage and retention
Storage refers to the process of placing the coded information into memory system for long term storage, similar to saving a file on the hard drive.Representation is the format in which the information is stored in memory, e.g.spatial, auditory, and semantic [1].One of the most important issues in memory storage is the retention of memory, which means how well the information is stored.Among many studies investigating this issue, the basic two most widely consented factors determining retention are: time lapse since encoding and frequency of repetition.It also depends on the initial strength determined by the encoding quality at learning [18].Information processing theories have suggested that human memory exists in an associated network where the nodes of memory are bidirectionally linked; the stronger the link means the easier to evoke the associated information [19].During each repetition, the item is re-encoded, thus the memory of one item may be bonded (linked) to various contexts.While the links between the item and most of the context information fade overtime, some association may be reinforced due to repetition.Also, as some information (context) is more readily associated with the item (target at the retrieving task), possibly due to their pre-existed relationship, the link between this context information and the item is likely to encoded with greater strength and retained for longer.For this reason, we can expect an uneven distribution and dynamic change of memory about the items and their related context.

Retrieval and forgetting
Retrieval is the process of recollecting information from long-term storage, and presenting the output [18], which can either be detailed information of the retrieving target or a judgement of whether the target exists in one's memory.These two types of outputs are referred to as remembered and known.The remember/know paradigm is widely used in studying the human memory, corresponding to recall and recognition tasks respectively.
According to psychology literature, forgetting (of the past) is usually caused by the inability to retrieve the item from memory rather than the lost or damage of storage.That is, the inability of tracing back to where that piece of memory is stored [1].It explains why cued recall tests (prompted with related information) usually results in better recall performance than free recall tests (with no provided cues nor subject to any specific order).Thus, we assume that a user can have better recollection of required information for searching (queries), if they are presented with relevant information as cues.As for free recall, most of the ideas that pop out are in fact either cued by the previous recalled piece of memory (thoughts) or triggered by external environment, which possesses certain elements that are associated with the piece of information in long term memory.Clustering is usually observed in free recall tasks.It can be considered as a more advanced version of chunking in retrieving, as it groups information according to some higher-level criterion, such as cross-modality similarity [16].Forgetting has also been argued as an mechanism of filtering out unwanted memory, that means it is possible that people forget things because they do not think they want to remember them (i.e.unimportant, or cause too much pain remembering that) [1].Yet, it is also possible they might want to use this previously unwanted information, but are unable to retrieve it due to forgetting.In this case, an HDM can potentially shows its advantage as a 'repository'.However, it also faces the same problem as forgetting in human memory of being unable to locate the information.

Context Factors that Influence Memory Retrieval
Retrieval and the popping out of associated (or clustered) ideas are triggered by the interaction of external information and internal context [20].Context is a vague concept.In the physical world, it usually refers to the external physical environment, including temporally and specially surrounding information, which is assumed to be encoded together, associating with each other, and acting as cues at retrieval.These types of related information present at the time of encoding or retrieval, are called external context, while the internal context is the activated information in human memory.When new information comes in, it broadcasts to the stored memory and the preexisting nodes which are active enough (above the 'to be perceived' threshold), these nodes react to the broadcast if they can be associated with the incoming information according the clustering rules.The threshold is determined by the effort or energy at the time of broadcasting.Those high priority links with best matched nodes pop out and construct the internal context which interacts with the input and leads to the searching target.It resides in short term storage waiting to associate or to reinforce association with the input information [21].

Human Memory in Information Re-finding Tasks
When an individual is motivated to re-find some information from their previously accessed information (e.g.files on their desktop), the first recalled pieces of information of the item will trigger clusters of memory which they belong to.These clusters of information form the internal context which contains both the information of the external context in the events when encoding the item, and those associated with the item based on (cross modality) similarity.There two general types of memory cluster, either episodic memory of the related information of the events during which the item was encoded, or semantic memory where the item conceptually resides in (e.g.information of the same the category).With the memory searching criteria provided by the system interface, the individual could decide which route or cluster to look into in order to retrieve the corresponding memory.For example, to search for (or re-find) previously accessed data, users need to look into their own memory to find information which is needed to perform the searching task.For example, the Microsoft Windows desktop search (WDS) allows the user to search by file name, path, file size, etc. while searching for his or her previous accessed files, the user needs to recall the information that the above fields require, in order to be able to perform the search.The recalled information will then be transformed into the form of a query to perform the IR task in the search engine.The matched results will be returned to the user to judge and decide whether one of the items is the requested target.When these results are presented, they act as an external input, which will modify the user's internal context.This change of context (usually new information is added) means that there is a possibility of recalling more related information, which does not belong to the clusters in the previous internal context.The newly recalled information may therefore lead to another round of searching.Admittedly, the user's memory may make mistakes, going to the wrong route (and misleading queries) or get a false alarm in recognition of suggested information.In these cases, an automatic estimation of the memory retrieval reliability would be a great help.

Related work on memory of personal data
A previous study tested people's memory about several types of content and context information of their own photos [12].The participants were required to free recall 3 tasks followed with an implicit evaluation of the features of their interface, which offered several categories of context fields.Their study found that outdoor/indoor classification, location and people were all well recalled and proved to be useful cues for retrieval.Location and time derived context such as weather, daylight status and season were also proved to be useful.In particular, local time and daylight status seem to be stronger cues than the date/time of the photo, or even actual year if the photo was taken long ago.Finally, people also seem to remember photo colours, but its role as a retrieval cue was not tested.
Further studies explored memory on extended types of data from a small HDM, with a concern of decay of memory over time [6] [11].Memory of context information (involved in this data collection) showed a generally consistent pattern with [12].Comparing with the participant's recall performances 6 month ago, we saw a varied degree of memory fade for different context information sources for the searching targets, e.g.semantic memory about textual information may decay faster than the episodic contextual information; also different behaviour of accessing the files influences and affects the memory recollection, e.g., self-generated data were generally better recalled than passively presented information.We also noticed that the keywords (content) which the participant selected to describe the documents 6 months ago were not necessarily the same as she recalled 6 months later.This finding is congruent with the memory model mentioned above, in that, as people's knowledge changes, their internal memory context may became difference from the target time (6 month ago) this item was encoded, therefore, the first few key words that the file links to changed.Thus, for the files whose retrieval is mainly based on metadata (e.g keywords), the mismatch between recalled features and the annotation might lead to a decline in their reliability for use in retrieval.
Traditional ways of automatically extracting keywords or summary content are based on statistics, e.g. term frequency (tf) and inverse document frequency (Idf).However, according to above findings, the most significant items as indicated by tf-idf weights may not necessarily be the ones that users are most likely to remember accurately for an item, although the frequent appearance of certain terms may to some extent improve the user's memory about them.A possible application of this finding is to modify the tf-idf score with recency and frequency components.For example, we may assume that the more recently frequently encountered terms are more important, as they are more likely to be recalled and therefore used it as queries.
According to various reasons mentioned above, static weighting of the metadata may be insufficient to achieve the best possible IR effectiveness.Besides, personal differences, e.g.different life style (for example, some people have most of their activity at the same place, thus location may not be distinctive for retrieval, while some people are frequent travellers, the locations may be a more distinctive and strongly associated context information) may also benefit from a more dynamic way of weighting the information.We believe that dynamic annotation or weighting of the annotations (or metadata) according to likelihood of being recalled correctly may be a solution for this issue.One of our goals is to explore how to use the above memory model to estimate the memory of individual metadata based on the data we capture.To estimate the recall possibility, an algorithm is needed to calculate and update the memory status of the data owner.According to the associated memory model, the ease or likelihood of memory being evoked and retrieved depends on the strength of links (either direct or indirect) to the provided cues.One essential step is to establish these links.

PILOT STUDY
We conducted a further new pilot study on memory of photos from the participants' own photo collection, to explore the types of association human memory uses.The reason for using photos is that they have embedded contextual information in themselves, which means no specific data collection for the experiment is needed.The study described in [12] has already provided very useful and detailed data on well remembered features for photo collections such as location, indoor/out door, people, and weather.In this study, we want to further explore the reasons and factors that influence memory and retrieval of photos.

Methods
Subjects: Three graduate students (all around 25 years old) participated in this study, two of them were interested in travelling and photographing, and the other one had some knowledge of multimedia IR.
Material: An electronic questionnaire was used to test their recall performance.It included one field for content, one for the photo's location (e.g.folder name and path on their computer), and other fields for contextual information, including 'your location', 'year', 'season', 'month', 'day' 'date', 'period of day', 'weather', 'people around'.This questionnaire also allowed the user to hide the content that they want to keep as private.Different from [6][11], the participant did not intend to collect data for re-finding task, which means that it was collected in a more natural setting.
Procedures: Instructions were given for each field before they started.They were asked to free recall any 20 of their photos from any period of their lifetime, and to input the photo's content and context into corresponding fields in the questionnaire, as soon as the photo popped into their mind.The participants were also asked to choose whether they took the photos themselves, which was assumed to indicate self-involvement, as according to our previous study, self-generated files were generally better remembered than passively present files.
A post-test face-to-face interview was also conducted to investigate why and how they recalled these particular photos, and if there are certain associations of neighbouring photos.This step aimed to explore why some photos are better remembered, and how the external context at the time of retrieval (e.g.previous photo) triggers memory activities.For example, one may recall a photo which is not so meaningful, but may be somehow related to the previously recalled one.Finally, the participants were asked to re-find these photos and check their recall results.

Results and Discussion
The results are generally consistent with the findings in [12].Experience related information such as the participants geographical location (98%), weather (or light status) (90%), the season (89%) and period of day (65%) were well recalled, while the exact day/date (12%) is seldom remembered unless the number is particularly bounded to the event, e.g. one participant only went to a certain place and met certain people on Fridays, thus if he remembered the photo was taken on those occasions, 'Friday' can be deduced.'Months' are well remembered by two participants for the photos taken during their travels (82%), but not by the participant who did not have much travel experience (32%).'Years' are usually well remembered (96%).unless the event was remote, e.g. more than 5 years.'People around' are also well recalled, though it cannot be checked only from the photos.'Locations' (path) of the photos also had sound recall, possibly because that the participants organize their collections very well.Yet due to the considerable number of photos in each folder, the participants reported a searching tool might be of great help.However, the self-generating effect (photo taken by the participants themselves) did not show the same advantage as it was in our previous pilot study.The result may have some ceiling effect due to that the first recalled photos might happen to be what the participants remembered well.This result suggests that in our future studies certain criteria of searching which increase the difficulty of recalling should be employed.
The reported reasons for recalling photos generally fell into five categories (or combinations of these categories): interesting (46%), novelty or impressive (31%), frequently seen (18%), recently viewed (15%), and 'no particular reason, but it popped into my mind' (6%).The last category suggested a possibility that previous ones triggered the recall of these photos.This kind of trigger was also found possible in other neighbouring photos.This finding gives an example of the memory clustering mechanism at retrieval.The association found in this study included: things in the same event (e.g. in the same trip), similar occasions (e.g.gathering together with certain group of people, or The 2nd BCS-IRSG Symposium on Future Directions in Information Access during festival celebrations), same location or similar type of location (e.g.train stations, parks, indoor).This implies a potential need to link files based on such kinds of information, and sheds light on interface design for presentation of results, e.g. to group the results by user preferred clustering approaches.Further study will be needed to explore other possible association or clustering approaches employed by human memory at retrieval time, with more data types.
When the participants looked back into their photo collections, there were a considerable number of photos that they could not recognize, and thus have no recollection of any information referring to these specific photos.This means these photos will not be searched.However, such photos might match some of the primary searching purpose, though they did not have these specific items in mind while performing the searching (re-finding) task.It would also be helpful to present these photos to them when what they actually want is a cluster of results, not specific files.Alternatively, the user may not wish to see the photos as all, and it would be interesting to explore means of suppressing the retrieval of items that are unlikely to be of interest to them at that moment.Again, the linking of files based on certain criteria, which the human memory uses to cluster information at retrieval, is desired.

Data Collection
In order to do further research on extended types of data over a much longer period, we have started a long term data collection exercise.This will be gathered from across one-year period with four participants.We are collecting computer activity, and other related data from their daily life, for example: • Computer activities are logged by desktop software, Microsoft Digital Memories1 (DM) and S'life, every time an application comes to the foreground.The full textual content of the documents, WebPages, and other applications, as well as the file name and path, etc. are be recorded.
• SenseCam image: The SenseCam [8] is a wearable camera, which passively takes photos with its fisheye lens, and store in 640x480 JPG format.It can be triggered by scheduled time (e.g.every 20s) or change of environment such as a detection of people, change of luminance.It takes about 3000 photo per day from one person's daily life.
Bluetooth is used to detect the surrounding Bluetooth devices, which can indicate the corresponding people (e.g. who have the Bluetooth on in their mobile photos) and objects (e.g.computers with Bluetooth on) [3].Content based technologies such as face detection may also be a supplement, but not the main approach to detect people, due to the low quality of SenseCam images.It is also because that the wearer may not necessarily know the content of the SenseCam images, e.g. who were in the photos, since they may not want to review the images captured everyday.Also, as they may remember their experience context, such as who was present in an event, people who were nearby may not be captured in their SenseCam images, but may be a useful retrieval cue.
• Geographical location: we use GPS function on Nokia N95 mobile phones to record the individual's location [3].
• Biometric devices: wearable biometric devices such as heart monitors and BodyMedia SenseWear armband are used record the participants' physical conditions, which can to some extent, indicate the wearer's motion, emotional status and arousal level [10].
• Similar to many other life logging projects, one major concern of our data collection is the privacy [4], not only of the participants whose life is 'recorded ', but also of the third party individuals who may be somehow involved in the recording.For example, people shown up in the photos taken by Sensecam, emails or messages sent to the participant involving the individual's private information, etc. Consent forms were signed by the participants, and they are allowed to delete the data they are unwilling to show the developers.

HDMs Model: Structuring Raw Data
We aim to develop a memory model for data in the HDMs which can estimate the strength of memory features.In this model, we define following objects: • Item: an attribute or related context information of the file, e.g. the title, the location.
• Link: There are difference types of links, as they are created according the rules of clustering memory.At this stage, we assume all the links to be bi-directional.

Weighting of Links
The 2nd BCS-IRSG Symposium on Future Directions in Information Access 1 http://research.microsoft.com/erp/memex/presentations/MSRDigital Memories 2006 Jim Gemmell Software.ppt