Self-Tracking Emotional States through Social Media Mobile Photography

This paper presents a preliminary breakdown of the results obtained in an exploratory study conducted through the mobile application Instagram. Our goal was to inspect the potential benefits of combining the self-reporting of emotions with everyday mobile photographic practices to learn more about users’ experiences. To do so, we instructed 25 participants to assess and report their emotional states using Instagram, during a 4-week period, according to a pre-established set of instructions. Participants also filled in pre and post-study questionnaires. We then analysed the 291 submissions obtained and the results from the questionnaires, focusing on three aspects: the categories of the photographs taken, the emotion labels used, and the feedback provided by the participants. We end by presenting and discussing a set of insights that might be useful in the design of mobile apps to improve emotional self-awareness and wellbeing.


INTRODUCTION
The self-tracking of emotional states (also known as Mood Tracking) consists in habitually monitoring our emotional states, over extended periods of time, to better understand how, and why, they vary.With that information in hand, we can better learn how to improve our emotional wellbeing.The selftracking of emotions is a typical exercise within the clinical practice in mental health, where therapists often ask their patients to keep mood diaries.This exercise can be beneficial, not only for those experiencing depression, bipolar disorder, and other dysfunctions; but also for those who wish to know more about themselves.In fact, mood tracking is becoming increasingly prevalent among the general public thanks to the myriads of applications that exist for that purpose.The process of self-tracking has two central aspects: collection (of relevant personal information) and reflection (to produce insights) [13].The work presented here focuses on the collection (selfreporting) of one's emotional states.

LITERATURE REVIEW
Self-reporting one's affective states is the most traditional method used to collect a person's emotional data within research [12].Still, a systematic literature review [7] established that the self-report of emotional states was under-researched and that additional studies involving users in real contexts and interfaces with new interaction styles are needed.
Unfortunately, self-tracking technologies often present issues of low adherence and high dropout rates due to barriers like lack of motivation, lack of time, or no immediate access to the tracking tool [9,13,23].Some social media mobile apps, however, seem to be able to overcome these barriers.Nowadays, people usually carry their mobile phones with them all the time and recurrently use apps like Instagram, which counts with more than 700 millions users as of April 2017 1 .For many of those users, sharing photos of their lives, along with their thoughts and feelings, has become a vital part of their daily routines, making social media a naturalistic setting, laden with emotional content.Another social media app, Facebook, lets its users add their current emotional state when they make a post (e.g., I am feeling angry) and to express their feelings about others' posts through "reactions" like "Love" or "Sad".Plus, on Twitter and Instagram individuals often use hashtags to express their emotions (e.g., #bored).There are many studies about emotions and social media, and while most focus on sentiment analysis -"the automatic extraction of sentiment-related information from text" [24], there is also research focused on directly exploring the potential of social media to help better understand mental health and depression [1,2,5,14,16], wellbeing [4] and other relevant public health matters [8].
We decided to use the application Instagram in our research because it offers a type of data that is particularly interesting in the study of behaviours related to emotions: photographs.The richness of photographs allows individuals to communicate messages in a way that would not be possible only with words [1].

Participants
To obtain the corpus of data that supports this paper, we ran a study with 25 participants (17 female), with ages between 25 and 63 years old (mean=34.4),over the course of 4 weeks.We gathered 291 submissions, with an average of 11.2 submissions per user.We recruited the participants via announcements placed on social networks, and we ran a prize draw for a 50€ voucher at the end of the study, as an incentive.All participants were informed about the procedures and agreed to a consent statement.We also asked for permission to use their photographs in future scientific publications.Additionally, we asked participants if they preferred to share their photos openly on Instagram or submit them privately, through the app's direct messaging system -3 participants chose the latter.Most participants (70.8%) used Instagram regularly before the study.

Procedure
We began the study with a questionnaire, meant to obtain demographic data and gauge the participants' habits regarding their wellbeing.A 4week data-collection period ensued, followed by another questionnaire, designed to get feedback on the participants' overall experience throughout the study.During the data-collection period, we asked the participants to share at least two photographs per week on their Instagram account, according to specific instructions, which were detailed in a document and summarized on a cheat sheet.Figure 1 exemplifies the application of these instructions (succinctly described in the list below).The elements of the instructions that directly concern the self-reporting of emotions are the emotion labels and two well-established dimensions of emotional responses: arousal and valence [20,21], which we decided to rename as energy and pleasantness, respectively, as we considered these to be simpler for the participants to understand and remember.Valence (pleasantness) describes the degree to which an emotion is either positive or negative, and arousal (energy) is the intensity of the felt emotion.We believe that understanding emotional states through these two dimensions can be very enlightening.Assessing the degree of arousal points to the vital relationship between emotions and body activation (more specifically, autonomic nervous system activity).Understating how emotions manifest themselves through physical cues in our bodies is vital to increase emotional self-awareness [6].Labelling emotional states and improving our emotional vocabulary is also essential to better recognize and communicate what we are feeling.The lists of emotions and colours initially used in the instructions stem from Plutchik's Psychoevolutionary Theory [17].We chose this particular theory because the list of (primary) emotions was short (8) and had an established relationship between colours and emotions.However, during the second week, some participants started to complain about the complexity of the tagging process.Thus, we decided to revise the instructions, at the beginning of the third week, with 72% of the participants adhering to the new rules.The adjustment consisted of letting the users freely add whatever emotions and colours they wanted, using hashtags (e.g., #happiness, #lilac).We were aware that this modification would complicate the data analysis, but since the study had an exploratory nature we decided to proceed with the adjustment.

DATA ANALYSIS
Since we received some of the entries via direct message, and some participants did not have a public profile on Instagram, we could not pull the data from Instagram's API.Instead, we collected it manually, by iteratively going through each participant's profile and downloading the photographs and transcribing the corresponding text.We then categorized the 291 photographs obtained and assessed the inter-rater reliability (IRR).We also examined all the emotion labels used in the entries and the feedback given by the participants in the final questionnaire.

Photographs
One of the authors (coder 1) used a thematic analysis procedure [3] to systematically find the underlying themes within the data, which resulted in a set of seven categories.Then, the remaining authors (coders 2 and 3) independently coded the pool of photos into the seven pre-established categories.The two authors did not receive any instructions other than the categories names.We computed the Cohen's Kappa for each pair of coders.The first pair (coder 1 and coder 2) obtained a very good strength of agreement (κ = 0.819).The second pair (coder 1 and coder 3) obtained a good strength of agreement (κ = 0.790).These results demonstrate consistency among the observational ratings given by the coders.The seven visual themes are Nature, Surroundings, Objects, People, Food, Animals and Pictorial Images.Figures 2 to 8 show some photos from the study from each category.There are some parallelisms between these categories and the ones reported in a recent study where 1000 photographs obtained through Instagram's API were methodically analysed [10].In that study, there was also a category labelled Food, two categories related to People (Friends and Selfie), one category similar to Animals (Pets), and two categories that could be potentially included in our Objects category (Gadgets and Fashion).

Emotion Labels
The chart in Figure 9 shows the number of posts that were tagged with each emotion label.The chart includes all the labels that appeared in at least four submissions.The label "joy" was by far the most employed label, appearing in a total of 159 posts (56,58%).Plus, if we group the label "joy" with the similar label "happy", we have a total of 175 posts (62,27%).The second most-used label was "anticipation", present in 64 submissions (22,77%), followed by "trust" and "surprise", with 26 posts each (9,25%).The label "sadness" was present in only 19 posts (6,76%).This data is consistent with what is described in the literature: individuals are more disposed to share positive events than negative ones, due to stigma and selfpresentation concerns [1].Furthermore, sharing only favourable and socially desirable images is associated with positive outcomes like higher self-esteem, since it helps people affirm positive views of themselves [1,25].

Number of posts
One participant said "I admit I was auto-censored to upload pictures in "bad days" because I'm not a person who wants to expose my problems to the world.",moreover, another stated, "I realized that in fact the tendency is to put into Instagram things that make me feel positive emotions.".

Participants' Feedback
According to the feedback provided in the final questionnaire, participants enjoyed the overall experience of assessing and reporting their emotional states while taking photographs with their mobile phones.From their comments (Table 1), it transpires that they found the study beneficial, especially regarding the self-assessment of emotional states.They reported that it made them reflect more and be further aware of their emotions."This is a good exercise to make us be more aware of our emotions, and I think that's half way for a healthy mind." "It was useful as a self-assessment and reflection about (what) I was doing." To the question about whether or not the emotion assessment made using the pleasantness (valence) and energy (arousal) dimensions was easy to perform, 88% of the participants answered positively.The remaining 12% said that the two dimensions were difficult to quantify.It was interesting to notice that some participants seemed to believe that the two dimensions always had to go in tandem (e.g., one could only feel high energy when pleasantness was high as well, and vice versa).One of the participants stated "I guess that being aware of my energy and pleasantness levels made me do something about that, especially when they were low.", while another claimed "I was surprised how sometimes, we can be overly tired but completely satisfied with the situation.",referring to a high pleasantness and low energy situation.This mental model is incorrect since the two dimensions can have opposite values.For instance, distress is characterized by high energy and low pleasantness, whereas contentment is its bipolar opposite [20].One participant commented on how "it became clear that energy and pleasantness aren't always aligned."

DISCUSSION
After reviewing the data collected during the study, we are inclined to believe that combining social mobile photography with the self-reporting of emotions is a viable way of incorporating this practice into people's everyday lives, as a means to fight issues of attrition commonly present in this type of technologies.We learned that people are generally not disposed to share their negative emotions with others, and thus it is important to offer users the possibility to keep entries private in a future mobile app so that they can feel at ease to explore their negative emotions as well.We also believe that it is imperative to better explain and illustrate the concepts of arousal and valence before asking users to deal with these dimensions.
Explaining the concepts better in the instructions and reporting interface, and experimenting with other labels, such as "body activation" instead of energy or arousal, might help.Lastly, we trust it is valuable to be aware of the categories of photographs that people usually take and share when designing and developing apps, and even perhaps use photos from the categories during the design process (e.g., mockups, personas, user stories).

CONCLUSIONS AND FUTURE WORK
This paper presented the first impressions of an exploratory, 4-week long study, where 25 participants submitted, through the mobile application Instagram, a total of 291 photographs and accompanying textual data regarding the selfreporting of emotional states.The primary goal of this experiment was to learn more about the process of self-assessing and reporting emotional states while taking photographs with a mobile app, and we intend to further explore this matter with further analysis of the data, interviews and participatory design sessions.We also intend to perform thorough individual analyses of each participant's data to understand the users better and produce reports to investigate how we can improve the design for the reflection aspect of the self-reporting of emotional states.Furthermore, we expect to use some of the gathered data, like photographs and captions, in the sketching process of a future mobile app for emotional wellbeing.Finally, thanks to the positive comments and reactions of the participants, we feel encouraged to keep exploring the frontiers between emotions, mobile phones, and photographs, to inform the design of future mobile technologies to promote emotional awareness and wellbeing.

Figure 1 :
Figure 1: A submission made by on the participants.

Figure 2 :
Figure 2: Set of photographs from the Nature category.

Figure 3 :
Figure 3: Set of photographs from the Surroundings category.

Figure 4 :
Figure 4: Set of photographs from the Objects category.

Figure 5 :
Figure 5: Set of photographs from the People category.

Figure 6 :
Figure 6: Set of photographs from the Food category.

Figure 7 :
Figure 7: Set of photographs from the Animals category.

Figure 8 :
Figure 8: Set of photographs from the Pictorial Images category.

Figure 9 :
Figure 9: Chart showing the number of posts tagged with each emotion label.