Exploring Movies through Interactive Visualizations

Videos and movies are important sources of information in the entertainment and learning form, having great power to affect us, perceptually, cognitively and emotionally. By integrating various media like image, sound and text along time, they are very rich and they are becoming pervasive on the Internet and interactive TV, increasing the need for new and powerful ways to access, browse and view them. Interactive visualization techniques have the potential to help handling with this rich but complex information: both the time when they were released, and the time along which their contents are weaved, in each movie. In previous work we have presented visualizations that allow the access to the movies released over a certain period of time based on genres and rankings, and to overview and browse movies based on their contents represented with tag clouds. In this paper, we explore the movies through visual representations of the different aspects of their contents, especially image, audio, and subtitles, with a focus on emotions. This approach has the ultimate goal of providing overviews, and browsing mechanisms based on interactive visualizations that may provide insights in analytical or more ludic uses, that may be efficient in conveying and accessing information, and are also easy, funny and aesthetical.


INTRODUCTION
state that video indexing should be similar to text, in the structural frame that informs it, making possibe to access and extract video data through segments such as shots or scenes.But it is not so easy to access this kind of information, due to the richness of its content.Sergei Eisenstein (1898Eisenstein ( -1948) ) stated once that movies were the only places where time passes differently, beyond reality time.Movies are complex because they imply several symbol systems such as image and audio, gathered as meaningful information that changes in time.Moreover, movies are conceptual objects, and therefore they carry metaphors and meanings that make clear the border that separates this art form from other visual narratives.
Visualization is a field where images are used for a better understanding of information, and when dealing with data that changes in time, this component is central.As a brief review, visualization techniques have been used for large datasets analysis, due to their power to deal with huge amounts of information, and to reveal patterns between the data.But users commonly like to be aware of the evolution of data over time, and in discovering trends and patterns (Aigner et al., 2007;Post et al., 2003;Silva & Catarci, 2000), even when precise and formal information is not needed.Temporal relations enable people to get insights about the data, and visualization can have a great role in enriching the process of understanding data, making it more ludic while keeping its complexity.That is the scope of our present work.
Our aim is to make possible to extract information that may be presented, and indexed by visualizing overviews and the structure of the movies contents in the different tracks of image, audio and subtitles.Static or dynamic interactive visualizations would allow to get summarized or more detailed information as a means to browse, and watch the movies along time in ways that may provide insights in analytical or more ludic and artistic uses, that may be efficient in conveying and accessing information, and also easy, funny and aesthetical.
Section 2 presents most relevant related work, followed by the presentation of our approach in section 3. The paper ends with conclusions and perspectives for future work in section 4.

RELATED WORK
The analysis of video contents can help handling with their cataloguing, and indexation.When gathered and meaningfully stored, this information is extremely useful in dealing with the huge amount of video information needing to be accessed.For related work in the area of content processing and retrieval, that falls outside the scope of the current paper, see (Gil et al., 2012).The next sections are focused on aspects related with video access and visualization.

Video Access
There are websites that provide information about movies, e.g.actors, directors, rating by users, title, keywords (e.g.IMDB (imdb.com)and Netflix (.com)), and websites that allow also to access and watch, comment, get recommendations, and publish videos (e.g.YouTube (.com) and vimeo (.com)).They do not support, though, the navigation through information conveyed in the different tracks, like subtitles, audio, and emotional info (either expressed or felt by the users), and they usually represent the video sets (e.g.resulting from a search) through lists.Film Finder (Ahlberg & Shneiderman, 1994) provides an interactive visualization that permits search and view by movie duration, title, genre, directors and actors, using colored spots, and movies space zooming for more or less detailed info.Videosphere (Bestiario) is a 3D sphere that represents and gives access to Ted's videos related by their semantics, displaying also a more interesting search.
Media Streams (Davis, 2000) allows to work on the structures of the videos, annotate, visualize, and retrieve information.Informedia Project (Hauptman, 2005) addresses a digital video library based on automatic video-content analysis that uses metadata to index, search, navigate, and retrieve videos' contents.It allows the search by speech and image, and provides the visual summarization of the scenes (mainly thumbnails representing scenes) for the search and image retrieval.Open Video (Geisler et al., 2001) allows the retrieval, storage and search of video and their contents.These applications do not provide, though, information about colours, movement, or emotions.

Video Visualization and Retrieval
Video Space (Rocha & Chambel, 2008) allows to interactively browse and visualize a 3D coloured representation of video spaces and video pixels, to explore their cultural and aesthetic properties.In the video space users access videos organized and related by themes: music and dance, mainly, authors, and countries, and inside the videos, they experience the feeling of navigating among their colored moving pixels in 3D.It draws on an artistic perspective of video access.ColorsInMotion (Martinho & Chambel, 2009) allows to interactively explore and visualize video contents based on colour, movement, and rhythm properties, using summarization of video and their scenes.Slit-scan imaging techniques produce static images to represent time-based phenomena (Levin et al.).From thin slices of film, that are extracted from a sequence of video frames, results a new image that summarizes the sequence.What Did I Miss? (Nunes et al., 2007) is a timeline visualization system that uses the slit-scan technique in order to explore a video history with detail.It is based on always-on media spaces broadcast video (VMS) (Bly et al., 1993;McEwan & Greenberg, 2005), and aims to give presence awareness among collaborators, encouraging the connection of people.Last Clock (Fig. 1) (Angesleva & Cooper, 2005) shows an overview of some space's dynamics and history, providing a record of what happens through time.The information captured includes the type of activity, the pace and volume of motion, and the moment in time when it had happened.Cinemetrics (Fig. 2) (Brodbeck) explores the movement in the movies, and it is displayed in a circular dynamic shape.Each scene is represented by a coloured shape that moves at different speeds, according to the detected movement.Artifacts of the Presence Era (Fig. 3) is a visualization concep-tually based on the geological layers in sedimentary rocks, and it visualizes the events in the physical space of a museum over time.The layers of the recorded video images are changed in its height by the sound of the environment.These works have similitudes with our presented visualizations although they consider only a few variables, or content categories, in what respects to the information provided.

VISUALIZING AND BROWSING MOVIES
We are extending MovieClouds (Gil et al., 2012), an interactive web application that adopts the tag cloud paradigm (for the power, flexibility, engagement and fun usually associated with clouds) and allows to access, explore and watch movies through the information conveyed in their contents, mainly audio, and subtitles, where most semantics is conveyed.Our previous contribution in the MovieClouds application (Jorge et al., 2012) was the means to provide support for visualizing and navigating the movies' space, in order to search for movies, get overviews and analyze information on genres and ratings throughout time of movie release, possibly finding patterns.We have included the time dimension at the movies' space level, in 2D and 3D representations, allowing the access to the movies to watch.In this work in progress, we are aiming for complementary goals, which are to present visualizations that provide new ways of overviewing and browsing movies through different aspects of their contents along the time inside the movies, especially image, audio, and subtitles, with a focus on emotions, through interactive static and dynamic visualizations.

Design Rationale
The proposed interactive visualizations intend to provide overviews that represent the whole content of the movies, allowing zooming for increased details, and the means to index and browse the contents in order to access the movie at the time of the selected parameter.The main idea is to allow the use of the application in a visual and ludic way.
Concentric wheel shapes were chosen on the clock metaphor, following the way people conceive the passing of time, along with rounded devices which are traditionally used for storing information about films and videos (reel, dvd, disk).In practical terms wheels have the affordance of rotating, and steering to influence the progress onwards and backwards, and in addition, they allow presenting the whole movie in more detail, in a thiner space, making better use of the screen, when compared with horizontal straight timelines.But we also adopt a linear stripe shape, inspired on the traditional filmstrip metaphor, which moves in-sync with the concentric wheels, and allows the viewing of the parameters of interest in the upside position, in more detail.
We considered Ben Shneiderman's (1996) Mantra "Overview first, zoom and filter, then details-ondemand" regarding two demanding levels of use, e.g. to be aware of the mood of the movie the user intends to watch by an overview of the audio and color representation perspectives, growing the complexity of the task according to the user's interest.The potential user is interested in movies and videos in what relates to their formal, and semantic properties of their contents.They might be researchers, professionals that aim for trends about movies throughout time, or just curious since this application implies an easy use, although its functionalities can only be of great usefulness for specialized tasks due to the kind, complexity and number of available information.
Next, we present three main visualizations that explore these concepts in three perspectives: 1) to visualize and explore content tracks or perspectives in a movie, related with image, sound, and subtitles, with a focus on emotions (sec.3.2); 2) to compare movies by selected content tracks (sec.3.3); and 3) to combine contents (e.g.movement and colors) in integrated views when comparing movies.

Visualizing Contents in a Movie
This visualization (Fig. 4) allows the overview, browsing and watching of the movie's content.
When browsing through the visualization, the users have access to the content tracks and they can also watch the movie.

Interactive Overview and Browsing
The movie wheel presents in the middle an overview as a tag cloud of the movie content, in a selected perspective or track content (most frequent word in subtitles, as default and in Fig. 5).Around, timelines represent the different content tracks (scene thumbnails as a start in Fig. 5).From this initial overview, in Fig. 4, it is possible to get the main concepts and visual properties of the movie.Content tracks as timelines: Since there are many content tracks available, and in order to make easier to relate and compare information, there are hidden, and floating information that the users can activate or move depending on their goals (Fig. 4 and 8).There is the wheel menu, which shows the whole movie in the screen, and the timeline menu, where the user can see the most representative frames of the scenes with more detail.The user might click on each track of interest, making them visible.It is possible to activate and cross information from the following tracks (from inside out): 1) subtitles; 2) audio events; 3) mood in audio; 4) felt emotions, 5) emotions in subtitles; 6) dominant colours; 7) movement; and 8) scene thumbnails.Selected tags in the chosen tag clouds get highlighted in colors, and make for colored dots to appear in the corresponding content track timelines to ease finding, comparing and browsing movie content.For the emotions, the Plutchik's (1980) colored model was chosen, and for the other tracks random colors are assigned in every click, for flexibility, and because there is no predefined mapping for such variety in semantics.
Beneath the timelines, tag clouds may overview each scene in a chosen perspective (e.g.subtitles in Figs. 4 and 7a), allowing to compare the evolution, and complementing the full movie tag cloud in the middle of the wheel.Also in the middle, the user may choose to have an animated tag cloud that changes as the scenes go by, as an alternative to the default static cloud that represents the whole movie.Dragging the wheel or the timeline: the user can make the movie go forward or backward towards the scene of interest.They both move in synchrony, allowing the user to observe the requested scenes with more detail in the white frame on the timeline (in the bottom), where the wheel and timeline scenes match (Fig. 6).This synchrony of wheel and timeline can be automatic (user choice) matching in the current moment, as the video plays.

Movie watching:
The user can choose the scene of interest, and by clicking on it, go and watch the movie at the time of that scene.It is possible to choose the place for the displaying of the movie, depending on the information of interest.E.g. if the user is interested in the subtitles of each scene (Fig. 7a) or in the visible tracks below the timeline (Fig. 7b) the floating display can be placed on the right (Fig. 4).To watch the video synchronized with the playing scenes matching with circular and timeline tracks, the movie can be played below the timeline, aligned with the current scene (Fig. 8).

Exemplified Interaction
In Fig. 4: 1) The users are aware of the main conceptual idea of the movie, by the most spoken words in the subtitles' tagclouds in the centre, and the most representative scenes of the movie around; 2) they click on the circular menu in order to see all the tracks available; 3) they select: subtitles, audio, mood, felt emotions, emotions, colours, and movement.All these tracks become visible, inside the initial wheel (Fig. 5); 4) they click on the linear menu, to select colours and movement (Fig 7b).In this case, the displaying of the movie is made on the right, in order not to hide the required information on the timeline.The users might also be interested in observing only one or two tracks, both in circular or linear way, and in this case, they will select only those properties on the menus.
The users might be interested in being aware of the moments in which the movie is more dynamic (more movement), and then cross that information with the other tracks available, e.g. to check if the audio events are also more frequent and agitated (e.g. a shooting), and compare to the emotions expressed in the movie (in the subtitles) and felt by the users (e.g.fear) during those moments.

Comparing Movies by Content Tracks
This visualization allows to relate and compare movies in chosen perspectives or tracks of their content (e.g.scene images in Fig. 9, allowing to get and idea of the main colors and visual rhythm by the number of scenes and image difference).

Interactive Overview and Browsing
The wheels present the chosen tracks in the selected movies .It is possible to select the movie thumbnails (Fig. 9) in order to watch the movie at that scene, and the user might be taken to the previous visualization in order to obtain more diverse information about each movie.

Exemplified Interaction
In Fig. 9: 1) Choose and select the movies of interest to analyze; 2) after selecting a track of interest, e.g. the most colored scene, each movie's wheel rotates, along with the respective timeline, until the requested scenes are vertically aligned.The requested information is marked in red, on the respective scene (Fig. 9-10); 3) the user can ask for the second highest frequency.The visualization will move accordingly, and a white mark will appear in the requested scenes; 4) It is possible to relate and compare the most dynamic and colored scenes of the movies, comparing their colors, and be aware of the moment those properties are in the movies.

Combined Content to Compare Movies
This visualization is also based on summarization, and thus, it is possible to be aware of the information throughout the movie's contents within a glance (Fig. 11), combining information in more than one track.It is based on linear timelines that may take different shapes to convey other types of information.In the example, the scene images of the movies are presented in a way that stress main colors, combined with dynamic information, such as movement in images or rhythm in audio (as chosen), along the movies.This visualization allows a comparison of different movies in a chosen combined perspective along the timeline.

Interactive Overview and Browsing
This visualization can be composed of one or more summarized movies, providing colors and rhythm along the chosen movies.It is possible to select tracks to compare, and relate.Mouse over on the summarized image allows the user to have a view of the respective frames.It is also possible to locate the selected frames in time, with time value displayed below the timeline (top of Fig. 1).

Exemplified Interaction
In the example (Fig. 11): 1) The user can choose a movie; 2) choose the tracks of interest, e.g.movement and dominant colors; 3) more movies can be selected in order to have comparative overviews among the movies and throughout time.

CONCLUSIONS AND FUTURE WORK
We presented our most recent work towards the interactive visualization of movies based on their contents.From overviews of the whole content to timelines where content may be viewed along time, allowing to index and watch the movies at any chosen moments.Movies may be overviewed and compared along the different perspectives of their content, for example to check what is happening in the audio when characters are talking about fear (in the subtitles), or if the dominant colors of the movie have a match with the theme of the conversations or the mood in the audio.Different movies may also be compared in chosen or combined perspectives of the content.Dynamic and synchronized interactions allow for more accurate perception and access to the movie contents.Most visualizations adopt circular clock and wheel metaphors, and the filmstrip timeline metaphor.They use tag clouds for overviews of semantic content (in the whole movie or individual scenes), mainly in subtitles, audio events (e.g.gun shot), audio mood (e.g.tense or playful) and felt emotions (e.g.happy or sad), and images and colors for visual properties.Preliminary user feedback was very encouraging and allowed to identify most strengths and challenging aspects that are being refined, along with the full development of the designed features.
Next steps include refining and extending current visualizations, based on our initial goals, our reflections, and a more thorough user evaluation that is being prepared, towards effective, rich, and expressive visualizations that can help to provide insights about movies, their contents and their impact on us.We intend to explore these richer content properties, as we get more movies processed in these perspectives, again in the timeline of movies released along the years, or even decades (e.g. is there a decade with more shooting in the movies, or more romantic dialogs?).The representation of series is also in our horizon.Their seasons and episodes could be represented in a way that captures their evolution.The system could also trace and present personal movies viewing histories, allowing users to reflect on the movies they tend to watch in different periods of their lives, and even compare the emotional impact the movies had on them along time, and relate that to their content.

ACKNOWLEDGMENTS
This work is partially supported by FCT through LASIGE Multiannual Funding and the ImTV research project (UTA-Est/MAI/0010/2009).

Figure 4 :
Figure 4: Visualizing Contents in a Movie.

Figure 6 :
Figure 6: Visualizing Contents in a Movie.The user can control the scenes to watch by dragging both the wheel and the timeline.And they synchronize in video play.

Figure
Figure 7a-b: Visualizing Contents in a Movie.The tracks might be visible or hidden, depending on the interest of the user: a) subtitles scene tag clouds; b) movement and dominant colors

Figure 8 :
Figure 8: Visualizing Contents in a Movie.The movie can be placed below the timeline if there is no information needed such as subtitles or horizontal tracks.

Figure 9 :
Figure 9: Comparing Movies by Content Tracks

Figure 10 :
Figure 10: Comparing Movies by Content Tracks.The navigation concept is similar to the first visualization.The user selects the tracks to match

Figure 11 :
Figure 11: Combined Content to Compare Movies.Four movies are shown in the visualization.On over, current scene and time on timeline are shown.