Storytelling and VR: Inducing emotions through AI characters

Recent forms of virtual reality (VR) have changed the way people interact with moving images in the entertainment and the games industry, as well as the way the content is created. Technological advances in VR have given an opportunity to create simulated environments that users can immerse themselves in, and sense almost as a real experience by combining film techniques and interactive media approaches. Storytelling in VR presents various challenges due to the spatial properties of the medium. Research suggests that engaging Non-Player Characters (NPCs) enhance storytelling and can do so by communicating emotions. Most VR war experiences use the concept of morale and emotions applied to a group of soldiers or individual characters. To address the need for more believable AI characters in VR, this project will investigate how emotions can be communicated more effectively in a VR war application. VR companies are increasingly using Artificial Intelligence (AI) and cloud technologies to develop a stronger ecosystem for NPCs. However, there are still significant number of limitations in terms of technology and immersive storytelling for VR with characters and props paying significant role for creating convincing VR experiences. This project will therefore aim to enhance storytelling in VR by inducing emotions through AI characters in a war environment inspired by realistic events from WWII.


INTRODUCTION
The term of virtual reality (VR) was firstly used in the 1980s by Jaron Lanier and can be defined in a variety of ways from painterly illusions to a threedimensional experience with usage of the headmounted display (HMDs) (Bucher 2018, p.3). VR's development has experienced peaks and troughs. The medium is still in its infancy, but after Facebook (now Meta) bought Oculus in 2019, VR has stepped in into the mainstream and brought the technology into people's homes (Bucher 2018, p.1). VR has become widely used in cinematography as well as the games industry.
The cinematic approach with a linear narrative concept does not utilise the full potential of VR, and instead the approach often used in games lets users discover the story. This has started discussions around storytelling for this media and how to create an immersive experience for the audience (Bucher 2018, p.6). Interactivity makes the biggest difference between VR and other storytelling mediums, due to spatial aspects (it is a three-dimensional, stereoscopic experience) and the ability for users to make decisions. Audiences can make decisions of where to look or how to interact with the environment and characters, which may have an impact on the narrative. Another major advantage of VR technology is that it can make the audience feel emotions (Ward 2017).
In order for storytelling in VR to be convincing, the non-player characters (NPCs) must also be believable. One of the ways this can be achieved is through a more accurate representation of human emotions. For war games, it may be possible to utilise concepts of morale and emotional reactions applied to an individual character or a group of soldiers (Mozgovoy & Umarov 2012, p.21-22). This project will investigate the implementation of an AI character which can exhibit emotions in an interactive VR war experience. The war environment will provide a basis to explore the emotions of NPCs in combat situations, and investigate how these can enhance immersive storytelling in VR. This paper will discuss the background and context for this project through a discussion of VR storytelling, emotion in games, and AI in VR. Following this, this paper will outline the proposed project.

VR AND STORYTELLING
Storytelling for VR has led to two dominant philosophical trends and perspectives in VR. The first of these focuses on viewers as the passive observants of the action and surroundings. The second approach allows the viewer to become the camera and interact with the environment. Stories created in the latter way begin to blur the line with games (Bucher 2018). Storytelling in VR should be less about telling the viewer a story and more about letting them discover it. Viewers dropped into VR worlds should have freedom to interact on their own. The capability of provoking emotion together with interactivity makes VR a powerful tool to create an immersive experience (Ward 2017).
Immersion can be understood as a process of engagement for users that relates to various aspects of involvement including emotions. This process is described in Calleja's player involvement model, which identifies six dimensions of involvement in digital games (Calleja, 2001). Emotional immersion is a type of involvement and dimension of immersion when the player feels emotionally aroused by the narrative elements of the game. Enhancing emotions may provide a way to improve immersion for VR storytelling.

EMOTIONS IN GAMES
The key theory of emotion in psychology and neuroscience research states that humans have a limited set of basic emotions such as fear, sadness, anger, happiness, and disgust (Ekman 1999). According to the theory of basic emotions, each specific emotion maps to one neural system. However, the circumplex model of affect proposed by Russell (1980) can be integrated and summarised as a two-dimensional space defined by valence (pleasant/unpleasant) and arousal (activated/disactivated) axes. Game designers usually explore and test a wide range of mechanics that yield emotional states they desire to trigger into the player. Emotional states such as engagement, fear, stress, frustration, and anticipation can dependent on the genre, the narrative and the objectives of the game (Yannakakis & Paiva 2014). Moreover, emotions can provide information to guide decision making. This idea is discussed by Damasio (1996), who stated that emotions are information providers and somatic markers represent by the gut feelings through a bodily feedback, which guides decision making.

AI AND EMOTIONS
Non-Player Characters (NPCs) may have a key role in enhancing emotional immersion. The most realistic NPCs in games are based on human emotions (Mozgovoy & Umarov 2012, p.2). This concept may be used in fighting and war games, to provide realistic human-like behaviours and indicate concepts or morale. If a war simulation can implement behaviours indicated by emotions in NPCs, and generate social interactions between them, and with the player, they may appear more believable. At the same time, this will improve immersion of the player into the combat simulation (Mozgovoy & Umarov 2012, p.21-22). Emotional states of NPCs should respond to current events and reflect changes in action in real time (Mozgovoy & Umarov 2012, p.23).
Research into the use of emotions in agent-based systems is mainly concerned with how to convey the emotions of NPCs to the player (Campisi & Caudell 2010). However, further improvements can be made by developing more sophisticated approaches through which NPCs can incorporate emotions in terms of their functioning through aspects such as decision making. This can be accomplished by drawing on approaches from the field of affective computing (Picard 1997), which provides ways in which computer systems can recognise, exhibit, or respond to human emotions.

MODELING AI BEHAVIOUR IN VR
Modelling realistic human behaviour, including decision-making and creativity is a complex challenge for games developers, and this also applies in VR. Behaviour trees (BTs) are a relatively new technique for developing behaviour models for AI characters. For a long time, finite state machines (FSMs) have been the dominant technique in game AI, but a major strength of BTs compared to FSMs is their modularity and ability to yield complex behaviours composed of simple tasks. The main difference between BTs and FSMs is that they are composed of behaviours rather than states. BTs employs a tree structure with a root node and a number of parent and corresponding child nodes representing behaviours (Yannakakis & Tolegius 2016).
The strongest trend in current VR research is adding believability to virtual agents (physical and behavioural). This is sought in adventure games, as well as serious VR games. Also, AI characters are frequently utilised as intelligent tutors ( Another example that uses AI to build believable characters is IMMERSE (Playabl Studios 2012). It is a gestural based, virtual training simulation for soldiers using AI techniques to teach social skills in non-English speaking environments, developed in Unity. AI presents the basic human dynamics skills that soldiers need to understand to enter into any social encounter regardless of the culture, group, or situation.

PROPOSED SOLUTION
The project will result in the development of a final prototype which implements the AI character in a vertical slice of virtual war scenario with usage of Unity game engine for Oculus Quest 2 platform.
An AI character will be developed as part of this project, based on utility-based techniques, so the agent acts based not only on what the goal is, but the best way to reach that goal (Figure 1). Each action will be evaluated separately, and the highest scoring action will be chosen. The Utility AI can be used as a Behaviour Tree and implement its branch structure (Rasmussen 2016). For example, the AI character will start interacting with the player while he passes a certain distance. To implement a complex behaviour system based on utility-based techniques for AI characters will be used Panda BT Free. This is a script-based Behaviour Tree engine that allows creating complex, scalable and reusable logics for the game and it is compatible with Unity game engine ( Figure  2). It eases up fast prototyping and iterative development (Begue 2019).

IMPLEMENTATION
The project is inspired by real events from The Dunkirk Evacuation (1940) during WWII called Operation Dynamo. The Royal Navy implemented evacuation of Allies troops cut off and surrounded by Nazi German troops from the beaches and harbour of Dunkirk.
Furthermore, the soldiers behaviours are inspired by memories of one of the soldiers, Harry Leigh-Dugmore, when German planes were bombing and machine gunning the beaches, he and his fellow soldier friends spread out as far as possible but then returned as nearly as they could to their position in the queue. He also brought down a Messerschmitt what been cheered as it crashed into the sea (Royal British Legion 2022). Inside the VR experience soldiers are also queuing on the beach and in the water. What is more, while bombarded by enemies they are attacking back or running away. Figure 5 is showing the model of AI BTs used in the final artefact. Each behaviour is composed of animations that illustrate current actions. Behaviours change after being successfully executed or regardless of action conditions.
Bombing is the initial event that leads to the next AI actions and emotional states (Figures 6 & 7). However, anger may increase while attacking, and fear while escaping. Successfully completed action may result in happiness. Utility AI techniques allow us to create an individual behavioural system based on the highest scoring action, which is also driven by current emotions.
Furthermore, the player functions (teleportation and interactions) are implemented with usage of XR Interaction Toolkit. The player has the freedom to choose the storyline of the narrative, and the way of interactions with the environment and characters. Due to this fact, the VR experience has a few possible storylines:  Exploring without any interactions.  Queuing with other soldiers.  Taking part in actions together with befriended soldiers (escaping from bombing, shooting at attacking planes). Figures 8 and 9 show screenshots from the VR prototype, illustrating key moments based on Harry Leigh-Dugmore's account. The AI characters respond to the situation, exhibiting emotional traits as they cheer a downed enemy aircraft (Figure 8), and queuing to board a boat (Figure 9).