Structures of Emotion: An online demonstration of an emotion recognition process

Structures of Emotion is an art performance that aims to examine the way humans and machines read and interpret emotional expressions. The work realises the difficulty of translating 'feelings' into words. We analyse the proprioceptive and cognitive complexity of emotion recognition, comparing human vs. computer vision’s biases while perceiving another person. Through this relationship emotions are reduced to a sensory input given by the subject's facial expressions. The relationship is mediated by a digital screen, which obstructs other biometric inputs that would otherwise be delivered in person. This ocularcentric, remote mode of communication has become a primary channel for social interactions.


INTRODUCTION
Structures of Emotion is an art performance that aims to examine the way humans and machines read and interpret emotional expressions. The work realises the difficulty of translating 'feelings' into words. We analyse the proprioceptive and cognitive complexity of emotion recognition, comparing human vs. computer vision's biases while perceiving another person. Through this relationship emotions are reduced to a sensory input given by the subject's facial expressions. The relationship is mediated by a digital screen, which obstructs other biometric inputs that would otherwise be delivered in person. This ocularcentric, remote mode of communication has become a primary channel for social interactions.
When we see someone smiling, does it necessarily mean that this person is 'Happy'? Our need to conceptualise and translate facial expressions into language is part of a natural learning process by which we attempt to understand the world. This process is innately reductive and biased due to the fact that we observe another person through only one perspective. The work examines how others see us and how this, in return, changes our behavioural responses. When we are told that we seem tired, angry, or sad the impact on our emotional response will vary depending on whether the feedback is produced by a human vs. a computer. Technologies that we design reflect our world views. The AI system used in this project is trained to recognise facial expressions according to a predetermined list of seven human-defined primary emotions. Systems built on this premise estimate aspects of an individual's identity or state of mind based on external appearances through Eurocentric codes. This design brings to mind pseudo-scientific physiognomic practices, which are notorious for their discriminatory nature and must be addressed in AI-based computer vision algorithms. The comparison between AI and human analysis of facial expressions reminds us that the technology is far from maturing beyond its maker, and both humans and machines still have much to learn.

DETAILS OF THE AI WEARABLE DEVICE
The device used in Structures of Emotion ( Figure  2) is composed of a micro-controller that hosts an off-the-shelf AI algorithm, pre-trained to recognise a person's face and estimate the emotions: Happy, Sad, Angry, Disgusted, Fearful, Surprised, and Neutral. The algorithm used for this device is outsourced from the online GitHub community (Omar178 2022).
The micro-processor is attached to the performer's head and a tiny camera is attached to the performer's forehead. This allows a live stream that enables real-time emotion recognition processing from the first-person perspective of the wearer. When the algorithm detects a person's face, it estimates their apparent emotion and displays it on an LED panel attached to the performer's chest. It also announces the estimated emotion out loud through a Bluetooth speaker connected to the wearable device. If the person who stands in front of the performer smiles, the algorithm will likely estimate them as happy. The LED panel will light up with the sentence "YOU SEEM HAPPY," and this sentence will also be announced out loud.

DETAILS OF THE DEMONSTRATION
The online demonstration of Structures of Emotion takes place on an online video conference. It includes two performers who attempt to analyse a participant's emotions; One performer is aided by the AI device described above and the other performer estimates emotions through the perspective of a human being, by simply using their own organic senses (Figure 1). The demonstration lasts about 10-15 minutes with each participant. The performers observe the participant's facial expressions and attempt to recognise their emotions. The participant is welcome to respond or remain silent. As the performance culminates a conversation transitions the relationship by inviting the participants to communicate their emotions and then analyse those of the performers and of the machine.

CONCLUSION
A strictly visual approach to emotion recognition is a reductive interpretation of the sensory information required to analyse the complex neurobiological system of human emotion. AI's simplistic vocabulary and its reliance on appearance make it unreliable. With that, it is essential to ask whether or not a human's interpretation of the emotions of others is any more accurate. The art demonstration of Structures of Emotion is aimed at provoking a conversation regarding biases that are implicit in both the human and nonhuman gaze. It inspires a reconsideration of the framework by which we understand emotions through the body and with language. Lastly, the work speculates a transhumanist future in which a wearable AI enables access to multiple perspectives. In this case, we strongly advocate that emotion recognition algorithms should be designed ethically as tools that promote human agency and autonomy over our emotional states.