Exploring the Relationship Between Music and Emotions with Machine Learning

As by the nature of perception, music holds a great deal of power over the emotions of the listener. Yet, when music is visually represented, the emotions experienced by the listener are typically neglected. This is perhaps due to the technical and economic barriers associated with emotion state retrieval. However, recent advancements in the accessibility of EEG and machine learning technologies makes this challenge significantly more accommodating.


INTRODUCTION
As by the nature of perception, music holds a great deal of power over the emotions of the listener. Yet, when music is visually represented, the emotions experienced by the listener are typically neglected. This is perhaps due to the technical and economic barriers associated with emotion state retrieval. However, recent advancements in the accessibility of EEG and machine learning technologies makes this challenge significantly more accommodating.
Therefore, the project reported in this paper uses these emerging creative technologies to explore the relationship between music and emotionswith the goal of provoking new artistic mediums in which emotional responses to music are visually depicted. In development, the project creates a new interface for the retrieval of emotional states in EEG data that is both lightweight and user-friendly.

EMOTION MEASUREMENT
For the purpose of emotion state measurement, the project makes use of the circumplex model of affect proposed by Russell (1980). In which, it is argued that values of emotional valence and activation can be plotted as a model of affective experience. This allows investigations to be grounded in two linear valuesrather than complex features of discrete emotional states.
As mentioned, this project uses EEG as a source of primary data. This technology is non-invasive and is believed to contain identifiers of emotional states (Reuderink et al., 2013). With this, many recordings containing positive and negative emotional valence and activation states were taken. These recordings were then used to train a neural network to classify positive and negative valence and activation values in live EEG data. This was done by converting the EEG data into spectrogram images and performing image classification through a convolutional neural network (CNN). In the spectrogram, each colour band represents a different brainwave frequency bandwidth (i.e., delta becomes red, gamma becomes blue)enabling the network to differentiate between the bandwidths. Inside each colour banding, are the four electrode positions (TP9, FP1, FP2, TP10). This method was chosen as it inherently contextualises the EEG data in both the spatial and temporal domainsallowing the network to interpret physical aspects, such as hemispherical activation (Pane et al., 2019), and temporal aspects, such as spikes and fluctuations.
The valence and activation classifications are run in real-time, with live EEG data. Then, the generated values are plotted against the circumplex model of affect. This produces a prediction for the strongest emotion felt by a participantat that point in time.

NEURAL SCORES APPLICATION
As the primary goal of this project was to provide others with access to emotion detection software, for use in artistic projects, a user-friendly desktop application was created. This application handles all the data processing and neural net operations. It then transmits the classified valence and activation values locally over OSCto be used in any other creative program that can access this data format (e.g., Max MSP, Processing, Unity).  In addition to the valence and activation values, the application also outputs a colour representation of the current stateas according to the circumplex colour model (Figure 1). This provides easy access to smooth emotion colour gradients, in RGB format.

MUSIC-EMOTION VISUALISATION
As stated, the production of this application hopes to provoke new artistic mediums in which emotions are considered. In regard to this project, the neural scores application is used to capture the emotions experienced by participants as they listen to music. Ultimately, with the goal of creating a new medium for music visualisation, which depicts the emotional journey music causes in the listenerrather than the qualities of the sound.
Due to the real-time capability of the neural scores application; there are numerous ways in which the emotional responses to music could be visualised. In this case, two possible methods were explored: particle system painting and reactive scenes. Each approach entails its own specific benefits and uses. For example, the reactive scene could be used to visualise the immediate emotional effect of music in a live audio scenario. Whereas, the particle system could paint the emotional journey in its entirety.
As project is ongoing, new ways of visualising this relationship are still being investigated. It is likely the final output will continue to show the emotional journey that music provokes in its listener, similar to the particle system. This is where the heart of the project lies; in the creation of a graphical score of sensation, a musical score of perceptiona neural score of emotion.