Graphs in Harmony Learning: A.I. assisted VR application

A development of learning technologies for distance education explores new possibilities of Virtual Reality (VR) and Artificial Intelligence (A.I.) application. The current state of art in development of A.I. assisted teaching agents is focused either on visual recognition, aiming to provide responsive systems to the learner’s implicit feedback (Lino, Rocha & Siza 2019; Zhang, Xia & Lim 2019), or on smart campuses ecosystem creation, with a heavy reliance on Chabot systems (Villegas-Ch, AriasNavarrete, & Palacios-Pacheco 2020).


INTRODUCTION
A development of learning technologies for distance education explores new possibilities of Virtual Reality (VR) and Artificial Intelligence (A.I.) application. The current state of art in development of A.I. assisted teaching agents is focused either on visual recognition, aiming to provide responsive systems to the learner's implicit feedback (Lino, Rocha & Siza 2019;Zhang, Xia & Lim 2019), or on smart campuses ecosystem creation, with a heavy reliance on Chabot systems (Villegas-Ch, Arias-Navarrete, & Palacios-Pacheco 2020).
In such a context, music theory teaching in a high school context should respond to a new demand of the society and this paper presents a VR application for music harmony learning with A.I. assistant, which fills the gap between traditional approaches in teaching music theory and an existing technological potential.

METHODOLOGY
A VR application for music harmony learning is based on the original methodology of diatonic harmony representation, which has shown its efficiency in experiments held with students during 2013-2015 (Shvets 2019). The methodology consists of the system of graphs and an augmented score representation of harmonic sequences. The system of graphs is organised in horizontal and vertical triads, reflecting logical relations between chords, whereas the augmented score contains a meaningful colour scheme for the chord functions' visual distinction, along with colour shades, revealing the chord structure.

VR APPLICATION
All listed elements of the methodology receive a tangible 3D representation in the VR context of the application, allowing multimodal interaction with the user. The content of the application is divided into three VR spaces -the "Entry Room", explaining the purposes of the application, the "Temple of Knowledge", which prepares the user to understand the content of the main activity, and the "Study Room", where the interaction with the system of graphs and augmented scores occurs (Figure 1).

A.I. ASSISTANT
Another innovative feature of the application is its A.I. cloud assistant, which gathers the information of the user activity and generates the content for tests based on the received data. The content generation is made via deployment of the pretrained A.I. model on a remote server.

Architecture
Communication of the device with the A.I. model is made via Flask framework with the data transferred in a JSON format -the VR application sends two chords from the sequence of a graph chosen by the user for testing and receives a third chord, generated by the model, as a response. The VR application then plays the corresponding sequence, asking the user to choose between the options in a graph (Figure 2). If the user chooses a correct graph slot, the chord is rendered in an augmented score representation (Figure 3).

Generative model
The generative model is based on long short-term memory (LSTM) deep neural network architecture, built in Keras framework. It consists of one Embedding layer, two LSTM layers and two Dense layers. The model input requires text format and outputs textual data as well.

SUMMARY
Summarising, this VR application supports learning process with augmented tangible 3D objects of chords, a system of meaningful colour scheme (for a chord function) and colour shades (for a chord structure), and proposes a level of abstraction in a form of the system of graphs, which puts in evidence the relations between chords in a given tonality. The A.I. assistance is used to generate new content for tests, rendered by VR application. Thus, the VR context together with A.I. models allows realising an activity-based approach in music harmony learning with a high degree of immersion, which contributes to the development of a music ear and to the enrichment of the learners' cognitive experience.