A Social Timeline for Exchanging Feedback about Musical Performances

This paper describes work in progress on the development of a social timeline that aims to enrich feedback for people learning music within an online community. This interface is novel because it uses multiple layers for group discussions rather than the single layer found in websites such as SoundCloud. Multi-layered timelines are commonly used for audio and video editing, but have not been used for collaborative annotation of music recordings (to avoid confusion we refer to the timeline as multi-layered not multi-track which normally refers to multiple audio channels). We tabulate key features of 15 competing systems to justify our claim that the multi-layered social timeline is a novel design pattern for exchanging feedback about musical performances. 49 classical music students used a prototype of the system over ﬁve months to upload and discuss rehearsals of an ensemble piece which was then publicly performed. Data from this study is currently being analysed so only a preliminary indication of results is given.


INTRODUCTION
This section describes the social timeline, outlines the political and economic drivers that inspired the project, states the aim and objectives and describes the method used to evaluate a prototype of the software with 49 classical music students.The structure of this paper is given.

Description of the social timeline
The social timeline is a website that allows you to upload and annotate music.Music can be uploaded in several ways: using a free iPhone App; uploading a file from a computer's hard drive; or recording directly into a browser.Once the recording has been uploaded and transcoded, it appears as a waveform with controls for playing, pausing and scrubbing the audio (figure 1a).Users can select and comment upon a region of the waveform, which then appears as a coloured rectangle next to an avatar of the user.The coloured blocks represent sections of the waveform that have been highlighted and annotated.Pop-up comments are revealed when the cursor hovers over a block.

Drivers for this research
Budget cuts, reduced teaching hours and increased classroom sizes have decreased the amount of personalised feedback given to music students.Students may challenge or dismiss feedback from teachers or peers, because it is unsubstantiated by evidence, or too general.They may also be unable to remember feedback, or recall it incorrectly.Practising an instrument is a lonely pursuit and learners may feel isolated and unsupported, especially if they are not having music lessons, which are an expensive luxury in economically deprived areas.This research is funded by a large European project called PRAISE which is developing a social network for learning music supported by technologies such as audio analysis, gesture analysis and intelligent agents (http://www.iiia.csic.es/praise).The evaluation of the social timeline will address one of this project's research questions: 'Under what conditions can social media systems enrich peer and tutor feedback for music learners ?'.

Aim and objectives
The social timeline aims to enrich social feedback about music learning.It has four objectives: 1) to record, represent and share multimedia data about aspects of music that are relevant to performance and learning; 2) to enable constructive dialogues about specific elements of a performance; 3) to encourage learners to reflect upon their creative practice; 4) to encourage the social learning of music.

Method
The social timeline was built using an iterative, evolutionary approach where eight successive prototypes were tested and refined following feedback from users (table 2).49 classical music students used prototype 6 of the timeline for five months to upload and discuss rehearsals of an ensemble piece which was then publicly performed.Students were divided into five groups (with nine or 10 students in each group) and completed a reflective journal containing a 1000 word account of their experiences.The dataset from Prototype 6 is currently being analysed in two ways: 1) A thematic analysis of the reflective journals The reflective journals were hand coded into themes that were relevant to the research question: 'under what conditions can social media systems enrich peer and tutor feedback for music learners ?' Braun (4) calls this method a semantic thematic analysis, to distinguish it from a 'full-fat' grounded theory, which requires an analysis to be directed towards theory development.Themes were discussed using selected quotations from participants, both from their journals, and from comments they made on the timeline.Discussion referred to academic literature as well as accounts from practitioners and companies such as Pixar who have published accounts of how feedback can support creative processes.
2) A grounded ontology of feedback from the timeline comments All comments from the timeline were imported into NVivo which was used to carry out a line-byline analysis of 200 of the 1954 comments.This analysis created categories for 1) type of feedback, 2) community actions, 3) interactions, and acted as a frame to categorise the remaining 90% of comments.Categories were revised when comments did not fit existing categories.

Organisation of this paper
This section introduced the social timeline and stated the project's aim, objectives and method.Section two compares key features of 15 competing music learning and distribution systems and claims that the multi-layered timeline is a novel design pattern for collaboratively annotating musical performances.Section three describes how the system was built.Section five presents some very interim results, and sections six concludes the paper by considering potential strengths and weaknesses of the system.

REVIEW OF COMPETING SYSTEMS
This section uses an analysis of 15 competing systems (table 1) to justify our claim that the multilayered social timeline is a novel design pattern for exchanging feedback about musical performances.
Within these system we found three common design patterns for social annotation of music: 1) spatially separated single-track timelines and threaded comments (figure 2a); 2) single track timelines integrated with stacked comments (figure 2b); 3) symbolic annotation of musical scores (figure 2d).

DEVELOPMENT OF EARLY PROTOTYPES
This section describes how prototypes of the social timeline were tested and refined to align them with user needs.Initial testing focused on improving basic functions such as uploading audio, scrubbing the timeline and selecting regions of the waveform for annotation.
The social timeline was built using an iterative, evolutionary approach where eight successive prototypes were tested and refined following feedback from users (table 2).

Method for evaluating prototypes
Prototypes 1-5 were evaluated using semi-structured interviews and screen recordings which captured user activity and reactions as they performed tasks such as uploading and annotating audio.The emphasis of these early evaluations was on usability, interface design and identifying appropriate functionality.
Prototype 6 was used by 49 classical music students and the data is currently being analysed.The method is described above in section 1.4.Prototype 8 was evaluated by two viola players (trainee and teacher), two drummers (trainee and teacher) and two conductors (trainee and teacher) who recorded performances of their choosing to MusicCircle using the gesture capture interface.During these performances the teacher gave advice to the trainee and referred to a projected image of the system.Participants annotated their performances using the social timeline and took part in a group discussion where they described their experience, evaluated the strengths and weaknesses of the system and help to design the next iteration of the prototype.Performances and interviews were recorded and uploaded to the MusicCircle website.

Prototype 1: embedded SoundCloud timeline
A 'minimal viable prototype' was developed in three weeks using an embedded version of the SoundCloud timeline (figure 3a).User evaluations revealed several limitations when using SoundCloud in an educational context: 1) comments are restricted to a single point in time and you cannot comment upon a region; 2) comments cannot overlap and have to be at least eight seconds apart; 3) comments cannot be edited after they are posted; 4) audio cannot be attached to comments.SoundCloud also unexpectedly modified their API causing the website to become temporarily unusable.In response to these limitations we designed a bespoke user interface which would give us full control over the functionality and user experience.

Prototypes 2 and 3: spatially separated threaded comments
Prototypes two and three used threaded comments, which are a common design pattern in competing software (table 1).Users reported that it was awkward to relate between selected regions of the waveform and the threaded comments below.This may be a consequence of the 'split attention principle', where spatial separation of text and graphics forces users to engage in unproductive cognitive effort (20).

Prototypes 4,5,6: multi-layered timeline with integrated comments
Prototypes four, five and six abandoned threaded comments in favour of a multi-layered timeline which allows collaborative annotation of multiple overlapping sections of a performance (figure 3b).
One of the core features of the social timeline is the region editor which allows users to select and deselect sections of the waveform for annotation.Making the region editor fluid and responsive was deceptively challenging, and it took four major revisions to achieve a satisfactory result.
Users reported a delay between posting a comment and it appearing to other users, this latency made the system seem sluggish and broke up the flow of dialogues.To fix this we added pop-up comments which proved visual cues about recent activity and a comment preview when the mouse hovers over a comment block.Users reported that these changes made the system feel more responsive and created the sense of a community working together at the same time.

Prototypes 7,8: Video and motion capture
The social timeline can be used to annotate any form of time based media and we have used it with video and motion capture data (figure 1b).

RESULTS
Prototype 6 was evaluated by 49 classical music students who uploaded 72 Tracks over the course of five months.There were 1954 comments using the timeline and 151 Replies.44 students filled in reflective journals describing their use of the system.All tracks were audio only.The data is currently being analysed so we are only able to give a preliminary indication of themes arising from the reflective journals.
1.The social timeline makes feedback more specific.
2. Sharing work in progress is initially daunting.
3. Learning alongside peers helps personal development.
4. Feedback can re-frame the individual against the group.
5. Sharing work in progress can demystify creative processes.
6. Feedback is valued from authorities who are perceived to be qualified.
7. Feedback should be authentic, but authenticity is complicated 8. Feedback is challenging to give and to receive.

CONCLUSION
This paper described work in progress on the development of a social multi-layered timeline.We believe that our system has the potential to enrich feedback for people learning music within an online community.From an initial analysis of the results we perceive the following strengths and weaknesses.

Potential weaknesses
Learners can have strong negative reactions to 'the violence of feedback'(15) from others and perceived criticisms may be amplified by the echo chamber of a social network.For this reason we provide guidance about how to give constructive feedback, and have a report button, which alerts the development team to inappropriate comments.
Intrusive covert activity by government agencies (9) and leading social networks (18; 17) have made people sensitive to the perception of 'big brother ' monitoring of private online activity.In response to this concern, we provide clear terms and conditions, written in plain English, which explain that any data collected will be anonymised and may be published, but that ownership and control resides with the user.
The social timeline may disrupt existing teaching practices, processes and power structures and may be viewed as de-humanising social interactions.

Potential strengths
The social timeline delivers personalised, specific feedback in the form of an active dialogue, which is a recommended way to help students develop their creative practice (12).It also allows people to listen and reflect upon their musical performances as they develop over time, which may help them to develop their listening and analytical skills (13).
The social nature of the timeline allows students, peers and teachers to reach a shared understanding about difficulties within a performance, and implement different strategies to resolve these issues.The timeline is being built as part of a non-commercial research project, so it will be offered for free to learners who cannot afford music lessons.The participatory design process should result in a system which has a realistic chance of meeting the 'real world' needs of users.

Figure 1 :
Figure 1: The social timeline: a) prototype 6 with multilayered audio; b) prototypes 7 and 8 with video and motion capture

Figure 2 :
Figure 2: Design patterns for annotating music and video: a) spatially separated single-track timeline and threaded comments in YouTube and Vimeo; b) single-track timelines integrated with stacked comments in SoundCloud and IndabaMusic; c) stacked multi-layered timelines visualising chords in Hooktheory and media clips in GarageBand and FinalCut Pro; d) symbolic annotation of musical scores in i-Maestro and Vemus

Prototype 7
was used by 128 computing students who uploaded 61 screen recording videos of software they had developed.They made 294 comments and gave 38 replies.

Figure 3 :
Figure 3: Initial prototypes: a) Embedded Soundcloud timeline showing a teacher commenting on a student's performance; b) concept design for a multitrack timeline

Table 2 :
Versions of the social timeline and primary users for evaluation