Designing and Studying a Multimodal Painting Installation in a Cultural Centre for Children

We here discuss a research project involving the design of a multimodal painting museum installation for children and study this relating to experience and engagement. During an ongoing PhD research examining multimodal interaction with museum installations, an opportunity arose to develop an installation and study the interaction with it. The installation was developed with a focus on tangible media combined with a GUI, spurred by a key interest to examine the interaction, experience and engagement outcomes of tangible media combined with other modalities, in the context of physical interaction with digital information. An overview of the study carried out is presented as well as a number of questions the study explored and initial observations.


INTRODUCTION
Typically, science museums adopt a strong handson interaction approach, while traditional museums assign a more passive, observatory role to visitors.However, this is shifting as traditional museums strive to incorporate technology and allow for a more active role for visitors.But a lot of the technology that has been implemented in museums only results in frantic button pushing, or provides visitors with the equivalent of a multimedia CDbooklet, with a lack of engagement with the actual topic and/or confusion (c.f.[6,7]).This PhD project aims to examine installations considering their modalities, particularly tangible media and its inappropriate or appropriate integration with other modalities, the context/topic and target audience.The research focuses on how the use of particular modalities or combination of modalities relating to the topic and target audience, influence engagement and experience.
A call for submissions for installations to the ARK, a cultural centre for children in Dublin provided a vehicle to move our research forward.It presented the opportunity to build an installation with a range of multimodal elements building on the principles of physical interaction and to examine the resulting interactions.Key issues for the study are to understand how physical interaction can contribute towards a better experience, engagement with the content for visitors, and to social engagement with peers and other visitors.This has us take both the perspective of the designer and the evaluator.We have the inside view of why specific layouts, shapes, sequence of events, colours, materials, etc. were chosen, and whether these were hoped to encourage particular behaviours, interaction and engagement.Studying the interaction of visitors can then reveal how effective these design decisions were for the desired outcome.
The study examined how visitors interact with the installation as a whole as well as regarding its individual elements.We consider how the interaction modalities exploit different sets of skills and capabilities (i.e.manual dexterity) [2] and what this means for the users' experience and engagement.Comparing the visitor/user interaction with the designer's perspective of intended interaction, we might be able to identify where the installation has intentionally and unintentionally encouraged certain interaction, understanding and engagement.The design of this installation creates a physical interaction that mimics the real life action of painting.On the other hand, the installation also explores tangibles and actions that are not totally familiar to the audience, such as using wooden cards and a slot for them to be placed in as key activation and selection tools.

CONCEPT AND INTERACTION OVERVIEW
The concept was generated around the exhibition theme of 'Awakening Curiosity, exploring nature, biodiversity and the world around us'.The target audience for the installation was children approximately aged 5.A key aspect of our installation design was to support multimodal interaction.
Thus, physical and visual communication was included in the design specification.The basic concept is that children pick an animal/organism to paint by choosing from a selection of wooden tokens, shown in part A of figure 1. Inserting this into a slot in the table, the image chosen appears on the table and projected screen to colour in (part B and C of figure 1).The children paint the image using a physical paintbrush and paint pots (parts D, E and F of figure 1).When the child has finished painting, they remove the wooden token from the slot (part G).Their individual painting is added to a collection of visitors paintings brought together to make up the wing patterns of a butterfly on the projected and screen image (part H of figure 1).
The installation design adapted the idea of selecting a page in a colouring book to select a drawing to colour in, by selecting a wooden card with a drawing on it.The concept was developed to collectively involve visitors in the creation of a new species of butterfly from smaller user-generated paintings.This concept was chosen to highlight how our actions affect other living organisms and portray the relationship of how different organisms affect other living organisms.By involving users in the creation of content it was hoped that visitors develop a feeling of ownership.This was anticipated to create an engaging experience where visitors are involved in creating the content rather than taking only a passive role regarding content, as is the case with standard information access points or databases of images or information.

Design Approach
Throughout the process an iterative design process was adopted.After generating initial concepts based on the exhibition theme and a multimodal interaction, a concept was selected to develop and run an explorative session with adult participants using paper prototypes.Then, a medium fidelity prototype, shown in figure 2, was developed which was evaluated with 16 adult participants, in 3 groups.The prototype evaluation was carried out using a partial Wizard of Oz technique, meaning the touch screen reacted to a real paint brush, but colour selection and token selection were simulated by a facilitator changing the screen and projection output, manually.While adults are not the target audience, this provided insights into usability issues, potential social interaction patterns, and suggested necessary concept changes.A second medium fidelity prototype was developed taking into account findings from the first evaluation.This was evaluated with 2 siblings aged 6 and 9 in a lab setting, before building the final installation for the exhibition.

SYSTEM DESCRIPTION
The installation (see figure 3), consists of a screen projection, an interactive touch screen which is synchronised to show the same visuals, a physical paintbrush, 10 physical paint pots, a tangible token slot, tangible wooden cards and an ambient audio track of wildlife sounds.
Physical wooden cards with laser inscribed drawings representing animals/organisms to colour in are used in the installation.The slot and cards are designed to have a similar appearance by using the same materials, colours and laser etching, thus implying a connection between the  Initially, we intended to utilize video camera tracking of two paintbrushes, using IR LEDs on the tip of each paint brush.While this worked in principle, due to software constraints and available resources we reverted to using one paintbrush on a HP touchsmart screen.We knew from early testing that this works quite well, although it has the disadvantage that the screen cannot differentiate different brushes and the screen would pick up any object touching it, not just the paintbrush.The paint pots are fitted with pulsating IR LEDs, which are detected by an IR sensor in the tip of the paintbrush.Once the IR sensor detects which paint pot the brush has been placed in, it feeds this information to Java and then to Flash, which changes the painting strokes' colour.Furthermore, the colour of an LED on the paintbrush changes to the chosen colour.This hardware and software communication set up can be seen in figure 4. The LED on the paintbrush indicates the paint pot colour and is essential to provide feedback to the user while simulating the paint on a real paintbrush.

RESEARCH QUESTIONS AND INITIAL FINDINGS
Meanwhile, the installation has been exhibited at the Cultural Centre for Children for 3 months during schools tours and public opening periods over the summer.We have carried out observational work and some video recording during both school tours and public opening times.A small number of school groups were observed at the exhibit (given their time limitations and low numbers per day) thus the main discussion will be based on the observations of the general public.
An element we chose to investigate was whether visitors immediately understand how to use the installation (i.e.'immediate apprehendability' [1]) Will people understand what to do with these physical objects (1) paintbrush and paint pots and (2) wooden cards and card slot in the table's surface?What will they do with them?Are there unexpected behaviours in using these physical tokens?Immediate apprehendability was present for interaction with the paintbrush as children were continually observed instantly picking up the brush and then trying to start painting.Upon doing so a prompt animation would appear on the screen showing an outline drawing of a person picking up a wooden card and putting it in the slot.A number of different actions occurred at this stage.(1) Many children understood the prompt and chose a card, then inserted in it the slot, (2) children would keep trying to paint while siblings or parents understood what to do and communicated to choose a card or they would choose a card together, (3) a parent or sibling/friend would read out written instructions placed at the top of the table, (4) if floor staff were present they would talk them through what to do.From the observations, it was evident that selecting a card to insert was not an obvious connection for visitors when they first approached the exhibit.The cards also are not the most enticing familiar objects of the exhibit, as children are initially drawn to the paintbrush.When children notice the paintbrush and pots it is apparent they understand the exhibit will involve painting and are motivatedby this.The exhibit thus supports visitors in understanding what the exhibit may involve and it's subject matter before interacting with it.However, this pattern of events provided an opportunity for children to embark on experimental learning, for social interaction and inclusion of others in the experience.
Upon inserting the card initially, many would remove the card from the slot immediately and  The majority of children completely took the illusion of the paintbrush being the sole control to enable painting.Even if they knew something else touching the screen would paint they continued using the brush, possibly, because they could only choose colours with the brush.They used the brush as an object and explicit mediator of control, handing it over to another person, or when other children were at the table waiting for a go, or siblings directly handed the brush to their sibling.Also, the key position of power/control is at the seat, children moved into this position while painting and others moved closer to it to as they waited for their turn.
Furthermore we are interested to see what social acts or communication emerge around these physical tokens.This is important because museums are a place of social interactionmuseum visits often are done in groups, either by families or as a school field trip [4].We thus want to know what aspects of the installation encourage social interactions among individual visitors and groups or discourage it and how?As a generalization, the exhibit highly promoted social interaction and included individuals who were not directly painting.In particular the use of wooden cards and the projected image showing the current painting encouraged interaction and included others in the experience.Parents, siblings and friends would verbally communicate with the child painting from observing at a distance the projected image or staying beside the table to help select colours, paint white over mistakes or praise the painting verbally.Occasionally, the painter would initiate conversation for example, asking what colour to choose or prompting others to observe their painting.
The wooden cards provided a reference for visitors to communicate around [8], children would take a card while not using the exhibit.They would bring the card over to the bench, show others what they intent to paint, hold it until they had their go using it as an expression of their intent, moving it in front of the projected image or staying at the side of the table with it in hand.The exhibit allowed others to choose what they would like to do and plan by browsing through the wooden cards while another child painted.The painter rarely showed any signs of disruption during this.Fernaeus and Tholander identify the ability to work offline and parallel as important qualities for users' interaction with technology [3].Similar as observed in [3], we saw how the wooden cards provide children with the ability to re-arrange these, draw one's attention to something using a physical reference while physically relocating oneself and plan ahead individually and together in parallel to the action of painting.Fernaeus and Tholander argue these elements contribute to increased social interactions and allow children to "act individually as well as collectively" [3].Furthermore, we want to know how the physical set up of the installation supports the visitors experience and social interaction.Are other visitors included, even if they are not painting?The table layout, size and height was designed considering the social interaction, target age groups ergonomics and anthropometric data and the physical limitations of the hardware.[9,10 Paint pots were dispersed on either side of the table so as not to exclude one side from being closer to the interactive screen.The tabletop is tilted slightly to allow visitors from all sides of the table to approach it and observe the interaction, while still implying a key position at the table for interaction.It can be assumed that children identified a clear control position at the installation as all but two children were observed locating themselves at the table where the slot was, in front of the chair placed at the installation.The layout of objects on the table allowed others that were not painting to explore the paint pots and cards without interfering with the painter.The painter was able to protect their card from being removed from the slot by others.We noticed younger siblings trying to remove it while others were painting and either the older sibling or parent holding it in place.
From an initial brief observational study with school groups that were led around by a tour guide we could see that visitor interaction is influenced by the directions and interpretations provided by the tour guide and teachers.Similar effects were found by a study carried out by Katriel looking at guided tours.[5] School groups were shown what to do with the installation.The observational study of school groups revealed positive social interaction among the students, encouraging the painter while they are not painting.However, teachers and guides occasionally need to ask children to wait for their turn.This indicates that without any supervision possible confrontations may emerge along with less outgoing students being somewhat excluded.As similarly observed with public groups, children picked up the cards and said to the guides and teachers "I want to put this one in next."They seem to be using the cards to indicate their intended actions.
Approximately only a third of visitors observed realized that the image was added to the overall butterfly and showed an interest in this.After a child finished painting and removed the card from the slot there was little to no time for them to reflect on the overall butterfly pattern and their painting.Typically, another card was inserted immediately, thus zooming in on one image to paint, or somebody touched the screen, stimulating the prompt animation to appear which covered the overall butterfly pattern.This affected people's understanding of the individual paintings relation to the overall butterfly.It also prohibited people from reflecting on their input.Many would realise their image was up on the butterfly, but once another person started to paint they were not able to view the overall butterfly.An integral element to exhibits is to support further interest and reflection.However, the installation hinders this by not providing an overview of the final butterfly pattern while somebody is painting.
Children commented that they liked to see what they were painting up on the main projected image as well as the table screen.They also pointed at the main projection showing it to others.During painting children used the projection for an overview when they were choosing a new colour or finished painting a section they would look up at the overall projection.On rare occasions painters would watch the projected image while painting.
From initial findings it is clear children are highly engaged with the exhibit.But what exactly they are engaging with in terms of their understanding of what the exhibit is about is to be further explored using video analysis.At this stage it appears the exhibit is about painting for visitors and less about creating a butterfly pattern collectively.It was rare to see visitors reflecting on where their image was on the butterfly or expressing they were adding to the pattern without floor staff prompting such thoughts.

FUTURE WORK
As the analysis of data is at the initial stages any questions brought up need to be further explored by analysing the observational notes in greater detail along with the video data captured.
It was hoped that a comparative study with an altered installation based solely on touchscreen interaction could be carried out during the exhibition.However, this was not feasible for organizational reasons.We hope to be able to conduct a brief comparative study in the future to investigate how the tangible objects affect the interaction with this exhibit in comparison to a solely screen based interaction.

Figure 3 .
Figure 3.The final installation in the Ark