Grumari : Digital storytelling

This project is part of a broad research investigation approaching questions related to design and the complex context of media focused in hypermedia languages. This project, aims to make a series of digital storytelling media available for deaf and hearing children, such as interactive ebooks thematically related to the preservation of the Cultural Heritage in the city of Rio de Janeiro, Brazil. This research is intended to generate materials for professionals of Design, Arts and related areas, for developing and producing storytelling using new digital languages. It is intended that this will widen the space for diffusion and cultural exchanges in and between artistic languages, in the following knowledge areas: visual arts, audio-visual media and music for digital narratives. In this paper reflections on the process of creating the digital storytelling artefact called Grumari will be presented, from the structure of the story, concepts of visual language, storyboard, and video, to considerations for music and sound design.


INTRODUCTION
This paper discusses a research project that was supported by an award to Professor Cristina Portugal, of a Post-doc. scholarship (2017-2018) from CAPES Foundation, an agency under the Brazilian Ministry of Education, in order to execute part of a research as a Visiting Scholar at the Royal College of Art.This research has the intention of developing two complementary studies, a theoretical research investigation and an experimental one.This paper will present the experimental study.The research project starts from the experience of the author during the development of a project called 'Design and contemporary digital technologies applied in the development of an interactive digital book for deaf and hearing children', which was founded by Public Call MCTI/CNPQ/Universal 14/2014.This project, aimed to make available a series of digital storytelling for deaf and hearing children, such as interactive digital books whose theme was the preservation of the Cultural Heritage of the city of Rio de Janeiro.
The first book of the series was presented as an paper with the title 'Design and Narrative for Deaf and Hearing Children', at the EVA London 2015 conference.The creation process of the second book from the series of books to be produced, named Grumari, will be presented in this paper.
The digital history project entitled Grumari allows us to understand the relationship between knowledge about the needs and potentialities of deaf children and the creation of digital books, which can stimulate their development and, consequently, facilitate their inclusion in society.Starting from the principles discussed in Kerckhove (1998), that each technology extends one of our faculties and transcends our physical limitations, we want to get the best extensions of our body.
The goal is to develop an e-book where the narrative will be made by multiple languages available in hypermedia systems.Digital narratives in this study prioritise the use of images for presenting a story.This is based on the precepts of the author for building the children's interactive digital narrative, which must take into consideration, besides the story text, images and the interactive technological resources.
The methodological process is based on studies of Janet Murray, Henry Jenkins, Johanna Drucker, Richard Williams and PIXAR Studios.The stages that will be presented are the developments of: story structure; character; visual language; video; sound design and music.
It was considered that hypermedia is a tool capable of exploring the senses of sight, hearing and touch, allowing the presentation of information in an affective and meaningful way.Thus, hypermedia is configured as a language because it has constituent elements, such as: static images, moving images, sounds, texts and hypertexts.These also have relationships between them, generating characteristics of hybridisation, interaction, navigation, multiplicity and nonlinearity.Therefore, it is a wide field for the research and creation of projects involves work in the field of Design in partnership with Art and other areas.The narratives with image, sound and written text complement each other and allow the reader to oscillate between one and another in order to understand the story.In the children's e-book the images, the written text, the sound and the design elements have an integrated relationship, where the meaning of the whole is greater than the sum of the constituent parts.Images tell what words cannot tell and vice-versa, because isolated elements are not able to tell the whole story.
The study presented here is the result of research performed by researcher Cristina Portugal who, since 2009, has been investigating design in the complex context of media, focusing on hypermedia languages.Aiming to apply those languages, Cristina has been developing, besides research about theoretical and aesthetic concepts for digital environments in the light of design, digital narratives together with her team.
During investigations made previously by Portugal (2009Portugal ( -2017) ) questions regarding acquisition of language were discussed, as were references to pedagogy; education sources for deaf children; studies of objects similar to the digital book; studies of possible interactions for the e-book for deaf and hearing children; and others.In this paper, as an unfolding of this research, reflections about the process of creating the digital narrative named Grumari will be presented.

CONCEPT OF DIGITAL NARRATIVE
When we create a story aiming to include deaf and hearing children, the first question that we think about is how to deliver the narrative and made it available digitally.We ask what makes digital narratives different from other ways of telling histories?The answer can come from several authors, however here we highlight the definition of Murray (2017), who considers the digital narrative as a continuation of the old tradition of telling stories, in a new media.In the same way that innovations happen in other media such as: language, writing, printing and new record technologies, new resources of inscription and transfer lead to new possibilities of representation.The author argues that, in the case of computing for genres beyond the narrative, we may identify four resourcesthe procedural, the participative, the encyclopaedic and the specialwhich, together, were taken into account for creating the digital narrative of Grumari.
The digital narrative may potentiate human intelligence according to Murray (2017) in the same way that happened with the invention of printing.This author elucidates the importance of studies regarding ways that representation can occur in digital media, including both the visualisation of information as well as other interactive aspects.
The author also adds, that the digital narrative must be taken as a multiple term instead of the less commonly used term of 'non-linearity', adding that we have the possibility of creating more complex story structures in this new medium, instead of only breaking traditional formats.
Taking into account what was exposed through this reading of relevant discourses, we opted to create a digital narrative for the story that was written as a way of communicating the importance of preserving the natural reserve of the Atlantic Forest, in the region named Grumari.
The digital narrative may still be conceptualised as a term used for stories told using technology.In other words, digital stories are multimedia presentations combining a variety of communicative elements inside a narrative structure.
The categories of narrative structures include: linear narrative; nonlinear narrative; interactive narrative; and graphic narrative.Media may include any combination such as: text, image, photo, video, audio, animation, elements of social network and/or interactive elements.
The interactive elements allow us to be actively involved with fictitious worlds as we experiment with them.Those new models of narrative indicate nothing less than the advancement of our culture, because they work to help us better understand the contemporary world.Burdick et al. (2012) presents relevant questions in the area of network information.
We live in one of those rare moments of opportunity for the humanities, not unlike other great eras of cultural historical transformation such as the shift from the scroll to the codex, the invention of the moveable type, the encounter with the New World, and the Industrial Revolution.Ours is an era in which the humanities have the potential to play a vastly expanded creative role in public life.(Burdick et al. 2012, p.vii).
Those concepts about provision of information by means of digital narratives were discussed, in depth, by the development team of this project.The next section will present the process of creating the digital narrative named Grumari.In order for this narrative to communicate effectively, a study was undertaken regarding the theme of the story, possible media to be used, and also, an investigation of questions regarding the target audience, deaf children (not exclusively).Considering the complexity of the project, in terms of both technological aspects, as well as the use of the Brazilian Language of Signals (LIBRAS), Portuguese and/or English, it was decided to first create a prototype as an animation (or 'teaser') because this provided a fast and economic way of defining and experiencing the digital book with deaf and hearing children.

REFLECTIONS ABOUT THE CREATIVE PROCESS OF THE DIGITAL NARRATIVE
The methodology was based on previous studies of Portugal (2009,2013,2015,2017).This essay expands the paper of Portugal (2017), named 'Design and Visual Arts for Digital Literature' presented previously at EVA London, where the full methodological process may be found.In short, the stages focused on: reading and analysing the text about digital narratives, visual language, UX and sound design.This project had a multidisciplinary team composed by designers, artists, illustrators, educators and music, aiming to discuss in a deeper way each one of the production stages of the project such as briefing, script, narration, motion design, music and sound design in digital environments.
In order to start the production of the story, the following question was considered: what makes a story to be interesting?According to Pixar there are elements that must be verified during the creation of a narrative to make it attractive.They are:  Story: a sequence of events that unfold through time. Perspective: a point of view or a way of seeing the world. World: the environment or set of rules where a story takes place. Character: the subjects or individuals we follow on the journey of the story. Protagonist: the main character(s), whose journey the story is about.
The next section the development process of the story named Grumari, taking into account authors such as Lambert, Janet Murray, Henry Jenkins, Richard Williams, among others, as well as the model of PIXAR Studios.

The story
The story was born during a tour in the region of 'Grumari Park', a reserve of the Atlantic Forest in the city of Rio de Janeiro.When we came across a resplendent and natural region, which is threatened by real estate growth, we started to idealise a story aiming to warn children about the importance of preserving the natural heritage of the city of Rio de Janeiro.
The narrative according to Brunner (1990) involves the knowledge, identity and rationality as people build their understanding of the world around them, the comprehension of themselves and their dialogue with other people.The narrative presents the meaning that people build "to themselves".In face of what was exposed, the question presented as the central theme of this storypreservation of the Atlantic Forest in the region of Grumari, city of Rio de Janeirobecomes more relevant because of the report published in 2/8/2018 in "Globo online", denouncing the scheme of irregular allotments in the region of Grumari.

Synopsis of the narrative:
In a reserve of the Atlantic Forest there is a civilization where very evolved beings live, a human community that developed having contact with nature only.The leader of the community has a daughter called Gaia.The girl loves to explore the tracks with her friend Buri.
Everything could be perfect in the Village, there life has unity, and the exuberant nature remains preserved.But in a fortress, in the summit of one of the more high mountains, lives a sorcerer called Bagual, who has the elixir of immortality; The ingredients needed to produce the elixir are obtained from the acid atmosphere of the cities, which does not exist in this paradisiacal place.
Considering the hindrance that (….) seems to be for his plans of immortality, Bagual tries to transform the reserve in a city with high towers of buildings, in order to finally make him an immortal being.

Structure of story
Every story told has a base or structure.The structure of a more basic story is "start, something happens and finishes".However, it may be complex, for instance, according to Lambert (2006), in consonance with PIXAR there are seven elements of a digital narrative.They are: 1. Point of view: Outlines the point of the story and the perspective from which the story is told.2. A dramatic question: Sets the tension of the story by identifying issues to be resolved.3. Emotional content: Engages the audience through common emotions and themes (love, pain, humour).4. The gift of your voice: Helps the audience to make meaning of images.5.The power of the soundtrack: Sets the mood of the story.6. Economy: Balances the auditory and visual tracks of meaning.7. Pacing: Sustains the attention of the audience by establishing and modifying the rhythm of the story.
Based on concepts mentioned above from PIXAR, we conceived the structure of the story for Grumari: 1. Once upon a time there was a sorcerer who had the formula of the elixir of immortality.2. Every day the sorcerer harvested the plant to make the elixir 3.One day he perceived that the plant was going to be extinct.4. So the sorcerer thought in a plan to avoid the plant of being extinct. 5.Because of that he went to the leader of the city in order to explain his evil plan.6.Until finally the sorcerer discovered that he could not go against nature.7. Since them, the Atlantic Forest remains preserved.

Concept
For creating the characters and scenarios a series of studies about the region of Grumari was made, about the time when the story happens, dresses, architecture, place, vegetation and other studies.
The characters are the beings that we follow during the trajectory of the each story.However, in order for characters to not only be generic ideas, it is necessary to create physical and emotional characteristics in order to give life to each one of them.
We defined that the story should have four main characters.The protagonist would be the sorcerer named Bagual and the other characters: the leader named Argo, his daughter Gaia and his friend Buri.
For each one of the characters external characteristics were defined.These characteristics included: dresses, designs and appearances; internal characteristics, such as personality and beliefs; and the desire that motivates the character to act.It was also necessary to establish the importance of each character, in other words, the demand, something that each character should make or learn in order to succeed, grow, fail or die.A session for generating ideas was performed with the creation team, and the characteristics for each character were defined, from the physical until the emotional aspects.

Visual language
Visual language refers to the way that images, symbols, icons and others are used to transmit ideas about the meaning of the story.Perspective, colour and shape may be used to represent a story aiming to guide the public to see, feel and understand the climate of the story.
According to Sudjic (2010, p.21), the role of the more sophisticated designers, today, is both being storytellers, making a design that speaks in a way that transmits those messages, as well as of solving formal and functional problems.For Sudjic (2010, p.50), it is the language of design that serves to suggest the gender of an object, many times by the less subtle mean, by colour, by size and by visual references.
Supporting this idea is Drucker (2014), who argues that this overview of approaches to formal principles of visual communication only skims the surface of a rich history.But the survey demonstrates the existence of carefully thought out foundations in which visual forms of knowledge can be understood.The systematic analysis of 'graphical language' remains crucial; its principles are the fundamental basis of the graphics.But they are not its end goal, which is the analysis and imaginative production of visualisations, visualised interpretation, and graphical user interfaces.(Drucker 2014, p.53)The communicative function of Design in digital environments demands familiarity with the formal rules coming from graphic compositions and from the way of organising visual, verbal, sound and kinaesthetic contents in systems.As discussed in Portugal (2013), design lends the hypermedia language syntaxes and visual values aimed at communication.
In face of what was exposed through means of sketches and the storyboard, issues about composition were also addressed: line, space, colour, tone, movement and shape; the visual language for visually representing the concept of the history was thereby defined.

Storyboard
Storyboard is a story telling technique by means of sequential frames using simple designs.It is a fast way of drafting all elements of a story, from structure until composition, according to the need of iterating and improving.Each frame of the storyboard represents a real scene of the animation, framed as it appears in its final presentation on the screen.
For developing a storyboard, the budget must be taken into consideration, as must deadlines for delivering the project and the development team of the storyboard.What must also be taken into account is the speed and ability of the team or illustrator in drafting.Thus, a choice can be made to either opt for an extremely detailed elaboration, presenting complex actions by showing each pose of a character, or, it is possible to opt for a more simplified workflow, ensuring that all main actions are shown.Here choice made was to develop a simple storyboard, however with all scenes needed to sufficiently develop the animation.

Video (animation)
Based on the storyboard, a story was developed, which readily utilised principles of video production such as: camera angles, edition, composition and movement of the camera and characters.
As the intention is to develop an interactive book, the option adopted by the project was to make first an animation (video-trailer) based in the concepts of Richard Williams (2009).The video has unique characteristics that make it more rich, interesting and complex than the other media.Thus, the animation, narration and sound were edited in a single video aiming to create a first prototype of the digital narrative as material for disclosure until the digital book is finished.For editing of the video some steps discussed by Drucker (2014) were considered: Because web environments are dynamic, it is tempting to take the basic language of motion picture editing and create analogies for each kind of shot (close up, establishing, tracking, detail, mid-range, pan, following, and so on), or transition between shots (cheat cut, parallel edit, cut away, dissolve, iris, jump, superimposition, wipe) match across shots (viewpoint, action, motion, scene, wipe, shot-reverse-shot, dissolve, jump-cut, etc.), or duration (long shot, overlapping, elliptical, simultaneous).But to reiterate, film editing relies on narrative theory, not just on visual principles of perception, and on the principles of temporal change, motion, animation, and dynamic graphical means are essential to its production.Web environments force cognitive processing across disparate and often unconnected areas of experience and representation.They frequently require multimodal processing of varied media (Drucker 2014, p.47).
All types of languages: image, text, audio, are mixed in the video in order to transmit a common message, building significations, transporting representations and spreading symbols.

Music, sound design and narration
In the first digital book integrating the series of books, which was presented in a previous paper of Portugal, C. et al. (2015) for the EVA London 2015 conference, there was no audio implementation.
However, the main intention of this project is to support the inclusion of deaf and hearing children in society together.If we want to include we cannot exclude.We therefore seek to set up an inclusive object, where disability is seen as difference and not deficit.So, audio was an important part discussed in this research project due to its favourable characteristics when applied in digital environments.A way of including narration, written text, video in LIBRAS, music and sound design that could support both the needs of deaf and hearing children was sought.
Audio has, as a characteristic, the power of intensifying the process of immersion and, in consequence, allows a greater involvement by the user of the media, making the user experience more interactive, developed and attractive.Audio is a powerful form of media and its use in digital environments must be considered both as an element of browsing as well as of immersion.The experience together with image, text and sound becomes complex and integrated for the user of the animation, and may impact the distinct senses of the spectator making use of only one artefact of communication.
For the creation of the music and sound design, a professional composer had to be part of the multidisciplinary team.New ideas were generated and experiments were analysed, for developing the music and sound design of the digital narrative, as presented in Portugal (2017).
Musician Felipe Alram composed the music for Grumari.The musician used a workstation of digital audio (Digital Audio Workstations [DAW]), which is a sequencer allowing recording, editing and playing digital audio.An upload of the music named Grumari was made in the DAW in order to start the process of composing the music, the sound design and the narration from the scenes were already animated, so the aim was to match the music to the content of the story and also, to have a meaning in itself by using the sound as a language for transmitting messages and provoking feelings and emotions.Figure 1 presents a screenshot from the DAW as used during the editing of audio of the Grumari animation.The animation, text, music, sound design and narration are already finished.However, the video of texts in LIBRAS (Incluir Tecnologia 2011) had not yet been recorded.In order to provide an example, we used a generic video found on the web.Given the complexity of the history, in order to make it available in digital format, the choice was made to make this prototype a faster and more economic way of defining and experiencing the project.

CONCLUSIONS
The research discussed in this paper is part of a project named 'Design * Technology: Contemporary design in digital environments', that is being developed in the School of Communication of RCA in London, with the support of Capes Foundation.This paper has presented reflections on the process of creating the digital narrative named Grumari.The aim is the development of a Book in bilingual digital format (Brazilian Sign Language --LIBRAS and Portuguese), which will help the process of reading for deaf children, but not exclusively.
The discussion take took place during the development process of this research project looked not only for a means of creating digital narratives, but at a wider debate regarding design in the complex context of media.Santaella (2013) believes that the changes this context presents relate to the way that we construct human languages.So, we started to think about what is relevant for members of the public, considering first the message and, then how to communicate it in the best way, depending on the media.Creating relevant content with purpose of making it available through the given medium is more important than focusing on the medium itself.The convergence of communicative approaches allows us to choose how and where we want to consume content as well as which types.It makes sense to make these technologies available inclusively for deaf and hearing children.
Back in 1996, Bill Gates was already approaching this question.To him, the advancement and democratization of hypermedia technologies would change communication and the way by which we consume it; and it would be essential to understand the possibilities for creating content considering the user experiences that are supported by interactivity.It was not only about virtual environments increasing the reach of the contents, disseminating content more quickly and at lower costs.But, it was also about the transformation of the relationship between user and content, as mediated by the type of hypermedia used and its technological resources (hyperlinks, videos, gifs, animations), which provide forms of interactivity.In other words, a new way of thinking should be born from this change, bringing the user, the content and the type of interaction to the centre of discussions.

Figure 1 :
Figure 1: Screen of the project of music from Felipe Alram in DAW.

Figure 2 :
Figure 2: Prototype of e-book of Grumari.The prototype of layout shown in Figure 2 displays one of the screens of the e-book in a digital device.